6 ms·
This is a very tiring criticism. Yes, this is true. But, it's an implementation detail (tokenization) that has very little bearing on the practical utility of t
by bglazer 9mo ago
This is a very tiring criticism. Yes, this is true. But, it's an implementation detail (tokenization) that has very little bearing on the practical utility of these tools. How often are you relying on LLM's to count letters in words?
- 101008 9mo agoIt's an example that shows that if these models aren't trained in a specific problem, they may have a hard time solving it for you.
- victorbjorklund 9mo agoNo, it is the issue with the tokenizer.
- Uehreka 9mo agoNo, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like searching a codebase for variants of a variable name in different cases).
- 8note 9mo agothe question remains: is the tokenizer going to be a fundamental limit to my task? how do i know ahead of time?
- worldsayshi 9mo agoWould it limit a person getting your instructions in Chinese? Tokenisation pretty much means that the LLM is reading symbols instead of phonemes. This makes me wonder if LLMs works better in Chinese.
- altruios 9mo agoAn analogy is asking someone who is colorblind how many colors are on a sheet of paper. What you are probing isn't reasoning, it's perception. If you can't see the input, you can't reason about the input.
- 9rx 9mo ago> What you are probing isn't reasoning, it's perception. Its both. A colorblind person will admit their shortcomings and, if compelled to be helpful like an LLM is, will reason their way to finding a solution that works around their limitations. But as LLMs lack a way to reason, you get nonsense instead.
- deleted 9mo ago[deleted]
- altruios 9mo agoWhat tools does the LLM have access to that would reveal sub-token characters to it? This assumes the colorblind person both believes it is true that they are colorblind, in a world where that can be verified, and possesses tools to overcome these limitations. You have to be much more clever to 'see' an atom before the invention of a microscope, if the tool doesn't exist: most of the time you are SOL.
- andy99 9mo agoAt this point if I was openAI I wouldn’t bother fixing this to give pedants something to get excited about.
- properbrew 9mo agoUnless they fixed this in 25 minutes (possible?) it correctly counts 1 `r`. https://chatgpt.com/share/6941df90-789c-8005-8783-6e1c76cdfcd2 https://chatgpt.com/share/6941df90-789c-8005-8783-6e1c76cdfc...
- iAMkenough 9mo agoThe criticism would stop if the implementation issue was fixed. It's an example of a simple task. How often are you relying on LLMs to complete simple tasks?
- 1970-01-01 9mo agoThe implementation detail is that we keep finding them! After this, it couldn't locate a seahorse emoji without freaking out. At some point we need to have a test: there are two drinks before you. One is water, the other is whatever the LLM thought you might like to drink after it completed refactoring the codebase. Choose wisely.