8 ms·
Couldn't we just make every human readable character a token? OpenAI's tokenizer makes "chess" "ch" and "ess". We could just make it into "c" "h" "e" "s" "s"
by skylerwiernik 2y ago
Couldn't we just make every human readable character a token?
OpenAI's tokenizer makes "chess" "ch" and "ess". We could just make it into "c" "h" "e" "s" "s"
- taeric 2y agoThis is just more tokens? And probably requires the model to learn about common groups. Consider, "ess" makes sense to see as a group. "Wss" does not. That is, the groups are encoding something the model doesn't have to learn. This is not much astray from "sight words" we teach kids.
- TZubiri 2y agoThis is just more tokens? Yup. Just let the actual ML git gud
- taeric 2y agoSo, put differently, this is just more expensive?
- TZubiri 2y agoExpensive in terms of computationally expensive, time expensive, and yes cost expensive. Worth noting that the relationship between characters to token ratio is probably quadratic or cubic or some other polynomial. So the difference in terms of computational difficulty is probably huge when compared to a character per token.
- Hendrikto 2y agoNo, actually much fewer tokens. 256 tokens cover all bytes. See the ByT5 paper: https://arxiv.org/abs/2105.13626 https://arxiv.org/abs/2105.13626
- tchalla 2y agoaka Character Language Models which have existed for a while now.
- cco 2y agoWe can, tokenization is literally just to maximize resources and provide as much "space" as possible in the context window. There is no advantage to tokenization, it just helps solve limitations in context windows and training.
- TZubiri 2y agoI like this explanation