4 ms·
The current gen llms tokenize numbers digit by digit unlike earlier llms.
by isaacfung 2y ago
The current gen llms tokenize numbers digit by digit unlike earlier llms.
- Last5Digits 2y agoThey don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.
- mike_hearn 2y agoNo, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.
- singularity2001 2y agoIf someone else is interested in the Bloomberg tokenizer: https://medium.com/generative-ai-insights-for-business-leaders-and/turning-text-into-data-how-bloomberggpt-tokenizes-financial-language-c17679fb72d0 https://medium.com/generative-ai-insights-for-business-leade...