I built something similar awhile back [1] and used OpenAI’s tokenizer playground [2] to recalculate tokens on a giant block of lorem ipsum text. I feel like thi

I built something similar awhile back [1] and used OpenAI’s tokenizer playground [2] to recalculate tokens on a giant block of lorem ipsum text. I feel like this gives a much more accurate representation. [1] https://dave.ly/tools/tokenflow/ https://dave.ly/tools/tokenflow/ [2] https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer