my group is currently working on a T5 model (and tokenizer) for html, as there are very few (if any) tokenizers that work well with HTML! You can try using GPT

my group is currently working on a T5 model (and tokenizer) for html, as there are very few (if any) tokenizers that work well with HTML! You can try using GPT4's tokenizer on your own HTML inputs below [1] ... there's definitely room for improvement! [1] https://tiktokenizer.vercel.app https://tiktokenizer.vercel.app