5 ms·
This means that current tokenisers are bad, and something better is needed if text rendering + image input is a better tokeniser.
by sojuz151 11mo ago
This means that current tokenisers are bad, and something better is needed if text rendering + image input is a better tokeniser.
- mr_toad 11mo agoHumans recognise two vastly different types of language input (auditory and visual). I doubt that one type of tokeniser is inherently superior.