10 ms·Aren’t Unicode characters generally treated as 2 tokens to avoid a huge vocabulary?by noddybear 3mo agoAren’t Unicode characters generally treated as 2 tokens to avoid a huge vocabulary?