7 ms·

Thanks! Isn’t “an Astronomer” a single word for the purpose of answering that question? Following your comment, I asked “Give me pairs of synonyms where the la

by born1989 1y ago

Thanks! Isn’t “an Astronomer” a single word for the purpose of answering that question? Following your comment, I asked “Give me pairs of synonyms where the last letter in the first is the first letter of the second” Claude 3.7 failed miserably. Chat GPT 4o was much better but not good

nearbuy 1y ago
Don't know about Claude, but at least with ChatGPT's tokenizer, it's 3 "words" (An| astronom|er).
colah3 1y ago
"An astronomer" is two tokens, which is the relevant concern when people worry about this.
philomath_mn 1y ago
That is a sub-token task, something I'd expect current models to struggle with given how they view the world in word / word fragment tokens rather than single characters.