5 ms·
Is this really how SOTA LLMs parse our queries? To what extent is this a simplified representation of what they really "see"?
by georgewsinger 1y ago
Is this really how SOTA LLMs parse our queries? To what extent is this a simplified representation of what they really "see"?
- jdspiral 1y agoYes, tokenization and embeddings are exactly how LLMs process input—they break text into tokens and map them to vectors. POS tags and SVOs aren't part of the model pipeline but help visualize structures the models learn implicitly.
- deleted 1y ago[deleted]
- helloplanets 1y agoThis is partly completely misleading and partly simplified, when it comes to SOTA LLMs. Subject–Verb–Object triples, POS tagging and dependency structures are not used by LLMs. One of the fundamental differences between modern LLMs and traditional NLP is that heuristics like those are not defined. And assuming that those specific heuristics are the ones which LLMs would converge on after training is incorrect.