_t89y

7 ms·

1.
by _t89y 3y ago
That's a really interesting thing to point out. NLP doesn't even work on language anymore. If it was adjacent to information retrieval before it is now a subfield of information retrieval. As long as it's grounded in Firth Mo
2.
by _t89y 3y ago
They've taken off because they have utility in information retrieval systems. They work for getting info into Google (Stanford) Knowledge Panels. I don't think it really goes any further than that. They are most useful to the few
3.
by _t89y 3y ago
There is an inherent structure in language. Embeddings do not and will not capture it. It's why they do not work. Their ability to form grammatical sentences with high accuracy is part of the illusion that you have been understood.
4.
by _t89y 3y ago
Easy there, Firthmiester. I'm familiar with the canon. If getting some desirable behavior in your application is good enough for you then feel free to ignore what I'm saying.
5.
by _t89y 3y ago
https://news.ycombinator.com/item?id=39680852
6.
by _t89y 3y ago
The Lambek calculus. Categorial grammars. Meanings are proofs. Not clusters of directional magnitudes in space.
7.
by _t89y 3y ago
Having a mid-century theory of natural language semantics isn't necessarily a bad thing. You just have to pick the right one.
8.
by _t89y 3y ago
Modeling language in a latent space is useful for certain kinds of analyses and certain aspects of language. It has its place as an empirical tool. That place is not the nuts and bolts of language itself. There are more suitable formalisms
9.
by _t89y 3y ago
Uh oh. LOL. Got some angry Firthers out there.
10.
by _t89y 3y ago
You can't fine-tune for understanding or reasoning. You can't "get better performance" on understanding. You're either equipped for it or you're not.
11.
by _t89y 3y ago
For describing semantics in natural language? Pretty much anything else.
12.
by _t89y 3y ago
Works for what? The leaderboards? BPE tiktokens, BPE GPT-2 tokens, SentencePiece, GloVe, word2vec, ..., take your pick, they all end up in a latent space of arbitrary dimensionality and arbitrary vocab size where they can be mapped onto ima
13.
by _t89y 3y ago
They make a mess of language. They are not a suitable representation. They are suitable for their efficiency in information retrieval systems and for sometimes crudely capturing semantic attributes in a way that is unreliable and uninterpre
14.
by _t89y 3y ago
It is true. And if you want to say anything about meaning this isn't even the right math.
15.
by _t89y 3y ago
It is meaningless to talk about cosine similarity of sentences, or words, at all. Choose whatever mapping you want. You'll still be in Firth Mode.
16.
by _t89y 3y ago
No understanding. Embeddings are a semantically vacuous representation and similarity is a semantically vacuous interpretation.
17.
by _t89y 3y ago
It's definitely not about semantics or language. As far as language is concerned similarity metrics are semantically vacuous and quantifying semantic similarity is a bogus enterprise.
18.
by _t89y 3y ago
Thanks for this perspective on the tradeoff between accuracy and efficiency and the insight that an adequately pre-trained model should be in a position to recover lost information from bad tokens. Tokenization, the gateway to word embeddin
19.
by _t89y 3y ago
2009: Porter stemming with NLTK 2013: LDA with MALLET 2015: spaCy 2018: BERT 2023: GPT-4 2024: every person is an NLP expert in four lines of LangChain code
20.
by _t89y 3y ago
Thanks for your reply. That's my first point. In 10 years we have word2vec, GloVe, GPT-2 and... tiktoken. lol. It's as if directional, numeric magnitudes in an embedding space of arbitrary dimensionality have magically captured or
21.
by _t89y 3y ago
Thanks for your reply. It's exactly like lexers for compilers. This parsing strategy coupled with the decision to then map the results into an embedding space of arbitrary dimensionality is why these models don't work and cannot b
22.
by _t89y 3y ago
It’s pretty wild how little discussion there's been about the core feature of these models. It's as if this aspect of their development has been solved. Basically all NLP publications today take these BPE tokens as a starting poin
23.
by _t89y 3y ago
Thanks for your reply. >>> I think it would help if you link "NeSy computation engine". I'm actually not familiar with this (not in the symbolic world, but interested. Just never had time, so if you got links here I&
24.
by _t89y 3y ago
Thanks for your comment. Please let me know what's not clear. Still no takers on my ML is CV comment below. Research on on-device ASR and computer vision is primarily driven by the same organizations that stand to benefit the most from
25.
by _t89y 3y ago
lol. Three computer vision researchers dislike this comment. Do any of you want to respond to it?
26.
by _t89y 3y ago
Domain adaptation across verticals is the only driver of innovation. Check out the NeSy computation engine. Its ``semantic parsing'' is domain parsing and its symbols are numeric. Scale if you want. It works for images. That'
27.
by _t89y 3y ago
Exactly. Google.
28.
by _t89y 3y ago
From the GPT-2 paper: ``Also thanks to all the Googlers who helped us with training infrastructure.''
29.
by _t89y 3y ago
It's because they don't understand language. You may have been mislead by their ability to generate language.
30.
by _t89y 3y ago
There's a typo in the title of the talk but don't worry I fixed it: Trends in Computer Vision ``In recent years, ML has completely changed our view of what is possible with computers'' In recent years ML has completely c

More ›