10 ms·
Maybe you haven't noticed that the "Bitter Lesson" had itself a "Bitter Lesson" - that scaling pure data and compute did not lead to AGI: diminishing training r
by mentalgear 6d ago
Maybe you haven't noticed that the "Bitter Lesson" had itself a "Bitter Lesson" - that scaling pure data and compute did not lead to AGI: diminishing training returns, GPT-5 disappointment, even openAI stating it was the last 'pure scale' model.
The path forward all big llm providers ("ai" labs) have gone is neuro-symbolic (even though they publicly would never labeled it as such to not admit critics like Gary Marcus were right - even though all their actions actually point in that direction).
- hbogert 6d agoNeuro symbolic, rly? Can you please elaborate what it is that made you conclude that?
- jephs 6d agoI think me just means neural network models RLed to Chain of Thought reasoning? The thinking tokens are the symbolic bit. Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely. I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.
- augment_me 6d agoQuite a leap to call a random embedding a neurosymbolic representation