7 ms·
I think this can't work because an LLM needs too much data, and before the internet there probably just wasn't enough to get close to what we have now
by elar_verole 1mo ago
I think this can't work because an LLM needs too much data, and before the internet there probably just wasn't enough to get close to what we have now
- jvanderbot 1mo agoEven simpler: Can GPT-2 anticipate and build Gwen/Deepseek? I think the answer is almost trivially "no", so I wonder what changed?
- ben_w 1mo agoLots of things changed, GPT-2 is small (1.5e9) and is also a base model, so it is only doing next-token/autocomplete rather than prompt-response like even the first ChatGPT-3.5 was doing.
- jebarker 1mo agoJust for the sake of clarity: all LLMs up to today are still only doing next-token/autocomplete. The training process got additional stages to shape the model weights, but standalone LLMs are still deployed essentially identically.
- ben_w 1mo agoIf you gave GPT-2 a question and ended with a "?", it might answer, but also it might write several more questions in a similar category. IMO, the mechanism isn't the important thing, the behaviour is. If you look at the step-by-step, we are also looking for the next word or motor action (and for whoever is about to suggest that we humans plan ahead, Transformer-based LLMs have been shown to also do this); as this is not a useful description of what it means to be a living brain, I'd say it's also not a useful description of what makes everything post-InstructGPT different from what came before.
- jebarker 1mo agoI agree completely - behaviorally the models have changed drastically due to RLHF, RLVR and now maybe even more so due to agentic harnesses. But the mechanism of prediction hasn’t changed, that was all I was clarifying.
- joefourier 1mo agoWhat about multi-token prediction and speculative diffusion? That’s a different mechanism of prediction, even if it serves only to accelerate decoding.
- jebarker 1mo agoAs you say, that's just an efficiency play and, as I understand it, doesn't change the behavior of the models beyond perhaps a small amount of sampling noise.
- wizzwizz4 1mo agoIf you frame it like so: <noob> Where do birds go when it rains? <expert> They then GPT-2 generally doesn't write more questions.
- ben_w 1mo agoGenerally. Sometimes it still did, in my experience.
- rowyourboat 1mo agoThat's not really a fair comparison, no? Modern LLMs are much more capable than GPT-2. We'ld need a modern LLM trained on exclusively old data, and that might be impossible
- hackernudes 1mo agoMaybe we can synthesize large amounts of limited information. I thought that new training data is mostly synthetic anyway.
- inigyou 1mo agoWhy couldn't an LLM, if it was smart enough, generate and consume its own data? I know the answer: because it leads to model collapse. But why is that? Wouldn't a smart model not collapse? It's seeming like they keep getting smarter because we keep pouring more of our own knowledge into them, not because they are actually getting smarter. And yes, sometimes a dumb but persistent bruteforcer can make new discoveries.
- Garlef 1mo ago> if it was smart enough and i think this is exactly the crux; the really big models need really big datasets and current gen LLMs get a lot of training data beyond "all books + all of the internet" the objection is then that producing this additional data would already confound it with pre "virtual cutoff date" knowledge (since the training data probably implies mathematical and SWE concepts that were developed post "virtual cutoff date")
- smusamashah 1mo agoIf it is smart enough to generate data it can consume to train itself better, it is already smart enough to not need to do that.
- inigyou 1mo agoIf a human is smart enough to do the Michelson-Morley experiment, they are smart enough to not need to do that.
- Plasmoid 1mo agoIt's because LLMs are entropy generators. That's not a bad thing for what people are doing. But to prevent model collapse you need a way to pump down the entropy. Much like in thermo, it's an expensive and slow process.
- Kinrany 1mo agoThey are already trained on generated data I believe
- naasking 1mo agoTypical LLM pretraining is very inefficient with data. NanoGPT slowrun shows that data can be used much more efficiently.