5 ms·
Where does this view come from? I’m not aware of any real evidence for this. Also consider our data center buildouts in 26 and 27 will be absolutely extraordina
by bubblelicious 10mo ago
Where does this view come from? I’m not aware of any real evidence for this. Also consider our data center buildouts in 26 and 27 will be absolutely extraordinary, and scaling is only at the beginning. You have a growing flywheel and plenty of synthetic data to break the data wall
- candiddevmike 10mo agoWe need a fundamental paradigm shift beyond transformers. Throwing more compute or data at it isn't pushing the needle.
- bubblelicious 10mo agoAnd you don’t think that’s already happening? Also where is your evidence for this?
- bigyabai 10mo ago> Also where is your evidence for this? The fact that "scaling laws" didn't scale? Go open your favorite LLM in a hex editor, oftentimes half the larger tensors are just null bytes.
- bubblelicious 10mo agoShow me a paper, this makes no sense of course scaling laws are scaling
- marcosdumay 10mo agoJust to point, but there's no more data. LLMs would always bottleneck on one of those two, as computing demand grows crazy quickly with the data amount, and data is necessarily limited. Turns out people threw crazy amounts of compute into it, so the we got the other limit.
- bigyabai 10mo agoSynthetic data works.
- marcyb5st 10mo agoThere's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model
- yorwba 10mo agoSynthetic data doesn't have to come from an LLM. And that paper only showed that if you train on a random sample from an LLM, the resulting second LLM is a worse model of the distribution that the first LLM was trained on. When people construct synthetic data with LLMs, they typically do not just sample at random, but carefully shape the generation process to match the target task better than the original training distribution.
- Alex2037 10mo ago[dead]
- Mistletoe 10mo agoYeah I’m constantly reminded of a quote about this- you can’t make another internet. LLMs already digested the one we have.
- bubblelicious 10mo agoEpoch has a pretty good analysis of bottlenecks here: https://epoch.ai/blog/can-ai-scaling-continue-through-2030 https://epoch.ai/blog/can-ai-scaling-continue-through-2030 There is plenty of data left, we don’t just train with crawled text data. Power constraints may turn out to be the real bottleneck but we’re like 4 orders of magnitude away
- skywhopper 10mo agoThere is zero evidence that synthetic data will provide any real benefit. All common sense says it can only reinforce and amplify the existing problems with LLMs and other generative “AI”.
- bubblelicious 10mo agoSounds like someone has no knowledge of the literature, synthetic data isn’t like asking ChatGPT to give you a bunch of fake internet data.
- ModernMech 10mo agoLet me put it this way: when ChatGPT tells me I've hit the "Free plan limit for GPT-5", I don't even notice a difference when it goes away or when it comes back. There's no incentive for me to pay them for access to 5 if the downgraded models are just as good. That's a huge problem for them.
- _aavaa_ 10mo agoIt is a problem easily solved with advertising.
- ModernMech 10mo agoNo, because as the history of hardware scaling shows us, things that run on supercomputers today will run on smartphones tomorrow. Current models already run fairly well on beefy desktop systems. Eventually models the quality of ChatGPT 4 will be open sourced and running on commodity systems. Then what? There's no moat.
- treis 10mo ago10-20 years of your data in the form of chat history Billions of users allowing them to continually refund their models Hell by then your phone might be the OpenAI 1. The world's first AI powered phone (tm)
- overfeed 10mo ago> The world's first AI powered phone Do you remember the Facebook phone? Not many people do, because it was a failed project, and that was back when Android was way more open. Every couple of years, a tech company with billions has the brilliant idea: "Why don't we have a mobile platform that we control?", followed by failure. Amazon is the only qualified success in this area.
- treis 10mo agoI agree that a slight twist on android doesn't make sense. A phone with a in integrated LLM with apps that are essentially prompts to the LLM might be different enough to gain market share.