5 ms·
There’s no reason to assume frontier-level intelligence eventually collapses all the way onto a midrange consumer GPU. In fact, there are quite a few reasons no
by jimbo808 1mo ago
There’s no reason to assume frontier-level intelligence eventually collapses all the way onto a midrange consumer GPU. In fact, there are quite a few reasons not to assume that (information-theoretic constraints, etc).
- christophilus 1mo agoBut, it could happen for a coding-focused model, or an accounting-focused model, etc. most tasks only need a subset of the total model to be done effectively.
- fooker 1mo agoThere's no information theoretic constraint we know of that prevents this. You will almost surely win a Turing award if you can prove this. It's almost a given that whatever is frontier intelligence today will run on a potato in a few years.
- jimbo808 1mo agoKinda silly to follow your “prove it” challenge with an absurd claim you most certainly cannot prove, much less support with evidence.
- fooker 1mo agoIt was not a "prove it" challenge. I'm pointing out that there's no known information theoretic constraint about the impossibility of frontier AI models being improved to fit/run on a small GPU. Please do not make up plausible sounding science facts.
- jimbo808 1mo agoPlease do not assert I am making a claim I’m not making. Information-theoretic constraints exist. My comment does not require some specific, hard constraint to have been clearly defined, for my point to be valid. If I were to say you could put a motorcycle in my car’s trunk, it would be perfect valid for me to say there are space constraints that make your idea unlikely. The same is true in this discussion, even though I have not computed the exact dimensions of the motorcycle and my car’s trunk.
- fooker 1mo agoYour claim was about frontier intelligence and midrange consumer GPUs. That's a pretty specific constraint. Sure, there could be some point between a midrange consumer GPU and a pocket calculator where you can't fit enough 'intelligence'. But we really have no idea if the constraint is information theoretic or something completely different. Demonstrating that is the hard part, not finding the exact number of bits. Talking about motorcycles in car trunks is just lazy false analogy here.
- deleted 1mo ago[deleted]
- jimbo808 1mo agoThen I give up. Best of luck.
- amazingamazing 1mo agoI will not claim a 5070, but there is already evidence in nature that you can get very good general intelligence with an order of magnitude less wattage. There are constraints of course- training takes way longer.
- fooker 1mo agoI wonder if we'll eventually find that Darwin style evolution gets us close to the global optima of intelligence given constraints like size and energy. We don't really have the tools to reason about this stuff yet. Exciting times.
- somenameforme 1mo agoCould you not say the exact same of image gen models? For those that haven't kept up with that domain, you can now efficiently run high quality image gen models on any plain old video card, with phenomenal results.
- xdertz 1mo agoWe could very well reach a point where models don't get better anymore, or where consumer models are good enough for 95% of the use-cases.
- conmod278 1mo agoCore reasoning model with plugins for specialized tasks like "Pip install" developed using the new science of AI neurosurgery.