7 ms·
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
by SPascareli13 8d ago
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
- actionfromafar 8d agoAm I missing some joke here?
- wtallis 8d agoTo the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
- SPascareli13 8d agoYet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
- andy_ppp 8d agoExcept Taalas is much faster than GPUs, orders of magnitude so. They aren’t going to get 100x faster at inference any time soon!
- SPascareli13 7d agoThere's a new SOTA model every few months, are you supposed to buy a new chip every new release?
- infecto 8d agoGo back and correct your idea that consumer hardware made asics obsolete. Then we can figure out if asic or asic like devices for inference will have no advantage.
- mitxela 8d agoYeah! Nobody needs chatjimmy.ai. Nobody needs their results to come back instantly instead of at 10 tokens per second. Nobody needs a CPU faster than a megahertz.
- wtallis 8d agoASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market. And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
- infecto 8d agoThis is factually wrong no? Bitcoin is asic only. The others all changed for other reasons unrelated to your thought.
- SPascareli13 7d agoMy thought was that ASICs turned out not to be worth it for crypto mining because consumer hardware evolved fast enough to do it, while also being cheaper and having some resale value, while ASICs are useless besides mining and have no resale value. So I'm extrapolating this same idea to LLM inference.
- mhast 7d agoFor the problems ASICs exist they vastly outperform general hardware. Typically both in absolute speed and efficiency. But it's only possible to make custom ASICs when you have a specific problem to solve. For newer crypto systems they can vary enough parameters that building a flexible enough ASIC to recoup the investment before the algorithm changes and makes your hardware useless. For problems where the problem to solve remain in the problem space the ASIC can solve there is no point to use a thing else.
- infecto 6d agoYou’re extrapolating on something that is false. Consumer hardware never caught up to asic.
- connicpu 8d agox86 has a built in instruction for doing AES. That's just moving the ASIC into the CPU core, not eliminating it.