5 ms·
The tokens per second performance numbers coming from Cerebrus/Talas are several orders of magnitude higher than models running on GPUs, which is such a huge st
by akiselev 2mo ago
The tokens per second performance numbers coming from Cerebrus/Talas are several orders of magnitude higher than models running on GPUs, which is such a huge step change that it will enable many more uses of LLMs that are impractical otherwise. I.e. think about gamers and burning in an LLM chip on a game console like a future Play Station - it doesn't matter if its a frontier LLM if it allows them to talk to in game characters in real time and have the LLM drive the storyline. The devs may be limited to training/RLHFing legacy models, but the performance enables many more use cases.
- quotemstr 2mo ago> Cerebrus/Talas They are fast, but they're still programmable accelerators, not a model burned into the gates.
- neynt 2mo agoCerebras is a programmable accelerator, but Taalas does burn the model into the gates.
- quotemstr 2mo agoAh, right. I misremembered. Does Taalas have a meaningful advantage here then?
- wyre 2mo ago6x. Taalas has Llama3.1 8B running at 18000 tok/s. Cerebras advertised that model at 3000 tok/s.
- quotemstr 2mo agoInteresting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?
- akiselev 2mo agoThe weights are in SRAM, so the LLM architecture is burned in but the weights can be updated.
- hedgehog 2mo agoNo the weights are in the metal layers, they cannot be updated.
- jfim 2mo agoThe base weights can't be updated but from what I recall it allows adding a low rank adapter to customize the model a little bit.
- hedgehog 2mo agoYes that's what I've read. As far as I know the approach should transfer well to hybrid model architectures like modern Qwen and sizes like 27B by using multiple chips. LoRA-steered Qwen 27B at 10K+ tokens per second would be transformative for some workflows.
- Tuna-Fish 2mo agoTaalas only uses SRAM for the KV cache and the activations, the weights are in mask rom in the metal layers. If they designed this right, it means that once they have a model, so long as they keep the hyperparameters fixed they can change the weights much faster than it takes to spin up a completely new chip, essentially at a cost of doing a minor revision.
- quotemstr 2mo agoIs the mask ROM really going to be worth it over Carmack's high-bandwidth-flash concept? I mean, sure, I could be convinced I guess, but it's not obvious.
- 2mo ago
- HNisCIS 2mo ago8B is still three orders smaller than current frontier though
- lsaferite 2mo agoTaalas claims significant cost, speed, and power differential with running on Nvidia (and similar) hardware. Their whole thesis is that with making the hardware cost less, run faster, and run with less power mixed with their claimed fast production cycle for new weights, companies would plan on purchasing and then relegating the other chips to different workloads at the next update cycle. And the chips have facilities for fine-tunes that can by dynamically loaded. The info I've seen has been very light on details, but if they can even get halfway where they are planning, it'll be a massive shift.
- stillpointlab 2mo agoI admit I would like a faster model - but even though I have faster models available I still go to Fable or GPT-5.6 90% of the time. So there is a gap between a potential preference and a revealed preference. Custom AI for things like facial recognition in cameras has existed for decades, before LLMs were a thing. I don't see that getting replaced. And on-device conversational intelligence might go that route as well, we'll have to wait and see. It's a lot of silicon to dedicated to a static non-changing thing. My money would be on programmable TPU-like things (Apple's NPU kind of stuff). It just seems more flexible to have an array of compute that you can load different models into, so you can update it, etc.
- sdfefcxv 2mo agoWhat you think doesn't matter unless its strictly for personal use. Your company will decide what makes economical sense.
- dehugger 2mo agoSome of us happen to be in positions to make those decisions at our respective companies. What we think does, in fact, matter.
- breuleux 2mo ago> even though I have faster models available I still go to Fable or GPT-5.6 90% of the time What about all the things you don't currently use an LLM for? If a specialized chip can run a model 100 times faster, you can suddenly use it for a lot of things at sub-second latency. You can write "make white transparent and add a red outline to x.png" instead of the corresponding imagemagick invocation and perceive little to no latency difference. You can hook it up to your browser and have it yank out all advertisements live, or tell it to highlight anything that might interest you, again, live. There's probably thousands of latent use cases nobody has thought of that would be enabled by a truly fast LLM, even a mediocre one. I don't think an on-device model needs to change much; it's already quite general in its capabilities.
- 2mo ago
- brian-armstrong 2mo agoThe models would have to get significantly better for this to help, though. Unedited LLM dialog is quite bad. Though, I wouldn't put it past AAAs to try anyway.
- F7F7F7 2mo agoThis is where treating these smaller models more like parts in a purpose built appliance ultimately benefits us.
- aurareturn 2mo agoI don't think ASICs is the long term answer here at the local LLM level. I think GPUs will still be. If you can have only one AI processor in your laptop (because they're big and expensive), it's going to be a GPU. This AI processor needs to inference LLMs, audio processing, image generation, video generation, etc. This is on top of normal graphics processing requirements such as video games, playing videos, decoding, encoding, etc. At the enterprise level, I can see some ASICs working once the market fully matures and improvements in architectures slow down drastically while demand for inference increases drastically. How far are we from this world? Maybe 5-10 years? It seems like model architectures are still changing rapidly and labs want fast experimentation that programmable GPUs offer. GPUs will still dominate in general - just like how CPUs still dominate despite ASICs.