11 ms·
Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, eve
by ericd 13d ago
Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs.
But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.
They may have had a little flexibility by supporting finetuning via LoRAs.
- liamwire 13d agoTo qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.
- runako 12d agoThis seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.
- ericd 12d agoYeah, it'd be a lot easier to maintain flow, less need to work on more than one session at once, etc. And then tool calls would be the limiting factor, especially network access. I hope AMD keeps the project moving forward post acquisition.
- ericd 12d agoYeah, I'm guessing this isn't unique, but I remember the first time I used ChatJimmy, I missed the fact that it had responded because I was still hitting the enter key, and getting ready to see tokens stream in, but they were already all sitting there, and I'd missed registering the visual diff somehow.