23 ms·
AMD acquires Taalas to boost inference performance by etching models in silicon
https://ir.amd.com/news-events/press-releases/detail/1296/amd-acquires-taalas-to-advance-compute-solutions-for-rapidly-growing-ai-inference-market https://ir.amd.com/news-events/press-releases/detail/1296/am...
https://chatjimmy.ai/ https://chatjimmy.ai/
- khoss 1mo agoI think the key application here is spec decoding. This solves the deprecation problem pretty nicely, and will likely have huge performance benefits
- gavinbuilds 1mo ago[flagged]
- proxysna 1mo agoReally hoped to see their hw out in the wild one day
- rvz 1mo agoDidn't even give them a chance to launch the hardware.
- MarkWayneNewton 1mo agoWhile this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.
- Legend2440 1mo agoThis is a new architecture. It's a non-vonn neumann device.
- badatnames 1mo agoWell so much for that dream. Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
- dogomatic 1mo agoWhat’s stopping competition?
- whythismatters 1mo agoThe demo: https://chatjimmy.ai/ https://chatjimmy.ai/
- senderista 1mo agoWow, feels like Google web search in 1999.
- joshvm 1mo agoIf you still want the experience, go and browse McMaster Carr. Wizards designed that website.
- senderista 1mo agoOh I have, though not for a while.
- eglintondust 1mo agoI'm inspired by this website. It's incredible.
- jodrellblank 1mo agoor LiveGrep fast search of the Linux kernel source code with regex support: https://livegrep.com/search/linux https://livegrep.com/search/linux
- senderista 1mo agoWow, I want something like that for my company's codebase.
- nsxwolf 1mo agoIt doesn’t believe it’s running on that chip, it’s arguing with me
- shaewest 1mo ago
- A_D_E_P_T 1mo agoThis is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.
- badatnames 1mo agoThey were too small for this to be a meaningfully sized purchase for AMD, there's real risk they get sucked into a team that ultimately delivers sqat, not to mention the chances of anything being delivered in an even remotely consumer-priced bracket are definitely out the window
- ycui7 1mo agoso qwen3.x-27b on hardware? or better deepseek-v4-flash on hardware .
- ilaksh 1mo agoI wrote them an email asking for PrismML Bonsai 27b Ternary which is like 6b or something crazy small and would be a lot easier for them to do initially.
- mdp2021 1mo agoThey were specializing their forthcoming system on 4-bit FP - which I understand is a structural decision. Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.
- syntaxing 1mo agoHonestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.
- smokel 1mo agoThe technical aspects of SOTA models are not publicly documented. How do you know if something is converging?
- _aavaa_ 1mo agoIf we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.
- nurumaik 1mo agoWill be capable and fast enough for 2-3 weeks until new sota drops
- amazingamazing 1mo agoIf it is capable today why would a new model change this?
- FridgeSeal 1mo agoBecause new stuff instantly makes anything prior bad and incapable and garbage of course! Did you forget the hype-machine speaking notes??? /s
- catchnear4321 1mo agoif capability is a commodity then the differentiator becomes taste.
- 1mo ago
- bhouston 1mo agoToronto Canada startup btw.
- cmrdporcupine 1mo agoSeems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller? Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
- kridsdale1 1mo agoWorks well, I remember driving by the ATI building as a kid.
- mikeayles 1mo agoAMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away. Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though. I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
- tandr 1mo agoWell, technically it is their hardware now...
- questionableans 1mo agoAnd their team, if they treat them well.
- Melatonic 1mo agoYeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically. I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
- Legend2440 1mo agoWhat you're describing is what Cerberas does. Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells. Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
- bob1029 1mo agoI feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.
- kridsdale1 1mo agoFPGA model storage?
- addaon 1mo agoNAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.
- mdp2021 1mo agoThe basis of Taalas is "compute in memory" electronics - past Von Neumann's separation of processor and memory. You need to be able to add|mul where the data (the weights) are stored.
- fellowniusmonk 1mo agoToken quantity will have a quality all its own.
- LarsDu88 1mo agoI'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
- larodi 1mo agoWe don’t really known (from the outside) how exactly do they move. Besides it may have not been truly viable 1-2 years ago…
- LPisGood 1mo agoI’m surprised Nvidia hasn’t partnered to make a Claude chip yet. It’s a win/win you can license them out, sell them when they become obsolete, etc.
- moshun 1mo agoConsidering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.
- amelius 1mo agoNot sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.
- walrus01 1mo agoImagine the size of chip needed to 'etch' something like Qwen 3.6 27B in size.
- flog 1mo agoIf someone has that sort of knowledge; how big a chip would be required? Is it possible?
- mdp2021 1mo agoWell, given the data above, roughly a 220b transistors chip for the HC1 tech.
- mdp2021 1mo agoNot too dissimilar to the first HC1 (6nm 815mm² 53B Transistors embedding an 8b LLM): > Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM
- golem14 1mo agoInteresting thought, because it's a yield question. How tolerant are models today to a few broken weights. If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)
- walrus01 1mo agoI wonder if you had a few percent of problems in the yield, if it would be functionally equivalent to the difference between a unsloth-published Q6 standard size GGUF vs. the nearly perfect precision of an unsloth Q8-K-XL. Or more like Q4 vs Q8 where a lot is lost.
- thepasch 1mo ago> How tolerant are models today to a few broken weights. Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses. I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence. I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.
- nojs 1mo agoCan anyone comment on the economics and likely turnaround times of this process, when it’s more mature? Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date? Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
- shangofox 1mo agoI mean even if it take a few months, it'll still be out of date. But there was a hypothetical when it came up in Feb, would you want Qwen 3.5 at like 10k tokens per second. At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?
- xienze 1mo agoThere's soooo much stuff that such a model is still capable of doing in the pursuit of getting a better overall answer. Imagine a powerful research agent that blasts out dozens of the small, cheap models to fetch and summarize one page each. Then the beefy researcher model performs the final analysis.
- 2001zhaozhao 1mo agoThere are always uses for outdated models. Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"
- edot 1mo agoJust looked this up, no longer true. Explore subagents inherit whatever model the parent is. And you can of course make other subagent configs.
- samtheprogram 1mo agoThat's solely so that you burn more money. It's totally unnecessary to assume the parent model. Sure, it could be upgraded from Haiku if there was a solid reason to, but...
- andrewvl 1mo agoIt must be a “super model”. What will be if new model released? New chips?
- downrightmike 1mo agoChip pops out like a gameboy cartridge. AI not working? Blow on it and jam it back in
- andrewvl 1mo agoAgree. Price is the question.
- downrightmike 1mo agoDepends on how smart you want your MegaMan to be
- tecoholic 1mo agoWith web search and tool call a decent current generation model at the speed of the chatjimmy could do a lot. People saying it would be out of date are missing the point. It’s not going to make much sense for frontier companies that’s chasing the SOTA. But for a lot of business use cases if someone can put GLM 5.2 and sell it as a box, it would make so much sense. My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
- equinumerous 1mo ago100% agree - you don't need the most up-to-date model to have something that's useful in agentic contexts. They could even produce chips with weights that make all the decision making/logical reasoning and have it delegate to other specialized agents. If it becomes cheap enough to print a run of custom chips, releasing a batch for each major advancement does not seem unreasonable for SOTA companies.
- cephei 1mo agoThere are so many use cases for supremely fast offline models. The first thing that comes to my mind is for real-time video processing or other non-textual content in real time.
- anigbrowl 1mo agoI wouldn't call it supremely fast but zippy and versatile, yes: https://shop.m5stack.com/products/ai-pyramid-computing-box-pro-8gb-version-ax8850 https://shop.m5stack.com/products/ai-pyramid-computing-box-p...
- 5555watch 1mo agoIn my understanding the first Deep Think / Pro models were already very good as they were doing some kind of parallel repeated reasoning, thus were slow and expensive. So if chatjimmy speeds enables a fast deep think level performance, I think that would be great.
- yumraj 1mo agoGiven the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
- sigmoid10 1mo agoI find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.
- deadbabe 1mo agoCan you give some examples of these tasks that require an hour or more of reasoning?
- xyzsparetimexyz 1mo agoThe recent maths prompts did. The 'you should find a breakthrough' one was several blocks of reasoning, each taking 90 minutes or so
- sigmoid10 1mo agoI've let it research and implement pretty complex stuff for my 3d engine. Like a realtime volumetric cloud rendering solution. The kind of stuff you can do yourself, but it would probably take weeks, if not months. I spent quite a bit discussing the design with it, knowing some details of how it should work from game dev conference talks. Once we agreed on a design and implementation plan with key steps and intermediaries, i just said "now implement it" and it happily designed algorithms, wrote and ran tests, generated and looked at screenshots and basically did everything you would expect from a really senior engine programmer for more than an hour of wall time. The end result was something that worked amazingly well after testing it myself and suggesting minor corrections that were my fault for not specifying them in the first place.
- msteffen 1mo agoThis is neat but IMO a little crazy. Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least. I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply. I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
- daishi55 1mo ago> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least. What are some examples?
- wmf 1mo agoThere's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience
- daishi55 1mo agoI wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.
- cubefox 1mo ago> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
- IsTom 1mo agoI think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.
- wmf 1mo agoThe methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.
- pyrolistical 1mo agoUmm I have an extra 35, do you have layer 6?
- octoberfranklin 1mo agoThey pipeline-parallelize across multiple chips. DeepSeek v4 Pro will be 30 chips.
- redmoonx 1mo agoIt obviously won’t be continuous delivery but could make sense if the lifecycle of a model (train, deploy, iterate (meaningfully) is about 1-2 years. In that case it fits nicely in the “this year’s model” already established with cars, phones, etc.
- ggm 1mo agoField reprogrammable, it's an FPGA on steroids. Field upgradable. Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
- mdp2021 1mo agoCan that be done when the whole idea is to store a multiplier into a handful of transistors?
- ggm 1mo agoI have no idea. It makes my comment a statement posted as a proxy for a question, a question you correctly pose explicitly. If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor. If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.
- XCSme 1mo agoWhy not have some a device/hardware that programs itself on-boot. Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.
- xyzsparetimexyz 1mo ago
- redox99 1mo agoIs there any LLM from exactly one year ago that would be worth running? In Aug 2025 you had - OpenAI o3 - Opus 4.1 - Gemini 2.5 Pro - Grok 4 Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free. Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
- daishi55 1mo agoThat is fkin wild. o3 was just a year ago? The progress is truly insane.
- redox99 1mo agoYeah I had to double check, o3 feels like it was ages ago. But GPT 5 came out Aug 7, so it's only one day off from my 1 year ago cutoff!
- mdp2021 1mo ago> Is there any LLM from exactly one year ago that would be worth running? Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).
- redox99 1mo agoIt's not just about it "being smart enough". It's about there being actual user demand when it needs to compete with the shiny new model. A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?
- mdp2021 1mo agoIf you have a 2 years old item working and the new 2 months old item offers little advantage over the old one, and its cost is nonzero, an important part of the aggregate demand will stick with the old one... So, when the models will be "good enough", you will probably use one as the "daily driver" for a long time for consolidated workflows (some of them enabled by the staggering collateral advantages of specialized hardware and obvious advantages of local hardware), and occasionally use other available models for exceptional tasks, and upgrade only when definitely advantageous - like normal goods.
- jackdoe 1mo agoCan you imagine in few years getting Fable level intelligence at 20k tokens per second? "You are not prepared" --Illidan Stormrage
- preommr 1mo agoPeople are missing the point if they think this is useless because frontier models keep changing every few months. We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software. I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
- ur-whale 1mo agoYeah, so https://chatjimmy.ai/ https://chatjimmy.ai/ ... the model is crap, but the speed is amazing. Worth checking out.
- empiricus 1mo agoWorth wondering why they used a crap model.
- jauntywundrkind 1mo agoCore rope memory is back baby! Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.) https://youtu.be/3MKRjt59hh4 https://youtu.be/3MKRjt59hh4
- laweijfmvo 1mo agoI’ve been using Gemma as my default (via Kagi) because it’s served on Cerebas hardware. The speed is honestly a game changer for day to day queries.
- drob518 1mo agoSo, Kimi K3 in silicon sometime soon?
- api 1mo agoI've had an endgame idea in mind for a while. Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges. You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
- wmf 1mo agoEach cartridge costs $1,000. Do you still want it?
- api 1mo agoMe? Probably not. A business or a hoster, sure. There'd probably end up being an aftermarket in used cartridges with slightly older but still good models on them.
- anigbrowl 1mo agoFor fast Kimi K3? You're damn right I do
- wmf 1mo ago$1,000 only gets you the Qwen 27B cartridge. For Kimi K3 it would be more like $100,000 (and the "cartridge" is the size of a refrigerator).
- trollbridge 1mo agoI would gladly pay $100,000 for local K3 running at 18,000 tok/sec.
- anigbrowl 1mo agoSome hurried goalpost-moving there! If we take Qwen 27B as the benchmark of equivalent power (rather than its default configuration with no fine-tuning), I can still think of applications. Auto summarization for legal document databases and other high security/offline contexts, contraband detection, moderating websites like HN etc :)
- roughly 1mo agoHow's that jive with the fact that they're introducing a new model every other week?
- drchickensalad 1mo agoThe new model every week is not necessary at this point really. What if you could run opus 5 for the next couple years at 1/20 the cost?
- roughly 1mo agoWhat's interesting about this is that I as a user would find this useful, but I think the AI industry as a whole would find it an absolute goddamn disaster. Opus 5 is a very good tool, but it is not a human-replacement-level intelligence, which means the entire revenue stream the industry's built on - labor replacement - is not met by this, and the only slightly charitable read of the industry's finances is that they're gonna bootstrap their way to creating the labor replacement hypothesis by getting people to spend money on Opus/etc, whereas if the actual product is a 1/20th the cost Opus-on-a-chip, the entire business and financing model that's tying up $N Trillion dollars of investment money goes out the window. Great for us, looks like a recession as far as the Market is concerned.
- jaggederest 1mo agoPipeline the burn into silicon, lower the latency as much as you can, for the 10-100x operation cost it's worth it. Imagine if frontier models cost $5/mtok and the 2nd or 3rd tier models cost $5/billion tokens for 3-month-old models.
- moralestapia 1mo agoTaalas is just a phenomenal startup from Toronto. My dearest congratulations to the founders. Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.
- yousif_123123 1mo agoIf things like this get traction, will we need all the datacenters?
- downrightmike 1mo agoYou are mistaken about what the datacenters are for
- tripledry 1mo agoWhat are they for?
- downrightmike 1mo agoopen air prison
- mNovak 1mo agoWhat I like about this, is that it significantly increases the probability of a sci-fi scenario where you're picking up a hot chip on the black market; rumor has it, Mythos 9 weights baked in...
- bigyabai 1mo agoPlug it in, and it's a old prototype with Gemma 5 weights baked onboard. Dammit, fucked by Craigslist again!
- NitpickLawyer 1mo agoBack in the kazaa and limewire days, you'd sometimes try to get a movie / episode from a series, wait hours / days for it to download, and when it was done you had a ~50/50 chance to actually watch what you wanted or an old german porn movie :/
- rullelito 1mo agoBetter then vice versa!
- arlort 1mo agosteadily heading towards the final goal of computers being composed of pluggable crystals as stargate intended
- matheusmoreira 1mo agoBlack market uncensored heretic Mythos weights...
- pragma_x 1mo agoWhat's old is new again. We'll be buying cartridges for AI capabilities on future architectures.
- sgc 1mo agoWhat does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses? Are we a couple years away, a decade away, or something else?
- mdp2021 1mo ago> What does it take to go from here to a model on a pcie card or an m.2 card It is already that. > Will "intelligence" become much like a gpu As an option among the implementations. > Are we a couple years away They could mass produce now, but it makes no sense at this rate of improvements in the models.
- sgc 1mo agoThanks for answering. This is an 8b model, which are mainly curiosities outside niche tasks. I guess I am asking how far we are away from having today's more generally useful frontier model equivalents widely available for everyday users in their personal pcs/laptops via a single pcie or m.2 drop in.
- mdp2021 1mo ago> how far we are away from having It depends on AMD now. What was planned after the 8b was a ~30b, which is already sufficient (or more, when running at ultra-high speed).
- andix 1mo agoIt would be quite ironic if this technology would render all those AI data centers practically useless. If the next step are just a much smaller amount of expensive chips, and the bottleneck becomes manufacturing those chips fast. Not building huge data centers and fighting for electrical power.
- linzhangrun 1mo agoThinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.
- DiscourseFan 1mo agoIt will be cool but also violent and terrible.
- pizzaiolo 1mo agoSo, like the present
- barbazoo 1mo agoWith more wealth concentrates at the top, yes.
- bigyabai 1mo agoPresumably wealth would concentrate upwards even if AI was never made.
- sanex 1mo agoYes it's a function of the monetary system. Absurd amounts of debt only certain people can access.
- DiscourseFan 1mo agoSo you're saying we need to abolish to monetary system?
- __MatrixMan__ 1mo agoLike some kind of pimple, which we can pop from all sides. And then we'll build something different. Something that works.
- galaxyLogic 1mo agoI think the big news is that AMD is getting into memory-business so they won't be so dependent on Hynix and what have you. Memory is the bottleneck currently.
- nolist_policy 1mo agoAMD doesn't even have their own fans.
- OddMerlin 1mo agoCongrats to the Taalas gang.
- concraper 1mo agoA massive L for Canada
- galaxyLogic 1mo ago"... the chip serve Meta’s Llama 3.1 8B at a blistering 16,960 tokens a second — when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. "
- ford 1mo agoI've been showing people chatjimmy for months - it's incredible. Both reasoning and tool use generation scale with TPS. Imagine 100x more reasoning on a model, or 100x parallel tool uses.
- tech234a 1mo agoSee also: Twitter statement from Taalas https://x.com/taalas_inc/status/2085458427757937097 https://x.com/taalas_inc/status/2085458427757937097
- est 1mo agoWaiting for intelligence on a stick, plugin an USB, characters in, characters out. 100% local and no leaks.
- hliyan 1mo agoQuestion: we currently emulate neural networks by performing matrix math in synchronous clock CPU architectures. Would it not be better to abandon synchronization and etch neuron synapses directly in silicon, keeping only the weights variable? I think some researchers are pursuing this, but I forget what the approach is called.
- stopyellingatme 1mo agoRelated: https://corticallabs.com/ https://corticallabs.com/ - neurons on chip
- freakynit 1mo ago"Neuromorphic chips" .... and I have the exact same question in mind.
- Perz1val 1mo agoThat'd be either analog computing or you're implementing floating point number math in each neuron. Maybe some kind of PWM calculations would be doable, do we have it researched or would be a new kind of math?
- Ey7NFZ3P0nzAe 1mo agoI don't understand what you'd etch instead of the weight? What's a neuron synapse to you that's not a weight?
- yunnpp 1mo agoI would've hoped the company stayed independent instead of being engulfed into a behemoth. I'd like to see more diversity in the hardware ecosystem, but I guess the economics of hardware manufacturing aren't there.
- kristianp 1mo agoI've been eagerly awaiting their 2nd gen HC2, which uses multiple chips to host a "mid sized reasoning" [1] model. Its due in summer according to the article, I wonder if it will ever be released in that form now. [1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-launches-hardcore-chip-with-insane-ai-inference-performance/ https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-la...
- NitpickLawyer 1mo ago> I wonder if it will ever be released in that form now. Yeah, I had the same thought. The key thing for them was the price point at which they could deliver a ~30B model. I would buy one today if it was ~1000$ and could run whatever the best 30B model is today, at those speeds advertised. Even if the model becomes superseded by model.5 in a few months, there's still a lot of things you can do with a "good enough" model for some tasks. And things like maj@x or generate 10 times and choose "at a glance" what you like (think frontend stuff) would be worth it. No idea if them selling to AMD is good or bad.
- num42 1mo agoI have used chatjimmy before, it is incredibly fast, waiting for latest SOTA model on the chips in future. Great!
- jijji 1mo agotaalas is great for llama 3.x 8B models, really bad for one board serving Kimi K3, it seems like you would bottleneck at a few hundred tokens no matter what you do.... spreading the big model against multiple cards seems the only way to get into the 1k+ tok/sec range. Another thing taalas is doing is masking the model weights into the silicon itself, not a flashable firmware, which would increase latency....
- 3836293648 1mo agoThey moved from HBM to dedicated silicon and only got a 48x speed up? That is so, so, so much less than I would've expected. Any numbers on how it scales?
- zkmon 1mo agoI guess the idea is, gains from inference speed could offset the cost of upgrading the chips to a new model when really required. I think general purpose models would consolidate and release frequency might flatten out, favoring this strategy.
- matheusmoreira 1mo ago> Once the chips are deployed you’re stuck with that model. At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.
- analog31 1mo agoWow, we're heading back to mask-programmed ROMs. I'm feeling young again.
- ratsbane 1mo agoSmart move by AMD. Chatjimmy is very fast and not very good, but I think it might become very fast AND very good.
- yassa9 1mo agoCan anyone imagine if a video generation model with the speed of ASICs baked into silicon ? real Sci-fi
- khanhnguyen8386 1mo ago[flagged]
- yigalirani 1mo agowhat prevents amd to just do what they do without acquiring them?
- hugmynutus 1mo agoHN is rightly pointing out putting a model into an ASIC is kind of dumb. HN is failing to understand that AMD knows this well. Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.
- christkv 1mo agoThere is a big risk in etching a model into silicon like this. We are still evolving what small models look like and improving their performance. When do you decide to etch one into silicon knowing that right now an improved one can be 3 months away.
- roschdal 1mo agoIs this the singularity?
- tonyhart7 1mo agoso in the future I can buy KIMI, GLM or whatever model that get "soldered" directly into GPU ???? so instead of RTX xx70 series, I can buy xxTA that have kimi integrated ??? is that right ??
- spacedoutman 1mo agoI feel like this will be the end of Taalas, AMD has for the most part of its history always chosen the wrong options.
- jdub 1mo agoalways? like amd64 vs itanium? (and AMD is still serious challenging Intel in x86 and GPUs today)
- rldjbpin 1mo agogiven their specs [1] show ~100 mm^2 of silicon for each billion params (albeit in 6 nm process), what will come first: nuclear fusion or one of these that can handle the multi-trillon param models of today? [1] https://taalas.com/products/ https://taalas.com/products/
- dave1010uk 1mo agoI'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personalised context. 2. AI interaction seems more like find-as-you-type than a back and forth. 3. AI produces many outputs to pick from. Either the human, AI, or another system can do the deciding. Even if it's last year's model, speeding up LLM inference could open up all sorts of opportunities.
- tybit 1mo agoYes, this is a great point and it’s even one that the Cerebras CEO spruiks for his own chips. Talaas has gone furthest on the spectrum here, so seems to have the most potential to evolve the use cases.
- amelius 1mo agoI think the most important direction will be: you spawn hundreds of agents at the same time, and let them work in a breadth-first search style. So you will not necessarily get your answers much faster, but they will be much more thoroughly researched. And if you do want faster responses, you can do that at the expense of quality.
- petra 1mo agoYou could probably do real-time deep research using that method, and than summarize and ask questions about the results. That is a probably a significant jump in search quality for many queries, that people didn't take the time to research properly.
- overfeed 1mo ago4. Pervasive , distributed dragnet surveillance under the misrepresentation that it's not a search until a human pulls the data. But a small on-device "E2E preserving" "safety" model that runs on your phone and snitches when illegal communication content is suspected. Edit: also consider centralized Room-641A-type surveillance when models summarize and/or flag all calls processed by public telephony
- rbanffy 1mo agoMy understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
- bjackman 1mo agoNo I think you are thinking of Etched/Sohu. Taalas' approach (at least for their demo'd product) is to bake the whole thing in completely. IIUC the optimiser can even see the weights while generating RTL. It's like there's an "uint8_t weights[] = " in the source code.
- fischeti 1mo agoThey are loaded on top in the sense that they are contained in the upper layer(s) of the chip. So when you want to change the weights, you have to produce fewer masks for fabrication, which reduces cost and time to market.
- rbanffy 1mo agoOh... That's not great - it'd be nice if it had a way to push updates without building a new chip. OTOH, maybe because of this our future cyberdecks will have cartridge ports.
- bjackman 1mo agoI believe the fully baked-in nature is pretty important for the perf they get. With a cartridge port you now have a bus between the weights and the compute and the weights and that bus can become a bottleneck. So I think we're looking at a spectrum here: - Fully fixed function - i.e. Taalas - Fixed function transformer unit (or whatever other AI architecture) with a "cartridge" for weights etc - i.e. Etched - Flexible TPU/GPU type stuff I think the middle of the spectrum is a bit of a dead space ATM because new models have recently been coming with significant updates to the architecture (like MoE, MTP) so by the time you have new weights you wanna load, you also want to replace the compute too. So really you probably either say "I can tolerate an old model, but I want it fast as FUCK" and go for Taalas-style, or you say "I want a near-frontier model" and you have to use flexible compute anyway. But, caveat: this comment seems to be making me sound more knowledgeable than I actually am. Take this with a grain of salt.
- perarneng 1mo ago"Generated in 0.007s • 14,370 tok/s" - for some use cases this will be awesome
- perarneng 1mo agoOn thing with this enormous speed is that you could create multiagent validation and correction and still have it super fast and achieve greater quality thanks to speed alone.
- Cort3z 1mo agoI believe this is big news. Speculations here on out: I imagine this being baked into consumer products, greatly increasing the local token capability for consumers. They will suck the cloud-oriented companies' milkshake. Most users do not need extremely capable models, they just need some automation to do better web-searches, and get simple facts etc. If it can do simple coding tasks too, but at thousands of tokens per second, in stead of tens or hundreds, the development will benefit so much. It will benefit AMD in other ways too. I imagine they can start selling physical chips, usb-drive like devices, that just does llm. If you want a newer, better, model, you simply go to a store and buy one. Need more capability, buy more drives. Similar to physx back in the day, but with usb-c and a smaller footprint.
- reqo 1mo agoOne advantage that I see in models that are implemented as code is that they can quickly and cheaply be modified using LoRAs. What would the equivalent be in hardware? Another piece of hardware you would attach like adding a graphics card to a computer?
- trash_cat 1mo agoPeople are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model? This move undercuts NVIDIA directly.
- Zenst 1mo agoWith AI models like Mixture of Experts, many of those experts will be the real target here, as polished, refined and little to no change, they become fine candidates for being locked into silicon. Who knows, add some SRAM in there and small changes to those experts could be carried out without needing new silicon. Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for some areas as you say, motor control for balance, vision systems, speach recognition systems etc, broken down, for robotoics, much is already there and just cost of battery/power holding much back.
- runtime_lens 1mo ago[flagged]
- Tade0 1mo agoAn "Expert" is really just an unfortunate name for what amounts to a dense part of a sparse matrix and that's also an oversimplification. It doesn't actually specialise in anything in particular that one can point to. For this reason you can really transfer them between models.
- 5555watch 1mo agoIs it still true? I'd assume you should be able to freeze the matrix and unfreeze an expert block, before feeding particularly chosen training data. Or that doesn't work?
- Tade0 1mo agoThat's more or less the idea behind Low-Rank Adaptation, or LoRA. There's also Mixture of LoRA Experts, which instead of slicing up the model and routing through that, routes through different LoRAs. But it all comes with tradeoffs, as you have to train and run the gating network doing the routing, which also comes at a cost.
- luciana1u 1mo ago[flagged]
- runtime_lens 1mo ago[flagged]
- TechTechTech 1mo agoI think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
- jameshart 1mo agoCommoditize your complements - still a winning strategy. If you make chips, you want models to be free.
- mixermachine 1mo agoScaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
- formerly_proven 1mo agoIt's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about half the area on ROM. There's a reason AI accelerators just use a ton of silicon area (each HBM3 die is >1000mm² of silicon). I imagine this is not terribly viable unless they make it a lot more space efficient e.g. using MLC ROM if they don't already, or use stacked dies with a ROM-optimized process. And then we're back to not cheap, though reticle chips were never in the cheap area to begin with.
- woadwarrior01 1mo agoI suspect this acquisition was a response to NVIDIA's acquisition of Groq. Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.
- nullc 1mo agoMight be an interesting motivation for looped LLMs to cut the gate count down. Perhaps even a collection of mixed programmable layers and baked layers in a loop.
- peter_d_sherman 1mo ago>"While the tech is blazing fast, if you hadn’t already figured it out, it comes with a pretty substantial downside. Once the chips are deployed you’re stuck with that model. Any change bigger than something like a LoRA adapter is going to require a re-spin of the chips, which is not only expensive but time-consuming." That is a downside to be sure, but from a pure business perspective, "that's not a bug -- it's a feature!"... from a pure business perspective it's the ability to sell and resell, to purchasing and re-purchasing customers, way into the future -- that is, recurring revenue from the perspective of the company being able to make those future recurring sales... In the above case, that company is AMD... (Also, on a related note, it would be interesting to see what open source / open hardware work has currently been done to offload LLM weights (and/or anything else that could be offloaded to silicon ASIC's) to FPGA's...)
- Iolaum 1mo agoTBH Taalas was a company I was existed about as a consumer. A dense model like gemma4-31b or qwen3.6-27b running at 10k t/s sounds like an awesome thing to have. Would be willing to pay GPU prices for it.
- harhargange 1mo agoMy 7900XT runs it at 35 tokens per second.
- Tuna-Fish 1mo agoNote the k in his figure. Taalas HC1 was clocked at 17000 tokens/s.
- eripa 1mo ago[flagged]
- galeos 1mo agoIs there scope to implement ternary models using this approach to minimise die area of the model parameters?
- mdp2021 1mo agoTo the best of my understanding it would make no sense - they are already using (I have to investigate how they achieved it technically) a single transistor to store the "weight", and they manage to encode an FP4 in it (through a table). No gain in having less than FP4, so.
- harhargange 1mo agoI have an AMD CPU as well as an AMD GPU. Their open-source philosophy and Linux compatibility, including a clean core distribution (No big.little for desktop) is what keeps me in. I think Taalas is in safe hands.
- ActionHank 1mo agoBig if it pans out. Economic and financial ripple effects would be huge aside from the obvious: - reduction in electricity usage - OpenAI / Anthropic are dead in the water unless they start to license their models to fabs. - Every single one of those GPUs that all of those massive data centers contain become paperweights.
- voxic11 1mo agoYou still need the GPUs for training new models though right? This is only suitable for inference.
- ActionHank 1mo agoFor sure, but I strongly doubt that we would need nearly as many.
- Transformanshen 1mo agoI can actually see this being useful for fairly narrow workloads in dedicated devices where the model doesn't need to change very often and low-latency inference matters more than flexibility I don't see it replacing general-purpose GPUs but it seems like a reasonable option for that kind of workload
- 12912312 1mo agoThis is a sign of the AI bubble bursting. The 2000 bubble also saw a flood of hardware accelerators from SSL to packet switching to firewalls to graphics, most of which failed.
- brap 1mo agoWhile the demo is incredible, I think that in most practical use-cases, models aren't very useful without tools (search, code execution, etc.). Even if we assume reasoning latency drops to ~0ms (AFAIK this demo doesn't include reasoning at all), these use-cases will still remain relatively slow due to I/O of tool calls.
- faxmeyourcode 1mo agoCompaction, busting the cache, and other issues like that will lose significance when you're running at 15k tokens per second like chatjimmy. Very interesting to think about what will change in the future.
- bot41 1mo agoI'm a big AI user, and have some minor experience training models, and follow some AI stories - but this is the coolest thing imo. The AI model/weights written directly on the chip!
- dabbz 1mo agoI see a lot of discourse about it being fast-to-deprecation. But I see it a different way personally. Modern LLMs are trying to do more with less. Focus on doing the right thing the first time. Even if we squeeze dumb LLMs, the significantly faster speed means quicker iterations. So a bad decision doesn't cost the time and inference costs that it cost before. It theoretically changes the scale of errant token spend. I compare it to the 1 thousand monkeys on a typewriter. In this case it's 1,000 monkeys with stale training data of everything ever written and the ability to search the web.
- vel0city 1mo agoI agree with this take in a lot of ways. If you slash the token cost and increase speed for each token 1,000x, who cares if it takes even 20x as many tokens to achieve the goal? And also, there are lots of tasks where models today are fine with doing. If you think of these things like appliances, who cares if it's not quite as powerful as the next generation? It was purchased to do a task, it still does that task very well. It feels like being in the 90s and asking "why buy a server today when they're going to be faster next year? Just keep renting mainframe time." Well maybe I just need a box to run our HR and payroll system, and this box manages to run it fine today.
- mafuy 1mo agoSounds to me like you would hire 20 barely-paid interns instead of 2 competent programmers.
- jedbrooke 1mo agoI’m surprised I haven’t seen anyone mention video models yet. I don’t know how many fps 17000 tok/s translates to exactly but it’s gotta be a lot. Might make real time AI video possible. Now that I think about it, real time AI video might be a clear case of “You scientists were so preoccupied with whether you could or not, you forgot to ask if you should”
- graiz 1mo agoTake an open model and put it on chip, it becomes a dedicated sub-agent. You can still use a frontier model to check things or orchestrate but having 1000 sub-agents becomes very possible. I was already long AMD but just added to my position.
- adityazero 1mo agoThere are market verticals where this makes a lot of sense. Embedded systems and IoT devices comes to mind. Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.
- 5555watch 1mo agoEven for consumers.. Every enthusiast now wants those specced out rigs to play with LLMs. Make a nice chip for that, and it will reduce some pressure on consumer RAM demand
- hacker_homie 1mo agoSo I guess, 1. This chip for an 8B model even if it was done at 5nm would still be twice the size of a conventional CPU die so what are the yields for this going to be like for even a 30B model? 2. They say 2 months but llama 3.1 was released 2024, ~2 years which is normal lead time for silicon, I suspect this would take longer if the architecture is not llama? 3. Can google do the same thing in house with their Gemma 4 series (two year lead time puts Gemma 4 on silicon April 2028) ? I can see the benefit for hyper scalers but at the rate of model turn over does this type of investment make sense?
- harhargange 1mo agoExactly, what will be the size of big models? Maybe they aren’t targeting big models but where do they give size estimates?