7 ms·

Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip t

Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this. Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a month, start to finish, for the physical processing. Maybe once LLM improvements asymptote further?

kushie 22d ago
tapeout could shrink but days per mask layer (DPML) does not have much margin..
- smallmancontrov 22d ago
  I'm not in industry, is DPML (which I assume is the time required to make a mask?) set by electron beam scan time or something?
vineyardmike 22d ago
How much of that 16mo is design versus just production? If there was a “plug and play” chip where you just BYO weights, how long would it take? The bigger issue seems to be that these chips can’t hold that many weights at the moment. (I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)
- RealityVoid 22d ago
  Talaas, from what I understand is building stuff just like that. The infra is the same and the weights layer is all you need to change. I guess you could half etch the chips and then finish them with the weights only. I think their turnaround is 6-8 Weeks. The size of the models fitting on the chips at the moment is llama 3 I think?
  - derefr 22d ago
    > I guess you could half etch the chips and then finish them with the weights only. Basically a https://en.wikipedia.org/wiki/Gate_array https://en.wikipedia.org/wiki/Gate_array. (The non-field-programmable kind.)
jeremyjh 22d ago
I think Sol is already good enough though.
- basilgohar 22d ago
  "640k (token context) should be enough for anyone."
  - jerf 22d ago
    I know what you're saying, but modulo things like losing track of what year it is as time passes by, a current frontier model is going to continue to be useful for many tasks for many years, even moreso if it's 5-10x faster due to the chip architecture. It's not that it would be the best forever, it's that it would be useful for plenty long enough to be worthwhile, even if there was better stuff available. In exactly the same way that this computer I'm typing this message on is not the latest and hottest cutting edge stuff. A 7 year old CPU, 7 year old Intel integrated graphics, an older NVMe disk, a mere 32GB of RAM... ok, that's one spec that's still pretty modern although it is slower RAM... but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge.
    - throwuxiytayq 22d ago
      > but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge While it’s still too early to tell, I don’t think that’s how intelligence scales. Better models get you better solutions even to trivial problems. The ceiling for getting it done better is very high even if you’re not doing anything complicated. And difficulty isn’t uniformly distributed anyway - it seems to me that “mostly simple” tasks often have annoying 1% tails that low-intelligence models struggle with. I think we’ll see people chasing the top models for quite a while, or indefinitely - depending on the cost curve.
    - dgently7 22d ago
      exactly, but the "goes out of date" is bad when we talk about software.. but this isnt software, its hardware. the youd have to buy a new one to get a better model is a FEATURE not a bug. like if im apple... and i can put a sol level llm in an iphone, market it as privacy first you own your data personal assistant, integrate it all over the os... and then when there is a better model/siri make all the users buy a new phone... thats how they "win" ai. the old standbys of better screens thinner cameras and batteries arent enough anymore. its basically tapped out. all modern phones are as thin as they need as big as they need as fast as they need and last all day on a battery... apple needs a new number to up thing that people can actually feel/see. model generations could be it... every year faster, smarter, more capbilities and integrations.
kurthr 22d ago
The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below. https://www.eetimes.com/taalas-specializes-to-extremes-for-extraordinary-token-speed/ https://www.eetimes.com/taalas-specializes-to-extremes-for-e... https://www.turingpost.com/p/taalas https://www.turingpost.com/p/taalas https://cambrian-ai.com/taalas-launches-hardcore-chip-with-insane-ai-inference-performance/ https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i... Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.
- MBCook 22d ago
  But that means your different chips all have different sets of weights and are different generations. If none of that is baked into the chip as now then all the chips are running the latest weights every time. Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.
  - geysersam 22d ago
    Why would it be useless in 3 months?
    - imtringued 22d ago
      Because on hacker news the only thing that matters is being in the current news cycle and not whether your business is profitable.
  - Certhas 22d ago
thoughtbefore 22d ago
It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer? If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.
- calebkaiser 22d ago
  Google rolled out TPUs in 2015. AWS released Inferentia and Trainium chips in 2020. If companies working on ML-specific chips was evidence that large transformer models have fully saturated their potential, the field would have been done circa GPT-2.
  - edgyquant 21d ago
    Neither of those companies core business model was serving llms
    - calebkaiser 21d ago
      What? Both of those companies absolutely serve LLMs, and both of them would love for serving LLMs to be an even bigger part of their business. Not only that, AWS is Anthropic's primary compute partner for training and inference. They literally use the newest generation of the Trainium chips I mentioned before: https://www.anthropic.com/news/anthropic-amazon-compute https://www.anthropic.com/news/anthropic-amazon-compute Chips are another axis for improvements in training and inference. Orgs large enough to explore the space have been doing it for at least a decade now. This is just a silly line of reasoning based on the faulty assumption that somehow, looking for increases in efficiency in training/inference means teams have reached some theoretical limit in model capability.
      - edgyquant 20d ago
        Notice how I used the words “core business” but that they didn’t do business at all
tintor 22d ago
They could etch the model architecture, without the weights into the chip. This way newly post-trained model can be loaded and served the same day.
- deleted 21d ago
  [deleted]
- deleted 21d ago
  [deleted]
nerdsniper 22d ago
> Maybe once LLM improvements asymptote further? Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".
lelanthran 22d ago
> Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). Yeah, with any luck it would put pressure on Nvidia to charge less, and not just to OpenAI. With a little more luck, we would see all the other players do the same thing, driving down the price of actual GPUs from GPU manufacturers.
thesz 21d ago
I made some analysis half a year ago: https://news.ycombinator.com/item?id=47109252 https://news.ycombinator.com/item?id=47109252 It appears that to have working ASIC with the LLM baked into it we need to place and route macroblocks, and not a great variety of them. These macroblocks can be pre-placed-and-routed, available as masks already and shared between different LLMs. Thus it appears that the tapeout delay can be substantially lower than a year.

kushie 22d ago

tapeout could shrink but days per mask layer (DPML) does not have much margin..

smallmancontrov 22d ago
I'm not in industry, is DPML (which I assume is the time required to make a mask?) set by electron beam scan time or something?

vineyardmike 22d ago

How much of that 16mo is design versus just production? If there was a “plug and play” chip where you just BYO weights, how long would it take? The bigger issue seems to be that these chips can’t hold that many weights at the moment. (I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)

RealityVoid 22d ago
Talaas, from what I understand is building stuff just like that. The infra is the same and the weights layer is all you need to change. I guess you could half etch the chips and then finish them with the weights only. I think their turnaround is 6-8 Weeks. The size of the models fitting on the chips at the moment is llama 3 I think?
- derefr 22d ago
  > I guess you could half etch the chips and then finish them with the weights only. Basically a https://en.wikipedia.org/wiki/Gate_array https://en.wikipedia.org/wiki/Gate_array. (The non-field-programmable kind.)

jeremyjh 22d ago

I think Sol is already good enough though.

basilgohar 22d ago
"640k (token context) should be enough for anyone."
- jerf 22d ago
  I know what you're saying, but modulo things like losing track of what year it is as time passes by, a current frontier model is going to continue to be useful for many tasks for many years, even moreso if it's 5-10x faster due to the chip architecture. It's not that it would be the best forever, it's that it would be useful for plenty long enough to be worthwhile, even if there was better stuff available. In exactly the same way that this computer I'm typing this message on is not the latest and hottest cutting edge stuff. A 7 year old CPU, 7 year old Intel integrated graphics, an older NVMe disk, a mere 32GB of RAM... ok, that's one spec that's still pretty modern although it is slower RAM... but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge.
  - throwuxiytayq 22d ago
    > but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge While it’s still too early to tell, I don’t think that’s how intelligence scales. Better models get you better solutions even to trivial problems. The ceiling for getting it done better is very high even if you’re not doing anything complicated. And difficulty isn’t uniformly distributed anyway - it seems to me that “mostly simple” tasks often have annoying 1% tails that low-intelligence models struggle with. I think we’ll see people chasing the top models for quite a while, or indefinitely - depending on the cost curve.
  - dgently7 22d ago
    exactly, but the "goes out of date" is bad when we talk about software.. but this isnt software, its hardware. the youd have to buy a new one to get a better model is a FEATURE not a bug. like if im apple... and i can put a sol level llm in an iphone, market it as privacy first you own your data personal assistant, integrate it all over the os... and then when there is a better model/siri make all the users buy a new phone... thats how they "win" ai. the old standbys of better screens thinner cameras and batteries arent enough anymore. its basically tapped out. all modern phones are as thin as they need as big as they need as fast as they need and last all day on a battery... apple needs a new number to up thing that people can actually feel/see. model generations could be it... every year faster, smarter, more capbilities and integrations.

kurthr 22d ago

The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below. https://www.eetimes.com/taalas-specializes-to-extremes-for-extraordinary-token-speed/ https://www.eetimes.com/taalas-specializes-to-extremes-for-e... https://www.turingpost.com/p/taalas https://www.turingpost.com/p/taalas https://cambrian-ai.com/taalas-launches-hardcore-chip-with-insane-ai-inference-performance/ https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i... Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.

MBCook 22d ago
But that means your different chips all have different sets of weights and are different generations. If none of that is baked into the chip as now then all the chips are running the latest weights every time. Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.
- geysersam 22d ago
  Why would it be useless in 3 months?
  - imtringued 22d ago
    Because on hacker news the only thing that matters is being in the current news cycle and not whether your business is profitable.
- Certhas 22d ago

thoughtbefore 22d ago

It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer? If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.

calebkaiser 22d ago
Google rolled out TPUs in 2015. AWS released Inferentia and Trainium chips in 2020. If companies working on ML-specific chips was evidence that large transformer models have fully saturated their potential, the field would have been done circa GPT-2.
- edgyquant 21d ago
  Neither of those companies core business model was serving llms
  - calebkaiser 21d ago
    What? Both of those companies absolutely serve LLMs, and both of them would love for serving LLMs to be an even bigger part of their business. Not only that, AWS is Anthropic's primary compute partner for training and inference. They literally use the newest generation of the Trainium chips I mentioned before: https://www.anthropic.com/news/anthropic-amazon-compute https://www.anthropic.com/news/anthropic-amazon-compute Chips are another axis for improvements in training and inference. Orgs large enough to explore the space have been doing it for at least a decade now. This is just a silly line of reasoning based on the faulty assumption that somehow, looking for increases in efficiency in training/inference means teams have reached some theoretical limit in model capability.
    - edgyquant 20d ago
      Notice how I used the words “core business” but that they didn’t do business at all

tintor 22d ago

They could etch the model architecture, without the weights into the chip. This way newly post-trained model can be loaded and served the same day.

deleted 21d ago
[deleted]
deleted 21d ago
[deleted]

nerdsniper 22d ago

> Maybe once LLM improvements asymptote further? Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".

lelanthran 22d ago

> Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). Yeah, with any luck it would put pressure on Nvidia to charge less, and not just to OpenAI. With a little more luck, we would see all the other players do the same thing, driving down the price of actual GPUs from GPU manufacturers.

thesz 21d ago

I made some analysis half a year ago: https://news.ycombinator.com/item?id=47109252 https://news.ycombinator.com/item?id=47109252 It appears that to have working ASIC with the LLM baked into it we need to place and route macroblocks, and not a great variety of them. These macroblocks can be pre-placed-and-routed, available as masks already and shared between different LLMs. Thus it appears that the tapeout delay can be substantially lower than a year.