5 ms·
Not a chip CEO, but I read this article and thought that they're working on some kind of application specific chip only for serving models. Similar to how an FP
by otterdude 3mo ago
Not a chip CEO, but I read this article and thought that they're working on some kind of application specific chip only for serving models. Similar to how an FPGA can optimize certain tasks.
Given constant weights / biases of a Transformer / DNN you could use pipelining to feed forward calculations through the array one layer at a time. For DNN's with thousands of layers you might see 1:1 speed up per layer channel.
I doubt they would undergo this process for marginal gains.
- xdavidliu 3mo agoi don't understand what the second paragraph is saying.
- otterdude 3mo agoBasically getting around the branch predictor problem with generalized compute architectures https://en.wikipedia.org/wiki/Branch_predictor https://en.wikipedia.org/wiki/Branch_predictor
- nine_k 3mo agoIn very crude terms, AFAICT, if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits this; would love to hear any ideas!
- cm2187 3mo agoRandom thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?
- 8note 3mo agohttps://www.anuragk.com/blog/posts/Taalas.html https://www.anuragk.com/blog/posts/Taalas.html
- lsaferite 3mo agohttps://taalas.com/ https://taalas.com/
- jwHollister 3mo agowow if they can get something like this working, what happens to all this infrastructure? Hyperscalers have to be assuming the lifespan of that stuff wrong considering the next gen will be 1000x more efficient.
- otterley 3mo agoThe question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.
- simondotau 3mo agoPresumably at some point the rapid progress of models will plateau, at least insofar as a model could be frozen in time and remain economically useful for the expected life of hardware. Especially if it comes with compelling benefits e.g. dramatically lower latency and/or dramatically higher performance per watt. If you can build chips that could run one specific LLM 100x faster than anything else, it would have a use case that nothing else could match.
- fragmede 3mo agohttps://www.cerebras.ai/ https://www.cerebras.ai/ is exactly that! Holy shit it's fast.
- guyomes 3mo ago> IDK how the custom hardware exploits this; would love to hear any ideas! You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2]. [1]: https://inria.hal.science/hal-04689673/document https://inria.hal.science/hal-04689673/document [2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.pdf https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...
- fulafel 3mo agoCurrent accelerators (TPUs, various onchip NPUs) are something close to this. Systolic array is the estabilished computer architecture term for flowing data from computation to computation without the overhead of a register file or von Neumann bottleneck.
- zgao 3mo agoYes, my statement was not about the quality or performance of the chip -- simply the tapeout timeline that was stated, by itself.
- kmacdough 3mo agoWith a striking lack of numbers, I'm not confident. I my experience, everything underspecified in a marketing release is unflattering. They're also not a chip designing company, but they're probably trying to keep up on the eyes of investors. As the article mentions, several of their competitors are chip designers and already have working procuction inference chips.
- SwellJoe 3mo agoWhen you have a few billion dollars you can hire chip people and partner with a chip company. That's not to say I expect they'll ship something competitive with Google's custom AI hardware on the first go, since Google has been at it for quite a while, but there's very few technical problems large sums of money won't solve.
- IX-103 3mo agoYeah, I'm not sure how competitive it is without any specs. Just from it being "inference only" that puts it on the same level as Google's 2015 TPUv1.