7 ms·

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3,

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server rack density) and serve these models reasonably well. It might work well if you have a single model with lots of customers, but as soon as you need more than a single model and a lot of finetunes/high rank LoRAs etc., these won't be usable. Or for any on-prem deployment since the main advantage is consolidating people to use the same model, together. [0]: https://wow.groq.com/groqcard-accelerator/ https://wow.groq.com/groqcard-accelerator/ [1]: https://twitter.com/tomjaguarpaw/status/1759615563586744334 https://twitter.com/tomjaguarpaw/status/1759615563586744334

pclmulqdq 3y ago
Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher throughput as batch size gets into the 100's.
- nabakin 3y ago
  I've been thinking the same but on the other hand, that would mean they are operating at a huge loss which doesn't scale
- foundval 3y ago
  (Groq Employee) It's hard to discuss Tok/sec/$ outside of the context of a hardware sales engagement. This is because the relationship between Tok/s/u, Tok/s/system, Batching, and Pipelining is a complex one that involves compute utilization, network utilization, and (in particular) a host of compilation techniques that we wouldn't want to share publicly. Maybe we'll get to that level of transparency at some point, though! As far as Batching goes, you should consider that with synchronous systems, if all the stars align, Batch=1 is all you need. Of course, the devil is in the details, and sometimes small batch numbers still give you benefits. But Batch 100's generally gives no advantages. In fact, the entire point of developing deterministic hardware and synchronous systems is to avoid batching in the first place.
- frozenport 3y ago
  I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1 I guess if you don't have any extra junk you can pack more processing into the chip?
  - foundval 3y ago
    (Groq Employee) Yes! Determinism + Simplicity are superpowers for ALU and interconnect utilization rates. This system is powered by 14nm chips, and even the interconnects aren't best in class. We're just that much better at squeezing tokens out of transistors and optic cables than GPUs are - and you can imagine the implications on Watt/Token. Anyways.. wait until you see our 4nm. :)
tome 3y ago
If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantees disappear very quickly.
- pclmulqdq 3y ago
  I don't know about HBM specifically, but DDR and GDDR at a protocol level are both deterministic. It's the memory controller doing a bunch of reordering that makes them non-deterministic. Presumably, if that is the reason you don't like DRAM, you could build your compiler to be memory-layout aware and have the memory controller issue commands without reordering.
  - tome 3y ago
    That could be possible. It's out of my area of expertise so I can't say for sure. My understanding was HBM forces on you specific access patterns and non-deterministic delays. Our compiler already deals with many other forms of resource-aware scheduling so it could take into account DRAM refreshes easily, so I feel like there must be something else that makes SRAM more suitable in our case. I'll have to leave that to someone more knowledgeable to explain though ...
  - johntb86 3y ago
    Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
    - pclmulqdq 3y ago
      You can control when those happen, too.
      - Temporary_31337 3y ago
matanyall 3y ago
Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configuration. Does that mean that factories are inefficient? You also say that H200's work reasonably well, and that's reasonable (but debatable) for synchronous, human interaction use cases. Show me a 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia.
- pbalcer 3y ago
  Just curious, how does this work out in terms of TCO (even assuming the price of a Groq LPU is 0$)? What you say makes sense, but I'm wondering how you strike a balance between massive horizontal scaling vs vertical scaling. Sometimes (quite often in my experience) having a few beefy servers is much simpler/cheaper/faster than scaling horizontally across many small nodes. Or I got this completely wrong, and your solution enables use-cases that are simply unattainable on mainstream (Nvidia/AMD) hardware, making TCO argument less relevant?
  - tome 3y ago
    We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.
    - nickpsecurity 3y ago
      Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior results in other low-latency domains.
moralestapia 3y ago
>The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM [...] IDGAF about any of that, lol. I just want an API endpoint. 480 tokens/sec at $0.27 per million tokens? Sign me in, I don't care about their hardware, at all.
- treesciencebot 3y ago
  there are providers out there offering for $0 per million tokens, that doesn't mean it is sustainable and won't disappear as soon as the VC well runs dry. Am not saying this is the case for Groq, but in general you probably should care if you want to build something serious on top of anything.
  - foundval 3y ago
    (Groq Employee) Agreed, one should care, and especially since this particular service is very differentiated by its speed and has no competitors. That being said, until there's another option at anywhere that speed.. That point is moot, isn't it :) For now, Groq is the only option that can let you build an UX with near-instant response times. Or a live agents that help with a human-to-human interaction. I could go on and on about the product categories this opens.
    - bethekind 3y ago
      Why go so fast? Aren't Nvidias products fast enough from a TPS perspective?
      - mike_hearn 3y ago
        OpenAI have a voice powered chat mode in their app and there's a noticeable delay of a few seconds between finishing your sentence and the bot starting to speak. I think the problem is that for realistic TTS you need quite a few tokens because the prosody can be affected by tokens that come a fair bit further down the sentence, consider the difference in pitch between: "The war will be long and bloody" vs "The war will be long and bloody?" So to begin TTS you need quite a lot of tokens, which in turn means you have to digest the prompt and run a whole bunch of forward passes before you can start rendering. And of course you have to keep up with the speed of regular speech, which OpenAI sometimes struggles with. That said, the gap isn't huge. Many apps won't need it. Some use cases where low latency might matter: - Phone support. - Trading. Think digesting a press release into an action a few seconds faster than your competitors. - Agents that listen in to conversations and "butt in" when they have something useful to say. - RPGs where you can talk to NPCs in realtime. - Real-time analysis of whatever's on screen on your computing device. - Auto-completion. - Using AI as a general command prompt. Think AI bash. Undoubtably there will be a lot more though. When you give people performance, they find ways to use it.
trsohmers 3y ago
Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these performance numbers would exceed several million dollars, so depreciating that over expected usage at the theoretical prices they have posted seems impractical, so from an actual performance per dollar standpoint they don’t seem viable, but do have a very cool demo of an insane level of performance if you throw cost concerns out the window. [0]: https://www.nextplatform.com/2023/11/27/groq-says-it-can-deploy-1-million-ai-inference-chips-in-two-years/?amp https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...
- tome 3y ago
  Thomas, I think for full disclosure you should also state that you left Groq to start a competitor (a competitor which doesn't have the world's lowest latency LLM engine nor a guarantee to match the cheapest per token prices, like Groq does.). Anyone with a serious interest in the total cost of ownership of Groq's system is welcome to email contact@groq.com.
  - trsohmers 3y ago
    I thought that was clear through my profile, but yes, Positron AI is focused on providing the best performance per dollar while providing the best quality of service and capabilities rather than just focusing on a single metric of speed. A guarantee to match the cheapest per token prices is sure a great way to lose a race to the bottom, but I do wish Groq (and everyone else trying to compete against NVIDIA) the greatest luck and success. I really do think that the great single batch/user performance by Groq is a great demo, but is not the best solution for a wide variety of applications, but I hope it can find its niche.
- Aeolun 3y ago
  I think that just means it’s for people that really want it? John doe and his friends will never have a need to have their fart jokes generated at this speed, and are more interested in low costs. But we’d recently been doing call center operations and being able to quickly figure out what someone said was a major issue. You kind of don’t want your system to wait for a second before responding each time. I can imagine it making sense if it reduces the latency to 10ms there as well. Though you might still run up against the ‘good enough’ factor. I guess few people want to spend millions to go from 1000ms to 10ms, but when they do they really want it.
imtringued 3y ago
I honestly don't see the problem. "just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.
- treesciencebot 3y ago
  per-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
  - QuadmasterXLII 3y ago
    load model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response, I would expect the model loading to take basically zero percent of the time in the above workflow
londons_explore 3y ago
> more than a single model and a lot of finetunes/high rank LoRAs I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone. The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.
- kcorbitt 3y ago
  This actually already exists! We did a writeup of the relevant optimizations here: https://openpipe.ai/blog/s-lora https://openpipe.ai/blog/s-lora
- azeirah 3y ago
  I recall a recent discussion about a technique to load the diff in weights between a lora and base model, zip it and transfer it on a per-needs basis.
- xzyaoi 3y ago
  There's also papers for hosting full-parameter fine-tuned models: https://arxiv.org/abs/2312.05215 https://arxiv.org/abs/2312.05215 Disclaimer: I'm one of the authors.

pclmulqdq 3y ago

Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher throughput as batch size gets into the 100's.

nabakin 3y ago
I've been thinking the same but on the other hand, that would mean they are operating at a huge loss which doesn't scale
foundval 3y ago
(Groq Employee) It's hard to discuss Tok/sec/$ outside of the context of a hardware sales engagement. This is because the relationship between Tok/s/u, Tok/s/system, Batching, and Pipelining is a complex one that involves compute utilization, network utilization, and (in particular) a host of compilation techniques that we wouldn't want to share publicly. Maybe we'll get to that level of transparency at some point, though! As far as Batching goes, you should consider that with synchronous systems, if all the stars align, Batch=1 is all you need. Of course, the devil is in the details, and sometimes small batch numbers still give you benefits. But Batch 100's generally gives no advantages. In fact, the entire point of developing deterministic hardware and synchronous systems is to avoid batching in the first place.
frozenport 3y ago
I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1 I guess if you don't have any extra junk you can pack more processing into the chip?
- foundval 3y ago
  (Groq Employee) Yes! Determinism + Simplicity are superpowers for ALU and interconnect utilization rates. This system is powered by 14nm chips, and even the interconnects aren't best in class. We're just that much better at squeezing tokens out of transistors and optic cables than GPUs are - and you can imagine the implications on Watt/Token. Anyways.. wait until you see our 4nm. :)

tome 3y ago

If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantees disappear very quickly.

pclmulqdq 3y ago
I don't know about HBM specifically, but DDR and GDDR at a protocol level are both deterministic. It's the memory controller doing a bunch of reordering that makes them non-deterministic. Presumably, if that is the reason you don't like DRAM, you could build your compiler to be memory-layout aware and have the memory controller issue commands without reordering.
- tome 3y ago
  That could be possible. It's out of my area of expertise so I can't say for sure. My understanding was HBM forces on you specific access patterns and non-deterministic delays. Our compiler already deals with many other forms of resource-aware scheduling so it could take into account DRAM refreshes easily, so I feel like there must be something else that makes SRAM more suitable in our case. I'll have to leave that to someone more knowledgeable to explain though ...
- johntb86 3y ago
  Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
  - pclmulqdq 3y ago
    You can control when those happen, too.
    - Temporary_31337 3y ago

matanyall 3y ago

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configuration. Does that mean that factories are inefficient? You also say that H200's work reasonably well, and that's reasonable (but debatable) for synchronous, human interaction use cases. Show me a 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia.

pbalcer 3y ago
Just curious, how does this work out in terms of TCO (even assuming the price of a Groq LPU is 0$)? What you say makes sense, but I'm wondering how you strike a balance between massive horizontal scaling vs vertical scaling. Sometimes (quite often in my experience) having a few beefy servers is much simpler/cheaper/faster than scaling horizontally across many small nodes. Or I got this completely wrong, and your solution enables use-cases that are simply unattainable on mainstream (Nvidia/AMD) hardware, making TCO argument less relevant?
- tome 3y ago
  We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.
  - nickpsecurity 3y ago
    Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior results in other low-latency domains.

moralestapia 3y ago

>The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM [...] IDGAF about any of that, lol. I just want an API endpoint. 480 tokens/sec at $0.27 per million tokens? Sign me in, I don't care about their hardware, at all.

treesciencebot 3y ago
there are providers out there offering for $0 per million tokens, that doesn't mean it is sustainable and won't disappear as soon as the VC well runs dry. Am not saying this is the case for Groq, but in general you probably should care if you want to build something serious on top of anything.
- foundval 3y ago
  (Groq Employee) Agreed, one should care, and especially since this particular service is very differentiated by its speed and has no competitors. That being said, until there's another option at anywhere that speed.. That point is moot, isn't it :) For now, Groq is the only option that can let you build an UX with near-instant response times. Or a live agents that help with a human-to-human interaction. I could go on and on about the product categories this opens.
  - bethekind 3y ago
    Why go so fast? Aren't Nvidias products fast enough from a TPS perspective?
    - mike_hearn 3y ago
      OpenAI have a voice powered chat mode in their app and there's a noticeable delay of a few seconds between finishing your sentence and the bot starting to speak. I think the problem is that for realistic TTS you need quite a few tokens because the prosody can be affected by tokens that come a fair bit further down the sentence, consider the difference in pitch between: "The war will be long and bloody" vs "The war will be long and bloody?" So to begin TTS you need quite a lot of tokens, which in turn means you have to digest the prompt and run a whole bunch of forward passes before you can start rendering. And of course you have to keep up with the speed of regular speech, which OpenAI sometimes struggles with. That said, the gap isn't huge. Many apps won't need it. Some use cases where low latency might matter: - Phone support. - Trading. Think digesting a press release into an action a few seconds faster than your competitors. - Agents that listen in to conversations and "butt in" when they have something useful to say. - RPGs where you can talk to NPCs in realtime. - Real-time analysis of whatever's on screen on your computing device. - Auto-completion. - Using AI as a general command prompt. Think AI bash. Undoubtably there will be a lot more though. When you give people performance, they find ways to use it.

trsohmers 3y ago

Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these performance numbers would exceed several million dollars, so depreciating that over expected usage at the theoretical prices they have posted seems impractical, so from an actual performance per dollar standpoint they don’t seem viable, but do have a very cool demo of an insane level of performance if you throw cost concerns out the window. [0]: https://www.nextplatform.com/2023/11/27/groq-says-it-can-deploy-1-million-ai-inference-chips-in-two-years/?amp https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...

tome 3y ago
Thomas, I think for full disclosure you should also state that you left Groq to start a competitor (a competitor which doesn't have the world's lowest latency LLM engine nor a guarantee to match the cheapest per token prices, like Groq does.). Anyone with a serious interest in the total cost of ownership of Groq's system is welcome to email contact@groq.com.
- trsohmers 3y ago
  I thought that was clear through my profile, but yes, Positron AI is focused on providing the best performance per dollar while providing the best quality of service and capabilities rather than just focusing on a single metric of speed. A guarantee to match the cheapest per token prices is sure a great way to lose a race to the bottom, but I do wish Groq (and everyone else trying to compete against NVIDIA) the greatest luck and success. I really do think that the great single batch/user performance by Groq is a great demo, but is not the best solution for a wide variety of applications, but I hope it can find its niche.
Aeolun 3y ago
I think that just means it’s for people that really want it? John doe and his friends will never have a need to have their fart jokes generated at this speed, and are more interested in low costs. But we’d recently been doing call center operations and being able to quickly figure out what someone said was a major issue. You kind of don’t want your system to wait for a second before responding each time. I can imagine it making sense if it reduces the latency to 10ms there as well. Though you might still run up against the ‘good enough’ factor. I guess few people want to spend millions to go from 1000ms to 10ms, but when they do they really want it.

imtringued 3y ago

I honestly don't see the problem. "just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.

treesciencebot 3y ago
per-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
- QuadmasterXLII 3y ago
  load model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response, I would expect the model loading to take basically zero percent of the time in the above workflow

londons_explore 3y ago

> more than a single model and a lot of finetunes/high rank LoRAs I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone. The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.

kcorbitt 3y ago
This actually already exists! We did a writeup of the relevant optimizations here: https://openpipe.ai/blog/s-lora https://openpipe.ai/blog/s-lora
azeirah 3y ago
I recall a recent discussion about a technique to load the diff in weights between a lora and base model, zip it and transfer it on a per-needs basis.
xzyaoi 3y ago
There's also papers for hosting full-parameter fine-tuned models: https://arxiv.org/abs/2312.05215 https://arxiv.org/abs/2312.05215 Disclaimer: I'm one of the authors.