7 ms·

Groq surpasses 1,200 tokens/sec with Llama 3 8B

43 points by YourCupOTea 2y ago

LorenDB 2y ago
Groq is an insane company. SambaNova (discussed yesterday[0]) is also very promising. However, what I really want to see is local AI accelerator chips a la Tenstorrent Grayskull that can boost local generation to hundreds of tokens per second while being more efficient than GPUs. [0]: https://news.ycombinator.com/item?id=40508797 https://news.ycombinator.com/item?id=40508797
- frozenport 2y ago
  Samba is on gen 4 silicon and still lagging, somebody over there is doing something wrong
  - snhbsqub 2y ago
    How are they lagging? They are running faster than anyone else at full precision and with many many fewer chips than Groq. Groq is not real.
    - frozenport 2y ago
      Well I've been using the groq public api, and its approx. the rates claimed. Economics and costs are hard to predict. For example, Groq is not using HBM chips. So probably the cards are a lot easier to source. Its not clear what the capacity of these systems are in terms of total users, or even tokens per second. Then you factor in cost. Then you realize all vendors will match a competitors pricing. Then you realize Groq doesn't sell chips. ¯\_(ツ)_/¯ The only thing you have is the public API to benchmark against: https://artificialanalysis.ai/ https://artificialanalysis.ai/
      - snhbsqub 2y ago
        - Groq has exactly 0 dollars in revenue - Groq requires 576 chips to run a single model - Groq can do low latency inference, but can't handle batches, and can't run a diversity of different models on each deployment - Groq quantizes the models, significantly affecting quality to get more speed (and don't communicate this to end users, which is very deceptive) - Groq can only run inference, cannot train on their systems - SambaNova has real revenue from big customers - SambaNova can run any model on a single node at the speed Groq requires - SambaNova can do low latency inference just like Groq, but can also run large batches and host hundreds of models on a single deployment - SambaNova does not quantize models unless explicitly stated - SambaNova can run training at perf competitive with Nvidia, as well as fastest inference in the world at full precision It really isn't a competition. Groq has done great as garnering hype in recent months, but it is a house of cards.
        frozenport 2y ago
        I think semi analysis commented that they have pipelines instead of batches[1]. So every clock cycle you're doing useful work rather than loading up people into batches. And thats why the arch will probably win for inference, for training you're basically competing with software eco system and silicon density. AKA NVIDIA can give TSMC more money to get more ALUs on the die. I think other places have attempted dataflow (FPGA etc) but they all basically had buffers (due to non-determinism in networks stack and even ram). SambaNova seems indistinguishable from an FPGA with a few clock cycles difference. I think they blew their shot with a Series D ($600 million???) where they made more of the same old. Maybe Intel will buy them to augment Altera? Looks like chasing parity with existing strategies. I buy the Groq hype because its something different, certainly the public demo helped. HN is about the future. [1] https://www.semianalysis.com/p/groq-inference-tokenomics-speed-but https://www.semianalysis.com/p/groq-inference-tokenomics-spe...
windowshopping 2y ago
Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?
- Me1000 2y ago
  Completely unrelated.
- verdverm 2y ago
  Different companies and efforts Groq - mainly hardware, the LPU (https://wow.groq.com/lpu-inference-engine/ https://wow.groq.com/lpu-inference-engine/) Grok - Elon's de jour AI endeavor
- porphyra 2y ago
  Unrelated --- groq wrote an angry blog post complaining about Elon's xAI's grok: https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/ https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/
  - BowBun 2y ago
    Not that angry! I appreciate the tone, though they deserve every right to protect their trademark.
    - flutas 2y ago
      > though they deserve every right to protect their trademark. And they can, Twitter (why everything gets claimed as his personal work I never know) isn't using their trademark. As they (Groq) themselves have said... > the difference of one consonant (q, k) only matters to scrabblers and spell checkers Grok the term has been around since at least 1961.[0] The fact that a company decided to take a common term (especially in the CS field), change one letter and trademark it doesn't mean nobody can use the original spelling at all. Funnily enough, Groq is trying to claim grok and groq are not associated terms in court filings while trying to bully another company with the same name: > The word “Groq” essentially did not exist before Ross created it and has no known meaning in any language beyond its intended association with Groq, Inc. vs that companies reply > The word “grok” originated in Robert Heinlein’s 1961 novel Stranger in a Strange Land. Merriam Webster defines “grok” as “to understand profoundly and intuitively.” The Oxford English Dictionary defines “grok” as “[t]o understand intuitively or by empathy; to establish rapport with.” Once Groq realized their trademark didn't include healthcare data, they tried to trademark...the other companies name. [0]: https://en.wikipedia.org/wiki/Grok https://en.wikipedia.org/wiki/Grok
andy_xor_andrew 2y ago
When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration. My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt. But... then I tried their demo, and... yeah, it's that good. Such an amazing company of talented individuals.
- saberience 2y ago
  The issue is that their chips need a huge amount of server blades and there's a big doubt whether this model actually scales. That is, how will Groq handle much larger models with a context of hundreds of thousands or millions of tokens? Right now this would require them to deploy a cluster with thousands of chips, versus 10 chips for say an NVidia system. The other issue they don't mention is power, space, efficiency etc. We want to run larger models with less power, fewer server blades, at lower cost. Not use more server blades, more chips, more power, etc.
  - verdverm 2y ago
    Cerebrus faces similar challenges with their wafer scale chips. If anything, Google's TPU advancements chart a viable course. I suspect both Groq and Cerebrus will overcome the challenges and offer competitive compute options, depending on the context
    - snhbsqub 2y ago
      SambaNova is the only one if the chip startups that is viable. It surprises me that people don’t see this.
- frozenport 2y ago
  8 year old unicorn++ with a public demo sounds credible?
behnamoh 2y ago
They're not responsive to my questions on Twitter, so I'm asking here: When will Groq support a real API (not experimental beta preview)? When will Groq support logprobs?! When will Groq actually tell us what their rate limit is?! Until these aren't answered, many of us can't actually build on Groq. Edit: It seems I'm getting downvoted by Groq employees...
- porphyra 2y ago
  Try asking in the groq discord [0]. Some groq employees are fairly responsive there. For groqcloud the rate limits are fairly clear [1]. For example, for llama3-8b-8192 you get 30 requests per minute, 14400 per day, and 30000 tokens per minute. That said, it's the beta free tier so it sometimes goes down randomly and the limits may be different once they start charging for it. I'm not affiliated with groq but I use groqcloud to make some simple chatbots since it's currently free. [0] https://discord.com/invite/n8KtCjfAug https://discord.com/invite/n8KtCjfAug [1] https://console.groq.com/settings/limits https://console.groq.com/settings/limits
deleted 2y ago
[deleted]

43 points by YourCupOTea 2y ago

Groq is an insane company. SambaNova (discussed yesterday[0]) is also very promising. However, what I really want to see is local AI accelerator chips a la Tenstorrent Grayskull that can boost local generation to hundreds of tokens per second while being more efficient than GPUs. [0]: https://news.ycombinator.com/item?id=40508797 https://news.ycombinator.com/item?id=40508797

frozenport 2y ago
Samba is on gen 4 silicon and still lagging, somebody over there is doing something wrong
- snhbsqub 2y ago
  How are they lagging? They are running faster than anyone else at full precision and with many many fewer chips than Groq. Groq is not real.
  - frozenport 2y ago
    Well I've been using the groq public api, and its approx. the rates claimed. Economics and costs are hard to predict. For example, Groq is not using HBM chips. So probably the cards are a lot easier to source. Its not clear what the capacity of these systems are in terms of total users, or even tokens per second. Then you factor in cost. Then you realize all vendors will match a competitors pricing. Then you realize Groq doesn't sell chips. ¯\_(ツ)_/¯ The only thing you have is the public API to benchmark against: https://artificialanalysis.ai/ https://artificialanalysis.ai/
    - snhbsqub 2y ago
      - Groq has exactly 0 dollars in revenue - Groq requires 576 chips to run a single model - Groq can do low latency inference, but can't handle batches, and can't run a diversity of different models on each deployment - Groq quantizes the models, significantly affecting quality to get more speed (and don't communicate this to end users, which is very deceptive) - Groq can only run inference, cannot train on their systems - SambaNova has real revenue from big customers - SambaNova can run any model on a single node at the speed Groq requires - SambaNova can do low latency inference just like Groq, but can also run large batches and host hundreds of models on a single deployment - SambaNova does not quantize models unless explicitly stated - SambaNova can run training at perf competitive with Nvidia, as well as fastest inference in the world at full precision It really isn't a competition. Groq has done great as garnering hype in recent months, but it is a house of cards.
      - frozenport 2y ago
        I think semi analysis commented that they have pipelines instead of batches[1]. So every clock cycle you're doing useful work rather than loading up people into batches. And thats why the arch will probably win for inference, for training you're basically competing with software eco system and silicon density. AKA NVIDIA can give TSMC more money to get more ALUs on the die. I think other places have attempted dataflow (FPGA etc) but they all basically had buffers (due to non-determinism in networks stack and even ram). SambaNova seems indistinguishable from an FPGA with a few clock cycles difference. I think they blew their shot with a Series D ($600 million???) where they made more of the same old. Maybe Intel will buy them to augment Altera? Looks like chasing parity with existing strategies. I buy the Groq hype because its something different, certainly the public demo helped. HN is about the future. [1] https://www.semianalysis.com/p/groq-inference-tokenomics-speed-but https://www.semianalysis.com/p/groq-inference-tokenomics-spe...

windowshopping 2y ago

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Me1000 2y ago
Completely unrelated.
verdverm 2y ago
Different companies and efforts Groq - mainly hardware, the LPU (https://wow.groq.com/lpu-inference-engine/ https://wow.groq.com/lpu-inference-engine/) Grok - Elon's de jour AI endeavor
porphyra 2y ago
Unrelated --- groq wrote an angry blog post complaining about Elon's xAI's grok: https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/ https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/
- BowBun 2y ago
  Not that angry! I appreciate the tone, though they deserve every right to protect their trademark.
  - flutas 2y ago
    > though they deserve every right to protect their trademark. And they can, Twitter (why everything gets claimed as his personal work I never know) isn't using their trademark. As they (Groq) themselves have said... > the difference of one consonant (q, k) only matters to scrabblers and spell checkers Grok the term has been around since at least 1961.[0] The fact that a company decided to take a common term (especially in the CS field), change one letter and trademark it doesn't mean nobody can use the original spelling at all. Funnily enough, Groq is trying to claim grok and groq are not associated terms in court filings while trying to bully another company with the same name: > The word “Groq” essentially did not exist before Ross created it and has no known meaning in any language beyond its intended association with Groq, Inc. vs that companies reply > The word “grok” originated in Robert Heinlein’s 1961 novel Stranger in a Strange Land. Merriam Webster defines “grok” as “to understand profoundly and intuitively.” The Oxford English Dictionary defines “grok” as “[t]o understand intuitively or by empathy; to establish rapport with.” Once Groq realized their trademark didn't include healthcare data, they tried to trademark...the other companies name. [0]: https://en.wikipedia.org/wiki/Grok https://en.wikipedia.org/wiki/Grok

andy_xor_andrew 2y ago

When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration. My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt. But... then I tried their demo, and... yeah, it's that good. Such an amazing company of talented individuals.

saberience 2y ago
The issue is that their chips need a huge amount of server blades and there's a big doubt whether this model actually scales. That is, how will Groq handle much larger models with a context of hundreds of thousands or millions of tokens? Right now this would require them to deploy a cluster with thousands of chips, versus 10 chips for say an NVidia system. The other issue they don't mention is power, space, efficiency etc. We want to run larger models with less power, fewer server blades, at lower cost. Not use more server blades, more chips, more power, etc.
- verdverm 2y ago
  Cerebrus faces similar challenges with their wafer scale chips. If anything, Google's TPU advancements chart a viable course. I suspect both Groq and Cerebrus will overcome the challenges and offer competitive compute options, depending on the context
  - snhbsqub 2y ago
    SambaNova is the only one if the chip startups that is viable. It surprises me that people don’t see this.
frozenport 2y ago
8 year old unicorn++ with a public demo sounds credible?

behnamoh 2y ago

They're not responsive to my questions on Twitter, so I'm asking here: When will Groq support a real API (not experimental beta preview)? When will Groq support logprobs?! When will Groq actually tell us what their rate limit is?! Until these aren't answered, many of us can't actually build on Groq. Edit: It seems I'm getting downvoted by Groq employees...

porphyra 2y ago
Try asking in the groq discord [0]. Some groq employees are fairly responsive there. For groqcloud the rate limits are fairly clear [1]. For example, for llama3-8b-8192 you get 30 requests per minute, 14400 per day, and 30000 tokens per minute. That said, it's the beta free tier so it sometimes goes down randomly and the limits may be different once they start charging for it. I'm not affiliated with groq but I use groqcloud to make some simple chatbots since it's currently free. [0] https://discord.com/invite/n8KtCjfAug https://discord.com/invite/n8KtCjfAug [1] https://console.groq.com/settings/limits https://console.groq.com/settings/limits

deleted 2y ago

[deleted]