6 ms·

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/0

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

moneywoes 3y ago
what’s the cost?
- jkachmar 3y ago
  right now we’re providing this access to public, anonymous users via this demo chat interface as an alpha test. we’ll be publishing information about API access, and pricing, shortly after the new year.
  - Frummy 3y ago
    You should add latex rendering
  - tome 3y ago
    Yup, we will be price competitive with OpenAI, and much faster!
mlazos 3y ago
How many chips are used for this demo? Do they have dram? I remember the earlier versions did not have dram. Are they also used for training or just inference?
- tome 3y ago
  I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total. Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!
  - convexstrictly 3y ago
    Could you explain the blockers to getting back-propagation working well on your chips?
    - tome 3y ago
      Our language processors have much lower latency and higher throughput than graphics processors so we have a massive advantage when it comes to inference. For language models particularly, time to first token is hugely important (and will probably become even more important as people start combining models to do novel things). Additionally, you probably care mostly about batch size 1. For training, latency is not the key issue. You generally want raw compute with a larger batch size. Backpropagation is just a numerical computation so you can certainly implement it on language processors, but the stark advantage we have over graphics processors in inference wouldn't carry over to training. Does that answer your question?
      - convexstrictly 3y ago
        Everything you say makes sense. Training is definitely more compute intensive than inference. Training is both memory throughput and compute constrained. Much research in speeding up training goes into optimizing HBM to SRAM communication. The equivalent for your chips would be communication from the SRAM of one chip to the SRAM of another, where it sounds like your architecture has a major memory throughput advantage over GPUs. So I assume you don't have a proportional compute advantage? By the way, it's great to see a non von Neumann architecture showing a major performance advantage in a real world application. And your chips are conceptually equivalent to chiplets; you should have a major cost advantage on bleeding edge process nodes if you scale up manufacturing. Overall very impressive!
        tome 3y ago
        I'm not an expert on the system architecture side of things. Maybe a Groqster who is can chime in. But the way I understand it is that you can't improve latency just by scaling, whereas you can improve throughput just by scaling, as long as it's acceptable to increase batch size. Increasing batch size is generally fine for training. It's a batch process! On the other hand, if someone comes up with a novel training process that is highly sequential then I'd expect Groq chips to do better than graphics processors in that scenario.
m1sta_ 3y ago
How easy is it for companies to setup private local servers using Grow hardware (cost and complexity). I've got money. I want throughout.
- tome 3y ago
  We've built and deployed racks at a number of organizations. Can you write a message to sales explaining your needs? https://groq.com/contact/ https://groq.com/contact/ Or if you give me your contact details I can pass them on.
cicce19 3y ago
Will you be selling individual cards? Are you looking for use cases in the healthcare vertical (noticed its not on your current list)? Working in the medical imaging space and could use this tech as part of the offering. Reach out at 16bit.ai
- tome 3y ago
  You can buy individual cards. For example Bittware is a reseller: https://www.bittware.com/products/groq/ https://www.bittware.com/products/groq/ But it might be best if you just contact us to explain your needs: https://groq.com/contact/ https://groq.com/contact/ I can also pass your details on to our sales team.
m3kw9 3y ago
Is it fixed to a certain llm architecture like llama2? How does it deal with different architectures like MOE for example
- tome 3y ago
  It's not fixed and our chip wasn't designed with LLMs in mind. It's a general purpose, low latency, high throughput compute fabric. Our compiler toolchain is also general purpose and can compile arbitrary high performance numerical programs without the need for handwritten kernels. Because of the current importance of ML/AI we're focusing on PyTorch and ONNX models as input, but it really could be anything. We can also deploy speech models like Whisper, for example, or image generation models. I don't know if we have any MOE architectures, but we'll be implementing Mixtral soon for sure!

moneywoes 3y ago

what’s the cost?

jkachmar 3y ago
right now we’re providing this access to public, anonymous users via this demo chat interface as an alpha test. we’ll be publishing information about API access, and pricing, shortly after the new year.
- Frummy 3y ago
  You should add latex rendering
- tome 3y ago
  Yup, we will be price competitive with OpenAI, and much faster!

mlazos 3y ago

How many chips are used for this demo? Do they have dram? I remember the earlier versions did not have dram. Are they also used for training or just inference?

tome 3y ago
I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total. Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!
- convexstrictly 3y ago
  Could you explain the blockers to getting back-propagation working well on your chips?
  - tome 3y ago
    Our language processors have much lower latency and higher throughput than graphics processors so we have a massive advantage when it comes to inference. For language models particularly, time to first token is hugely important (and will probably become even more important as people start combining models to do novel things). Additionally, you probably care mostly about batch size 1. For training, latency is not the key issue. You generally want raw compute with a larger batch size. Backpropagation is just a numerical computation so you can certainly implement it on language processors, but the stark advantage we have over graphics processors in inference wouldn't carry over to training. Does that answer your question?
    - convexstrictly 3y ago
      Everything you say makes sense. Training is definitely more compute intensive than inference. Training is both memory throughput and compute constrained. Much research in speeding up training goes into optimizing HBM to SRAM communication. The equivalent for your chips would be communication from the SRAM of one chip to the SRAM of another, where it sounds like your architecture has a major memory throughput advantage over GPUs. So I assume you don't have a proportional compute advantage? By the way, it's great to see a non von Neumann architecture showing a major performance advantage in a real world application. And your chips are conceptually equivalent to chiplets; you should have a major cost advantage on bleeding edge process nodes if you scale up manufacturing. Overall very impressive!
      - tome 3y ago
        I'm not an expert on the system architecture side of things. Maybe a Groqster who is can chime in. But the way I understand it is that you can't improve latency just by scaling, whereas you can improve throughput just by scaling, as long as it's acceptable to increase batch size. Increasing batch size is generally fine for training. It's a batch process! On the other hand, if someone comes up with a novel training process that is highly sequential then I'd expect Groq chips to do better than graphics processors in that scenario.

m1sta_ 3y ago

How easy is it for companies to setup private local servers using Grow hardware (cost and complexity). I've got money. I want throughout.

tome 3y ago
We've built and deployed racks at a number of organizations. Can you write a message to sales explaining your needs? https://groq.com/contact/ https://groq.com/contact/ Or if you give me your contact details I can pass them on.

cicce19 3y ago

Will you be selling individual cards? Are you looking for use cases in the healthcare vertical (noticed its not on your current list)? Working in the medical imaging space and could use this tech as part of the offering. Reach out at 16bit.ai

tome 3y ago
You can buy individual cards. For example Bittware is a reseller: https://www.bittware.com/products/groq/ https://www.bittware.com/products/groq/ But it might be best if you just contact us to explain your needs: https://groq.com/contact/ https://groq.com/contact/ I can also pass your details on to our sales team.

m3kw9 3y ago

Is it fixed to a certain llm architecture like llama2? How does it deal with different architectures like MOE for example

tome 3y ago
It's not fixed and our chip wasn't designed with LLMs in mind. It's a general purpose, low latency, high throughput compute fabric. Our compiler toolchain is also general purpose and can compile arbitrary high performance numerical programs without the need for handwritten kernels. Because of the current importance of ML/AI we're focusing on PyTorch and ONNX models as input, but it really could be anything. We can also deploy speech models like Whisper, for example, or image generation models. I don't know if we have any MOE architectures, but we'll be implementing Mixtral soon for sure!