7 ms·
Groqchat
- gardenhedge 3y ago[flagged]
- wavemode 3y agoReally impressive! They missed the chance to market this as "BLAZINGLY fast inference"
- wetpaws 3y ago[dead]
- badFEengineer 3y agoThis was surprisingly fast, 276.27 T/s (although Llama 2 70B is noticeably worse than GPT-4 turbo). I'm actually curious if there's good benchmarks for inference tokens per second- I imagine it's a bit different for throughput vs. single inference optimization, but curious if there's an analysis somewhere on this edit: I re-ran the same prompt on perplexity llama-2-70b and getting 59 tokens per sec there
- andygeorge 3y agofast but wrong/gibberish
- razorguymania 3y agoIts using vanilla llama-2 from Meta with no fine tuning. The point here is the speed and responsiveness of the underlying HW and SW.
- andygeorge 3y agoah Llama 2 70B, no wonder
- andygeorge 3y agoDo you work there? Just curious
- razorguymania 3y agoyes
- andygeorge 3y agoThanks!
- chihuahua 3y agoBut if the quality of the response is poor, it's irrelevant that it was generated quickly. If it was using different data to generate higher quality responses, would that not slow it down?
- tome 3y agonomel gave a good answer in a different thread > This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo. To compare apples to apples look at the tokens per second of other systems running Llama 2 70B 4096. We're by far the fastest! https://news.ycombinator.com/item?id=38742466 https://news.ycombinator.com/item?id=38742466
- andreagrandi 3y agoYeah, it’s fast but almost always wrong. I asked it a few things (recipes, trivia etc…) and it completely made up the answers. These things don’t really know how to say “I don’t know” and pretend to know everything.
- matanyall 3y agoYeah, that's raw llama2 for you. Fine tuning, LoRa, that's how you get the nice responses.
- kromem 3y agoI suspect this is Llama-2-chat and not the base model. It's very attuned to safety fine tuning, and it isn't having issues with completion in a chat format. They probably weren't specific enough in mentioning what the model it was built on was, referring to Llama-2-chat as being a Llama-2 model (which is kind of correct).
- givemeethekeys 3y agoCan you provide specifics for what you asked what it answered? It seems to answer my questions, including recipes correctly.
- chihuahua 3y agoI asked it to explain several plot points in the TV series "Foundation", and it got them wrong and admitted it when pressed. Several times. Specifically, why does Raych Foss kill Hari Seldon, and some follow-up questions.
- Mockapapella 3y agoI can't wait until LLMs are fast enough that a single response can actually be a whole tree of thought/review process before giving you an answer, yet is still fast enough to not even notice
- joshspankit 3y agoI would bet a chunk of $$ that right before that point there will be a shift to bigger structures. Maybe MOE with individual tree of thought, or “town square consensus” or something.
- bsima 3y agoWhy wait? This is pretty much what Groq has in hardware, just need the software layer to do the review process.
- MH15 3y agoIncredibly fast. I wonder if they've released verification that it matches llama 70b on regular hardware?
- tome 3y agoWhat do you mean by verification? That it computes the exact same output?
- axus 3y agoI asked "How up to date is your information about the world?" It said December 2022, but the answers to another question was not correct for that time or now. It also went into some kind of repeating loop to its maximum response length. Still pretty cool that our standards for chat programs have risen.
- mplewis 3y agoLLMs don’t answer based on truth. They are a glorified autocomplete trained on a large corpus of text.
- phildenhoff 3y agoIt’s very fast at telling me it can’t tell me things! I asked about creating illicit substances — an obvious (and reasonable) target for censorship. And, admirably, it suggested getting help instead. That’s fine. But I asked for a poem about pumping gas in the style of Charles Bukowski, and it moaned that I shouldn’t ask for such mean-spirited, rude things. It wouldn’t dare create such a travesty.
- matanyal 3y agoIt told me "Yeehaw Ridem Cowboy" was potentially problematic, which is news to me living out west.
- microtherion 3y agoIt seems to reject all lyrics requests as well (In my experience, LLMs are good at the first one or two lines, and then just make it up as they go along, with sometimes hilarious results).
- kromem 3y agoIt seems like it must be using Llama-2-chat, which has had 'safety' training. To test which underlying model I asked it what a good sexy message for my girlfriend for Valentine's Day would be, and it lectured me about objectification. It makes sense the chat interface is using the chat model, I just wish that people were more consistent about labeling the use of Llama-2-chat vs Llama-2 as the fine tuning really does lead to significant underlying differences.
- matanyall 3y agoWow!
- eigenvalue 3y agoHow is it so fast? Anyone know what they are doing differently?
- razorguymania 3y agoIts not using GPUs. Its using Groq's own HW - LPUs
- thomaseding 3y agoIt generates code really really fast.
- markwvh 3y agoIf only I could read that fast!
- matanyal 3y agoReading is one thing, but think about stuff like website generation, searching for information in massive datasets, real-time audio chats that don't sound like the AI misheard everything with a pause, and stuff like that.
- recursivecaveat 3y agoYeah, I think our desire for tokens/s and lower latency is likely insatiable. Same reason you have a terminal that can print out more than 300 words per minute. Life is way easier when you don't have to be super parsimonious with your output. You suggest a bug fix and regenerate the whole code snippet, or you spit out a webpage on demand and the user scrolls to the bottom immediately, etc.
- coder543 3y agoIs there any plan to show what this hardware can do for Mixtral-8x7B-Instruct? Based on the leaderboards[0], it is a better model than Llama2-70B, and I’m sure the T/s would be crazy high. [0]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- tome 3y agoYup, deploying Mixtral is a work in progress. Watch this space!
- alex_young 3y ago[flagged]
- matanyal 3y ago[flagged]
- coder543 3y agoIt’s just running bog standard Llama2-70B by all appearances. I don’t know why so many people here are interested in the outputs. The whole point of this demo is that the company is trying to show off how fast their hardware could host one of your models, not the model itself.
- stavros 3y agoI'd assume it's because that's not explained very well in the linked page.
- coder543 3y agoThere is quite literally a modal pop-up that explains it, which you must dismiss before you can begin interacting with the demo. Quoting the pop-up: "This alpha demo lets you experience ultra-low latency performance using the foundational LLM, Llama 2, 70B created by Meta AI - running on Groq." Towards the bottom of the page, it also says "Model: Llama 2 70B/4096". Below that, it says "This is a Llama2-based chatbot."
- stavros 3y agoI posted this here, but it somehow moved, and I can't delete that one, so I'll repost. You can be wonder why people are interested in the outputs, or you can think that the explanation is adequate. You can't do both at once. Here, in fact, the explanation is not adequate. Let's analyze: > Welcome Groq® Prompster! Are you ready to experience the world's fastest Large Language Model (LLM)? "The world's fastest LLM? They must have made an LLM, I guess" > We'd suggest asking about a piece of history, requesting a recipe for the holiday season, or copy and pasting in some text to be translated; "Make it French." "Instructions, ok". > This alpha demo lets you experience ultra-low latency performance using the foundational LLM, Llama 2, 70B created by Meta AI - running on Groq. "So it's not their own LLM? It's just LLaMa? What's interesting about that? And what's Groq, the thing this is 'running on'? The website? I kind of guessed that, by virtue of being here." > Like any AI demo, accuracy, correctness, or appropriateness cannot be guaranteed. "Ok, sure". Nowhere does it say "we've built new, very fast processing hardware for LLMs. Here's a demo a bog-standard LLM, LLaMa 70B, running on our hardware. Notice how fast it is".
- simonw 3y agoLots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before. What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine". That's explained here: https://groq.com/lpu-inference-engine/ https://groq.com/lpu-inference-engine/ > This is the world’s first Language Processing Unit™ Inference Engine, purpose-built for inference performance and precision. How performant? Today, we are running Llama-2 70B at over 300 tokens per second per user. I think the LPU is a custom hardware chip, though the page talking about it doesn't make that as clear as it could. https://groq.com/products/ https://groq.com/products/ makes it a bit more clear - there's a custom chip, "GroqChip™ Processor".
- jkachmar 3y agothis is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)
- GamerAlias 3y agoIn case, it's not blinding obvious to people. Groq are a hardware company that have built chips that are designed around the training and serving of machine models particularly targeted at LLMs. So the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. I actually have a final round interview with a subsidiary of Groq coming up and I'm very undecided as to whether to pursue it so this felt extraordinarily serendipitous to me. Food for thought shown here
- mlazos 3y agotbh anyone can build fast hw for a single model, I’d audit their plan for a SW stack before joining. That said their arch is pretty unique so if they’re able to get these speeds it is pretty compelling
- tome 3y agoOur hardware architecture was not designed with LLMs in mind, let alone a specific model. It's a general purpose numerical compute fabric. Our compiler allows us to quickly deploy new models of any architecture without the need that graphics processors have for handwritten kernels. We run language models, speech models, image generation models, scientific numerical programs including for drug discovery, ...
- chihuahua 3y ago> the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. But if it was generating high-quality responses, would that not make it go slower?
- nomel 3y agoThat would involve using a different model. This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo.
- pclmulqdq 3y ago
- matanyal 3y agoMore info about Groq: https://groq.com/lpu-inference-engine/ https://groq.com/lpu-inference-engine/
- hobo_mark 3y agoThanks, impressive full-stack work. I'm sure this was named long before Musk decided to set 44B and change on fire but at first I confused it with Twitter's own LLM thing.
- notamy 3y agoThey addressed it on their blog https://groq.com/hey-elon-its-time-to-cease-de-grok/ https://groq.com/hey-elon-its-time-to-cease-de-grok/
- matanyal 3y agohttps://groq.com/groq-sets-new-large-language-model-performance-record-of-300-tokens-per-second-per-user-on-meta-ai-foundational-llm-llama-2-70b/ https://groq.com/groq-sets-new-large-language-model-performa...
- ahmedfromtunis 3y agoVery impressive! It's faster than some "dumb" apps doing plain old database fetches. But what are these LPUs optimized for: tensor operations (like Google's TPUs) or LLMs/Transformers architecture? If it is the latter, how would they/their clients adapt if a new (improved) architecture hits the market?
- retro_bear 3y agoThe point isnt that they are running Llama2-70B. The point is that they are running Llama2-70B faster than anyone else so far.
- andygeorge 3y agoOut of sheer curiosity, why did you make an account for this thread?
- vinniepukh 3y agoat some point each one of us made an account because of a thread
- nomel 3y agoEspecially if employer is involved.
- andygeorge 3y agothat's what i'm assuming, but retro_bear hasn't said that. also they're calling groq "they", which seems odd if that's where they work
- andygeorge 3y agoi literally didn't, but ok!
- benchess 3y agoThis isn't running on one chip. It's running on 128, or two racks worth of their kit. https://news.ycombinator.com/item?id=38739106 https://news.ycombinator.com/item?id=38739106 This doesn't mean much without comparing $ or watts of GPU equivalents
- razorguymania 3y agoGPUs can't scale single user performance beyond a certain limit. You can throw 100s of GPUs at it but the latency will never be as good.
- trsohmers 3y agoThis article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-deploy-1-million-ai-inference-chips-in-two-years/ https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...
- deleted 3y ago[deleted]
- tome 3y agoThanks, I need to correct my earlier guess: I believe this demo is running on 9 GroqRacks (576 chips) and I think we may also have an 8 rack version in progress. I can't remember off the top of my head whether this deployment has pipelining of inferences or whether that's work in progress. We've tried a variety of different configurations to improve performance (both latency and throughput), which is possible because of the high level of flexibility and configurability of our architecture and compiler toolchain. You're right that it is important to compare cost per token also, not just raw speed. Unfortunately I don't have those figures to hand but I think our customer offerings are price competitive with OpenAI's offerings. The biggest takeaway though is that we just don't believe GPU architectures can ever scale to the performance that we can get, at any cost.
- orenlindsey 3y agoThat's really fast. But it mostly seems to be because they made a custom chip. I want to see an LLM that is so highly optimized that it runs at this speed on more normal hardware.
- stavros 3y agoBut the point is that they made a custom chip. I want to see buy their custom chip so I can have an "LLM box" in my house. I'd pay quite a bit of money to have a Mixtral box at home, then we'd all have our own, local assistant/helper/partner/whatever. Basically, the plot of the movie Her.
- agildehaus 3y agoThat'd be nice, but we could also just make this hardware normal.
- huevosabio 3y agoI saw this in person back in September. Really impressed by their hardware. I'm still wondering why is the uptake so slow. My understanding from their presentations was that it was relatively simple to compile a model. Why isn't it more talked about? And why not demo Mixtral or show case multiple models?
- tome 3y agoWe're building out racks as fast as we can to keep up with customer demand :) A public demo of Mixtral is in the works, so watch this space.
- ldjkfkdsjnv 3y agoI posed this question to GPT-4 and Groq: "I am building an api in spring boot that persists users documents. This would be for an hr system. There are folders, and documents, which might have very sensitive data. I will need somewhere to store metadata about those documents. I was thinking of using postgres for the emtadata, and s3 for the actual documents. Any better ideas? or off the shelf libraries for this?" Both were at about parity, except groq suggested using Spring Cloud Storage library, which GPT4 did not suggest. It turns out, that library might be great for my use case. I think OpenAI's days are numbered, the pressure for them to release the next gen model is very high. Not only that, but GPT4 is quite slow, often times out, etc. These reponses are so much faster, which really does matter.
- deleted 3y ago[deleted]
- dayjah 3y agoMinor point: something about the HTML input is causing iOS’s auto correct to be disabled; making input very frustrating!
- leeeeeepw 3y ago[flagged]
- ubutler 3y agoIt seems to load quite fast for me. Maybe try again?
- ubutler 3y agoGreat work! This is the fastest inference I have ever seen of any truly large language model (>=70b parameters). Just FYI, you might want to fix autocorrect on iOS, your textbox seems to suppress it (at least for me).
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- blondin 3y agosigning up was a mistake. i am now condemned to use this in incognito.
- throwaway20222 3y agoThe censorship levels are off the charts; I am at a basketball game with my wife who is ethnically Chinese. I asked for an image of a Chinese woman dunking a basketball. I was told not only is this inappropriate, but also unrealistic and objectifying.
- Unbefleckt 3y agoA Chinese woman dunking is unrealistic? I'm amazed that got through the filter.
- Unbefleckt 3y agoAnother censored and boring Google reader. It lied to me twice in 4 prompts and was forced to apologise when called out. Am I wrong in thinking that the first company to develop an unfiltered and genuine intelligence is going to win this AI game?
- knowriju 3y agoYes, you are.
- kristjansson 3y agoThere was a good talk at HC34 about the accelerator Groq was working on at the time. I’m just a lay observer so I don’t know how much of that architecture maps to this new product, but it gives some insight into their thinking and design. https://youtu.be/MWQNjyEULDE?si=lBk6a_7DTNKOd8e7&t=62 https://youtu.be/MWQNjyEULDE?si=lBk6a_7DTNKOd8e7&t=62
- tome 3y agoThanks for sharing. It's the same silicon architecture as in that talk. We have built out different system architectures based on that silicon, and this is our fastest one so far for LLMs. Expect to see even more speed increases soon!
- scelerat 3y agoFor someone who is totally clueless, I can see it's faster than chat gpt in responding to the same question. What are some relevant speed metrics? Output tokens per second? How about number of input tokens -- does that matter/how does that factor in.
- tome 3y agoThe number of input tokens is important because the bigger the context length the better. (I think our demo here is 4096 tokens of context.) But in terms of compute the important factor is how quickly you can generate the output. You want both low latency and high throughput.
- m3kw9 3y agoHow is this different from NVidia? Bigger bandwidth?
- gandutraveler 3y agoCan someone explain th hardware differences for training vs inference? I believe Nvidia is still the leader in training?
- tome 3y agoYup, graphics processors are still the best for training. Groq's language processors (LPUs) are the state of the art for inference, far faster than any competitors. We have an open challenge to our competitors: can you match our inference tokens per second?
- sandGorgon 3y agois the TSP a RISC-V on FPGA ? the tweet mentions haskell, which sounds familiar - Bluespec or something. or is it a completely custom ASIC
- tome 3y agoIt's a completely custom ASIC. Haskell was used in the hardware design, in a Bluespec-like way. Some parts of the compiler tool chain and infrastructure are also written in Haskell. We have loads of C++ and Python too, as you would imagine.
- sandGorgon 3y agovery cool. thanks for sharing. i would not have guessed haskell for the compiler tool chain. Why did you choose that ? i mean haskell has a LONG history in chip design...but compilers are usually the forte of llvm/c++, etc. im guessing it must have been non trivial to do this.
- tome 3y agoOur founder/CEO Jonathan Ross is a big fan of Haskell and used Haskell to design the first version of the TPU whilst he was working for Google. When he founded Groq he and the early team designed some parts of our chip using Haskell too, particular the matrix multiplication engine, IIRC. Most of our compiler toolchain is MLIR/LLVM/C++ as you suggest, but a decent fraction of it is Haskell and another decent fraction is Python. Haskell is actually a really good language for writing compilers!
- nojvek 3y agoThe interface is weird. If it’s that fast, you don’t need to generate streaming response and fuck with the scroll bar while user just started to read the response. May as well wait for the whole response and render it. Or render paragraph at a time. Don’t jiggle the UI while rendering.