9 ms·
Cerebras launches inference for Llama 3.1; benchmarked at 1846 tokens/s on 8B
- freediver 2y agoYep it is fast. Now what exactly is Llama 8B useful is another matter - what are some good use cases? One scenario I can think of is rolepaying - but I would assume that the slow streaming speed was kind of a feature there.
- rgbrgb 2y agoSpeed is useful for batch tasks or doing a bunch of serial tasks quickly. E.g. "take these 1000 pitch decks and give me 5 bullets on each", "run this prompt 100 times and then pick the best response", "detect which of these 100k comments mention the SF Giants".
- drdaeman 2y ago8B is not exactly great for roleplaying, if we put the bar any high. It is just not sophisticated enough, as it has very limited "reasoning"-like capabilities and can normally make sensible conclusions only about very basic things (like if it's raining, maybe character will get wet). It can and will hallucinate about stuff like inventories or rules - and it's not a context length thing. If there are multiple NPCs, things get worse, as they're starting to all mix up. 70B does significantly better in this regard. Nowhere close to perfection, but the frequency of WTFs about LLM's output are [subjectively] drastically lower. Speed can be useful in RP if we'd run multiple LLM-based agents (like "plot", "goal checker", "inventory", "validation", "narrator") that function call each other to achieve some goal.
- wkat4242 2y agoThese wafers only have 44GB of RAM though. Very curious why the quantity is so low considering the chips are absolutely massive. It's SRAM though so very fast, comparable to cache in a modern CPU. But I assume being fast and loading the whole model there is the point.
- adgjlsfhk1 2y agoprobably largely because dram is a lot more dense than sea.
- bottlepalm 2y agoSurveillance states and intelligence agencies. Or maybe a MMO with a town of NPCs.
- benopal64 2y agoWhy can't the MMO with a town of NPCs have an intelligence agency too?
- seldo 2y agoFor agentic use cases, where you might need several round-trips to the LLM to reflect on a query, improve a result, etc., getting fast inference means you can do more round-trips while still responding in reasonable time. So basically any LLM use-case is improved by having greater speed available IMO.
- freediver 2y agoThe problem with this is tok/sec does not tell you what time to first token is. I've seen (with Groq) where this is large for large prompts, nullifying the advantage of faster tok/sec.
- halJordan 2y agoWhat kind of answer are you looking for? Just start asking it questions. The constant demand for a magic silver bullet use case applicable to every person in the country is wild. If you have to ask, you're not using it. What exact use case did google.com enable you to do that made it worthwhile for everyone to immediately start using? It let you access nytimes.com? Access amazon.com? No, it let you ask off the wall, asinine, long tail questions no one else asked.
- mikewarot 2y agoWhy is it so gosh darned slow? If you've got enough transistors to hold 44 gigabytes of RAM, you've got enough to have the whole model in stored with no need for off-chip transfers. I'd expect tokens out at 1 Ghz aggregate. Anything less than 1 Mhz is a joke.... ok, not a joke, but surprisingly slow.
- twothreeone 2y agoEven if they could generate tokens at that speed on the chip (which maybe they can in theory?) you need to get user tokens onto the chip and the resulting model tokens off again and transport them to the user as well. This means at some point the I/O becomes the bottleneck, not the compute. I also suspect it will get faster still, from the announcement it didn't sound like it's "optimal" yet.
- cma 2y agoUser tokens onto the chip and output tokens out are tiny.
- twothreeone 2y agoNot if you're serving tens of thousands of users at the same time.
- cma 2y agoStill tiny at 100,000.
- chessgecko 2y agoOn die communication isn’t free, a lot of things here are sequential and within matrix multiplies the cores have to transfer output and mem loads have to be distributed. It’s really fast but not like one cycle
- mikewarot 2y ago
- ChrisArchitect 2y ago[dupe] More discussion on official post: https://news.ycombinator.com/item?id=41369705 https://news.ycombinator.com/item?id=41369705
- phkahler 2y agoThe winner will be one of two approaches: 1) Getting great performance using regular DRAM - system memory. 2) Bringing the compute to the RAM chips - DRAM is accessed 64Kb per row (or more?) and at ~10ns per read you can use small/slow ALUs along the row to do MAC operations. Not sure how you program that though. Current "at home" inference tends to be limited by how much RAM your graphics card has, but system RAM scales better.
- rfoo 2y agoThe winner, unfortunately, will be on cloud inference.
- eth0up 2y agoI'll probably get stoned for asking here, but... since you seem knowledgeable on the subject: I just got llama3.1-8b (standard and instruct). However, I cannot do anything with it on my current hardware. Can you recommend the best AI model that I: 1) can self host 2) run on 16GB ram with no dedicated graphics card and an old intel i5 3) use on Debian without installing a bunch of exo-repo mystery code? Any recommendation, directly or semi related would be appreciated - I'm doing my 'research' but haven't made much progress nor had any questions answered.
- arcanemachiner 2y agoSetting up Ollama via Docker was the easiest way for me to get up and running. Not 100% sure if it fits your constraints, but highly recommended.
- programd 2y agoAnother option is to download and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models quite nicely.
- 2y ago
- russ 2y agoHere’s an AI voice assistant we built this weekend that uses it: https://x.com/dsa/status/1828481132108873979?s=46&t=uB6padbn5E1yf7Bu3ioG1Q https://x.com/dsa/status/1828481132108873979?s=46&t=uB6padbn...
- cheptsov 2y agoVery interested in playing with their hardware and cloud. Also I wonder if it’s possible to try cloud without contacting their sales.
- ein0p 2y ago8b models won’t even need a server a year from now. Basically the only reason to go to the server a year or two from now will be to do what edge devices can’t do: general purpose chat, long context (multimodal especially), data augmented generation that relies on pre-existing data sources in the cloud, etc. And on the server it’s very expensive to run batch size 1. You want to maximize the batch size while also keeping an eye on time to first token and time per token. Basically 20-25 tok/sec generation throughput is a good number for most non-demo workloads. TTFT for median prompt size should ideally be well under 1 sec. But I’m happy they got this far. It’s an ambitious vision, and it’s extra competition in a field where it’s severely lacking.
- bkitano19 2y agoTime to first token is as important to know for many use cases, rarely are people reporting it
- Gcam 2y agoSee here for our TTFT metric benchmarks: https://artificialanalysis.ai/models/llama-3-1-instruct-70b/providers#latency https://artificialanalysis.ai/models/llama-3-1-instruct-70b/...
- wkat4242 2y agoWow one chip taking up a whole wafer. I bet their yields are low, though I assume they're not using the bleeding edge process but a slightly older one that's totally worked out. Still the price of one of these would be nuts if they'd sell them. Upwards of 1 million?
- Havoc 2y agoGuessing it’s set up in a way where they can just disable dead cores
- twothreeone 2y agoProcess defects can be located and routed around statically on the chip, it's described e.g. here: https://youtu.be/8i1_Ru5siXc?t=810 https://youtu.be/8i1_Ru5siXc?t=810