11 ms·
Qwen 3.8 27B available on Cerebras at 1500 tokens/s
- TawResearch 15d ago[flagged]
- gardnr 15d agoI used their Coding Plan for a few months. It is genuinely difficult to keep up with the models. The output is so fast. Qwen 3.8 27B is likely one of the strongest models they've hosted so far. Edit: it looks like this is only available on a API token pricing. Does anyone know if they have rolled out prompt caching yet? It used to get pretty expensive for agentic coding tasks with no prompt caching.
- altertable 15d agoAgreed, but in our SAAS I can tell some UX will sky-rocket to next level with this
- jasongill 15d agoIt appears that they do support Prompt Caching: https://inference-docs.cerebras.ai/capabilities/prompt-caching https://inference-docs.cerebras.ai/capabilities/prompt-cachi...
- abtinf 15d ago> How are cached tokens priced? > There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model. Well, talk about flipping the narrative.
- the_duke 15d agoIt doesn't reduce the price though.
- jasongill 15d agoGood catch, I guess I got lost in the marketing speak of the page!
- deleted 15d ago[deleted]
- cute_boi 15d agoi believe they used to have monthly plan, what happened to that?
- eli 15d agoStrongest model that they host on the public endpoint. They do a super fast version of GPT 5.6 Sol for OpenAI and have bigger open models on dedicated endpoints.
- singpolyma3 15d agoThe coding plan is gone now right?
- gardnr 15d agoLast time I got one, I had to log into a Discord server and wait for "the drop" and IIRC Daniel Kim was giving them out based on who was there at the time. They were gone in less than a minute. This was ~8 months ago.
- porphyra 15d agoWhy do they only host small models rather than the 2.4T version? Is the I/O and interconnect between the wafers bad due to the limited beachfront relative to the massive size of the chip?
- altertable 15d agoMostly economics I'm sure
- gardnr 15d agoThey make a giant inference chip. Their inference service is basically just advertising for their core value prop: hardware. The CEO was on Gradient Dissent a couple years ago: https://www.youtube.com/watch?v=qNXebAQ6igs https://www.youtube.com/watch?v=qNXebAQ6igs
- codexon 15d agoThe wafer only has space for 44 gb of sram. If they offload ram they lose the speedup of having everything on 1 chip (the whole point of cerebras).
- minimaltom 15d ago[dead]
- porphyra 15d agoThey can host larger models by pipelining it on multiple wafers. Each wafer stores one layer and N layers can serve an N * 44 gb model with N concurrency. The limitation would of course be inter-wafer I/O, which my comment was getting at. That's probably how they can serve bigger models like GPT 5.6 Sol [1]. [1] https://www.cerebras.ai/blog/accelerating-gpt-5-6-sol-ultrafast-with-openai https://www.cerebras.ai/blog/accelerating-gpt-5-6-sol-ultraf...
- codexon 15d agoI never said offloading was impossible. It will result in a large slowdown. It would look bad for cerebras if other people are hosting the 27b version and show a higher TPS than cerebras.
- Marciplan 15d agoused their Code product with GLM4.7. its fun but if the model is bad it just doesn’t do much useful. Hope they add such models to Code too :)
- altertable 15d agoYeah GLM 4.7 is from another decade at the speed we're going
- foundfontic 15d agoI really wish they had their customer support somewhere else than Discord, which seems to think I'm a bot and doesen't accept my email or phone numbe
- londons_explore 15d agodiscord support can fix such issues
- threecheese 15d agoIf you need customer support to access customer support, something is wrong; no?
- Zambyte 15d agoDiscord is simply a liability.
- peri-cl 15d ago(Was anyone able to create an account just now? I tried but onboarding falls into a redirect loop) (update: I got my answer. support@ replied and said my email domain is on their blacklist. It was just me (and I've resolved it)).
- bakies 15d agoyeah - used sign in with google
- trvz 15d agoNormal people: tok/s or t/s Psychopaths: tok/SEC
- scotty79 15d agoI like tps
- verdverm 15d agodo you get reports on them?
- actionfromafar 15d agoGot you covered.
- altertable 15d agook fair, caps lock kept ON /o\
- tacone 15d agoNoticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon. For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.
- srcreigh 15d agoGreat observation. That’s not enough context even for some one shot xhigh requests. When I put Qwen3.8 27B xhigh towards adding scope proxying to the Guice library, it one shotted a great impl using 250k context before stopping. Part of the greatness of the model is that it just keeps going until it gets a great result. 128k context is disappointing.
- walrus01 15d ago128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.
- fulafel 15d agoIsn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc. I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.
- walrus01 15d ago128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator. 128k is pretty much only good for chat/conversational/question asking (including tool calls for searching things and spitting back/parsing a set of results) or human interactive agent purposes.
- jasongill 15d agoIt would be great if they made their inference capacity for this model available via OpenRouter; the fastest provider on OpenRouter right now is at ~80tps https://openrouter.ai/qwen/qwen3.8-27b#providers https://openrouter.ai/qwen/qwen3.8-27b#providers They do appear to host other models on OpenRouter so maybe Qwen3.8 will be there soon: https://openrouter.ai/provider/cerebras https://openrouter.ai/provider/cerebras
- zackangelo 15d agoWe're serving it around 150-200tok/s (uses our new speculative decoding implementation on a DFlash2 draft model). https://mixlayer.com https://mixlayer.com, LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.
- deleted 15d ago[deleted]
- danielklnstein 15d agoI tried in your playground and got 14.2 tok/s?
- zackangelo 15d agoapologies we just got a sudden burst of new users and traffic, it's scaling up now.
- zackangelo 15d agojust added 8 more H200s to the cluster, if you (or anyone else) runs into issues please feel free to drop me a message: zack at mixlayer.com
- danielklnstein 15d agoWorks much better now! Got 103.9 tok/s, not quite 200 - but still amazing! Thanks for sharing
- byako 15d ago[flagged]
- eli 15d agoIf you read the reasoning trace for Qwen 3.8, it does a whole lot of "uh" and "But, wait..." too
- howunfortunate 15d agoYou're absolutely right - filler words are genuinely load-bearing
- Zambyte 15d agoAt 1500 tps, "uh" is about 0.7 ms, instead of 200-300 ms for a human.
- miohtama 15d agoYour brain can wash laundry and cook pasta, so there is still a long way to go
- qiine 15d ago(requires additional fleshy bits sold separately)
- davrosthedalek 15d agoregarding my brain, my mother might disagree on the laundry part.
- dgellow 15d agoYour brain updates itself constantly and maintains your whole body, LLMs are static. Still, 1500tokens/s is indeed wild
- ripbozo 15d agofyi this is an AI bot account
- polygot 15d agoUt oh, might be down: "Unable to connect to the server. Please check your connection and try again." when sending a message to Qwen 3.8 27B.
- vb-8448 15d agoAt that speed it's too pricey for agentinc tasks.
- yipinwong 15d agoThe target audience is who needs raw speed. Having the choice is good as you can make a trade-off between speed, perf, and quality. Until last year, people had a single AI god they believed in (mostly Anthropic stuff). Now we have power to make choices (open-weights, SOTA, speed-optimized, etc) the same way you do for system designs.
- vb-8448 15d agoIt's not a criticism, I was really looking forward to trying out such a powerful model at this speed. But I burn my 5$ allowance in 10 minutes ... and only because I was hitting rate limits, without it would probably be less than a minute.
- yipinwong 15d agoI hear ya... the best option is to use company budget as normies will rack up ridciulous amount soon with that raw speed.
- dshat 15d agoI'm saddened that Gemma4 is replaced by Qwen 3.8 on PayGo plan. Gemma4 31B is not coding model but it is excellent at intent understanding and task execution used in agentic software. This just shows that real world dominant usage for llms so far is to code generate. And not to augment business products. They must had barely anyone using Gemma to remove it from that tier.
- fulafel 15d agoWhat are the best benchmarks/leaderboards that compare task completion time between provider+model combos?
- freehorse 15d agoI have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.
- bitexploder 15d agoThe thing I didn’t realize for a while is 27B is rather smart. As many (or more) activated parameters as the flash models of the universe that we know about. It reasons very well. It just doesn’t have a lot of knowledge.
- nicce 15d agoThey seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.
- bitexploder 15d agoBut running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
- nicce 15d agoYou can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.
- bitexploder 15d agoI believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.
- darkbatman 15d agoI have been their user for more than year even used coding plans, though for normal coding the quota will definitely be a blocker if you are using opencode because rpm are bit less. Good for products/api though.
- the_duke 15d agoFunnily enough the pricing isn't that much worse than on openrouter, where the best price at the moment is $0.24 in / $2.55 out, vs $1 / $1.5 on Cerebras. Sure, 4x input , but cheaper output. Though Cerebras doesn't have prompt caching, so not great for agentic workloads. (they do, but it doesn't affect the price.
- srcreigh 15d agoIt is 15x more expensive. Openrouter usually charges like 1/4 for cached input. Most of the cost for agentic coding is input tokens, you pay for the whole context at each tool call or message. Output tokens is just a small rate
- drchaim 15d agoThe idea of custom software on the fly is coming
- pllbnk 15d agoJust a couple days ago I learned about ninfer (https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.
- beastman82 15d agocan't second ninfer enough. amazing tech
- lowbloodsugar 15d agoOk, I need to try that. I'm getting 45tok/s with vLLM on my 6000. >600tok/s concurrent, but 45tok/s single request.
- pllbnk 15d agoEven without ninfer I would get over 80 on LM studio with default settings, so it should be noticeably more on 6000. You might want to try different a different inference engine or settings.
- jakswa 15d agodang only for certain nvidia GPUs, had my hopes up
- aizk 14d agoIs there an equivalent but for 4090s?
- pllbnk 14d agoThe repository has many forks, suggesting that folks are trying to (vibe) code support for different GPUs. Might be worth a shot.
- hexa00 15d agoJust tried it on a medium size coding/debug problem on an existing codebase, observations: - Input doesn't look faster than other models, it spends a lot of time reading Read about 5M tokens - Output is awesome, super fast as you expect from the 1500t/sec I think that's correct - Tool call is failing more than say DS4, which leads to time wasted on retries (complex tools like browser control for example) - Shell commands are still somewhat of a bottleneck The net effect is that I spend about the same time waiting, and I still need to read that output so, at least for coding, it actually reconciles me with the 100-200t/sec you can get on DS4 or the like. Maybe that's a good sweet spot after all and faster t/sec is not where the bottleneck is. Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy
- peri-cl 15d ago> "Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy" I don't believe Cerebras has a cached input pricing? They don't list one on the model page: https://inference-docs.cerebras.ai/models/qwen-3.8-27b https://inference-docs.cerebras.ai/models/qwen-3.8-27b edit: See the sibling discussion, https://news.ycombinator.com/item?id=49554520#49555094 https://news.ycombinator.com/item?id=49554520#49555094 ("Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate")
- olivermuty 15d agoThey have cache, but it costs the same indeed, no idea what the point of the cache is
- lostmsu 15d agoThey don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!
- 15d ago
- nostrebored 15d ago150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billing is not available on Enterprise accounts. Please contact your team for further questions. ``` We have no team (they removed themself from our slack channel after we talked about rate limits). Perplexingly, none of this even shows up in the request, which gives: ``` {"message":"Model does not exist or you do not have access to it.","type":"not_found_error","param":"model","code":"model_not_found"} ``` When the error is really about billing. I always want to like Cerebras, but I get the vibe that as a tokens in tokens out consumer you are not valued at all.
- olivermuty 15d agoCerebras the tech is awesome, cerebras the company is a trainwreck
- dd8601fn 15d agoIs this the chatjimmy asic approach with a bigger model?
- ericd 15d agoNo, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.
- vel0city 15d agoI thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning. Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.
- gpugreg 15d agoI was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size of 55217 tokens, while Qwen3.8-27B was not even close to being done with a 64178 context window. This is a very efficient way to burn your money, but I would not recommend it for programming. On the positive side, I got a $5 signup bonus, so it wasn't my own money.
- d2p 15d ago> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?
- gpugreg 15d agoCached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.
- nullbio 15d agoCached tokens counting toward the limit is ridiculous.
- perching_aix 15d agothen it's basically useless lol, wtf, this has to be a defect
- selcuka 15d agoIt's PR: https://news.ycombinator.com/item?id=49556302 https://news.ycombinator.com/item?id=49556302
- srcreigh 15d agoHow many years until chips like this are available to consumers?
- eli 15d agoI just did a little anecdotal test. Had pi + cerebras review a recent commit and asked a few quick followups on it. Worked great. The Cerebras session cost me $1.60 and took a total of 5.1 mins. I did get a few brief 429 rate limit errors in there. The p50 speed was 890 tok/s and 0.64s TTFT. Using OpenRouter averages, that would've cost $0.29 (no cache discount at Cerebras!) and would've taken about 14.4 minutes. So on this one short session, cerebras was 5.6x more expensive in exchange for being 2.8x faster. Or, another way, $1.32 buys back about 9 minutes of your time. Not a bad trade IMHO but the cache situation is a real bummer. The longer your session the more relatively expensive Cerebras gets. The "good" news is you're also limited by its short context window. (Also, I used to be on the Cerebras coding plan and the support is pretty bad for end users. My guess is these public endpoints are really just product demos for potential enterprise customers.)
- irthomasthomas 15d agoThanks! Is there something about their platform that prevents caching? Or are they just not passing on the discount?
- eli 15d agoThe session had a 91.4% cache hit rate. They just give zero discount.
- imtringued 15d agoIt sounds to me that they don't have enough capacity and they want to discourage people from using the service. There is nothing about their architecture that prevents reusing the KV cache other than the opportunity cost of keeping the memory occupied.
- orliesaurus 15d agoQwen 3.8 27B is an exceptional model for coding and ranks as one of the best local models for coding....BUT in my head I am confused why a company that's IPO'd doesn't invest in RL'd super specialized, super-damn-fast models for very specific tasks - instead of giving us the OSS GPT model from what feels like 200 years ago
- kroaton 15d agoEspecially since they still serve Codex-Spark, which is dogshit.
- anthonypasq 15d agoalmost of their business is hosting Sol ultra fast or whatever for OpenAI to use internally
- grav 15d agoShould be available in OpenCode once this lands: https://github.com/anomalyco/models.dev/pull/6199/changes https://github.com/anomalyco/models.dev/pull/6199/changes
- irthomasthomas 15d agoIt's going to cost a fortune in opencode without prompt caching.
- ecshafer 15d agoI have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"
- FeepingCreature 15d agoI have a self hosted Qwen 3.8 27B and I find it unbelievably cracked and dedicated. It's at least credibly attempted everything I've thrown at it. Just today I had it write a toy compiler with a JIT backend just to test out a concept, and that was with 4-bit quantization and 8-bit KV cache. Something has to be going wrong with your deployment.
- codazoda 15d agoI want a Qwen 3.8 27B hosted locally but I don't quite have the RAM for it. And, I don't want to buy the RAM until I prove I can use it. Yesterday I did have success with Gemma-4-12b with 128k context. It fits in my RAM and it's relatively fast on my hardware. I had to give it prompts that are quite a bit different from the way I use foundation models, but I did get it to work quite well. I feel like I could learn it's differences and get good at using it for real work.
- apitman 14d agoYou could try renting a GPU or VPS
- pyrolistical 15d agoI run it locally at q4_k_xl on a r9700 with kv cache bf16 and while it thinks a lot, it’s still fast enough to do the task. This model had its knowledge replaced with reasoning ability. The chain of thought what makes this reasoning effective. So this is why you need to let it think and don’t quantize the kv cache.
- FeepingCreature 15d ago
- deleted 15d ago[deleted]
- RomanPushkin 15d agoThe question is whether Cerebras is available... I've been trying to get https://www.cerebras.ai/code https://www.cerebras.ai/code for at least 1 year now. It's all sold out. Always. I once joined their Discord, waited for the drop, and it all sold out in seconds. I haven't had enough time to put my card details. Somebody recommended that I should put my card details in advance, lol. The next time I hear about them I am laughing, because when I could enjoy these powers? How many years I should be sitting in a waitlist...
- codazoda 15d agoDo I understand their pricing correctly? This is $10 per month for a developer account PLUS you pay $1.49/M for output tokens and $0.99/M for input tokens on Qwen 3.8 27b with a 128k context? EDIT: Or, maybe it's just token pricing, but $10 is the minimum? Maybe it's that. https://www.cerebras.ai/pricing https://www.cerebras.ai/pricing
- low_tech_punk 15d agoNo. You buy a minimum of $10 worth of credit, then use it at $1.49/M rate. There is no recurring charge. There is a separate subscription based plan, which is sold out now.
- codazoda 15d agoGot it. But, they also charge the same for cached tokens, so that probably closes the gap on Foundation models quite a bit.
- ma2kx 15d agoI guess Cerebras didnt intend the model for agentic coding but rather for small one shot task like title generation. At least thats why I use the free tier for.
- low_tech_punk 15d agomy hunch is that their hardware can't cache
- karim79 15d agoTokens are the new latest and greatest nonsensical shit on the planet. It's amusing. I can't wait to see the world in 1-2 years and the hilarity of looking back on this day.
- mrngld 15d agoSomeone on horseback probably said that after seeing a car broke down on the side of the road for the first time. Well, cars only got better. We're going to be talking about how to leverage this new tool for many years to come.
- jing09928 15d ago[flagged]
- forlorn 15d agoIs Kimi 3 available anywhere like that?
- walrus01 15d agoI don't see the point of paying for external inference on Qwen 3.8 27B with a bunch of arbitrary limits, when you can run it locally without ridiculous memory requirements. Even the unsloth Q8-XL version of it with full context and extra llama-server --cache-ram (like 10GB instead of 8GB) fits in 64GB. Paying for external inference for a much larger model like qwen 3.8-flash-next Q8 with full context makes a lot more sense, since the model consumes something like 188GB RAM when fully loaded into an inference engine. Yeah I guess this is cool and all that it runs at some ridiculous token/s rate but if the actual usage of it is highly limited... What's the point? I'd rather have a much slower tok/s rate that can chew on things 24x7.
- WithinReason 15d agoNow imagine having it running 10x faster on a Taalas chip on a card you can buy for $1k. I hope that future happens
- apatheticonion 15d agoI just want an API that takes these crazy small / cost effective open weight models and charges peanuts for access. Think, DeepSeek Flash (before the price hikes) prices. If I can run this on a 32gb card while they have a datacenter with wholesale electricity prices, why are we not seeing "cents per billion tokens" pricing?
- mrngld 15d agoBecause your 32gb card isn't running this at 1500 token/s. Serving these things at scale with the enormous context windows real use demands and doing some with usable performance takes a lot of expensive hardware. Yes, their margin on straight inference is allegedly really high, but that's severely offset by high capital costs. If you want to spend a new car worth of money and still not serve as fast as Cerebras because you can't simply buy their mammoth custom chips, then yes you too can self host a huge Deepseek or GLM model.
- apatheticonion 13d agoYes, that's what I'm saying. 200t/s is more than enough to 5 - 10x my productivity and the intelligence of current open weight models covers 90% of both my guided and software factory workflows
- naklitechie 15d agoHas reliability improved? It's flaky in Production.
- latentsea 15d agoWhat we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.
- gilgoomesh 15d agoEasy. I just need... 50 times more speed. Better download some more RAM.
- kilroy123 15d agoThis too is my dream. But I want fable level intelegence. Not Qwen or deepseek flash.
- mncharity 15d ago> 1500 t/s [...] all you can eat tokens as fast as you can eat them Pfft, you can eat tokens far far faster than that. Many orders of magnitude more. Just switch from frontier-style bespoke artisanal pets to cloud burst-parallel, latent subspace exploring/exploiting/searching, mass ensembles of cow herds. There's N-Version Programming. Work the problem in English, in Chinese, in Haskell, Lisp, Rust, etc. Then work ports to the target lang. There's design space sampling. Work the problem emphasizing performance, or security, or monitoring, readability, etc. Then work a synthesis. There's non-determinism sampling. Work the problem order 10 or 100 times. Then work to combine the best bits from each. There's sample synthesis. NP-hard aggregation of insights. There's genetic exploration. Work populations of trees of work variants under selective pressure. There's repo quantum superpositions of implementation space. The unspecified remains indeterminate - state space collapse occurs not upon each edit/commit, but as JIT-synthesized fuzzing/search upon each execution. There's maintaining a pretty dev UI, but that >>10k tok/s is trivial, because like symbiotic adversary cocreation, fine-grain agent swarms, scenario analysis/forecasting, etc, etc, it is unlike the preceding items... which scale combinatorially. "All you need is 1500 t/s"? "All you need is 640k RAM" is only 5 orders of magnitude off from 64 GB. It takes "All you need is a single Intel 3101's 64 bits", to get 9 orders of magnitude from 64 GB. Then datacenters...
- latentsea 14d ago
- register 15d agoI tried it. Not impressed. The gain in development speed is only marginal as there are other bottlenecks that affect overall development speed. Probably gains might be more significant for pure content generation tasks rather than software development
- storch 15d agoBut what happened to Gemma 4, it disappeared from the catalog? Has the Gemma 4 Cerebras API been "deprecated"?
- alescalaios 15d ago[dead]
- laweijfmvo 15d agoKagi used to serve gemma4 31b on cerebras, and i got so used to the speed that i basically stopped using other models. now that they no longer offer it, i find myself reaching for a model a lot less
- Roark66 15d agoDon't they have very low limits? What do people use these tiny limits for? I started measuring my Claude Max x5 use and last week (they gave me 50% more) I used 1.3B input tokens. Some 130M were cache writes, rest was cached. And 5M output. This puts things in perspective. We're taking thousands of bucks weekly even if I managed to switch to Kimi K3. What is the majority of this use? Infrastructure upgrades, troubleshooting and so on. Ingesting quite a bit of documentation at beginning of each session. Sessions run from few hours to a month long and 1M context usually hovers near 30-60%.
- khalic 15d agoI can't wait for Qwen 27B on an ASIC
- dev0p 15d agoI wonder how much it would cost? It's something that's been bugging me for a while: once we reach a "good enough" small model, and qwen 3.8 27b is already getting damn close to it, does it make sense to just bake weights and everything directly into an ASIC, and use that for highly optimized inference? AFAIK only groq and cerebras are moving in that direction, and only to be providers themselves... It would be a dream to buy one for <$1k.
- nrub 15d agoThis is exactly what Taalas has already done, and the reason they got quickly acquired by AMD. Their chip runs Llama 3.1 8B at 17k tok/s, and even if llama 3.1 is dated, I can think of many problems I could use it for, especially at those speeds. They're certain to be working on a newer set of weights by now.
- onesandofgrain 15d agoTried it limits are shit
- explorigin 14d agoYep, dumber than Deepseek4 at 10x the price. Cool tech demo though.
- deleted 14d ago[deleted]
- gabri200 14d agoI tried to get Qwen3.8 27B working properly under high concurrency, and while the quality level is spectacular for the size, the performance wasn't the best, even with MTP. Unless you have a very big infrastructure, it's difficult to run a dense model concurrently with high throughput.I suppose that's why almost all large models are now MoE. On the other hand, 1500 tok/s is an impressive speed, and that speed is very important for agent tasks, so a service like this instead of local infrastructure might make sense, although it also depends on your busines constraints.
- iamleppert 14d agoCompletely unusable last time I tried it. You got hit with rate limits after the first few minutes of using it. It's fast but cannot sustain its claimed speed before it immediately hits its rate limit. What good is it if you can't finish a task? It's like having a sports car that can only drive 180 mph for 5 seconds every minute and then has to cool down for an hour. Borderline fraudulent to advertise to developers when it's completely impractical to use. Their only support is their company Slack channel.
- lsb 12d agoThe context window is limited to 64k or 128k. If you’re using this with a coding agent, there’s going to be a lot of compactions. I found that I had a subagent whose compacted context plus prompts and such was over the window and it errored out in opencode.