7 ms·
I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration. > In our evaluations, GPT-5.6 Sol on Ultrafast mode answe
by iamcoder18 1mo ago
I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
- piyh 1mo agoFeels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.
- sscaryterry 1mo agoYep, https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law
- beng-nl 1mo agoI wish, but I don’t think this qualifies as single thread performance. Cerebras is whole wafer computation though right? So you get the result sooner, but the same amount of computation has occurred, just with huge parallelism.
- wrsh07 1mo agoSeems like they will do Sol first while capacity constrained? I can't imagine the margins they'll be charging
- sscaryterry 1mo agoI think this is what they need to actually have a margin :)
- zozbot234 1mo agoAnswering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
- hgoel 1mo agoPresumably when they're talking about benchmarking, they're using a fixed amount of resources, like everyone else does when benchmarking?
- desmondl 1mo agoThe animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
- whimsicalism 1mo agoi assume they meant serially....
- jcims 1mo ago>Answering 2,500 independent questions is an embarrassingly parallel workload So is calculating the total time required to answer all of the questions.
- baq 1mo agoAmadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
- moralestapia 1mo agoSuch a deep and smart insight. But why is that important if they're measuring time?
- moralestapia 1mo ago
- sixtyj 1mo agoOutput from Cerebras with GPT model is 750 tokens per second. Don’t blink. (Chatjimmy has 14,200 TPS.)
- mips_avatar 1mo agoUnfortunately AMD bought them, so I don't think we will get to see another release from them.
- sixtyj 1mo agoAha, thanks, that’s fresh; press release is from Aug 6 https://ir.amd.com/news-events/press-releases/detail/1296/amd-acquires-taalas-to-advance-compute-solutions-for-rapidly-growing-ai-inference-market https://ir.amd.com/news-events/press-releases/detail/1296/am...
- tomrod 1mo agoChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
- dzhiurgis 1mo agoThe knowledge of ChatJimmy is terrible. Even Qwen on my iPhone is better.
- headPoet 1mo agoChatJimmy isn't a model, it's Llama 3.1 8B hardwired into silicon. The point isn't to be a good llm, but to showcase the speedup that's possible
- senordevnyc 1mo agoHaha, at first I thought you meant that the knowledge of the existence of an LLM that’s so fast is terrible because it’s ruined every other LLM for you!
- 1mo ago
- rvz 1mo agoBeen waiting since Cerebras-GPT. [0] [0] https://news.ycombinator.com/item?id=35490837 https://news.ycombinator.com/item?id=35490837
- christoff12 1mo agoAn irrational gripe of mine is how GPT uses 7× instead of 7x. I recognize that the former is the multiplication symbol, but I don't think it should be used that way.
- Daneel_ 1mo agoIf anything, writing "7x" is actually just a poor emulation of writing "7×". Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.
- TacticalCoder 1mo agoBut speech evolves. And we all pronounce "ten ex", not "ten times". Actual speech usage very clearly pronounce the letter 'x'. So it's 10x. And no need for Unicode codepoints.
- Daneel_ 1mo agoI fully agree that speech evolves, but maybe this is just in the US? I've never really heard anyone say it here in Australia, it's usually "times", although admittedly almost no one uses the phrase/format here.
- christoff12 1mo agoAh, yes. My gripe is certainly US American flavored.
- shussjjs 1mo agoA number plus an “x” is used beyond Trumpistan. I have never in my life heard anyone say “ten ex”.
- _flux 1mo ago
- andai 1mo agoI'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.) In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models). Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting). Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
- trollbridge 1mo agoLuna and Terra are great, but so are DeepSeek Flash, MiMo, Qwen 3.6, and even Gemma for some things. MiMo Pro has had UltraSpeed for a while.
- epolanski 1mo agoI feel odd to use these models, because it feels like a faster model doesn't feel that much faster if it spends reading files, making edits and running checks. It feels too situational.
- andai 1mo agoWhich one of those takes time?
- phoghed 1mo agoLuna has impressive results on DeepSWE. I mostly split work between Luna and Sol. If something seems simple enough I always try it with Luna first.
- shepherdjerred 1mo agoLuna is incredible considering the cost. It’s very good for routine work.
- 1mo ago
- aetherspawn 1mo agoI discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed better, and we lost less tokens due to guardrails shutting down sessions (I feel like it’s illegal to take $50 of someone’s token money and then shut down a session with guardrails before they get an answer, and yet Anthropic do it to us constantly… either take our money and commit, or trigger the guardrails immediately)
- minraws 1mo agoWait isn't Fable like 2x more expensive if we compare under 272k tokens
- ChadMoran 1mo agoThe comment you're replying to almost feels like it was written by a bot or am I crazy?
- w4yai 1mo agoI agree. Weird to use <“> and <”> characters. Maybe written on phone, but certainly not on keyboard.
- gunalx 1mo agoCommon on non us keebs though.
- jrflo 1mo agoOther languages use different characters for quotes, if anything that's an indication that's not written by a LLM because it's not favoring the standard English character. https://en.wikipedia.org/wiki/Quotation_mark#Specific_language_features https://en.wikipedia.org/wiki/Quotation_mark#Specific_langua...
- krzyk 1mo agoIsn't ultrafast just making hundreds of subagents?
- anentropic 1mo agoNo, that's Claude "ultracode"