10 ms·

I think the fun takeaway from this is that GPT 5.4 is probably 45B active parameters and GPT 5.6 Sol is closer to 50B.

logicallee 1mo ago
(Where did you see that?) This was also interesting: "CS-4 delivers more than 1,000 tokens per second on models exceeding 10 trillion parameters." Was it known that there were 10 trillion parameter models in use? I think the frontier providers keep the size of their models carefully hidden.
- ewild 1mo ago
  It's rumored fable is around that 10T number
  - walrus01 1mo ago
    If this is true, it's even more impressive that some of the open weight models that are <3.5T in size, approx 33% of its size, are within a few points of it in the artificial analysis leaderboard.
    - habosa 1mo ago
      GLM 5.3 is "only" 753B parameters. Much much smaller.
    - manquer 1mo ago
      Not necessarily, there could be diminishing returns on mere parameters count . There is nothing to say for example a 1 Quadrillion parameter model will be vastly more intelligent than current SOTA especially since new training data is largely synthetic today
      - HDBaseT 1mo ago
        That's precisely what he is saying, there is diminishing returns (or optimization left on the table).
        manquer 1mo ago
        I read it as it is impressive because smaller models 2.5T are squeezing similar returns as 10T models despite being 1/4th size not that there beyond 2T today the number or parameters do not have much meaning
        verdverm 1mo ago
        or the latest qwen3.8 27B doing so well at ~1/100 the size of K3
        rnewme 1mo ago
        What about general knowledge you can get out of it before hallucinations start?
        walrus01 1mo ago
        It did OK on schlongbench v1.0 (test of a specific niche word that doesn't make it into smaller LLMs) but it sure does love to count words https://pastes.io/r8F1AY8h https://pastes.io/r8F1AY8h
        verdverm 1mo ago
        I do not rely on any LLM of any size for general knowledge baked into the weights, they all hallucinate and that is the wrong way to hold them imo I think there is some merit in that smaller models cannot memorize so much of the training data, i.e. that they are less likely to do copyright infringement, and by analogy not having memorized SDK / API surfaces that have since changed from the training data
        walrus01 1mo ago
    - mlmonkey 1mo ago
      You want to take a look at the "Scaling Laws" paper, so you can extrapolate from these numbers.
      - stymaar 1mo ago
        This paper, as well as the Chinchilla one, aged like milk though.
    - scosman 1mo ago
      And GLM is only 0.7T! But these labs distill off the larger models. Both officially at the labs with the big ones, and unofficially. We need the giant models to get the smaller models.
  - johnnyApplePRNG 1mo ago
    Fable is most definitely nowhere near 10T. The cost to train and infer that would be insane, even by today's standards.
    - nl 1mo ago
      Fable is strongly believed to be around 10T. The most conservative estimate I've seen is 8T. Eg: https://www.reuters.com/technology/bytedance-targets-mega-ai-model-nearing-anthropics-mythos-ft-reports-2026-08-07/ https://www.reuters.com/technology/bytedance-targets-mega-ai... That reports Mythos as 8T and Fable as 5T, but I think they mean Opus as 5T, which is widely known, eg: https://eu.36kr.com/en/p/3760679047267075?ref=explainx https://eu.36kr.com/en/p/3760679047267075?ref=explainx Both Grok and Bytedance are training 10T models.
      - andai 1mo ago
        Wasn't Opus ~1.5T and Fable is about twice that?
      - stymaar 1mo ago
        The fact that Musk claims Opus is 5T to justify why Grok is far behind should be taken with a massive grain of salt given he's a recidivist mythomaniac. Honestly if Opus is 5T parameters while being matched by the biggest open models that are at least twice smaller, it would mean that the US is already behind China in the AI race, despite a significant edge in compute.
        WinstonSmith84 1mo ago
        Yes. And Opus goes a very long way compared to Fable, Anthropic isn't doing any favour, it's clearly just 2 models with a very different amount of parameters.
        nl 1mo ago
        The open models don't really match Opus. For example I regularly do Fable+Opus agentic coding runs over 24 hours without intervention. I think I've had GLM do a run that was a few hours. That's the closest I've had an open model come on that kind of work.
- alightsoul 1mo ago
  pretty sure 10 trillion parameters is now the norm among closed ai labs, given that nvidia also references the same 10 trillion number for their nvl72 racks
  - KronisLV 1mo ago
    Pretty bad efficiency then unless that only applies to Fable class but even then - Kimi K3 is around 3T and does similarly well in most benchmarks.
    - esafak 1mo ago
      As others have noted, most benchmarks stress the torso, not the tail.
- nl 1mo ago
  Mythos/Fable are around 10T: > According to FT, industry estimates say Anthropic's most advanced Mythos 5 has about 8 trillion parameters and Fable 5 about 5 trillion https://www.reuters.com/technology/bytedance-targets-mega-ai-model-nearing-anthropics-mythos-ft-reports-2026-08-07/ https://www.reuters.com/technology/bytedance-targets-mega-ai... I believe this report has confused Opus (which is known to be around 5T) and Fable. Other reports say 10T. See for example https://eu.36kr.com/en/p/3760679047267075?ref=explainx https://eu.36kr.com/en/p/3760679047267075?ref=explainx where Musk talks about the models being trained on Colossus2
  - zozbot234 1mo ago
    > I believe this report has confused Opus (which is known to be around 5T) and Fable. 5T for Opus feels quite high though. DeepSeek V4 Pro is a mere 1.6T and often described as a match with Opus in overall quality. Even the largest open models in common use are around 2.8T.
    - Implicated 1mo ago
      > and often described as a match with Opus in overall quality It's not. Idk about who has more T's but, unfortunately, DS4 pro is not a match to Opus, at least not Opus 4.8.
    - nl 1mo ago
      It's not an Opus match. The difference is very visible in long tail applications. Exactly where you'd expect parameter count to matter.
  - YmiYugy 1mo ago
    I'm confused. I thought Mythos 5 and Fable 5 were exactly the same model just with a different security layer in front of it. Could they mean the Mythos 5 Preview?
    - nl 1mo ago
      Yes. One reason why I think that report has confused Fable and Opus.
- redox99 1mo ago
  You don't really need to train a 10T model to test cerebras against a 10T model. You can feed it an untrained (randomly initialized) model and benchmark it. Result will be gibberish but performance the same.
yorwba 1mo ago
You cannot infer this because they only show the tokens per second per user. One way to get a higher number is to have fewer users per chip. I'm pretty sure Cerebras has a confidentiality agreement with OpenAI, and this press release was carefully constructed to avoid leaking details about the model weights. For example, the graph of tokens per second vs. tokens per second per user doesn't have any numbers that would allow you to translate between the two. (And in any case the relationship depends on the model.)
- petu 1mo ago
  They show that CS-4 can't really do batching (or rather it can't properly benefit from it), total throughput barely changes (25%?): https://cdn.sanity.io/images/e4qjo92p/production/6a132331880d41c8ea20b584f4fd37c270741692-1920x1080.png https://cdn.sanity.io/images/e4qjo92p/production/6a132331880... Which I think makes it feasible to approximate activation from CS-4 tokens per second per user.
Copenjin 1mo ago
Didn't they say that they can support bigger models now?
- rbanffy 1mo ago
  Memory capacity on the WSE is the same as before, but access to off-wafer memory is much slower, so the sweet spot is a given fixed balance of memory and compute. They have announced a partnership with AMD in which CPU/GPU hardware is used for part of the workload and the WSE-3 machines are used for inference for specialized smaller models, but I'm not really sure of the details on that. And there is, of course, the educated guesses about what WSE-4 will be, one being adding a LOT of stacked SRAM or DRAM to tip the balance towards memory (which could also be done by having a few different tile designs with various configurations of compute and memory capacity). I am curious about which way they'll go.

(Where did you see that?) This was also interesting: "CS-4 delivers more than 1,000 tokens per second on models exceeding 10 trillion parameters." Was it known that there were 10 trillion parameter models in use? I think the frontier providers keep the size of their models carefully hidden.

ewild 1mo ago
It's rumored fable is around that 10T number
- walrus01 1mo ago
  If this is true, it's even more impressive that some of the open weight models that are <3.5T in size, approx 33% of its size, are within a few points of it in the artificial analysis leaderboard.
  - habosa 1mo ago
    GLM 5.3 is "only" 753B parameters. Much much smaller.
  - manquer 1mo ago
    Not necessarily, there could be diminishing returns on mere parameters count . There is nothing to say for example a 1 Quadrillion parameter model will be vastly more intelligent than current SOTA especially since new training data is largely synthetic today
    - HDBaseT 1mo ago
      That's precisely what he is saying, there is diminishing returns (or optimization left on the table).
      - manquer 1mo ago
        I read it as it is impressive because smaller models 2.5T are squeezing similar returns as 10T models despite being 1/4th size not that there beyond 2T today the number or parameters do not have much meaning
        verdverm 1mo ago
        or the latest qwen3.8 27B doing so well at ~1/100 the size of K3
        rnewme 1mo ago
        What about general knowledge you can get out of it before hallucinations start?
        walrus01 1mo ago
        It did OK on schlongbench v1.0 (test of a specific niche word that doesn't make it into smaller LLMs) but it sure does love to count words https://pastes.io/r8F1AY8h https://pastes.io/r8F1AY8h
        verdverm 1mo ago
        I do not rely on any LLM of any size for general knowledge baked into the weights, they all hallucinate and that is the wrong way to hold them imo I think there is some merit in that smaller models cannot memorize so much of the training data, i.e. that they are less likely to do copyright infringement, and by analogy not having memorized SDK / API surfaces that have since changed from the training data
        walrus01 1mo ago
  - mlmonkey 1mo ago
    You want to take a look at the "Scaling Laws" paper, so you can extrapolate from these numbers.
    - stymaar 1mo ago
      This paper, as well as the Chinchilla one, aged like milk though.
  - scosman 1mo ago
    And GLM is only 0.7T! But these labs distill off the larger models. Both officially at the labs with the big ones, and unofficially. We need the giant models to get the smaller models.
- johnnyApplePRNG 1mo ago
  Fable is most definitely nowhere near 10T. The cost to train and infer that would be insane, even by today's standards.
  - nl 1mo ago
    Fable is strongly believed to be around 10T. The most conservative estimate I've seen is 8T. Eg: https://www.reuters.com/technology/bytedance-targets-mega-ai-model-nearing-anthropics-mythos-ft-reports-2026-08-07/ https://www.reuters.com/technology/bytedance-targets-mega-ai... That reports Mythos as 8T and Fable as 5T, but I think they mean Opus as 5T, which is widely known, eg: https://eu.36kr.com/en/p/3760679047267075?ref=explainx https://eu.36kr.com/en/p/3760679047267075?ref=explainx Both Grok and Bytedance are training 10T models.
    - andai 1mo ago
      Wasn't Opus ~1.5T and Fable is about twice that?
    - stymaar 1mo ago
      The fact that Musk claims Opus is 5T to justify why Grok is far behind should be taken with a massive grain of salt given he's a recidivist mythomaniac. Honestly if Opus is 5T parameters while being matched by the biggest open models that are at least twice smaller, it would mean that the US is already behind China in the AI race, despite a significant edge in compute.
      - WinstonSmith84 1mo ago
        Yes. And Opus goes a very long way compared to Fable, Anthropic isn't doing any favour, it's clearly just 2 models with a very different amount of parameters.
      - nl 1mo ago
        The open models don't really match Opus. For example I regularly do Fable+Opus agentic coding runs over 24 hours without intervention. I think I've had GLM do a run that was a few hours. That's the closest I've had an open model come on that kind of work.
alightsoul 1mo ago
pretty sure 10 trillion parameters is now the norm among closed ai labs, given that nvidia also references the same 10 trillion number for their nvl72 racks
- KronisLV 1mo ago
  Pretty bad efficiency then unless that only applies to Fable class but even then - Kimi K3 is around 3T and does similarly well in most benchmarks.
  - esafak 1mo ago
    As others have noted, most benchmarks stress the torso, not the tail.
nl 1mo ago
Mythos/Fable are around 10T: > According to FT, industry estimates say Anthropic's most advanced Mythos 5 has about 8 trillion parameters and Fable 5 about 5 trillion https://www.reuters.com/technology/bytedance-targets-mega-ai-model-nearing-anthropics-mythos-ft-reports-2026-08-07/ https://www.reuters.com/technology/bytedance-targets-mega-ai... I believe this report has confused Opus (which is known to be around 5T) and Fable. Other reports say 10T. See for example https://eu.36kr.com/en/p/3760679047267075?ref=explainx https://eu.36kr.com/en/p/3760679047267075?ref=explainx where Musk talks about the models being trained on Colossus2
- zozbot234 1mo ago
  > I believe this report has confused Opus (which is known to be around 5T) and Fable. 5T for Opus feels quite high though. DeepSeek V4 Pro is a mere 1.6T and often described as a match with Opus in overall quality. Even the largest open models in common use are around 2.8T.
  - Implicated 1mo ago
    > and often described as a match with Opus in overall quality It's not. Idk about who has more T's but, unfortunately, DS4 pro is not a match to Opus, at least not Opus 4.8.
  - nl 1mo ago
    It's not an Opus match. The difference is very visible in long tail applications. Exactly where you'd expect parameter count to matter.
- YmiYugy 1mo ago
  I'm confused. I thought Mythos 5 and Fable 5 were exactly the same model just with a different security layer in front of it. Could they mean the Mythos 5 Preview?
  - nl 1mo ago
    Yes. One reason why I think that report has confused Fable and Opus.
redox99 1mo ago
You don't really need to train a 10T model to test cerebras against a 10T model. You can feed it an untrained (randomly initialized) model and benchmark it. Result will be gibberish but performance the same.

yorwba 1mo ago

You cannot infer this because they only show the tokens per second per user. One way to get a higher number is to have fewer users per chip. I'm pretty sure Cerebras has a confidentiality agreement with OpenAI, and this press release was carefully constructed to avoid leaking details about the model weights. For example, the graph of tokens per second vs. tokens per second per user doesn't have any numbers that would allow you to translate between the two. (And in any case the relationship depends on the model.)

petu 1mo ago
They show that CS-4 can't really do batching (or rather it can't properly benefit from it), total throughput barely changes (25%?): https://cdn.sanity.io/images/e4qjo92p/production/6a132331880d41c8ea20b584f4fd37c270741692-1920x1080.png https://cdn.sanity.io/images/e4qjo92p/production/6a132331880... Which I think makes it feasible to approximate activation from CS-4 tokens per second per user.

Copenjin 1mo ago

Didn't they say that they can support bigger models now?

rbanffy 1mo ago
Memory capacity on the WSE is the same as before, but access to off-wafer memory is much slower, so the sweet spot is a given fixed balance of memory and compute. They have announced a partnership with AMD in which CPU/GPU hardware is used for part of the workload and the WSE-3 machines are used for inference for specialized smaller models, but I'm not really sure of the details on that. And there is, of course, the educated guesses about what WSE-4 will be, one being adding a LOT of stacked SRAM or DRAM to tip the balance towards memory (which could also be done by having a few different tile designs with various configurations of compute and memory capacity). I am curious about which way they'll go.