7 ms·

Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit https://huggingface.co/prism-ml/Te

by smeeth 2mo ago

Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.

dwa3592 2mo ago
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
- dTal 2mo ago
  >in case they start to ban open source AI development Is that likely, do you think?
  - dwa3592 2mo ago
    dario has been saying open source models are dangerous. who knows who is listening to him.
    - mdp2021 2mo ago
      The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)". There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
  - submain 2mo ago
    NVidia seems to want them. They have a lot of lobbying power.
acchow 2mo ago
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
- yieldcrv 2mo ago
  Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
- smeeth 2mo ago
  1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact. even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
  - Aurornis 2mo ago
    > yes. https://www.alphaxiv.org/abs/2607.bonsai-27b https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks. Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6. I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
flockonus 2mo ago
There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat! https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit

by smeeth 2mo ago

dwa3592 2mo ago

agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.

dTal 2mo ago
>in case they start to ban open source AI development Is that likely, do you think?
- dwa3592 2mo ago
  dario has been saying open source models are dangerous. who knows who is listening to him.
  - mdp2021 2mo ago
    The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)". There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
- submain 2mo ago
  NVidia seems to want them. They have a lot of lobbying power.

acchow 2mo ago

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

yieldcrv 2mo ago
Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
smeeth 2mo ago
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact. even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
- Aurornis 2mo ago
  > yes. https://www.alphaxiv.org/abs/2607.bonsai-27b https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks. Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6. I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.

flockonus 2mo ago

There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat! https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit