7 ms·

TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1

bensyverson 2mo ago
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
NitpickLawyer 2mo ago
There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
- PcChip 2mo ago
  this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values
  - zawaideh 2mo ago
    It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
    - throwawayffffas 2mo ago
      I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer. The way they do it is packing like the other comment says. Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.
  - petu 2mo ago
    packing multiple trits together e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing https://compilade.net/blog/ternary-packing
    - JoshTriplett 2mo ago
      It's impressive how close to optimal this is. You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59). At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.
  - edflsafoiewq 2mo ago
    It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128. Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT. It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
    - snthpy 2mo ago
      Thanks. This relates to some questions I've got. I was playing around with the previous generation smaller models and found that i wasn't getting any speedups from the T1 and T2 binary/ternary models compared to standard Q4 quants of straight qwen3.6 models. I was wondering whether unpacking of the ternary encoding was impacting inference speed? If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.
  - lioeters 2mo ago
    > never heard of a bit ever having more than two possible values It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.
Dwedit 2mo ago
1.6 bits if you want the most practical way to pack five 3-state numbers into a single byte. But even then, they usually pack four 4-state numbers instead.

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.

NitpickLawyer 2mo ago

There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

PcChip 2mo ago
this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values
- zawaideh 2mo ago
  It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
  - throwawayffffas 2mo ago
    I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer. The way they do it is packing like the other comment says. Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.
- petu 2mo ago
  packing multiple trits together e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing https://compilade.net/blog/ternary-packing
  - JoshTriplett 2mo ago
    It's impressive how close to optimal this is. You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59). At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.
- edflsafoiewq 2mo ago
  It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128. Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT. It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
  - snthpy 2mo ago
    Thanks. This relates to some questions I've got. I was playing around with the previous generation smaller models and found that i wasn't getting any speedups from the T1 and T2 binary/ternary models compared to standard Q4 quants of straight qwen3.6 models. I was wondering whether unpacking of the ternary encoding was impacting inference speed? If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.
- lioeters 2mo ago
  > never heard of a bit ever having more than two possible values It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.

Dwedit 2mo ago

1.6 bits if you want the most practical way to pack five 3-state numbers into a single byte. But even then, they usually pack four 4-state numbers instead.