10 ms·
BitNet: Inference framework for 1-bit LLMs
Paper: https://arxiv.org/pdf/2310.11453 https://arxiv.org/pdf/2310.11453
- ilovesamaltman 6mo ago[flagged]
- QuadmasterXLII 6mo agoheadline hundred billion parameter, none of the official models are over 10 billion parameters. Curious.
- Tuna-Fish 6mo agoThe project is an inference framework which should support 100B parameter model at 5-7tok/s on CPU. No one has quantized a 100B parameter model to 1 trit, but this existing is an incentive for someone to do so.
- est 6mo ago> quantized a 100B parameter model to 1 trit I had the same question, after some debates with Chatgpt, it's not the "quantize" for post-training we often witness these days, you have to use 1 trit in the beginning since pre-train.
- 152334H 6mo agobut there is no trained 100b param model? "can run a 100B BitNet" is about the inference implementation, not about the existence of any such model
- webXL 6mo agoI think they used a dummy model or else they would have linked to it. Just google '1-bit 100b model' and you'll only see references to this project without any download links.
- radarsat1 6mo agoI'm curious if 1-bit params can be compared to 4- or 8-bit params. I imagine that 100B is equivalent to something like a 30B model? I guess only evals can say. Still, being able to run a 30B model at good speed on a CPU would be amazing.
- regularfry 6mo agoAt some point you hit information limits. With conventional quantisation you see marked capability fall-off below q5. All else being equal you'd expect an N-parameter 5-bit quant to be roughly comparable to a 3N-parameter ternary, if they are trained to the same level, just in terms of the amount of information they can possibly hold. So yes, 100B ternary would be within the ballpark of a 30B q5 conventional model, with a lot of hand-waving and sufficiently-smart-training
- cubefox 6mo agoI assume that theoretically, 1-bit models could be most efficient because modern models switched from 32 bit to 16 bit to 8 bit per parameter (without quantization).
- regularfry 6mo agoIt's not clear where the efficiency frontier actually is. We're good at measuring size, we're good at measuring FLOPS, we're really not very good at measuring capability. Because of that, we don't really know yet whether we can do meaningfully better at 1 bit per parameter than we currently get out of quantising down to that size. Probably, is the answer, but it's going to be a while before anyone working at 1 bit per param has sunk as many FLOPS into it as the frontier labs have at higher bit counts.
- cubefox 6mo agoThe thing with efficiency is that it is relative to both inference and training compute. If you do quantization, you need a more powerful higher precision model to quantize from, which doesn't exist if you want to create a frontier model. In this case the question is only whether you get better inference and/or training performance from training e.g. a native 1 bit model. Currently the optimal training precision seems to be 8 bit (at least used by DeepSeek and some other open weight companies). But this might change with different training methods optimized for 1-bit training, like from this paper I linked before: https://proceedings.neurips.cc/paper_files/paper/2024/hash/718a3c5cf135894db6e718725f52ef9a-Abstract-Conference.html https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...
- syntaxing 6mo agoMisleading title but this is pretty exciting. Interesting how this is based on llama cpp. Its nice to see some momentum since they released the paper in 2023
- LuxBennu 6mo agoThe title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on commodity CPUs. If 5-7 tok/s on a single CPU for 100B-class models is reproducible, that's a real milestone for on-device inference. Framework is ready. Now we need someone to actually train the model.
- rustyhancock 6mo agoYes. I had to read it over twice, it does strike me as odd that there wasn't a base model to work with. But it seems the biggest model available is 10B? Somewhat unusual and does make me wonder just how challenging it will be to train any model in the 100B order of magnitude.
- wongarsu 6mo agoApproximately as challenging as training a regular 100B model from scratch. Maybe a bit more challenging because there's less experience with it The key insight of the BitNet paper was that using their custom BitLinear layer instead of normal Linear layers (as well as some more training and architecture changes) lead to much, much better results than quantizing an existing model down to 1.58 bits. So you end up making a full training run in bf16 precision using the specially adapted model architecture
- naasking 6mo agoWhat's unusual about it? It seems pretty standard to train small models to validate an approach, and then show that training scales with model size to 8B to 14B parameter models, which is what they did.
- wongarsu 6mo agoI've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be orders of magnitude simpler than what Nvidia is making
- nickcw 6mo ago> bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next). One bit or one trit? I am confused!
- drsopp 6mo ago"1-bit LLMs" is just marketing. The Shannon entropy of one letter with a 3 symbol alphabet (-1, 0, 1) is 1.58.
- Dwedit 6mo agoLog Base 2 of 3 = ~1.5849625, so that's the limit to how well you can pack three-state values into bits of data. For something more practical, you can pack five three-state values within a byte because 3^5 = 243, which is smaller than 256. To unpack, you divide and modulo by 3 five separate times. This encodes data in bytes at 1.6 bits per symbol. But the packing of 5 symbols into a byte was not done here. Instead, they packed 4 symbols into a byte to reduce computational complexity (no unpacking needed)
- rasz 6mo ago>1-bit model >packed 4 symbols into a byte microslop, typical bunch of two-bit frauds!
- cubefox 6mo agoYeah, "1.58 bit" is 1 trit with three states, since log2(3)≈1.58. So it's not a inference framework for 1-bit models (two states per parameter) but for 1.58 bit models (three states per parameter). Annoying that they try to mix up the two.
- silon42 6mo agoI always hope for "just a bunch of if statements" ... this is not it.
- itsthecourier 6mo agohttps://github-production-user-asset-6210df.s3.amazonaws.com/54800242/377447164-7f46b736-edec-4828-b809-4be780a3e5b1.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20260311%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260311T131116Z&X-Amz-Expires=300&X-Amz-Signature=be8d1f20e916f54fdd7c2a0a639041a9171f2f59db84e7fd54a8135662ff8829&X-Amz-SignedHeaders=host https://github-production-user-asset-6210df.s3.amazonaws.com... demo shows a huge love for water, this AI knows its home
- _fw 6mo agoAlso, very influenced by the literature of Jenkins (2010).
- giancarlostoro 6mo agoOne of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think. Wikipedia is nice, but there's so much room for human error and bias there.
- embedding-shape 6mo agoYour worry about Wikipedia is that there is "much room for human error and bias", yet earlier you seem to imply that a LLM that has access to the www somehow would have less human error and bias? Personally, I'd see it the other way around.
- giancarlostoro 6mo agoWhen GPT 3.5 became a thing, it had crawled a very nuanced set of websites, this is what I mean. You basically curate where it sources data from.
- utopiah 6mo ago> validating outputs for LLM companies How? They can validate thousands if not millions of queries but nothing prevent the millions-th-and-one from being a hallucination. People who would then pay extra for a "Encyclopedia Britanica validated LLM" would then, rightfully so IMHO, complain that "it" suggested them to cook with a dangerous mushroom.
- uniq7 6mo agoSince Google Search already includes an AI summary, your minimally viable "LLM" can be just an HTTP GET call
- intrasight 6mo agoIt's not so much a "minimally viable LLM" but rather an LLM that knows natural language well but knows nothing else. Like me - as an engineer who knows how to troubleshoot in general but doesn't know about a specific device like my furnace (recent example). And I don't think that LLM could just Google or check Wikipedia. But I do agree that this architecture makes a lot of sense. I assume it will become the norm to use such edge LLMs.
- Arcuru 6mo agoIt's good to see this getting some continued development. I looked into it last year[1] and I thought it showed a lot of promise so I've been very disappointed that I never saw a newer model. [1] - https://jackson.dev/post/dont-sleep-on-bitnet/ https://jackson.dev/post/dont-sleep-on-bitnet/
- cubefox 6mo agoI think this approach is not so interesting because it's just quantization of a full precision model. So it speeds up inference (at a quality penalty) but not training. It would be more interesting to train an actually binary model directly, without any floating point multiplication, like in this paper: https://proceedings.neurips.cc/paper_files/paper/2024/hash/718a3c5cf135894db6e718725f52ef9a-Abstract-Conference.html https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...
- algoth1 6mo agoHeadline: 100B. Falcon 3 family: 10B. An order of magnitude off
- bee_rider 6mo agoWhat’s the lower limit on the number of bits per parameter? If you use CSR-style sparse matrices to store the weights can it be less than 1?
- simonw 6mo agoAnyone know how hard it would be to create a 1-bit variant of one of the recent Qwen 3.5 models?
- regularfry 6mo agoThere are q2 and q1 quants, if you want an idea of how much performance you'd drop. Not quite the same implementation-wise, but probably equivalent in terms of smarts.
- nikhizzle 6mo agoAlmost trivial using open source tools, the question is how it performs without calibration/fine tuning.
- wongarsu 6mo agoThe results would probably be underwhelming. The bitnet paper doesn't give great baselines to compare to, but in their tests a 2B network trained for 1.58bits using their architecture was better than Llama 3 8B quantized to 1.58bits. Though that 2B network was about on par with a 1.5B qwen2.5. If you have an existing network, making an int4 quant is the better tradeoff. 1.58b quants only become interesting when you train the model specifically for it On the other hand maybe it works much better than expected because llama3 is just a terrible baseline
- philvas 6mo agosteve jobs would have loved the microsoft repo with demo on mac
- rarisma 6mo agoNo 100b model. My disappointment is immeasurable and my day is ruined.
- devnotes77 6mo ago[flagged]
- regularfry 6mo agoLower down the stack you also get better cache efficiency. More parameters per byte means more parameters per cache line.
- WhitneyLand 6mo agoIf they had a big result like, native 1.58-bit quality clearly matches top peers, they would be saying that prominently in the repo. The engineering/optimization work is nice, but this is not what people have been waiting for, as much as, can’t the Bitnet idea that seemed promise really deliver in a competitive way.
- StilesCrisis 6mo agoThe output from this model is horrible! It's GPT-2 level babble and repeats entire paragraphs verbatim. It also reuses the same fake citation `(Jenkins, 2010)` over and over again. From the start of their video (which scrolls by fast enough that you don't see the slop clearly...) ``` Ecosystem Services and their impact on the Ecosystem Ecosystem services refer to the services provided by ecosystems to the human society. These services include water, air, energy, nutrients, and soil (Jenkins, 2010). For instance, water is the most important service provided by an ecosystem and it helps in the conservation of water, irrigation and sanitation (Jenkins, 2010). On the other hand, air provides the oxygen needed for life. The water cycle is a significant ecosystem service because it involves the cycling of water among the different parts of an ecosystem. It also involves the movement of water through the atmosphere, from one place to another. It is also the process of evaporation and condensation of water from the atmosphere. It also involves the movement of water from the air to the soil and water into the oceans. The water cycle is a significant ecosystem service because it involves the cycling of water among the different parts of an ecosystem. It also involves the movement of water through the atmosphere, from one place to another. It is also the process of evaporation and condensation of water from the atmosphere. It also involves the movement of water from the air to the soil and water into the oceans. ```
- naasking 6mo agoIt's a two year old base model that's only 3B parameters, trained on only 100B tokens. It's still a research project at this point.
- gardnr 6mo agoThe new model they just released has impressive benchmark results: https://huggingface.co/microsoft/bitnet-b1.58-2B-4T https://huggingface.co/microsoft/bitnet-b1.58-2B-4T Except on GSM8K and math...
- naasking 6mo agoThanks for the link, the GSM8K result actually leads the pack in that table, but math is indeed underwhelming. Qwen 2.5 is in the lead, but bitnet isn't far behind and it takes 1/6th as much memory during inference, and was trained on less than 1/4 the number of tokens. Pretty cool.
- logicallee 6mo agoIt might interest you to know that one or two months ago, I had Claude port BitNet to WebGPU from the reference implementation, so that it runs right in your browser as a local model. After some debugging, the port seemed to work, but the model didn't function as well as the reference implementation so I'll have to work on it for a while. You can see a debugging session livestreamed here[1]. The released model file was about a gigabyte, it fits in most people's GPU's. We were also able to successfully fine-tune it right in the browser. There's a lot that you can do when the model size is that small, yet still powerful. Our next step is that we want to put up a content distribution network for it where people can also share their diffs for their own fine-tuned model. I'll post the project if we finish all the parts. [1] https://www.youtube.com/live/x791YvPIhFo?is=NfuDFTm9HjvA3nzN https://www.youtube.com/live/x791YvPIhFo?is=NfuDFTm9HjvA3nzN
- naasking 6mo agoI think the README [1] for the new CPU feature is of more interest, showing linear speedups with number of threads. Up to 73 tokens/sec with 8 threads (64 toks/s for their recommended Q6 quant): https://github.com/microsoft/BitNet/blob/main/src/README.md https://github.com/microsoft/BitNet/blob/main/src/README.md
- a1o 6mo ago> A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2 With how much RAM? How much storage does it requires?
- yubainu 6mo ago[flagged]
- herf 6mo agohttps://arxiv.org/pdf/2310.11453 https://arxiv.org/pdf/2310.11453 The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model. You can build it and run some models, but the selection is limited because it has to be trained from scratch. I imagine inference speed is faster compared with modern PTQ (4- and 8-bit quants) though.
- deleted 6mo ago[deleted]
- leventilo 6mo agoThe energy numbers are the real story here, 70-82% reduction on CPU inference. If 1-bit models ever get good enough, running them on commodity hardware with no GPU budget changes who can deploy LLMs. That's more interesting than the speed benchmarks imo.
- lemonish97 6mo agoI wonder when we begin to see the dividends of all the NPU PCs come into play. AMD have been doing some good work with their NPU/iGPU hybrid inference kernels. If these larger models could be scaled down to run on NPUs, you'd see much better power advantages, compared to running them on the CPU.
- Havoc 6mo agoYou can already run some models on the NPUs in the Rockchip RK3588 SBCs which are pretty abundant. A claude 4.6 they are most certainly not, but if you get through the janky AF software ecosystem they can run small LLMs reasonably well with basically zero CPU/GPU usage
- throwa356262 6mo agoAre the NPUs really that powerful? I was under the impression that they were primarily designed for low power use.
- lemonish97 6mo agoThey seem to be getting better or more powerful. The newer Intel Panther lakes and AMD Ryzen are over 50 TOPS now, IIRC
- cheema33 6mo ago> I wonder when we begin to see the dividends of all the NPU PCs come into play. A few months ago I used Whisper from OpenAI, an automatic speech recognition system released in 2002, on my modern 20-core Intel CPU to convert audio from a video file to text. It worked fine. Took a while and the machine got hot and the fans kicked in. I then found the Intel's optimized version of whisper that used NPU. It required a lot more steps to get working, but in the end it did work and was about 6x faster. And the machine remained cool and silent in the process. Since then I have become a fan of the NPUs. They are not NVIDIA GeForce RTX 5090, but they are significantly better than a modern CPU.
- knodi123 6mo agoWhy would they film a demo video of it spewing out barely-coherent rambling repetitive drivel? If your model sucks at writing essays, maybe just tell us that, and film a demo of it doing something it IS good at?
- patchnull 6mo ago[flagged]
- perfmode 6mo agoThe quality cliff question is the right one to be asking. There's a pattern in systems work where something that scales cleanly in theory hits emergent failure modes at production scale that weren't visible in smaller tests. The loss landscape concern is exactly that kind of thing, and nobody has actually run the experiment. That said, I think the comparison to improving GGUF quantization isn't quite apples to apples. Post-training quantization is compressing a model that already learned its representations in high precision. Native ternary training is making an architectural bet that the model can learn equally expressive representations under a much tighter constraint from the start. Those are different propositions with different scaling characteristics. The BitNet papers suggest the native approach wins at small scale, but that could easily be because the quantization baselines they compared against (Llama 3 at 1.58 bits) were just bad. A full-precision model wasn't designed to survive that level of compression. The real tell will be whether anyone with serious compute (not Microsoft, apparently) decides the potential inference cost savings justify a full training run. The framework existing lowers one barrier, but the more important barrier is that a failed 100B training run is extremely expensive, and right now there's not enough evidence to derisk it. Two years of framework polish without a flagship model is a notable absence.
- andai 6mo ago>Meanwhile GGUF Q2 and Q3 quantizations on llama.cpp keep getting better Can you tell me more about this? It's been about a year since I looked into it, but it looked like performance dropped hard below Q4. I'd love to see more about this. Also what's a good way to run them? I mostly use Ollama which only goes down to Q4. I think it supports HF urls though?
- password4321 6mo agoThis recent discussion is still open and may provide some helpful info: How to run Qwen 3.5 locally https://news.ycombinator.com/item?id=47292522 https://news.ycombinator.com/item?id=47292522
- aplomb1026 6mo ago[flagged]
- janalsncm 6mo agoThey have a demo video in the readme. I think they are trying to convey that BitNet is fast, which it is. But it is worth taking a moment to pause and actually see what the thing is doing so quickly. It seems to keep repeating that the water cycle is the main source of energy for all living things on the planet and then citing Jenkins 2010. There are also a ton of sentence beginning with “It also…” I don’t even think it’s correct. The sun is the main source of energy for most living things but there’s also life near hydrothermal vents etc. I don’t know who Jenkins is, but this model appears to be very fond of them and the particular fact about water. I suppose fast and inaccurate is better than slow and inaccurate.
- 5o1ecist 6mo ago[dead]
- hinkley 6mo agoDo LLMs have a way to look at or consider dependent variables? Seems like that could end up as a situation where a fractional number of bits or bytes per parameter might make sense. Particularly with adverbs and adjectives, negators.
- htk 6mo agoSo Microsoft is actually using 2 bits instead of 1.58. In this case they could represent -1, 0, 1, 2. As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured. Does that make sense?
- hrimfaxi 6mo agoCan you explain your third statement? > As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured.
- deleted 6mo ago[deleted]
- DoctorOetker 6mo agoIn the human brain most synapses are indeed excitatory, while a minority is inhibitory. No concise HN comment will give you a complete picture of whats currently known about the human brain, so a platitude necessarily follows: We call the nearly touching interfaces between neurons synapses, small packets / droplets of neurotransmitter are sent across this interface from the source to the target neuron. Such signals can be excitatory (promote the probability of excitation of the target firing soon) or inhibitory (inhibits the probability of the target firing soon). There are 2 types of sensitive areas on your average neuron: the dendrites (long branching tentacles, that receive excitatory signals) and the cell body where all the signals are accumulated to a local instantaneous "sum" is also sensitive to synaptic activation, but the synapses on the cell body are inhibitory, when sufficiently inhibited the neuron will refuse to fire its axons, so the inhibitory synapses on the cell body can gate the cumulative signal and prevent it from triggering this neuron temporarily. If the neuron does fire, this propagates along the axons (another type of branching tentacles, which lead to yet other neurons, sometimes touching them excitatorily at their dendrite, sometimes touching a neuron inhibitorily at their cell body. I hope that helped?
- vermilingua 6mo agoIt is really truly incredible that this mess of microscopic meat plumbing encodes everything we see, think, and do. Terrifying and amazing all at once.
- Herring 6mo agoIf this stuff was so revolutionary, don't you guys think Qwen/DeepSeek would have snapped it up already? Both those teams are highly innovative, picking up and inventing new techniques all the time. Hell, Deepseek-v3 was one of the first to do large scale fp8 training.
- docybo 6mo ago[flagged]
- faldore 6mo agoWhy did you call it a 100b parameter model? it is not 100b parameters. they published a 1b parameter and a 2b parameter model. Furthermore, it was published 11 months ago, it's not a new release.
- kristopolous 6mo agoI don't see the news here ... there's https://huggingface.co/collections/microsoft/bitnet https://huggingface.co/collections/microsoft/bitnet which is last updated 12/2025 ... am I just paying more attention here or is there something actually new about this? Also as far as I know, this is more of a research curiosity - BitNet really doesn't perform that well on evals. I think Qwen3.5 2B is the best you can get in the ~1GB class.
- almaight 6mo agoCould this ternary model be more easily replicated on the Taalas HC1?