25 ms·
Run DeepSeek R1 Dynamic 1.58-bit
- cubefox 2y agoFor anyone wondering why "1.58" bits: 2^1.58496... = 3. The weights have one of the three states {-1, 0, 1}.
- dist-epoch 2y agoThey say something else: > We managed to selectively quantize certain layers to higher bits (like 4bit), and leave most MoE layers (like those used in GPT-4) to 1.5bit
- cubefox 2y agoThat was just improper rounding from 1.58 to 1.5. They say 1.58 in other places and explicitly link to https://arxiv.org/abs/2402.17764 https://arxiv.org/abs/2402.17764
- deleted 2y ago[deleted]
- tarruda 2y agoWould be great if the next generation of base models was designed to be inferred with 128GB of VRAM while 8bit quantized (which would fit in the consumer hardware class). For example, I imagine a strong MoE base with 16 billion active parameters and 6 or 7 experts would keep a good performance while being possible to run on 128GB RAM macbooks.
- Davidzheng 2y agoWould be great, but unfortunately i think intelligence at that compute scale will be limit by hardware not its model. Though at hardware limit I would expect it to be roughly human level especially if optimized for a particular domain.
- tarruda 2y agoI remember that Llama 3 was trained on data curated by Llama 2 and it resulted in a model with a significant performance boost (even though it was trained by a previous generation model of the same size). Maybe using a strong reasoning model such as R1 the next generation, even more performance can be extracted from smaller models.
- danielbln 2y agoThat's already happening, and is in fact even part of the R1 training pipeline. An intermediate small reasoning model churns out training data for RL a larger model, rinse and repeat. Deepseek also showed model distillation with synthetic reasoning data to work quite well.
- alchemist1e9 2y agoIt’s a pretty neat paradigm and I see an abstract connection to how brains dream and produce their own synthetic training data while sleeping that supplements their real data used while awake.
- Davidzheng 2y agoIs your first claim in the R1 paper? I didn't see it when I looked
- danielhanchen 2y agoSo I remember Deepseek used float8 for training - Character AI also used int8 for training - it is indeed possible, but sometimes training can be unstable - Deepseek to my knowledge is actually the first lab to use float8 at a large scale without causing loss spikes - they used FP8 tensor cores, then every 4th matrix multiply, they accumulated to a FP32 accumulator - it seems like the Hopper Tensor Cores accumulation mechanism might not be actual FP32 accumulation. I wrote more here: https://x.com/danielhanchen/status/1872719599029850391 https://x.com/danielhanchen/status/1872719599029850391
- raghavbali 2y ago> Unfortunately if you naively quantize all layers to 1.58bit, you will get infinite repetitions in seed 3407: “Colours with dark Colours with dark Colours with dark Colours with dark Colours with dark” or in seed 3408: “Set up the Pygame's Pygame display with a Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's”. This is really interesting insight (although other works cover this as well). I am particularly amused by the process by which the authors of this blog post arrived at these particular seeds. Good work nonetheless!
- littlestymaar 2y agoCan't this kind of repetition be dealt with at the ~~decoder~~ (edit: sampler) level, like for any models? (see DRY ~~decoder~~ sampler for instance: https://github.com/oobabooga/text-generation-webui/pull/5677 https://github.com/oobabooga/text-generation-webui/pull/5677)
- ErikBjare 2y agoYou can deal with this through various sampling methods, but it doesn't actually fix the fried model.
- danielhanchen 2y agoOh yes one could provide a repetition penalty for example - the issue is it's not just repetition that's the issue. I find it rather forgets what it already saw, and so hence it repeats stuff - it's probably best to backtrack, then delete the last few rows in the KV cache. Another option is to employ min_p = 0.05 to force the model not to generate low prob tokens - it can help especially in the case when the 1.58bit model generates on average 1/8000 tokens or so an "incorrect" token (for eg `score := 0`)
- reichardt 2y agoYou likely mean sampler, not decoder. And no, the stronger the quantization, the more the output token probabilities diverge from the non-quantized model. With a sampler you can't recover any meaningful accuracy. If you force the sampler to select tokens that won't repeat, you're just trading repetitive gibberish for non-repetitive gibberish.
- ThePhysicist 2y agoIn general, how do you run these big models on cloud hardware? Do you cut them up layer-wise and run slices of layers on individual A100/H100s?
- teekert 2y agoWas wondering the same, but for HPC clusters :)
- amelius 2y agoYou could do that, and add pipelining to improve speed.
- phire 2y agoMy understanding is with MoE (Mixture of Experts), you can and should shard it horizontally. The whole model is 600GB, but only 37GB is active during the evaluation of any single output token. So you can load a different active subset of the MoE into each 89GB GPU, sharding it across something like 32 different GPUs (or can you get away with less? Wouldn't be surprised if they can infer on 8x H800 gpus). Some parameters are common, others are independent. Queries can be dynamically routed between GPUs, potentially bouncing between GPUs as much as once per output token, depending on which experts they need to activate. Though, I suspect it's normal to stick on one MoE subset for several output tokens. This has a secondary benefit that as long as the routing distribution is random, queries should be roughly load balanced across all GPUs.
- yorwba 2y agoEach MoE layer has its own router, and it activates 8 (out of 256) experts at a time. There's no reason to expect all of them to stay on the same GPU, so you're pretty much guaranteed to have to do all-to-all communication between the GPUs in your cluster after every layer for every token.
- phire 2y agoInteresting. I had assumed the performance advantage for MoE came from minimising traffic between GPUs. But if it's per layer routing, then it's going to massively increase inter-gpu traffic compared to vertical slicing. I guess that means the performance advantage actually comes when batching thousands of queries? The MoE routing would mean that on each MoE layer, each GPU shard gets a batch of queries that will all hit roughly the same subset of experts (and read the same weights from memory). The batches then shuffle between each MoE layer to re-optimise. It's kind of like GPU raytracing where you get large performance gains by running coherency sorting on rays and batching similar rays together.
- upghost 2y agoThanks for the run instructions, unsloth. Deepseek is so new it's been breaking most of my builds.
- danielhanchen 2y agoGlad they were helpful! :)
- marcodiego 2y agoThis is an important step. Especially for beginners or people who are not in the loop, being able to easily type some simple commands to download, install dependencies, compile and run everything needed for a LLM AI model gives a feeling sci-fi; it's almost like you can have a helping brain at home. One thing I've being thinking about doing is to combine one of those LLM models running in llama.cpp, feed it with the output of whisper.cpp and connect its output to some TTS model. I wonder how far from Wheels and Roadie from the Pole Position tv series.
- homarp 2y agosee also https://news.ycombinator.com/item?id=42846588 https://news.ycombinator.com/item?id=42846588
- terryjiao 2y ago[dead]
- apples_oranges 2y agoRandom observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.
- anakaine 2y agoI disagree with cancelling the OpenAI subscription. I've been getting some help from o1 for both python and php recently, and o1 was doing massively better for the python stuff (it ran, deepseeks didn't and wont with prompt refinement).
- gradus_ad 2y agoWere you running a local model?
- neom 2y agoAlso for some philosophical stuff DeepSeek just won't do it. I'm working on an essay about spirituality and sometimes it just responds that it doesn't know how to work on those types of problems and we should do something fun like math or games, claud tends to reply with something more like "I have to be honest with you, reincarnation is not real" and ChatGPT doesn't seem to care about that kinda thing at all.
- wqaatwt 2y agoJust don’t ask it about anything related to Tiananmen square or president Pooh.. I’d guess they didn’t quite a bit of fine tuning to censor some more sensitive topics which probably impacts the output quality for other non technical subjects.
- greenavocado 2y agoWould fine-tuning by using a LoRA paper over the censorship to a large degree?
- deleted 2y ago[deleted]
- miohtama 2y agoFlappy Bird in Python is the new Turing test
- danielhanchen 2y ago:) It's my goto test :) I did amp it up by adding 10 conditions and made a scoring card - I found the original R1 to sometimes forget "import os" or miss some lines as well, so I thought it was at least a good check! I also like to ask the models to create a simple basic Minecraft type game where you can break pieces and store them in your inventory, but disallow building stuff
- miohtama 2y agoI feel any AI can fix those problems when they can finally act. The problem AIs cannot run or debug code, or even book a hotel for me. When that is solved and an AI can interact with the code like a human does, it can fix its problems like a human does.
- merman 2y agoExactly! Why can’t LLMs run their own code?
- Applejinx 2y agoRampancy.
- whimsicalism 2y agothey can, feel free to inference and give it an interpreter
- mclau156 2y agohopefully we eventually push them to make more classic games like motherlode
- brap 2y agoAs someone who is out of the loop, what’s the verdict on R1? Was anyone able to reproduce the results yet? Is the claim that it only took $5M to train generally accepted? It’s a very bold claim which is really shaking up the markets, so I can’t help but wonder if it was even verified at this point.
- huijzer 2y ago> Is the claim that it only took $5M to train generally accepted? Based on Nvidia being down 18% yesterday I would say the claim is generally accepted.
- deskamess 2y ago> Nvidia being down 18% The only part of DeepSeek-R1 I do not like. I hope it's over, but I am not holding my breath.
- coffeebeqn 2y agoNvidia is now up only 1906% over 5 years. What a disgrace
- samvher 2y agoIt crashed all the way back to June 2024 levels, eons of progress wiped out
- tarruda 2y agoIt is still unconfirmed since no one outside of deepseek reproduced it. If confirmed, Nvidia could go down even more
- Wheaties466 2y agobased on information and background they thoroughly gave when releasing their research its pretty easy to put together that it did take them significantly less resources to train this model. only having specific parameters available at a time instead of activating everything all at once is pretty ingenious. that and they just happened to be undergoing a large scale "cyber attack"
- mtrovo 2y agoWow, an 80% reduction in size for DeepSeek-R1 is just amazing! It's fantastic to see such large models becoming more accessible to those of us who don't have access to top-tier hardware. This kind of optimization opens up so many possibilities for experimenting at home. I'm impressed by the 140 tokens per second speed with the 1.58-bit quantization running on dual H100s. That kind of performance makes the model practical for small or mid sized shops to use it for local applications. This is a huge win for people working on agents that require low latency that only local models could support.
- danielhanchen 2y agoI was pleasantly surprised by 140 tokens/s as well! I literally thought I did something wrong but it was real!
- paradite 2y agoBtw completely off topic, but your comment triggered the internal classification in my brain, and it looks like AI-generated. Not accusing you anything. Could be that you happen to write in a way similar to LLMs. Could be that we are influenced by LLM writing styles and are writing more and more like LLMs. Could be that the difference between LLM generated content and human-generated content is getting smaller and harder to tell.
- ahmeneeroe-v2 2y agoVery funny, I didn't mentally jump to LLM, but the language was so lifeless that I stopped reading. Amazing that OP confirmed you're correct (and good use of LLM @OP).
- j_bum 2y ago+1 my LLM spidy senses were tingling. It’s the exclamation point in the first paragraph, the concise and consistent sentence structure, and the lack of colloquial tone. OP, no worries if you’re real. I often read my own messages or writing and worry that people will think I’m an LLM too.
- CodeCompost 2y agoCan I run this on ollama?
- benoitg 2y agoYes, the instructions are in the OP.
- sylware 2y agosite is javascript walled 80%? On 2 H100 only? To get near chatgpt 4? Seriously? The 671B version??
- fsflover 2y ago> site is javascript walled I use Qubes OS to protect myself from the JS.
- sylware 2y agoThat site should work with a noscript/basic (x)html browser.
- whimsicalism 2y agothey have not benchmarked the quantized model.
- Jasondells 2y agoAn 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skeptical about how practical this really is for most people. Like, yeah, you can run it on 24GB VRAM or even with just 20GB RAM, but "slow" is an understatement—those speeds would make even the most patient person throw their hands up. And then there’s the whole repetition issue. Infinite loops with "Pygame’s Pygame’s Pygame’s" kind of defeats the point of quantization if you ask me. Sure, the authors have fixes like adjusting the KV cache or using min_p, but doesn’t that just patch a symptom rather than solve the actual problem? A fried model is still fried, even if it stops repeating itself. On the flip side, I love that they’re making this accessible on Hugging Face... and the dynamic quantization approach is pretty brilliant. Using 1.58-bit for MoEs and leaving sensitive layers like down_proj at higher precision—super clever. Feels like they’re squeezing every last drop of juice out of the architecture, which is awesome for smaller teams who can’t afford OpenAI-scale hardware. "accessible" still comes with an asterisk. Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? For that price, I’d rather build a rig with used 3090s and get way more bang for my buck (though, yeah, it’d be a power hog). Cool tech—no doubt—but the practicality is still up for debate. Guess we'll see if the next-gen models can address some of these trade-offs.
- danielhanchen 2y agoOh the repetition issue is only on the non dynamic quants :) If you do dynamic quantization and use the 1.58bit dynamic quantized model the repetition issue fully disappears! Min_p = 0.05 was a way I found to counteract the 1.58bit model generating singular incorrect tokens which happen around 1 token per 8000!
- smcleod 2y agomin_p is great, do you apply a small amount of temperate as well?
- danielhanchen 2y ago
- CHB0403085482 2y agoDeepSeek R1 in a nutshell youtube.com/watch?v=Nl7aCUsWykg
- petesergeant 2y agoIt is going to be truly fucking revolutionary if open-source models are and continue to be able to challenge the state of the art. My big philosophical concern is that AI locks Capital into an absolutely supreme and insurmountable lead over Labour, and into the hands of oligarchs, and the possibility of a future where that's not case feels amazing. It pleases me greatly that this has Trump riled up too, because I think it means he's much less likely to allow existing US model-makers to build moats, as I think he's -- even as a man who I don't think believes in very much -- absolutely unwilling to let the Chinese get the drop on him over this.
- fullstackchris 2y agoI have no doubt open source will catch up (it already has, eh?) at the end of the day, it's just creative / new iterations on what is ultimately the transformer architecture... the amount of "secret" moat-like stuff that OpenAI was doing was bound to be figured out or exceeded eventually, like everything in tech... Not to make fun of OpenAI and the great work they've done but it's kinda like if I went out in the 90s and said I'm going to found a company to have the best REST APIs. You can always found a successful tech company, but you can't found a successful tech company on a technological architecture or pattern alone.
- amusingimpala75 2y ago> DeepSeek-R1 has been making waves recently by rivaling OpenAI's O1 reasoning model while being fully open-source. Do we finally have a model with access to the training architecture and training data set, or are we still calling non-reproducible binary blobs without source form open-source?
- stackedinserter 2y agoIt sounds like if they owe you the training architecture and training data set.
- chris_pie 2y agoIt absolutely doesn't. It sounds like further diluting the term "open-source" isn't great.
- cubefox 2y agoI assume when people say "open source model" they mean "open weights model". The "open source" term doesn't really make sense here, since machine learning models are not compilations of source code. (Though DeepSeek has published several papers with details on their training process. It's more than just open weights.)
- afro88 2y agoThe size reduction while keeping the model coherent is incredible. But I'm skeptical of how much effectiveness was retained. Flappy bird is well known and the kind of thing a non-reasoning model could het right. A better test would be something off the beaten path that R1 and o1 get right that other models don't.
- whimsicalism 2y agoyeah it is pretty unclear how lobotomized it is without benchmark. i’ve gotten full fp8 running on 8xh100, probably going to keep doing that
- hendersoon 2y agoThe size reduction is impressive but unless I missed it, they don't list any standard benchmarks for comparison so we have no way to tell how it compares to the full-size model.
- techwiz137 2y agoHow can you have a bit and a half exactly? It doesn't make sense.
- dosinga 2y agoIt's not a bit and a half. It is 1.58 or really log(3) / log(2) since it allows for three values, -1, 0 an 1
- TheTaytay 2y agoDanielhanchen, your work is continually impressive. Unsloth is great, and I’m repeatedly amazed at your ability to get up to speed on a new model within hours of its release, and often fix bugs in the default implementation. At this point, I think serious labs should give you a few hour head start just to iron out their kinks!
- danielhanchen 2y agoOh thanks a lot! Appreciate it :) We're always open to collaborating with anyone!
- bluesounddirect 2y agoHi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here . I 100% expect some downvotes from the ccp.
- bluesounddirect 2y agohttps://apnews.com/article/deepseek-china-generative-ai-internet-security-concerns-c52562f8c4760a81c4f76bc5fbdebad0 https://apnews.com/article/deepseek-china-generative-ai-inte... TA
- tivert 2y ago> Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here . And that's a really important strategic advantage China has versus America, which has such an insane fixation on pure(ish) free markets and free trade that it gives away its advantages in strategic industry after strategic industry. Some people falsely infer from the experience with the Soviet Union that freer markets always win geopolitical competition, but that's false.
- cynicalpeace 2y ago> And that's a really important strategic advantage China has versus America, which has such an insane fixation on pure(ish) free markets and free trade that it gives away its advantages in strategic industry after strategic industry. > Some people falsely infer from the experience with the Soviet Union that freer markets always win geopolitical competition, but that's false. The data we have is 500 years of free markets in the western world and the verdict is overwhelmingly: Yes, more freedom means more winning. Just invite some incompetent bureaucrat over your house to dictate how you should cook and you'll quickly agree.
- tivert 2y ago
- MyFirstSass 2y agoIs this akin to the quants already being done to various models when you download a GGUF at 4 bits for example, or is this variable layer compression something new that can also be make existing smaller models smaller so we can fit more into say 12 or 16 gb's of vram?
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- beernet 2y agoBig fan of unsloth, they have huge potential, could definitely need some experienced GTM people though, IMO. The pricing page and messages sent there are really not good.
- danielhanchen 2y agoOh thanks :) Yes agreed we do need better GTM - temporarily it's still me and my brother running Unsloth, so for now we're just prioritizing many more engineering releases :)
- mclau156 2y agoIs the new LLM benchmark to create flappy bird in pygame?
- indigodaddy 2y agoIs there any small DS or qwen model that could run on say an M4 Mac Mini Standard (16G) ?
- danesparza 2y agoJust ask it about Taiwan (not kidding). I'm not sure I can trust a model that has such a focused political agenda.
- slewis 2y agoIt would be really useful to see these evaluated across some of the same evals that the original R1 and deepseek's distills were evaluated on.
- xiphias2 2y agoHas it been tried on 128GB M4 MacBook Pro? I'm gonna try it, but I guess it will be too slow to be usable. I love the original DeepSeek model, but the distilled versions are too dumb usually. I'm excited to try my own queries on it.
- emseetech 2y agoI'm downloading it now and will report back. (I've been using the 32B and while it could always be better, I'm not unhappy with it)
- TheTaytay 2y agoHow'd it go, and which client are you using? :)
- emseetech 2y agoPretty rough. Using LM Studio, trying to load the model throws an error of "insufficient system resources." I disabled this error, set the context length to 1024 and was able to get 0.24 tokens per second. Comparatively, the 32B distill model gets about 20 tokens per second. And it became incredibly flaky, using up all available ram, and crashing the whole system a few times. While the M4 Max 128GB handles the 32B well, it seems to choke on this. Here's to hoping someone works on something in-between (or works out what the ideal settings are because nothing I fiddled with helped much).
- htk 2y agoThere's a terminal command to increase the maximum vram MacOS can use, you can try that as you're probably going over the limit and the system is resorting to treat as system ram. (I ran into this problem a couple of times using ollama).
- xiphias2 2y agoMaybe VLLM is better at inferencing MoE (also you can set the number of experts to use). In theory half of the model fits to RAM, so it should be GPU limited if memory management is smart.
- Pxtl 2y agoIs there any good quick summary of what's special about DeepSeek? I know it's OSS and incredibly efficient, but news laymen are saying it's trained purely on AI info instead of using a corpus of tagged data... which, I assume, means it's somehow extracting weights or metadata or something from other AIs. Is that it?
- rahimnathwani 2y agoIs there any good quick summary of what's special about DeepSeek? Yes, section 2.3 of the Deepseek R1 paper summarizes the training part you're asking about, in less than a page. https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSee...
- DogRunner 2y ago>For optimal performance, we recommend the sum of VRAM + RAM to be at least 80GB+. Oh nice! So I can try it in my local "low power/low cost" server at home. My homesystem does run in a ryzen 5500 + 64gb RAM + 7x RTX 3060 12gb So 64gb RAM plus 84gb VRAM I dont want to brag around, but point to solutions for us tinkerers with a small budget and high energy costs. such system can be build for around 1600 euro. The power consumption is around 520 watt. I started with a AM4 Board (b450 Chipset) and one used RTX 3060 12gb which cost around 200 Euro used if you are patient. There every additional GPU is connected with the pcie riser/extender to give the cards enough space. After a while I had replaces the pcie cards with a single pcie x4 to 6x PCIe x1 extender. It runs pretty nice. Awesome to learn and gain experience
- tucnak 2y agoHow are you arriving at those numbers? ryzen 5500 + 7x3060 + cooling ~= 1.6 kW off the wall, at 360 GB/s memory bandwidth, and considering your lane budget, most of it will be wasted in single PCIe lanes. After-market unit price of 3060's is 200 eur, so 1600 is not good-faith cost estimate. From the looks of it, your setup is neither low-power, nor low-cost. You'd be better served with a refurbished mac studio (2022) at 400GB/s bandwidth fully utilised over 96 GB memory. Yes, it will cost you 50% more (considering real cost of such system closer to 2000 eur) however it would run at a fraction of power use (10x less, more or less) I get it that hobbyists like to build PC's, but claiming that sticking seven five year out of date low-bandwidth GPU's in a box is "low power/low cost" is a silly proposition. You're advocating for e-waste
- benjiro 2y agoThe issue is that you are taking max GPU power draw, as a given. Running a LLM does not tax a GPU the same way a game does. There is a rather know Youtuber, that ran LLMs on a 4090, and the actual power draw was only 130W on the GPU. Now add that this guy has 7x3060 = 100% miner. So you know that he is running a optimized profile (underclocked). Fyi, my gaming 6800 draws 230W, but with a bit of undervolting and sacrificing 7% performance, it runs at 110W for the exact same load. And that is 100% taxed. This is just a simple example to show that a lot of PC hardware runs very much overclocked/unoptimized out of the box. Somebody getting down to 520W sounds perfectly normal, for a undervolted card that gives up maybe 10% performance, for big gains in power draw. And no, old hardware can be extreme useful in the right hands. Add to this, its the main factor that influences the speed tends to be more memory usage (the more you can fit and the interconnects), then actual processing performance for running a LLM. Being able to run a large model for 1600 sounds like a bargain to me. Also, remember, when your not querying the models, the power will be mostly the memory wakes + power regulators. Coming back to that youtuber, he was not constantly drawing that 130W, it was only with spikes when he ran prompts or did activity. Yes, running from home will be more expensive then a 10$ copilot plan but ... nobody is also looking at your data ;)
- ggm 2y agoIf I invested in a 100x machine because I needed 100 of x to run, and somebody shows how 10x can work, why have I not just become the holder of 10 10x machines, and therefore have already achieved capex to exploit this new market? I cannot understand why "openai is dead" has legs: repurpose the hardware and data and it can be multiple instances of the more efficient model.
- stevenhuang 2y agobecause of discounted cash flow/valuation models. you invest in a 100x machine expecting a revenue of X, but now you can only charge X/100 because R1 shows that AI inference can be done much more efficiently. see the price decrease of ChatGPT and addition of free O3 etc. this reduction of future cash flows, ceteris paribus, implies that the present value of these cash flows decrease. this then results in massive repricing to the downside as market participants update their forecasts. what you are missing is that to assume as you do, you must make the additional assumption that demand for additional compute is infinite. Which may very well be the case, but it is not guaranteed compared to the present realized fact that R1 means lower revenues for AI inference providers -> changes the capex justification for even more hardware -> NVDA receives less revenue.
- ggm 2y agoThanks. Always a mistake to assume the price of something is bound to your own cost of doing it: the price is bound to the other guys cheaper price!
- patleeman 2y agoIncredible work by the Unsloth brothers again. It’s really cool to see bitnet quantization implemented like this.
- Dwedit 2y agoIs this actually 1.58 bits? (Log base 2 of 3) I heard of another "1.58 bit" model that actually used 2 bits instead. "1.6 bit" is easy enough, you can pack five 3-state values into a byte by using values 0-242. Then unpacking is easy, you divide and modulo by 3 up to five times (or use a lookup table).