10 ms·
GLM-5.2 – How to Run Locally
- tsouth2 3mo ago[dead]
- xrd 3mo agoSo close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970 https://news.ycombinator.com/item?id=48629970
- uberex 3mo agoFunny I casually asked Gemini and it said 500k for unquantized with decent throughput.
- stymaar 3mo agoThis is why you shouldn't believe uncritically an answer from an LLM (neither should you do for any answer from a human either though).
- andy_ppp 3mo agoBut I did my research online and the sun cycle is every 11 years and something something global warming is a hoax every single year now.
- j45 3mo agoLLMs aren't discrete calcluators or estimators of things unless framed and guided to do so.
- uberex 3mo agoGood job I didn't use a vanilla LLM without tool use harness then.
- colinsane 3mo agoi asked gemini and it replied with "Error: 400 Your prompt was blocked by safety filters. Please revise and try again."
- digitaltrees 3mo agoI asked and it said “403 forbidden - careful peon attempts to bypass the late stage capitalism api with your monetary offerings in exchange for you daily tokens will get you perma banned right to jail”.
- matheusmoreira 3mo agoSafety from competition!
- nijave 3mo agoThat's fair for new hardware. You probably want to prompt "homelab" or "used hardware" to compare what's in this thread.
- mgambati 3mo agoWith 2 wouldn’t have good results. Ideal range for coding is at least Q8.
- kibibu 3mo agoAccording to this very article, 4-bit dynamic is essentially lossless
- Aurornis 3mo agoWatch out. Those claims are often made based on KL-divergence over some arbitrary corpus, not performance in the real world or benchmarks. I’ve found that I need to go a couple steps past whatever quantizations are good enough in the KL-divergence testing to get good performance in real tasks with long context. So when Q4 is claimed to be lossless I end up with Q5 or Q6 for actual long-context tasks.
- cheema33 3mo agoI have the RAM, but not the VRAM. What kind of speed/tps could you expect from a 3090 with 24GBs of RAM? I am somewhat tempted to pick a GPU with 24GBs of RAM.
- phamilton 3mo agoGeneration is basically just memory bandwidth math. Each token has to read all the active weights. I think that's around 40B parameters active. At a 4-bit quant that's 20GB. With 100GB/s (replace with whatever your bandwidth is) and you get 5 tokens per second.
- ekidd 3mo agoA GPU with 24GBs of RAM is mostly useful for running a very carefully squeezed Qwen3.6 27B (4-bit Unsloth quants, 8-bit K/V cache, possibly MTP, 128k context). This is a fun little model that's smart enough to do debugging, refactoring, and implementing "clean" specs that don't force it to make complicated design choices. I've seen it rip through a 9-year-old Terraform AWS config, and (without using the network) correctly identify nearly everything that would need to be upgraded or migrated for modern AWS. But if I give it some poorly conceived spec with lurking design headaches, then it goes on an endless thinking binge and ultimately fails. Speed-wise, I don't have numbers, but it feels subjectively faster than Opus in Claude Code. YMMV. Once you go above "a used 3090 at a decentish price", then I strongly recommend renting cloud GPUs or at least testing models using paid APIs. This allows testing your use case before spending piles of money.
- elliotbnvl 3mo ago$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe. NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM. You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill.
- __m 3mo agoHow fast will the hardware become outdated? Are there big improvements expected in the next 3 years?
- easygenes 3mo agoM5 Ultra will ship before end of year, likely. Though with current RAM shortage, likely max spec will be 256GB and in short supply. In late 2027 or early 2028, Nvidia will release Vera Rubin DGX Spark, likely with double or better the performance of current Blackwell, though unclear if memory capacity will go up much from current 128GB. Two to four of those will run models like this decently. In 2028 we should expect Vera Rubin RTX discrete lineup, including the replacement to the RTX PRO 6000. Likely memory spec will be minimum 128GB. Good chance of up to 200GB. Two to four of those will run NVFP4 models in this class very well.
- deleted 3mo ago[deleted]
- jiqiren 3mo agoI hope all this speculation comes true. Right now this ram crunch is ridiculous.
- hajile 3mo agoIt might be M6 Ultra and I think the real reason for stopping selling top-tier units was to avoid mid-generation price hikes and increasing demand for the more expensive next-gen systems that I assume will come with 512gb (maybe 1TB) of RAM and a massive markup to match.
- ijidak 3mo agoCrossing my fingers that this boom jumpstarts 90's like improvements in computing hardware. I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses. Most of the money and energy went to mobile for the last fifteen years. Affordable local inference might be the gravy train the server, desktop, and laptop manufacturers need to get back in gear.
- linzhangrun 3mo agoPhysical limitation of the manufacturing process may be more significant factor, starting from the TSMC 10nm ten years ago
- gruez 3mo ago>I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses. No, we're running into limits of moore's law, and it's showing in prices for new nodes, where they're getting denser but not cheaper.
- horsawlarway 3mo agoIt's true we hit limits, but I feel like a lot of it was "limits" in the sense that the tradeoff stopped being worth the cost, so we optimized in other areas. So we hit limits on clock speed in the early 2000s (ex - the 4ghz wall) but it also turned out that mobile as the driver for sales meant no one really cared much about clock speed compared to performance/watt. Clock speed mattered, but only relative to how many watts it took to get it (and above 4ghz... too many watts). But we've seen a 15x improvement over the last 20 years. Performance/Watt is WAY up. My guess is that LLMs are going to drive another "improvement cycle" in areas that we didn't care much about before. I've built about 10 personal desktop machines (1 every ~4 years) and I can honestly say that I didn't care much about memory bandwidth prior to 2021. In the same way that I didn't care much about how many watts my pentium 4 was using in 2005. But now... now I care a lot about memory bandwidth. I care about memory speeds and total system ram in a manner I really, really didn't before. So I think we're going to see a big shift to machines built on unified ram with a crazy focus on squeezing memory bandwidth and total ram capacity as far as we can. My bet is that we'll get a similar 10-15x improvement by 2040 in unified system ram designs. I fully expect to see 2tb unified ram desktops and 200gb unified ram phones be relatively common on a 20 year timeline, assuming we see similar levels of geopolitical stability (ex - world war 3 throws a wrench into things).
- bbor 3mo agoI’m kinda lost here… do y’all really have machines in your houses with hundreds of gigs of RAM?? Am I just behind the times? The page advertises the 8-bit quant as taking ~800GB, which seems like it would require at least 3 consumer motherboards fully stacked w/ 4x64GB cards each. Maybe “locally” has slowly come to imply “…on your homelab”?
- cpburns2009 3mo agoRAM wasn't expensive even a year ago. I maxed out a used Dell Precision T5610 with 128 GB DDR3 for $250 in 2021.
- numpad0 3mo agoDRAM prices at mid-2025 rates were ~$2.5/GB for DDR5, and ~$1.5/GB for DDR4. "Hundreds of gigs" of RAM used to be under $500. 128GB of cheapest RAM used to be like $200. It seemed to go over heads for a lot of people that you could get hypothetical future machines on CS/CE textbooks were attainable for that little, for some reason - there seemed to be some fixation on the idea that 16GB is all you need.
- Gracana 3mo agoYou don't have to have a server, workstation motherboards support lots of memory channels. I was lucky to buy a lot of RAM before prices skyrocketed. I knew I wanted to play with this stuff, so I spent what felt like a lot of money at the time to buy 8x96GB DDR5-6400 RDIMMs. Now the same RAM costs at least 6x more.
- woodrowbarlow 3mo ago[dead]
- nijave 3mo agoI got a 2U rackmount with 192Gi DDR4 for $1.1k USD in 2023. Around 1.5 yrs ago, server RAM could be had pretty cheap--especially slower LRDIMMs (I wanna say 512Gi DDR4 was <$500 USD). I checked a couple old ServeTheHome threads and seeing maybe around $50/32GB RDIMM although thought it was cheaper than that for a little while
- zuzululu 3mo agowonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a local LLM disconnected from the cloud.
- Iolaum 3mo agoAt full quantization GLM 5.2 may be close to GPT 5.4. But at Q2 or whatever one needs in order to run it on a pro-sumer device it will be worse. Also I m not sure where you are getting the under 2k value. I bought a Framework desktop 128GB last year and my setup was around 2.7k. The same setup now sells for around 4.7k.
- kccqzy 3mo agoThe AMD 395 supports up to 128GB unified RAM. So still not enough even at 1-bit quant unfortunately.
- monksy 3mo ago96gb vram is the max it supports.
- cpburns2009 3mo agoThat's the max you can statically allocate in the BIOS. It's best to leave that at the minimum (500 MB I think), and let the drivers dynamically allocate. You can use up to about 120 GB on Linux.
- selfhoster11 3mo agoUnder Linux it is allegedly 110GB, but I’m not sure.
- benjiro29 3mo ago"GLM 5.2 is just shy of GPT 5.4"... If your running the full model. As in have 750 (FP8) to 1.5TB(FP16) of memory available. Do not mix the benchmark results of GLM 5.2 FP16/FP8 with FP4 or FP2. * FP4 will mean a accuracy loss of about 3%. Not noticeable but more chance for mistakes. * FP2 ... what is what most people are able to run at home, for a "reasonable" price. Your looking at over 17% loss in accuracy. At that point, your running at less then claude-sonnet-4.6, as the issues compound with accuracy losses. And reasonable priced is still in the ~ $5000 range (192GB + GPU 32GB active/kv cache system). For that price your using a Codex / Claude Pro subscription for the next 4+ years with better models (by default), let alone with a FP2 GLM 5.2 version. And your looking at < 10 fps. A MacStudio with 512GB will net you 18 a 20fps+ with FP4, but ... i mean, those used to be $10.000. Unfortunately the local hardware cost is a major issue for running large models like that. Edit: Its funny whenever the issue of cost and what you need to give up vs the subscription services, there are always people who downvote in bad faith.
- pheggs 3mo agoI feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?
- fsuts 3mo agoWhy do you think they are rushing to IPO!!
- fny 3mo agoThe RAM requirements are still pretty painful.
- yieldcrv 3mo agoequilibrium in one or two more years on the consumer/prosumer side think Apple M6 or M7 with a currently unforeseen denser memory style, 256gb RAM a couple inference or cache improvements on the algorithmic side, using less ram for context windows and doubling token speed again denser open source models, packing more experts for smaller active layers it'll still be expensive but like $8,000 - $13,000 instead of $450,000 worth of B200s
- stingraycharles 3mo agoFairly certain that model sizes and computational requirements will grow as the price for LLM compute drops.
- 3stacks 3mo agoMaybe there's a conversation to be had about how much is enough... Unless something beyond my imagination happened, I would be happy enough with Opus 4.5 levels of productivity
- stingraycharles 3mo agoThis really sounds like “640kb should be enough”. I’m sorry, but I just can’t imagine us running smaller models than we are using right now in 5-10 years from now.
- andai 3mo agoHow is this model half the size of DeepSeek V4 Pro? Is it because DeepSeek did more aggressive cost cutting on the attention mechanism?
- skiing_crawling 3mo ago"it can fit" on 256GB of RAM, but it will be heavily quantized and still run very slowly. The headline number is not token generation, its prompt processing. So if you get 10 tok/s and an API gives you 20-30 tok/s, it doesn't seem that bad on its face, but a mac studio or any other machine that's not loading all of it into GPU will do PP 20-50X slower than a purely GPU based setup, which is what actually makes this unusable without $50k in GPUs. On top of that, you will still be heavily quantized.
- gerdesj 3mo agoA nvidia spark thingie has 128GB unified RAM. They also have a dual port version of one of these things: https://www.nvidia.com/content/dam/en-zz/Solutions/networking/ethernet-adapters/connectx-7-datasheet-Final.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/networkin.... ie 2 x 100GB/s ports, they may even be 2 x 200GB/s. Once I've got my paws on one, I'll know more. You can cluster these beasts too. Two and three (with two IP subnets) is fairly obvious. Four or more might need a switch depending on how much network latency affects things. Apple seem to have forgotten about M series with gobs of RAM. I can't get the Apple shop to show more than 96GB of unified RAM and that costs a kidney.
- Computer0 3mo ago128 gb of much slower ram than Apple.
- dannyw 3mo agoDGX Spark is ~273GB/s. That’s about M5 Pro territory, and twice as fast as the M5. You’d have to go to the M5 Max, or M3 Ultra, to get higher memory bandwidth than the Spark.
- hajile 3mo agoIf you are trying to get more than 64gb of RAM or doing tons of inferencing, you're getting a Max or Ultra anyway.
- nullc 3mo agoJust running cpu only w/ Q6 on 9684X I get about 1tok/s ... also still get about 1tok/s/stream when running 16 in parallel.
- nullc 3mo ago6.5t/s if I add one RTX R6000 gpu (where is uses about 25GB).
- Wowfunhappy 3mo ago> The full model requires 1.51TB of disk space ...a bit of an odd question: how well do LLMs losslessly compress, as in for cold storage? I definitely don't have the hardware to run this model at any kind of reasonable speed (and I don't want to use a super aggressive quantization that would kill performance). Even so, I think it would be cool to retain an offline copy, in case... I don't really know, a solar flare destroys the internet some day, or maybe a zombie apocalypse. It would just be cool to have. But 1.5 TB is a bit too much! If it could be compressed down into something semi kind of reasonable, that would be fun!
- SirMadam 3mo agoSOTA LLM specific compression achieves around ~54%! https://arxiv.org/abs/2505.06252v3 https://arxiv.org/abs/2505.06252v3
- redox99 3mo agoProbably not at all, considering weights are randomly initialized.
- gcr 3mo agoThere are two forms of compression relevant to LLMs: 1. Reduce the number of parameters 2. Reduce the resolution of each parameter (quantization) For 1, changing the architecture is typically only possible by the labs producing the models, which is why each OSS model release tends to feature a small number of carefully chosen model sizes (for example, Gemma4 comes in e2B, e4B, 12B, 26Ba4B, and 31B sizes). Generally, models with higher parameter counts have more world knowledge. For coding models, this shows up as a stronger command of uncommon libraries/languages. Very small models (<20B) also lack “smarts.” Reducing the resolution of each parameter is easier which is why lots of practitioners have their own quantizations, but this makes it harder for a model to “think” fluently. Interacting with heavily quantized models feels like interacting with someone who didn’t get any sleep the night before. Models that have higher-fidelity quantization take more RAM and have higher “smarts,” but don’t necessarily have more world knowledge. Models with aggressive quantization tend to be more likely to make rookie mistakes, emit malformed tool calls, get stuck in loops, or even exhibit signs of “neuroticism” / “distress” in their thinking tokens. Parameter counts = world knowledge, quantization = “smarts.” This is a soft rule of thumb, the difference isn’t very strong.
- CGamesPlay 3mo agoCan somebody help me understand the Quantization Analysis? It says "dynamic 4-bit UD-Q4_K_XL and dynamic 5-bit UD-Q5_K_XL are generally lossless" while showing a top-1% token agreement on the chart of 97.5%. Not what I would consider "generally lossless". Is this implying that some post-processing is going to account for the 2.5% loss? Beam search?
- dannyw 3mo agoGenerally 97.5% token agreement is very positive. Like the article explains, the difference isn’t the model thinking the capital of France isn’t Paris, but rather maybe saying “The capital of France is Paris” instead of “Paris is the capital of France”.
- hxii 3mo agoAny time I see one of these posts about models of this size a quote comes to mind – "Your Scientists Were So Preoccupied With Whether Or Not They Could, They Didn’t Stop To Think If They Should". Only a select few have the hardware required to run this to begin with, and even then the forecasted performance makes me wonder if it’s worth it at all.
- segmondy 3mo agoCompletely worth it. At 6tk a second. If I can get 2 hrs of token generation. That's 2hrs * 3600secs * 6tk = 43200 tokens, at about 10tk to a line of code, that's about 4320 lines. Let's even trim it more and slice it by half. That's 2160 lines of code a day. Most professional programmers can't deliver that much consistently in a day. The key to a model this large is (1) Use it to plan, generate lots of plan and farm out to a smaller model. Then for very specific and complicated portions precisely prompt for what you need.
- uberex 3mo agoThats not a complete reasoning. Even frontiers need to revisit and fix things. Add 10 loops to that and it is 20 hours. Still great compared to a 2023 human, but why am I not just paying pocket money for Claude Pro instead?
- segmondy 3mo agoYou're talking about agentic workflow. Agentic is cruise controls. Race car drivers shift manually for more precision and to go faster. If the only way you know how to code with AI is agentic, then you are putting yourself on a crutch.
- uberex 3mo agoYou are saying you can one shot without loops on something like GLM-5.2?
- ramgine 3mo agoI have up to 1tb of ddr4 in my server but it only has a 12gb vram 3060. Would getting a 24gb vram make this a viable system or am I throwing money away?
- dofm 3mo agoCan't run this myself. But I do like Unsloth Studio, quite a lot. It's nicely designed.
- snootypoot 3mo agoif sam altman didnt exist i could afford to run this
- numlock86 3mo agoif sam altman didn't exists this model would most likely not exist as well
- VaporJournalAPP 3mo ago[flagged]
- jonathanhefner 3mo ago> Runing GLM-5.2 on local hardware Do the runes make it smarter or just run faster (or both)?
- nicman23 3mo agodepends on the color
- segmondy 3mo agoI run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at home. It's great for planning, one shot prompting once you have a plan or all the context you need. This entire hardware cost $2400 when it was built. If you're willing to be resourceful, you can find ways to run these models at home. I often get the silly question of why, and suggestions about how much I can save using cloud API, but the Fable drama has opened up eyes on why it's good for us to be independent. Thanks team unsloth, Q4_K_XL is solid, if you are going to grab a quant, make sure to get the K_XL variant if it can fit.
- ikari_pl 3mo agoI can work out max 90GB to the agents. Advise. :)
- redox99 3mo agoThat's crazy good for $2400.
- edg5000 3mo agoVery cool. So it's not just about GPU VRAM which I incorrectly thought. I though you'd need 512 GB GPU VRAM. I don't think it cost only 2400; 512GB ram would be more expensive though back in the day. But not mortgage-grade 200.000 which I estimated myself (which assumed running in 100% VRAM; overkill for a single user probably).
- segmondy 3mo agoyou can use system ram with a system like llama.cpp which offloads to system ram. token generation is a function of system bandwidth, the faster the bandwidth the better. so I'm on 8 channel 2400mhz. if I had a 12 ddr channel, I would get 1.5x the speed at 2400mhz. of course ddr5 is much faster, so a 12 ddr at 4800mhz will provide 3x the speed for token generation or roughly 18tk/sec. prompt processing is all about compute, so the more cpu cores you have the faster it can do PP.
- CHUNK_CHUNK 3mo ago[flagged]
- cws_ai_buddy 3mo ago[flagged]
- boringspinner 3mo ago[dead]
- Frannky 3mo agoThere is a push from multiple directions at the same time: - new AI desktops with GB10s. They are relatively cheap and you can cluster them and load 1TB of VRAM - Nvidia, amd, intel, Cerebras etc pushing new hardware - oss models getting crazy good, like glm 5.2 - flash models getting very good like deepseek V4 flash - quantizations - harnesses being able to use different models (big for difficult stuff, small for grunt work) So hopefully soon for the ones who want to break free from APIs, we will be able to host at home a cluster of AI desktops at a reasonable price with Opus-level capabilities, can't wait!!
- matheusmoreira 3mo agoHope you're right! Can't wait!
- khafra 3mo agoI feel like "relatively" is doing a lot of work, there: at about $4k per GB10, that's $36k for a 1TB cluster. Cheap compared to equivalent H200's, but out of reach for home labs that aren't funded with OpenAI or Anthropic RSUs.
- snarfy 3mo agoWhen the AI bubble pops those hardware prices will pop too.
- Tepix 3mo agoMy hope is on Intel Crescent Island with 480GB. I don't need 8x H200 performance (and cost), but I would like to run GLM 5.2 Q8.
- drudolph914 3mo agoGLM 5.2 is the first time I'm actually excited about AI! I'm not the most bullish on AI code for several few reasons, but the biggest reason is the ownership model. We all know we're near the tail end of the "subsidized pricing" window for AI, and I've been hoping for so long to get an open weight model that is _close enough_ to the SOTA before this window closes - and we actually got it! I'm excited to be able to in the near future run GLM locally, and use these things like a tool instead of living in this for-rent model for the rest of my life. I'm excited to actually enjoy programming again
- numlock86 3mo agoIs this really worth it, though? Throughout the years my experience with quantized models has been that they feel like a lobotomized version of the original. Doesn't matter if it's an LLM, dedicated diffusion model or some other dedicated task. Sure, they get the job done. But a lot worse. The only ones that can somewhat hold up are the ones provided by the vendor directly. Gemma4 comes to mind. However I suspect they have some secret sauce other than just "let's quantize this" since they have the original model and its data at hand. There should be more native 4bit, 1.25bit and likewise models. Those actually work great while making them smaller in comparison. But I guess there is some reason for them being pretty niche.
- edg5000 3mo agoOne advantage about local LLM: You could serialize the context yourself, without being constrained by APIs. And let's not forget, the Big 2 encrypt their thinking. If you use custom clients, which is a very grey area alreay, being able to produce the context string raw is a big bonus. Takes away a lot of annoying constraints and needless mystique/obfuscation. But I don't know how usable GLM 5.2 is vs the Big 2.
- c7b 3mo agoCan someone explain the math to me? Why is 1-bit only ten percent less memory than 2-bit?
- incognito124 3mo agoKeyword dynamic, the parameters are quantized on a case by case basis
- idonotknowwhy 3mo ago2 reasons. First, it's not really "1 bit", actually much closer to 2-bit. IQ1_M is actually 1.75bit and IQ2_XXS is 2.06bit This is from the ./llama-quantize --help with most of the quant types and their size in bpw: https://pastebin.com/bCUqGfeE https://pastebin.com/bCUqGfeE And to elaborate on the "dynamic" aspect inconito said in the other comment, if you click on one of the .gguf files in huggingface: https://huggingface.co/unsloth/GLM-5.2-GGUF/blob/main/UD-IQ1_M/GLM-5.2-UD-IQ1_M-00002-of-00006.gguf https://huggingface.co/unsloth/GLM-5.2-GGUF/blob/main/UD-IQ1... There are a lot of Q5_K, Q6_K, etc tensors. Only the routed experts (ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight) are heavily quantized, and it looks like the down_proj is actually iq3_xxs for this model.
- jzer0cool 3mo ago1 bit requirement (1-bit 223 GB wowza). What you all recommend with 24-48 vram, or is this approach much out dated now.
- suyash 3mo agoWe really need a quantized version for regular laptop
- Havoc 3mo agoI bet OpenAI and Anthropic hate the timing of glm 5.2. Kinda shows they have a headstart rather than a magic moat
- achrono 3mo agoNope, GLM 5.2 is only the latest and greatest in a long line of open-weights models. There are even fully open source models that are comparable to o1-mini (OLMo), or almost-fully-open ones that are comparable to o3 (Nemotron). I'm super grateful to the open labs (who, importantly, do not have the word 'Open' in their name), all the more so to the likes of Ai2. There is no magic moat indeed. It is math, engineering and of course copious amounts of data (and the political maneuvering required to secure it, e.g. how most everyone has trained on Anna's Archive by this point).
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- ofcyes 3mo ago[dead]
- walrus01 3mo agoI really don't think anyone is going to have a good time trying to run it on anything with 256GB of RAM no matter what the post says. 512 is the much more realistic minimum. I'm fortunate enough to have two 512GB RAM dual xeon workstations in my home office that I bought cheap before the price rise to mess around with things...
- zkmon 3mo agoI have high respect for unsloth's work, helping millions to get started with local AI, but this post appears kind of download bait. Offloading too many layers to CPU is not going to work at all. I have tried this many times and had to rm -rf on those heavy hf cache folders. Also I doubt 1-bit or 2-bit quants of GLM 5.2, running mostly outside of VRAM can beat Q8_0 of Qwen3.6-27B fully loaded in VRAM - on usefulness.
- iaw 3mo agoI run 3bit GLM5.2 and full precision Qwen3.6-27B. GLM is much much closer to frontier models in it's breadth and ability to plan. If you just need to implement Python code from an existing plan Qwen is your choice but it has problem succeeding with more complex tasks that GLM5.2 does not. As I type this my local GLM5.2 is troubleshooting bugs that Qwen would not be able to handle.
- zkmon 3mo agoNot sure how much of your GLM is offloaded to CPU. I was contending the suggestion of using system RAM + VRAM.
- maxignol 3mo agoLucky me, I never go out without my 256gb unified ram mac x)
- lisa_luoyf 3mo ago[flagged]
- cjbprime 3mo agoI've got access to a 192GB RAM Mac Studio, which is below the stated minimum RAM. Can swapping off fast disk be used to make it work out, especially since it's MoE?
- walrus01 3mo agoSeems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.
- smallerize 3mo agoThe LLM tools are smart enough to keep the weights on the disk and read-write stuff in RAM.
- cjbprime 3mo agoBoth sentences are likely wrong. It's a written lifespan (technically an "erased" one), not a read lifespan. The weights are only needed from disk read-only. And Mac NVMe interfaces are surprisingly fast. Edit: Oh, I think you maybe thought I meant swapping working RAM off disk? I didn't. I meant swapping weights off disk into RAM on-demand.
- walrus01 3mo agoYes, I meant that 192GB of RAM even with the worst quantization would result in such a large model going deep into disk swap when it entirely runs out of RAM. At least 100GB worth, if MacOS will even allow that without freezing or crashing or OOM killing the process.
- cjbprime 3mo agoYou still have a core misunderstanding. Only one layer of weights is required in memory at a time. A forward pass can be over-simplified as a matrix multiplication of each layer, one at a time. There is no swapping of working RAM. We're just talking about loading the weights read-only data into RAM on-demand for each layer. It is only as slow as your storage interface.
- bilekas 3mo ago> this can directly fit on a 256GB unified memory Mac And yet Apple won't sell them to you anymore. And I'm not too confident it will be even possible to hand then 10k to get one again.
- chakintosh 3mo agoBreaking even in 2069
- jessinra98 3mo agoAnyone here tried both Qwen and GLM families on the same setup and found a clear winner for one task vs the other?
- stackedinserter 3mo agoTLDR: realistically, you can't.
- antirez 3mo agoDwarfStar work in progress numbers: I see 14 tokens/sec generation, that slopes to 10 t/s with longer 10k or more context size. Consider that the indexed attention requires evaluating 2048 selected rows, 2x DeepSeek and with less compression, so the performances with larger contexts here to south faster. Prefill can be 180 t/s on small contexts to 150 t/s and less with larger contexts. I used DeepSeek v4 PRO in this conditions, it is usable but it is far from the 35 t/s 400 t/s prefill you get with DeepSeek v4 Flash 2 bit on a MacBook m5 max. But likely my implementation is yet not optimized enough, so a bit more performance can be obtained. I'm using 4 bit quants. The model is also definitely less sparse than DeepSeek v4, so it activates a bigger percentage of parameters. If it works decently at 2-bit, that would be a win even for machines where 4-bit fits, since this would mean 2x memory (equivalent) bandwidth basically for the routed experts. Local inference needs really hard a 1.2 / 1.5 T/s memory bandwidth system with 512GB and 2/3 times the GPU compute of Mac Studio M3 Ultra, at an affordable 10/15k price point. A variant with 1TB memory would also be welcomed at 20k price point.
- reasonabl_human 3mo agoThank you for your work on DwarfStar! It is truly helping democratize access to frontier tech.
- rbrener 3mo agoIt is quite cool indeed..!
- zozbot234 3mo ago10k context is not a whole lot, this model theoretically supports up to 1M. But the KV cache storage takes up a whole lot more memory capacity at full context than DeepSeek V4 Pro, let alone Flash. (About ~96GB according to readily available KV cache calculators, might be more in practice. For comparison DeepSeek Flash is ~10GB and Pro is at least in that ballpark.) So I'm not sure that this model is a good deal for memory-constrained machines unless you're specifically interested in very short contexts only. This could still be worth it if it came with a game-changing increase in smarts but that seems a bit unlikely so far. It will be interesting to see how this model does under a SSD streaming scenario, the lower sparsity should ideally be favorable. > Local inference needs really hard a 1.2 / 1.5 T/s memory bandwidth system with 512GB and 2/3 times the GPU compute of Mac Studio M3 Ultra, at an affordable 10/15k price point. A variant with 1TB memory would also be welcomed at 20k price point. Are these realistic specs at present? Not that clear to me, 1.5 T/s seems really high.
- draginol 3mo agoThe most interesting part of this to me is not the benchmark table, but the packaging. A model like GLM-5.2 being available as GGUF, usable through llama.cpp/Ollama/vLLM/SGLang/LM Studio, and wrapped for local agent workflows changes the category. It stops being an impressive open model exists and starts becoming this is something a small team can actually put into its development stack. For instance, company buys an RX6000 setup for say $15k total. They could use this for handling data heavy sifting that would otherwise be a lot of Claude tokens. It doesn't need to be as good as frontier-best. Just good enough. I could see a business of people packaging this and handing it to companies who want Help Desk bots without any extra setup.
- giancarlostoro 3mo ago> For instance, company buys an RX6000 setup for say $15k total. They could use this for handling data heavy sifting that would otherwise be a lot of Claude tokens. Considering they might be spending thousands per month on API costs already, dropping 15K to save on one process might not be bad. On the other hand, also an opportunity to sell GLM 5.2 inference at near cost to other companies for less than whatever Claude costs. In theory it costs anywhere from $0.51 to less than $2 an hour to run it and use it 24/7 that's still wildly cheaper than calling Opus which doesn't bill per hour, but per million tokens, drastically higher. Hell, you could probably bill at $5 per GPU hour and still be cheaper. Whether you're looking to self-host or sell hosting for it, it looks way cheaper regardless. I think most decent open models will continue to fit in at least 32GB of VRAM so a 6000 Pro GPU is more than enough. alternatively, even on a 5090 you can get a reasonable amount of inference for way less than paying for Opus, Qwen would be your friend there though.
- storus 3mo agoSo a minimum of 3x RTX Pro 6000 to run 1-bit at ~76% accuracy or MacStudio 512GB RAM to run 4-bit at ~97% accuracy.
- iaw 3mo agoNo. Unsloth has CPU offloading. It'll be slow but it'll work even with SSD offloading.
- c121618 3mo ago[flagged]
- jingpostmedia 3mo ago[flagged]
- viduus 3mo ago[flagged]
- peptech 3mo ago[dead]