6 ms·
> In practice, it'll be incredible slow and you'll quickly regret spending that much money on it instead of just using paid APIs until proper hardware gets chea
by biddit 9mo ago
> In practice, it'll be incredible slow and you'll quickly regret spending that much money on it instead of just using paid APIs until proper hardware gets cheaper / models get smaller.
Yes, as someone who spent several thousand $ on a multi-GPU setup, the only reason to run local codegen inference right now is privacy or deep integration with the model itself.
It’s decidedly more cost efficient to use frontier model APIs. Frontier models trained to work with their tightly-coupled harnesses are worlds ahead of quantized models with generic harnesses.
- theLiminator 9mo agoYeah, I think without a setup that costs 10k+ you can't even get remotely close in performance to something like claude code with opus 4.5.
- cmrdporcupine 9mo ago10k wouldn't even get you 1/4 of the way there. You couldn't even run this or DeepSeek 3.2 etc for that. Esp with RAM prices now spiking.
- coder543 9mo ago$10k gets you a Mac Studio with 512GB of RAM, which definitely can run GLM-4.7 with normal, production-grade levels of quantization (in contrast to the extreme quantization that some people talk about). The point in this thread is that it would likely be too slow due to prompt processing. (M5 Ultra might fix this with the GPU's new neural accelerators.)
- benjiro 9mo ago> $10k gets you a Mac Studio with 512GB of RAM Because Apple has not adjusted their pricing yet for the new ram pricing reality. The moment they do, its not going to be a $10k system anymore but in the $15k+... The amount of wafers going to AI is insane and will influence not just memory prices. Do not forget, the only reason why Apple is currently immunity to this, is because they tend to make long term contracts but the moment those expire ... then will push the costs down consumers.
- tonyhart7 9mo agogenerous of you to predict apple only make it 50% expensive
- deleted 9mo ago[deleted]
- embedding-shape 9mo ago> $10k gets you a Mac Studio with 512GB of RAM, which definitely can run GLM-4.7 with normal, production-grade levels of quantization (in contrast to the extreme quantization that some people talk about). Please do give that a try and report back the prefill and decode speed. Unfortunately, I think again that what I wrote earlier will apply: > In practice, it'll be incredible slow and you'll quickly regret spending that much money on it I'd rather place that 10K on a RTX Pro 6000 if I was choosing between them.
- coder543 9mo ago> I'd rather place that 10K on a RTX Pro 6000 if I was choosing between them. One RTX Pro 6000 is not going to be able to run GLM-4.7, so it's not really a choice if that is the goal.
- bigyabai 9mo agoYou definitely could, the RTX Pro 6000 has 96 (!!!) gigs of memory. You could load 2 experts at once at an MXFP4 quant, or one expert at FP8.
- coder543 9mo agoNo… that’s not how this works. 96GB sounds impressive on paper, but this model is far, far larger than that. If you are running a REAP model (eliminating experts), then you are not running GLM-4.7 at that point — you’re running some other model which has poorly defined characteristics. If you are running GLM-4.7, you have to have all of the experts accessible. You don’t get to pick and choose. If you have enough system RAM, you can offload some layers (not experts) to the GPU and keep the rest in system RAM, but the performance is asymptotically close to CPU-only. If you offload more than a handful of layers, then the GPU is mostly sitting around waiting for work. At which point, are you really running it “on” the RTX Pro 6000? If you want to use RTX Pro 6000s to run GLM-4.7, then you really need 3 or 4 of them, which is a lot more than $10k. And I don’t consider running a 1-bit superquant to be a valid thing here either. Much better off running a smaller model at that point. Quantization is often better than a smaller model, but only up to a point which that is beyond.