6 ms·
what's the hardware needed to run the trillion parameter model?
by enigma101 10mo ago
what's the hardware needed to run the trillion parameter model?
- trvz 10mo agoTo start with, an Epyc server or Mac Studio with 512GB RAM.
- criddell 10mo agoI looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting... I'm guessing an Epyc machine is even less.
- graeme 10mo agoHow does the mac studio load the trillion parameter model?
- deleted 10mo ago[deleted]
- petu 10mo agoBy using ~3 bit quantized model with llama.cpp, Unsloth makes good quants: https://docs.unsloth.ai/models/tutorials-how-to-fine-tune-and-run-llms/kimi-k2-how-to-run-locally#model-uploads https://docs.unsloth.ai/models/tutorials-how-to-fine-tune-an... Note that llama.cpp doesn't try to be production-grade engine, more focused on local usage.
- CamperBob2 10mo agoIt's an MoE model, so it might not be that bad. The deployment guide at https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main/docs/deploy_guidance.md https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main... suggests that the full, unquantized model can be run at ~46 tps on a dual-CPU machine with 8× NVIDIA L20 boards. Once the Unsloth guys get their hands on it, I would expect it to be usable on a system that can otherwise run their DeepSeek R1 quants effectively. You could keep an eye on https://old.reddit.com/r/LocalLlama https://old.reddit.com/r/LocalLlama for user reports.
- wg0 10mo agoAre such machines available in the A class clouds such as Azure/AWS/Google?