5 ms·
DGX Spark is a biiiiit costly but neat to hear! Do you know what kinda throughput you’re getting on that kinda setup? (I have a secondary problem of being “lo
by rtpg 4d ago
DGX Spark is a biiiiit costly but neat to hear!
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
- anon373839 4d agoIt is costly, especially right now. I don’t think you can make a case for it on cost savings! The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth. I think these numbers may improve because the model is pretty new and optimizations aren’t done.
- int_19h 4d agoI don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models. The only reason to run locally is privacy.
- anon373839 3d agoPrivacy is a great reason, but independence is another. It’s very nice knowing that you’re going to get the same reliable product every time you call the model. Nothing is going to change unless you decide to change it.