5 ms·

The value prop really depends on what you're doing. If you're just vibe coding with giant frontier models, yes, the value will be worse. Especially now, where

The value prop really depends on what you're doing. If you're just vibe coding with giant frontier models, yes, the value will be worse. Especially now, where GPU prices have spiked another 20% last month. For some tasks where owning the setup and full kv cache matters, the payoff calculation is ridiculously in favor of running your own deployment. For instance for some batch classifications jobs where the prefix cache hit rate will be >95%. The calculus also changes if you just use AI as a light tool while coding and don't need the giant models; qwen3 27B runs at 80TPS on a 5090 properly deployed.

drewnick 1mo ago
The KV cache is a great example. When you own hardware, vllm etc can keep your KV cache warm indefinitely for bursty large token loads vs clouds will purge quite often. One of my tasks has a very large but very static system prompt and instructions, think something like 128k tokens. On our own hardware, we keep that cached, sprinkle in the 8k of goodness needed for output, and DSV4 Flash 0731 can generate our output at ~50 tps on $10k worth of hardware. I've been benchmarking vs publics clouds (which are tbf insanely cheap!) and it's basically a break even in 24-36 months if nothing changes, which it could for better or worse. Combined with the security and stability of internal hardware, to me it's in "no brainer" territory for this workload, even if models don't improve. Our only "risk" is better cheaper hardware or cloud costs, which given the trajectory is murky at best short of a bust.