9 ms·

although i initially thought it didn't make sense financially to run this kind of model locally, i did run the numbers and for heavy users this could justify bu

by guybedo 21d ago

although i initially thought it didn't make sense financially to run this kind of model locally, i did run the numbers and for heavy users this could justify buying $10k worth of hardware with a ROI over a few months, less than a year. I was looking at my token usage, mostly from subsidized codex/grok subscriptions and i'm a somewhat heavy user. The thing is i would actually use even more tokens if it wasn't for the weekly quotas. In the end, with a $10k investment and running this kind of model, estimating a 2x increase in token usage because i wouldn't have weekly quotas and comparing to glm api prices, this thing could pay for itself in less than a year. Obviously i'm paying subscription price right now, so the math doesn't work. Although using local ai removes all weekly quotas. Keep a subscription to have access to frontier models for planning work, and local hardware + glm-5.3 flash for implementation, e2e testing, qa work 24/7. It's not that crazy of an idea and the numbers aren't that bad.

minraws 21d ago
You aren't going to get nearly as much token usage locally from DGX Sparks or even M5 Ultra (though it might be close, unsure would need to get my mittens on it to clarify). You will get around 2-4 concurrent streams of aggregate tokens at best for such a model and around 0.5B output tokens per month assuming you use loops and run it when you are sleeping. That's 500 (per mill) * 0.5$ = 250$ only at most. Then there is maintanence and efficiency costs due to electricity usage and such, any down time, etc. You will be lucky if you can squeeze more than 200$ of value out of it in a month. I don't think people should buy local hardware for money reasons, by the time you will pay off a 10K USD machine, 2-3K USD machine will catch up and beat it by a significant margin. Unless your expectation is that we will be in hardware winter for the next 10+ years. At 200$ per month it will take around 200 * 50 = 10k, that is, 50 months, so around 4-5 years. Again assuming you are making the most of your hardware somehow, very hard to do in practice. I don't recommend people to use compute as investment or payoff thing, but if you have the money to burn and can afford it why not, maybe with some software optimizations it will be cheaper but then again Z.ai is currently offering 50% discount and providers will offer cheaper rates for sure. But either way you will never be able to burn more than 200$ worth of token on a cheap hardware device, because inference becomes more profitable the more you scale it up, you have separate prefill and decode engines/systems, and a lot of nuance, but assume for every 10x increase in infra you increase margins by 5-10%. So from 10K to 100K to 1M to 10M to 100M.. I don't think this curve continues beyond 100M but I have no idea about that scale unless some AI lab is interested in hiring me lol. So a 100M infra will have ~30% better margins than you at 10K, then there is software optimizations but that's cheap enough, though some of it is only viable at scale. Either way assume 10K is the price of privacy if you really want to buy it. Don't worry about making the most out of the usage, you will always be in a net loss but I would assume for you 10K doesn't matter.
- guybedo 21d ago
  yeah i mostly agree, especially compared to subsidized subscription cost. But for a heavy user who has enough work to be done so that the box runs almost 24/7 at say 50tok/sec, the math gets interesting against API prices. And it can be interesting compared to subscription in the sense that you don't have the quota anymore. That means there's probably a lot of things you're not doing because of the quotas that you could do now. It depends heavily on the tok/sec obviously and the very best solution financially remains subscriptions. But the idea remains entertaining and not that disconnected from reality
  - minraws 21d ago
    At 50tps for single stream you are going to get 50 * 60 * 60 * 24 * 30 = 130M out tokens of GLM 5.3 Flash... That's less than what 40$ at current API rates... So if you are willing to pay 200$ per month you will get much better limits paying API rates. You can't run large Kimi K3 models on 10K worth of hardware either way, you need to spend like 50K USD minimum. Just pay for the API rates or get a low cost provider that uses higher batching, you can get shittier tps but much better prices, probably go as low as 20$ for as much usage as you can ever get from a 10K USD machine from GLM 5.3 Flash... The issue is nothing expensive runs on these devices and cheap stuff isn't worth running locally, eletricity costs ~12cents/kwh in us iirc, so at 330W M5 Ultra will burn around 8 * 0.12 = ~1$ per day extra in electricity so the electricity is going to cost you the same as the API rates(30$ per month). I truly don't think you are accounting for the costs here properly. But again if money truly doesn't matter it's much better for privacy and better than paying one of the shady AI labs who are doing god knows what with your data.
    - colingauvin 21d ago
      Your point isn't lost on me, but a few other considerations: 1) Rates are theoretically discounted for GLM 5.3 Flash right now, by 50%. 2) Hardware costs have continued ascending with no sign of letting off, so it's unlikely that a DGX Spark depreciates to zero in one year. 3) Compare performance in terms of difficult tasks/$ over the last 6 months, 3 months, etc. Open weights are a ratchet. In terms of intelligence per $, a Spark is never going to be a worse deal tomorrow than it is today, at least until the entire platform is replaced or obsoleted. 71 days ago the best model you could run on two Sparks was an aggressive Q3 quant of Qwen 3.5 397B (AA 34). 70 days ago it was a mixed-quant of GLM 5.2 (AA 53). 30 days ago it was full fat DeepSeek 4 Flash (AA 53). Today it's GLM 5.3 Flash (AA57) and/or Qwen 3.8 Next (Unknown). Sometime this week it will likely become mixed-quant GLM 5.3 (AA 60). So in < 80 days we have almost doubled the benchmark score. And that curve is still accelerating. If you view it as "cost per token of model vs API" then yes it's a bad deal. If you view it as "cost of task per $" then it has almost doubled in value in less than 3 months. All of this, imo, API and hardware, is still massively underpriced.

by guybedo 21d ago

minraws 21d ago

You aren't going to get nearly as much token usage locally from DGX Sparks or even M5 Ultra (though it might be close, unsure would need to get my mittens on it to clarify). You will get around 2-4 concurrent streams of aggregate tokens at best for such a model and around 0.5B output tokens per month assuming you use loops and run it when you are sleeping. That's 500 (per mill) * 0.5$ = 250$ only at most. Then there is maintanence and efficiency costs due to electricity usage and such, any down time, etc. You will be lucky if you can squeeze more than 200$ of value out of it in a month. I don't think people should buy local hardware for money reasons, by the time you will pay off a 10K USD machine, 2-3K USD machine will catch up and beat it by a significant margin. Unless your expectation is that we will be in hardware winter for the next 10+ years. At 200$ per month it will take around 200 * 50 = 10k, that is, 50 months, so around 4-5 years. Again assuming you are making the most of your hardware somehow, very hard to do in practice. I don't recommend people to use compute as investment or payoff thing, but if you have the money to burn and can afford it why not, maybe with some software optimizations it will be cheaper but then again Z.ai is currently offering 50% discount and providers will offer cheaper rates for sure. But either way you will never be able to burn more than 200$ worth of token on a cheap hardware device, because inference becomes more profitable the more you scale it up, you have separate prefill and decode engines/systems, and a lot of nuance, but assume for every 10x increase in infra you increase margins by 5-10%. So from 10K to 100K to 1M to 10M to 100M.. I don't think this curve continues beyond 100M but I have no idea about that scale unless some AI lab is interested in hiring me lol. So a 100M infra will have ~30% better margins than you at 10K, then there is software optimizations but that's cheap enough, though some of it is only viable at scale. Either way assume 10K is the price of privacy if you really want to buy it. Don't worry about making the most out of the usage, you will always be in a net loss but I would assume for you 10K doesn't matter.

guybedo 21d ago
yeah i mostly agree, especially compared to subsidized subscription cost. But for a heavy user who has enough work to be done so that the box runs almost 24/7 at say 50tok/sec, the math gets interesting against API prices. And it can be interesting compared to subscription in the sense that you don't have the quota anymore. That means there's probably a lot of things you're not doing because of the quotas that you could do now. It depends heavily on the tok/sec obviously and the very best solution financially remains subscriptions. But the idea remains entertaining and not that disconnected from reality
- minraws 21d ago
  At 50tps for single stream you are going to get 50 * 60 * 60 * 24 * 30 = 130M out tokens of GLM 5.3 Flash... That's less than what 40$ at current API rates... So if you are willing to pay 200$ per month you will get much better limits paying API rates. You can't run large Kimi K3 models on 10K worth of hardware either way, you need to spend like 50K USD minimum. Just pay for the API rates or get a low cost provider that uses higher batching, you can get shittier tps but much better prices, probably go as low as 20$ for as much usage as you can ever get from a 10K USD machine from GLM 5.3 Flash... The issue is nothing expensive runs on these devices and cheap stuff isn't worth running locally, eletricity costs ~12cents/kwh in us iirc, so at 330W M5 Ultra will burn around 8 * 0.12 = ~1$ per day extra in electricity so the electricity is going to cost you the same as the API rates(30$ per month). I truly don't think you are accounting for the costs here properly. But again if money truly doesn't matter it's much better for privacy and better than paying one of the shady AI labs who are doing god knows what with your data.
  - colingauvin 21d ago
    Your point isn't lost on me, but a few other considerations: 1) Rates are theoretically discounted for GLM 5.3 Flash right now, by 50%. 2) Hardware costs have continued ascending with no sign of letting off, so it's unlikely that a DGX Spark depreciates to zero in one year. 3) Compare performance in terms of difficult tasks/$ over the last 6 months, 3 months, etc. Open weights are a ratchet. In terms of intelligence per $, a Spark is never going to be a worse deal tomorrow than it is today, at least until the entire platform is replaced or obsoleted. 71 days ago the best model you could run on two Sparks was an aggressive Q3 quant of Qwen 3.5 397B (AA 34). 70 days ago it was a mixed-quant of GLM 5.2 (AA 53). 30 days ago it was full fat DeepSeek 4 Flash (AA 53). Today it's GLM 5.3 Flash (AA57) and/or Qwen 3.8 Next (Unknown). Sometime this week it will likely become mixed-quant GLM 5.3 (AA 60). So in < 80 days we have almost doubled the benchmark score. And that curve is still accelerating. If you view it as "cost per token of model vs API" then yes it's a bad deal. If you view it as "cost of task per $" then it has almost doubled in value in less than 3 months. All of this, imo, API and hardware, is still massively underpriced.