5 ms·

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

SXX 3mo ago
I guess you missed recent news. Problem is that cloud LLM might just sliently sabotage your work by downgrading output model with no notice. Or cloud LLM might just refuse to sell to you because it dont like your passport.
- swiftcoder 3mo ago
  This is not really a problem for the open-weight models, you can always give your money to an inference provider in a different jurisdiction
- yorwba 3mo ago
  So you're buying expensive hardware as insurance for the case that your cloud provider turns against you and you have to switch to another of the twenty offering the same model https://openrouter.ai/z-ai/glm-5.2 https://openrouter.ai/z-ai/glm-5.2 or in the worst case buy the same hardware later? How does that make sense?
  - deleted 3mo ago
    [deleted]
  - brookst 3mo ago
    It’s rationalization for what people want to do anyway. Like buying a new car today and taking on gas, parking, etc, expenses in case the bus route you’re using goes away at some point in the future. It’s not an economic decision, it’s a desire to have the new car dressed up in what-ifs.
    - drptech 3mo ago
      [dead]
    - CamperBob2 3mo ago
      Yes, it is understandable that people who are subject to being kicked off the bus at random times through no fault of their own, or who sometimes find that the bus slows to 8 miles per hour and makes them late for work, or who are tired of arguing with the bus driver who refuses to take them to the liquor store, the casino, or the titty bar, may aspire to own a car, even a crappy one. Any more tortured metaphors in store for us?
  - drptech 3mo ago
    [dead]
walrus01 3mo ago
Not everyone lives in a place where electricity is $0.20 a kWh. For instance BC Hydro residential rates are $0.11 (CAD) for the first tier and $0.14 for the second tier of consumption in a month. At current exchange rate $0.14 CAD is $0.099 USD a kWh. Hydro Quebec is even cheaper. At a theoretical 6 tok/s, 86400 seconds in a day, approx 500,000 tokens of GLM5.2 output for 2 bucks a day seems like a pretty good bargain to me. Of course not counting the one time cost of the hardware to run it. But I see people dropping $4000-5000 on all kinds of much less useful stuff. Additionally in a place where people use electric baseboard heating or electric in floor radiant heating, or really any other heating element based system in winter that's less efficient than a heat pump, additional electrical from a computing load is basically "free" since you would be spending that same money otherwise to heat your house. If a computer with 512GB of RAM is dumping the waste heat into your room, it accomplishes a portion of the same thing as a baseboard. Not to mention there is a whole other less measurable benefit of having a locally hosted model that can't be turned off or arbitrarily restricted by a service provider, and where all of your queries and context cache aren't subject to surveillance by any third party.
- tmountain 3mo ago
  Lots of people have solar. Green AI, imagine that!
- discordance 3mo ago
  Where I live prices are often higher than 20c/kWh, but lets take your example and halve it (10c/kWh) so it's ~$1.40/day or ~$500/year. On Openrouter, the cheapest GLM 5.2 provider costs $3/MTok (at 44 tps). Assuming most use is output tokens, that's still the equivalent of 450k token/day, so we're in the same ball park, but without the capex for 2 3090's and the machine. Self hosted only makes economic sense if your priority is being in control / avoiding surveillance.
  - walrus01 3mo ago
    That's true, there's a lot of places where power is considerably more expensive than $0.20 USD/kWh. But also the 600W figure assumes that it's fully loaded 24x7x365. Running a system that will be 600W under max CPU usage on all cores and RAM and a few 3090-class GPUs, that same system might be only 90W or around there when idle at 0.00 unix load. If we say: (600 * 24 * 31)/1000 = 446kWh in a month at full load 24 hours a day But it could be less, such as: (90 * 12 * 31)/1000 = 33.48 kWh of idle time in a month, and 223kWh of "full load" 600W time in a month, if it's at full load only 12 hours a day. If you're the only user accessing it and you only "use" it 12 hours a day, that cumulative yearly dollar figure would be almost halved. Or even less if a person is using it in bursts and intermittently throughout an 8 hour workday.
dzjkb 3mo ago
how do you rent 2 3090s for $2.80/day?
matheusmoreira 3mo ago
We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I think the main reason not to run locally is to get the full models instead of quantized versions.
- traceroute66 3mo ago
  > We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted privacy-conscious options out there. Two names to look at are Tinfoil[1] and Privatemode (from Edgeless Systems)[2]. Tinfoil[1] is, sadly, US-based. EU-sovereignty-option is on their long-term radar. But they do have GLM-5.2 today. Privatemode[2] is a German company (Edgeless Systems) with EU-based servers. But sadly no GLM-5.2 today, it is on their mid-long term radar though. Both Tinfoil and Privatemode operate on the same concept of the LLM operating in a secure enclave and you have end-to-end attestation and encryption. Tinfoil have not been independently audited, it is somewhere on their long-term radar. Privatemode have been thoroughly independently audited with documentation available on request. Both of them are API-tokens-only. So if you're currently one of those people throwing $200 a month down the pan at Anthropic/OpenAI for a so-called-alleged 'unlimited' plan, then neither Tinfoil or Privatemode will be the place for you. [1]https://tinfoil.sh/ https://tinfoil.sh/ [2] https://www.privatemode.ai/ https://www.privatemode.ai/
  - patates 3mo ago
    > Apple next product refresh I have this feeling that it'll be very expensive and still scarce. Normally I wouldn't say this about Apple, because their pricing is part of their brand, but this time the demand (both by data-centers and prosumers) is the force majeure.
    - traceroute66 3mo ago
poulpy123 3mo ago
which hyper scaler would you suggest ?
throwawayffffas 3mo ago
So in my experience with 2 7900XTs with models that sit fully in VRAM it's more like 400W the gpus spend a lot of time waiting for each other.
- culopatin 3mo ago
  How’s running models with AMD these days?
DrScientist 3mo ago
Depends on whether you've also gone for self-hosted electricity generation or not.
segmondy 3mo ago
No one locally runs full load all day. The only way to see that is if you're training. We are talking about inference. I limit my GPU to 300watts. You can limit them down to 200w. Since everything is not in GPU and the bottleneck is between CPU/system ram. The GPUs don't even get to spike, I see 160w-180w for each GPU during inference. So redo your calculation again. Figure about 6 hrs of daily inference, and we are down to roughly $125 a year. Thanks again for your speculation.
deleted 3mo ago
[deleted]
downut 3mo ago
I have rooftop solar and I have been building credit with my electric utility even though the daily high temperature is well over 100F outside and a comfortable 75F inside. That includes running three AMD 12 thread 128GB systems with obsolete GPUs 24x7x365. I'm not a gamer, so 6 years ago I went low-end low-power GPUs. Boy am I dumb. Currently running the qwen3.6:27b, 35b, and gemma4:31b models just fine. As soon as VRAM prices drop to sanity I'm going to load up and I could care less about the power draw. Some parts of the future are absolutely great.
- downut 3mo ago
  I am fascinated that I got down voted. I mean, isn't what I'm doing here nearly ideal? Or maybe not: why? My solar panels shade my roof under the incessant sun of the Sonoran Desert and turn a fraction of the insolation into electrical power that allows me to do almost SOTA local LLM stuff inside my house for free[1] that the parent commenter thought to be economically infeasible. Of course it's slow! So what! Right now I'm transcribing to text a podcast with whisper.cpp and it will take about as much time as the original podcast duration but I will be able to read it in 1/20th of the time. Alternative interpretation of a downvote: we should all be enslaved to corporate electrical generation provided to "local" electric utility corporations so that we are economically incentivized to use cloud LLM providers. That's weird, no? Teach me. [1] It's a small nice house that cost ~$330K not too far off from my city center. This isn't rich privilege boasting.
bawana 3mo ago
Even on a macStudio w 512 gig memory?
eptcyka 3mo ago
Isn't that still cheaper than the 100 or 200$ plan that Anthropic wants from you?

SXX 3mo ago

I guess you missed recent news. Problem is that cloud LLM might just sliently sabotage your work by downgrading output model with no notice. Or cloud LLM might just refuse to sell to you because it dont like your passport.

swiftcoder 3mo ago
This is not really a problem for the open-weight models, you can always give your money to an inference provider in a different jurisdiction
yorwba 3mo ago
So you're buying expensive hardware as insurance for the case that your cloud provider turns against you and you have to switch to another of the twenty offering the same model https://openrouter.ai/z-ai/glm-5.2 https://openrouter.ai/z-ai/glm-5.2 or in the worst case buy the same hardware later? How does that make sense?
- deleted 3mo ago
  [deleted]
- brookst 3mo ago
  It’s rationalization for what people want to do anyway. Like buying a new car today and taking on gas, parking, etc, expenses in case the bus route you’re using goes away at some point in the future. It’s not an economic decision, it’s a desire to have the new car dressed up in what-ifs.
  - drptech 3mo ago
    [dead]
  - CamperBob2 3mo ago
    Yes, it is understandable that people who are subject to being kicked off the bus at random times through no fault of their own, or who sometimes find that the bus slows to 8 miles per hour and makes them late for work, or who are tired of arguing with the bus driver who refuses to take them to the liquor store, the casino, or the titty bar, may aspire to own a car, even a crappy one. Any more tortured metaphors in store for us?
- drptech 3mo ago
  [dead]

walrus01 3mo ago

Not everyone lives in a place where electricity is $0.20 a kWh. For instance BC Hydro residential rates are $0.11 (CAD) for the first tier and $0.14 for the second tier of consumption in a month. At current exchange rate $0.14 CAD is $0.099 USD a kWh. Hydro Quebec is even cheaper. At a theoretical 6 tok/s, 86400 seconds in a day, approx 500,000 tokens of GLM5.2 output for 2 bucks a day seems like a pretty good bargain to me. Of course not counting the one time cost of the hardware to run it. But I see people dropping $4000-5000 on all kinds of much less useful stuff. Additionally in a place where people use electric baseboard heating or electric in floor radiant heating, or really any other heating element based system in winter that's less efficient than a heat pump, additional electrical from a computing load is basically "free" since you would be spending that same money otherwise to heat your house. If a computer with 512GB of RAM is dumping the waste heat into your room, it accomplishes a portion of the same thing as a baseboard. Not to mention there is a whole other less measurable benefit of having a locally hosted model that can't be turned off or arbitrarily restricted by a service provider, and where all of your queries and context cache aren't subject to surveillance by any third party.

tmountain 3mo ago
Lots of people have solar. Green AI, imagine that!
discordance 3mo ago
Where I live prices are often higher than 20c/kWh, but lets take your example and halve it (10c/kWh) so it's ~$1.40/day or ~$500/year. On Openrouter, the cheapest GLM 5.2 provider costs $3/MTok (at 44 tps). Assuming most use is output tokens, that's still the equivalent of 450k token/day, so we're in the same ball park, but without the capex for 2 3090's and the machine. Self hosted only makes economic sense if your priority is being in control / avoiding surveillance.
- walrus01 3mo ago
  That's true, there's a lot of places where power is considerably more expensive than $0.20 USD/kWh. But also the 600W figure assumes that it's fully loaded 24x7x365. Running a system that will be 600W under max CPU usage on all cores and RAM and a few 3090-class GPUs, that same system might be only 90W or around there when idle at 0.00 unix load. If we say: (600 * 24 * 31)/1000 = 446kWh in a month at full load 24 hours a day But it could be less, such as: (90 * 12 * 31)/1000 = 33.48 kWh of idle time in a month, and 223kWh of "full load" 600W time in a month, if it's at full load only 12 hours a day. If you're the only user accessing it and you only "use" it 12 hours a day, that cumulative yearly dollar figure would be almost halved. Or even less if a person is using it in bursts and intermittently throughout an 8 hour workday.

dzjkb 3mo ago

how do you rent 2 3090s for $2.80/day?

matheusmoreira 3mo ago

We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I think the main reason not to run locally is to get the full models instead of quantized versions.

traceroute66 3mo ago
> We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted privacy-conscious options out there. Two names to look at are Tinfoil[1] and Privatemode (from Edgeless Systems)[2]. Tinfoil[1] is, sadly, US-based. EU-sovereignty-option is on their long-term radar. But they do have GLM-5.2 today. Privatemode[2] is a German company (Edgeless Systems) with EU-based servers. But sadly no GLM-5.2 today, it is on their mid-long term radar though. Both Tinfoil and Privatemode operate on the same concept of the LLM operating in a secure enclave and you have end-to-end attestation and encryption. Tinfoil have not been independently audited, it is somewhere on their long-term radar. Privatemode have been thoroughly independently audited with documentation available on request. Both of them are API-tokens-only. So if you're currently one of those people throwing $200 a month down the pan at Anthropic/OpenAI for a so-called-alleged 'unlimited' plan, then neither Tinfoil or Privatemode will be the place for you. [1]https://tinfoil.sh/ https://tinfoil.sh/ [2] https://www.privatemode.ai/ https://www.privatemode.ai/
- patates 3mo ago
  > Apple next product refresh I have this feeling that it'll be very expensive and still scarce. Normally I wouldn't say this about Apple, because their pricing is part of their brand, but this time the demand (both by data-centers and prosumers) is the force majeure.
  - traceroute66 3mo ago

poulpy123 3mo ago

which hyper scaler would you suggest ?

throwawayffffas 3mo ago

So in my experience with 2 7900XTs with models that sit fully in VRAM it's more like 400W the gpus spend a lot of time waiting for each other.

culopatin 3mo ago
How’s running models with AMD these days?

DrScientist 3mo ago

Depends on whether you've also gone for self-hosted electricity generation or not.

segmondy 3mo ago

No one locally runs full load all day. The only way to see that is if you're training. We are talking about inference. I limit my GPU to 300watts. You can limit them down to 200w. Since everything is not in GPU and the bottleneck is between CPU/system ram. The GPUs don't even get to spike, I see 160w-180w for each GPU during inference. So redo your calculation again. Figure about 6 hrs of daily inference, and we are down to roughly $125 a year. Thanks again for your speculation.

deleted 3mo ago

[deleted]

downut 3mo ago

I have rooftop solar and I have been building credit with my electric utility even though the daily high temperature is well over 100F outside and a comfortable 75F inside. That includes running three AMD 12 thread 128GB systems with obsolete GPUs 24x7x365. I'm not a gamer, so 6 years ago I went low-end low-power GPUs. Boy am I dumb. Currently running the qwen3.6:27b, 35b, and gemma4:31b models just fine. As soon as VRAM prices drop to sanity I'm going to load up and I could care less about the power draw. Some parts of the future are absolutely great.

downut 3mo ago
I am fascinated that I got down voted. I mean, isn't what I'm doing here nearly ideal? Or maybe not: why? My solar panels shade my roof under the incessant sun of the Sonoran Desert and turn a fraction of the insolation into electrical power that allows me to do almost SOTA local LLM stuff inside my house for free[1] that the parent commenter thought to be economically infeasible. Of course it's slow! So what! Right now I'm transcribing to text a podcast with whisper.cpp and it will take about as much time as the original podcast duration but I will be able to read it in 1/20th of the time. Alternative interpretation of a downvote: we should all be enslaved to corporate electrical generation provided to "local" electric utility corporations so that we are economically incentivized to use cloud LLM providers. That's weird, no? Teach me. [1] It's a small nice house that cost ~$330K not too far off from my city center. This isn't rich privilege boasting.

bawana 3mo ago

Even on a macStudio w 512 gig memory?

eptcyka 3mo ago

Isn't that still cheaper than the 100 or 200$ plan that Anthropic wants from you?