13 ms·

It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud bea

by rkangel 21d ago

It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.

WASDx 20d ago
I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.
- rkangel 20d ago
  These companies have displayed zero respect for everyone's intellectual property getting these models trained. I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
  - WASDx 20d ago
    Then use other providers hosting open models. Companies and individuals already put their whole code base on the cloud. I'm genuinely interested in privacy-oriented use cases where ZDR is not enough.
    - Gigachad 20d ago
      I'm not that worried about the codebase itself. I'm worried about the fact coding agents poke around the terminal and system so much that there is almost a certainty that some of your other personal data ends up in the context somewhere which is getting logged in to a training dataset by random hosting providers.
    - applfanboysbgon 20d ago
      ZDR is built on trust. Given that end-to-end encryption fundamentally doesn't work with LLMs, as they need the content to be unencrypted to operate on it[1], you have no way to prove that once your plaintext data is on somebody else's server they aren't doing whatever the hell they please with it. All you have to rely on is their pinky promise that they won't do anything with it. Trust is a valid option, much of our society runs on trust, but you can eliminate the need for trust whatsoever by running on your own hardware. [1] Yes, I'm aware of experiments to operate on encrypted prompts, but these are only research attempts, not something that could actually be used with frontier models in production.
- elorant 20d ago
  Privacy isn’t only, I don’t want anyone to have access to my data. It could also be, I don’t want anyone to know my use case because it’s niche and highly profitable.
joemazerino 20d ago
And compliance.
jaggederest 20d ago
I think the biggest reason is to own the stack so your model can't be changed out from under you, but maybe I care about that too much.
- deleted 20d ago
  [deleted]
- zackify 20d ago
  You do, there's like 20 providers for any model on openrouter. You can also just spin bedrock or gcp and download the weights for later if you're worried. It's never going to make cost sense when the token rate is so low with how expensive ram is
  - jaggederest 20d ago
    What if the internet goes away?
    - zackify 19d ago
      Starlink? It's never gone anymore
- Aurornis 20d ago
  > I think the biggest reason is to own the stack so your model can't be changed out from under you, The concern would be future regulations that prohibit you from buying a hosted version of the model. Even that could be bypassed with a VPN to another country but it's more work to go through the payments. As long as there is demand for a model, it will be hosted by multiple providers.
  - jaggederest 20d ago
    What if the model is hopelessly obsolete, and thus no demand, but I want that specific model? Owning the weights and hardware is not just solving for one problem. It eliminates all the classes of problems that occur outside of your building, if you have a solar and battery setup. Also, on a more practical basis, what if the way it's served is bad. Maybe I want my specific KV setup, or ultra low quant for entertaining garbage at 200 tk/s
deleted 20d ago
[deleted]
truncate 20d ago
I'm actively uninspired to write high quality code when using Anthropic/OpenAI models given the high chance I'm a customer as well as used as dataset generation tool for them. But currently cloud does beat costs of hardware ownership, particularly with ridiculously high RAM/GPU/SSD costs....again due to these same companies.
somenameforme 20d ago
I think that's overly pessimistic. Here's [1] a video of somebody running it on a ~$6000 rig and getting around 14T/s for complex prompts (about double that for simpler prompts). Payback time is going to depend on your electric cost/consumption. In most domains cloud providers end up charging a significant premium rather than a offering a scale enabled discount, relative to local at retail costs. That will almost certainly end up being the case with LLMs as well, if it isn't already. Furthermore we continue to follow the path that image gen neural networks took. In that domain hardware requirements reached a peak and then started sharply declining to where we are today where a plain old video card can rapidly generate images that took a supercomputer not that long ago. So it's reasonable to assume that performance of such a system could potentially even increase over time. [1] - https://www.youtube.com/watch?v=ZWS2JVN2iBI https://www.youtube.com/watch?v=ZWS2JVN2iBI
- millicentricism 20d ago
  With roughly 2.7 million seconds per month, times 14 tokens per second, you are getting 38.5 million tokens a month at most. That’s less than 164USD worth of GLM5.3 tokens on the inference market. So that 6000 USD rig will take 3 years to break even - and only if it runs continuously. And this is being generous, as it’s not even taking quantisation into account.
  - somenameforme 20d ago
    I think if you steel-man what I'm saying, what you're saying falls apart. 14 tokens per second was rare. It only dropped that low in one scenario where he had it single shot an entire game (flappy bird clone) from scratch, with different assets, all self created, and so on. It ended up resulting in the LLM doing stuff like plotting out a some odd 100 item long to-do list, requerying it repeatedly, and so on. And it succeeded. Also as the video mentions, the guy wasn't very familiar with what he was doing, and so there are almost certainly various optimizations on the config side he could work out, especially as he was using a 5 GPU system, which default configs are probably not well optimized for. But I think we've rapidly moving along the same path as image gen stuff. Local generation has gone from purely theoretic, to requiring supercomputers to run relatively incapable models, to where we are today - where with a fairly basic high end setup, he's comfortably running a frontier level model. There's definitely an argument for going local that's only growing stronger by the day.
comandillos 20d ago
I mean, I think it depends. At home 3 of us we use AI for multiple reasons, from coding apps to asking general questions, and if we would have to pay equivalent subscriptions that would be ~1k a year on AI + submitting all your data to external services. I payed around ~8k on 2 DGX Sparks that, at the moment, serves perfectly fine as a ChatGPT/Claude replacement at home (DS4 Flash peaking at ~170 tokens per sec with 6 concurrent sequences), and even once the technology is obsolete for inference in a few years, I will still have 2 pretty powerful machines for whatever I need + some pretty fast NVME Storage. I don't think its a terribly bad idea.
alexpotato 20d ago
> It is absolutely not worth buying hardware to run models for purely (long term) cost reasons This is especially true when it's trivial to have the LLM itself write you a script/tool that can rent a GPU node for you (via API calls to providers) and then download and set up an open weight model for you.
jrm4 20d ago
It seems absurdly naive to rely on "oh, the cloud AI of the future will definitely be as open and priced the same way it is right now." And not "Hey, these companies have a history of giving you something nice now, and rugpulling you either in quality or price later." Your "absolutely" seems silly.

by rkangel 21d ago

WASDx 20d ago

I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.

rkangel 20d ago
These companies have displayed zero respect for everyone's intellectual property getting these models trained. I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
- WASDx 20d ago
  Then use other providers hosting open models. Companies and individuals already put their whole code base on the cloud. I'm genuinely interested in privacy-oriented use cases where ZDR is not enough.
  - Gigachad 20d ago
    I'm not that worried about the codebase itself. I'm worried about the fact coding agents poke around the terminal and system so much that there is almost a certainty that some of your other personal data ends up in the context somewhere which is getting logged in to a training dataset by random hosting providers.
  - applfanboysbgon 20d ago
    ZDR is built on trust. Given that end-to-end encryption fundamentally doesn't work with LLMs, as they need the content to be unencrypted to operate on it[1], you have no way to prove that once your plaintext data is on somebody else's server they aren't doing whatever the hell they please with it. All you have to rely on is their pinky promise that they won't do anything with it. Trust is a valid option, much of our society runs on trust, but you can eliminate the need for trust whatsoever by running on your own hardware. [1] Yes, I'm aware of experiments to operate on encrypted prompts, but these are only research attempts, not something that could actually be used with frontier models in production.
elorant 20d ago
Privacy isn’t only, I don’t want anyone to have access to my data. It could also be, I don’t want anyone to know my use case because it’s niche and highly profitable.

joemazerino 20d ago

And compliance.

jaggederest 20d ago

I think the biggest reason is to own the stack so your model can't be changed out from under you, but maybe I care about that too much.

deleted 20d ago
[deleted]
zackify 20d ago
You do, there's like 20 providers for any model on openrouter. You can also just spin bedrock or gcp and download the weights for later if you're worried. It's never going to make cost sense when the token rate is so low with how expensive ram is
- jaggederest 20d ago
  What if the internet goes away?
  - zackify 19d ago
    Starlink? It's never gone anymore
Aurornis 20d ago
> I think the biggest reason is to own the stack so your model can't be changed out from under you, The concern would be future regulations that prohibit you from buying a hosted version of the model. Even that could be bypassed with a VPN to another country but it's more work to go through the payments. As long as there is demand for a model, it will be hosted by multiple providers.
- jaggederest 20d ago
  What if the model is hopelessly obsolete, and thus no demand, but I want that specific model? Owning the weights and hardware is not just solving for one problem. It eliminates all the classes of problems that occur outside of your building, if you have a solar and battery setup. Also, on a more practical basis, what if the way it's served is bad. Maybe I want my specific KV setup, or ultra low quant for entertaining garbage at 200 tk/s

deleted 20d ago

[deleted]

truncate 20d ago

I'm actively uninspired to write high quality code when using Anthropic/OpenAI models given the high chance I'm a customer as well as used as dataset generation tool for them. But currently cloud does beat costs of hardware ownership, particularly with ridiculously high RAM/GPU/SSD costs....again due to these same companies.

somenameforme 20d ago

I think that's overly pessimistic. Here's [1] a video of somebody running it on a ~$6000 rig and getting around 14T/s for complex prompts (about double that for simpler prompts). Payback time is going to depend on your electric cost/consumption. In most domains cloud providers end up charging a significant premium rather than a offering a scale enabled discount, relative to local at retail costs. That will almost certainly end up being the case with LLMs as well, if it isn't already. Furthermore we continue to follow the path that image gen neural networks took. In that domain hardware requirements reached a peak and then started sharply declining to where we are today where a plain old video card can rapidly generate images that took a supercomputer not that long ago. So it's reasonable to assume that performance of such a system could potentially even increase over time. [1] - https://www.youtube.com/watch?v=ZWS2JVN2iBI https://www.youtube.com/watch?v=ZWS2JVN2iBI

millicentricism 20d ago
With roughly 2.7 million seconds per month, times 14 tokens per second, you are getting 38.5 million tokens a month at most. That’s less than 164USD worth of GLM5.3 tokens on the inference market. So that 6000 USD rig will take 3 years to break even - and only if it runs continuously. And this is being generous, as it’s not even taking quantisation into account.
- somenameforme 20d ago
  I think if you steel-man what I'm saying, what you're saying falls apart. 14 tokens per second was rare. It only dropped that low in one scenario where he had it single shot an entire game (flappy bird clone) from scratch, with different assets, all self created, and so on. It ended up resulting in the LLM doing stuff like plotting out a some odd 100 item long to-do list, requerying it repeatedly, and so on. And it succeeded. Also as the video mentions, the guy wasn't very familiar with what he was doing, and so there are almost certainly various optimizations on the config side he could work out, especially as he was using a 5 GPU system, which default configs are probably not well optimized for. But I think we've rapidly moving along the same path as image gen stuff. Local generation has gone from purely theoretic, to requiring supercomputers to run relatively incapable models, to where we are today - where with a fairly basic high end setup, he's comfortably running a frontier level model. There's definitely an argument for going local that's only growing stronger by the day.

comandillos 20d ago

I mean, I think it depends. At home 3 of us we use AI for multiple reasons, from coding apps to asking general questions, and if we would have to pay equivalent subscriptions that would be ~1k a year on AI + submitting all your data to external services. I payed around ~8k on 2 DGX Sparks that, at the moment, serves perfectly fine as a ChatGPT/Claude replacement at home (DS4 Flash peaking at ~170 tokens per sec with 6 concurrent sequences), and even once the technology is obsolete for inference in a few years, I will still have 2 pretty powerful machines for whatever I need + some pretty fast NVME Storage. I don't think its a terribly bad idea.

alexpotato 20d ago

> It is absolutely not worth buying hardware to run models for purely (long term) cost reasons This is especially true when it's trivial to have the LLM itself write you a script/tool that can rent a GPU node for you (via API calls to providers) and then download and set up an open weight model for you.

jrm4 20d ago

It seems absurdly naive to rely on "oh, the cloud AI of the future will definitely be as open and priced the same way it is right now." And not "Hey, these companies have a history of giving you something nice now, and rugpulling you either in quality or price later." Your "absolutely" seems silly.