10 ms·

I'm about to leave a shallow comment, but I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely b

I'm about to leave a shallow comment, but I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? So the fact that publicly available information is conflicted is probably a sign that at the very least, the numbers aren't amazing. Yes I know there's no evidence and this is lazy reasoning. But there's probably a bit of truth to this line of thought.

whatshisface 4mo ago
Inference has traditionally been far less expensive than training. One public example is the fact that hobbyists can run StableDiffusion ($600k training costs[1]) on their personal computers. Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The model of providing inference for anything near the operational costs (like a utility would), would the delta from the norm if it were true. [1] https://x.com/emostaque/status/1563870674111832066 https://x.com/emostaque/status/1563870674111832066
- thesz 4mo ago
  The difference between training and inference is 1) one have to keep intermediate results for backward pass in training and 2) computation for training double because of the backward pass. Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute. One of the ways out of this unfortunate situation is to use something like Stochastic Average Gradient Descent [1]. Examples there are mostly concerned with regularized logistic regression, which makes problem more or less convex. Neural networks are inherently non-convex. Still, maybe some ideas from there can be utilized in the context of neural networks, like use of estimated Lipshitz constant to derive curvature and appropriate learning step. [1] https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdf
  - janalsncm 4mo ago
    So one way to think about it is roughly, Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
    - thesz 4mo ago
      Multiply "inference + backwards pass (~2x inference cost) + activations (vram overhead)" by batch size (thousands) to get to the actual RAM and compute cost. Optimizer like ADAM adds only two or three model-sized overhead. And last, but not least, you need only one hidden layer kept in RAM for inference, but you need all of them (61 for Deepseek models) kept in RAM for computing gradient for one sample.
      - galaxyLogic 4mo ago
        Does it matter what is the difference in size of needed inputs for inference vs. training?
      - xyhopguy 4mo ago
        Microbatch size is a hyperparameter, it can be set to 1 and work just as effectively. With gradient accumulation it's equivalent even. Large batch sizes are used to increase parallelism, and sometimes to reduce variance in the loss signal (at the cost of increased bias). Batch size is frequently limited by compute bottlenecks well before memory.
      - mcv 4mo ago
        And of course you do all of this for every object in your training set, which is going to be larger than the total number of uses for any individual user.
  - whatshisface 4mo ago
    That is an estimate of the relative cost of one training step, but you have to multiply it by the number of training steps, an unknown quantity.
  - mike_hearn 4mo ago
    It's all got much more complex than that in recent years. Training now involves large amounts of inference for RL rollouts and similar. You can't disentangle them computationally like that. "Inference" is just the word used to mean serving customer traffic now, and "training" means creating the model you serve.
- vanviegen 4mo ago
  I think in your StableDiffusion example, a lot more than $600k will have been spend on electricity alone for inference (on those personal computers you mention). So inference is more expensive then training.
Tuna-Fish 4mo ago
Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it. Also, inference costs are bound to go way down with more optimized architectures. GPUs are fundamentally not great at inference. No platform where the weights are streamed from a large pool of memory is. If the models ever quiet down, there will be massive step changes in cost/token, energy/token and tokens/second, as models are etched into silicon ala https://chatjimmy.ai/ https://chatjimmy.ai/
- golem14 4mo ago
  Why would any company brag about their margins ? Yet they do, to attract investors.
  - Tuna-Fish 4mo ago
    The key AI labs are not public companies, they are at liberty to brag about their margins to potential investors in private.
    - bwhiting2356 4mo ago
      this is changing soon
      - joelthelion 4mo ago
        Not really, how much of a public company are you when 5% of your capital is public ?
        tverbeure 4mo ago
        The percentage is irrelevant for this discussion. As soon as you’re public, you need to report detailed financial numbers.
lumost 4mo ago
For equal capability tokens, there has been about a 10x drop in cost every 6 months. We are still chasing the best because the best is moving rapidly, but it’s a simple thought experiment to work out what the cost to serve an 8B model from 2 years ago is in a world of 2T models. Note: parameter counts are illustrative. Concretely, qwen3.6 27B delivers opus 4.5 capability at 1/27th the cost on openrouter. Single chip llama3 8b performance can exceed 17k tokens/sec.
- byzantinegene 4mo ago
  8B models would be consider obsolete in the world of 2T models, at least if we're talking about the competitiveness of OpenAI/Anthropic. The only reason why they are valued so highly is their supposed dominance at the top end.
  - lumost 4mo ago
    The main story of agent use cases is in enterprise so far. An enterprise will only pay for a model capable of handling the task and no more. Most enterprise's see no need to hire PhDs as factory line workers. Coding is an interesting case as [1] the pace of progress has been absurd and [2] it's hard to put an upper bound on required capability. However hard to put a bound on and will are different, it's quite possible that the average engineer will cease to see the benefit of rapid progress - or that their employer will be satisfied with lower tier models. How smart of a model do you need to build a high quality CRUD app for internal users? Or build a scalable web service?
    - byzantinegene 4mo ago
      yes, which is why the revenue growth story is not looking so great for Anthropic/OpenAI, when open-source alternatives are not far behind with much lower costs.
- joshuahedlund 4mo ago
  > For equal capability tokens, there has been about a 10x drop in cost every 6 months Is this still happening? Opus 4.5 was six months ago, can you get its capabilities for 1/10 cost now? Are we on track to get the same for 4.6 in a couple months?
neuronexmachina 4mo ago
> If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? Google seems to pretty regularly post about how their TPU and algorithm advancements have been decreasing energy costs for both inference and training.
no-name-here 4mo ago
> I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? Unless to the grandparent commenter’s point they’re using it to obscure their large prisoner’s dilemma (training) cost?
brookst 4mo ago
What other companies brag about lowered costs? Isn’t that just a complicated way of asking customers to demand lower prices?

whatshisface 4mo ago

Inference has traditionally been far less expensive than training. One public example is the fact that hobbyists can run StableDiffusion ($600k training costs[1]) on their personal computers. Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The model of providing inference for anything near the operational costs (like a utility would), would the delta from the norm if it were true. [1] https://x.com/emostaque/status/1563870674111832066 https://x.com/emostaque/status/1563870674111832066

thesz 4mo ago
The difference between training and inference is 1) one have to keep intermediate results for backward pass in training and 2) computation for training double because of the backward pass. Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute. One of the ways out of this unfortunate situation is to use something like Stochastic Average Gradient Descent [1]. Examples there are mostly concerned with regularized logistic regression, which makes problem more or less convex. Neural networks are inherently non-convex. Still, maybe some ideas from there can be utilized in the context of neural networks, like use of estimated Lipshitz constant to derive curvature and appropriate learning step. [1] https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdf
- janalsncm 4mo ago
  So one way to think about it is roughly, Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
  - thesz 4mo ago
    Multiply "inference + backwards pass (~2x inference cost) + activations (vram overhead)" by batch size (thousands) to get to the actual RAM and compute cost. Optimizer like ADAM adds only two or three model-sized overhead. And last, but not least, you need only one hidden layer kept in RAM for inference, but you need all of them (61 for Deepseek models) kept in RAM for computing gradient for one sample.
    - galaxyLogic 4mo ago
      Does it matter what is the difference in size of needed inputs for inference vs. training?
    - xyhopguy 4mo ago
      Microbatch size is a hyperparameter, it can be set to 1 and work just as effectively. With gradient accumulation it's equivalent even. Large batch sizes are used to increase parallelism, and sometimes to reduce variance in the loss signal (at the cost of increased bias). Batch size is frequently limited by compute bottlenecks well before memory.
    - mcv 4mo ago
      And of course you do all of this for every object in your training set, which is going to be larger than the total number of uses for any individual user.
- whatshisface 4mo ago
  That is an estimate of the relative cost of one training step, but you have to multiply it by the number of training steps, an unknown quantity.
- mike_hearn 4mo ago
  It's all got much more complex than that in recent years. Training now involves large amounts of inference for RL rollouts and similar. You can't disentangle them computationally like that. "Inference" is just the word used to mean serving customer traffic now, and "training" means creating the model you serve.
vanviegen 4mo ago
I think in your StableDiffusion example, a lot more than $600k will have been spend on electricity alone for inference (on those personal computers you mention). So inference is more expensive then training.

Tuna-Fish 4mo ago

Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it. Also, inference costs are bound to go way down with more optimized architectures. GPUs are fundamentally not great at inference. No platform where the weights are streamed from a large pool of memory is. If the models ever quiet down, there will be massive step changes in cost/token, energy/token and tokens/second, as models are etched into silicon ala https://chatjimmy.ai/ https://chatjimmy.ai/

golem14 4mo ago
Why would any company brag about their margins ? Yet they do, to attract investors.
- Tuna-Fish 4mo ago
  The key AI labs are not public companies, they are at liberty to brag about their margins to potential investors in private.
  - bwhiting2356 4mo ago
    this is changing soon
    - joelthelion 4mo ago
      Not really, how much of a public company are you when 5% of your capital is public ?
      - tverbeure 4mo ago
        The percentage is irrelevant for this discussion. As soon as you’re public, you need to report detailed financial numbers.

lumost 4mo ago

For equal capability tokens, there has been about a 10x drop in cost every 6 months. We are still chasing the best because the best is moving rapidly, but it’s a simple thought experiment to work out what the cost to serve an 8B model from 2 years ago is in a world of 2T models. Note: parameter counts are illustrative. Concretely, qwen3.6 27B delivers opus 4.5 capability at 1/27th the cost on openrouter. Single chip llama3 8b performance can exceed 17k tokens/sec.

byzantinegene 4mo ago
8B models would be consider obsolete in the world of 2T models, at least if we're talking about the competitiveness of OpenAI/Anthropic. The only reason why they are valued so highly is their supposed dominance at the top end.
- lumost 4mo ago
  The main story of agent use cases is in enterprise so far. An enterprise will only pay for a model capable of handling the task and no more. Most enterprise's see no need to hire PhDs as factory line workers. Coding is an interesting case as [1] the pace of progress has been absurd and [2] it's hard to put an upper bound on required capability. However hard to put a bound on and will are different, it's quite possible that the average engineer will cease to see the benefit of rapid progress - or that their employer will be satisfied with lower tier models. How smart of a model do you need to build a high quality CRUD app for internal users? Or build a scalable web service?
  - byzantinegene 4mo ago
    yes, which is why the revenue growth story is not looking so great for Anthropic/OpenAI, when open-source alternatives are not far behind with much lower costs.
joshuahedlund 4mo ago
> For equal capability tokens, there has been about a 10x drop in cost every 6 months Is this still happening? Opus 4.5 was six months ago, can you get its capabilities for 1/10 cost now? Are we on track to get the same for 4.6 in a couple months?

neuronexmachina 4mo ago

> If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? Google seems to pretty regularly post about how their TPU and algorithm advancements have been decreasing energy costs for both inference and training.

no-name-here 4mo ago

> I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? Unless to the grandparent commenter’s point they’re using it to obscure their large prisoner’s dilemma (training) cost?

brookst 4mo ago

What other companies brag about lowered costs? Isn’t that just a complicated way of asking customers to demand lower prices?