6 ms·

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do

by lbriner 1mo ago

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.

jgalt212 1mo ago
> we started using AI tools and ended up spending millions per year This is how AWS made its fortune.
- chadash 1mo ago
  Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend. Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business. And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
  - andrekandre 1mo ago
    > And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually. the best thing to do is avoid the first cigarette (but everyone at the office smokes, and if you want a promotion you'll smoke with them too)
ankitmathur 1mo ago
Something underlying a lot of this is that pricing models for enterprise coding tools have changed from seat-based to consumption-based pretty quickly, as AI usage has exploded. For months, engineers were able to use unlimited AI for no marginal cost, but that's changed quickly. In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.
therealdrag0 1mo ago
On this planet? They’re not saying they regret doing it, or that it was a mistake. They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized. Pretty standard business or life iteration.
pwendell 1mo ago
The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user growth as well and BOOM suddenly you're at a $10M run rate within 60 days. And it's now impossible to forecast future growth. It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations. The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
- emanuelecella 1mo ago
  [flagged]
K3UL 1mo ago
Weirdly a lot of the come from company that sell Ai credits in some capacity, and who are also selling (or will soon) some kind of AI gateway or router
habosa 1mo ago
There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new. 1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage. 2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money). 3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter. 4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions. 5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change. 6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates. I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.
- napoleond 1mo ago
  I think my startup can help: https://unbiased.ai https://unbiased.ai Happy to give you (or anyone here) some trial credits if interested! Email address in my profile.
- deleted 1mo ago
  [deleted]
nitwit005 1mo ago
Someone did notice, as they panicked at the cost. What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.
dan_q 1mo ago
> On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering. Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers. The point of AI is: 1. It's bad engineering and everyone knows it. 2. Management is pushing it through anyway. 3. Deal with it, asshole. The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control. Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.
- andrekandre 1mo ago
  > The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control. it may be where you are at (not disagreeing at all with your experience), but where i am the management really are 'true believers' so to speak... they are all-in and mesmerized by the tech (call it fomo or whatever)

by lbriner 1mo ago

jgalt212 1mo ago

> we started using AI tools and ended up spending millions per year This is how AWS made its fortune.

chadash 1mo ago
Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend. Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business. And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
- andrekandre 1mo ago
  > And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually. the best thing to do is avoid the first cigarette (but everyone at the office smokes, and if you want a promotion you'll smoke with them too)

ankitmathur 1mo ago

Something underlying a lot of this is that pricing models for enterprise coding tools have changed from seat-based to consumption-based pretty quickly, as AI usage has exploded. For months, engineers were able to use unlimited AI for no marginal cost, but that's changed quickly. In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.

therealdrag0 1mo ago

On this planet? They’re not saying they regret doing it, or that it was a mistake. They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized. Pretty standard business or life iteration.

pwendell 1mo ago

The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user growth as well and BOOM suddenly you're at a $10M run rate within 60 days. And it's now impossible to forecast future growth. It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations. The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.

emanuelecella 1mo ago
[flagged]

K3UL 1mo ago

Weirdly a lot of the come from company that sell Ai credits in some capacity, and who are also selling (or will soon) some kind of AI gateway or router

habosa 1mo ago

There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new. 1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage. 2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money). 3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter. 4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions. 5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change. 6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates. I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.

napoleond 1mo ago
I think my startup can help: https://unbiased.ai https://unbiased.ai Happy to give you (or anyone here) some trial credits if interested! Email address in my profile.
deleted 1mo ago
[deleted]

nitwit005 1mo ago

Someone did notice, as they panicked at the cost. What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.

dan_q 1mo ago

> On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering. Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers. The point of AI is: 1. It's bad engineering and everyone knows it. 2. Management is pushing it through anyway. 3. Deal with it, asshole. The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control. Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.

andrekandre 1mo ago
> The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control. it may be where you are at (not disagreeing at all with your experience), but where i am the management really are 'true believers' so to speak... they are all-in and mesmerized by the tech (call it fomo or whatever)