7 ms·

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.

nurettin 2mo ago
This is why I always fail loud rather than pick a stupid default.
DelightOne 2mo ago
Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
01284a7e 2mo ago
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo https://www.youtube.com/watch?v=3fGHaVn5rGo
- akdev1l 2mo ago
  Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
  - qurren 2mo ago
    You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
    - tyre 2mo ago
      Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
      - geodel 2mo ago
        "Former" seems to an important detail here.
        gleenn 2mo ago
        If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
        FabCH 2mo ago
        Not the OP, but: The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
        switchbak 2mo ago
        If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else. Or in my case, actively ignore any and all recruiting from that sesspool.
  - 01284a7e 2mo ago
    If only there was some way to get anomaly detection services [1] inside of AWS... 1: https://aws.amazon.com/what-is/anomaly-detection/ https://aws.amazon.com/what-is/anomaly-detection/
  - mcpherrinm 2mo ago
    While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills. I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_and_tsunami https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped. I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure) It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
    - oenton 2mo ago
      > there's definitely this sort of monitoring in place. Surely somebody was paged. Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
  - Groxx 2mo ago
    They've already got anomaly detection: their users.
  - el1s7 2mo ago
    I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
  - dclowd9901 2mo ago
    Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
- mvdtnz 2mo ago
  Why would you think there are "no tests"?
  - 27183 2mo ago
    We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
    - nullorempty 2mo ago
      Technically, there could be a test. It could just be wrong!
      - 27183 2mo ago
        If a tree falls in the forest and nobody hears it... [edit] Testing your tests, like testing your backups, is a good idea
        fragmede 2mo ago
        Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.
        27183 2mo ago
        Yeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done. You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
- deltaray3 2mo ago
  It's just like in Superman III
- CobrastanJorji 2mo ago
  There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
  - qeternity 2mo ago
    But these aren't the right services where the test should be, right? There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X". It's the third service that multiplies these things that failed. Where are the tests for that?
    - DrewADesign 2mo ago
      To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice. I’m totally guessing though.
      - lostlogin 2mo ago
        > human-or-LLM-driven error. In a computer system, dont those categories cover pretty much everything except a meteor strike?
- vasco 2mo ago
  I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
  - simmerup 2mo ago
    I’d be concerned as it throws into doubt their entire accounting mechanism
  - yawaramin 2mo ago
    The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
  - znsnsksjiaja 2mo ago
    If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there. They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
- SlightlyLeftPad 2mo ago
  It’s fractions of a penny Peter.
- bas 2mo ago
  Who needs tests when you have vibes?
- andai 2mo ago
  >no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect. I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing! Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes. https://en.wikipedia.org/wiki/Normalization_of_deviance https://en.wikipedia.org/wiki/Normalization_of_deviance
  - rtpg 2mo ago
    write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes. If you write the tests the agents have an easier time writing the code!
  - antihipocrat 2mo ago
    I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase. The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
    - andai 2mo ago
      Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little. The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!
- hsbauauvhabzb 2mo ago
  What could a banana possibly cost Michael, ten dollars?
- roadhero 2mo ago
  we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D
- DonHopkins 2mo ago
  ...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.
- nonameiguess 2mo ago
  This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode. There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless. This seems to happen quite a bit with software companies.
  - spydum 2mo ago
    I have bad news if you think banks are the pinnacle of cyber security practices... Maybe some rare few are, but by and large they generally stink. They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
    - antonvs 2mo ago
      The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains. So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average. This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.
pudgywalsh 2mo ago
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
- AlotOfReading 2mo ago
  Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.
  - gleenn 2mo ago
    One of the Mars landers famously failed due to unit conversion errors from metric to standard.
    - golem14 2mo ago
      Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.
    - blemasle 2mo ago
      I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
      - Dan_- 2mo ago
        I know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”
      - fc417fc802 2mo ago
        Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures. That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
crossroadsguy 2mo ago
Had it been half a million dollars or something or say like a few hundred dollars?
sscaryterry 2mo ago
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
- edelbitter 2mo ago
  [dead]
- 8note 2mo ago
  its so far out as to be obviously not real a smaller error by say, just one or two orders of magnitude are much more believable as a reader
  - sscaryterry 2mo ago
    I've seen people have panic attacks over much smaller amounts. Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
  - creatonez 2mo ago
    Not everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates
- shermantanktop 2mo ago
  But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions? Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
  - picofarad 2mo ago
    Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars. I'm stressed thinking about it, like from a theoretical standpoint.
Twirrim 2mo ago
I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
- hotstickyballs 2mo ago
  CSV files are widely known to be used by the most frugal companies so of course it is.
  - fragmede 2mo ago
    Have you seen what RDS costs‽ AWS couldn't possibly afford that!
  - berkes 2mo ago
    I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far. Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt. I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day. Because CSV is the "interface" that will bring your system down if not properly designed.
- anvuong 2mo ago
  That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
dev_l1x_be 2mo ago
Imagine a programming language that has physical measurement unit support so this could have never happened.
- yawaramin 2mo ago
  Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/units-of-measure https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
  - dev_l1x_be 2mo ago
    Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).
QuinnyPig 2mo ago
Was this the day that gp3 EBS volumes came out by any chance?
- donavanm 2mo ago
  Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.
dlenski 2mo ago
> Services emit metering values that arent directly tied to prices. Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both. I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
merek 2mo ago
Were there cases of finalized invoices and fund transfers while the error was in place? The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.
mlinhares 2mo ago
The last thing a billing system should have is defaults, this is such an insane thing.
- thewhitetulip 2mo ago
  Hey man, vibe coders gonna vine code. No wait. Loop enginering. We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!
mlitwiniuk 2mo ago
What apology letters? Never got anything, not even a reply to my support tickets.
AjeetTester2026 2mo ago
[dead]

nurettin 2mo ago

This is why I always fail loud rather than pick a stupid default.

DelightOne 2mo ago

Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?

01284a7e 2mo ago

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo https://www.youtube.com/watch?v=3fGHaVn5rGo

akdev1l 2mo ago
Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
- qurren 2mo ago
  You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
  - tyre 2mo ago
    Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
    - geodel 2mo ago
      "Former" seems to an important detail here.
      - gleenn 2mo ago
        If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
        FabCH 2mo ago
        Not the OP, but: The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
      - switchbak 2mo ago
        If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else. Or in my case, actively ignore any and all recruiting from that sesspool.
- 01284a7e 2mo ago
  If only there was some way to get anomaly detection services [1] inside of AWS... 1: https://aws.amazon.com/what-is/anomaly-detection/ https://aws.amazon.com/what-is/anomaly-detection/
- mcpherrinm 2mo ago
  While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills. I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_and_tsunami https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped. I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure) It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
  - oenton 2mo ago
    > there's definitely this sort of monitoring in place. Surely somebody was paged. Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
- Groxx 2mo ago
  They've already got anomaly detection: their users.
- el1s7 2mo ago
  I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
- dclowd9901 2mo ago
  Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
mvdtnz 2mo ago
Why would you think there are "no tests"?
- 27183 2mo ago
  We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
  - nullorempty 2mo ago
    Technically, there could be a test. It could just be wrong!
    - 27183 2mo ago
      If a tree falls in the forest and nobody hears it... [edit] Testing your tests, like testing your backups, is a good idea
      - fragmede 2mo ago
        Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.
        27183 2mo ago
        Yeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done. You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
deltaray3 2mo ago
It's just like in Superman III
CobrastanJorji 2mo ago
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
- qeternity 2mo ago
  But these aren't the right services where the test should be, right? There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X". It's the third service that multiplies these things that failed. Where are the tests for that?
  - DrewADesign 2mo ago
    To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice. I’m totally guessing though.
    - lostlogin 2mo ago
      > human-or-LLM-driven error. In a computer system, dont those categories cover pretty much everything except a meteor strike?
vasco 2mo ago
I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
- simmerup 2mo ago
  I’d be concerned as it throws into doubt their entire accounting mechanism
- yawaramin 2mo ago
  The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
- znsnsksjiaja 2mo ago
  If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there. They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
SlightlyLeftPad 2mo ago
It’s fractions of a penny Peter.
bas 2mo ago
Who needs tests when you have vibes?
andai 2mo ago
>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect. I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing! Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes. https://en.wikipedia.org/wiki/Normalization_of_deviance https://en.wikipedia.org/wiki/Normalization_of_deviance
- rtpg 2mo ago
  write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes. If you write the tests the agents have an easier time writing the code!
- antihipocrat 2mo ago
  I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase. The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
  - andai 2mo ago
    Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little. The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!
hsbauauvhabzb 2mo ago
What could a banana possibly cost Michael, ten dollars?
roadhero 2mo ago
we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D
DonHopkins 2mo ago
...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.
nonameiguess 2mo ago
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode. There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless. This seems to happen quite a bit with software companies.
- spydum 2mo ago
  I have bad news if you think banks are the pinnacle of cyber security practices... Maybe some rare few are, but by and large they generally stink. They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
  - antonvs 2mo ago
    The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains. So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average. This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.

pudgywalsh 2mo ago

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

AlotOfReading 2mo ago
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.
- gleenn 2mo ago
  One of the Mars landers famously failed due to unit conversion errors from metric to standard.
  - golem14 2mo ago
    Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.
  - blemasle 2mo ago
    I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
    - Dan_- 2mo ago
      I know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”
    - fc417fc802 2mo ago
      Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures. That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.

crossroadsguy 2mo ago

Had it been half a million dollars or something or say like a few hundred dollars?

sscaryterry 2mo ago

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

edelbitter 2mo ago
[dead]
8note 2mo ago
its so far out as to be obviously not real a smaller error by say, just one or two orders of magnitude are much more believable as a reader
- sscaryterry 2mo ago
  I've seen people have panic attacks over much smaller amounts. Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
- creatonez 2mo ago
  Not everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates
shermantanktop 2mo ago
But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions? Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
- picofarad 2mo ago
  Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars. I'm stressed thinking about it, like from a theoretical standpoint.

Twirrim 2mo ago

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!

hotstickyballs 2mo ago
CSV files are widely known to be used by the most frugal companies so of course it is.
- fragmede 2mo ago
  Have you seen what RDS costs‽ AWS couldn't possibly afford that!
- berkes 2mo ago
  I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far. Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt. I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day. Because CSV is the "interface" that will bring your system down if not properly designed.
anvuong 2mo ago
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

dev_l1x_be 2mo ago

Imagine a programming language that has physical measurement unit support so this could have never happened.

yawaramin 2mo ago
Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/units-of-measure https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
- dev_l1x_be 2mo ago
  Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).

QuinnyPig 2mo ago

Was this the day that gp3 EBS volumes came out by any chance?

donavanm 2mo ago
Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.

dlenski 2mo ago

> Services emit metering values that arent directly tied to prices. Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both. I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.

merek 2mo ago

Were there cases of finalized invoices and fund transfers while the error was in place? The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.

mlinhares 2mo ago

The last thing a billing system should have is defaults, this is such an insane thing.

thewhitetulip 2mo ago
Hey man, vibe coders gonna vine code. No wait. Loop enginering. We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!

mlitwiniuk 2mo ago

What apology letters? Never got anything, not even a reply to my support tickets.

AjeetTester2026 2mo ago

[dead]