6 ms·
same here, I am still in shock. took me 10 minutes to find the 'operational issue' message in the dashboard. longest 10 minutes of my life.
by dirkk0 2mo ago
same here, I am still in shock. took me 10 minutes to find the 'operational issue' message in the dashboard. longest 10 minutes of my life.
- charles_f 2mo agoCan you not set spending limits in AWS?
- reformd 2mo agohe did, 140 billion :D
- inigyou 2mo agoNo you can't. Spending limits imply realtime billing backend flows and they also imply deleting all your data so that you don't pay for storage.
- prmoustache 2mo agoStorage could switch to read only. That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
- handoflixue 2mo agoRealtime billing seems entirely within the abilities of AWS. "Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
- Planktonne 2mo agoThey could do it; they don't want to.
- minitoar 2mo agoWhat is a storage-based billing story?
- kgwgk 2mo agoOnce upon a time in a cloud kingdom far, far away a big, beautiful bill was issued based on storage causing much disconcertion. Etc.
- handoflixue 2mo agoLike kgwgk already said - people post the occasional horror story where they run up a huge AWS bill, either due to a run-away process or getting hacked. It seems to almost always be about Compute, not Storage.
- everforward 2mo agoStorage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”. Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend. GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
- inigyou 2mo agoStorage-based billing stories. When an account is hijacked it's always for compute, not storage.
- everforward 2mo agoOh, I also don’t think I’ve ever seen that but I’m not surprised. Even if you could steal a huge amount of storage, filling it with data would take ages and the cat and mouse game of moving the data as hacks get uncovered would be untenable. I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
- benterix 2mo agoI heard this false justification already in 2007, in spite of many customers asking for it. Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
- inigyou 2mo agoBig cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.
- bcrosby95 2mo agoWith AI it should take like a weekend.
- inigyou 2mo agoI think they already tried rewriting billing with AI. Very smartly they only tried rewriting the estimator first. This post is about the outcome of it.
- thewhitetulip 2mo agoWeekend is too long bro! With AI they can weite 50 implementations in a single prompt. They clearly are using tooo many humans and that's why they're lagging More AI will 100% fix the issue lol
- SAI_Peregrinus 2mo ago> and they also imply deleting all your data so that you don't pay for storage. Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
- inigyou 2mo agoThat's even more work to implement. And now you store files on a second account that pays for only one day a month to not get deleted.
- 0cf8612b2e1e 2mo agoNo wiggle room to come up with a workable solution. Let’s go shopping instead.
- SAI_Peregrinus 2mo agoHow would that work? If you only pay for one day, your access (read and write) gets cut off after the first month. Then if you don't pay the bill for the entire grace period, your data gets deleted.
- boristsr 2mo agoNo, alerts but not limits.
- perching_aix 2mo agoNot only can you not set limits, even the alarms are not real time. So it is entirely possible to get on the hook for terrifying amounts of money and not know until it's all too late.
- masafej536 2mo agoIf you owe AWS 140B dollars its their problem ;)