6 ms·
Amazon S3 will no longer charge for several HTTP error codes
- toomuchtodo 2y agoRelated: Jeff Barr acknowledges S3 unauthorized request billing issue - https://news.ycombinator.com/item?id=40221108 https://news.ycombinator.com/item?id=40221108 - May 2024 (18 comments) How an empty S3 bucket can make your AWS bill explode - https://news.ycombinator.com/item?id=40203126 https://news.ycombinator.com/item?id=40203126 - April 2024 (111 comments)
- jsheard 2y agoThe system works! Just raise your concerns and they'll get around to it in [checks notes] 18 years https://twitter.com/cperciva/status/1785402732976992417 https://twitter.com/cperciva/status/1785402732976992417
- cbsmith 2y agoIn fairness, the issue was attended to within weeks after it recently got attention.
- recursive 2y agoSo, another way of saying this is, it took more than a decade to get attention?
- surgical_fire 2y agosomething something customer obsession
- pavel_lishin 2y agoWe prefer to call it "eventual consistency".
- cbsmith 2y agoRealistically, Amazon didn't have the scale/resources to mitigate/manage this problem for customers back in the day. It also wasn't a target like it is now. Even a decade ago, it was a comparatively small problem that was no doubt simpler to address this on a case-by-case basis. Being responsive isn't about having infinite resources. It's about prioritization. I doubt at the time this was anywhere near the top of the list for them to fix even for the person who tweeted years ago.
- hiatus 2y agoYou made more or less the exact same comment on a recent thread. What does it add to this conversation? https://news.ycombinator.com/item?id=40221193 https://news.ycombinator.com/item?id=40221193
- j-pb 2y agoIt gave me pleasure.
- spacebanana7 2y agoTo be fair it is quite remarkable customer service story & relevant to the article.
- Quinner 2y agoAdds historical context as to a duration of an extremely long-lasting problem?
- jsheard 2y agoI did, but I think it's worth stressing that this didn't actually have the quick two week turnaround you might assume if you first heard about it from the billing horror story posted on here recently. It's been known about forever and only became a priority when it turned into a PR issue.
- tyingq 2y agoAs an HN reader it added valuable context for me that I was unaware of.
- treve 2y agoIn the same timespan Microsoft released Windows 1 all the way up to XP
- DaiPlusPlus 2y agoWhy have things stagnated?
- ranger_danger 2y agoPeople put up with it.
- SSLy 2y agoactual innovation has just moved to different problems, now that the desktop OS has been mostly "solved" – until SteamOS showed a fresher way forward, that is.
- CSMastermind 2y agoAhh I see the problem. The steps to get it resolved were not to tell the team about it. The steps were to raise a big enough fuss that it would undermine customer trust if the team didn't fix it.
- swyx 2y agotwitter support tier is the highest of all!
- ceejayoz 2y ago> For buckets configured with website hosting, applicable request and other charges will still apply when S3 returns a custom error document or for custom redirects. I was wondering about that one.
- Joel_Mckay 2y agoBezos loss-leader product-manager pushes hook deeper into worm. I fail to see this as progress, YMMV =3
- beeeeerp 2y agoNow please do this for NXDOMAIN on Route53. This can be a big problem with acquired domains.
- SushiHippie 2y agoI just searched for this and this documentation entry came up: https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/configuring-route53-for-cost-protection-from-nxdomain-attacks.html https://docs.aws.amazon.com/whitepapers/latest/aws-best-prac... I can't believe that their 'fix' is to set a wildcard dns entry, this feels somewhat like a joke. Does this mean that a NXDOMAIN response costs more than a successful response?
- aeyes 2y agoIt has the same cost as a successful response which can quickly add up to a few hundred dollars per month with a couple of DNS enumeration scans. Google Cloud and Azure also bill DNS like this. Unless you need some of the advanced features you really shouldn't host your DNS in the big cloud providers.
- technion 2y agoCloudflare don't bill like this, part of why I moved off route 53.
- withinboredom 2y agoNeither does Digital Ocean, for that matter.
- beeeeerp 2y agoThat’s not entirely true - aliases to AWS resources are free (their suggested “workaround”).
- mike_d 2y agoYou should never actually use Route53 for your domains. Delegate a subdomain like cloud.yourcompany.net to R53 and use that.
- dmw_ng 2y agoCan't imagine a change like this would be made without some analysis.. would love an internal view into a decision like this, I wonder if they already have log data to compute financial loss from the change, or if they have sampling instrumentation fancy enough to write/deploy custom reports like this quickly. In any case 2 weeks seems like an impressive turnaround for such a large service, unless they'd been internally preparing to acknowledge the problem for longer
- pdimitar 2y agoAre you for real? Legitimately baffled by your comment. How about the financial losses of customers that could be DDoS-ed into bankruptcy through no fault of their own? Keeping S3 bucket names secret is not always easy.
- xmprt 2y agoI was thinking this too. You're giving AWS a lot of credit if you think they're not going to do some kind of analysis about how much they were making (albeit illegitimately) from invalid responses. I'm just surprised that they either didn't do the analysis beforehand or that if they did do the analysis beforehand (like the parent commenter suspected), how they were able to get the report for that analysis out so quickly.
- dmw_ng 2y agoI prefer your version: Barr replies to a tweet before gatecrashing the next S3 planning session. "A customer is hurting, folks!". The call immediately falls silent with only occasional gasps heard from stunned engineers, and the gentle weeping of a PM. I wonder if Amazon offers free therapy following an incident like this
- pdimitar 2y agoNot billing you because a script kiddie ran a script on your S3 bucket is a good start of a therapy, I'd say. :)
- londons_explore 2y ago
- chadhutchins10 2y agoWe've done it. Now let's re-engineer our apps to use error codes for 200 responses and get free S3 usage.
- hunter2_ 2y agoIf I understand TFA, you'd need to find a way to get S3 (which offers no server-side script execution, only basic file delivery) to emit an error code (403 specifically) alongside a response of useful data. Good luck...
- dgacmu 2y agoWell, you can probably send out one bit a time by updating your ACLs on a clock (with which your clients are also roughly synchronized) and distinguishing between 403 and 404. take an awful lot of time to get that data out, though.
- ornornor 2y ago> take an awful lot of time to get that data out, though. That’s what glacier is for!
- saltcured 2y agoIt seems to me you could just use static ACLs and create (or not) object names to cause this 403 vs 404 distinction? The drawback is that you'll be paying for the minimum retention of minimum-sized objects, not to mention all the other bucket management traffic you are using. So you're going to have a lot of consumers of the same bit stream before you've somehow made the covert, "free" egress a net positive value versus a regular object. I imagine AWS can trivially put in place some throttling of error responses to make this impractical. Ignoring these economic issues, imagine a content-addressing scheme like /stream-identifier/bitnumber which you can then poll to fetch one bit per request. Populate an object (which will return 403) for 1 bits and omit an object (which will return 404) for 0 bits. You also need to know some stream length or "end of stream" limit. Otherwise you can't tell if you've read past the end or are really fetching 0 bits of a longer stream. One strategy might be to use an 8b/10b encoding so you can detect when you're not getting a valid symbol anymore. You could treat that as end of stream if it is supposed to be static, or go into some polling mode to wait for more symbols to be posted. Hybrid strategies might use regular objects or recursive use of these streams to publish metadata streams that tell you about the available stream names, lengths, and encoding schemes.
- lapcat 2y agoAWS is full of dark patterns. You can sign up for the so-called "free" tier and then too easily, unwittingly enable something that suddenly charges you hundreds of dollars before you know it (by getting a bill at the end of the month), even if you're not doing anything with the service except looking around. AWS doesn't give any warning to free tier members that a configuration change is going to cost you, and their terms are also very confusing. For example, PostgreSQL is advertised as free, but "Aurora PostgreSQL" is quite costly.
- ranger_danger 2y ago> AWS doesn't give any warning It does if you ask it to. You can get billing alerts if current costs are projected to go over a threshold.
- OptionOfT 2y agoBut it's kinda the same as a trial where you have to put in a credit card number. If they auto-charge once the trial is over, I don't like them. That is a dark pattern. Equally, with this, AWS could very well ask you, the user, what you'd like to do if you surpass the free tier. Charge? Or turn it all off. On top of that they could instate default thresholds so that you, the person who just started their free trial, does not get bill shock when you forget to turn of that $200/h machine.
- paulddraper 2y agoAlmost all trials tell you how much they'll charge you.
- lapcat 2y agoMy threshold is $0. I was never expecting to get billed on the free tier.
- hughesjj 2y agoYou have to manually set this up though. There's so much UX for a prospective new AWS dev that could be improved. Say a 1 click "do you want billing alerts with that?" Template, or a "do you want to lock down expensive nonfree stuff?" option to set some soft limits to zero out of the gate (nescessitating a self serve support case to unblock). It's frustrating. It's been this way for over a decade yet you'll still see new customers cutting themselves on the nuances of free tier. I get that AWS is 'enterprise real deal do what I say', but I don't think that means you should completely exclude the customer story of any new developers just getting their feet wet. It's an area of customer obsession the business has regrettably lacked, and if you go by the continuous stories of people messing it up on HN/twitter/reddit/etc, it only becomes more glaring how little the new guys are being taken care of.
- cratermoon 2y agoFrom the previous story, "S3 requests without a specified region default to us-east-1 and are redirected as needed. And the bucket’s owner pays extra for that redirected request." So will Amazon continue charge for the redirected 403?
- dangoodmanUT 2y agowe canceled them successfully ig
- moi2388 2y agoThere needs to be a law that says any user needs to set any limit on any service or subscription, and then the costs can not surpass this until the budget is upped by the user. At the same time, there should be real-time cost analysis, breakdown per service and predicted costs per day.
- usr1106 2y agoA law in which country? Well, GDPR showed a bit that rather global impact is possible. If you offer an open service on the internet you need to be prepared that users and misusers will cause costs. However, if you block it for public access you as a customer are not offering a public service. It's the cloud provider offering a public service so it seems just a basic legal principle that it's the cloud provider who pays for misuse (attempts to access something that is not public). But of course big corporations are not known for fair contracts respecting legitimate interest of the customer before legal action is on the horizon. I wonder what made AWS wake up here.
- moi2388 2y agoAgreed. Don’t know what made them wake up, but I did file a complaint about their free tier dark patterns and the Luxembourg EU GDPR office got involved after my countries GDPR office tried it first, and apparently is busy with some bigger investigation, so that investigation might’ve spooked them (not my own application I don’t think)