11 ms·
Google Cloud: $72,000 bill overnight
- jsnell 6y agoDupe (just a restatement of the original blog posts): https://news.ycombinator.com/item?id=25372336 https://news.ycombinator.com/item?id=25372336
- blindm 6y ago> Google let go of our bill as a one-time gesture How many times do they have to do that? Because if it is a high number, they would be operating at a loss.
- gonzo41 6y agoThat 72K is probably not anywhere near how cheap it was for google to provide the service.
- ncfausti 6y agoThis was my thought as well. What other kinds of businesses or services let you run up a bill of tens of thousands of dollars and then say, "Ok, you made a mistake, you can take it back"? Any educated guesses on what this compute might actually cost Google? I assume they're able to do this because the fixed costs have mostly been paid for already and the marginal cost of the electricity, system wear, and bandwidth are negligible, but I'm not sure.
- gonzo41 6y agoIt would cost them dollars. An IPhone costs < $100 to make. Obviously there's a development cost to it all. it's also easy to suggest that there's probably a feature missing atm :D
- gravypod 6y agoIn previous companies I've seen AWS and Azure do the same for >$10k bills from small startups. 1x$10,000 invoice is no where near as much as a lifetime of hosting a unicorn startup. Most cloud providers will even give you $50k to $100k in credits for 1 year if you are a startup from a good incubator/investor. It gives you incentives to not to care about how much you're spending. By the 1 year mark you're probably making some money, raising another series, and attempting to dramatically scale up your business so now you don't have the time to clean up the tech debt you've created to lower costs. They are absolutely making a bunch of money this way.
- imtringued 6y agoCloud providers somehow turned a commodity business into a high margin business. The costs are way lower than you think. The other factor is that keeping customers means more profit than throwing them out, even if they have made a mistake.
- LaundroMat 6y agoI had an € 1.200 bill from Google once for using their reverse geolocation API for a month. I complained and got a canned response saying something like "Fine, here's your money back but next time you're paying". It probably helped that I had been on the free tier up until then and complained that I never got a warning that I had surpassed the free tier amount of API calls. I'm not using the service anymore,
- dap 6y agoI have seen similar stories with AWS. It’s somewhat shocking to me that there’s no way to ask to get cut off above some dollar limit. Is every customer risking unbounded liability?
- lukeschlather 6y agoThe liability is on Google's side, mostly. There are hard limits in terms of the number of instances you can create without deliberately asking to spend more money, and these hard limits are set based on what Google is willing to write off for an overnight mistake.
- vinni2 6y agoWhat about prepaid credit cards with payment limit? If the payment failed the service will be terminated? Or does aws continue and send an invoice anyway?
- kristaps 6y agoGoogle once (falsely?) considered my Revolut debit card a pre-paid card and refused to accept it for GCS billing. The error message wasn't anything generic either - it stated specifically that pre-paid cards are not accepted.
- doktorhladnjak 6y agoAren't Revolut's cards prepaid? I was under the impression it's not full, real bank account since they make it easy to transfer money in out. I imagine they're making money off the issuing bank part of the interchange fees
- robjan 6y agoThey still send the invoice. I accidentally left a couple of small files in an S3 bucket for years. Eventually my card expired on that account and they continued bothering me until I contacted support and we agreed to waive the bill. I imagine they would have been more forceful if it was a larger bill.
- motohagiography 6y agoThis happened to me with Linkedin advertising, where my budget of a couple hundred dollars got re-charged to my card up to a couple thousand dollars without notifications. They handled all the complaints not in email, but their web based interface, and memory holed their commitment to refunding me the money after multiple back and forths. This isn't a mistake, the design is their business model. While we don't have a specific formal definition and name for it in the category of dark patterns, I'd like to name it "scumbag billing," where we got scumbagged.
- gazelleeatslion 6y agoOr, if you cancel your Amazon account it doesn’t immediately stop all AWS billing. It is — kid you not — recommended to terminate your individual services to avoid additional billing. Technical limitation of a trillion dollar company? I say scum bag billing
- raphaelj 6y agoWhat? How is that even legal ?
- deleted 6y ago[deleted]
- Red_Leaves_Flyy 6y agoCan you afford better lawyers than Amazon? If not there's your answer...
- gazelleeatslion 6y agoSpeculating but I wager they sell smaller collections in bulk to debt collectors at discount to try and collect vs running in house
- gazelleeatslion 6y agoLogin and click cancel account and read the language. I’m sure it’s because of some of their intertwined or niche services and not everything but is very real. You can cancel your credit card, but I imagine big Amazon and Co shops/sells their collections to the best on paper deal debt collectors (aka, truly the scummiest and worst ones). If you have ever been pursued for unpaid debt like this, despite consumer legal protections, it is years of Hell, legal threats/letters, calls, and other gray intimidation. All while wondering if your credit score will just be nuked over night. Source: Victim of identity fraud
- ExcavateGrandMa 6y agohuhmygod 72k! that's kind of a legal scam and they calls it a service? :Ð Also instead taking yourself for an hacker who can handle a compute instance... now you are fully aware you can't :Ð Programming without exception... or even without common sense... is a russian rolleta... Sincerely sorry for you...
- codecamper 6y agoex Googler does not have 75,000. Hmm... Google not the gig it used to be?
- raphaelj 6y agoSomeone should make a list of all the providers that support some kind of billing budget. It took me way more time than expected to find a CDN that has a budget limit for my personal project (I'm using Bunny CDN)
- franciscop 6y agoI've been looking at BunnyCDN and it looks pretty impressive. Is it as good as their landing pages look? Is it like a CDN+DNS so I can point my nameservers there and then configure the CNAMEs inside? (I'm fairly unfamiliar with DNS in general). I've been pretty happy with Cloudflare, but at some point I added my credit card (silly me) and now I live scared of a DDOS costing me a lot of money.
- minxomat 6y agoDNS is in the works, but at the moment no, you need an external DNS with apex CNAME support (e.g. cloudflare)
- dedene 6y agoI thought Cloudflare does not charge per Gb bandwidth but a fixed fee/month?
- manigandham 6y agoCloudflare has free bandwidth, including DDOS attacks.
- raphaelj 6y agoI've been super happy with BunnyCDN for my personal SaaS. Their UI is super nice, and it has been super easy to setup. Costs are really easy to predict, which is awesome for me.
- minxomat 6y agoI can also recommend Bunny. Their support is also top notch.
- mikelward 6y ago
- ionwake 6y agoMy problem with AWS billing was I asked sales/support to call me so I could check with them what the cost would be to test one of their higher end GPU servers. They said I would just pay for the minutes used that it wouldnt add up to anything. I tested the server for about 5 minutes and was charged a couple hundred dollars for "spinning up " the instance. Something the AWS sales guy assured me on the phone would not happen. I still dont know why I didnt appeal I guess I know better than to try.
- gonzo41 6y agoNext time don't ring, write and email. I've worked in financial sales. What's in the contract counts more than anything anyone ever says.
- wdb 6y agoBut is an email that says you only pay for the minutes counts as an contract?
- vnkatesh 6y agoCounts as legal paper trail at least.
- gonzo41 6y agoIt just creates a record. Doesn't have to be enforceable. but it means the person doing support becomes accountable for the things they've said. Chances are there's telephony recordings of your call. So you could escalate looking to hear those recordings. Emails' just easier in this regard.
- deleted 6y ago[deleted]
- jamesrr39 6y ago#1 reason why I have never, and never plan to (while it remains this way), use GCP/AWS etc while it would be on my personal bank card. Instead I use DigitalOcean where you have droplet limits that you can set, and the ability the pre-pay if you pay by PayPal, and never enter my bank card. If anyone from DO (or another provider) is reading this, any chance of pre-payment from bank cards? After reading enough of these articles, this could really swing a cloud provider choice for a small company. (Pre-paid gift vouchers would be cool as well, give someone $10 to spend for Christmas).
- rantwasp 6y agothat’s a non-argument. you can do the same with an EC2 instance. you know exactly what it’s gonna cost. it’s this fancy services with “elastic” pricing models that usually get you
- jamesrr39 6y agoSure, you know the price with renting one machine, and if what your doing is not a web app. But what about when you get way more network traffic than your app expected (I've seen HN submissions with exactly this)? And what if you built in some kind of scaling, automatically renting extra machines when you get traffic spikes? They have your card, you pay the $$$.
- corytheboyd 6y agoI don’t have any skin in the game here, but what if you don’t build auto-scaling, which keeps this comparison fair. How does pricing differ now?
- jamesrr39 6y agoThe difference would be that for a prepaid service, you can build it in, knowing that once it eats through enough $, that's it; there is no more credit to take, power off the service. Whereas the billing by credit card, it will keep on going, and you can end up with these huge bills to pay. But to answer your question (about AWS EC2 vs a DO droplet?), about other costs, you still have data transfer costs, which is currently: AWS (for US East: Ohio): Inbound: - first GB free - then $0.09/GB after (until 10TB, then you go to the next tier, paying a little less per GB. Outbound - Well, I couldn't figure it out. The page was too complicated for me! (I think it's $0.01/GB? From this text: "Data transferred “in” to and “out” from public or Elastic IPv4 address is charged at $0.01/GB in each direction") Source: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer DO: Inbound - free Outbound Free tier: depends on which droplet and how long you keep the droplet powered on for, but for the cheapest $5/month powered on all month, you get 1TB free. After free tier: $0.01/GB Source for Inbound: https://www.digitalocean.com/docs/billing/bandwidth/ https://www.digitalocean.com/docs/billing/bandwidth/ Source for outbound calculator: https://www.digitalocean.com/pricing/bandwidth/ https://www.digitalocean.com/pricing/bandwidth/ Anyone who understands it better than me (especially the AWS pricing), please feel free to comment, I'd genuinely be interested to understand it better; with the way it's documented, I don't really understand it very well.
- zenexer 6y ago> The ex-Googler reflected that he missed the possibility of pages that link back to each other, causing "infinite recursion." Although tangential to the billing issue, this is reckless. If you’re building a crawler of any kind, please, please, please prioritize ensuring this doesn’t happen so I don’t have to wake up at 3 AM. I run the infrastructure for a moderate-sized site with probably about a hundred million pages or so. We can handle the HN hug-of-death just fine. But poorly-made crawlers that recurse like this? They’re increasingly problematic. If your solution to fixing your crawler is “throw more concurrency at it and ignore the recursion,” and suddenly your requests start timing out, that’s a pretty damn strong hint that you’re ruining someone’s day. From my perspective, this will look like an attack. I’ll see thousands of IP addresses repeatedly requesting the same pages, usually with generic user agent headers. Which ones are actual attacks, and which are just poorly-made crawlers? Well, if you’ve got a generic user agent string that doesn’t link to a contact page, and you’re circumventing rate limiting by changing your IP address, and you had the bright idea to let your test code run overnight, I’m going to treat it as an attack. At 3 AM, I’m not inclined to differentiate between negligence and malice. This is happening more and more often, and I partially blame it on the ease of “accidentally” obtaining a ridiculous quantity of cloud resources. People deploy shoddy test code and go to bed. They turn it off in the morning when they see the bill. It’s become so prevalent that our company has come up with an internal term for these crawlers that spin up a new thread/container for every page: snowballing crawlers. Save a sysadmin: don’t snowball. Oh, and include a useful user agent header so we can contact you instead of your cloud provider.
- CyberDildonics 6y agoMaybe it would work to put a marker argument (like the IP address as base64) in the URL when there might be snowballing traffic so you can see if it comes back at you. That could be used to send a page with all the links taken out, or just be rate limited.
- zenexer 6y agoTricks like that don’t work with sites that are receiving a lot of traffic. Also, the exact solution you’ve described is a liability—IP addresses leak when people send each other links, and having unique URLs like that can cause issues with caching. Sure, we could store tokens in a database, but then you’ve just moved the bottleneck to the database. We do have various ways to combat these issues; like any website of sufficient size, we have pretty complex methods of detecting problematic traffic and assessing the risk of any given request or session. However, no solution is perfect, and with the number of broken crawlers we see, some will inevitably cause problems. To be clear, we can adjust our code and block them—that’s not an issue. The issue is that I have to wake up at 3 AM to do it, and even if it’s blocked, dealing with that traffic can be expensive. This guy got his $72k bill forgiven, but don’t expect the websites on the other end to be so lucky. (Yes, yes, ingress bandwidth is often free, but it’s never that simple. Scaling up? Bezos takes a cut. More database traffic? Pay the Bezos tax. Replication of enormous logs to other providers? Bezos hungry!) Negligence is negligence. If you get in a car and drive recklessly without proper training, even if you didn’t intend to hurt anyone, you’re not going to get a lot of sympathy when you mow down a pedestrian. Likewise, I have little sympathy for people who face enormous bills for abusing powerful tools. That’s not to say cloud providers don’t have billing problems. The delays are unacceptable, and the budgeting tools are often unintuitive or, as was likely the case here, outright inadequate. But in no universe was deploying code that spun up a container for every URL encountered a good idea. Should such a mistake result in a $72k bill? Eh, probably not. I doubt this person will make the same mistake again, even with the bill forgiven. Or maybe they’ll just blame Google and attempt the same thing on AWS.
- Viliam1234 6y agoA month or two ago some product was advertised (ahem, upvoted) on Hacker News that offered a free tier. It sounded interesting and I wanted to try, but the website didn't allow to create an account for the free tier specifically. Instead it was like: create a general account, and if your usage remains below X it is free, and as soon as it goes above X you agree to pay. With no way to cap usage, i.e. no way to cap spending. In other words, the only way to access the "free trial" is to give a blanket promise to pay unlimited amount of money if something goes wrong. There is no way I would agree to that, so I just closed the browser tab and forgot about the whole thing. That is, until this debate reminded me of it. For an online service, implementing the cap should be quite simple, so if it is not available, I am going to assume this is intentional.