7 ms·
AWS deleted my 10-year account and all data without warning
- alper 1y ago> An AWS consultant who’d been covering my bills disappeared, citing losses from the FTX collapse. Yeah, well. The dog ate my homework.
- jbrw 1y agoMan, so glad I moved away from AWS.
- averrois 1y agoAWS has perfected the art of killing startups...
- tchbnl 1y ago>Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t. I put them in one provider Ah, but that's still one basket.
- Bratmon 1y agoDoes... Does the writer of this piece think the phrase only applies to literal baskets?
- jsiepkes 1y agoWild that people don't realize that these "separate" systems in AWS all share things like the same control plane.
- PartiallyTyped 1y agoThat is wrong in every way possible. Each service is isolated, has its own control plane, and often even split into multiple cells. Source: I worked there. https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html https://docs.aws.amazon.com/wellarchitected/latest/reducing-...
- akerl_ 1y agoYou’ve missed the point by leaning into your pedantry. The point is that from the perspective of vendor/platform risk (things like account closure, billing disputes, etc), all of AWS is the same basket, even if you use multiple regions and services.
- PartiallyTyped 1y agoI dismissed the claim that they share a single control panel across services because it is false. I dismissed that claim alone and talked about nothing else. The comment I replied to added nothing to the conversation except an unsubstantiated and plainly false remark. > Wild that people don't realize that these "separate" systems in AWS all share things like the same control plane. This comment does not say anything about all being in one basket. It only claims that all services run the same control plane which is so far from truth it must have been made from somebody utterly unaware of how big cloud teams operate. Seems that this community has lost the ability to reason and converse and would rather be outraged.
- jsiepkes 1y ago> must have been made from somebody utterly unaware of how big cloud teams operate. Nice, an attack on the person rather then the argument. Made from an anonymous Internet account. > Seems that this community has lost the ability to reason and converse and would rather be outraged. This might be a good time to reflect on your own online behavior.
- senderista 1y agoNot sure why you’re being downvoted, this is 100% correct from someone else who worked there.
- ImPostingOnHN 1y ago> Not sure why you’re being downvoted If you're not sure why someone is being downvoted, there's a good chance there is a misunderstanding somewhere, and a good way for you to understand is to wait for people to comment, and then read those comments. Alternatively (and in a more general sense), if you want to take a more active role in your learning, a good way to learn is to ask questions, and then read the answers.
- wewewedxfgdf 1y agoSo you restored from your off cloud backup, right? Tell me you have off cloud backups? If not, then I know its brutal, but AWS is responsible for their part in the disaster, and you are responsible for yours - which is not being able to recover at all.
- stefan_ 1y agoThis is really the longest, most self-aggrandizing sermon yet on "I stored my data on this other computer I don't own", complete with conspiracy theory and all. Store your data on your own disks, then at least you will blame yourself, not .. Java command line parsers?
- huksley 1y agoIf you are given only 5 days to comply with some request, that's how complicated your infra at AWS should be - so you can migrate to another provider in that time. Just use EC2 and basic primitives which are easy to migrate (ie S3, SES)
- S0y 1y ago>Just use EC2 and basic primitives which are easy to migrate (ie S3, SES) If that's your whole infra you really shouldn't be on AWS in the first place.
- adastra22 1y agoA bit ironic when that entire stack was invented at AWS.
- NewJazz 1y agoVMs were invented at AWS? Blob storage?
- adastra22 1y agoEC2? S3? Yes. That kind of "cloud" tech was invented at AWS. Nothing like the ec2 API existed before Amazon. It's what made AWS big. Maybe my graybeard is showing, but I remember when that kind of pre-containerization cloud provisioned VM resources was a radically new idea.
- cnst 1y agoFreeBSD jail predates Amazon Web Services, and so does SWsoft's Virtuozzo that was subsequently open-sourced as OpenVZ. For Amazon SES, DJB's qmail made it possible to send a ridiculous amount of subscription emails very efficiently, too. I've been using a VPS powered by Virtuozzo since like 2002 or 2003, how is EC2 all that different? Just the API? Per https://en.wikipedia.org/wiki/Virtuozzo_(company) https://en.wikipedia.org/wiki/Virtuozzo_(company), SWsoft was founded in 1997, and publicly released Virtuozzo in 2000. Per https://en.wikipedia.org/wiki/FreeBSD_jail https://en.wikipedia.org/wiki/FreeBSD_jail, FreeBSD jail was committed into FreeBSD in 1999 "after some period of production use by a hosting provider", and released with FreeBSD 4.0 in 2000. Per https://en.wikipedia.org/wiki/Qmail https://en.wikipedia.org/wiki/Qmail, qmail was released in 1998. Today, lots of AWS services are basically just re-packaged OSS packages.
- reactordev 1y agoMy only question is: where the hell was your support rep? Every org that works with AWS in any enterprise capacity has an enterprise agreement and an account rep. They should have been the one to guide you through this. If you were just yolo’ing it on your own identification without a contract, well, that’s that. You should have converted over to an enterprise agreement so they couldn’t fuck you over. And they will fuck you over.
- akerl_ 1y agoThey’ve really buried the lede here: this reads like the person paying for the account was not the post author, and AWS asked the payer (who from their perspective is the owner of the account) for information. That person wasn’t around to respond.
- floating-io 1y agoWhile that is certainly true, the idea that they can so rapidly decimate your data without the possibility to restore is still terrifying if that's your only copy of that data. They should have to hold it for at least 90 days. In my opinion, it should be more like six months to a year. In my mind, it's exactly equivalent to a storage space destroying your car five days after you miss a payment. They effectively stole and destroyed the data when they should have been required to return it to the actual owner. Of course, that's my opinion of how it should be. AFAIK, there is no real legal framework, and how it actually is is entirely up to your provider, which is one reason I never trust them.
- adastra22 1y agoIt sounds like it wasn’t OP’s data though, which is an important distinction.
- nerdponx 1y agoIt was his, along with his clients' data.
- anonymars 1y agoThat's not the impression I got > When AWS demanded this vanished payer validate himself, I pointed out that I already had my own Wise card on file—the same card I’d used to pay before the payer arrangement, kept active specifically in case the payer disconnected while I was traveling or offline
- akerl_ 1y agoThe post suggests that even if AWS’s policy had been to hold the data for a year, the same thing would have happened, because they deleted the data early due to operator error. Similarly, a physical storage company can totally make a mistake and accidentally destroy your stuff if they mix up their bookkeeping, and your remedy is generally either to reach an amicable settlement with them or sue them for your damages.
- floating-io 1y agoMy fear of this sort of thing happening is why I don't use github or gitlab.com for primary hosting of my source code; only mirrors. I do primary source control in house, and keep backups on top of that. It's also why nothing in my AWS account is "canonical storage". If I need, say, a database in AWS, it is live-mirrored to somewhere within my control, on hardware I own, even if that thing never sees any production traffic beyond the mirror itself. Plus backups. That way, if this ever happens, I can recover fairly easily. The backups protect me from my own mistakes, and the local canonical copies and backups protect me from theirs. Granted, it gets harder and more expensive with increasing scale, but it's a necessary expense if you care at all about business continuity issues. On a personal level, it's much cheaper though, especially these days.
- wewewedxfgdf 1y agoI once said to the CTO of the company I worked for "do we back up our source code"? He said, "no, it's on github". I said no more.
- Dylan16807 1y agoIf nobody has the repo checked out, what are the odds it's important?
- bryant 1y ago> If nobody has the repo checked out, what are the odds it's important? Oh boy. Tons of apps in maintenance mode run critical infrastructure and see few commits in a year.
- Dylan16807 1y agoAnd the people using it multiple times a year delete it afterwards?
- RealStickman_ 1y ago
- lightedman 1y agoLesson to learn: Never use Amazon or anyone else.
- 0x696C6961 1y agoOr just pay your bills.
- 3eb7988a1663 1y agoBilling mistakes happen all the time. Even (especially?) at multinationals with dedicated payment departments.
- saltysalt 1y agoThis is why I use a local NAS for offline backups.
- dboreham 1y agoThis is good but not really enough. You need another backup to cover the case where this backup is burned to a crisp when your house catches fire. And that second backup needs to be in another geographic region to guard against regional disasters such as meteor impact, super volcano eruption (possibly not a concern for you but it is for me), etc.
- ProofHouse 1y agoThis has happened to me too destroyed five years of my life no joke. Obviously it wasn’t just the set up and pipelines that took only 4 to 6 months but as a chain reaction to collapsed the entire startup. It was so unexpected. Lesson learned.
- roncesvalles 1y agoIf it takes more than a day to re-setup your cloud infra on a fresh account, consider investing time in going IaC-first.
- slashdave 1y agoPut your valuables into a safe deposit box. Or, buy some stocks. Some accident occurs. You don't pay your bill, address changes, etc. You have at least two entire years to contact the holder and claim your property. After that point, it is passed to the state as unclaimed property. You still have an opportunity to claim it. Digital data? Screw that! One mistake, everything deleted.
- akerl_ 1y agoPhysical storage providers make the same kind of mistakes all the time, accidentally emptying the wrong storage unit or trashing the wrong safety deposit box. You have potentially stronger civil remedies for recouping on those damages, but not always.
- yardie 1y agoCloud user here. If you would read your contracts, and it doesn't matter which cloud service you use, they all have the same section on Share Responsibility. https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode... You, the customer, are responsible for your data. AWS is only responsible for the infrastructure that it resides on.
- crote 1y agoThat's exactly the problem, isn't it? AWS is not responsible for what you do with your data. AWS, on its own, is not going to make backups for you so you can recover from a failed DB migration you did yourself. AWS cannot be held responsible for your admin fat-fingering and deleting the entire prod environment. However, AWS is responsible for the underlying infrastructure. The website you linked clearly shows "storage" as falling under AWS's responsibility: your virtual hard drives aren't supposed to just magically disappear! If they can just nuke your entire setup with an "Oops, sorry!", what's all the talk about redundancy and reliability supposed to be worth? At that point, how are they any different from Joe's Discount Basement Hosting?
- dmead 1y agoReddit deleted my 20 year old account with several warnings
- tguvot 1y agoHappened to me as well (permaban). Without any warning. You can request via help site data takeout under ccpa or gdpr. Took about two weeks but i got all my data
- dmead 1y agoYou still get access to log in and delete comments. I'll probably just use a tool to delete all my comments. Fuck em.
- tguvot 1y agoi can't login. i can change password but no login for me.
- dmead 1y agoProbably because I was a mod of a moderately popular local sub. I started losing my shit at people posting anti trans hysteria. Part of me wants to send spez a bill for 14 years of deleting dick pics and local kkk posting. And then there was COVID...
- spiralcoaster 1y agoThe amount of self-aggrandizing and lack of self awareness tells me this author is doing to do all of this again. This post could be summed up with "I should have had backups. Lesson learned", but instead they deflect to whining about how their local desktop is a mess and they NEED to store everything remotely to stay organized. They're going to dazzle you with all of their hardened bunker this, and multiple escape route that, not realizing all of their complex machinery is metaphorically running off of a machine with no battery backup. One power outage and POOF!
- gblargg 1y agoThe author doesn't grasp what putting all your eggs into one basket means: > Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t. I put them in one provider, with what should have been bulletproof redundancy: That's one basket. A single point of failure. "But it should have been impossible to fail!" Backups are to handle the "impossible" failure (in reality nothing is 100% reliable).
- Tohsig 1y agoFully agree. It's the same reason why you wouldn't put a repo on Github and then mirror that repo to Github. At a minimum any good mirror would be on a different provider, and ideally (if we get real picky) on a completely different cloud service.
- davidhyde 1y agoThis one time, traveling through Asia, a simple merchant transaction triggered a fraud alarm on my card. The default for my bank at the time was to cancel my card automatically. This was before the days where cards could become unblocked. I had to travel to another city to pick up a new card in 10 working ways. This was a Mastercard credit card. I thought I was smart traveling with both a mastercard and a Visa card. Well, the Visa card was automatically cancelled too. Due to the same event. No cards for me to use to get to that city and I had to resort to a dodgy western union transfer to move forward. Also, try booking a flight with cash, it’s not fun. My point is that the basket that eggs are put in is not always clear in hindsight. I wasn’t even aware that Mastercard and visa shared fraud alerts and that they were automatically linked. The author’s article is not about backups, it’s about accountability.
- hyperman1 1y agoThe java command line theory seems strange to me. Java has no standard command line parser in the JDK (Standard library). Apache commons cli probably comes closest to a standard, and they support --gnu-style-long-options just like everybody else. The jvm(runtime) has some long non-POSIX options, but that's not very relevant here.
- seuros 1y agoYeah, it's fishy. I never claimed the Java theory is confirmed, just that it’s what an insider told me after the fact. They said a dry-run flag was passed in the --gnu-style form, but the internal tool expected -dry, and since Java has no native CLI parser, it just ignored it and ran for real. Supposedly the dev team was used to Python-style CLIs, and this got through without proper testing.
- foundry27 1y agoEditorial comment: It’s a bit weird to see AI-written (at least partially; you can see the usual em-dashes, it’s-not-X-it’s-Y) blog posts like this detract from an author’s true writing style, which in this case I found significantly more pleasant to read. Read his first ever post, and compare it to this one and many of the other recent posts: https://www.seuros.com/blog/noflylist-how-noflylist-got-cleared-for-production-3cgf/ https://www.seuros.com/blog/noflylist-how-noflylist-got-clea... I’m not much of a conspiracy theorist, but I could imagine a blog post almost identical to this one being generated in response to a prompt like “write a first-person narrative about: a cloud provider abruptly deleting a decade-old account and all associated data without warning. Include a plot twist”. I literally cannot tell if this story is something that really happened or not. It scares me a little, because if this was a real problem and I was in the author’s shoes, I would want people to believe me.
- ageitgey 1y agoI didn't get an AI vibe from this post. Grammar checkers add em dashes, too. If anything, an AI tool would have written a shorter, less rambling post.
- foundry27 1y agoIt’s easy to be fooled, myself included it seems :) For context, here’s a handful of the ChatGPT cues I see. - “wasn’t just my backup—it was my clean room for open‑source development” - “wasn’t standard AWS incompetence; this was something else entirely” - “you’re not being targeted; you’re being algorithmically categorized” - “isn’t a system failure; the architecture and promises are sound” - “This isn’t just about my account. It’s about what happens when […]” - “This wasn’t my production infrastructure […] it was my launch pad for updating other infrastructure” - “The cloud isn’t your friend. It’s a business” I counted about THIRTY em-dashes, which any frequent generative AI user would understand to be a major tell. It’s got an average word count in each sentence of around ~11 (try to write with only 11 words in each sentence, and you’ll see why this is silly), and much of the article consists of brief, punchy sentences separated by periods or question marks, which is the classic ChatGPT prose style. For crying out loud, it even has a table with quippy one-word cell contents at the end of the article like what ChatGPT generates 9/10 times when asked for a comparison of two things. It’s just disappointing. The author is undermining his own credibility for what would otherwise be a very real problem, and again, his real writing style when you read his actual written work is great.
- 486sx33 1y ago[dead]
- lowbloodsugar 1y ago>For years, I’ve watched developers on Reddit and Facebook desperately seeking US or EU billing addresses, willing to pay $100+ premiums to avoid MENA region assignment. >When I asked why, a colleague warned me: “AWS MENA operates differently. They can terminate you randomly.” Huh.
- cnst 1y agoMENA = Middle East / North Africa. (I did have to look it up.)
- infinitedata 1y agoIf you are not in the US don’t use AWS. There is some shady stuff happening here, also Wise is not trusted by anyone
- palepa 1y ago[dead]
- sitzkrieg 1y agoi hosted a single static webpage critical of israel on cloudfront in aws govcloud and the account was gone within a month
- SergeAx 1y agoLessons learned: 1) Keep my backups with a different infrastructure provider 2) Never allow third parties to pay for critical services 3) The moment I am asked for "verification," the emergency plan kicks in
- bitdeep 1y agoTL;DR; REKT by a extra - because java just don’t stop on invalid commands, ex: —dry is just ignored.
- garrickmurphy 1y ago[dead]
- bengray 1y ago[dead]
- Kate5477 1y ago[dead]