5 ms·
AWS S3 Having Problems Again? (Monday 12PM Pacific)
We're seeing similar problems to last night -- lots of 503s from S3. Anyone else?
- Stovoy 11y agoYes, seeing the same thing.
- needcaffeine 11y agoJust started again in us-east-1.
- kernel_sanders 11y agoSame for us
- mej10 11y agoYep!
- dkuebric 11y agoYep, same. Lots of latency too--here's what we're seeing: http://kuebri.ch/bucket/s3_latency_081015.png http://kuebri.ch/bucket/s3_latency_081015.png
- astroteller 11y ago>This webpage is not available Heh.
- dkuebric 11y agoI'd wanted to put this on imgur instead of my micro instance, but I'm guessing imgur is backed by S3 because their upload was broken :)
- zeeta6 11y agoWhat tool is that?
- mstkrft 11y agoSame here :(
- ninjakeyboard 11y agobad day.
- azundo 11y agoWe're seeing similar symptoms here as well.
- AnonNo15 11y agoSeeing it too. 15:00 EST
- toomuchtodo 11y ago503s galore. Is anyone seeing issues in other S3 regions?
- RyanGWU82 11y agoLooks like it got better around 12:20 PM, about 10 minutes after the incident started. We haven't seen any problems in the last few minutes.
- RyanGWU82 11y ago... and errors started up again at 1:00 PM.
- andrebrov 11y agoWe had problems with AWS ML tonight
- atopuzov 11y agoAmazon yet again lying to it's customers about the status of the service is the only real issue I see here>. Services fail, it's a fact of life but at least admit it's broken and that the issue is being fixed instead of blatantly lying and saying minor disruptions.
- eric_h 11y agohttp://status.aws.amazon.com http://status.aws.amazon.com appears to indicate that they did have problems and have now resolved them.
- autotune 11y agoWhat happened to that 99.99% availability? Either way this just got posted at reddit.com/r/sysadmin which might be useful to some for tracking error rate: https://pulse.turbobytes.com/results/55c8751aecbe400bf80005f2/ https://pulse.turbobytes.com/results/55c8751aecbe400bf80005f...
- ceejayoz 11y agoTheir SLA guarantees 99.9% on a monthly basis. The 99.99% mentioned on the product page isn't guaranteed at all. As for what happened, my money is on this: https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3-introduces-new-usability-enhancements/ https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3... > You can now increase your Amazon S3 bucket limit per AWS account... Amazon S3 now supports read-after-write consistency for new objects added to Amazon S3 in US Standard region. The 100 bucket limit used to be an absolute, unchangeable hard limit - rare for AWS and thus likely something deep in the architecture from S3 being one of their first services - so I suspect the lifting of that limit involved some fairly major changes to the backend.
- StabbyCutyou 11y agoThey actually would let you increase that, but only up to a certain point and only if you specifically requested it. I don't see them mention the absolute ceiling being lifted, so that is probably still in place somewhere. I'd wager it's more likely that read-after-write change.
- KenCochrane 11y agoFrom Amazon: "Hello, We have just become aware of EC2 network connectivity issues in the US-EAST-1 region. The impact of this issue is loss of network connectivity to EC2 instances in US-EAST-1. The AWS support and engineering teams are actively working on bringing closure to this issue. I will share additional information as soon as I learn more about this issue."
- Negitivefrags 11y agoI'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I feel like AWS has way too many moving parts to be stable. It's very tempting for them to reuse bits of infrastructure everywhere which increases the chances that if something goes wrong somewhere it will break your stuff. So for example, hosting instance images on S3 means that when S3 has issues, now EC2 has issues.
- ceejayoz 11y ago> I'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I haven't rented servers in about two years, but yes. My old MediaTemple dedicated servers and the pair we had colocated at a local telco's large datacenter experienced occasional network outages and whatnot just like AWS does. IMO, AWS made load balancing and fault-tolerant setups much more accessible to small businesses. At the time of our switch, getting a load balancer at our colo was quite pricey whereas AWS charges $15ish a month for it. Getting a three-tier setup on AWS was easy, whereas at the colo we'd have to pay for at least 6U of space even for relatively small amounts of traffic.
- cbsmith 11y ago> I feel like AWS has way too many moving parts to be stable. Well, everyone has different definitions of stable. > I'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I don't think it is inflammatory, but perhaps you have an idealized notion of how well teams manage uptime, and more importantly, failures, on average. Sure, a great team will do fantastic, but an average team will... not. That said, it seems likely that AWS is hitting some kind of a rough patch right now. There are, however, other cloud services whose uptime records for the last year would be the envy of your typical company's internal hosting services, and I'd expect AWS to return to that fold shortly.
- spydum 11y agoit's not all about uptime. It's about upfront capital costs. It also highly depends on the service and solution you build. I know in traditional EC2 space, over the long term you absolute can self-host for far less money, but you are talking about 5-year TCO. Things like S3 bring another advantage: a globally distributed data store is HARD to build and supply on your own dime. I would argue things like that are much harder to beat self-hosted.
- arturhoo 11y agoWe had problems while connecting to S3 standard US region from us-east-1 at 19UTC but it was solved 20 minutes later. edit: seeing connectivity issues again at 19h50UTC
- ronreiter 11y agoYes, same here.
- thspimpolds 11y ago"12:28 PM PDT Between 12:03 PM to 12:19 PM PDT we experienced elevated errors for requests made to Amazon S3 in the US-STANDARD Region. The issue has been resolved and the service is operating normally" Our AWS TAM called us. I don't think he wanted the nasty call I gave him at 4:30am
- edgan 11y agoThe us-east-1 region gets treated differently than all other regions by AWS. Part of the reason it gets treated differently it is the default, and hence the most popular. It also doesn't help that it is on the east coast, and experiences more weather. For the above reasons, and that I work in the SF bay area, I put everything in us-west-2. us-west-2 sometimes has it's own issues, but nothing quite at the level of us-east-1.
- numbsafari 11y agoWe're all just huddling in us-east-1 waiting for the day us-west-2 falls into the ocean...
- AlphaSite 11y agoEast1 is absurdly huge, isn't it? I saw statements to the effect that its nearly 60% of AWS's capacity.
- toomuchtodo 11y agous-east-1 data is replicated to another region on the west coast behind the scenes. Its supposed to fail over automatically. http://shlomoswidler.com/2009/12/read-after-write-consistency-in-amazon.html http://shlomoswidler.com/2009/12/read-after-write-consistenc... > Aha! I had forgotten about the way Amazon defines its S3 regions. US-Standard has servers on both the east and west coasts (remember, this is S3 not EC2) in the same logical “region”. The engineering challenges in providing read-after-write consistency in a smaller geographical area are greatly magnified when that area is expanded. The fundamental physical limitation is the speed of light, which takes at least 16 milliseconds to cross the US coast-to-coast (that’s in a vacuum – it takes at least four times as long over the internet due to the latency introduced by routers and switches along the way).
- mdellabitta 11y agoIIRC, the AWS console itself is hosted out of us-east-1. Which means you're always somewhat exposed to whatever failure modes it has.
- 11y ago
- bhz 11y agoWe saw a short burst of 503s a short while ago, but we have not seen any since. Hopefully we do not see any more though. Also, for the record, S3 has been very stable for us otherwise. We have been rather happy with AWS overall.
- onyxraven 11y agoSame, though just as I write this we see another spike of errors.
- bhz 11y agoOk, no 503s but just got a very small burst of 500s, "com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 500, AWS Service: Amazon S3, AWS Request ID: -redacted-, AWS Error Code: InternalError, AWS Error Message: We encountered an internal error. Please try again., S3 Extended Request ID: -redacted-" :/
- matwood 11y agoWe have also seen a higher rate of port scans/attacks today. I wonder if it is AWS wide causing system overload issues.
- wbharding 11y agoIndeed. As I write this we're in the midst of our third S3 outage of the day. The past two were eventually documented on the AWS Service Dashboard. The latest one has not yet received its tiny status icon to indicate an outage. It's one thing that S3 keeps going down today; we run our own server cluster and I accept that 100% uptime isn't possible. But it's aggravating that they can't at least figure out how to give timely updates on their dashboard when something is broken. We inevitably learn of S3 outages through our internal error reporting systems before AWS posts it to their status page. When they do finally post, it is usually a tiny "information" icon, even when reporting a problem that makes the service unusable. The laggy, misleading nature of their status page gives the impression they must be tying bonuses to the status icons. Can't fathom why else they would be so inept when it comes to keeping us updated when something is wrong. Surely they have sufficient internal monitoring to pick up on these outages long before they update their customers.
- ceejayoz 11y agoIt shouldn't be, but I've found https://twitter.com/ylastic https://twitter.com/ylastic by far the best way to find early info on AWS issues.
- kordless 11y agoInteresting this article was bumped from the front page so quickly. Makes you wonder...