8 ms·

Tell HN: CloudFront Is Having an Outage

53 points by klinskyc 3y ago

Sites hosted via Cloudfront (e.g https://app.circleci.com/) are having an outage and returning 421s and 500s currently

snacktaster 3y ago
All our HTTPS (ACM) sites are returning: > 421 ERROR > The request could not be satisfied. > The distribution does not match the certificate for which the HTTPS connection was established with. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
gregmfoster 3y ago
Seeing 421 on our AWS amplify splash page as well
ssabev 3y ago
Well, that explains wtf is happening. You guys seeing this on us-east-1 or across?
- romanhotsiy 3y ago
  us-east-1 only for us
- flangola7 3y ago
  Seeing it across 3 regions at minimum
- Aaronstotle 3y ago
  saw it across us-east-1 mainly
- neuronexmachina 3y ago
  As the saying goes: Friends don't let friends rely on us-east-1. (It was the very first region, and it shows)
  - hinkley 3y ago
    I will never understand why anyone in our infrastructure team thought for a moment that putting our backup datacenter in us-east-1. And why nobody else tried to get him fired. That's like buying a backup generator from a guy that wants to meet you in the Walmart parking lot.
    - booi 3y ago
      Whoa them fightin words. I bought a generator at a Walmart parking lot and it was fine. I only got stabbed twice. Generator didn’t actually work though
  - messe 3y ago
    Having worked at AWS: us-east-1 is probably the easiest region to deploy in (it's old, it's popular, every AWS service is available there), but might just be the worst region to rely on (it's old, it's popular, and every AWS service is available there—but only most of the time). I vividly remember when AWS Fault Injection Simulator first launched, posting a meme in the #aws-memes slack channel posting a drake meme with the first panel being "Using AWS Fault Injection Simulator", and the second "Deploying in us-east-1".
dave4420 3y ago
My employer’s sites are hosted via CloudFront and seem to be fine. The underlying buckets are in Dublin, in case that makes any difference.
akhayam 3y ago
AWS Status page doesn't show anything yet: https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
- NullInvictus 3y ago
  Short of something within a few hundred feet of Bezo's desk being actually on fire, AWS's health page never seems to actually report issues.
  - scrum-treats 3y ago
    Bezos is no longer the CEO. It's Andy Jassy. I agree, AWS (and Amazon Alexa + Ring) seem to be skirting and/or choosing active silence on that "Lambda" outage that took out us-east-1. Alexa had at the very least two full scale outages this year, and Ring was successfully hacked as well. Amazon's response: "looks like a Lambda... more later" then nothing, nothing, and "it must be a 3rd party skill." AWS health page: GREEN. AWS Post-event summaries: non-existent after 2021. This is quite troubling when you consider what AWS holds in its coffers, along with the number of customers AWS currently has. No transparency, not even strategic transparency, is a definite red flag.
    - thrashh 3y ago
      Between the companies I've worked with and the incidents I've been through, I'd say AWS is well-known to be "very slow" at updating their status reports when something goes down. Even had AWS liaisons tell us that things were definitely down even though the status page was completely green.
      - jwestbury 3y ago
        GMs/directors of AWS services are judged on how many "dashboard posts" they have (i.e., how many times their service posts to the status page). Incidentally, these same people are the ones in charge of approving posts to the status page for their service. Guess why this is an issue. Everyone on the ground at AWS knows this, and most AWS employees hate it. Source: Former AWS engineer for >5 years.
klinskyc 3y ago
It looks like they are back up now
akhayam 3y ago
Lots of users seem to be impacted for a short time period: https://isdown.app/integrations/aws/amazon-cloudfront https://isdown.app/integrations/aws/amazon-cloudfront
sgammon 3y ago
Seems stable now

53 points by klinskyc 3y ago

Sites hosted via Cloudfront (e.g https://app.circleci.com/) are having an outage and returning 421s and 500s currently

All our HTTPS (ACM) sites are returning: > 421 ERROR > The request could not be satisfied. > The distribution does not match the certificate for which the HTTPS connection was established with. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

gregmfoster 3y ago

Seeing 421 on our AWS amplify splash page as well

ssabev 3y ago

Well, that explains wtf is happening. You guys seeing this on us-east-1 or across?

romanhotsiy 3y ago
us-east-1 only for us
flangola7 3y ago
Seeing it across 3 regions at minimum
Aaronstotle 3y ago
saw it across us-east-1 mainly
neuronexmachina 3y ago
As the saying goes: Friends don't let friends rely on us-east-1. (It was the very first region, and it shows)
- hinkley 3y ago
  I will never understand why anyone in our infrastructure team thought for a moment that putting our backup datacenter in us-east-1. And why nobody else tried to get him fired. That's like buying a backup generator from a guy that wants to meet you in the Walmart parking lot.
  - booi 3y ago
    Whoa them fightin words. I bought a generator at a Walmart parking lot and it was fine. I only got stabbed twice. Generator didn’t actually work though
- messe 3y ago
  Having worked at AWS: us-east-1 is probably the easiest region to deploy in (it's old, it's popular, every AWS service is available there), but might just be the worst region to rely on (it's old, it's popular, and every AWS service is available there—but only most of the time). I vividly remember when AWS Fault Injection Simulator first launched, posting a meme in the #aws-memes slack channel posting a drake meme with the first panel being "Using AWS Fault Injection Simulator", and the second "Deploying in us-east-1".

dave4420 3y ago

My employer’s sites are hosted via CloudFront and seem to be fine. The underlying buckets are in Dublin, in case that makes any difference.

akhayam 3y ago

AWS Status page doesn't show anything yet: https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status

NullInvictus 3y ago
Short of something within a few hundred feet of Bezo's desk being actually on fire, AWS's health page never seems to actually report issues.
- scrum-treats 3y ago
  Bezos is no longer the CEO. It's Andy Jassy. I agree, AWS (and Amazon Alexa + Ring) seem to be skirting and/or choosing active silence on that "Lambda" outage that took out us-east-1. Alexa had at the very least two full scale outages this year, and Ring was successfully hacked as well. Amazon's response: "looks like a Lambda... more later" then nothing, nothing, and "it must be a 3rd party skill." AWS health page: GREEN. AWS Post-event summaries: non-existent after 2021. This is quite troubling when you consider what AWS holds in its coffers, along with the number of customers AWS currently has. No transparency, not even strategic transparency, is a definite red flag.
  - thrashh 3y ago
    Between the companies I've worked with and the incidents I've been through, I'd say AWS is well-known to be "very slow" at updating their status reports when something goes down. Even had AWS liaisons tell us that things were definitely down even though the status page was completely green.
    - jwestbury 3y ago
      GMs/directors of AWS services are judged on how many "dashboard posts" they have (i.e., how many times their service posts to the status page). Incidentally, these same people are the ones in charge of approving posts to the status page for their service. Guess why this is an issue. Everyone on the ground at AWS knows this, and most AWS employees hate it. Source: Former AWS engineer for >5 years.

klinskyc 3y ago

It looks like they are back up now

akhayam 3y ago

Lots of users seem to be impacted for a short time period: https://isdown.app/integrations/aws/amazon-cloudfront https://isdown.app/integrations/aws/amazon-cloudfront

sgammon 3y ago

Seems stable now