7 ms·

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in t

by bgentry 7mo ago

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.

Imustaskforhelp 7mo ago
> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.
- tokyobreakfast 7mo ago
  > this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.
  - flymasterv 7mo ago
    Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.
    - ok_dad 7mo ago
      What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
      - flymasterv 7mo ago
        Their lack of multiple AZ’s isn’t the guy making 30k a year’s problem.
      - stonogo 7mo ago
        In that case, the hero was the person who avoided relying on a single AZ when they deployed to cloud.
        Imustaskforhelp 7mo ago
        100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted? attacking datacenters within a particular region so that service would have a hard time. I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this. That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.
      - Imustaskforhelp 7mo ago
        > What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time. Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.
  - thatguy0900 7mo ago
    I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.
    - tokyobreakfast 7mo ago
      on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely
- pwarner 7mo ago
  But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
  - Imustaskforhelp 7mo ago
    > But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target? My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now). > but in the fog of war maybe the data center wasn't the target? We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?
- subscribed 7mo ago
  I wouldn't risk it. Both Americans and Israelis are known for double taps. Surely Iran can adapt their tactics too.
- zozbot234 7mo ago
  > this would require human intervention Amazon has self-propelled robots that handle their logistics and fulfillment, don't they? Send in the robots.
jiggawatts 7mo ago
This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc… Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances. A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for. “Oh, that’s in case there’s a fire!” “What does it do?” “It cuts… the power… uhh… for the safety of the fire department.” “So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?” “… yes.” “Not 100% then, is it?”
- vntok 7mo ago
  Should have pushed it.
  - dminvs 7mo ago
    at The Planet in Dallas c. 2002 the EPO button was exposed with no cover, and in very very close proximity to the "Exit" button for the doors... one day, a colo customer hit the wrong button on the way out, and uhh, there was an outage
    - vntok 7mo ago
      How did you find out about the outage?
    - throwaway81523 7mo ago
      Obligatory far side cartoon: https://static1.srcdn.com/wordpress/wp-content/uploads/2023/09/far-side-wings-fall-off.jpg https://static1.srcdn.com/wordpress/wp-content/uploads/2023/...
  - skywhopper 7mo ago
    My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days! That was when we installed the Big Clear Button Cover.
dehrmann 7mo ago
> impacted by objects Well that's a way of phrasing it. I assume these were missiles.
- sebastiennight 7mo ago
  Most likely, those were debris from the interception of missiles flying overhead and being destroyed on their way to a military target. AFAIK, there have been no confirmed signs of civilian sites being targeted directly, and it would also be unlikely that actual missiles would cause so little damage that you could patch your datacenter up and get it ready to go within hours.
  - breppp 7mo ago
    That's incorrect, there have been multiple hotels being attacked and recently oil facilities in SA
    - sebastiennight 7mo ago
      Can you please share sources?
      - breppp 7mo ago
        refineries, gas fields - https://www.reuters.com/business/energy/saudi-aramco-shuts-ras-tanura-refinery-after-drone-strike-source-says-2026-03-02/ https://www.reuters.com/business/energy/saudi-aramco-shuts-r... dubai airport, residential buildings, hotels, ports - https://www.reuters.com/world/middle-east/several-loud-blasts-heard-over-dubai-doha-second-day-witnesses-say-2026-03-01/ https://www.reuters.com/world/middle-east/several-loud-blast... https://www.lemonde.fr/en/international/article/2026/03/01/iran-attacks-luxury-hotels-and-airports-in-dubai_6750972_4.html https://www.lemonde.fr/en/international/article/2026/03/01/i... Bahrain - https://bh.usembassy.gov/security-alert-update-5-u-s-embassy-manama-bahrain-u-s-government-personnel-moved-from-hotels-march-1-2026/ https://bh.usembassy.gov/security-alert-update-5-u-s-embassy... You can google for other targets in Qatar, Oman, Jordan, Iraq, Cyprus and Kuwait
deleted 7mo ago
[deleted]
xg15 7mo ago
Impact assessment: yes.

by bgentry 7mo ago

Imustaskforhelp 7mo ago

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.

tokyobreakfast 7mo ago
> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.
- flymasterv 7mo ago
  Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.
  - ok_dad 7mo ago
    What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
    - flymasterv 7mo ago
      Their lack of multiple AZ’s isn’t the guy making 30k a year’s problem.
    - stonogo 7mo ago
      In that case, the hero was the person who avoided relying on a single AZ when they deployed to cloud.
      - Imustaskforhelp 7mo ago
        100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted? attacking datacenters within a particular region so that service would have a hard time. I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this. That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.
    - Imustaskforhelp 7mo ago
      > What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time. Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.
- thatguy0900 7mo ago
  I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.
  - tokyobreakfast 7mo ago
    on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely
pwarner 7mo ago
But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
- Imustaskforhelp 7mo ago
  > But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target? My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now). > but in the fog of war maybe the data center wasn't the target? We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?
subscribed 7mo ago
I wouldn't risk it. Both Americans and Israelis are known for double taps. Surely Iran can adapt their tactics too.
zozbot234 7mo ago
> this would require human intervention Amazon has self-propelled robots that handle their logistics and fulfillment, don't they? Send in the robots.

jiggawatts 7mo ago

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc… Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances. A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for. “Oh, that’s in case there’s a fire!” “What does it do?” “It cuts… the power… uhh… for the safety of the fire department.” “So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?” “… yes.” “Not 100% then, is it?”

vntok 7mo ago
Should have pushed it.
- dminvs 7mo ago
  at The Planet in Dallas c. 2002 the EPO button was exposed with no cover, and in very very close proximity to the "Exit" button for the doors... one day, a colo customer hit the wrong button on the way out, and uhh, there was an outage
  - vntok 7mo ago
    How did you find out about the outage?
  - throwaway81523 7mo ago
    Obligatory far side cartoon: https://static1.srcdn.com/wordpress/wp-content/uploads/2023/09/far-side-wings-fall-off.jpg https://static1.srcdn.com/wordpress/wp-content/uploads/2023/...
- skywhopper 7mo ago
  My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days! That was when we installed the Big Clear Button Cover.

dehrmann 7mo ago

> impacted by objects Well that's a way of phrasing it. I assume these were missiles.

sebastiennight 7mo ago
Most likely, those were debris from the interception of missiles flying overhead and being destroyed on their way to a military target. AFAIK, there have been no confirmed signs of civilian sites being targeted directly, and it would also be unlikely that actual missiles would cause so little damage that you could patch your datacenter up and get it ready to go within hours.
- breppp 7mo ago
  That's incorrect, there have been multiple hotels being attacked and recently oil facilities in SA
  - sebastiennight 7mo ago
    Can you please share sources?
    - breppp 7mo ago
      refineries, gas fields - https://www.reuters.com/business/energy/saudi-aramco-shuts-ras-tanura-refinery-after-drone-strike-source-says-2026-03-02/ https://www.reuters.com/business/energy/saudi-aramco-shuts-r... dubai airport, residential buildings, hotels, ports - https://www.reuters.com/world/middle-east/several-loud-blasts-heard-over-dubai-doha-second-day-witnesses-say-2026-03-01/ https://www.reuters.com/world/middle-east/several-loud-blast... https://www.lemonde.fr/en/international/article/2026/03/01/iran-attacks-luxury-hotels-and-airports-in-dubai_6750972_4.html https://www.lemonde.fr/en/international/article/2026/03/01/i... Bahrain - https://bh.usembassy.gov/security-alert-update-5-u-s-embassy-manama-bahrain-u-s-government-personnel-moved-from-hotels-march-1-2026/ https://bh.usembassy.gov/security-alert-update-5-u-s-embassy... You can google for other targets in Qatar, Oman, Jordan, Iraq, Cyprus and Kuwait

deleted 7mo ago

[deleted]

xg15 7mo ago

Impact assessment: yes.