5 ms·

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products.

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products. It was not related to the Astra launch. We don't comment on other providers' outages.

dboreham 13d ago
Straight from the turkey's beak.
swader999 13d ago
Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.
- eli 13d ago
  It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
- theaxeonthanksg 13d ago
  [dead]
- OfficialTurkey 13d ago
  Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
  - swader999 13d ago
    Hopefully you get a decked out command center too.
- stillpointlab 13d ago
  Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
- alanfranz 12d ago
  It’s a standard term for a role during an incident.
theaxeonthanksg 13d ago
[dead]
deleted 13d ago
[deleted]
wnmurphy 13d ago
Curious then as to why the incident coincided with similar issues with Anthropic and xAI.
- wnmurphy 13d ago
  Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
  - reilly3000 13d ago
    Thank you. This should be pinned or something.
    - wnmurphy 13d ago
      xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.
ncb_asj 13d ago
Shouldn't Astra be the Incident Commander?
- sshine 13d ago
  Secretly, they are.
grist_4_da_chil 13d ago
[dead]
honeybadger1 12d ago
At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?

dboreham 13d ago

Straight from the turkey's beak.

swader999 13d ago

Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.

eli 13d ago
It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
theaxeonthanksg 13d ago
[dead]
OfficialTurkey 13d ago
Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
- swader999 13d ago
  Hopefully you get a decked out command center too.
stillpointlab 13d ago
Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
alanfranz 12d ago
It’s a standard term for a role during an incident.

theaxeonthanksg 13d ago

[dead]

deleted 13d ago

[deleted]

wnmurphy 13d ago

Curious then as to why the incident coincided with similar issues with Anthropic and xAI.

wnmurphy 13d ago
Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
- reilly3000 13d ago
  Thank you. This should be pinned or something.
  - wnmurphy 13d ago
    xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.

ncb_asj 13d ago

Shouldn't Astra be the Incident Commander?

sshine 13d ago
Secretly, they are.

grist_4_da_chil 13d ago

[dead]

honeybadger1 12d ago

At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?