6 ms·

The OpenAI outage lasted only 15 minutes and when it happened everyone started to use the other models which created super heavy load for them. This then cascad

The OpenAI outage lasted only 15 minutes and when it happened everyone started to use the other models which created super heavy load for them. This then cascaded into them all being down. Does this really need an explanation?

JoeAltmaier 12d ago
Not that uncommon. It's so easy to just put up the new thing and make the old thing the failure route. But the old thing nearly never had the bandwidth for today's traffic. A famous EBay outage some years ago was just such a scenario.
serf 12d ago
it wasn't timed like a cascade, and it relies on the premise that every single frontier provider is working so efficiently that they spend exactly what they need to provide for their exact market with perfect margins. I do not believe personally that 1) they can forecast their load that perfectly 2) they chose to remain that inflexible in a world where they are at each others' throats and a single meme can cause bursts of activity.
- jonas21 12d ago
  Or it could simply mean that they're operating at the limit of the capacity they were able to purchase and do not have headroom to handle load spikes. From what I understand, that's the situation Anthropic is in. And since Anthropic is now leasing a large portion of xAI's datacenter capacity, it's plausible that an Anthropic load spike could cause issues for xAI as well. Your comment makes it sound like they can just push a button and spin up more capacity -- but at this scale and in this GPU-constrained environment, that's not really how it works.
- hparadiz 12d ago
  9 AM PST / 12 Noon EST on weekday. All my co workers immediately went "oh codex is down lemme try Claude". Multiply that by millions. Easy to see how they all went down. The OpenAI downtime also coincided exactly with their tweets announcing GPT-6 and about an hour before they started to role it out. It was 1000% a cascade. I would bet on it.
gonzalohm 12d ago
That's not necessarily the reason. Less technical people are usually bound to just one provider

JoeAltmaier 12d ago

Not that uncommon. It's so easy to just put up the new thing and make the old thing the failure route. But the old thing nearly never had the bandwidth for today's traffic. A famous EBay outage some years ago was just such a scenario.

serf 12d ago

it wasn't timed like a cascade, and it relies on the premise that every single frontier provider is working so efficiently that they spend exactly what they need to provide for their exact market with perfect margins. I do not believe personally that 1) they can forecast their load that perfectly 2) they chose to remain that inflexible in a world where they are at each others' throats and a single meme can cause bursts of activity.

jonas21 12d ago
Or it could simply mean that they're operating at the limit of the capacity they were able to purchase and do not have headroom to handle load spikes. From what I understand, that's the situation Anthropic is in. And since Anthropic is now leasing a large portion of xAI's datacenter capacity, it's plausible that an Anthropic load spike could cause issues for xAI as well. Your comment makes it sound like they can just push a button and spin up more capacity -- but at this scale and in this GPU-constrained environment, that's not really how it works.
hparadiz 12d ago
9 AM PST / 12 Noon EST on weekday. All my co workers immediately went "oh codex is down lemme try Claude". Multiply that by millions. Easy to see how they all went down. The OpenAI downtime also coincided exactly with their tweets announcing GPT-6 and about an hour before they started to role it out. It was 1000% a cascade. I would bet on it.

gonzalohm 12d ago

That's not necessarily the reason. Less technical people are usually bound to just one provider