6 ms·

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several mi

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

philipwhiuk 10mo ago
So their parser broke again I guess. And no staged rollout I assume?
- tialaramex 10mo ago
  Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.
  - rhdunn 10mo ago
    Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem then there's no point in conducting them.
    - Xunjin 10mo ago
      Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?
      - tialaramex 10mo ago
        However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern 1. Accident happens 2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration 3. Regulator declined this recommendation, arguing it's too expensive to do X, or people already do X, or even (hilariously) both 4. Go to 1. Too often, what happens is that eventually 5. Extremely Famous Accident Happens, e.g. killing loved celebrity Space Cowboy 6. Investigators conclude Accident would not happen if people did X, remind regulator that they have previously recommended requiring X 7. Press finally reads dozens of previous reports and so News Story says: Regulator killed Space Cowboy! 8. Regulator decides actually they always meant to require X after all
        ethbr1 10mo ago
        As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.* On the one hand, you'd like to prevent the thing the regulation is seeking to prevent. On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing). "Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you downscope/target regulations to get better good ROI) *Yes, the easy pessimistic take is 'industry fights all regulation on cost grounds', but the fact that the argument is abused doesn't mean it doesn't have some underlying merit
        tialaramex 10mo ago
        I think conventionally the verb is "to steelman" with the intended contrast being to a strawman, an intentionally weak argument by analogy to how straw isn't strong but steel is. I understood what you meant by "strongman" but I think that "steelman" is better here. There is indeed a good reason regulators aren't just obliged to institute all recommendations - that would be a lot of new rules. The only accident report I remember reading with zero recommendations was a MAIB (Maritime accidents) report here which concluded that a crew member of a fishing boat has died at sea after their vessel capsized because they both they and the skipper (who survived) were on heroin, the rationale for not recommending anything was that heroin is already illegal, operating a fishing boat while on heroin is already illegal, and it's also obviously a bad idea, so, there's nothing to recommend. "Don't do that". Cost is rarely very persuasive to me, because it's very difficult to correctly estimate what it will actually cost to change something once you decided it's required - based on current reality where it is not. Mass production and clever cost reductions resulting from the normal commercial pressures tend to drive down costs when we require something but not before (and often not after we cease to require it either) It's also difficult to anticipate all benefits from a good change without trying it. Lobbyists against a regulation will often try hard not to imagine benefits after all they're fighting not to be regulated. But once it's in action, it may be obvious to everyone that this was just a better idea and absurd it wasn't always the case. Remember when you were allowed to smoke cigarettes on aeroplanes? That seems crazy, but at the time it was normal and I'm sure carriers insisted that not being allowed to do this would cost them money - and perhaps for a short while it did.
    - meindnoch 10mo ago
      >don't blame the people involved (unless they are acting maliciously) Or negligently.
      - jq-r 10mo ago
        That still shouldn't be a part of post mortem, more of a performance review item.
        tempaccount420 10mo ago
        They should be performantly removed.
    - __turbobrew__ 10mo ago
      The aviation industry regularly requires certifications, check rides, and re-qualifications when humans mess up. I have never seen anything like that in tech. Sometimes the solution is to not let certain people do certain things which are risky.
  - kypro 10mo ago
    > They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear. Or they could say, "we want to continue to prioritise speed of security rollouts over stability, and despite our best efforts, we do make mistakes, so sometimes we expect things will blow up". I guess it depends what you're optimising for... If the rollout speed of security patches is the priority then maybe increased downtime is a price worth paying (in their eyes anyway)... I don't agree with that, but at least it's an honest position to take. That said, if this was to address the React CVE then it was hardly a speedy patch anyway... You'd think they could have afforded to stagger the rollout over a few hours at least.
    - lima 10mo ago
      It's just poor risk management at this point. Making sure that a configuration change doesn't crash the production service shouldn't take more than a few seconds in a well-engineered system even if you're not doing staged rollout.
uyzstvqs 10mo ago
Ah yes, Cloudflare's worst enemy: The configuration change.
- hinkley 10mo ago
  On fridays, yes.
meindnoch 10mo ago
React (a frontend JS framework) can now bring down critical Internet infrastructure. I will repeat it because it's so surreal: React (a frontend JS framework) can now bring down critical Internet infrastructure.
- phplovesong 10mo ago
  [flagged]
  - mvandermeulen 10mo ago
    What was the AI slop part?
    - GaryBluto 10mo ago
      When something goes wrong, people are starting to immediately assume it's because of the thing they don't like.
- cryptonym 10mo ago
  That's Next.js, not React. Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.
  - philipwhiuk 10mo ago
    I think the "argument" is that it's a critical vuln so they can't "go slow". So now a vuln check for a component deployed on, being generous, 1% of servers causes an outage for 30% of the internet. The argument is dumb.
  - JimDabell 10mo ago
    > That's Next.js, not React. React seems to think that it was React: https://react.dev/blog/2025/12/03/critical-security-vulnerability-in-react-server-components https://react.dev/blog/2025/12/03/critical-security-vulnerab...
reassess_blind 10mo ago
I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.
- Traubenfuchs 10mo ago
  "Please don‘t block the rollout pipleline with a simple react security patch update."
- lima 10mo ago
  They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.
  - tehlike 10mo ago
    You can selectively bypass many roll out procedures in a properly designed system.
    - lima 10mo ago
      If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.
  - ethbr1 10mo ago
    For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?
aatd86 10mo ago
so it's react again in the end .. zzzzzzz
o_m 10mo ago
I wonder if this is the new normal? Weekly Cloudflare outages that breaks huge parts of the internet.
pepoluan 10mo ago
So. Another regex problem?

philipwhiuk 10mo ago

So their parser broke again I guess. And no staged rollout I assume?

tialaramex 10mo ago
Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.
- rhdunn 10mo ago
  Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem then there's no point in conducting them.
  - Xunjin 10mo ago
    Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?
    - tialaramex 10mo ago
      However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern 1. Accident happens 2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration 3. Regulator declined this recommendation, arguing it's too expensive to do X, or people already do X, or even (hilariously) both 4. Go to 1. Too often, what happens is that eventually 5. Extremely Famous Accident Happens, e.g. killing loved celebrity Space Cowboy 6. Investigators conclude Accident would not happen if people did X, remind regulator that they have previously recommended requiring X 7. Press finally reads dozens of previous reports and so News Story says: Regulator killed Space Cowboy! 8. Regulator decides actually they always meant to require X after all
      - ethbr1 10mo ago
        As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.* On the one hand, you'd like to prevent the thing the regulation is seeking to prevent. On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing). "Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you downscope/target regulations to get better good ROI) *Yes, the easy pessimistic take is 'industry fights all regulation on cost grounds', but the fact that the argument is abused doesn't mean it doesn't have some underlying merit
        tialaramex 10mo ago
        I think conventionally the verb is "to steelman" with the intended contrast being to a strawman, an intentionally weak argument by analogy to how straw isn't strong but steel is. I understood what you meant by "strongman" but I think that "steelman" is better here. There is indeed a good reason regulators aren't just obliged to institute all recommendations - that would be a lot of new rules. The only accident report I remember reading with zero recommendations was a MAIB (Maritime accidents) report here which concluded that a crew member of a fishing boat has died at sea after their vessel capsized because they both they and the skipper (who survived) were on heroin, the rationale for not recommending anything was that heroin is already illegal, operating a fishing boat while on heroin is already illegal, and it's also obviously a bad idea, so, there's nothing to recommend. "Don't do that". Cost is rarely very persuasive to me, because it's very difficult to correctly estimate what it will actually cost to change something once you decided it's required - based on current reality where it is not. Mass production and clever cost reductions resulting from the normal commercial pressures tend to drive down costs when we require something but not before (and often not after we cease to require it either) It's also difficult to anticipate all benefits from a good change without trying it. Lobbyists against a regulation will often try hard not to imagine benefits after all they're fighting not to be regulated. But once it's in action, it may be obvious to everyone that this was just a better idea and absurd it wasn't always the case. Remember when you were allowed to smoke cigarettes on aeroplanes? That seems crazy, but at the time it was normal and I'm sure carriers insisted that not being allowed to do this would cost them money - and perhaps for a short while it did.
  - meindnoch 10mo ago
    >don't blame the people involved (unless they are acting maliciously) Or negligently.
    - jq-r 10mo ago
      That still shouldn't be a part of post mortem, more of a performance review item.
      - tempaccount420 10mo ago
        They should be performantly removed.
  - __turbobrew__ 10mo ago
    The aviation industry regularly requires certifications, check rides, and re-qualifications when humans mess up. I have never seen anything like that in tech. Sometimes the solution is to not let certain people do certain things which are risky.
- kypro 10mo ago
  > They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear. Or they could say, "we want to continue to prioritise speed of security rollouts over stability, and despite our best efforts, we do make mistakes, so sometimes we expect things will blow up". I guess it depends what you're optimising for... If the rollout speed of security patches is the priority then maybe increased downtime is a price worth paying (in their eyes anyway)... I don't agree with that, but at least it's an honest position to take. That said, if this was to address the React CVE then it was hardly a speedy patch anyway... You'd think they could have afforded to stagger the rollout over a few hours at least.
  - lima 10mo ago
    It's just poor risk management at this point. Making sure that a configuration change doesn't crash the production service shouldn't take more than a few seconds in a well-engineered system even if you're not doing staged rollout.

uyzstvqs 10mo ago

Ah yes, Cloudflare's worst enemy: The configuration change.

hinkley 10mo ago
On fridays, yes.

meindnoch 10mo ago

React (a frontend JS framework) can now bring down critical Internet infrastructure. I will repeat it because it's so surreal: React (a frontend JS framework) can now bring down critical Internet infrastructure.

phplovesong 10mo ago
[flagged]
- mvandermeulen 10mo ago
  What was the AI slop part?
  - GaryBluto 10mo ago
    When something goes wrong, people are starting to immediately assume it's because of the thing they don't like.
cryptonym 10mo ago
That's Next.js, not React. Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.
- philipwhiuk 10mo ago
  I think the "argument" is that it's a critical vuln so they can't "go slow". So now a vuln check for a component deployed on, being generous, 1% of servers causes an outage for 30% of the internet. The argument is dumb.
- JimDabell 10mo ago
  > That's Next.js, not React. React seems to think that it was React: https://react.dev/blog/2025/12/03/critical-security-vulnerability-in-react-server-components https://react.dev/blog/2025/12/03/critical-security-vulnerab...

reassess_blind 10mo ago

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

Traubenfuchs 10mo ago
"Please don‘t block the rollout pipleline with a simple react security patch update."
lima 10mo ago
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.
- tehlike 10mo ago
  You can selectively bypass many roll out procedures in a properly designed system.
  - lima 10mo ago
    If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.
- ethbr1 10mo ago
  For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?

aatd86 10mo ago

so it's react again in the end .. zzzzzzz

o_m 10mo ago

I wonder if this is the new normal? Weekly Cloudflare outages that breaks huge parts of the internet.

pepoluan 10mo ago

So. Another regex problem?