5 ms·

Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare W

Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them. Wise was just down which is a pretty big one. Also odd how some websites were down this time that previously weren't down with the global outage in November

gowthamgts12 10mo ago
CDN was also affected for some customers. we were down with 500.
thinkindie 10mo ago
we were not affected too and we realised it was Cloudflare because Linear was down and they were mentioning an upstream service. Also Ecosia was affected, and I then realised they might be relying on Cloudflare too.
m_mueller 10mo ago
Maven Repository was down for me for a while, now it recovered.
reassess_blind 10mo ago
Yeah it's strange. My sites that are are proxied through Cloudflare remained up, but Supabase was taken offline so some backends were down. Either a regional PoP style issue, or a specific API or service had to be used to be affected.
archon810 10mo ago
Our locations excluded from Cloudflare WAF were up, but the rest was down. I think WAF took a dump.
kryptn 10mo ago
was interesting, some of our stuff failed, but some other stuff that used cloudflare indirectly didn't.
themly 10mo ago
CDN was definitely down also. We were widely impacted by it with 500's.
cryptonym 10mo ago
> Looking forward to the post mortem This is becoming a meme.
- meandmycode 10mo ago
  This has to be setting off some alarm bells internally, a well written postmortem on an occasional issue, great, but when your postmortem talks about learnings and improvements yet major outages keep happening, it becomes meaningless..
da_grift_shift 10mo ago
The excuse: >A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. >The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. >We will share more information as we have it today. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q
- madjam002 10mo ago
  It's quite an unfortunate coincidence that React has indirectly been the reason for two recent issues at Cloudflare haha
  - brobdingnagians 10mo ago
    Two's a coincidence, three's a pattern; I guess we will have to wait until next month to see if it becomes a pattern. Was there a particular aspect of the React Server Components that made it easy to have this problem appear? would it have been caught or avoided in another framework or language?
- GoblinSlayer 10mo ago
  Who sent an xml request?
gritzko 10mo ago
The entire Cloud/SaaS story had a lot of happy-path cost optimization. The particular glitch that triggered the domino effect may be irrelevant relative to the fact that the effect reproduces.

gowthamgts12 10mo ago

CDN was also affected for some customers. we were down with 500.

thinkindie 10mo ago

we were not affected too and we realised it was Cloudflare because Linear was down and they were mentioning an upstream service. Also Ecosia was affected, and I then realised they might be relying on Cloudflare too.

m_mueller 10mo ago

Maven Repository was down for me for a while, now it recovered.

reassess_blind 10mo ago

Yeah it's strange. My sites that are are proxied through Cloudflare remained up, but Supabase was taken offline so some backends were down. Either a regional PoP style issue, or a specific API or service had to be used to be affected.

archon810 10mo ago

Our locations excluded from Cloudflare WAF were up, but the rest was down. I think WAF took a dump.

kryptn 10mo ago

was interesting, some of our stuff failed, but some other stuff that used cloudflare indirectly didn't.

themly 10mo ago

CDN was definitely down also. We were widely impacted by it with 500's.

cryptonym 10mo ago

> Looking forward to the post mortem This is becoming a meme.

meandmycode 10mo ago
This has to be setting off some alarm bells internally, a well written postmortem on an occasional issue, great, but when your postmortem talks about learnings and improvements yet major outages keep happening, it becomes meaningless..

da_grift_shift 10mo ago

The excuse: >A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. >The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. >We will share more information as we have it today. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

madjam002 10mo ago
It's quite an unfortunate coincidence that React has indirectly been the reason for two recent issues at Cloudflare haha
- brobdingnagians 10mo ago
  Two's a coincidence, three's a pattern; I guess we will have to wait until next month to see if it becomes a pattern. Was there a particular aspect of the React Server Components that made it easy to have this problem appear? would it have been caught or avoided in another framework or language?
GoblinSlayer 10mo ago
Who sent an xml request?

gritzko 10mo ago

The entire Cloud/SaaS story had a lot of happy-path cost optimization. The particular glitch that triggered the domino effect may be irrelevant relative to the fact that the effect reproduces.