4 ms·

Well, you can never be sure that he didn't: https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage

by sakisv 10mo ago

itzjacki 10mo ago
Oh don't you worry. We are very much talking about the global outage as if he was the root cause. Like good colleagues :)
- rapnie 10mo ago
  Hmm, wait a minute.. maybe he was the cause! (no, kidding. just upping the pressure as a good peer :)
  - bryanrasmussen 10mo ago
    are we truly good if we don't start a class action suit against this hapless scapegoat?!
    - conorcleary 10mo ago
      Just join the one we've started over in this cubicle!
srmarm 10mo ago
> May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances. I'd love to know more about what those specific circumstances were!
CableNinja 10mo ago
Damn, imagine being the customer responsible for that, oof
- whstl 10mo ago
  I would be fine if it was my "fault", but I'm sure people in business would find a way to make me suffer. But on a personal level, this is like ordering something at a restaurant and the cook burning the kitchen because they forgot to take out your pizza out of the oven or something. I would be telling it to everyone over beers (but not my boss).
  - sakisv 10mo ago
    I would be tempted to put it on my CV :D
- WJW 10mo ago
  What do you imagine would be the result if you brought down cloudflare with a legitimate config update (ie not specifically crafted to trigger known bugs) while not even working for them? If I were the customer "responsible" for this outage, I'd just be annoyed that their software is apparently so fragile.
Bloomy22 10mo ago
I'm pretty sure I crashed Gmail using something weird in its filters. It was a few years ago. Every time I did something specific (I don't remember what), it would freeze and then display a 502 error for a while.
nevf1 10mo ago
It's also what was the cause of the Azure Front Doors global outage two weeks ago - https://aka.ms/air/YKYN-BWZ https://aka.ms/air/YKYN-BWZ "A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions."
- boruto 10mo ago
  This is actually pretty nice and amazing that they publish video format incident retrospectives.

Well, you can never be sure that he didn't: https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage

by sakisv 10mo ago

Well, you can never be sure that he didn't: https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage

itzjacki 10mo ago

Oh don't you worry. We are very much talking about the global outage as if he was the root cause. Like good colleagues :)

rapnie 10mo ago
Hmm, wait a minute.. maybe he was the cause! (no, kidding. just upping the pressure as a good peer :)
- bryanrasmussen 10mo ago
  are we truly good if we don't start a class action suit against this hapless scapegoat?!
  - conorcleary 10mo ago
    Just join the one we've started over in this cubicle!

srmarm 10mo ago

> May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances. I'd love to know more about what those specific circumstances were!

CableNinja 10mo ago

Damn, imagine being the customer responsible for that, oof

whstl 10mo ago
I would be fine if it was my "fault", but I'm sure people in business would find a way to make me suffer. But on a personal level, this is like ordering something at a restaurant and the cook burning the kitchen because they forgot to take out your pizza out of the oven or something. I would be telling it to everyone over beers (but not my boss).
- sakisv 10mo ago
  I would be tempted to put it on my CV :D
WJW 10mo ago
What do you imagine would be the result if you brought down cloudflare with a legitimate config update (ie not specifically crafted to trigger known bugs) while not even working for them? If I were the customer "responsible" for this outage, I'd just be annoyed that their software is apparently so fragile.

Bloomy22 10mo ago

I'm pretty sure I crashed Gmail using something weird in its filters. It was a few years ago. Every time I did something specific (I don't remember what), it would freeze and then display a 502 error for a while.

nevf1 10mo ago

It's also what was the cause of the Azure Front Doors global outage two weeks ago - https://aka.ms/air/YKYN-BWZ https://aka.ms/air/YKYN-BWZ "A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions."

boruto 10mo ago
This is actually pretty nice and amazing that they publish video format incident retrospectives.