6 ms·

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

by SerCe 10mo ago

bayesnet 10mo ago
And a well-written one at that. Compared to the AWS port-mortem this could be literature.
- hn-idiots 10mo ago
  [flagged]
  - tclancy 10mo ago
    I feel like your username really brings something extra to the party. Now go home.
    - eastdakota 10mo ago
      Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.
      - yahoozoo 10mo ago
        Based CEO
        hn-idiots 10mo ago
        [flagged]
- philipwhiuk 10mo ago
  Except it fails to document anything about the actions they made to Warp in London during the resolution.
  - eastdakota 10mo ago
    There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
    - philipwhiuk 10mo ago
      Thanks!
yen223 10mo ago
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
- thesh4d0w 10mo ago
  The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.
  - NewJazz 10mo ago
    I'm sure he wanted traffic to flow for those few hours but it didn't :p
- tom1337 10mo ago
  I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0] [0] https://news.ycombinator.com/item?id=45588305 https://news.ycombinator.com/item?id=45588305
- madeofpalk 10mo ago
  From what I've observed, it depends on whether you're an "engineering company" or not.
- eastdakota 10mo ago
  Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
SilverElfin 10mo ago
[flagged]
- mh- 10mo ago
  Why give this sort of content more visibility/reach? I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.
  - SilverElfin 10mo ago
    Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit. The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.
    - mh- 10mo ago
      Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.
      - DoctorOW 10mo ago
        I'm not the person you were replying to, but there is a rule I often see about not directly replying/quote tweeting because "engagement" appears to boost support for the ideas expressed. The recommendation then, would be to screenshot it (often with the username removed) and link to that.
andrewinardeer 10mo ago
Plenty are capable. Most don't bother.
BrtByte 10mo ago
It's not just a PR-friendly summary either... they included real technical detail, timestamps, even code snippets
eastdakota 10mo ago
* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.
- ignoramous 10mo ago
  That's all well & good, but I'm curious... > Spent some time after we got things under control talking to customers. Then went home. What did sama / Fidji say? ;) Turnstile couldn't have been worth that.

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

by SerCe 10mo ago

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

bayesnet 10mo ago

And a well-written one at that. Compared to the AWS port-mortem this could be literature.

hn-idiots 10mo ago
[flagged]
- tclancy 10mo ago
  I feel like your username really brings something extra to the party. Now go home.
  - eastdakota 10mo ago
    Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.
    - yahoozoo 10mo ago
      Based CEO
      - hn-idiots 10mo ago
        [flagged]
philipwhiuk 10mo ago
Except it fails to document anything about the actions they made to Warp in London during the resolution.
- eastdakota 10mo ago
  There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
  - philipwhiuk 10mo ago
    Thanks!

yen223 10mo ago

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

thesh4d0w 10mo ago
The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.
- NewJazz 10mo ago
  I'm sure he wanted traffic to flow for those few hours but it didn't :p
tom1337 10mo ago
I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0] [0] https://news.ycombinator.com/item?id=45588305 https://news.ycombinator.com/item?id=45588305
madeofpalk 10mo ago
From what I've observed, it depends on whether you're an "engineering company" or not.
eastdakota 10mo ago
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.

SilverElfin 10mo ago

[flagged]

mh- 10mo ago
Why give this sort of content more visibility/reach? I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.
- SilverElfin 10mo ago
  Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit. The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.
  - mh- 10mo ago
    Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.
    - DoctorOW 10mo ago
      I'm not the person you were replying to, but there is a rule I often see about not directly replying/quote tweeting because "engagement" appears to boost support for the ideas expressed. The recommendation then, would be to screenshot it (often with the username removed) and link to that.

andrewinardeer 10mo ago

Plenty are capable. Most don't bother.

BrtByte 10mo ago

It's not just a PR-friendly summary either... they included real technical detail, timestamps, even code snippets

eastdakota 10mo ago

* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.

ignoramous 10mo ago
That's all well & good, but I'm curious... > Spent some time after we got things under control talking to customers. Then went home. What did sama / Fidji say? ;) Turnstile couldn't have been worth that.