5 ms·
"Heads should probably roll here..." Google has a blameless post-mortem process (see https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-boo
by Guidii 5y ago
"Heads should probably roll here..."
Google has a blameless post-mortem process (see https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/) which essentially boils down to:
0) We want to fix the situation. By the time the post-mortem happens, that should already be done.
1) We want to make sure this never happens again. In order to do that we need to understand as much as we can about what happened.
2) We want everyone to share everything they know about the event. If the participants worry about fallout, they will be compelled to cover up.
3) Most failures happen at a number of points, so you need to dig in deeply if you want to get full value out of the "event".
4) Good post-mortems lead to process fixes to reduce (or eliminate?) the chance of similar events, or to make fixes throughout the code base.
The other thing is that after the post-mortem, all of the folks involved have learned from it. Usually they're better engineers because of that expensive lesson you've just paid for.
Disclaimer: Googler, participant in several post-mortems, opinions my own.
- tmp_anon_22 5y agoEveryone should support blameless postmortems. I'm more ruing the impending "oh we're soooo sorry" PR babble where it reads as a subtle "** off pleb".
- Mindwipe 5y agoBlameless post-mortems are good, but there have been rather a lot of bootloader level disasters on ChromeOS/Android and where they intersect at this point and it is probably worth wondering why this process does not seem to be generating sufficient improvement to make customer devices work better. One notes that even years later Google still haven't figured out why the Pixel C occasionally just forgot it's own passcode and encryption key and locked users out. Thousands of reports, and publicly acknowledged as an issue. Never addressed. Pixel devices have had multiple bootloader issues and are currently experiencing a widespread Widevine downgrade for months and not fixed.
- TechBro8615 5y agoSounds nice, but how much is this really true in practice? If one guy on the team seriously screws up, won't the rest of the team naturally trust him less?