6 ms·
Wolf Incident Postmortem
- layer8 4y agoA very good observation: https://www.jefftk.com/p/wolf-incident-postmortem#109658988836786118 https://www.jefftk.com/p/wolf-incident-postmortem#1096589888...
- Communitivity 4y agoThis is awesome. Job well done, and hats off sir.
- h2odragon 4y agoSensor was faulty, sensor replaced, system functioning normally.
- pixiemaster 4y agoCruel.
- thebetatester 4y agoIt took me longer than I care to admit to figure out this wasn't talking about a real incident. I figured flock was about some Kubernetes/VM thing.
- michaelnoguera 4y agoContext for those wondering: The Boy Who Cried Wolf https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf#The_fable https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf#The_fab...
- cialowicz 4y agoWell done. Seems like a good prototype to show someone who is unfamiliar with how a postmortem should be written up.
- paulgb 4y agoIndeed. Bookmarked it for exactly this.
- dvh 4y agoReminds me of an admin story where each incident ended with updating /etc/motd
- msla 4y agoHere I thought Jeff K was a sly fox, not a wolf.
- 0xbadcafebee 4y agoThe only wolf evidence are alerts by a now-missing sentry saying "wolf" twice and "real wolf" once. Yet the conclusion due to missing sentry is 'wolf', and not only that, that there must have been only one wolf rather than a pack. If you don't have supporting evidence, this conclusion may be faulty, and the real fault may return. My initial reaction is that there is no wolf. The sentinel began exhibiting a mental bug that reoccurred for 2 days until the sentinel broke and wandered off. I think the resolution should have an action item to continue closely monitoring sentinels for further bugs, and collect metrics to show proof of wolf. And, you know, maybe come up with an additional wolf countermeasure (if the fault was a wolf, it will probably come back)
- abeppu 4y agoI really prefer when the "postmortem" label is metaphorical.
- barbazoo 4y agoI don't get the joke but I browsed a bit, got to the story about "MA RMV Overloaded" [0] where they tried to renew their driver license. I was baffled by the fact that it was possible at all and even faster to renew your official driver license through a private company (AAA) than through their DMV office. [0] https://www.jefftk.com/p/ma-rmv-overloaded https://www.jefftk.com/p/ma-rmv-overloaded
- mlyle 4y agoIt's about the "boy who cried wolf" vs how incident management works. False positives can make real issues get an insufficient response when they occur (e.g. sentinel getting eaten).
- jandrese 4y agoFalse positives can be a far more serious issue than people are willing to admit. There are plenty of loss of life incident reports that include wording like "the operator disabled the alert system due to numerous false positives" near the beginning. An example: https://www.youtube.com/watch?v=1zDcsjHyxr8 https://www.youtube.com/watch?v=1zDcsjHyxr8
- kayodelycaon 4y agoIt's a reference to this story: https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf :)
- msla 4y agoI remember Jeff K's old website: https://www.somethingawful.com/hosted/jeffk/ https://www.somethingawful.com/hosted/jeffk/