5 ms·
The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data cent
by lovedswain 5y ago
The global outage drinking game:
bad DNS config push - 1 shot
routing loop - 2 shots
third party advertising your routes - 3 shots
power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots
Wolves ate through fiber - 5 shots
And it was a full moon - 6 shots
Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots
And service developers left the company to found a startup - 8 shots
Expired internal SSL certificate - 9 shots
Daylight savings changeover-induced database corruption - 10 shots
Windows Update - 11 shots
- ASalazarMX 5y ago> Wolves ate through fiber - 5 shots I hope it's wolves, that would create a new category in https://cybersquirrel1.com/ https://cybersquirrel1.com/
- partcyborg 5y agoYou forgot: Hunter shot your fiber relay with a hunting rifle - 6 shots
- nabla9 5y agoApple decided to shut down Facebook.
- godelmachine 5y agoDST can be such a hassle. It affects our reports and change requests.
- erik_seaberg 5y agoAn adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.
- erik_seaberg 5y ago5 is often broken down to “rain fade” and “backhoe fade.” My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
- lovedswain 5y agoIn a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart
- erik_seaberg 5y agoI heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.
- reaperducer 5y agoCoincidentally, I'm in the same predicament with Facebook, itself. I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook. Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
- dylan604 5y agoisn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?
- reaperducer 5y agoBecause the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.
- 5y ago
- alfalfasprout 5y agoDon't forget cosmic rays!