7 ms·
How Complex Systems Fail (1998)
- feyman_r 25d agoI may have shared this before on a different submission: John Gall’s books are really good on this topic: General Systemantics [https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics]
- littlecranky67 25d agoGall's law is amongst my favorite ones and with decades of experience in software development, I have to say it holds absolutely true: > “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” — John Gall, Systemantics (1975)
- rglover 25d agoLove Systemantics. That was one of the first systems books I read and where I felt like I found someone practiced who understood how systems actually function (a realistic, honest view that systems always fail, it's just a question of when, why, and how).
- deleted 25d ago[deleted]
- rowyourboat 25d agoAll of this sounds just like any air crash investigation I ever read
- shash 25d agoAnd industrial accident investigation (except the ones with low regulation or whatever). And market or supply chain collapse, and civilization collapse (late Bronze Age anyone?)
- tptacek 25d agoRichard Cook was a UChicago anaesthesiologist who took up safety systems research after studying patient safety; some of his work is rooted in Three Mile Island, and some of it comes from aviation safety.
- Balooga 25d agoYes, and the cause is almost always due to many small errors, the Swiss Cheese Model[1] [1] - https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- jedberg 25d ago> Failure free operations require experience with failure. This is why we created Chaos Engineering. By constantly forcing failure, it made us always create systems in defense of that failure, and gave us great data on where the tipping point is for different systems within a particular failure mode.
- AlotOfReading 25d agoI've always struggled to apply this to the systems I work on. If the system fails, someone potentially dies, though in practice they've never been more than hospitalized. To avoid that, huge amounts of effort are expended on failure modelling and testing, but that doesn't eliminate unknown unknowns. That discrepancy has made front page news a couple times.
- jedberg 25d agoWhen I was at Netflix and reddit, one thing I said often was, "Luckily, we are not a bank". And it sounds like you are working on even more critical systems than that. Chaos engineering doesn't really apply to data critical or safety critical systems. You can't just break them in the real world to see how they fail. You have it exactly right -- it has to be modeled and tested in lab conditions. Safety critical systems are not a place for YOLO development.
- YZF 25d agoIt's still a tool in the tool box. Somewhat analogous to accelerated life testing in non-software products. You induce conditions that make failures more likely to occur.
- obscurette 25d agoIt's much more universal and complicated than that. One of the big issues schools and pedagogy in general struggle with is that our environment is far too safe in too many ways. For kids there is too few ways to learn from failures. Attempts to solve the issue look often like "Hey, kids, let's fall over now all at once in safest way possible and learn from it!". But it doesn't work at all. Real failures have to be unexpected, related to your decisions and really hurt so that you can learn from them. PS. Btw, I am certain that this is the main cause of the mental health crisis amongst young people.
- mohamedkoubaa 25d agoI can't tell if the article is describing how complex systems fail or if they are using failure characteristics to define complex systems.
- shash 25d agoIt’s more about failure. It’s right there on top.
- yipinwong 25d agoI think there are a few common themes to the failure reasons, but cannot get my hands on it. This seems like a list of reasons while I am looking for more abstract directions on how to prevent them. --- I am trying not to use AIs to just do that for me to tinkle my neurons.
- shash 25d agoI think, part of the point is that it’s not possible to have a recipe to prevent failures. They are cascades of many events coming together to fail in an a priori non obvious way. Or so I read the [site? article?]
- deleted 25d ago[deleted]
- haemdahl 25d ago[dead]
- squirrel 25d agoThe definitive work on this topic is Normal Accidents, with a modern retelling in Meltdown. https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcsik_book) https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcs...
- tptacek 25d agoI'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more. But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up. All practitioner actions are gambles. I should have that inked somewhere.
- ErroneousBosh 25d ago> I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/ https://grugbrain.dev/
- tptacek 25d agoI am also a broken record about Grug-brainedness, but I don't need to be so repetitive about it because so many other people are too.
- ErroneousBosh 25d agoMy Grug-brained hill to die on is that there are many pieces of technology like the LM741 opamp, the Rover V8, Morris A-series and Peugeot XUD engines, and a big SQL database running on a thumping great Unix box, that have remained essentially unchanged for decades, worked well when they were new, and are still a perfectly good solution today for so much of what you want to do. You know what they have in common? A lack of complexity. Lovely simple technology, fix it with a hammer. Drive it to the moon and back. Drive it across the desert. Bring tools though!
- icantevenhold 25d agoOne of the great documents of our civilisation
- sandeepkd 25d agoOverall a good collection on the complex systems, somehow it missed the part how complex system came into the existence in the first place itself. > Human expertise in complex systems is constantly changing I feel this is single most important factor responsible to both making system complex and at the same time improving them depending on who the people are and how they take failures and breakdowns. I find it funny but we are in an era where folks building ML systems do not seem to remember the direction in which to open the screws. They do have clear expertise in something new but clearly lack in some other areas
- tptacek 25d agoLiterally the first item in the list addresses the necessity of complexity in the systems it's discussing.
- sandeepkd 25d agoA hazardous system does not necessarily has to be complex. The article is a generalized list which makes it bit harder to reason with on this topic. A big part of the complexity gets added over the time as a part of failures, effort to introduce more fail safes, processes to avoid the same failure again. Complexity begets complexity, which is why I was referring to the dependency on the human elements who are involved in evaluating the failure and remedies. tldr; Removing a process is risky and thankless for most part, no one wants to take that risk. So after every failure a new process gets added and thus you end up with an ever growing complex system
- deleted 25d ago[deleted]
- evidaxis 25d ago[flagged]
- nemesis17 25d agoWorth reading Leveson’s works on safety engineering.
- throwaway231013 25d agoHighly recommend "The Logic of Failure" by Dietrich Dörner (1997)
- hn974izqdv 25d agoGood take, no notes
- anonymars 25d ago"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously." This very much resembles Admiral Cloudberg's write-up of the National Airport collision: "No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost." https://admiralcloudberg.medium.com/reaping-the-whirlwind-inside-the-potomac-river-midair-collision-0475416f2b0f https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
- deleted 25d ago[deleted]
- esafak 25d ago> After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. What does that mean? If you knew that the precursors were why did you not set alerts?
- cpgxiii 25d agoThe "naive notion of system performance" is that the system is primarily operating in the ideal regime, and thus all deviations deserve alerts (and receive attention). The reality is that often the system is operating often enough in some failure mode that any such alerts would be regularly triggered by accepted behavior, and thus the alerts are systematically ignored/disabled/normalized. E.g. nominally you should never be mixing traffic types (aircraft and helicopters, civilian and military) in close proximity to a major airport and in a regime where TCAS is unlikely to offer sufficient protection. So in theory, any mixing should immediately trigger an alert and investigation to develop new procedures. But in practice, if you routinely allow such mixing under what you believe are "safe" practices (and get away with such mixing for a long time) then when a real accident happens you will have plenty of "proto-accidents" to look back on, but the warning signs from those near-accidents will have become accepted practice.
- zero_k 25d agoThis is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly: Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse. [1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf https://www.msoos.org/largefiles/safety2.pdf
- ChrisMarshallNY 25d agoWhenever I see this [excellent] paper, I am always struck by the first sentence, in the first section: > All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature. (Emphasis mine) I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.
- DJBunnies 25d agoIs it not merely an example of such failure?
- deleted 25d ago[deleted]
- elisbce 25d ago"Catastrophe requires multiple failures – single point failures are not enough." I disagree with this point. From what I have seen and experienced, there are so many SPOFs in the real-world complex systems that do not have adequate defense or even any defense at all. The only reason catastrophe didn't occur (often) is only because the natural failure rate is extremely low. Like the probability of air traffic controller making a critical mistake is very low, maybe once every few years, but when it happens, the SPOF will almost certainly cause a disaster, and the system (ie. the way air traffic is controlled today) has been around for decades and is so outdated but the cost of changing the entire process around the world is simply way too high so we just take this as "normal".
- stAInley 25d agoThe operators know complex systems work. Words I've spoken, heard, or both: "Just rerun it, works the second time." "That's been failing for a while now. Just ignore the alert." "This procedure is old. We've added a step 64b in pencil. I can't remember why." "Is it meant to sound like that?" "What's the tape for?" "Careful you don't bump this button." "The database name has changed, the post-it has the new one. That's a '1' by the way." "Accept the failure, keep going." "Because you're the person touching the system the most, you're the most likely person to break it." "I know it says call on failure, but they never answer."
- bohnohboh 25d agoa nice compliment is The Myth of Progress by Tom Wessels. Preview here: https://www.perlego.com/book/2236739/the-myth-of-progress-toward-a-sustainable-future-pdf https://www.perlego.com/book/2236739/the-myth-of-progress-to...
- RachelF 25d agoEconomists would be wise to read this. The world economy is now a rather complex system, and many of the safeguards from previous generations have been removed in the interests of "efficiency".
- owen-hill 25d ago[flagged]