14 ms·
Count to ten when a plane goes down
- feld 12y agoReset all computers in the embassy with F7? No warning prompt? Fire the idiot who wrote that function.
- mullingitover 12y agoStole the words right out of my keyboard. Who puts the 'reset every workstation in the building' button right next to the 'reset just one workstation' button with no confirmation prompt?
- fekberg 12y agoHe might have been that times IT Administrator? We've come a long way over these past 30 years, but think about it for a second, if you're an IT Administrator today handling all the office machines, you probably have the power to click 1 button to turn them all off? (Depending on the setup of course..) There's obviously more access rights involved today. Doesn't answer "who" put the button in there, I'm just thinking out-loud! :)
- matchu 12y agoThe issue wasn't whether he allowed to press it; it was just way too easy to press accidentally. Even if I have the authority to press the big red button, I'd like it to be behind glass and far from the light switch.
- Florin_Andrei 12y ago> if you're an IT Administrator today handling all the office machines, you probably have the power to click 1 button to turn them all off? To err is human. To push the error to thousands of instances online at once - that's devops.
- krrrh 12y agocrontab -e crontab -r
- samstave 12y agocrontab -ri
- jauer 12y agoIt could very well have been a minicomputer system where he was on the operator console and the other "computers" were terminals.
- kysol 12y agoMy thoughts exactly. With that sort of "global" function, you would expect some sort of countdown timer prompt on each terminal that could be cancelled.
- oh_sigh 12y agoThere was probably something like a warning prompt that came up, but he may have been used to disregarding the message because it was always what he intended to do.
- hueving 12y agoIt is possible there was a warning for both functions and the operator just acknowledged the warning assuming it was for the single reset without reading it. This is a common problem with warning too often in interfaces.
- baddox 12y agoIt's still pretty obviously a horrible idea to have the two keys right next to each other with confirmations that are even remotely similar.
- swilliams 12y agoHoly moly. That's almost like the one Far Side cartoon with a "Wings Stay On" "Wings Fall Off" switch. http://imgur.com/AosYvGn http://imgur.com/AosYvGn
- smacktoward 12y agoIn fairness, it was a different world back then. There were so few people administering computer networks that you could generally assume someone who was doing so had been thoroughly trained; and the thing about highly trained people is that they tend to view things like failsafes and safeties as pointless time-wasters. "I know what I'm doing when I hit F7, but the damn system makes me sit there for 30 seconds before it does what I told it to do! Piece of junk." The result was that software in that era tended to come with a lot more sharp edges. The age of the Recycle Bin that would save you from yourself didn't arrive until administering systems became something the general public was expected to do.
- austinz 12y agoThis is very true, which begs the question - why was a 23-year-old summer intern placed in charge of the embassy computer system, or even given access to its central console? He may have have self-taught facility with computers, but I find it hard to believe he'd have much experience at that age with the sort of large, mission-critical institutional computer system described in the article. I wonder if he had a supervisor sysadmin that he was working under, but given how he described his boss, that seems unlikely as well.
- analog31 12y agoYou might be surprised. What I observed during that time was that a lot of, if not most, senior people were clueless about what computers were actually doing. The potential "reach" of that intern might just not have occurred to people. And we still have national security / diplomacy disasters resulting from relatively low level people having access to vital computer systems beyond anybody's imagination.
- efnx 12y agoEven allowing for the era, providing the opportunity for a possibly disastrous effect to take place due to someone dropping the keyboard or knocking it with their elbow is inexcusable. Accidents happen - they should be minimized and steps should be taken towards quick recovery but they should be expected.
- 12y ago
- michaelneale 12y agoThat was one my my reactions too (yeah it was a different age). My other reaction was about the total loss due to a restart - in what could have easily happened by faults in many other systems - presumably there was no way to save data as work went no, no journaling or anything. Indeed this was a different age.
- dade_ 12y agoTwo servers side by side. Broken KVM that only switched the monitor. The screen was Windows NT, but the keyboard in front of the monitor was connected to another server running OS/2. Ctrl-Alt-Del The Windows screen did nothing, but the sudden hard drive activity on the OS/2 server told me what I needed to know. Not thirty seconds later, I had visitors. I was around 21 at the time, so yeah... Experience.
- raverbashing 12y agoI don't remember how it is on the newer versions, but try using fdisk in an old linux distro and see how many confirmations you get when deleting a partition. Or just the old rm -rf / Older stuff asks for much less confirmations.
- taylorbuley 12y agoNo thanks. It makes me shudder to just see this printed in the comments section!
- grecy 12y agoI was leaving my old job and handing in my MacBook Pro. After getting permission from the network guys (who were going to format it anyway) I ran rm -rf / as root. It was surprisingly boring, taking a very long time to delete all my files (I should have deleted them first). Eventually it got around to deleting fonts, which caused things to render a little strange, but after 60 minutes nothing much had changed and it was still chugging, so we shut it down and went to the bar for my last "Friday night drinks".
- yoblin 12y agoI did the same thing but had remotely SSH mounted some of our production servers. Won't make that mistake again.
- gordjw 12y agoI managed to run (sudo) mv /* /tmp one day. I was trying to move everything out of a folder to tmp (mv ./* /tmp), which was fine. Then I cd to /, and, wanting to rerun the command I'd run right before the mv, pressed up twice and enter quickly... Well, it failed once it finished moving /bin/mv to tmp. Of course cp, and a lot of other helpful commands come before mv alphabetically. It wasn't too bad, just needed a boot from a live disc to move everything back, but I still get nervous whenever / and * are in the same command.
- 12y ago
- xophe 12y agoYou must have missed the main point of this essay, so I'll reiterate here. "take a deep breath, count to ten"
- SonicSoul 12y agoagreed 100% (if there truly wasn't a prompt). OP took the fall because the head administrator was too ashamed to admit that their system was so poorly designed to allow for this to happen.
- deleted 12y ago[deleted]
- discardorama 12y agoIt's easy to have this attitude now. But if you ask people who were in IT 30 years ago, they'll tell you that systems in those days had _many_ sharp edges. You were expected to know your way around, and the consequences of mistakes were pretty severe.
- WalterBright 12y agoI was in IT 30 years ago, and this was never the attitude. What's different today: 1. we know a heluva lot more about human factors design 2. we have a heluva lot more excess computer power that can be devoted to human factors
- RachelF 12y agoI'd add: 3. many more people are using computers.
- mseebach 12y agoYou know what's totally plausible? That hitting F6 and F7 prompted for confirmation, but using the same prompt - and he just hit "Y", "Return" as he'd done 1000 times before (just like everybody does with the UAC dialog in Windows 7 today), and that bit didn't make it into the story because it's totally irrelevant to the point he's making. Heck, it probably wasn't even F6 or F7. If he's a normal human, he likely can't remember.
- JustSomeNobody 12y agoThink about the resource constraints of systems back then.
- openjck 12y agoSome people are saying that a warning would have helped. Remember that warning functions are easy to ignore. Never use a warning when you mean undo. http://alistapart.com/article/neveruseawarning http://alistapart.com/article/neveruseawarning
- hliyan 12y agoThat won't help here. You cannot "undo" a workstation reset, or any other action that results in a state propagation.
- openjck 12y agoSure you can. Pressing the button would start a timer. While the timer is running, the user would have the opportunity to review their selection (maybe even with a simulation of what effect the selection has) and could undo the request if necessary. Only after the timer expires would the action actually be taken. This is how the "undo send" feature of Gmail works. http://mashable.com/2010/08/22/how-to-undo-send-in-gmail/ http://mashable.com/2010/08/22/how-to-undo-send-in-gmail/
- _pmf_ 12y agoI don't know whether armchair usability trivia helps here. Not every system deals with ephemeral web drivel; some systems interact with the real world and have impact.
- openjck 12y agoAre you saying that usability does not need to be considered in the design of critical systems? Human factors grew out of the need to build safe and error-resistant weaponry in World War II. Poor attention to human factors and user interface design was a factor in the Three Mile Island disaster. http://en.wikipedia.org/wiki/Human_factors#In_aviation http://en.wikipedia.org/wiki/Human_factors#In_aviation http://en.wikipedia.org/wiki/Three_Mile_Island_accident#Human_factors:_Confusion_over_valve_status http://en.wikipedia.org/wiki/Three_Mile_Island_accident#Huma... With respect, to call this armchair usability and to imply that some users are just stupid is to completely misunderstand what usability is.
- ryanobjc 12y agoI disagree that it was appropriate that you were fired, but interesting story all about.
- kysol 12y agoFor security reasons I think that it might have been justified, considering the events that had just taken place. Still a crappy way to go out though.
- brown9-2 12y agoWhat security reasons? Firing the author didn't change what had already happened.
- kysol 12y agoPeople were bat shit crazy in the middle of that Cold War. If someone randomly decided to turn off machines without notice, even if they said "whoops accident, my bad", their actions would have instantly thought of as sabotage. I'm not agreeing with the outcome.
- sitinaud 12y agoThat fact that he wasn't too concerned about having accidentally reset all the computers in the building suggests that he may not have had an appropriate temperament/attitude for a sysadmin managing critical systems.
- Karellen 12y agoOr, you know, he had a perfectly good reason to think that accidentally resetting all the computers in the building at that time would not be a problem: "Not long after I arrived in my office, I received a call from a secretary in the Agriculture Department who liked to play a computer game before her workday started. Her favorite game had a bug that regularly froze her workstation. [...] I realized that I had mistakenly hit F7 and reset all the workstations in the embassy. This realization didn’t bother me much, because no one except the Agriculture section secretary was usually on the computer system this early in the morning." I'm sure I'd have thought something like: "Phew! Glad I made that mistake now, rather than at 11am when everyone was half-way through their morning's work. Likely no harm done at all, and I'm going to be really careful with that command in the future. Yup, definitely dodged a bullet there..."
- mikegreco 12y agoThe author states they felt it was appropriate when they were fired. In what world would it be appropriate to get fired for a single, simple, incredibly easy to make mistake? Doubly insane when there were exactly zero safeguards in place to prevent the mistake from being made.
- endtime 12y ago> In what world Japan, I guess? I've never been there but the story was consistent with my impression of their work culture.
- HillRat 12y agoIt was the AMEMB in Tokyo, though the basic principles apply to ay bureaucracy answering to political masters. Interns don't get AFSA (or, at the time, AFGE) union representation, and somebody's head was going to roll for that mistake, even though the company that programmed a non-confirmed global reset into a single keypress was truly at fault. Fair? Nope. Inevitable? Yep.
- jpatokal 12y agoThis was the American Embassy, which follows American work culture. Also, in Japanese companies, it's basically impossible to fire people. They can, however, be assigned to a desk in a windowless room and be given nothing to do for several years, until they take the hint and "voluntarily" quit.
- patio11 12y agoc.f. http://www.nytimes.com/2013/08/17/business/global/layoffs-illegal-japan-workers-are-sent-to-the-boredom-room.html?pagewanted=all&_r=0 http://www.nytimes.com/2013/08/17/business/global/layoffs-il... Suffice it to say that I am aware of situations created by a societal expectation of lifetime employment which make the above article look positively sane. (And I recently learned that, in some cases, what I had assumed was just an ironclad social contract actually is legally enforceable, which blows my mind.)
- lotsofmangos 12y agoI often suspect that most of the work involved in keeping a power hierarchy going, is involved with trying to pretend that this kind of shit doesn't happen all the time.
- lamontcg 12y agoAnd then conspiracy theorists latch onto this kind of shit, but believe that it must be malicious silliness... Why didn't Reagan respond immediately? Well, he was waiting to hear from Chancellor Gorkon that the KAL flight had been successfully beamed aboard and was en route to Pluto, of course... Clearly they'd have their shit together better so it couldn't have been a 23-year old rebooting all the computers accidentally and wiping out hours of critical work -- that would just be ridiculous...
- Loughla 12y agoConspiracy theorists are just 20th and 21st century prophets, really. They search for meaning in an all too often meaningless world. It's comforting to think that people can control the direction of every choice in the world, and that someone is at the helm. It's uncomfortable to think about the daily series of random, unconnected decisions that drive the direction of our species.
- lotsofmangos 12y agoI'm not sure that it is more comforting to think that there is someone at the helm, as much as anyone who aspires to be considered to be at the helm has to keep pushing that story, so it gets repeated more often and with better special effects than the story about there not being anyone at the helm. Actually being in control of stuff is very difficult, but convincing people that you are in control of stuff is pretty easy as we are all suckers for narrative. The main ways to disrupt a power narrative is to spread other narratives or for a situation to occur that upsets the existing narrative, so getting people to make up new ones. This explains why totalitarian governments can collapse so quickly, which wouldn't be possible if the people running them were actually in control of anything.
- Schwolop 12y agoThanks for posting this. I found https://news.ycombinator.com/item?id=8062683 https://news.ycombinator.com/item?id=8062683 yesterday but yours appears to be the direct link to the author's blog, which I had missed.
- deleted 12y ago[deleted]
- kosei 12y agoReally brave of the author to share this story. I know most people would be afraid to admit this kind of a public "mistake".
- joewaltman 12y agoGreat story....thanks for your willingness to share.
- deleted 12y ago[deleted]
- jackschultz 12y agoThe issue form most new stories is that even when the truth comes out, the great majority of people will never hear the actual facts. One issue is because news stations move on from caring about the story quickly. Or, the bigger issue in my opinion, is that people won't believe the new, correct facts since the old ones will have been engrained in their head. Solving both these issues would be really helpful for society, but are obviously damn hard to solve since we haven't really gotten anywhere in this space.
- Aardwolf 12y agoWhen there is such chaotic news story, I usually switch from news to Wikipedia. That has all the facts and continues the story even after all media lost interest.
- userbinator 12y ago"With great power comes great responsibility." Incidentally, "features" like this are why I don't trust systems that have some centralised control - IMHO giving any one individual (or organisation, in many cases these days) such power over others is not a good thing.
- tokenadult 12y agoI remember this time sequence very well because I was living in Taiwan when the incident happened. Yes, people who lived in east Asian time zones saw news reports that appeared to be based on knowledgeable sources that the plane might have landed safely with all passengers alive. This explanation of why the Western-aligned diplomats and military officials based in east Asia didn't have complete information when they were interviewed by the press is quite interesting, and explains puzzling memories I have from that day.
- dictum 12y agoFurther reading for those who want to be disabused of the concept of human errors: https://en.wikipedia.org/wiki/The_Design_of_Everyday_Things https://en.wikipedia.org/wiki/The_Design_of_Everyday_Things
- steven2012 12y agoWow. I got goosebumps when I read that article. I'm old enough to actually remember when KAL 007 was shot down, and while I wasn't old enough to hear about the conspiracy theories, I do remember the thing about people being safe and landing in Russia. To think that this was just a small mistake on the part of someone, which caused international ripple effect, and who later blogged about it is really something incredible.
- joshuaheard 12y agoI thought the headline meant to count to ten when a plane goes down...while you are in it!
- instakill 12y agoMe too. I still don't know how counting to 10 will help you press the right key on your keyboard though.
- ck2 12y agoHow about when Russia returned the data recorders after years of refusing to South Korea - made a press spectacle of it - and then South Korea discovered the recorders were empty and missing the data tapes when the press was gone. Or the US navy crew who received medals after shooting down the Iranian airline. Once there is loss of life, it is 100% politics afterwards with little to no practicality, just look at all the mass shootings where there were zero changes afterwards. We simply do not value life, it is politics first.
- nness 12y ago> Or the US navy crew who received medals after shooting down the Iranian airline. You make it seem like they received the medal for having shot down the plane. In reality, those who were awarded medals, were awarded Tour of Duty medals for their time spend in a combat zone. I believe the distinction is important, particularly since that class of medals are routinely awarded to individuals during their time in the military.
- ck2 12y agoIf a police officer shot and killed innocent bystanders, should they get achievement awards for doing their job otherwise? My answer would be no, you failed at your job regardless. Same thing with military.
- abcd_f 12y ago> That Korean announcement and the slow response by the US President — both caused by delayed real information — caused decades of conspiracy theories. I appreciate that the OP was a part of the situation, but conspiracy theories were not caused by this. It was time of very high tension between the US and Soviet Union. So when a plane veers off the course into not just Soviet airspace, but into an explicitly cordoned off top secret area, ignores all communication attempts, ignores the presence of fighter jets and just keeps on flying, then the situation itself is a fertile soil for conspiracy theories.
- brudgers 12y ago'It was a time of very high tension' doesn't quite capture how different it was. Through the glass of a yellow newspaper box, the Miami News headline that the Soviets had shot down a plane carrying a Congressman. My first thought was "This is the war." Not 'a' but 'the'. The primary stance of the US military was squared off against the USSR and had been for more than 30 years.
- jheriko 12y agoI hope you fought that firing... ... incompetence like that comes from having F6 next to F7 and no checks or authorisation needed for a potentially dangerous action etc. Processes should be designed for people to make the common mistakes... its what they do.
- jheriko 12y agonm. just seen the follow up. :)
- jere 12y ago>And let’s hope that there is no stupid 23-year-old with his finger on an important keyboard in this information chain. No. This is something you would read in Design of Everyday Things where Don Norman would totally shame the the engineers who made that system. Software shouldn't be designed with the assumption that no one makes errors.
- ak39 12y agoWhat I find incredible to believe is that this problem could have happened without the F7 erroneous keystroke by a human. A simple power outage could have resulted in this exact same catastrophe. Why didn't the backups work? System wasn't "robust" enough. (Did I just use the word "robust"?)
- tootie 12y agoAlan Cooper's About Face is a great book on interaction design for computers and one of his axioms is "Hide the escape lever". Basically make sure the ejector seat control isn't right next to the throttle.
- deleted 12y ago[deleted]
- peterwwillis 12y ago> On this day, I highlighted her workstation and hit the F6 key to reset. But my screen went temporarily black and then seemed to be starting again. I realized that I had mistakenly hit F7 and reset all the workstations in the embassy. Ugh. Those with automation capabilities: keep this lesson in mind, because it will happen to you in production one day. 'dsh -a reboot' is incredibly easy to type and can have disastrous effects. Creating abstraction layers around common admin tasks can help catch simple mistakes and give prompts before dangerous behavior.
- noisy_boy 12y agoFirstly, firing the intern doesn't make sense - it was a mistake waiting to happen and he just happened to do it at the wrong time. Secondly, the punishment meted out should be: 1. Proportional to the degree of carelessness (in this case not that much since he accidentally hit a wrong key adjacent to the right one, didn't mow down anybody while driving drunk) 2. Inversely proportional to the likelihood of the error (in this case the likelihood was very high since the reset key was a. uncovered/single-press b. right next to single reset key). 3. Proportional to intention (this was a completely unintentional error) If you say, that the punishment should also be dependent on the degree of damage, I would say that the responsibility of managing the risk of such damage wasn't his but of the person responsible for implementing such a high risk design. If such a person is not around, find the person who approved such a design. Government departments are usually very good with paper trail.
- acdha 12y agoHe's used a longer version of that story to view this as a management lesson – it's definitely not just “blame the intern”: http://globis.jp/774-2 http://globis.jp/774-2
- kyrra 12y agoSorta weird how the 2 articles differ from one another about being "fired". Count to ten article: > I, naturally, felt terrible and was, appropriately, fired. Honesty Wins article: > But, naturally, that day was my last day of work at the American Embassy. But, not because I was fired; although, I might have been fired if that day didn’t just happen to be the last scheduled day of my summer internship.
- acdha 12y agoBureaucratic jit-jitsu: http://johncbeck.tumblr.com/post/92502108047/so-what-did-you-do-after-you-got-fired-from-the http://johncbeck.tumblr.com/post/92502108047/so-what-did-you... “Oh, don't worry boss, we sacked the fool who made that mistake!”
- 12y ago
- hyperliner 12y ago"My boss, a >> Japanese << computer engineer named Itoh, poked his head in the door. " hmmm, I am pretty sure Mr. Itoh was not Japanese working in the American embassy. I am pretty sure he was American.
- billmalarky 12y agoIf he was a first gen American his culture would have been greatly shaped by Japanese culture.
- disputin 12y agoScapegoat. The ritual expulsion of the evil spirits wrapped neatly in a little parcel to appease the elders and thereby prevent them blaming each other - harmony continues in the hall of power. Meanwhile the problem was in the process, not the employee, so nothing has been fixed, and the guy who had learned the lesson is no longer there, and so the problem will recur with the next lamb to the slaughter.
- justizin 12y agoIt's a sign of poor management that someone has to be fired when something goes wrong, outages are learning situations for all involved, and it is widely held that the person who took the action that caused an outage is not responsible, but that all involved are responsible. See John Allspaw's Swiss Cheese Theory : http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ http://www.kitchensoap.com/2012/02/10/each-necessary-but-onl... . [ Edit: I guess it's not Allspaw's model, but he applies it to systems engineering rather well - http://en.wikipedia.org/wiki/Swiss_cheese_model http://en.wikipedia.org/wiki/Swiss_cheese_model ] "Accidents emerge from a confluence of conditions and occurrences that are usually associated with the pursuit of success, but in this combination—each necessary but only jointly sufficient—able to trigger failure instead." The person who pushed the button is not at fault, the manager is not at fault, the guy who designed the button is not at fault - all are jointly responsible. Blaming the intern does, however, reflect extremely poorly on Itoh and everyone else in the chain of command. A superior who demands retribution for a simple mistake that happened to cause him or her pain is basically worthless. But, I forget, we're talking about Ronald Reagan.
- deleted 12y ago[deleted]
- privatedan 12y agoMy first thought after reading the article was that it was ridiculous to fire/scapegoat the author for hitting the wrong key, too. This has happened to me before, where a single keystroke ( in my case, a line break in a config file ) caused me to take down a production system. My punishment? Designing a more robust system that would protect itself from a badly formatted config file. To this day, ten years later, a similar error has not been repeated, despite several attempts of people to push bad config files to our production systems. If I had been instead fired, no doubt a similar, but perhaps not exact, error would have been repeated every year or so. If I had made the same mistake twice without any attempts to fix the situation long term, then, yes, I think that would have been a fire-able offense. If you're working with people who care primarily about their own positions and egos without regard to the team as a whole, well, be prepared to be thrown under the bus when it comes time for those people to protect themselves.