13 ms·

Type in the exact number of machines to proceed

554 points by vii 6y ago

dgritsko 6y ago
Similar idea as GitHub's "type the exact name of this repository if you want to delete it" confirmation dialog. Maybe that's really what you want to do, but in case that's not actually what you meant to do, having a few extra hoops to jump through seems like a good idea.
- Hokusai 6y ago
  > having a few extra hoops to jump through seems like a good idea. I think that there is more to that. You need to consciously type the name of the repo that you want to remove. Windows used to add a lot of jumps to get something done, and the result was mindless clicking the "yes" button and realizing 1 second later that you deleted important information. That extra hoops need to be cognitive meaningful.
  - Cthulhu_ 6y ago
    Yes, and infrequent; the main issue with Windows (Vista mainly) was that it appeared far too often. Even with 7, when you're setting it up for the first time for example, I think it shows up too often. Same with Terms & Conditions. If you want your customers to truly have read and understood them, you have to show them a short quiz at the end of it. You're required to do a quiz in Europe nowadays if you want to engage in stock trading.
- wjdp 6y ago
  Do you type the repo name, or just copy/paste or select/middle click it? Half of me would want them to put `user-select: none` on that text. The other half has to archive 10+ repos and would hate that!
- segfaultbuserr 6y ago
  Some disk management software also has "type the exact label of this partition to reformat it" to prevent accidental data loss.
- edanm 6y ago
  That's what I thought of immediately as well! I've seen that pattern in a few other places too, and I always think it's a really good UX choice.
xamuel 6y ago
I wish it were possible for similar prompts to appear before all sorts of policy-makers and bureaucrats. "It appears you are about to institute a policy which will require 400 million patients to sign an additional waiver every time they visit a clinic, this will waste a total of 354,921 human hours within the next year alone. Please type 354,921 to proceed."
- jackhack 6y ago
  closely related: the Paperwork Reduction Act of 1995 https://digital.gov/resources/paperwork-reduction-act-44-u-s-c-3501-et-seq/ https://digital.gov/resources/paperwork-reduction-act-44-u-s... it requires the office of management and business to calculate the impact of records-keeping requirements impact on time and privacy, among other things. I do not believe it has resulted in a reduced recordskeeping burden. For the most part I simply see an estimate of how long it will take to complete my tax forms and permits, on the form itself. Perhaps others have different views.
  - mulmen 6y ago
    Hard to say, knowing the cost of a new process could have informed a new design or requirements. We don’t know what the other path held. But I believe in general having more information allows us to make better decisions so this is a good act.
- Joker_vD 6y ago
  Yeah, it's quite surreal. "Hey, privacy is important, so let it make so that to handle people's private data, you'll need a permission from them". All right, now whenever you try to e.g. send a (paper) mail, you have to sign the waiver that yes, you do allow the post office to see and handle your name and your mail address. Not only that, all such waivers seem to be written as "I hereby allow <insert the legal entity> to handle my private data in whatever way they want to", so we're back on square one, just with more perfunctory paperwork required.
- mulmen 6y ago
  How do you know it was a waste? Maybe that was time well spent.
roydivision 6y ago
Reminds me of the proposal to keep the nuclear launch codes inside the body of an innocent volunteer, so the President would have to kill the person to get the codes. https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-lau.html https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-...
- chrisseaton 6y ago
  I've never understood this idea. If you believe we should never use nuclear weapons, then don't have them at all. If you believe there is a case where it may be moral and rational to use nuclear weapons, why would you want to put a potential barrier in the way of their use? You could have a situation where everyone was agreed to use them but the president was physically unable to harm the aide to use them. You can know that something is the right thing to do but not have the courage to physically harm someone to do it. An interlock that you may not be able to unlock for reasons unrelated to the task at hand is a bad interlock.
  - gumby 6y ago
    > I've never understood this idea. It's an attempt to make an abstraction concrete. Think of it as the trolley problem in real life. Stalin is famously supposed to have said, "one death is a tragedy, 100,000 is a statistic". Cynical or not it is how humans think. > If you believe we should never use nuclear weapons, then don't have them at all. Strategic game theory and Mutual Assured Destruction depend on the possibility that the other guy will use them if you do, and may be the only way to prevent their use. Interestingly this is one reason why you want the other guy to know your procedures, capabilities, deployments etc. Secret weapons have no deterrent value.
    - chrisseaton 6y ago
      > Think of it as the trolley problem in real life. Well exactly... doesn't that show you that it's a bad idea? People don't know if they could bring themselves to throw the switch even if everyone thinks it makes rational sense. You're taking a rational, well-considered, strategic decision... and making the interlock a messy personal emotional one unrelated to the actual issue at hand. That sounds like the wrong way around to be doing things?
educationcto 6y ago
Terraform prints out the number of resources changed and at least requires a "yes" to proceed. Not quite as onerous as described but at least prevents some type of fat-fingering. Basically all changes with Terraform are risky as they usually involved bringing up and down infrastructure. Terraform will perform the following actions: # google_compute_instance.vm_instance will be created + resource "google_compute_instance" "vm_instance" { + ... <more> Plan: 2 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes
- brodouevencode 6y ago
  IaC is a real time saver, but inherently dangerous.
- caymanjim 6y ago
  This is exactly the problem the author is referring to. With Terraform, you always type "yes" to proceed, so it turns into muscle memory. You stop reading the output, and you're already typing "yes" before you even see the prompt. Terraform's output is also verbose, and many changes show up as "1 to add, 0 to change, 1 to destroy" because they don't separately list a "replace" category. It's pretty bad; you've got cognitive overload, confusing output summary, and a predetermined continue answer. And this is often an action you're performing under duress. I've been bitten by it plenty of times.
jancsika 6y ago
It would be neat to print out an esoteric error that gets a single result in Google, where the "forum" in the result has a rando answer about using a certain esoteric flag. Then you search the logs to see who is trying the command with the esoteric flag and "fix the glitch with payroll" for those employees.
JoeAltmaier 6y ago
Makes it harder to nest that command inside a script - you have to parse out the number and paste it back? Or do I misunderstand - should it still prompt the user in the middle of the process when that step arrives? That would be problematical if it were included in a web page or whatever.
- ccakes 6y ago
  The very point of this is to make it difficult to do what you’re describing. If the tool could potentially touch a large number of machine, even if you’re super sure you got it right you should still prompt the user
  - JoeAltmaier 6y ago
    Or write a script that carefully calculates the number of machines and gets it right. I guess you wouldn't use this prompting script then?
- larrik 6y ago
  I believe this would be as part of the script you are writing, not the scripts you are calling.
- rad_gruchalski 6y ago
  Hopefully there’s an API to fetch that count :)
csmattryder 6y ago
I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling https://en.wikipedia.org/wiki/Pointing_and_calling
- MaxBarraclough 6y ago
  Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.
  - blantonl 6y ago
    Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.
    - staunch 6y ago
      And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
      - nemosaltat 6y ago
        Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.
larrik 6y ago
I've seen this sort of thing in a few places, and I really do think it's a great idea.
deleted 6y ago
[deleted]
Cthulhu_ 6y ago
I for one can't fathom any organization managing a million devices / servers / VMs / whatnot. I'm having enough trouble with one, and my biggest employers had maybe a few dozen at best, and they already had a dedicated ops team that worked mainly with infrastructure-as-code.
sidpatil 6y ago
Hmm, it's conceptually like a combination of a CAPTCHA and a launch code.
jaclaz 6y ago
Side question. How many/which companies have more than one million Linux machines?
- notacoward 6y ago
  At least Facebook (where OP worked), Amazon, Google, and Microsoft. Probably Netflix, maybe Apple. There might be a couple more, but no more than that because we've already accounted for a pretty high percentage of worldwide shipments for servers, disks, etc. Fun fact: when you're that big, your demand creates its own inflation and you have to consider that in projections.
  - jaclaz 6y ago
    Thanks, so a handful at most, and the "usual" ones, I always thought that those companies keep their machines connected in (redundant) "sets" and that a command affecting all of them was more a case for "never" rather than "once in a while".
    - jeffbee 6y ago
      Google, at least, has a thing that is supposed to prevent widespread disruption at the machine level, called the "Safe Removal Service"[1]. This is a good idea that in practice isn't perfect. If you write a tool that does not consult SRS, or your service doesn't declare a SRS policy, there can be surprises. A particular outage that I will never forget took out Gmail delivery worldwide in an instant, because the change was not expected to be disruptive and therefore did not integrate with SRS. As it turned out the change disabled the machines where it was applied, and the process of selecting a subset of machines to canary the change was not independent of the way in which Gmail assigns services to machines, so in the space of a few seconds they created a global outage. https://twitter.com/bgrant0607/status/1134536670504554496 https://twitter.com/bgrant0607/status/1134536670504554496
  - kube-system 6y ago
    If by "machine" we also mean things outside of a 19" rack, I would wager that large telecoms probably have way more devices running Linux than FAANG. Imagine the network of cable modems that Comcast alone must operate. What percentage of their 28+ million broadband customers rent Comcast owned/managed modems? Almost all of them except the tech-savvy crowd? And that's just one device type.
vsnf 6y ago
I do this with a git pre-push hook to the main branch of my repositories. It displays a prompt in red and forces me to type in the name of the branch. The result of one too many mindlessly accidental pushes.
willvarfar 6y ago
I am so adding this to a query api I have, where its all too easy to leave off constraints and end up asking for massive data sets by mistake. Thinking I can probably enhance it by forcing the user to type in the number as text rather than numeric, so they can't cut-n-paste. Kind of force them to type in "I am sure I want all data ever" or something.
- recursive 6y ago
  I don't think this is useful for an api. This is only useful when humans are the direct user of the component. Automated users, like those of an API will dutifully provide the required safety value.
nemo1618 6y ago
Notably, Discord does something like this when you @everyone in a large channel: "You're about to push a notification to 12,000 people, are you sure you want to do that...?"
- pwinnski 6y ago
  Sounds like a yes/no answer is expected? If so, that is exactly what Rachel is suggesting is not enough.
  - jerf 6y ago
    In this case, usually the very fact that a popup unexpectedly popped up is enough. I use Konsole as my main shell, and like several other shells now it has a "You're about to paste 100KB, yes/no?", and I don't mindlessly click "yes" because it is already a "cache miss" to see that dialog at all.
  - justincredible 6y ago
    She also suggests localizing the number will prevent copy paste somehow.
- raverbashing 6y ago
  Slack should take a note of this. Especially for rogue @here notifications
eznzt 6y ago
Debian already does this, it asks you to type something like "yes do as I asked" if you want to remove a package that is considered to be part of the core.
vondur 6y ago
That may have helped when Emory University's IT dept. accidentally sent a wipe and reformat command using Microsoft's SCCM to all of the Windows computers and servers on campus back in 2014. https://it.slashdot.org/story/14/05/17/051214/emory-university-sccm-server-accidentally-reformats-all-computers-campus-wide https://it.slashdot.org/story/14/05/17/051214/emory-universi...
jasonpeacock 6y ago
Raskin talks about the futility of this in his book The Humane Interface. Basically, what happens is the brain switches operating context from "I want to do something" to "resolve this interruption (confirmation box)" and you don't relate the one to the other - you're so focused on getting rid of the interruption that the original task is forgotten until after the interruption is gone. Then you switch back to the original task that had been interrupted by the confirmation box and then you realize you made a mistake. It's much better to engineer "undo" ability into systems - like delaying commands (GMail's "Undo Send" does this), or caching previous state, etc.
- andrewflnr 6y ago
  That's exactly why it's not a "confirmation box", but requires you to slow down and think for half a second. She even talked about mitigating copy-paste, which is the next obvious way people could habituate. Also, while undo is great, it's not always technically feasible. The tools in question are basically for modifying the layer that implements undo for your end users, and are often themselves fundamentally irreversible. Undo for raw hard disks involves forensic analysis at best.
  - robaato 6y ago
    Or you have commands which randomly reverse the meaning of the confirmation prompt: Continue: yes or no? Don't continue: yes or no? As long as operators know to expect this, they also know to wait and actually read the prompt before answering (as in turn of auto reaction)...
  - jasonpeacock 6y ago
    The problem (I probably didn't paraphrase Raskin well) is when you slow down & think for a half a second, you context switch from "I need to do operation" to "I need to make this dialog box go away". No matter what tasks are required to make the dialog box go away - doing math, retyping a message, clicking a randomly ordered box - that becomes the top task in your head and you "forget" about the original task until you finish this task. Once you resolve the interruption, you switch context back to the original task and then you still have that "oh crap" moment. Yes, sometimes undo is very difficult, and can require a system designed to support that ability as a first-class feature from the start. Many systems you can perform rollbacks, but there are definitely destructive actions - in which case you should have test stacks to validate your actions in advance, and peer review. (e.g. dual keys to launch the missiles)
Ayesh 6y ago
I have an old laptop with a dead battery, and for a BIOS upgrade, it prevents me from updating without 50% battery. I have to type "danger" to bypass this restriction, and I thought it was pretty cool. Another good UI pattern is in Firefox, that it disables the Run button on downloads for a few seconds.
- duskwuff 6y ago
  Disabling the "run" button for a few seconds was actually done to mitigate another risk -- sites cueing the user to click in a particular location, then triggering the confirmation dialog with the "run" button right where the user was about to click.
illumin8 6y ago
This is a great idea, and I'd like to point out that having such a system in place would have prevented one of the largest Internet outages in recent memory - the Amazon S3 outage in 2017: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/ > At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.
- zedpm 6y ago
  It's kind of funny, since various operations performed in the AWS web console use this model (e.g. type the name of the resource you're trying to delete). As an organization, they're aware of this approach and think it's useful, but (presumably) didn't use it in their own internal tooling.
  - audience_mem 6y ago
    Perhaps those were added after they learnt their lesson.
harikb 6y ago
I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.
- Xophmeister 6y ago
  We've been known to use something like --yes-i-really-mean-it-this-time for really dangerous options. It's a like built-in solemnisation step.
- austinl 6y ago
  I like this format in general, since it communicates the command is severe/irreversible. Heroku implements a similar confirmation when performing destructive actions. Commands require your to pass a `--confirm ${APP NAME}` flag, so the original command itself does nothing. Of course, this doesn't prevent you including those flags in makefiles, etc. I once dropped a table in a side project by accident because I took the wrong tab autocomplete suggestion in a makefile.
- yobert 6y ago
  I did this with "--im-not-scared" for production mode :D
  - csours 6y ago
    We had one that required "BADIDEA" to run
- jrumbut 6y ago
  Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.
  - dingaling 6y ago
    I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o
jerf 6y ago
https://news.ycombinator.com/item?id=24907002 https://news.ycombinator.com/item?id=24907002 Looks like https vs http link.
- dang 6y ago
  We've merged the threads now. Thanks!
Animats 6y ago
Yes. Github does that when you delete a repository. You have to confirm by typing in the name of the repository you are deleting.
tigger0jk 6y ago
I've typically used pdsh https://github.com/chaos/pdsh https://github.com/chaos/pdsh for these types of commands, and I don't think they have any such safety options. The only protection is to be wracked with fear whenever you type pdsh. Obviously this fear wanes with use, and eventually you don't think about a command for long enough before you do it and hit enter on a regrettable one.
luhn 6y ago
One of the largest AWS outages to date was caused by a scenario like this. [1] A mistyped commanded removed too many servers from an S3 subsystem, overloading the remaining servers and crashing the subsystem. The failure snowballed until the entire S3 region was down, which then caused issues with dependent services like EBS, ALB, and Lambda. They couldn't even update the status page because that also depended on S3. [1] https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- HenryKissinger 6y ago
  I remember that. The AWS dashboard was all green checkmarks... because the red checkmarks icons the dashboard was supposed to display were stored inside the crashed servers.
- jodrellblank 6y ago
  >"overloading the remaining servers and crashing the subsystem. The failure snowballed until" the entire Eastern Seaboard was without power? https://youtu.be/XetplHcM7aQ?t=693 https://youtu.be/XetplHcM7aQ?t=693 (James Burke's Connections, ref. cascading power cut 1965)
TravHatesMe 6y ago
Reminds me of a study done where a test was given with questions that weren't difficult but likely to make a silly error. Around 85% of participants got at least one question wrong, but when they repeated the same test with a difficult-to-read font, that number dropped to ~25% or so. That's another way to make your brain work, use a terrible font.
- apricot 6y ago
  > That's another way to make your brain work, use a terrible font. And suddenly my complex analysis prof who wrote his exams in Comic Sans is vindicated!
andrewfromx 6y ago
i've done this before by displaying unix epoc and asking the user to copy/paste that value WITHIN a 3 second window as an env var. i.e. if you up arrow and run same TIMESTAMP=1603827448 ./foo it won't work because 1603827448 is now way too old.
- myroon5 6y ago
  One of the main benefits is explicitly acknowledging relevant context. Timestamps don't provide additional relevant context
ComodoHacker 6y ago
In role-playing games, it's a common practice to confirm deletion of your character by typing in some word, like 'delete' or character name.
kbenson 6y ago
This is a topic near and dear to my heart, as I'm often that person arguing to make some slightly less automated because the small trade-off in time is insurance against some of the worst mistakes you can have. Automation to the point of removing humans leads to stupid problems that a human wouldn't make if they looked at what was going on. So we automate tot he point where we minimize human contact, presenting a summary of actions that as humans we can apply our wonderful brains to and prevent those problems. Except some percentage of the time we don't actually pay attention, and depending on how the human interaction was introduced instead of complete automation, some percentage (or multiple!) of errors still sneak through. Automation to the point of minimal human contact where you assume the human will read the presented information and make an informed decision doesn't work. The point is that we want a human to understand what is being asked, so taking some step to ensure they do understand is warranted. It will never be perfect, but adding steps like she proposes are definitely a step in the right direction, IMO.
bronco21016 6y ago
It amazes me that something like this can be done by a single person. In aviation any time input is given to the machine, it's entered by one human (typically pilot flying) and then verified by the other human (typically pilot monitoring) before being committed to or executed. For example... when a new altitude is assigned by ATC, say FL300, the pilot flying will spin it in the selector window and keep his hand or finger there until the second pilot agrees with and confirms the selection by reading FL300 out of the selector window. I know there are meat bags in these giant tubes so that changes attitudes towards safety etc. However, it seems to me that when organizations start putting the power to halt nearly the entire business in the hands of one person, there should be some slightly different attitudes. A breaking change in a million servers could easily cost hundreds of thousands or maybe even millions in lost revenue or employee productivity. I'm just an outsider though. Perhaps this level of attention is practiced at some shops. It's just interesting to me how in some fields we settle on pretty uniform standard practices whereas others are seen as non-human-life threatening so it's just shoot first, ask questions later.
- rachelbythebay 6y ago
  Best practice for using the "weaponized" version of the tool when you had powers to actually hit all of them at once was to paste the command into IRC and get some of your fellow peeps to eyeball it and make sure it was sane. <me> team: hey, sanity check this please: hsh -A "dumb_thing && other_thing --foo --bar" <teammate> shipit [ I type the command ] <me> ok, running as job 1234 The last part was a courtesy done so that they could watch the progress of it too without having to dig to find my request. It also meant they could kill it easily if something went wrong and they couldn't raise me for some reason. Tools like this are best used outside the solo realm.
  - im3w1l 6y ago
    I think an automated tool would be preferable since there is no 100% foolproof guarantee that what you type in irc is the same as what you type in the terminal.
- crispyambulance 6y ago
konjin 6y ago
Finally the Roman numeral converter I programmed in university will be useful.
gabeio 6y ago
I do like this idea, this is I assume why github makes you type the repo name out in full. I wish AWS followed suit, when deleting any RDS (database) instance on AWS all you have to type is "delete me"... very easy to copy and paste as well as just know what you need to type and be on autopilot. I have even poked support about it and their response was underwhelming.
remram 6y ago
A similar system is molly-guard [1], which replaces the reboot/halt/poweroff/... commands with scripts that make you type in the name of the machine before proceeding. Avoids shutting down the wrong machine because you forgot where you SSH'd. [1]: https://manpages.debian.org/buster/molly-guard/molly-guard.8.en.html https://manpages.debian.org/buster/molly-guard/molly-guard.8...
- b6z 6y ago
  Many years ago, I made that mistake two or three times, rebooting the wrong machine. Since then, I use molly-guard on all my remote machines. Never happened again.
heelix 6y ago
Back in the Spiderman 2 days, I worked for a content management company that was supporting a really, really big website. I believe they were playing host file games for Stage/Prod. Was in the room on when they demo'ed something, did a restart of the system - and every pager in the room went off. Yah...
rossjudson 6y ago
This resonates with me. Years ago I took down a service in a cell accidentally (Googlers might empathize: never 'borg' when you meant to 'borgcfg'). If I had been asked to enter the exact number of tasks I was about to nuke, I might have thought twice ;)
- scottlamb 6y ago
  I've certainly deliberately downed an enormous number of tasks, though, as part of a cluster turn-down. I love the technique of requiring the operator to echo a key fact, but in the case you're describing I think the key fact is not how many tasks but that that they're serving live traffic. So: * You could ask the operator to echo the qps figure...but really any number other than zero is likely to be an error, so it can just error out in that case without needing the confirmation. * Even if it is serving zero qps now, if it's not explicitly drained at the load balancer, downing it is likely to be a mistake. So even better to check that. Only once in my career have I taken down jobs serving live traffic. (They were serving 100% errors.) It was deliberate, but even so I wouldn't have minded having to supply a --yes-i-know-im-downing-live-jobs. edit: and if for some reason my assumption is wrong and downing undrained things becomes routine...well, you'd want to fix that, but as a short term measure going back to the confirming a number rather than the force option would be appropriate. Is certainly not good to have an override that's routinely used.
  - jeffbee 6y ago
    The way we approached this on my SRE team was semi-manual with improved ergonomics. We embedded the live traffic graph in the turndown tool, so it would be right in your face before you took the destructive action. Of course it was always possible to go one level down on the tooling and do everything manually, but it wasn't the usual way.
    - scottlamb 6y ago
      Seems reasonable, but as you might have seen, rossjudson did accidentally-ish go to a lower layer: he wrote "never 'borg' when you meant to 'borgcfg'". And you're still relying on someone actually looking at the graph in their face which isn't as sure a thing as it'd be if they had to echo something back as Rachel is advocating for. (For the benefit of non-Googlers/Xooglers: borg is a lower-level tool mostly used when everything else has gone wrong and borgcfg is a higher-level, more routine tool. These days people often layer things on top of that as well, because we love piling up abstraction layers. This approach is completely successful because abstraction layers never leak and solve every problem without making anything hard to debug at all. /s) In my ideal world, even the lowest layer a human ever uses would do safety checks by default. Eg, imagine if the job specification included "query this safety check service on change" and the borg tool (as part of querying the existing job on a cancel/rm command) discovered that and honored it. Most people/jobs would use a safety check that fails taking down a job unless the load balancer reports all relevant services have that job drained. The safety check service could also specify a confirmation prompt (similar to what Rachel is advocating) that could be customizable (like qps or percent of global capacity rather than just number of tasks). The safety check would be effective no matter what layer you use, and there'd be no good reason to use one that would cause prompt fatigue. The outage rossjudson described (and I know he's not the only one who has done exactly this!) would have been avoided.
RobRivera 6y ago
Having babysat my fair share of critical clusters, i support this advice
Darkphibre 6y ago
Reminds me of when the Fortune 50 company (150k employees) I worked for rolled out new firewall restrictions that blocked the DNS port. To all machines. Employee and servers alike. Yes. Including the DNS servers. Took them a day or two to work out how to roll that one back.
- zamadatix 6y ago
  The first use of a new security product my manager insisted we roll out (as a duplicate to an existing tool from another group) was to quarantine a change in a system file that seemed to be spreading through all of the PCs. Except the change was to quarantine explorer.exe which was being changed with a patch that just got pushed out. The net result was about 6 hours of the desktop group wondering "why the hell are all of the PCs not logging in right after this patch" followed by about a month of rolling tickets from seldom used computers that had just been powered off. His excuse was it only showed a file hash in the main screen and you had to view details to see the name plus he had a 3 day change open to roll out the system. Never understood how he got away with that one but such things did catch up to him about 2 years later.
woliveirajr 6y ago
Once I had to deal with some software-RAID in Linux (mdadm it is), around 2007. There was some -force option that would just print information explaining what it would do and, to perform the real action, you needed to type another flag (that should never be revealed). Edit: added name of software
temporallobe 6y ago
This is similar to a UI solution a colleague and I came up with. The action the user could kick off was unstoppable and irreversible (a large batch job), and it seemed like even a confirmation prompt was too easy to simply click through. So we had the UI present a modal dialog asking the user to type in a specific word in all caps to confirm the action. Worked like a charm.
- D-Coder 6y ago
  I did a similar thing with a Star Trek program many years ago. One of the commands (22? 23?) was to detonate the warp engines in the hope of taking the enemy with you. After hitting the wrong number once, I added a confirmation that presented a random six-digit number that you had to enter before it accepted the command.
aqme28 6y ago
Nitpicking > "This might be as simple as printing the number with your locale's version of numerical separators, like "123,456" or "123.456" or "123 456" or whatever else you might use where you are. The trick is then to NOT accept that as input, but instead demand that they remove the separator and jam it in as just digits. " It's easier to just strip non-digit characters than to parse the input for them and respond accordingly. This is a confirmation step with basically a checksum, so you're not going to get many false positives.
- Kerrick 6y ago
  Stripping the non-digit characters would allow "123,456" to validate instead of only accepting "123456" -- which defeats the whole purpose of printing the number with numerical separators (to prevent copy/paste).
  - aqme28 6y ago
    If you're worried about copy-paste, make it a random code.
lqet 6y ago
Github has been doing this for quite a while know when you try to delete a repository - you have to type in the exact repository name to confirm.
- bmaupin 6y ago
  Which I always mindlessly copy and paste...
  - coder543 6y ago
    If you really think that’s an issue, pasting could be disabled for that input field. Would that make you happier? It hasn’t been an issue for me, since repo names aren’t usually super long and onerous to type.
    - gruez 6y ago
      > If you really think that’s an issue, pasting could be disabled for that input field. Would that make you happier? many (most?) HN users probably have that disabled, because too many sites abuse it to block password managers, for "security reasons" .
      - coder543 6y ago
        I disagree with the many/most. Many/most are probably using uBlock Origin, which doesn’t try to prevent things like blocking pasting (to my knowledge). I’m sure some are using NoScript-like features... but that’s not the same as specifically preventing websites from preventing paste. It’s just a sledgehammer. I can’t name an extension to do that one task (and/or similar tasks) off the top of my head, and I’m reasonably familiar with discussions in these parts. uBlock Origin is known to be very popular, unlike an obscure “allow paste” extension. But, that’s just like, my opinion... as they say. The point I was making is that copying and pasting seems like more effort than just typing the repo name. Do you commonly encounter long, inscrutable repo names? Do you delete repos frequently enough to have built up the habit of copying and pasting the repo name into the delete box? If it is common enough, disabling paste would actually benefit the user based on the premise of the article.
wotton 6y ago
Marketo, the marketing automation platform, does this when you try to do things to large data sets, very useful.
wotton 6y ago
Marketo, the marketing automation platform, does this when you try to do things to lots of people, super useful.
bnastic 6y ago
Promise Pegasus (thunderbolt storage) comes with a GUI that does the same thing - to shut it down you have to type “CONFIRM” before clicking the button
outworlder 6y ago
> 1221425541 machines will be affected "Do you care? (Y/N)" Cattle, people. Not pets. Just make sure you don't hit all machines simultaneously and are rolling, instead. Since the post is talking about automation anyway, assume that any machine that can go down will go down. Ensure that any such disruption will be minimal. Oops, you just killed the production database? Whatever, who cares, it has just failed over anyway (or, for a distributed one, a new node was elected, data started replicating, etc). If one considers having to SSH to a machine to be an anti-pattern, it's amazing how much crap goes away. In the more generalized case, where it's not about machines, then it makes more sense. Maybe you are running a query that's going to perform updates across multiple clusters. It still should not be done by hand with direct production access - unless you are in the middle of a declared (and urgent!) incident and everything is on fire. In which case there's a bunch of people watching over your shoulder (or more likely, screen sharing in a conference call). The same job you have (hopefully) run in QA you should be able to re-target to production. Make the question just be a way to "unlock" your automation - for instance, by not copying credentials or environment information until the proper confirmation has been received. One should still have an escape hatch for when (not IF) things go wrong.
- joshuamorton 6y ago
  Killing all of your cattle is still a concern.
tialaramex 6y ago
So, related obviously correct designs: 1. Git's Force-with-lease. Git push's "force" is too powerful, you will likely regret this much power, but it's tempting. So force-with-lease is the same power but conditional on you telling git what exactly the state was that you're overriding. This has two benefits, one is like Rachel's, it is an opportunity for a human to stop for a moment and consider, wait, why are we overriding this state? To find out what it is we might as well read... oh the state says it's an "emergency fix. Call Jerry". Maybe, just maybe, I ought to call Jerry before I force overwrite it? But the other is about race conditions which Rachel doesn't specifically address. If you are very careful to check that the state you want to overwrite with force is indeed a state that should be overridden, nothing prevents it meanwhile changing and then you overwrote state you didn't even know existed. But force-with-lease fixes that because your lease won't match. I believe Force-with-lease is a pattern that ought to be far more widespread. I've used several configuration management tools that let somebody say "Temporarily don't mess with config on these machines" and some of them let you write a reason like "James is rebuilding the RAID arrays" but none of them have that force-with-lease pattern that would be let me say "I know James is rebuilding the RAID arrays, this change must happen anyway but if anything else is blocking the change then reject it and let me know". 2. Prefer Undo to Confirmation. If the computer can undo the action, even if that's a bunch of work and you'd rather not bother, put that work in and enable undo. Humans always know they "really" wanted to do the thing you're asking them to confirm so it's somewhat futile to ask, but they often realise they didn't want to afterwards and will undo it if you make that possible. Not everything can be undone. Undo factory reset isn't a thing. But lots of things you can't undo it was just laziness, try to do better in your own software. Your users (which might include you) will be grateful.
- coder543 6y ago
  Related but semi-random: it slightly annoys me that force-with-lease goes through the entire effort of force pushing if it thinks the remote is identical to the local. It’s not going to change anything either way, and it could save me the second or two of waiting on it to do nothing. If local is already identical to the last known state of the remote, and I’m trying to force push, the actual error is that I didn’t edit the local branch in the way I thought I had when I decided it was time to force push. (I realize there is a possible error message case if the remote has changed... but I don’t feel like this command is the best one to use to discover whether the remote has changed, if you have no changes you actually intend to force push.)
ineedasername 6y ago
Oh god this would have saved me so much stress once. It was early in my career, and part of my duties was to run a merge/purge process on dupe records. I'd select the dupes for merge using a checkbox, but the vendor's interface for this just had a "confirm" button. So, I confirmed. However I'd selected the "select all" box and.... confirmed. Merging every. single. record. into one (1) record. I was fortunate, the vendor was able to roll back the changes, and nothing was lost. I also had a very good mentor-like boss who avoided reaming me out before we knew if there was a solution or not, and when there was he simply told me "I'm sure you've learned your lesson, but don't do that again."
cle 6y ago
Even better than you confirming your own action, is someone else confirming it. If the stakes are high, require two people to turn the keys, instead of just one.
gitgud 6y ago
> "I've worked at a few places that had a large number of Linux boxes. I'm talking about well over a million." A few places!? What is an example of this?
- throwawaygh 6y ago
  My guess: Rackspace, Google, and Facebook.
diebeforei485 6y ago
I'm thinking this could also be useful for cases where colleges mistakenly email all applicants saying they'd been accepted, when they in fact had not been.
mcintyre1994 6y ago
AWS sometimes does something similar to this like “enter the name of the thing you’re trying to delete to confirm”. I think it makes sense because you can have such a huge difference between how much you care about certain s3 buckets or CloudFormation deploys etc. In true AWS fashion it’s inconsistent between services though.
- nucleardog 6y ago
  To their credit, even if it’s unintentional, every time one of those screens pop up I have to stop and think about what I’m doing because every screen wants something different from me!
rcarmo 6y ago
This reminded me that a few years back I worked at a place where (notoriously) Puppet would occasionally go over some random box and remove access to people, just because. Or to all the machines, on one occasion. (It was actually some sort of race condition when we massively updated per-project access permissions and asked for SSH keys to be redeployed, but it was annoying as heck, and sure to happen whenever you really needed to access that particular machine.)
regularfry 6y ago
I've seen this implemented as "Please type: My username is $USERNAME and I will not cry over spilt milk" but that was more to guard against support tickets.

554 points by vii 6y ago

Similar idea as GitHub's "type the exact name of this repository if you want to delete it" confirmation dialog. Maybe that's really what you want to do, but in case that's not actually what you meant to do, having a few extra hoops to jump through seems like a good idea.

Hokusai 6y ago
> having a few extra hoops to jump through seems like a good idea. I think that there is more to that. You need to consciously type the name of the repo that you want to remove. Windows used to add a lot of jumps to get something done, and the result was mindless clicking the "yes" button and realizing 1 second later that you deleted important information. That extra hoops need to be cognitive meaningful.
- Cthulhu_ 6y ago
  Yes, and infrequent; the main issue with Windows (Vista mainly) was that it appeared far too often. Even with 7, when you're setting it up for the first time for example, I think it shows up too often. Same with Terms & Conditions. If you want your customers to truly have read and understood them, you have to show them a short quiz at the end of it. You're required to do a quiz in Europe nowadays if you want to engage in stock trading.
wjdp 6y ago
Do you type the repo name, or just copy/paste or select/middle click it? Half of me would want them to put `user-select: none` on that text. The other half has to archive 10+ repos and would hate that!
segfaultbuserr 6y ago
Some disk management software also has "type the exact label of this partition to reformat it" to prevent accidental data loss.
edanm 6y ago
That's what I thought of immediately as well! I've seen that pattern in a few other places too, and I always think it's a really good UX choice.

xamuel 6y ago

I wish it were possible for similar prompts to appear before all sorts of policy-makers and bureaucrats. "It appears you are about to institute a policy which will require 400 million patients to sign an additional waiver every time they visit a clinic, this will waste a total of 354,921 human hours within the next year alone. Please type 354,921 to proceed."

jackhack 6y ago
closely related: the Paperwork Reduction Act of 1995 https://digital.gov/resources/paperwork-reduction-act-44-u-s-c-3501-et-seq/ https://digital.gov/resources/paperwork-reduction-act-44-u-s... it requires the office of management and business to calculate the impact of records-keeping requirements impact on time and privacy, among other things. I do not believe it has resulted in a reduced recordskeeping burden. For the most part I simply see an estimate of how long it will take to complete my tax forms and permits, on the form itself. Perhaps others have different views.
- mulmen 6y ago
  Hard to say, knowing the cost of a new process could have informed a new design or requirements. We don’t know what the other path held. But I believe in general having more information allows us to make better decisions so this is a good act.
Joker_vD 6y ago
Yeah, it's quite surreal. "Hey, privacy is important, so let it make so that to handle people's private data, you'll need a permission from them". All right, now whenever you try to e.g. send a (paper) mail, you have to sign the waiver that yes, you do allow the post office to see and handle your name and your mail address. Not only that, all such waivers seem to be written as "I hereby allow <insert the legal entity> to handle my private data in whatever way they want to", so we're back on square one, just with more perfunctory paperwork required.
mulmen 6y ago
How do you know it was a waste? Maybe that was time well spent.

roydivision 6y ago

Reminds me of the proposal to keep the nuclear launch codes inside the body of an innocent volunteer, so the President would have to kill the person to get the codes. https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-lau.html https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-...

chrisseaton 6y ago
I've never understood this idea. If you believe we should never use nuclear weapons, then don't have them at all. If you believe there is a case where it may be moral and rational to use nuclear weapons, why would you want to put a potential barrier in the way of their use? You could have a situation where everyone was agreed to use them but the president was physically unable to harm the aide to use them. You can know that something is the right thing to do but not have the courage to physically harm someone to do it. An interlock that you may not be able to unlock for reasons unrelated to the task at hand is a bad interlock.
- gumby 6y ago
  > I've never understood this idea. It's an attempt to make an abstraction concrete. Think of it as the trolley problem in real life. Stalin is famously supposed to have said, "one death is a tragedy, 100,000 is a statistic". Cynical or not it is how humans think. > If you believe we should never use nuclear weapons, then don't have them at all. Strategic game theory and Mutual Assured Destruction depend on the possibility that the other guy will use them if you do, and may be the only way to prevent their use. Interestingly this is one reason why you want the other guy to know your procedures, capabilities, deployments etc. Secret weapons have no deterrent value.
  - chrisseaton 6y ago
    > Think of it as the trolley problem in real life. Well exactly... doesn't that show you that it's a bad idea? People don't know if they could bring themselves to throw the switch even if everyone thinks it makes rational sense. You're taking a rational, well-considered, strategic decision... and making the interlock a messy personal emotional one unrelated to the actual issue at hand. That sounds like the wrong way around to be doing things?

educationcto 6y ago

Terraform prints out the number of resources changed and at least requires a "yes" to proceed. Not quite as onerous as described but at least prevents some type of fat-fingering. Basically all changes with Terraform are risky as they usually involved bringing up and down infrastructure. Terraform will perform the following actions: # google_compute_instance.vm_instance will be created + resource "google_compute_instance" "vm_instance" { + ... <more> Plan: 2 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes

brodouevencode 6y ago
IaC is a real time saver, but inherently dangerous.
caymanjim 6y ago
This is exactly the problem the author is referring to. With Terraform, you always type "yes" to proceed, so it turns into muscle memory. You stop reading the output, and you're already typing "yes" before you even see the prompt. Terraform's output is also verbose, and many changes show up as "1 to add, 0 to change, 1 to destroy" because they don't separately list a "replace" category. It's pretty bad; you've got cognitive overload, confusing output summary, and a predetermined continue answer. And this is often an action you're performing under duress. I've been bitten by it plenty of times.

jancsika 6y ago

It would be neat to print out an esoteric error that gets a single result in Google, where the "forum" in the result has a rando answer about using a certain esoteric flag. Then you search the logs to see who is trying the command with the esoteric flag and "fix the glitch with payroll" for those employees.

JoeAltmaier 6y ago

Makes it harder to nest that command inside a script - you have to parse out the number and paste it back? Or do I misunderstand - should it still prompt the user in the middle of the process when that step arrives? That would be problematical if it were included in a web page or whatever.

ccakes 6y ago
The very point of this is to make it difficult to do what you’re describing. If the tool could potentially touch a large number of machine, even if you’re super sure you got it right you should still prompt the user
- JoeAltmaier 6y ago
  Or write a script that carefully calculates the number of machines and gets it right. I guess you wouldn't use this prompting script then?
larrik 6y ago
I believe this would be as part of the script you are writing, not the scripts you are calling.
rad_gruchalski 6y ago
Hopefully there’s an API to fetch that count :)

csmattryder 6y ago

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling https://en.wikipedia.org/wiki/Pointing_and_calling

MaxBarraclough 6y ago
Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.
- blantonl 6y ago
  Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.
  - staunch 6y ago
    And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
    - nemosaltat 6y ago
      Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.

larrik 6y ago

I've seen this sort of thing in a few places, and I really do think it's a great idea.

deleted 6y ago

[deleted]

Cthulhu_ 6y ago

I for one can't fathom any organization managing a million devices / servers / VMs / whatnot. I'm having enough trouble with one, and my biggest employers had maybe a few dozen at best, and they already had a dedicated ops team that worked mainly with infrastructure-as-code.

sidpatil 6y ago

Hmm, it's conceptually like a combination of a CAPTCHA and a launch code.

jaclaz 6y ago

Side question. How many/which companies have more than one million Linux machines?

notacoward 6y ago
At least Facebook (where OP worked), Amazon, Google, and Microsoft. Probably Netflix, maybe Apple. There might be a couple more, but no more than that because we've already accounted for a pretty high percentage of worldwide shipments for servers, disks, etc. Fun fact: when you're that big, your demand creates its own inflation and you have to consider that in projections.
- jaclaz 6y ago
  Thanks, so a handful at most, and the "usual" ones, I always thought that those companies keep their machines connected in (redundant) "sets" and that a command affecting all of them was more a case for "never" rather than "once in a while".
  - jeffbee 6y ago
    Google, at least, has a thing that is supposed to prevent widespread disruption at the machine level, called the "Safe Removal Service"[1]. This is a good idea that in practice isn't perfect. If you write a tool that does not consult SRS, or your service doesn't declare a SRS policy, there can be surprises. A particular outage that I will never forget took out Gmail delivery worldwide in an instant, because the change was not expected to be disruptive and therefore did not integrate with SRS. As it turned out the change disabled the machines where it was applied, and the process of selecting a subset of machines to canary the change was not independent of the way in which Gmail assigns services to machines, so in the space of a few seconds they created a global outage. https://twitter.com/bgrant0607/status/1134536670504554496 https://twitter.com/bgrant0607/status/1134536670504554496
- kube-system 6y ago
  If by "machine" we also mean things outside of a 19" rack, I would wager that large telecoms probably have way more devices running Linux than FAANG. Imagine the network of cable modems that Comcast alone must operate. What percentage of their 28+ million broadband customers rent Comcast owned/managed modems? Almost all of them except the tech-savvy crowd? And that's just one device type.

vsnf 6y ago

I do this with a git pre-push hook to the main branch of my repositories. It displays a prompt in red and forces me to type in the name of the branch. The result of one too many mindlessly accidental pushes.

willvarfar 6y ago

I am so adding this to a query api I have, where its all too easy to leave off constraints and end up asking for massive data sets by mistake. Thinking I can probably enhance it by forcing the user to type in the number as text rather than numeric, so they can't cut-n-paste. Kind of force them to type in "I am sure I want all data ever" or something.

recursive 6y ago
I don't think this is useful for an api. This is only useful when humans are the direct user of the component. Automated users, like those of an API will dutifully provide the required safety value.

nemo1618 6y ago

Notably, Discord does something like this when you @everyone in a large channel: "You're about to push a notification to 12,000 people, are you sure you want to do that...?"

pwinnski 6y ago
Sounds like a yes/no answer is expected? If so, that is exactly what Rachel is suggesting is not enough.
- jerf 6y ago
  In this case, usually the very fact that a popup unexpectedly popped up is enough. I use Konsole as my main shell, and like several other shells now it has a "You're about to paste 100KB, yes/no?", and I don't mindlessly click "yes" because it is already a "cache miss" to see that dialog at all.
- justincredible 6y ago
  She also suggests localizing the number will prevent copy paste somehow.
raverbashing 6y ago
Slack should take a note of this. Especially for rogue @here notifications

eznzt 6y ago

Debian already does this, it asks you to type something like "yes do as I asked" if you want to remove a package that is considered to be part of the core.

vondur 6y ago

That may have helped when Emory University's IT dept. accidentally sent a wipe and reformat command using Microsoft's SCCM to all of the Windows computers and servers on campus back in 2014. https://it.slashdot.org/story/14/05/17/051214/emory-university-sccm-server-accidentally-reformats-all-computers-campus-wide https://it.slashdot.org/story/14/05/17/051214/emory-universi...

jasonpeacock 6y ago

Raskin talks about the futility of this in his book The Humane Interface. Basically, what happens is the brain switches operating context from "I want to do something" to "resolve this interruption (confirmation box)" and you don't relate the one to the other - you're so focused on getting rid of the interruption that the original task is forgotten until after the interruption is gone. Then you switch back to the original task that had been interrupted by the confirmation box and then you realize you made a mistake. It's much better to engineer "undo" ability into systems - like delaying commands (GMail's "Undo Send" does this), or caching previous state, etc.

andrewflnr 6y ago
That's exactly why it's not a "confirmation box", but requires you to slow down and think for half a second. She even talked about mitigating copy-paste, which is the next obvious way people could habituate. Also, while undo is great, it's not always technically feasible. The tools in question are basically for modifying the layer that implements undo for your end users, and are often themselves fundamentally irreversible. Undo for raw hard disks involves forensic analysis at best.
- robaato 6y ago
  Or you have commands which randomly reverse the meaning of the confirmation prompt: Continue: yes or no? Don't continue: yes or no? As long as operators know to expect this, they also know to wait and actually read the prompt before answering (as in turn of auto reaction)...
- jasonpeacock 6y ago
  The problem (I probably didn't paraphrase Raskin well) is when you slow down & think for a half a second, you context switch from "I need to do operation" to "I need to make this dialog box go away". No matter what tasks are required to make the dialog box go away - doing math, retyping a message, clicking a randomly ordered box - that becomes the top task in your head and you "forget" about the original task until you finish this task. Once you resolve the interruption, you switch context back to the original task and then you still have that "oh crap" moment. Yes, sometimes undo is very difficult, and can require a system designed to support that ability as a first-class feature from the start. Many systems you can perform rollbacks, but there are definitely destructive actions - in which case you should have test stacks to validate your actions in advance, and peer review. (e.g. dual keys to launch the missiles)

Ayesh 6y ago

I have an old laptop with a dead battery, and for a BIOS upgrade, it prevents me from updating without 50% battery. I have to type "danger" to bypass this restriction, and I thought it was pretty cool. Another good UI pattern is in Firefox, that it disables the Run button on downloads for a few seconds.

duskwuff 6y ago
Disabling the "run" button for a few seconds was actually done to mitigate another risk -- sites cueing the user to click in a particular location, then triggering the confirmation dialog with the "run" button right where the user was about to click.

illumin8 6y ago

This is a great idea, and I'd like to point out that having such a system in place would have prevented one of the largest Internet outages in recent memory - the Amazon S3 outage in 2017: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/ > At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

zedpm 6y ago
It's kind of funny, since various operations performed in the AWS web console use this model (e.g. type the name of the resource you're trying to delete). As an organization, they're aware of this approach and think it's useful, but (presumably) didn't use it in their own internal tooling.
- audience_mem 6y ago
  Perhaps those were added after they learnt their lesson.

harikb 6y ago

I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.

Xophmeister 6y ago
We've been known to use something like --yes-i-really-mean-it-this-time for really dangerous options. It's a like built-in solemnisation step.
austinl 6y ago
I like this format in general, since it communicates the command is severe/irreversible. Heroku implements a similar confirmation when performing destructive actions. Commands require your to pass a `--confirm ${APP NAME}` flag, so the original command itself does nothing. Of course, this doesn't prevent you including those flags in makefiles, etc. I once dropped a table in a side project by accident because I took the wrong tab autocomplete suggestion in a makefile.
yobert 6y ago
I did this with "--im-not-scared" for production mode :D
- csours 6y ago
  We had one that required "BADIDEA" to run
jrumbut 6y ago
Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.
- dingaling 6y ago
  I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o

jerf 6y ago

https://news.ycombinator.com/item?id=24907002 https://news.ycombinator.com/item?id=24907002 Looks like https vs http link.

dang 6y ago
We've merged the threads now. Thanks!

Animats 6y ago

Yes. Github does that when you delete a repository. You have to confirm by typing in the name of the repository you are deleting.

tigger0jk 6y ago

I've typically used pdsh https://github.com/chaos/pdsh https://github.com/chaos/pdsh for these types of commands, and I don't think they have any such safety options. The only protection is to be wracked with fear whenever you type pdsh. Obviously this fear wanes with use, and eventually you don't think about a command for long enough before you do it and hit enter on a regrettable one.

luhn 6y ago

One of the largest AWS outages to date was caused by a scenario like this. [1] A mistyped commanded removed too many servers from an S3 subsystem, overloading the remaining servers and crashing the subsystem. The failure snowballed until the entire S3 region was down, which then caused issues with dependent services like EBS, ALB, and Lambda. They couldn't even update the status page because that also depended on S3. [1] https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

HenryKissinger 6y ago
I remember that. The AWS dashboard was all green checkmarks... because the red checkmarks icons the dashboard was supposed to display were stored inside the crashed servers.
jodrellblank 6y ago
>"overloading the remaining servers and crashing the subsystem. The failure snowballed until" the entire Eastern Seaboard was without power? https://youtu.be/XetplHcM7aQ?t=693 https://youtu.be/XetplHcM7aQ?t=693 (James Burke's Connections, ref. cascading power cut 1965)

TravHatesMe 6y ago

Reminds me of a study done where a test was given with questions that weren't difficult but likely to make a silly error. Around 85% of participants got at least one question wrong, but when they repeated the same test with a difficult-to-read font, that number dropped to ~25% or so. That's another way to make your brain work, use a terrible font.

apricot 6y ago
> That's another way to make your brain work, use a terrible font. And suddenly my complex analysis prof who wrote his exams in Comic Sans is vindicated!

andrewfromx 6y ago

i've done this before by displaying unix epoc and asking the user to copy/paste that value WITHIN a 3 second window as an env var. i.e. if you up arrow and run same TIMESTAMP=1603827448 ./foo it won't work because 1603827448 is now way too old.

myroon5 6y ago
One of the main benefits is explicitly acknowledging relevant context. Timestamps don't provide additional relevant context

ComodoHacker 6y ago

In role-playing games, it's a common practice to confirm deletion of your character by typing in some word, like 'delete' or character name.

kbenson 6y ago

This is a topic near and dear to my heart, as I'm often that person arguing to make some slightly less automated because the small trade-off in time is insurance against some of the worst mistakes you can have. Automation to the point of removing humans leads to stupid problems that a human wouldn't make if they looked at what was going on. So we automate tot he point where we minimize human contact, presenting a summary of actions that as humans we can apply our wonderful brains to and prevent those problems. Except some percentage of the time we don't actually pay attention, and depending on how the human interaction was introduced instead of complete automation, some percentage (or multiple!) of errors still sneak through. Automation to the point of minimal human contact where you assume the human will read the presented information and make an informed decision doesn't work. The point is that we want a human to understand what is being asked, so taking some step to ensure they do understand is warranted. It will never be perfect, but adding steps like she proposes are definitely a step in the right direction, IMO.

bronco21016 6y ago

It amazes me that something like this can be done by a single person. In aviation any time input is given to the machine, it's entered by one human (typically pilot flying) and then verified by the other human (typically pilot monitoring) before being committed to or executed. For example... when a new altitude is assigned by ATC, say FL300, the pilot flying will spin it in the selector window and keep his hand or finger there until the second pilot agrees with and confirms the selection by reading FL300 out of the selector window. I know there are meat bags in these giant tubes so that changes attitudes towards safety etc. However, it seems to me that when organizations start putting the power to halt nearly the entire business in the hands of one person, there should be some slightly different attitudes. A breaking change in a million servers could easily cost hundreds of thousands or maybe even millions in lost revenue or employee productivity. I'm just an outsider though. Perhaps this level of attention is practiced at some shops. It's just interesting to me how in some fields we settle on pretty uniform standard practices whereas others are seen as non-human-life threatening so it's just shoot first, ask questions later.

rachelbythebay 6y ago
Best practice for using the "weaponized" version of the tool when you had powers to actually hit all of them at once was to paste the command into IRC and get some of your fellow peeps to eyeball it and make sure it was sane. <me> team: hey, sanity check this please: hsh -A "dumb_thing && other_thing --foo --bar" <teammate> shipit [ I type the command ] <me> ok, running as job 1234 The last part was a courtesy done so that they could watch the progress of it too without having to dig to find my request. It also meant they could kill it easily if something went wrong and they couldn't raise me for some reason. Tools like this are best used outside the solo realm.
- im3w1l 6y ago
  I think an automated tool would be preferable since there is no 100% foolproof guarantee that what you type in irc is the same as what you type in the terminal.
crispyambulance 6y ago

konjin 6y ago

Finally the Roman numeral converter I programmed in university will be useful.

gabeio 6y ago

I do like this idea, this is I assume why github makes you type the repo name out in full. I wish AWS followed suit, when deleting any RDS (database) instance on AWS all you have to type is "delete me"... very easy to copy and paste as well as just know what you need to type and be on autopilot. I have even poked support about it and their response was underwhelming.

remram 6y ago

A similar system is molly-guard [1], which replaces the reboot/halt/poweroff/... commands with scripts that make you type in the name of the machine before proceeding. Avoids shutting down the wrong machine because you forgot where you SSH'd. [1]: https://manpages.debian.org/buster/molly-guard/molly-guard.8.en.html https://manpages.debian.org/buster/molly-guard/molly-guard.8...

b6z 6y ago
Many years ago, I made that mistake two or three times, rebooting the wrong machine. Since then, I use molly-guard on all my remote machines. Never happened again.

heelix 6y ago

Back in the Spiderman 2 days, I worked for a content management company that was supporting a really, really big website. I believe they were playing host file games for Stage/Prod. Was in the room on when they demo'ed something, did a restart of the system - and every pager in the room went off. Yah...

rossjudson 6y ago

This resonates with me. Years ago I took down a service in a cell accidentally (Googlers might empathize: never 'borg' when you meant to 'borgcfg'). If I had been asked to enter the exact number of tasks I was about to nuke, I might have thought twice ;)

scottlamb 6y ago
I've certainly deliberately downed an enormous number of tasks, though, as part of a cluster turn-down. I love the technique of requiring the operator to echo a key fact, but in the case you're describing I think the key fact is not how many tasks but that that they're serving live traffic. So: * You could ask the operator to echo the qps figure...but really any number other than zero is likely to be an error, so it can just error out in that case without needing the confirmation. * Even if it is serving zero qps now, if it's not explicitly drained at the load balancer, downing it is likely to be a mistake. So even better to check that. Only once in my career have I taken down jobs serving live traffic. (They were serving 100% errors.) It was deliberate, but even so I wouldn't have minded having to supply a --yes-i-know-im-downing-live-jobs. edit: and if for some reason my assumption is wrong and downing undrained things becomes routine...well, you'd want to fix that, but as a short term measure going back to the confirming a number rather than the force option would be appropriate. Is certainly not good to have an override that's routinely used.
- jeffbee 6y ago
  The way we approached this on my SRE team was semi-manual with improved ergonomics. We embedded the live traffic graph in the turndown tool, so it would be right in your face before you took the destructive action. Of course it was always possible to go one level down on the tooling and do everything manually, but it wasn't the usual way.
  - scottlamb 6y ago
    Seems reasonable, but as you might have seen, rossjudson did accidentally-ish go to a lower layer: he wrote "never 'borg' when you meant to 'borgcfg'". And you're still relying on someone actually looking at the graph in their face which isn't as sure a thing as it'd be if they had to echo something back as Rachel is advocating for. (For the benefit of non-Googlers/Xooglers: borg is a lower-level tool mostly used when everything else has gone wrong and borgcfg is a higher-level, more routine tool. These days people often layer things on top of that as well, because we love piling up abstraction layers. This approach is completely successful because abstraction layers never leak and solve every problem without making anything hard to debug at all. /s) In my ideal world, even the lowest layer a human ever uses would do safety checks by default. Eg, imagine if the job specification included "query this safety check service on change" and the borg tool (as part of querying the existing job on a cancel/rm command) discovered that and honored it. Most people/jobs would use a safety check that fails taking down a job unless the load balancer reports all relevant services have that job drained. The safety check service could also specify a confirmation prompt (similar to what Rachel is advocating) that could be customizable (like qps or percent of global capacity rather than just number of tasks). The safety check would be effective no matter what layer you use, and there'd be no good reason to use one that would cause prompt fatigue. The outage rossjudson described (and I know he's not the only one who has done exactly this!) would have been avoided.

RobRivera 6y ago

Having babysat my fair share of critical clusters, i support this advice

Darkphibre 6y ago

Reminds me of when the Fortune 50 company (150k employees) I worked for rolled out new firewall restrictions that blocked the DNS port. To all machines. Employee and servers alike. Yes. Including the DNS servers. Took them a day or two to work out how to roll that one back.

zamadatix 6y ago
The first use of a new security product my manager insisted we roll out (as a duplicate to an existing tool from another group) was to quarantine a change in a system file that seemed to be spreading through all of the PCs. Except the change was to quarantine explorer.exe which was being changed with a patch that just got pushed out. The net result was about 6 hours of the desktop group wondering "why the hell are all of the PCs not logging in right after this patch" followed by about a month of rolling tickets from seldom used computers that had just been powered off. His excuse was it only showed a file hash in the main screen and you had to view details to see the name plus he had a 3 day change open to roll out the system. Never understood how he got away with that one but such things did catch up to him about 2 years later.

woliveirajr 6y ago

Once I had to deal with some software-RAID in Linux (mdadm it is), around 2007. There was some -force option that would just print information explaining what it would do and, to perform the real action, you needed to type another flag (that should never be revealed). Edit: added name of software

temporallobe 6y ago

This is similar to a UI solution a colleague and I came up with. The action the user could kick off was unstoppable and irreversible (a large batch job), and it seemed like even a confirmation prompt was too easy to simply click through. So we had the UI present a modal dialog asking the user to type in a specific word in all caps to confirm the action. Worked like a charm.

D-Coder 6y ago
I did a similar thing with a Star Trek program many years ago. One of the commands (22? 23?) was to detonate the warp engines in the hope of taking the enemy with you. After hitting the wrong number once, I added a confirmation that presented a random six-digit number that you had to enter before it accepted the command.

aqme28 6y ago

Nitpicking > "This might be as simple as printing the number with your locale's version of numerical separators, like "123,456" or "123.456" or "123 456" or whatever else you might use where you are. The trick is then to NOT accept that as input, but instead demand that they remove the separator and jam it in as just digits. " It's easier to just strip non-digit characters than to parse the input for them and respond accordingly. This is a confirmation step with basically a checksum, so you're not going to get many false positives.

Kerrick 6y ago
Stripping the non-digit characters would allow "123,456" to validate instead of only accepting "123456" -- which defeats the whole purpose of printing the number with numerical separators (to prevent copy/paste).
- aqme28 6y ago
  If you're worried about copy-paste, make it a random code.

lqet 6y ago

Github has been doing this for quite a while know when you try to delete a repository - you have to type in the exact repository name to confirm.

bmaupin 6y ago
Which I always mindlessly copy and paste...
- coder543 6y ago
  If you really think that’s an issue, pasting could be disabled for that input field. Would that make you happier? It hasn’t been an issue for me, since repo names aren’t usually super long and onerous to type.
  - gruez 6y ago
    > If you really think that’s an issue, pasting could be disabled for that input field. Would that make you happier? many (most?) HN users probably have that disabled, because too many sites abuse it to block password managers, for "security reasons" .
    - coder543 6y ago
      I disagree with the many/most. Many/most are probably using uBlock Origin, which doesn’t try to prevent things like blocking pasting (to my knowledge). I’m sure some are using NoScript-like features... but that’s not the same as specifically preventing websites from preventing paste. It’s just a sledgehammer. I can’t name an extension to do that one task (and/or similar tasks) off the top of my head, and I’m reasonably familiar with discussions in these parts. uBlock Origin is known to be very popular, unlike an obscure “allow paste” extension. But, that’s just like, my opinion... as they say. The point I was making is that copying and pasting seems like more effort than just typing the repo name. Do you commonly encounter long, inscrutable repo names? Do you delete repos frequently enough to have built up the habit of copying and pasting the repo name into the delete box? If it is common enough, disabling paste would actually benefit the user based on the premise of the article.

wotton 6y ago

Marketo, the marketing automation platform, does this when you try to do things to large data sets, very useful.

wotton 6y ago

Marketo, the marketing automation platform, does this when you try to do things to lots of people, super useful.

bnastic 6y ago

Promise Pegasus (thunderbolt storage) comes with a GUI that does the same thing - to shut it down you have to type “CONFIRM” before clicking the button

outworlder 6y ago

> 1221425541 machines will be affected "Do you care? (Y/N)" Cattle, people. Not pets. Just make sure you don't hit all machines simultaneously and are rolling, instead. Since the post is talking about automation anyway, assume that any machine that can go down will go down. Ensure that any such disruption will be minimal. Oops, you just killed the production database? Whatever, who cares, it has just failed over anyway (or, for a distributed one, a new node was elected, data started replicating, etc). If one considers having to SSH to a machine to be an anti-pattern, it's amazing how much crap goes away. In the more generalized case, where it's not about machines, then it makes more sense. Maybe you are running a query that's going to perform updates across multiple clusters. It still should not be done by hand with direct production access - unless you are in the middle of a declared (and urgent!) incident and everything is on fire. In which case there's a bunch of people watching over your shoulder (or more likely, screen sharing in a conference call). The same job you have (hopefully) run in QA you should be able to re-target to production. Make the question just be a way to "unlock" your automation - for instance, by not copying credentials or environment information until the proper confirmation has been received. One should still have an escape hatch for when (not IF) things go wrong.

joshuamorton 6y ago
Killing all of your cattle is still a concern.

tialaramex 6y ago

So, related obviously correct designs: 1. Git's Force-with-lease. Git push's "force" is too powerful, you will likely regret this much power, but it's tempting. So force-with-lease is the same power but conditional on you telling git what exactly the state was that you're overriding. This has two benefits, one is like Rachel's, it is an opportunity for a human to stop for a moment and consider, wait, why are we overriding this state? To find out what it is we might as well read... oh the state says it's an "emergency fix. Call Jerry". Maybe, just maybe, I ought to call Jerry before I force overwrite it? But the other is about race conditions which Rachel doesn't specifically address. If you are very careful to check that the state you want to overwrite with force is indeed a state that should be overridden, nothing prevents it meanwhile changing and then you overwrote state you didn't even know existed. But force-with-lease fixes that because your lease won't match. I believe Force-with-lease is a pattern that ought to be far more widespread. I've used several configuration management tools that let somebody say "Temporarily don't mess with config on these machines" and some of them let you write a reason like "James is rebuilding the RAID arrays" but none of them have that force-with-lease pattern that would be let me say "I know James is rebuilding the RAID arrays, this change must happen anyway but if anything else is blocking the change then reject it and let me know". 2. Prefer Undo to Confirmation. If the computer can undo the action, even if that's a bunch of work and you'd rather not bother, put that work in and enable undo. Humans always know they "really" wanted to do the thing you're asking them to confirm so it's somewhat futile to ask, but they often realise they didn't want to afterwards and will undo it if you make that possible. Not everything can be undone. Undo factory reset isn't a thing. But lots of things you can't undo it was just laziness, try to do better in your own software. Your users (which might include you) will be grateful.

coder543 6y ago
Related but semi-random: it slightly annoys me that force-with-lease goes through the entire effort of force pushing if it thinks the remote is identical to the local. It’s not going to change anything either way, and it could save me the second or two of waiting on it to do nothing. If local is already identical to the last known state of the remote, and I’m trying to force push, the actual error is that I didn’t edit the local branch in the way I thought I had when I decided it was time to force push. (I realize there is a possible error message case if the remote has changed... but I don’t feel like this command is the best one to use to discover whether the remote has changed, if you have no changes you actually intend to force push.)

ineedasername 6y ago

Oh god this would have saved me so much stress once. It was early in my career, and part of my duties was to run a merge/purge process on dupe records. I'd select the dupes for merge using a checkbox, but the vendor's interface for this just had a "confirm" button. So, I confirmed. However I'd selected the "select all" box and.... confirmed. Merging every. single. record. into one (1) record. I was fortunate, the vendor was able to roll back the changes, and nothing was lost. I also had a very good mentor-like boss who avoided reaming me out before we knew if there was a solution or not, and when there was he simply told me "I'm sure you've learned your lesson, but don't do that again."

cle 6y ago

Even better than you confirming your own action, is someone else confirming it. If the stakes are high, require two people to turn the keys, instead of just one.

gitgud 6y ago

> "I've worked at a few places that had a large number of Linux boxes. I'm talking about well over a million." A few places!? What is an example of this?

throwawaygh 6y ago
My guess: Rackspace, Google, and Facebook.

diebeforei485 6y ago

I'm thinking this could also be useful for cases where colleges mistakenly email all applicants saying they'd been accepted, when they in fact had not been.

mcintyre1994 6y ago

AWS sometimes does something similar to this like “enter the name of the thing you’re trying to delete to confirm”. I think it makes sense because you can have such a huge difference between how much you care about certain s3 buckets or CloudFormation deploys etc. In true AWS fashion it’s inconsistent between services though.

nucleardog 6y ago
To their credit, even if it’s unintentional, every time one of those screens pop up I have to stop and think about what I’m doing because every screen wants something different from me!

rcarmo 6y ago

This reminded me that a few years back I worked at a place where (notoriously) Puppet would occasionally go over some random box and remove access to people, just because. Or to all the machines, on one occasion. (It was actually some sort of race condition when we massively updated per-project access permissions and asked for SSH keys to be redeployed, but it was annoying as heck, and sure to happen whenever you really needed to access that particular machine.)

regularfry 6y ago

I've seen this implemented as "Please type: My username is $USERNAME and I will not cry over spilt milk" but that was more to guard against support tickets.