13 ms·

This is the way. I wish this were taught in computer science class, development bootcamps, operations team onboarding, anywhere there is a procedure that is eve

by throwaway20371 5y ago

This is the way. I wish this were taught in computer science class, development bootcamps, operations team onboarding, anywhere there is a procedure that is even slightly complicated to automate. It is the absolute best solution there is. * Documentation of the entire procedure is contained in one place. No need to go sifting through 20 different sources of documentation. This lowers the human emotional barrier to "just get it done", as people will always avoid things they aren't comfortable/familiar with, or don't have all the steps to. This central point of documentation also enables rapidly improving the process by letting people see all the steps in one place, which makes it easier to fix/collapse/remove steps. * Automation in small pieces over time avoids the trap of "a project" where one or more engineers have to be dedicated to this one task for a long period of time. Most things shouldn't be automated unless there is demonstrably greater value in the cost of automating them than the cost of not doing so. Automating only the most valuable/costly pieces first gives immediate gains without sinking too much into the entire thing. * One unified "method" to encapsulate any kind of process means your organization can ramp up on processes easier, reducing overall organizational cost. * In the absence of any other similar process, you are guaranteed to save time and money. I would say that the only potential downside is if someone decides to "engineer" this method, making it more and more and more complicated, until it loses its value. KISS is a requirement for it to be sustainable.

csdvrx 5y ago
> only potential downside is if someone decides to "engineer" this method It can be engineered, if you follow a gradual process. On servers, I keep a log of what was deployed in a root directory following the sequential number _ goal format (ex: 00_partitions ... 90_web_server) It is not fancy, most of the logs are not even scripts: many are just ASCII text files, that will only be used as a checklist if the same "goal" has to be achieved again. For example, 00_partition may be "gdisk /dev/nvme0n1" followed by a copy-pasted list of the partitions and some quick description about why it was done that way. But that's on the first iteration only: the next iteration turns that into "do-nothing" script, the next iteration into a better script with basic checks (supporting both /dev/nvme0n1 and /dev/sda), then exception handling (if partitions already exist, etc), and so on: this gradual complexification process avoids the "premature optimization" of creating infrastructure-as-code for what you rarely need, while optimizing and fine tuning the parts you most often need. Someone will certainly mention Terraform, or Ansible, or something else - yes, they exist and they are nice, but if you are doing everything there, you are over-engineering and wasting time: not everything needs your equal attention! If you only install a webserver once in a blue moon, make a .txt checklist of the steps you followed. But if you leave and breathe nginx options and certificate deployment, fully automatize all that, including the obscure details of what may fail if you use let's encrypt with some specific DNS configuration! And if you don't know yet which is which, start small (a .txt checklist will cost you a few minutes) and the next time you find yourself doing the same thing, do it better using the previous artifact (the .txt file) to create a better one (a script, then a better script etc)
- chousuke 5y ago
  I would argue that all deployments (no matter how small) should have configuration management. In the simplest case, a deployment consists of: 1. Install packages 2. Install configuration files, possibly from templates. 3. Configure services to start on boot. This kind of automation is trivial to do with almost any tool, but there's no reason not to use something like Ansible that's designed for infrastructure automation, because you get encrypted secrets, templating and idempotency with zero effort and the result can be stored in a git repository somewhere. The Ansible playbook is as fast to write as installing and configuring things manually, and after some practice, it's faster and the resulting system is of higher quality. Even if you end up using the automation you write only once, it still has value because it doubles as a formalized description of what you did, and can be stored together with additional documentation. Over time you will also accumulate a library of bits and pieces that you can copy over to new setups, further improving your speed and quality.
  - crispyambulance 5y ago
    > [...] all deployments (no matter how small) should have configuration management [...] Ansible [...] with zero effort [...] No. It's a powerful tool, Ansible, but let's not get carried way. There's a ton of complexity behind the scenes. If you over-do it you end up with a ream of ugly yaml and you're fighting with the tool as much as you are any real problems.
  - Arch-TK 5y ago
    This is assuming you already know Ansible.
    - chousuke 5y ago
      Sure, but then again, typing shell commands into text files is assuming you already know shell commands. You have to spend time to learn your tools at some point. For simple configuration management, Ansible is a straight upgrade to most shells because of idempotency alone, never mind the fancier features like the more advanced modules, multi-node orchestration, or encrypted vaults. The YAML syntax is dumb and it has its issues, for sure, but it still does even the simple things much better than plain old shell. Anyone who has any familiarity at all with UNIXy systems can learn Ansible from zero well enough in a day or two for it to start becoming truly useful, and if you don't have the foundation for that... why on Earth are you setting up a web server? I mean, it's of course fine to tinker with things for learning, but I was assuming a real deployment scenario.
nerdponx 5y ago
The problem I see is that someone will inevitably update the procedure (or make a change that unknowingly requires a change in the procedure) and not update the script. Either because they are pressed for time or because they forgot. Same as any other documentation. The solution ultimately is for PMs to get it into their heads that software and infrastructure require maintenance like anything else, and consistently refusing to schedule time for software/dev-tool maintenance (such as updating documentation) has the same effect as refusing to schedule time for physical equipment maintenance. Then and only then do engineers have the freedom to set up mandatory procedures and checklists for their work, the way all engineers should be allowed and encouraged to do.
- masukomi 5y ago
  > The problem I see is that someone will inevitably update the procedure (or make a change that unknowingly requires a change in the procedure) and not update the script why would your procedure be to do anything _other_ than "run script foo and do what it says"? If your procedure is not that, then your procedure doesn't reflect reality, and thus is outdated documentation that needs to be updated. if the steps of the procedure only exist within the script then there's only one place to update it. And yes, this suggests the script should be very readable.
  - nerdponx 5y ago
    > If your procedure is not that, then your procedure doesn't reflect reality, and thus is outdated documentation that needs to be updated. Configurations change all the time. There is no technological safeguard against someone forgetting to write down the change in the playbook script; it has to be organizational.
    - csdvrx 5y ago
      A script comparing the md5 or the timestamp of the configuration files against the md5 or the timestamps of the log entry in charge of these files can do that I mean, if /etc/hosts is more recent than /log-directory/03-static-hosts-in-etc or the md5 you have recorded for this file, a daemon can easily create a ticket / send an email to whoever was logged at the time of the change.
neo2006 5y ago
If you are going through writing the do-nothing script anyway why not do a do-something script and remove the error prone human out of the loop, it also can serve as documentation and if you are disciplined enough to never make a change other then trough the automation you can have the benefit of source controlling it and have documentation with historical context
- renewiltord 5y ago
  Simple. Do nothing script: echo “Go to your Google Account settings and set a vacation auto responder” Do something script: import oauth2 # I have already lost, there is no salvation in life, without truth or peace there is only the void that beckons
- computronus 5y ago
  I happen to have just written a do-nothing script, so I can answer why I found it helpful vs. a do-something script. - I am still developing the procedure in the script, so it is premature to automate. A do-nothing script still benefits you by telling you exactly what to do - in my case, spitting out exact commands to run - but you can assess its steps before performing them. - The script still gathers a lot of information and associates it together in order to figure out the correct commands. That work is valuable all by itself. - Even though you have to run commands yourself, it's copy-and-paste vs. hand-typing, so it's already less error-prone. - The script documents the procedure even without automation, so the benefit is immediate. Now, I think that it's better for a do-nothing script to _evolve_ into a do-something script. But, if that effort is delayed or never happens, at least you've got something.
- mdoms 5y ago
  You have comprehensively missed the point. And your questions are answered in the article: > At first glance, it might not be obvious that this script provides value. Maybe it looks like all we’ve done is make the instructions harder to read. But the value of a do-nothing script is immense: > * It’s now much less likely that you’ll lose your place and skip a step. This makes it easier to maintain focus and power through the slog. > * Each step of the procedure is now encapsulated in a function, which makes it possible to replace the text in any given step with code that performs the action automatically. > * Over time, you’ll develop a library of useful steps, which will make future automation tasks more efficient. > A do-nothing script doesn’t save your team any manual effort. It lowers the activation energy for automating tasks, which allows the team to eliminate toil over time.
cseleborg 5y ago
> It is the absolute best solution there is. I prefer checklists. With a checklist, I can mark my progress as I go through the motions and, more importantly, interrupt the work even for several days, before I pick it up again. Checklists are much, much easier to adapt. They can also hold more information than progress indication, like important outputs of the procedure that need to be used somewhere else later, etc. They can be archived in case I need to understand how I did it on that particular occasion one year ago. Google Docs added checklist a while ago and I think they are very handy. And very KISS.
- csdvrx 5y ago
  > Checklists are much, much easier to adapt. They can also hold more information than progress indication, like important outputs of the procedure that need to be used somewhere else later, etc. echo "## Step 2/10 : preparing the SSH key for xxx" (...) KEY=$( cat .ssh/id_rsa.pub ) echo "## Do not forget to use this important output somewhere else later: $KEY" > They can be archived in case I need to understand how I did it on that particular occasion one year ago. (...) # 20191102 # TODO: 2 years ago, we decide to use 4096 bits key, make sure to check the length # in case I forget, how to do that, the number of bits can be forced with -b 4096 # 20201102 # WONTFIX: use ecdsa for the specific host yyyy So I stand with the author: this is the absolute best solution there is: it can be as simple or as thorough as you need, while being very low tech and simple to use. And when you find yourself needing to make say an Ansible configuration, you already have most of what you need.
  - MaulingMonkey 5y ago
    This fails on !wsl windows (no `cat`.) Or on *nix if `ssh-keygen` is missing (minimal VM image? new box?) Or on *nix if I absentmindedly used `bash` specific syntax and I'm now in a real `sh` shell. And this is for basic SSH setup! An aborted script has a good chance of having left things in a broken, half-constructed state, such that simply re-executing the script results in more errors. Attempting to manually resume the script midway will have likely discarded important env variables that I must now figure out how to reconstruct. ...which isn't to say the scripting approach doesn't have value - poor checklist discipline by my coworkers has led me to painstakingly automate more than one perfectly fine checklist, and some stuff is quite scripting friendly - but I've also written my share of scripts that ate up more time in debugging and troubleshooting than I ever saved by writing them in the first place.
magicalhippo 5y ago
I'm primarily a Windows guy, but I'm dabbling more and more with Linux lately. Especially after I got a few Raspberry Pi's and various clones set up. What I've ended up doing is to just write Word documents containing what I do. I keep my Word documents in a single directory on my OneDrive, so I can access them on all my machines. Raspberry Pi iSCSI server? One document for that, containing the links to guides used, commands I've run and scripts made. I use headers to organize the sections, like one for compiling the custom kernel, one for configuring iSCSI on the Pi, one for using it on the NAS etc. Need to create a new Docker SMB mount? Once the right incantation has been found, a small script is made and also added to the right Word document, ready to be pasted into the next machine I might need it on. It's not terribly pretty and not at all fancy, but I found it's low enough barrier that it's easy to do and maintain, and it's very helpful to have it all in one place. Had I been a Linux guy first and foremost I would probably have used something else than Word, but it works and I find the separation between the regular font for comments and monospace for script makes it easy to quickly distinguish. It's also easy to add screenshots for clarifications etc.
- lucb1e 5y ago
  I think using an executable format like in the article, you'll find it's not actually that much harder to read than a Word document with proportional fonts. And I'm not sure where you need screenshots if it's about doing things on a non-Windows system, even if that seems like an odd transition at first :)
  - magicalhippo 5y ago
    Yeah might have to try it out. The images could be for things like performance graphs, interactive menu choices, photos of GPIO wiring etc. Not a huge loss to not have them, but given it's dead easy to add to a Word document, why not?
- csdvrx 5y ago
  > Had I been a Linux guy first and foremost I would probably have used something else than Word, but it works and I find the separation between the regular font for comments and monospace for script makes it easy to quickly distinguish. It's also easy to add screenshots for clarifications etc. As a windows girl, I recommend you give a try to the notebooks like RStudio: you can add screenshots (or script them with ahk, nircmd etc to automatically screenshot at some points of the execution) and execute blocks of commands in about anything. The notebook approach has the additional nice feature of stopping execution as soon as a block fails, giving you the opportunity to fix it before you continue from that point, something often more tedious with Linux scripts (where you need to commend the beginning if your script isn't idempotent) Ideally, you would always write idempotent scripts, but who've got time for that :) In practice, if you try to avoid wasting effort, it's often the icing on the cake, once everything else has been done. So I like notebook environments for this simple feature: piecewise execution with verbose output (similar to bash -xe) That said, a directory per machine (and a subdirectory with all the drivers and specific software) on Onedrive with a RTF file full of screenshots (because wordpad is everywhere!) is how I work most of the time :)

csdvrx 5y ago

> only potential downside is if someone decides to "engineer" this method It can be engineered, if you follow a gradual process. On servers, I keep a log of what was deployed in a root directory following the sequential number _ goal format (ex: 00_partitions ... 90_web_server) It is not fancy, most of the logs are not even scripts: many are just ASCII text files, that will only be used as a checklist if the same "goal" has to be achieved again. For example, 00_partition may be "gdisk /dev/nvme0n1" followed by a copy-pasted list of the partitions and some quick description about why it was done that way. But that's on the first iteration only: the next iteration turns that into "do-nothing" script, the next iteration into a better script with basic checks (supporting both /dev/nvme0n1 and /dev/sda), then exception handling (if partitions already exist, etc), and so on: this gradual complexification process avoids the "premature optimization" of creating infrastructure-as-code for what you rarely need, while optimizing and fine tuning the parts you most often need. Someone will certainly mention Terraform, or Ansible, or something else - yes, they exist and they are nice, but if you are doing everything there, you are over-engineering and wasting time: not everything needs your equal attention! If you only install a webserver once in a blue moon, make a .txt checklist of the steps you followed. But if you leave and breathe nginx options and certificate deployment, fully automatize all that, including the obscure details of what may fail if you use let's encrypt with some specific DNS configuration! And if you don't know yet which is which, start small (a .txt checklist will cost you a few minutes) and the next time you find yourself doing the same thing, do it better using the previous artifact (the .txt file) to create a better one (a script, then a better script etc)

chousuke 5y ago
I would argue that all deployments (no matter how small) should have configuration management. In the simplest case, a deployment consists of: 1. Install packages 2. Install configuration files, possibly from templates. 3. Configure services to start on boot. This kind of automation is trivial to do with almost any tool, but there's no reason not to use something like Ansible that's designed for infrastructure automation, because you get encrypted secrets, templating and idempotency with zero effort and the result can be stored in a git repository somewhere. The Ansible playbook is as fast to write as installing and configuring things manually, and after some practice, it's faster and the resulting system is of higher quality. Even if you end up using the automation you write only once, it still has value because it doubles as a formalized description of what you did, and can be stored together with additional documentation. Over time you will also accumulate a library of bits and pieces that you can copy over to new setups, further improving your speed and quality.
- crispyambulance 5y ago
  > [...] all deployments (no matter how small) should have configuration management [...] Ansible [...] with zero effort [...] No. It's a powerful tool, Ansible, but let's not get carried way. There's a ton of complexity behind the scenes. If you over-do it you end up with a ream of ugly yaml and you're fighting with the tool as much as you are any real problems.
- Arch-TK 5y ago
  This is assuming you already know Ansible.
  - chousuke 5y ago
    Sure, but then again, typing shell commands into text files is assuming you already know shell commands. You have to spend time to learn your tools at some point. For simple configuration management, Ansible is a straight upgrade to most shells because of idempotency alone, never mind the fancier features like the more advanced modules, multi-node orchestration, or encrypted vaults. The YAML syntax is dumb and it has its issues, for sure, but it still does even the simple things much better than plain old shell. Anyone who has any familiarity at all with UNIXy systems can learn Ansible from zero well enough in a day or two for it to start becoming truly useful, and if you don't have the foundation for that... why on Earth are you setting up a web server? I mean, it's of course fine to tinker with things for learning, but I was assuming a real deployment scenario.

nerdponx 5y ago

The problem I see is that someone will inevitably update the procedure (or make a change that unknowingly requires a change in the procedure) and not update the script. Either because they are pressed for time or because they forgot. Same as any other documentation. The solution ultimately is for PMs to get it into their heads that software and infrastructure require maintenance like anything else, and consistently refusing to schedule time for software/dev-tool maintenance (such as updating documentation) has the same effect as refusing to schedule time for physical equipment maintenance. Then and only then do engineers have the freedom to set up mandatory procedures and checklists for their work, the way all engineers should be allowed and encouraged to do.

masukomi 5y ago
> The problem I see is that someone will inevitably update the procedure (or make a change that unknowingly requires a change in the procedure) and not update the script why would your procedure be to do anything _other_ than "run script foo and do what it says"? If your procedure is not that, then your procedure doesn't reflect reality, and thus is outdated documentation that needs to be updated. if the steps of the procedure only exist within the script then there's only one place to update it. And yes, this suggests the script should be very readable.
- nerdponx 5y ago
  > If your procedure is not that, then your procedure doesn't reflect reality, and thus is outdated documentation that needs to be updated. Configurations change all the time. There is no technological safeguard against someone forgetting to write down the change in the playbook script; it has to be organizational.
  - csdvrx 5y ago
    A script comparing the md5 or the timestamp of the configuration files against the md5 or the timestamps of the log entry in charge of these files can do that I mean, if /etc/hosts is more recent than /log-directory/03-static-hosts-in-etc or the md5 you have recorded for this file, a daemon can easily create a ticket / send an email to whoever was logged at the time of the change.

neo2006 5y ago

If you are going through writing the do-nothing script anyway why not do a do-something script and remove the error prone human out of the loop, it also can serve as documentation and if you are disciplined enough to never make a change other then trough the automation you can have the benefit of source controlling it and have documentation with historical context

renewiltord 5y ago
Simple. Do nothing script: echo “Go to your Google Account settings and set a vacation auto responder” Do something script: import oauth2 # I have already lost, there is no salvation in life, without truth or peace there is only the void that beckons
computronus 5y ago
I happen to have just written a do-nothing script, so I can answer why I found it helpful vs. a do-something script. - I am still developing the procedure in the script, so it is premature to automate. A do-nothing script still benefits you by telling you exactly what to do - in my case, spitting out exact commands to run - but you can assess its steps before performing them. - The script still gathers a lot of information and associates it together in order to figure out the correct commands. That work is valuable all by itself. - Even though you have to run commands yourself, it's copy-and-paste vs. hand-typing, so it's already less error-prone. - The script documents the procedure even without automation, so the benefit is immediate. Now, I think that it's better for a do-nothing script to _evolve_ into a do-something script. But, if that effort is delayed or never happens, at least you've got something.
mdoms 5y ago
You have comprehensively missed the point. And your questions are answered in the article: > At first glance, it might not be obvious that this script provides value. Maybe it looks like all we’ve done is make the instructions harder to read. But the value of a do-nothing script is immense: > * It’s now much less likely that you’ll lose your place and skip a step. This makes it easier to maintain focus and power through the slog. > * Each step of the procedure is now encapsulated in a function, which makes it possible to replace the text in any given step with code that performs the action automatically. > * Over time, you’ll develop a library of useful steps, which will make future automation tasks more efficient. > A do-nothing script doesn’t save your team any manual effort. It lowers the activation energy for automating tasks, which allows the team to eliminate toil over time.

cseleborg 5y ago

> It is the absolute best solution there is. I prefer checklists. With a checklist, I can mark my progress as I go through the motions and, more importantly, interrupt the work even for several days, before I pick it up again. Checklists are much, much easier to adapt. They can also hold more information than progress indication, like important outputs of the procedure that need to be used somewhere else later, etc. They can be archived in case I need to understand how I did it on that particular occasion one year ago. Google Docs added checklist a while ago and I think they are very handy. And very KISS.

csdvrx 5y ago
> Checklists are much, much easier to adapt. They can also hold more information than progress indication, like important outputs of the procedure that need to be used somewhere else later, etc. echo "## Step 2/10 : preparing the SSH key for xxx" (...) KEY=$( cat .ssh/id_rsa.pub ) echo "## Do not forget to use this important output somewhere else later: $KEY" > They can be archived in case I need to understand how I did it on that particular occasion one year ago. (...) # 20191102 # TODO: 2 years ago, we decide to use 4096 bits key, make sure to check the length # in case I forget, how to do that, the number of bits can be forced with -b 4096 # 20201102 # WONTFIX: use ecdsa for the specific host yyyy So I stand with the author: this is the absolute best solution there is: it can be as simple or as thorough as you need, while being very low tech and simple to use. And when you find yourself needing to make say an Ansible configuration, you already have most of what you need.
- MaulingMonkey 5y ago
  This fails on !wsl windows (no `cat`.) Or on *nix if `ssh-keygen` is missing (minimal VM image? new box?) Or on *nix if I absentmindedly used `bash` specific syntax and I'm now in a real `sh` shell. And this is for basic SSH setup! An aborted script has a good chance of having left things in a broken, half-constructed state, such that simply re-executing the script results in more errors. Attempting to manually resume the script midway will have likely discarded important env variables that I must now figure out how to reconstruct. ...which isn't to say the scripting approach doesn't have value - poor checklist discipline by my coworkers has led me to painstakingly automate more than one perfectly fine checklist, and some stuff is quite scripting friendly - but I've also written my share of scripts that ate up more time in debugging and troubleshooting than I ever saved by writing them in the first place.

magicalhippo 5y ago

I'm primarily a Windows guy, but I'm dabbling more and more with Linux lately. Especially after I got a few Raspberry Pi's and various clones set up. What I've ended up doing is to just write Word documents containing what I do. I keep my Word documents in a single directory on my OneDrive, so I can access them on all my machines. Raspberry Pi iSCSI server? One document for that, containing the links to guides used, commands I've run and scripts made. I use headers to organize the sections, like one for compiling the custom kernel, one for configuring iSCSI on the Pi, one for using it on the NAS etc. Need to create a new Docker SMB mount? Once the right incantation has been found, a small script is made and also added to the right Word document, ready to be pasted into the next machine I might need it on. It's not terribly pretty and not at all fancy, but I found it's low enough barrier that it's easy to do and maintain, and it's very helpful to have it all in one place. Had I been a Linux guy first and foremost I would probably have used something else than Word, but it works and I find the separation between the regular font for comments and monospace for script makes it easy to quickly distinguish. It's also easy to add screenshots for clarifications etc.

lucb1e 5y ago
I think using an executable format like in the article, you'll find it's not actually that much harder to read than a Word document with proportional fonts. And I'm not sure where you need screenshots if it's about doing things on a non-Windows system, even if that seems like an odd transition at first :)
- magicalhippo 5y ago
  Yeah might have to try it out. The images could be for things like performance graphs, interactive menu choices, photos of GPIO wiring etc. Not a huge loss to not have them, but given it's dead easy to add to a Word document, why not?
csdvrx 5y ago
> Had I been a Linux guy first and foremost I would probably have used something else than Word, but it works and I find the separation between the regular font for comments and monospace for script makes it easy to quickly distinguish. It's also easy to add screenshots for clarifications etc. As a windows girl, I recommend you give a try to the notebooks like RStudio: you can add screenshots (or script them with ahk, nircmd etc to automatically screenshot at some points of the execution) and execute blocks of commands in about anything. The notebook approach has the additional nice feature of stopping execution as soon as a block fails, giving you the opportunity to fix it before you continue from that point, something often more tedious with Linux scripts (where you need to commend the beginning if your script isn't idempotent) Ideally, you would always write idempotent scripts, but who've got time for that :) In practice, if you try to avoid wasting effort, it's often the icing on the cake, once everything else has been done. So I like notebook environments for this simple feature: piecewise execution with verbose output (similar to bash -xe) That said, a directory per machine (and a subdirectory with all the drivers and specific software) on Onedrive with a RTF file full of screenshots (because wordpad is everywhere!) is how I work most of the time :)