4 ms·

SUPER helpful. Very grateful!

sliken 4y ago
Generally new admins tend towards creating physical or virtual machines, logging in as root, editing files, starting services, and backing them up. Which is fine, a great way to learn, a less great way to be in production. Upgrades can be painful, it's easy to break things is non-obvious ways, and there's no easy "undo". Generally I'd recommend either using containers and maintain your build scripts in git, or use some configuration management software like ansible or puppet (there are many others as well) so you can install a new OS (virtually or physically), then have ansible/puppet/whatever add config files, packages, services etc to get things working. Of course then you keep ansible/puppet/whatever files in git AND back them up. That way if there's some major upgrade of OS or software you can spin up a test env, test it, and then decide to promote/replace an existing machine ... or just ditch it.
- kickaha 4y ago
  Really good advice. I still have flashes of “Ohhhh righhhhhht it’s a VIRTUAL machine---I can...” make a snapshot, clone it and test new stuff, etc. etc. etc. I’ll be happy when it fully sinks in.
  - sliken 4y ago
    Generally I try to avoid VPSs. They are long lived, it's easy to accumulate undocumented state, and have substantial memory, disk I/O, and network I/O overhead. You may end up with a years old VPS with many changes over the years, and ugly things accumulate. Maybe you built a package from source, but now it's got a security vulnerability, but your forgot it's there. So apt upgrade doesn't help. Not to mention any security fix assumes you aren't already compromised. A rebuilt container (I suggest proxmox or docker) rebuilt will remove any previous existing compromise. Sure you can clone an old VPS... but you still don't know exactly what changes have been made, and if it was compromised the clone is as well. Much better to have docker (or similar) build script, a ansible play, or a puppet manifest. They will clearly list needed packages, config files, services, at least in puppet's case you'll have the "trifecta". The trifecta is a mapping between a service, the required packages, and the config files. So Puppet can notice a config file change, restart a service, and if a package is missing install it. So puppet is to a degree self healing, even when an admin edits a file on the guest/container/VPS instead of in puppet and it's fixed on the next puppet run. That way years later it's not big deal to upgrade the OS, or replace one piece of the stack with another, then apply ALL the changes you've done so far, and test the result. I can tell you from experience it's a nightmare to inherit somebody else's VPS that has 3 python installs, tons of services, and requires manual intervention on boot to start/stop services. Would be MUCH better if it was just managed by config management and everything except the base OS is controlled by ansible/puppet. Often custom VPSs are called pets or snowflakes (with a negative connotation), much better to make them easy to replace as needed, like cows (with a positive connotation) as the saying goes.
    - kickaha 4y ago
      Wow, thanks. This is a good and useful perspective. I shall endeavor to make my VPS a source of nutrition rather than a source of affection. :-)
    - sliken 4y ago
      Can't reply to your reply, it's too deeply nested. Even better than a VPS with multiple services is (IMO) a container per service. So you get a nice clear single function container, but don't have to pay the memory and performance overhead of having a VPS per service. Containers have zero memory/performance overhead (it's based on cgroups, which normal linux processes have anyways), minimal disk overhead (not need for an OS install for each one), and saves a ton of ram (each VPS requires at least a linux kernel).

Generally new admins tend towards creating physical or virtual machines, logging in as root, editing files, starting services, and backing them up. Which is fine, a great way to learn, a less great way to be in production. Upgrades can be painful, it's easy to break things is non-obvious ways, and there's no easy "undo". Generally I'd recommend either using containers and maintain your build scripts in git, or use some configuration management software like ansible or puppet (there are many others as well) so you can install a new OS (virtually or physically), then have ansible/puppet/whatever add config files, packages, services etc to get things working. Of course then you keep ansible/puppet/whatever files in git AND back them up. That way if there's some major upgrade of OS or software you can spin up a test env, test it, and then decide to promote/replace an existing machine ... or just ditch it.

kickaha 4y ago
Really good advice. I still have flashes of “Ohhhh righhhhhht it’s a VIRTUAL machine---I can...” make a snapshot, clone it and test new stuff, etc. etc. etc. I’ll be happy when it fully sinks in.
- sliken 4y ago
  Generally I try to avoid VPSs. They are long lived, it's easy to accumulate undocumented state, and have substantial memory, disk I/O, and network I/O overhead. You may end up with a years old VPS with many changes over the years, and ugly things accumulate. Maybe you built a package from source, but now it's got a security vulnerability, but your forgot it's there. So apt upgrade doesn't help. Not to mention any security fix assumes you aren't already compromised. A rebuilt container (I suggest proxmox or docker) rebuilt will remove any previous existing compromise. Sure you can clone an old VPS... but you still don't know exactly what changes have been made, and if it was compromised the clone is as well. Much better to have docker (or similar) build script, a ansible play, or a puppet manifest. They will clearly list needed packages, config files, services, at least in puppet's case you'll have the "trifecta". The trifecta is a mapping between a service, the required packages, and the config files. So Puppet can notice a config file change, restart a service, and if a package is missing install it. So puppet is to a degree self healing, even when an admin edits a file on the guest/container/VPS instead of in puppet and it's fixed on the next puppet run. That way years later it's not big deal to upgrade the OS, or replace one piece of the stack with another, then apply ALL the changes you've done so far, and test the result. I can tell you from experience it's a nightmare to inherit somebody else's VPS that has 3 python installs, tons of services, and requires manual intervention on boot to start/stop services. Would be MUCH better if it was just managed by config management and everything except the base OS is controlled by ansible/puppet. Often custom VPSs are called pets or snowflakes (with a negative connotation), much better to make them easy to replace as needed, like cows (with a positive connotation) as the saying goes.
  - kickaha 4y ago
    Wow, thanks. This is a good and useful perspective. I shall endeavor to make my VPS a source of nutrition rather than a source of affection. :-)
  - sliken 4y ago
    Can't reply to your reply, it's too deeply nested. Even better than a VPS with multiple services is (IMO) a container per service. So you get a nice clear single function container, but don't have to pay the memory and performance overhead of having a VPS per service. Containers have zero memory/performance overhead (it's based on cgroups, which normal linux processes have anyways), minimal disk overhead (not need for an OS install for each one), and saves a ton of ram (each VPS requires at least a linux kernel).