7 ms·

Server Setup Basics for Self Hosting

169 points by joelp 2y ago

tiffanyh 2y ago
Love seeing devops post on HN. Wish it included server monitoring as a section.
- everforward 2y ago
  Server monitoring for self-hosting is kind of hard because it basically necessitates either buying another server to monitor the first, or paying for SaaS. From personal experience, I would just pay someone else for a SaaS monitoring solution. It will almost universally be cheaper and more reliable. If you really wanted to run your own, Prometheus is probably the way to go. Local storage should be fine as a data store for self-hosted. Grafana can be used for dashboarding, and either Grafana or AlertManager can do the alerting component. It’s really not all that worth it for self-hosted scale, though. Running all that in the cloud is going to cost basically the same as buying a DataDog license unless you’re at 3-ish hosts, and more than that if you’re doing clustered monitoring so you aren’t blind if your monitoring host is down.
- neitsab 2y ago
  Funny how that kind of posts is now called "DevOps", while 10 years ago it was simply called "system administration" ;-) Besides I fail to see any DevOps tenets in it, quite the opposite: a shell script at the bottom is little in the way of reliable automation. To me this post reads more like someone relatively new to server management wanted to share their gathered tips and tricks, i.e. me 10 years ago when I started my self-hosting journey :-D
abhinavk 2y ago
> You want to use SSH (Secure Shell) and make sure that SSH is the only way to log in. Some distributions (like openSuSE) also enable KbdInteractiveAuthentication by default so just disabling PasswordAuthentication won't work.
- _blk 2y ago
  I'm a bit sceptical of the choice of port 2222 as an alternative. At that point you might as well leave 22, but otherwise it's a good intro. If you're serious about starting post the sections into [insert AI service name] and start asking questions.
  - 0cf8612b2e1e 2y ago
    Anything other than 22 is an improvement in just the reduced log volume.
    - dartos 2y ago
      Yeah, you basically dodge most automated attacks by getting off port 22
- dsissitka 2y ago
  This is one of those things I like to verify: david@desktop:~$ nmap -p 22 --script ssh-auth-methods becomesovran.com Starting Nmap 7.92 ( https://nmap.org ) at 2024-08-25 23:31 EDT Nmap scan report for becomesovran.com (162.213.255.209) Host is up (0.066s latency). rDNS record for 162.213.255.209: server1.becomesovran.com PORT STATE SERVICE 22/tcp open ssh | ssh-auth-methods: | Supported authentication methods: | publickey | gssapi-keyex | gssapi-with-mic | password |_ keyboard-interactive Nmap done: 1 IP address (1 host up) scanned in 0.86 seconds david@desktop:~$ As far as I can tell AuthenticationMethods publickey is the right way to do it these days but I'd love to know if that's not the case.
  - yjftsjthsd-h 2y ago
    I've just been doing ssh -v localhost echo 2>&1 | grep continue (obviously replacing "localhost" with whatever server you want, and you can put anything you want where "echo" is but that's the best no-op I've come up with)
    - 2y ago
remram 2y ago
> Differential backups back up all the changes since the last full backup (...) An incremental backup backs up data that was changed since the last backup I'm not sure I understand the distinction?
- magicalhippo 2y ago
  They both do delta backups, but incremental bases it's delta on previous backup, while differential between the last full backup. To restore from an incremental you need the last full backup and all the incrementals inbetween. If you do say a full backup every month, you'd need up to 30 good incremental backup sets to be able to restore. For the differential you just need the last full backup in addition. Obviously the differential one might take more and more space, depending on the changes.
  - remram 2y ago
    I see, thanks. I only use Restic so this is not relevant to me, but I think I understand the trade-off.
- dsissitka 2y ago
  Differential backups are always: Full Backup -> Differential Backup Incremental backups are: Full Backup -> Incremental Backup [-> Incremental Backup ...] At least that's how it is with Macrium.
- betaby 2y ago
  https://rdiff-backup.net/ https://rdiff-backup.net/ explains
- pudgyblues 2y ago
  With differential backups there's only 2 artifacts, the full and the diff. If you make another differential backup you overwrite the previous diff so it's always the changes since the last full backup. With incremental it's full backup + inc1 + inc2 +... forever, each backup depends on the previous.
hobobaggins 2y ago
I'd switch to Userify if you have a team to distribute keys for, because it's ultra-lightweight and also keeps you from messing up permissions on the ssh key/directory, which I've done too many times! (also it does sudo which is quite nice) Also, restarting ssh will not boot you out of the session (your session has already been forked as a different process), so leave your terminal window open (to fix any screwups) and then log in on a separate window on the new port and just make sure you can get in. For backups, don't set up logins from your main server(s) to your backup server; log in from your backup server to your main server. That way, if someone breaks into your main server, they can't get into your backup server.
AtlasBarfed 2y ago
So... about docker, did they backtrack on their licensing landgrab? About a year ago I swear everyone was going to podman, but in the last few months I see nothing but docker references. Podman is supposed to be drop-in. Well, it was advertised. I haven't touched anything in six months.
- dartos 2y ago
  IIRC docker had some heat for their docker desktop licenses. I think podman is more secure and simpler, but not as ergonomic to have locally (it’s not quite a drop in for docker. No real docker compose support for example) Podman is the default for k8s last I heard
- bongobingo1 2y ago
  I think Podman is still pretty fringe in the grand count. It also may suffer from an aspect of those that know about it and use it are less likely to write a blog post about it. I use it and prefer it, much so. Mostly because of rootless (I know docker has made attempts to improve this in the last year or so), not futzing with my iptables and a better handling of pushing images between hosts (it's been over a year since I touched any of that infra, I just remember it being more of hassle with docker which took a "our way or no way" approach). The biggest issue I have with Podman is the pace of its improvement against the rate of Debian releases!
- mkesper 2y ago
  Only use docker engine ("moby"). Docker desktop makes no sense on a Linux system as it introduces a VM into the mix, further adding complexity and reducing performance. https://docs.docker.com/engine/ https://docs.docker.com/engine/
Cyph0n 2y ago
Great post! I (relatively) recently switched my primary home server over to NixOS and am now a huge fan of it as a distribution for self-hosting. Here is how setting this all up would like in NixOS (modulo some details & machine-specific configuration). It's <100 lines, can be executed/configured with a single CLI command (even from a different machine!), rolled back easily if things go wrong, and can be re-used on any NixOS machine :) { networking = { # Server hostname hostName = "myserver"; # Firewall firewall = { enable = true; allowedTCPPorts = [ 80 443 2222 ]; }; }; # Users users.users = { newuser = { isNormalUser = true; home = "/home/newuser"; hashedPassword = "my-hashed-pwd"; openssh.authorizedKeys.keys = [ "my-pub-key" ]; }; }; # SSH services.openssh = { enable = true; ports = [ 2222 ]; settings = { PermitRootLogin = "no"; PasswordAuthentication = false; AllowUsers = [ "newuser" ]; }; extraConfig = '' Protocol 2 # Use only SSH protocol version 2 MaxAuthTries 3 # Limit authentication attempts ClientAliveInterval 300 # Client alive interval in seconds ClientAliveCountMax 2 # Maximum client alive count ''; }; services.fail2ban.enable = true; # Nginx + SSL via LetsEncrypt services.nginx = { enable = true; recommendedOptimisation = true; recommendedProxySettings = true; recommendedTlsSettings = true; virtualHosts = { "example.com" = { locations."/" = { proxyPass = "http://localhost:8080"; proxyWebsockets = true; }; forceSSL = true; enableACME = true; }; }; }; security.acme = { acceptTerms = true; defaults.email = "myemail@gmail.com"; certs."example.com" = { dnsProvider = "cloudflare"; environmentFile = ./my-env-file; }; }; # Logrotate services.logrotate = { enable = true; configFile = pkgs.writeText "logrotate.conf" '' /var/log/nginx/*.log { weekly missingok rotate 52 compress delaycompress notifempty create 0640 www-data adm sharedscripts postrotate [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid` endscript } ''; }; # Bonus: auto-upgrade from GH repo system.autoUpgrade = { enable = true; flake = "github:myuser/nixos-config"; flags = [ "-L" # print build logs "--refresh" # do not use cached Flake ]; dates = "00:00"; allowReboot = true; randomizedDelaySec = "45min"; }; }
crdrost 2y ago
Qq, do people doing their own server setup like this use containerization at all? When I looked at it, it was like “yeah you can run Docker or k3s,” and I think Hashicorp had their own version, but it seemed like folks didn't really bother? Also like setting up virtual networks among VPSes seemed like it required advanced wizardry.
- lolinder 2y ago
  I have an old desktop with Debian where I run a bunch of random things (Home Assistant, Pihole, Seafile, Jellyfin, ...) and I run everything with Podman Compose. I have enough things that I'm 100% confident I'd have run into dependency issues by now without containerization, but with Docker files it's trivial to keep them separate. As a bonus, compose.yml files are basically the lingua franca for describing deployments these days, so you can almost always find an example in the official docs for any given service you might want to host and get lots of help.
- ozr 2y ago
  > do people doing their own server setup like this use containerization at all? Depends on what you're deploying, really. If it's one Go service per host, there's no real need. Just a unit file and the binary. Your deployment scheme is scp and a restart. For more complicated setups, I've used docker compose. > Also like setting up virtual networks among VPSes seemed like it required advanced wizardry. Another 'it depends'. If you're running a small SaaS application, you probably don't need multiple servers in the first place. If you want some for redundancy, most providers offer a 'private network', where bandwidth is unmetered. Each compute provider is slightly different: you'll want to review their docs to see how to do it correctly. Tailscale is another option for networking, which is super easy to setup.
  - cassianoleal 2y ago
    There’s rarely, if ever, a _need_ for containerisation. Even for a single static binary though, there are benefits like network and filesystem segregation, resource allocation, …
- 2y ago
erros 2y ago
You may want to update this post to disable password authentication, and thus you'll no longer need to install fail2ban. An important goal is to tighten your attack surface, not expand it. At this point you will still have an exposed SSHd server, so I'd recommend throwing the server under tailscale. You can setup the SSHd listener to use your tailscale IP or setup tailscale for SSH via ACLs (https://tailscale.com/tailscale-ssh https://tailscale.com/tailscale-ssh). Additionally you can further tighten controls of incoming logins with the use of AllowGroups to tighten your controls on which groups can log into the system. This would mitigate a scenario where an adversary is able to escalate enough privileges to write an .authorized_keys file to a non-privileged user which may have a shell still configured. Finally, unless you're treating this server as a bastion host of sorts, you probably should disable forwarding for agents or X11 etc. We've seen a lot of adversaries move laterally due to this agent forwarding.
- Semaphor 2y ago
  > You may want to update this post to disable password authentication Probably not, as that’s one of the first things they do. That said, I feel like all this fail2ban stuff is very much cargo culting in the selfhosting community. I’ve had my VPS SSH server on port 22 with no fail2ban for slightly over a decade, exposed to the public internet (home server is behind tailscale, VPS hosts the stuff I always want accessible from everywhere). Bots try it, they fail, the end. Maybe I’m missing something, but I have yet to find a good reason for the added complexity.
  - ftrobro 2y ago
    This and possibly unknown similar flaws is a good reason for not advertising ssh: https://arstechnica.com/security/2024/07/regresshion-vulnerability-in-openssh-gives-attackers-root-on-linux/ https://arstechnica.com/security/2024/07/regresshion-vulnera...
  - dingaling 2y ago
    fail2ban is great for reducing clutter in logs. When I'm trying to debug something using auth.log I don't want to sift through 800 attempts by some IP to SSH in as root. It gives just enough info about the origin and nature of attempted intruders without overwhelming detail.
jks 2y ago
I recommend checking out Caddy <https://caddyserver.com/ https://caddyserver.com/>, which replaces both Nginx and Certbot in this setup. Tailscale <https://tailscale.com/ https://tailscale.com/> can remove the need to open port 22 to the world, but I wouldn't rely on it unless your VPS provider has a way to access the server console in case of configuration mistakes.
- jim180 2y ago
  I've recently discovered, that Caddy config file has a neat support for imports: https://pastebin.com/vVQYrpmj https://pastebin.com/vVQYrpmj
- Semaphor 2y ago
  Caddy also simplifies many common Nginx configurations with a one-liner. The biggest hurdle is when you don’t have a simple configuration, as all the examples are usually only for Nginx ;)
- calgoo 2y ago
  For Tailscale backup access, another way is to block port 22 on a firewall and then only unblock it if you need access.
  - nehal3m 2y ago
    If you depend on the host behind Tailscale to access the firewall from the inside then that's not going to work. Most colos I have hardware at offer a separate network for iDRAC/ILO/your flavor of OOB management, I like to use the console through that to open/close stuff like this.
- InvOfSmallC 2y ago
  Regarding tailscale, be sure to remove the expiration flag on your server. That's how I lost mine.
chadsix 2y ago
And for those of you that don't have an external IP, you can use services that provide egress for you like IPv6.rs. [1] [1] I'm DevOps there! ;)
- ahofmann 2y ago
  Thanks for the link, product looks good. Maybe you should work a bit on the website, this is how it looks at my pixel phone on Firefox: https://photos.app.goo.gl/txpxvDQAMYQWRSmi7 https://photos.app.goo.gl/txpxvDQAMYQWRSmi7
solatic 2y ago
Especially when writing a tutorial for beginners - please use the long-form flags (e.g. sudo usermod --append --groups sudo newuser) instead of short-form flags (e.g. sudo usermod -aG sudo newuser). Short-form flags make commands look like arcane voodoo magic. They make sense only to help you save time entering commands if you know them by heart already. Tutorials are read by beginners who are not necessarily familiar with the commands in the first place - long-form flags help communicate what these commands are actually doing and thus make for a more effective tutorial.
- lhousa 2y ago
  "Acronyms seriously suck" ~Elon
  - benterix 2y ago
    At least it would be funny if signed by RMS.[0] [0] He wouldn't as he was a fan of recursive ones.
- noahjk 2y ago
  I would go as far as to say short flags should never be shared or saved (besides man pages or similar). Long flags help anyone who needs to review something in the future, even the author. Perfect for scripts of all sorts, tutorials, anything checked into git, etc.
  - sebazzz 2y ago
    Powershell recommendations, for instance, are that you keep aliases and short-form for yourself, and long-form for scripts and tutorials. Remove-Item -Path X:\test\ -Recurse -Force del X:\test -rec -for
- deviantintegral 2y ago
  Yes please! This came up enough for us we standardized on this for all documentation and CI scripts. https://architecture.lullabot.com/adr/20211006-avoid-command-aliases/ https://architecture.lullabot.com/adr/20211006-avoid-command...
jw_cook 2y ago
At the end of the article, there's a link to a script[1] that does the steps covered in the article. That got me thinking: how do other self-hosters/homelabbers here go about automating their server setups? None/purely manual? One big shell script? Multiple scripts wrapped in a Makefile (or justfile, or other command runner)? More enterprisey provisioning/automation tools like Ansible, Puppet, etc.? [1] https://git.sovbit.dev/Enki/sovran-scripts https://git.sovbit.dev/Enki/sovran-scripts
- jmathai 2y ago
  One big shell script has worked really well for me. One project on AWS ran the script when new EC2 instances with a particular tag/label were spun up and that's how we scaled horizontally. What's nice about is that it doesn't require any specialized knowledge beyond bash - and that's something which is pretty easy to learn and great to know. It also attracts, IMO, the type of developers who avoid chasing new trends.
  - Sammi 2y ago
    I have a folder of scripts. One main script that calls into the other scripts, just so I can keep my head straight. But one large script might work just as well for you. This sets up everything I need so I can treat my servers as livestock instead of pets - that is, so I can easily slaughter and replace them whenever I want, instead of being tied to them like a pet.
- atoav 2y ago
  For my use cases I found that just having a (updated) note with the things I would usually do works best. This is because I would not deploy everything anywhere and manually being aware of each step instead of hiding it within a script is somewhat a feature (e.g. you can easily insert a custom extra step etc). If I would do basically the same over and over I'd probably go with a script, ansible cookbook or similar, but as of now the manual route is totally fine.
  - ranger207 2y ago
ValtteriL 2y ago
A monitoring setup with for instance Prometheus+Grafana would be a great addition to this. And then maybe automating all of it with something like Ansible.
davidmitchell2 2y ago
While these seems to be secure... tampering with default settings always cause PITA; especially during automated upgrades. In addition, ssh port changes are all security thru obscurity.
- Sammi 2y ago
  Just closing well known ports will mean less drive-by sniffing. Which is an improvement. Doesn't mean you are now completely safe - it's just an improvement. At the very least it will make your logs smaller, as they won't be as full of drive-by sniffing. Security is an onion, you can add layers. There is no perfect security. You can add hurdles and hope you make yourself too difficult for you adversary. Some hurdles add more than others, and not using well known ports is on the lesser end of the scale. You might still find it worthwhile, just so you have cleaner logs to sift through.
deleted 2y ago
[deleted]
voidUpdate 2y ago
With SSH Keys, do you have to just carry around your private key everywhere on a usb or something if you want to connect from multiple locations? Sometimes I find myself somewhere I've never been and I want to connect to my server. With a password that's easy, I just type it in, but I can't exactly create a new private key to access the server when I don't have access to the server in the first place
- fragmede 2y ago
  password managers can store ssh private keys, though there's still the problem of you've been mugged, have no cellphone wallet or keys, and need to break back into your life.
- SSLy 2y ago
  Some password managers expose SSH agent socket, so you can do that instead of fumbling with files.
nihilius 2y ago
Here are some "First Things on a Server" Notes. https://gist.github.com/klaushardt/07f5e3068355aafc2dce660a54be1ea7 https://gist.github.com/klaushardt/07f5e3068355aafc2dce660a5... Ansible/Puppet or NixOS would be better, but this is what works in Self Hosting.
thijsb 2y ago
Why the `sudo ufw allow outgoing`? Wouldn't it be worth to deny all to prevent extrusion and only open ports for services that need to communicate externally?
Sandbag5802 2y ago
Hey I just want to say thank you for the write up. I just got into the hobby of self hosting my own applications and it's quite a bit. I appreciated your sections about logging and user management.
deleted 2y ago
[deleted]
enkimin 2y ago
Article author here. Glad some people found this useful and to those with suggestions, ill keep those in mind when updating the post. Cheers.

169 points by joelp 2y ago

tiffanyh 2y ago

Love seeing devops post on HN. Wish it included server monitoring as a section.

everforward 2y ago
Server monitoring for self-hosting is kind of hard because it basically necessitates either buying another server to monitor the first, or paying for SaaS. From personal experience, I would just pay someone else for a SaaS monitoring solution. It will almost universally be cheaper and more reliable. If you really wanted to run your own, Prometheus is probably the way to go. Local storage should be fine as a data store for self-hosted. Grafana can be used for dashboarding, and either Grafana or AlertManager can do the alerting component. It’s really not all that worth it for self-hosted scale, though. Running all that in the cloud is going to cost basically the same as buying a DataDog license unless you’re at 3-ish hosts, and more than that if you’re doing clustered monitoring so you aren’t blind if your monitoring host is down.
neitsab 2y ago
Funny how that kind of posts is now called "DevOps", while 10 years ago it was simply called "system administration" ;-) Besides I fail to see any DevOps tenets in it, quite the opposite: a shell script at the bottom is little in the way of reliable automation. To me this post reads more like someone relatively new to server management wanted to share their gathered tips and tricks, i.e. me 10 years ago when I started my self-hosting journey :-D

abhinavk 2y ago

> You want to use SSH (Secure Shell) and make sure that SSH is the only way to log in. Some distributions (like openSuSE) also enable KbdInteractiveAuthentication by default so just disabling PasswordAuthentication won't work.

_blk 2y ago
I'm a bit sceptical of the choice of port 2222 as an alternative. At that point you might as well leave 22, but otherwise it's a good intro. If you're serious about starting post the sections into [insert AI service name] and start asking questions.
- 0cf8612b2e1e 2y ago
  Anything other than 22 is an improvement in just the reduced log volume.
  - dartos 2y ago
    Yeah, you basically dodge most automated attacks by getting off port 22
dsissitka 2y ago
This is one of those things I like to verify: david@desktop:~$ nmap -p 22 --script ssh-auth-methods becomesovran.com Starting Nmap 7.92 ( https://nmap.org ) at 2024-08-25 23:31 EDT Nmap scan report for becomesovran.com (162.213.255.209) Host is up (0.066s latency). rDNS record for 162.213.255.209: server1.becomesovran.com PORT STATE SERVICE 22/tcp open ssh | ssh-auth-methods: | Supported authentication methods: | publickey | gssapi-keyex | gssapi-with-mic | password |_ keyboard-interactive Nmap done: 1 IP address (1 host up) scanned in 0.86 seconds david@desktop:~$ As far as I can tell AuthenticationMethods publickey is the right way to do it these days but I'd love to know if that's not the case.
- yjftsjthsd-h 2y ago
  I've just been doing ssh -v localhost echo 2>&1 | grep continue (obviously replacing "localhost" with whatever server you want, and you can put anything you want where "echo" is but that's the best no-op I've come up with)
  - 2y ago

remram 2y ago

> Differential backups back up all the changes since the last full backup (...) An incremental backup backs up data that was changed since the last backup I'm not sure I understand the distinction?

magicalhippo 2y ago
They both do delta backups, but incremental bases it's delta on previous backup, while differential between the last full backup. To restore from an incremental you need the last full backup and all the incrementals inbetween. If you do say a full backup every month, you'd need up to 30 good incremental backup sets to be able to restore. For the differential you just need the last full backup in addition. Obviously the differential one might take more and more space, depending on the changes.
- remram 2y ago
  I see, thanks. I only use Restic so this is not relevant to me, but I think I understand the trade-off.
dsissitka 2y ago
Differential backups are always: Full Backup -> Differential Backup Incremental backups are: Full Backup -> Incremental Backup [-> Incremental Backup ...] At least that's how it is with Macrium.
betaby 2y ago
https://rdiff-backup.net/ https://rdiff-backup.net/ explains
pudgyblues 2y ago
With differential backups there's only 2 artifacts, the full and the diff. If you make another differential backup you overwrite the previous diff so it's always the changes since the last full backup. With incremental it's full backup + inc1 + inc2 +... forever, each backup depends on the previous.

hobobaggins 2y ago

I'd switch to Userify if you have a team to distribute keys for, because it's ultra-lightweight and also keeps you from messing up permissions on the ssh key/directory, which I've done too many times! (also it does sudo which is quite nice) Also, restarting ssh will not boot you out of the session (your session has already been forked as a different process), so leave your terminal window open (to fix any screwups) and then log in on a separate window on the new port and just make sure you can get in. For backups, don't set up logins from your main server(s) to your backup server; log in from your backup server to your main server. That way, if someone breaks into your main server, they can't get into your backup server.

AtlasBarfed 2y ago

So... about docker, did they backtrack on their licensing landgrab? About a year ago I swear everyone was going to podman, but in the last few months I see nothing but docker references. Podman is supposed to be drop-in. Well, it was advertised. I haven't touched anything in six months.

dartos 2y ago
IIRC docker had some heat for their docker desktop licenses. I think podman is more secure and simpler, but not as ergonomic to have locally (it’s not quite a drop in for docker. No real docker compose support for example) Podman is the default for k8s last I heard
bongobingo1 2y ago
I think Podman is still pretty fringe in the grand count. It also may suffer from an aspect of those that know about it and use it are less likely to write a blog post about it. I use it and prefer it, much so. Mostly because of rootless (I know docker has made attempts to improve this in the last year or so), not futzing with my iptables and a better handling of pushing images between hosts (it's been over a year since I touched any of that infra, I just remember it being more of hassle with docker which took a "our way or no way" approach). The biggest issue I have with Podman is the pace of its improvement against the rate of Debian releases!
mkesper 2y ago
Only use docker engine ("moby"). Docker desktop makes no sense on a Linux system as it introduces a VM into the mix, further adding complexity and reducing performance. https://docs.docker.com/engine/ https://docs.docker.com/engine/

Cyph0n 2y ago

Great post! I (relatively) recently switched my primary home server over to NixOS and am now a huge fan of it as a distribution for self-hosting. Here is how setting this all up would like in NixOS (modulo some details & machine-specific configuration). It's <100 lines, can be executed/configured with a single CLI command (even from a different machine!), rolled back easily if things go wrong, and can be re-used on any NixOS machine :) { networking = { # Server hostname hostName = "myserver"; # Firewall firewall = { enable = true; allowedTCPPorts = [ 80 443 2222 ]; }; }; # Users users.users = { newuser = { isNormalUser = true; home = "/home/newuser"; hashedPassword = "my-hashed-pwd"; openssh.authorizedKeys.keys = [ "my-pub-key" ]; }; }; # SSH services.openssh = { enable = true; ports = [ 2222 ]; settings = { PermitRootLogin = "no"; PasswordAuthentication = false; AllowUsers = [ "newuser" ]; }; extraConfig = '' Protocol 2 # Use only SSH protocol version 2 MaxAuthTries 3 # Limit authentication attempts ClientAliveInterval 300 # Client alive interval in seconds ClientAliveCountMax 2 # Maximum client alive count ''; }; services.fail2ban.enable = true; # Nginx + SSL via LetsEncrypt services.nginx = { enable = true; recommendedOptimisation = true; recommendedProxySettings = true; recommendedTlsSettings = true; virtualHosts = { "example.com" = { locations."/" = { proxyPass = "http://localhost:8080"; proxyWebsockets = true; }; forceSSL = true; enableACME = true; }; }; }; security.acme = { acceptTerms = true; defaults.email = "myemail@gmail.com"; certs."example.com" = { dnsProvider = "cloudflare"; environmentFile = ./my-env-file; }; }; # Logrotate services.logrotate = { enable = true; configFile = pkgs.writeText "logrotate.conf" '' /var/log/nginx/*.log { weekly missingok rotate 52 compress delaycompress notifempty create 0640 www-data adm sharedscripts postrotate [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid` endscript } ''; }; # Bonus: auto-upgrade from GH repo system.autoUpgrade = { enable = true; flake = "github:myuser/nixos-config"; flags = [ "-L" # print build logs "--refresh" # do not use cached Flake ]; dates = "00:00"; allowReboot = true; randomizedDelaySec = "45min"; }; }

crdrost 2y ago

Qq, do people doing their own server setup like this use containerization at all? When I looked at it, it was like “yeah you can run Docker or k3s,” and I think Hashicorp had their own version, but it seemed like folks didn't really bother? Also like setting up virtual networks among VPSes seemed like it required advanced wizardry.

lolinder 2y ago
I have an old desktop with Debian where I run a bunch of random things (Home Assistant, Pihole, Seafile, Jellyfin, ...) and I run everything with Podman Compose. I have enough things that I'm 100% confident I'd have run into dependency issues by now without containerization, but with Docker files it's trivial to keep them separate. As a bonus, compose.yml files are basically the lingua franca for describing deployments these days, so you can almost always find an example in the official docs for any given service you might want to host and get lots of help.
ozr 2y ago
> do people doing their own server setup like this use containerization at all? Depends on what you're deploying, really. If it's one Go service per host, there's no real need. Just a unit file and the binary. Your deployment scheme is scp and a restart. For more complicated setups, I've used docker compose. > Also like setting up virtual networks among VPSes seemed like it required advanced wizardry. Another 'it depends'. If you're running a small SaaS application, you probably don't need multiple servers in the first place. If you want some for redundancy, most providers offer a 'private network', where bandwidth is unmetered. Each compute provider is slightly different: you'll want to review their docs to see how to do it correctly. Tailscale is another option for networking, which is super easy to setup.
- cassianoleal 2y ago
  There’s rarely, if ever, a _need_ for containerisation. Even for a single static binary though, there are benefits like network and filesystem segregation, resource allocation, …
2y ago

erros 2y ago

You may want to update this post to disable password authentication, and thus you'll no longer need to install fail2ban. An important goal is to tighten your attack surface, not expand it. At this point you will still have an exposed SSHd server, so I'd recommend throwing the server under tailscale. You can setup the SSHd listener to use your tailscale IP or setup tailscale for SSH via ACLs (https://tailscale.com/tailscale-ssh https://tailscale.com/tailscale-ssh). Additionally you can further tighten controls of incoming logins with the use of AllowGroups to tighten your controls on which groups can log into the system. This would mitigate a scenario where an adversary is able to escalate enough privileges to write an .authorized_keys file to a non-privileged user which may have a shell still configured. Finally, unless you're treating this server as a bastion host of sorts, you probably should disable forwarding for agents or X11 etc. We've seen a lot of adversaries move laterally due to this agent forwarding.

Semaphor 2y ago
> You may want to update this post to disable password authentication Probably not, as that’s one of the first things they do. That said, I feel like all this fail2ban stuff is very much cargo culting in the selfhosting community. I’ve had my VPS SSH server on port 22 with no fail2ban for slightly over a decade, exposed to the public internet (home server is behind tailscale, VPS hosts the stuff I always want accessible from everywhere). Bots try it, they fail, the end. Maybe I’m missing something, but I have yet to find a good reason for the added complexity.
- ftrobro 2y ago
  This and possibly unknown similar flaws is a good reason for not advertising ssh: https://arstechnica.com/security/2024/07/regresshion-vulnerability-in-openssh-gives-attackers-root-on-linux/ https://arstechnica.com/security/2024/07/regresshion-vulnera...
- dingaling 2y ago
  fail2ban is great for reducing clutter in logs. When I'm trying to debug something using auth.log I don't want to sift through 800 attempts by some IP to SSH in as root. It gives just enough info about the origin and nature of attempted intruders without overwhelming detail.

jks 2y ago

I recommend checking out Caddy <https://caddyserver.com/ https://caddyserver.com/>, which replaces both Nginx and Certbot in this setup. Tailscale <https://tailscale.com/ https://tailscale.com/> can remove the need to open port 22 to the world, but I wouldn't rely on it unless your VPS provider has a way to access the server console in case of configuration mistakes.

jim180 2y ago
I've recently discovered, that Caddy config file has a neat support for imports: https://pastebin.com/vVQYrpmj https://pastebin.com/vVQYrpmj
Semaphor 2y ago
Caddy also simplifies many common Nginx configurations with a one-liner. The biggest hurdle is when you don’t have a simple configuration, as all the examples are usually only for Nginx ;)
calgoo 2y ago
For Tailscale backup access, another way is to block port 22 on a firewall and then only unblock it if you need access.
- nehal3m 2y ago
  If you depend on the host behind Tailscale to access the firewall from the inside then that's not going to work. Most colos I have hardware at offer a separate network for iDRAC/ILO/your flavor of OOB management, I like to use the console through that to open/close stuff like this.
InvOfSmallC 2y ago
Regarding tailscale, be sure to remove the expiration flag on your server. That's how I lost mine.

chadsix 2y ago

And for those of you that don't have an external IP, you can use services that provide egress for you like IPv6.rs. [1] [1] I'm DevOps there! ;)

ahofmann 2y ago
Thanks for the link, product looks good. Maybe you should work a bit on the website, this is how it looks at my pixel phone on Firefox: https://photos.app.goo.gl/txpxvDQAMYQWRSmi7 https://photos.app.goo.gl/txpxvDQAMYQWRSmi7

solatic 2y ago

Especially when writing a tutorial for beginners - please use the long-form flags (e.g. sudo usermod --append --groups sudo newuser) instead of short-form flags (e.g. sudo usermod -aG sudo newuser). Short-form flags make commands look like arcane voodoo magic. They make sense only to help you save time entering commands if you know them by heart already. Tutorials are read by beginners who are not necessarily familiar with the commands in the first place - long-form flags help communicate what these commands are actually doing and thus make for a more effective tutorial.

lhousa 2y ago
"Acronyms seriously suck" ~Elon
- benterix 2y ago
  At least it would be funny if signed by RMS.[0] [0] He wouldn't as he was a fan of recursive ones.
noahjk 2y ago
I would go as far as to say short flags should never be shared or saved (besides man pages or similar). Long flags help anyone who needs to review something in the future, even the author. Perfect for scripts of all sorts, tutorials, anything checked into git, etc.
- sebazzz 2y ago
  Powershell recommendations, for instance, are that you keep aliases and short-form for yourself, and long-form for scripts and tutorials. Remove-Item -Path X:\test\ -Recurse -Force del X:\test -rec -for
deviantintegral 2y ago
Yes please! This came up enough for us we standardized on this for all documentation and CI scripts. https://architecture.lullabot.com/adr/20211006-avoid-command-aliases/ https://architecture.lullabot.com/adr/20211006-avoid-command...

jw_cook 2y ago

At the end of the article, there's a link to a script[1] that does the steps covered in the article. That got me thinking: how do other self-hosters/homelabbers here go about automating their server setups? None/purely manual? One big shell script? Multiple scripts wrapped in a Makefile (or justfile, or other command runner)? More enterprisey provisioning/automation tools like Ansible, Puppet, etc.? [1] https://git.sovbit.dev/Enki/sovran-scripts https://git.sovbit.dev/Enki/sovran-scripts

jmathai 2y ago
One big shell script has worked really well for me. One project on AWS ran the script when new EC2 instances with a particular tag/label were spun up and that's how we scaled horizontally. What's nice about is that it doesn't require any specialized knowledge beyond bash - and that's something which is pretty easy to learn and great to know. It also attracts, IMO, the type of developers who avoid chasing new trends.
- Sammi 2y ago
  I have a folder of scripts. One main script that calls into the other scripts, just so I can keep my head straight. But one large script might work just as well for you. This sets up everything I need so I can treat my servers as livestock instead of pets - that is, so I can easily slaughter and replace them whenever I want, instead of being tied to them like a pet.
atoav 2y ago
For my use cases I found that just having a (updated) note with the things I would usually do works best. This is because I would not deploy everything anywhere and manually being aware of each step instead of hiding it within a script is somewhat a feature (e.g. you can easily insert a custom extra step etc). If I would do basically the same over and over I'd probably go with a script, ansible cookbook or similar, but as of now the manual route is totally fine.
- ranger207 2y ago

ValtteriL 2y ago

A monitoring setup with for instance Prometheus+Grafana would be a great addition to this. And then maybe automating all of it with something like Ansible.

davidmitchell2 2y ago

While these seems to be secure... tampering with default settings always cause PITA; especially during automated upgrades. In addition, ssh port changes are all security thru obscurity.

Sammi 2y ago
Just closing well known ports will mean less drive-by sniffing. Which is an improvement. Doesn't mean you are now completely safe - it's just an improvement. At the very least it will make your logs smaller, as they won't be as full of drive-by sniffing. Security is an onion, you can add layers. There is no perfect security. You can add hurdles and hope you make yourself too difficult for you adversary. Some hurdles add more than others, and not using well known ports is on the lesser end of the scale. You might still find it worthwhile, just so you have cleaner logs to sift through.

deleted 2y ago

[deleted]

voidUpdate 2y ago

With SSH Keys, do you have to just carry around your private key everywhere on a usb or something if you want to connect from multiple locations? Sometimes I find myself somewhere I've never been and I want to connect to my server. With a password that's easy, I just type it in, but I can't exactly create a new private key to access the server when I don't have access to the server in the first place

fragmede 2y ago
password managers can store ssh private keys, though there's still the problem of you've been mugged, have no cellphone wallet or keys, and need to break back into your life.
SSLy 2y ago
Some password managers expose SSH agent socket, so you can do that instead of fumbling with files.

nihilius 2y ago

Here are some "First Things on a Server" Notes. https://gist.github.com/klaushardt/07f5e3068355aafc2dce660a54be1ea7 https://gist.github.com/klaushardt/07f5e3068355aafc2dce660a5... Ansible/Puppet or NixOS would be better, but this is what works in Self Hosting.

thijsb 2y ago

Why the `sudo ufw allow outgoing`? Wouldn't it be worth to deny all to prevent extrusion and only open ports for services that need to communicate externally?

Sandbag5802 2y ago

Hey I just want to say thank you for the write up. I just got into the hobby of self hosting my own applications and it's quite a bit. I appreciated your sections about logging and user management.

deleted 2y ago

[deleted]

enkimin 2y ago

Article author here. Glad some people found this useful and to those with suggestions, ill keep those in mind when updating the post. Cheers.