9 ms·

Show HN: A CLI tool I made to self-host any app with two commands on a VPS

452 points by mightymoud 2y ago

MarvinJWendt 2y ago
Author of PTerm here, nice work! Let me know if you ever need help or have feedback about PTerm :)
Canada 2y ago
Very well presented, the README.md looks great.
- mightymoud 2y ago
  Thanks! This comment really makes my day!
Hexigonz 2y ago
Ohhhh I like this. I really enjoy the flyctl CLI tools from Fly.io, which simplifies in a similar manner, but it's platform specific. Good work
silasb 2y ago
Nice, I'm working in the same space as you (not opensource, personal project). We landed on the same solution, encoding the commands inside Golang and distributing those via SSH. I'm somewhat surprised not to see this more often. I'm guessing supporting multiple linux versions could get unwieldy, I focused on Ubuntu as my target. Differences that I see. * I modeled mine on-top of docker-plugins (these get installed during the bootstrapping process) * I built a custom plugin for deploying which leveraged https://github.com/Wowu/docker-rollout https://github.com/Wowu/docker-rollout for zero-downtime deployments Your solution looks much simpler than mine. I started off modeling mine off fly.io CLI, which is much more verbose Go code. I'll likely continue to use mine, but for any future VPS I'll have to give this a try.
- mightymoud 2y ago
  hahah seems like we went down the same rabbit hole. I also considered `docker-rollout` but decided to write my own script. Heavily inspired by the docker-rollout source code btw. Just curious, why did you decide to go with docker plugins?
jjkmk 2y ago
Looks really good, going to test it out.
superkuh 2y ago
I don't know about you but I find the single command $ sudo apt install $x to be much faster, offers wider range of software, more reliable, less fragile, easier to network, and more secure when it comes to running applications on an Ubuntu VPS. The only thing the normal way of running applications is less good at (compared to this dependency manager manager) is "Zero downtime".
- LVB 2y ago
  I’m not sure what you’re comparing that to. This project is about easily deploying your own app/side-project, which wouldn’t be available via apt.
  - superkuh 2y ago
    99% of what people run in docker is just normal applications.
    - indigodaddy 2y ago
      Not sure how true this statement is in general, but it’s definitely not true of course for what the project described as the use case, eg your own side project/app, which you’d obviously not be able to “apt install.” Unless OP meant like the supporting hosting/proxy infra like Apache/nginx, which yeah, that’s what this project is trying to avoid/abstract for the user to have to deal with. At the end of the day if you use this tool I guess all you’d need to worry about (given the tool is stable and works obviously) would be apt upgrades of the OS and even that you can automate, and then just figure out your reboot strategy. For me, I don’t even want to deal with that, so I happily use fly.
      - mightymoud 2y ago
        Respect! Fly is an absolute beast and to me is best in class for sure!
- mightymoud 2y ago
  I think this is just miscommunication - I meant more in a side-project/application that you made yourself. Not an application package you install on ubuntu
pqdbr 2y ago
This looks really nice, congrats! 1) I see Kamal was an inspiration; care to explain what differs from it? I'm still rocking custom Ansible playbooks, but I was planning on checking out Kamal after version 2 is released soon (I think alongside Rails 8). 2) I see databases are in your roadmap, and that's great. One feature that IMHO would be game changer for tools like this (and are lacking even in paid services like Hatchbox.io, which is overall great) is streaming replication of databases. Even for side projects, a periodic SQL dump stored in S3 is generally not enough nowadays, and any project that gains traction will need to implement some sort of streaming backup, like Litestream (for SQLite) or Barman with streaming backup (for Postgres). If I may suggest this feature, having this tool to provision a Barman server in a different VPS, and automate the process of having Postgres stream to it would be game changer. One barman server can actually accommodate multiple database backups, so N projects could do streaming backup to one single barman server. Of course, there would need to be a way to monitor if the streaming is working correctly, and maybe even help the user with the restoration process. But that effectively brings RTO down to near 0 (so no data loss) and can even allow point in time restoration.
- indigodaddy 2y ago
  Pretty sure that fly.io for example supports litestream as I remember seeing some fly doc related to litestream when I was looking a few days ago for my own project. Would also make sense that they do given Litestream’s creator is currently Fly’s VP of Product (I believe).
  - ctvo 2y ago
    Yes, fly.io is associated with Litestream, but... how is that related to the above thread or this tool?
    - indigodaddy 2y ago
      Quoted from the parent comment: “One feature that IMHO would be game changer for tools like this (and are lacking even in paid services like Hatchbox.io, which is overall great) is streaming replication of databases.” And then I mentioned that I believe fly.io has litestream support. I think it’s fairly relevant to the comment/thread.
LVB 2y ago
This looks good, and I’m a target user in this space. One thing I’ve noticed is the prevalence of Docker for this type of tool, or the larger self-managed PaaS tools. I totally get it, and it makes sense. I’m just slow to adapt. I’ve been so used to Go binary deployments for so long. But I also don’t really like tweaking Caddyfiles and futzing with systemd unit files, even though the pattern is familiar to me now. Been waffling on this for quite a while…
- kokanee 2y ago
  I'm a waffler on this as well, increasingly leaning away from containers lately. I can recall one time in my pre-Docker career when I was affected by a bug related to the fact that software developed on Mac OS ran differently than software running on CentOS in production. But I have spent untold countless hours trying to figure out various Docker-related quirks. If you legitimately need to run your software on multiple OSes in production, by all means, containerize it. But in 15 years I have never had a need to do that. I have a rock solid bash script that deploys and daemonizes an executable on a linux box, takes like 2 seconds to run, and saves me hours and hours of Dockery.
  - bantunes 2y ago
    I don't understand how running a single command to start either a single container or a stack of them with compose, that then gets all the requirements in a tarball similar and just runs is seen as more complicated than running random binaries, setting values on php.ini, setting up mysql or postgres, demonizing said binaries and making sure libraries and the like are in order.
    - bluehatbrit 2y ago
      I suppose like anything, it's a preference based on where the majority of your experience is, and what you're using it for. If you're running things you've written and it's all done the same way, docker probably is just an extra step. I personally run a bunch of software I've written, as well as open source things. So for me docker makes everything significantly easier, and saves me installing a lot of rubbish I don't understand well.
4star3star 2y ago
I like what I'm seeing, though I'm not sure I have a use case. On a VPS, I'll typically run a cloudflared container and configure a Cloudflare tunnel to that VPS. Then, I can expose any port and point it to a subdomain I configure in the CF dashboard. This gives https for free. I can expose services in containers or anything else running on the VPS. I'll concede there's probably a little more hands on work doing things this way, but I do like having a good grip on how things are working rather than leaning on a convenient tool. Maybe you could convince me Sidekick has more advantages?
- skinner927 2y ago
  I must be an old simpleton, but why get cloudflare involved? You can get https for free with nginx and letsencrypt.
  - mightymoud 2y ago
    It's a tunnel. So VPS can only be reached through cloudflare. It's not only for https, but more for security and lockdown
    - mediumsmart 2y ago
      excellent and if cloudflare thinks your IP is iranian its going to get a really secure lockdown.
      - nine_k 2y ago
        More seriously, it also helps when you're a target of a DDoS. It's always a balancing act between outsourcing your heavy lifting, and having to trust that party and depend on them.
        newaccount74 2y ago
        I've been running my business on VPS from Linode and Hetzner for the last 13 years and have never had an issue with DoS. I think the benefits of Cloudflares DoS protection are vastly oversold and absolutely unnecessary for 99% of businesses. (I think the false positives where some users randomly get captchas are actually bad for business)
bluehatbrit 2y ago
This is super nice, and I'm a big fan of the detailed readme with screenshots. I'll definitely be trying it out, although I do have a pretty nice setup now which will be hard to pull away from. It's ansible driven, lets me dump a compose file in a directory, along with a backup and restore shell script, and deploys it out to my server (hetzner dedicated via server auction). It's really nice that this handles TLS/SSL, that was a real pain for me as I've been using nginx and automating cerbot wasn't the most fun in the world. This looks a lot easier on that front!
- mightymoud 2y ago
  Sounds like you have a great setup. My vision is to make a setup like yours more accessible really w/o having to play with low level config like ansible. I think you should try to replace nginx with Traefik - it handles certs out of the box!
  - bluehatbrit 2y ago
    Mine is dead simple. I just have a repo with all my ansible in it, and have a nested module called "service". It takes in an app name, domain name, backup schedule, and a true/false on whether it should get a public nginx setup. Then it finds the compose file based on the app name. It templates in the domain name wherever needed in the compose file, and if it's meant to be public it'll setup a nginx config (which runs on the host, not in docker). If the folder with the compose file has a backup.sh and restore.sh it also copies those over, and sets up a cron for the backup schedule. It's less than 70 lines of yaml, plus some more for restart handlers. The only bit that irks me is the initial tls/ssl setup. Certbot changes the nginx config to insert the various certificates, which then makes my original nginx config out of date. I really like nginx and have used it for a long time so feel comfortable with it, but I've been considering traefik and caddy for a while just to get around this. Although another option for me is to use a cloudflare tunnel instead, and then ignoring certificate management altogether. This is really attractive because it also means I can close some ports. I'll have to find some time to play around with traefik and caddy first though!
    - 2y ago
devmor 2y ago
Wow this is super handy! I have paid tools that function like this for a couple of specific stacks but this seems like an amazing general purpose tool. Considering the ease of setup the README purports, a few hours of dealing with this might save me a couple hundred bucks a month in service fees.
- mightymoud 2y ago
  Glad you found this useful. Let me know if you have specific features in mind.
  - devmor 2y ago
    I didn't see anything in the readme about deploy hooks - do you have a feature that lets users run arbitrary commands after the image is deployed? I have common use cases for both pre (ex. database migrations) and post (ex. Resource caching, worker spinup) traffic switchover.
    - mightymoud 2y ago
      Yup deploy hooks are on my mind. Just didn't put them on Readme. Shouldn't be very hard to implement. Might do this first before docker-compose support.
achempion 2y ago
This looks amazing, congrats on the release! Really looking forward for the database hosting feature as well (and probably networking and mounting data dirs). As a side note, any reason why you decided against using docker in swarm mode as it should have all these features already built it?
- mightymoud 2y ago
  Correct me if I'm wrong, Docker Swarm mode is made to manage multi node clusters. This is meant for only one single VPS.
  - achempion 2y ago
    You can use docker swarm just for single VPS. - install docker - run docker swarm init - create yaml that describes your stack (similar to docker-compose) - run docker stack deploy That's basically it. My go-to solution when I need to run some service on single VPS. If you want to just run a single container, you can also do this with `docker service create image:tag`
    - 3np 2y ago
      I thought docker-swarm had been considered neglected to the point of dead and without a future for a few years now. Is this impression incorrect/outdated? EDIT: So apparently what used to be known as "Docker Swarm" has been posthumously renamed to "Swarm Classic"/"Classic Swarm" and is indeed dead, abandoned, and deprecated. The project currently known as "Docker Swarm" is a younger completely different project which appears actively maintained. "Classic" still has roughly twice the GH stars and forks compared to the new one. I can't be the only one who's dismissed the latter, assuming it to be the former. Very confusing naming and branding, they would probably have more way more users if they had not repurposed the name like this. https://github.com/docker-archive/classicswarm https://github.com/docker-archive/classicswarm > Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit https://github.com/docker/swarmkit
  - evantbyrne 2y ago
dvaun 2y ago
Awesome! Love that it's written in Go—I've recently tested the language for some use cases at work and find it great. I'll dive into your repo to see if I can learn anything new :)
rafaelgoncalves 2y ago
This really looks nice! Congrats!
Sn0wCoder 2y ago
This looks great. Just bookmarked and then had to double check that I did not just bookmark it a few weeks ago. Turns out I had bookmarked Caddy which is similar but does not deploy the app and don’t think supports Docker. It was the auto CERT that was what I was interested in and what had stuck out in my mind. Have certbot setup and never think about it again, until my server needed to be rebuilt, and I started researching. Good to go for a few months, but my hosting will be up here in a year and going to switch providers and upgrade my setup to 2+ gig so I can run docker reliably. Thanks for posting this one just moved to the top of the list.
- indigodaddy 2y ago
  In what sense would Caddy not support Docker? You can use caddy on the host itself to proxy to a docker container, and you could also have Caddy as a Docker container to proxy to other Docker containers (would just need an initial incoming iptables rule to the caddy container for the latter scenario— although caddy might have instructions somewhere on a more elegant way than iptables to get the connections to the Docker caddy container not sure)
  - Sn0wCoder 2y ago
    Hey indigo, Thank you for pointing this out. When I was looking to install caddy, I was specifically looking for something without using docker since my VPS is 1g / 1cpu and that is what I based my comment off. When was reading the sidekick docs seemed by running one command that it would first install sidekick and then install the cert/app all with one docker file but now I am not even sure about that. Appreciate you pointing that out, now I am back into analysis paralysis on which one I should use
spelunker 2y ago
Looks great! I similarly got frustrated about the complexity of doing side-project ops stuff and messed around with Kamal, but this goes the extra mile by automatically setting up TLS as well. I'll give it a try!
AndrewCopeland 2y ago
Its a simple cli in go It uses docker There is no k8s Handles certs Zero down time I would love for it to support docker-compose as some of my side projects needs a library in python but I like having my service be in go, so I will wrap the python library in a super simple service. Overall this is awesome and I love the simplicity, with the world just full of serverless, AI and a bunch of other "stuff". Paralysis through analysis is really an issue and when you are just trying to create a service for yourself or an MVP, it can be a real hinderance. I have been gravitating towards Taskfile to perform similar tasks to this. God speed to you and keep up the great work.
- mightymoud 2y ago
  Thanks man! I'm working on the docker-compose support. I got it working locally, but the ergonomics are really hard to get right, cus compose files are so flexible. I was even considering using the `sidekick.yaml` file as the main config and then turn that into docker compose - similar to what fly.io does with fly.toml. But I wanna keep this docker centric... so yeah I am still doing more thinking around this
replwoacause 2y ago
Can’t wait to try this out..!
throwaway984393 2y ago
[dead]
aag 2y ago
This could be great for my projects, but I'm confused about one thing: why does it need to push to a Docker registry? The Dockerfile is local, and each image is built locally. Can't the images be stored purely locally? Perhaps I'm missing something obvious. Not using a registry would reduce the number of moving parts.
- mightymoud 2y ago
  Locally here means the locally on your laptop locally, not locally on your VPS. Contrary to popular opinion, I believe your source code shouldn't be on your prod machine - a docker image is all you need. Lots of other projects push your code to VPS to build the image there then use it. I see no point in doing that...
  - sdf4j 2y ago
    The docker registry can be avoided by exporting/importing the docker image over ssh.
    - mightymoud 2y ago
      [dead]
    - aflukasz 2y ago
      At the cost of losing layers reusage at transport time.
- 3np 2y ago
  You can easily set up a Docker/CNCF registry[0] container running locally. It can be run either as a caching pull-through mirror for a public registry (allowing you to easily run public containers in an environment without internet access) or as a private registry for your own image (this use-case). So if you want both features, you currently need two instances. Securing it for public use is a bit less trivial but for local use it's literally a 'run' or two. So you can do 'docker build -t localhost/whatever' and then 'docker run localhost/whatever'. Also worth checking out podman to more easily run everything rootless. If all you need is to move images between hosts like you would files, you don't even need a registry (docker save/load). [0]: https://distribution.github.io/distribution/ https://distribution.github.io/distribution/
joseferben 2y ago
this looks amazing! i’m building https://www.plainweb.dev https://www.plainweb.dev and i’m looking for the simplest way to deploy a plainweb/plainstack project. looks like sidekick has the same spirit when it comes to simplicity. in the plainstack docs i’ve been embracing fly.io, but reliability is an issue. and sqlite web apps (which is the core of plainstack) can’t have real zero downtime deployments, unless you count the proxy holding the pending request for 30 seconds while the fly machine is deployed. i tried kamal but it felt like non-ruby and non-rails projects are second class citizens. i was about to document deploying plainstack to dokku, but provisioning isn’t built-in. my dream deployment tool would be dokku + provisioning & setup, sidekick looks very close to that. definitely going to try this and maybe even have it in the blessed deploy path for plainstack if it works well!
- mightymoud 2y ago
  Planweb looks really promising. Keen to get in touch and see how I can help support the vision of Plainweb with Sidekick then you can reference it in the docs. I'll reach out on twitter
johnklos 2y ago
"to self-host any app" Docker != app. Perhaps it'd be more accurate to say, "to host any Docker container"?
- prmoustache 2y ago
  I am not saying docker is the best way to run any app, but could you share a kind of app can't you host in a docker? I mean on rootless containers, yes, a lot of apps that need access to the underlying system might not work, but they are usually system stuff, not the kind you want to host on a VPS anyway. But when running as root I can't think of many.
  - johnklos 2y ago
    There are many, many things that can't be hosted in Docker. For instance, literally everything that I run on many of my servers can't, because Docker only supports certain platforms and architectures. Things that we want to run on systems with modest amounts of CPU and memory can't be run via Docker for obvious reasons. Stuff that needs hardware access can't. Stuff that needs finer resolution of permissions can, but there's a lot more work that needs to be done. But even ignoring those, if I'm going to spend all the time needed to containerize everything in to Docker images myself, why wouldn't I just run the programs directly and not deal with the overhead and extra work?
InvOfSmallC 2y ago
Can I run more than one app on the same VPS with this solution? I now run more than one app into one single VPS.
- mightymoud 2y ago
  [dead]
sigmonsays 2y ago
tools like this are pretty sweet but I would rather just run it myself. docker-compose with a load balancer (traefik) is fairly straightforward and awesome. the TLS setup is nice but I wildcard that and just run certgen myself. The main thing I think that's missing is some sort of authentication or zero trust system, maybe vpn tunnel provisioner. Most services I self host I do not want to be made public due to security concerns.
funshed 2y ago
Nice, you should probably explain what traefik, sops and age will do. First time I've heard of sops, very handy!
tegiddrone 2y ago
Looks nice! Something I'd want in front is some sort of basic app firewall like fail2ban or CrowdSec to ban vuln scanners and other intrusion attempts. It is a nice thing about Cloudflare since they provide some of this protection.
turtlebits 2y ago
What about this is highly available? On a single VPS? Does this only support a single app? Nice project but the claims (production ready? Load balance on a single server?) are a bit ridiculous.
- dewey 2y ago
  Highly available is overrated for most use cases, especially for any side projects.
- closewith 2y ago
  In my experience, single apps on VPSes have far higher availability in practice than the majority of convoluted deployments.
dewey 2y ago
I'd also suggest people to take a look at Dokku, it's a very mature project with a similar scope and was discussed here a few weeks ago: https://news.ycombinator.com/item?id=41358020 https://news.ycombinator.com/item?id=41358020 I wrote up my own experiences too (https://blog.notmyhostna.me/posts/selfhosting-with-dokku-and-docker-compose https://blog.notmyhostna.me/posts/selfhosting-with-dokku-and...) and I can only recommend it. It is ~3 commands to set up an app, and one push to deploy after that.
- rgrieselhuber 2y ago
  I've kept a list of these tools that I've been meaning to check out. In scope, do they cover securing the instance? Is there any automation for creating networks of instances?
  - dewey 2y ago
    > In scope, do they cover securing the instance? Most of these I checked don't, but a recent Ubuntu version is perfectly fine to use as-is. > Is there any automation for creating networks of instances? Not that I'm aware, it would also defeat the purpose of these tools a bit that are supposed to be simple. (Dokku is "just" a shell script).
- mimischi 2y ago
  Been using dokku for probably 8 years now? (or something close to that; it used to be written entirely in bash!) Hosting private stuff on it, and an application at $oldplace probably also still runs on this solid setup. Highly recommended, and the devs are a great sport!
- FloatArtifact 2y ago
  Part of me dies every time I see projects not integrating robust restoring and backup systems.
  - Aeolun 2y ago
    None of my hobby projects across 15 years or so have ever needed backups or restoring. I can agree it would be nice to have, but it’s a far cry from necessary.
nhatcher 2y ago
This looks fantastic TBH! Can't wait to give it a go. Congratulations. I've long thought something like this should be possible. The only thing I've done is document carefully my own steps: https://www.nhatcher.com/post/a-cto-on-a-shoestring/ https://www.nhatcher.com/post/a-cto-on-a-shoestring/
gf297 2y ago
What's the purpose of encrypting the env file with sops, when the age secret key is stored on the VPS? If someone has access to the encrypted env file, they will also have access to the secret key, and can decrypt it.
- mightymoud 2y ago
  [dead]
- nsluss 2y ago
  only if they can login as the owner of the key. Most of my team can get on our vps, only a couple could get our age key.
trey-jones 2y ago
"Wow, this really looks significantly better than my own CLI tools" I'm going to have to look into this pterm thing.
hkon 2y ago
Have used caprover. Good that more tools enter this space.
singhrac 2y ago
Any possibility you’d add support for a Mac Mini deployment? I think the extra complexity would be from changing the Docker images, but of course the devil is in the details. I just have a Mac Mini and it would be great to self-host some stuff.
- brirec 2y ago
  As someone who used to love hosting things on a Mac mini, have you tried installing Linux on it to use as a dedicated server? If you do, it should handle this just like any other platform you could install it on.
Humphrey 2y ago
Love this! That said, I achieve the same thing manually using Docker Compose & some shell scripts. It takes a bike longer, but it has forced me to learn the lower level tools that helpers like Sidekick use. Also, all of these tools have great documentation on getting up and running, but SIGNIFICANTLY LESS INFO ON HOW TO MAINTAIN OVER THE LONG TERM. If I was going to start using a tool like Sidekick, Kamal, or Dokku I would want clear answers to the following: - How do I keep my VPS host up and running with latest security updates? - How do I update to more recent versions of Docker? - How do I update services that maintain state (eg. update to a new Postgres version) - How do I seamlessly migrate to a new host (perhaps as a way to solve the above)? - How should I manage and serve static resources & user media? (store on host or use cloud storage?) - How do I manage database migrations during an update, and how do I control that process to avoid downtime during an update? I just spent an entire evening transferring a side project to a new VPS because I needed to update Postgres. The ideal self-hosting solution would make that a 20 min task.
jfdi 2y ago
These are great. Having tooling to get stuff out fast and as safely as possible to get to iterating openly. Here’s a bash script I posted a while back on a different thread that does similar thing if of interest for anyone. It’s probably less nice than op’s for ex it only works with digitalocean (which is great!) - but it’s simple small and mostly readable. also assumes docker - but all via compose, with some samples like nginx w auto-ssl via le. https://github.com/thomaswilley/tide.sh https://github.com/thomaswilley/tide.sh
mattfrommars 2y ago
I consider myself bit techsavy knowing Linux and basic scripting. But does anyone have a resource or link that explains the idea to make a service which OP shared here? Because frankly, I'd feel lost reading the code from one file at a time without knowing where to start. Plus it's written in Go which I have I am not familiar with.
- mightymoud 2y ago
  I wrote a intro guide to pterm and cobra-cli here https://mightytuts.com/blog/golang-cli-introduction-with-cobra-and-pterm/ https://mightytuts.com/blog/golang-cli-introduction-with-cob... I also have a video on YT if that is your thing
interstice 2y ago
Really like this! Funnily enough I was just rabbit holing into terraform + ansible in an effort to do essentially this but with an anycast network. The thinking was to mirror apps across locations with a single deploy. I don't suppose you're plannning something similar with this one?
- apitman 2y ago
  Anycast is a really underrated feature and I'm surprised no major VPS providers offer it. It's one of my favorite things about fly.io
vander_elst 2y ago
How does it compare to ansible? I didn't immediately find a link
vickodin 2y ago
Very interesting. The similarity to the sidekiq is a bit confusing, but it doesn't really matter.
strzibny 2y ago
Given the choice of Docker and Traefik I would love to know what's the exact difference to Kamal? And btw Kamal will soon have a new improved version with a custom proxy.
udev4096 2y ago
I made something similar sometime ago (without using docker): http://github.com/thebigbone/deploypot http://github.com/thebigbone/deploypot
pmarreck 2y ago
I would love this, but with Nix.
worik 2y ago
Nice work. I wonder, though. Why Ubuntu? Why not Debian? With all due respect to Cannonical, Ubuntu is not really suitable. It is not aimed at developers Unless it has changed since I left it in a fury, it takes too much control away from you with the Snap system.
mutant 2y ago
I just tried this out on DO and Linode. Worked great! Very cool stack.
lenkite 2y ago
Gorgeous tool! Thanks for making this.

452 points by mightymoud 2y ago

MarvinJWendt 2y ago

Author of PTerm here, nice work! Let me know if you ever need help or have feedback about PTerm :)

Canada 2y ago

Very well presented, the README.md looks great.

mightymoud 2y ago
Thanks! This comment really makes my day!

Hexigonz 2y ago

Ohhhh I like this. I really enjoy the flyctl CLI tools from Fly.io, which simplifies in a similar manner, but it's platform specific. Good work

silasb 2y ago

Nice, I'm working in the same space as you (not opensource, personal project). We landed on the same solution, encoding the commands inside Golang and distributing those via SSH. I'm somewhat surprised not to see this more often. I'm guessing supporting multiple linux versions could get unwieldy, I focused on Ubuntu as my target. Differences that I see. * I modeled mine on-top of docker-plugins (these get installed during the bootstrapping process) * I built a custom plugin for deploying which leveraged https://github.com/Wowu/docker-rollout https://github.com/Wowu/docker-rollout for zero-downtime deployments Your solution looks much simpler than mine. I started off modeling mine off fly.io CLI, which is much more verbose Go code. I'll likely continue to use mine, but for any future VPS I'll have to give this a try.

mightymoud 2y ago
hahah seems like we went down the same rabbit hole. I also considered `docker-rollout` but decided to write my own script. Heavily inspired by the docker-rollout source code btw. Just curious, why did you decide to go with docker plugins?

jjkmk 2y ago

Looks really good, going to test it out.

superkuh 2y ago

I don't know about you but I find the single command $ sudo apt install $x to be much faster, offers wider range of software, more reliable, less fragile, easier to network, and more secure when it comes to running applications on an Ubuntu VPS. The only thing the normal way of running applications is less good at (compared to this dependency manager manager) is "Zero downtime".

LVB 2y ago
I’m not sure what you’re comparing that to. This project is about easily deploying your own app/side-project, which wouldn’t be available via apt.
- superkuh 2y ago
  99% of what people run in docker is just normal applications.
  - indigodaddy 2y ago
    Not sure how true this statement is in general, but it’s definitely not true of course for what the project described as the use case, eg your own side project/app, which you’d obviously not be able to “apt install.” Unless OP meant like the supporting hosting/proxy infra like Apache/nginx, which yeah, that’s what this project is trying to avoid/abstract for the user to have to deal with. At the end of the day if you use this tool I guess all you’d need to worry about (given the tool is stable and works obviously) would be apt upgrades of the OS and even that you can automate, and then just figure out your reboot strategy. For me, I don’t even want to deal with that, so I happily use fly.
    - mightymoud 2y ago
      Respect! Fly is an absolute beast and to me is best in class for sure!
mightymoud 2y ago
I think this is just miscommunication - I meant more in a side-project/application that you made yourself. Not an application package you install on ubuntu

pqdbr 2y ago

This looks really nice, congrats! 1) I see Kamal was an inspiration; care to explain what differs from it? I'm still rocking custom Ansible playbooks, but I was planning on checking out Kamal after version 2 is released soon (I think alongside Rails 8). 2) I see databases are in your roadmap, and that's great. One feature that IMHO would be game changer for tools like this (and are lacking even in paid services like Hatchbox.io, which is overall great) is streaming replication of databases. Even for side projects, a periodic SQL dump stored in S3 is generally not enough nowadays, and any project that gains traction will need to implement some sort of streaming backup, like Litestream (for SQLite) or Barman with streaming backup (for Postgres). If I may suggest this feature, having this tool to provision a Barman server in a different VPS, and automate the process of having Postgres stream to it would be game changer. One barman server can actually accommodate multiple database backups, so N projects could do streaming backup to one single barman server. Of course, there would need to be a way to monitor if the streaming is working correctly, and maybe even help the user with the restoration process. But that effectively brings RTO down to near 0 (so no data loss) and can even allow point in time restoration.

indigodaddy 2y ago
Pretty sure that fly.io for example supports litestream as I remember seeing some fly doc related to litestream when I was looking a few days ago for my own project. Would also make sense that they do given Litestream’s creator is currently Fly’s VP of Product (I believe).
- ctvo 2y ago
  Yes, fly.io is associated with Litestream, but... how is that related to the above thread or this tool?
  - indigodaddy 2y ago
    Quoted from the parent comment: “One feature that IMHO would be game changer for tools like this (and are lacking even in paid services like Hatchbox.io, which is overall great) is streaming replication of databases.” And then I mentioned that I believe fly.io has litestream support. I think it’s fairly relevant to the comment/thread.

LVB 2y ago

This looks good, and I’m a target user in this space. One thing I’ve noticed is the prevalence of Docker for this type of tool, or the larger self-managed PaaS tools. I totally get it, and it makes sense. I’m just slow to adapt. I’ve been so used to Go binary deployments for so long. But I also don’t really like tweaking Caddyfiles and futzing with systemd unit files, even though the pattern is familiar to me now. Been waffling on this for quite a while…

kokanee 2y ago
I'm a waffler on this as well, increasingly leaning away from containers lately. I can recall one time in my pre-Docker career when I was affected by a bug related to the fact that software developed on Mac OS ran differently than software running on CentOS in production. But I have spent untold countless hours trying to figure out various Docker-related quirks. If you legitimately need to run your software on multiple OSes in production, by all means, containerize it. But in 15 years I have never had a need to do that. I have a rock solid bash script that deploys and daemonizes an executable on a linux box, takes like 2 seconds to run, and saves me hours and hours of Dockery.
- bantunes 2y ago
  I don't understand how running a single command to start either a single container or a stack of them with compose, that then gets all the requirements in a tarball similar and just runs is seen as more complicated than running random binaries, setting values on php.ini, setting up mysql or postgres, demonizing said binaries and making sure libraries and the like are in order.
  - bluehatbrit 2y ago
    I suppose like anything, it's a preference based on where the majority of your experience is, and what you're using it for. If you're running things you've written and it's all done the same way, docker probably is just an extra step. I personally run a bunch of software I've written, as well as open source things. So for me docker makes everything significantly easier, and saves me installing a lot of rubbish I don't understand well.

4star3star 2y ago

I like what I'm seeing, though I'm not sure I have a use case. On a VPS, I'll typically run a cloudflared container and configure a Cloudflare tunnel to that VPS. Then, I can expose any port and point it to a subdomain I configure in the CF dashboard. This gives https for free. I can expose services in containers or anything else running on the VPS. I'll concede there's probably a little more hands on work doing things this way, but I do like having a good grip on how things are working rather than leaning on a convenient tool. Maybe you could convince me Sidekick has more advantages?

skinner927 2y ago
I must be an old simpleton, but why get cloudflare involved? You can get https for free with nginx and letsencrypt.
- mightymoud 2y ago
  It's a tunnel. So VPS can only be reached through cloudflare. It's not only for https, but more for security and lockdown
  - mediumsmart 2y ago
    excellent and if cloudflare thinks your IP is iranian its going to get a really secure lockdown.
    - nine_k 2y ago
      More seriously, it also helps when you're a target of a DDoS. It's always a balancing act between outsourcing your heavy lifting, and having to trust that party and depend on them.
      - newaccount74 2y ago
        I've been running my business on VPS from Linode and Hetzner for the last 13 years and have never had an issue with DoS. I think the benefits of Cloudflares DoS protection are vastly oversold and absolutely unnecessary for 99% of businesses. (I think the false positives where some users randomly get captchas are actually bad for business)

bluehatbrit 2y ago

This is super nice, and I'm a big fan of the detailed readme with screenshots. I'll definitely be trying it out, although I do have a pretty nice setup now which will be hard to pull away from. It's ansible driven, lets me dump a compose file in a directory, along with a backup and restore shell script, and deploys it out to my server (hetzner dedicated via server auction). It's really nice that this handles TLS/SSL, that was a real pain for me as I've been using nginx and automating cerbot wasn't the most fun in the world. This looks a lot easier on that front!

mightymoud 2y ago
Sounds like you have a great setup. My vision is to make a setup like yours more accessible really w/o having to play with low level config like ansible. I think you should try to replace nginx with Traefik - it handles certs out of the box!
- bluehatbrit 2y ago
  Mine is dead simple. I just have a repo with all my ansible in it, and have a nested module called "service". It takes in an app name, domain name, backup schedule, and a true/false on whether it should get a public nginx setup. Then it finds the compose file based on the app name. It templates in the domain name wherever needed in the compose file, and if it's meant to be public it'll setup a nginx config (which runs on the host, not in docker). If the folder with the compose file has a backup.sh and restore.sh it also copies those over, and sets up a cron for the backup schedule. It's less than 70 lines of yaml, plus some more for restart handlers. The only bit that irks me is the initial tls/ssl setup. Certbot changes the nginx config to insert the various certificates, which then makes my original nginx config out of date. I really like nginx and have used it for a long time so feel comfortable with it, but I've been considering traefik and caddy for a while just to get around this. Although another option for me is to use a cloudflare tunnel instead, and then ignoring certificate management altogether. This is really attractive because it also means I can close some ports. I'll have to find some time to play around with traefik and caddy first though!
  - 2y ago

devmor 2y ago

Wow this is super handy! I have paid tools that function like this for a couple of specific stacks but this seems like an amazing general purpose tool. Considering the ease of setup the README purports, a few hours of dealing with this might save me a couple hundred bucks a month in service fees.

mightymoud 2y ago
Glad you found this useful. Let me know if you have specific features in mind.
- devmor 2y ago
  I didn't see anything in the readme about deploy hooks - do you have a feature that lets users run arbitrary commands after the image is deployed? I have common use cases for both pre (ex. database migrations) and post (ex. Resource caching, worker spinup) traffic switchover.
  - mightymoud 2y ago
    Yup deploy hooks are on my mind. Just didn't put them on Readme. Shouldn't be very hard to implement. Might do this first before docker-compose support.

achempion 2y ago

This looks amazing, congrats on the release! Really looking forward for the database hosting feature as well (and probably networking and mounting data dirs). As a side note, any reason why you decided against using docker in swarm mode as it should have all these features already built it?

mightymoud 2y ago
Correct me if I'm wrong, Docker Swarm mode is made to manage multi node clusters. This is meant for only one single VPS.
- achempion 2y ago
  You can use docker swarm just for single VPS. - install docker - run docker swarm init - create yaml that describes your stack (similar to docker-compose) - run docker stack deploy That's basically it. My go-to solution when I need to run some service on single VPS. If you want to just run a single container, you can also do this with `docker service create image:tag`
  - 3np 2y ago
    I thought docker-swarm had been considered neglected to the point of dead and without a future for a few years now. Is this impression incorrect/outdated? EDIT: So apparently what used to be known as "Docker Swarm" has been posthumously renamed to "Swarm Classic"/"Classic Swarm" and is indeed dead, abandoned, and deprecated. The project currently known as "Docker Swarm" is a younger completely different project which appears actively maintained. "Classic" still has roughly twice the GH stars and forks compared to the new one. I can't be the only one who's dismissed the latter, assuming it to be the former. Very confusing naming and branding, they would probably have more way more users if they had not repurposed the name like this. https://github.com/docker-archive/classicswarm https://github.com/docker-archive/classicswarm > Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit https://github.com/docker/swarmkit
- evantbyrne 2y ago

dvaun 2y ago

Awesome! Love that it's written in Go—I've recently tested the language for some use cases at work and find it great. I'll dive into your repo to see if I can learn anything new :)

rafaelgoncalves 2y ago

This really looks nice! Congrats!

Sn0wCoder 2y ago

This looks great. Just bookmarked and then had to double check that I did not just bookmark it a few weeks ago. Turns out I had bookmarked Caddy which is similar but does not deploy the app and don’t think supports Docker. It was the auto CERT that was what I was interested in and what had stuck out in my mind. Have certbot setup and never think about it again, until my server needed to be rebuilt, and I started researching. Good to go for a few months, but my hosting will be up here in a year and going to switch providers and upgrade my setup to 2+ gig so I can run docker reliably. Thanks for posting this one just moved to the top of the list.

indigodaddy 2y ago
In what sense would Caddy not support Docker? You can use caddy on the host itself to proxy to a docker container, and you could also have Caddy as a Docker container to proxy to other Docker containers (would just need an initial incoming iptables rule to the caddy container for the latter scenario— although caddy might have instructions somewhere on a more elegant way than iptables to get the connections to the Docker caddy container not sure)
- Sn0wCoder 2y ago
  Hey indigo, Thank you for pointing this out. When I was looking to install caddy, I was specifically looking for something without using docker since my VPS is 1g / 1cpu and that is what I based my comment off. When was reading the sidekick docs seemed by running one command that it would first install sidekick and then install the cert/app all with one docker file but now I am not even sure about that. Appreciate you pointing that out, now I am back into analysis paralysis on which one I should use

spelunker 2y ago

Looks great! I similarly got frustrated about the complexity of doing side-project ops stuff and messed around with Kamal, but this goes the extra mile by automatically setting up TLS as well. I'll give it a try!

AndrewCopeland 2y ago

Its a simple cli in go It uses docker There is no k8s Handles certs Zero down time I would love for it to support docker-compose as some of my side projects needs a library in python but I like having my service be in go, so I will wrap the python library in a super simple service. Overall this is awesome and I love the simplicity, with the world just full of serverless, AI and a bunch of other "stuff". Paralysis through analysis is really an issue and when you are just trying to create a service for yourself or an MVP, it can be a real hinderance. I have been gravitating towards Taskfile to perform similar tasks to this. God speed to you and keep up the great work.

mightymoud 2y ago
Thanks man! I'm working on the docker-compose support. I got it working locally, but the ergonomics are really hard to get right, cus compose files are so flexible. I was even considering using the `sidekick.yaml` file as the main config and then turn that into docker compose - similar to what fly.io does with fly.toml. But I wanna keep this docker centric... so yeah I am still doing more thinking around this

replwoacause 2y ago

Can’t wait to try this out..!

throwaway984393 2y ago

[dead]

aag 2y ago

This could be great for my projects, but I'm confused about one thing: why does it need to push to a Docker registry? The Dockerfile is local, and each image is built locally. Can't the images be stored purely locally? Perhaps I'm missing something obvious. Not using a registry would reduce the number of moving parts.

mightymoud 2y ago
Locally here means the locally on your laptop locally, not locally on your VPS. Contrary to popular opinion, I believe your source code shouldn't be on your prod machine - a docker image is all you need. Lots of other projects push your code to VPS to build the image there then use it. I see no point in doing that...
- sdf4j 2y ago
  The docker registry can be avoided by exporting/importing the docker image over ssh.
  - mightymoud 2y ago
    [dead]
  - aflukasz 2y ago
    At the cost of losing layers reusage at transport time.
3np 2y ago
You can easily set up a Docker/CNCF registry[0] container running locally. It can be run either as a caching pull-through mirror for a public registry (allowing you to easily run public containers in an environment without internet access) or as a private registry for your own image (this use-case). So if you want both features, you currently need two instances. Securing it for public use is a bit less trivial but for local use it's literally a 'run' or two. So you can do 'docker build -t localhost/whatever' and then 'docker run localhost/whatever'. Also worth checking out podman to more easily run everything rootless. If all you need is to move images between hosts like you would files, you don't even need a registry (docker save/load). [0]: https://distribution.github.io/distribution/ https://distribution.github.io/distribution/

joseferben 2y ago

this looks amazing! i’m building https://www.plainweb.dev https://www.plainweb.dev and i’m looking for the simplest way to deploy a plainweb/plainstack project. looks like sidekick has the same spirit when it comes to simplicity. in the plainstack docs i’ve been embracing fly.io, but reliability is an issue. and sqlite web apps (which is the core of plainstack) can’t have real zero downtime deployments, unless you count the proxy holding the pending request for 30 seconds while the fly machine is deployed. i tried kamal but it felt like non-ruby and non-rails projects are second class citizens. i was about to document deploying plainstack to dokku, but provisioning isn’t built-in. my dream deployment tool would be dokku + provisioning & setup, sidekick looks very close to that. definitely going to try this and maybe even have it in the blessed deploy path for plainstack if it works well!

mightymoud 2y ago
Planweb looks really promising. Keen to get in touch and see how I can help support the vision of Plainweb with Sidekick then you can reference it in the docs. I'll reach out on twitter

johnklos 2y ago

"to self-host any app" Docker != app. Perhaps it'd be more accurate to say, "to host any Docker container"?

prmoustache 2y ago
I am not saying docker is the best way to run any app, but could you share a kind of app can't you host in a docker? I mean on rootless containers, yes, a lot of apps that need access to the underlying system might not work, but they are usually system stuff, not the kind you want to host on a VPS anyway. But when running as root I can't think of many.
- johnklos 2y ago
  There are many, many things that can't be hosted in Docker. For instance, literally everything that I run on many of my servers can't, because Docker only supports certain platforms and architectures. Things that we want to run on systems with modest amounts of CPU and memory can't be run via Docker for obvious reasons. Stuff that needs hardware access can't. Stuff that needs finer resolution of permissions can, but there's a lot more work that needs to be done. But even ignoring those, if I'm going to spend all the time needed to containerize everything in to Docker images myself, why wouldn't I just run the programs directly and not deal with the overhead and extra work?

InvOfSmallC 2y ago

Can I run more than one app on the same VPS with this solution? I now run more than one app into one single VPS.

mightymoud 2y ago
[dead]

sigmonsays 2y ago

tools like this are pretty sweet but I would rather just run it myself. docker-compose with a load balancer (traefik) is fairly straightforward and awesome. the TLS setup is nice but I wildcard that and just run certgen myself. The main thing I think that's missing is some sort of authentication or zero trust system, maybe vpn tunnel provisioner. Most services I self host I do not want to be made public due to security concerns.

funshed 2y ago

Nice, you should probably explain what traefik, sops and age will do. First time I've heard of sops, very handy!

tegiddrone 2y ago

Looks nice! Something I'd want in front is some sort of basic app firewall like fail2ban or CrowdSec to ban vuln scanners and other intrusion attempts. It is a nice thing about Cloudflare since they provide some of this protection.

turtlebits 2y ago

What about this is highly available? On a single VPS? Does this only support a single app? Nice project but the claims (production ready? Load balance on a single server?) are a bit ridiculous.

dewey 2y ago
Highly available is overrated for most use cases, especially for any side projects.
closewith 2y ago
In my experience, single apps on VPSes have far higher availability in practice than the majority of convoluted deployments.

dewey 2y ago

I'd also suggest people to take a look at Dokku, it's a very mature project with a similar scope and was discussed here a few weeks ago: https://news.ycombinator.com/item?id=41358020 https://news.ycombinator.com/item?id=41358020 I wrote up my own experiences too (https://blog.notmyhostna.me/posts/selfhosting-with-dokku-and-docker-compose https://blog.notmyhostna.me/posts/selfhosting-with-dokku-and...) and I can only recommend it. It is ~3 commands to set up an app, and one push to deploy after that.

rgrieselhuber 2y ago
I've kept a list of these tools that I've been meaning to check out. In scope, do they cover securing the instance? Is there any automation for creating networks of instances?
- dewey 2y ago
  > In scope, do they cover securing the instance? Most of these I checked don't, but a recent Ubuntu version is perfectly fine to use as-is. > Is there any automation for creating networks of instances? Not that I'm aware, it would also defeat the purpose of these tools a bit that are supposed to be simple. (Dokku is "just" a shell script).
mimischi 2y ago
Been using dokku for probably 8 years now? (or something close to that; it used to be written entirely in bash!) Hosting private stuff on it, and an application at $oldplace probably also still runs on this solid setup. Highly recommended, and the devs are a great sport!
FloatArtifact 2y ago
Part of me dies every time I see projects not integrating robust restoring and backup systems.
- Aeolun 2y ago
  None of my hobby projects across 15 years or so have ever needed backups or restoring. I can agree it would be nice to have, but it’s a far cry from necessary.

nhatcher 2y ago

This looks fantastic TBH! Can't wait to give it a go. Congratulations. I've long thought something like this should be possible. The only thing I've done is document carefully my own steps: https://www.nhatcher.com/post/a-cto-on-a-shoestring/ https://www.nhatcher.com/post/a-cto-on-a-shoestring/

gf297 2y ago

What's the purpose of encrypting the env file with sops, when the age secret key is stored on the VPS? If someone has access to the encrypted env file, they will also have access to the secret key, and can decrypt it.

mightymoud 2y ago
[dead]
nsluss 2y ago
only if they can login as the owner of the key. Most of my team can get on our vps, only a couple could get our age key.

trey-jones 2y ago

"Wow, this really looks significantly better than my own CLI tools" I'm going to have to look into this pterm thing.

hkon 2y ago

Have used caprover. Good that more tools enter this space.

singhrac 2y ago

Any possibility you’d add support for a Mac Mini deployment? I think the extra complexity would be from changing the Docker images, but of course the devil is in the details. I just have a Mac Mini and it would be great to self-host some stuff.

brirec 2y ago
As someone who used to love hosting things on a Mac mini, have you tried installing Linux on it to use as a dedicated server? If you do, it should handle this just like any other platform you could install it on.

Humphrey 2y ago

Love this! That said, I achieve the same thing manually using Docker Compose & some shell scripts. It takes a bike longer, but it has forced me to learn the lower level tools that helpers like Sidekick use. Also, all of these tools have great documentation on getting up and running, but SIGNIFICANTLY LESS INFO ON HOW TO MAINTAIN OVER THE LONG TERM. If I was going to start using a tool like Sidekick, Kamal, or Dokku I would want clear answers to the following: - How do I keep my VPS host up and running with latest security updates? - How do I update to more recent versions of Docker? - How do I update services that maintain state (eg. update to a new Postgres version) - How do I seamlessly migrate to a new host (perhaps as a way to solve the above)? - How should I manage and serve static resources & user media? (store on host or use cloud storage?) - How do I manage database migrations during an update, and how do I control that process to avoid downtime during an update? I just spent an entire evening transferring a side project to a new VPS because I needed to update Postgres. The ideal self-hosting solution would make that a 20 min task.

jfdi 2y ago

These are great. Having tooling to get stuff out fast and as safely as possible to get to iterating openly. Here’s a bash script I posted a while back on a different thread that does similar thing if of interest for anyone. It’s probably less nice than op’s for ex it only works with digitalocean (which is great!) - but it’s simple small and mostly readable. also assumes docker - but all via compose, with some samples like nginx w auto-ssl via le. https://github.com/thomaswilley/tide.sh https://github.com/thomaswilley/tide.sh

mattfrommars 2y ago

I consider myself bit techsavy knowing Linux and basic scripting. But does anyone have a resource or link that explains the idea to make a service which OP shared here? Because frankly, I'd feel lost reading the code from one file at a time without knowing where to start. Plus it's written in Go which I have I am not familiar with.

mightymoud 2y ago
I wrote a intro guide to pterm and cobra-cli here https://mightytuts.com/blog/golang-cli-introduction-with-cobra-and-pterm/ https://mightytuts.com/blog/golang-cli-introduction-with-cob... I also have a video on YT if that is your thing

interstice 2y ago

Really like this! Funnily enough I was just rabbit holing into terraform + ansible in an effort to do essentially this but with an anycast network. The thinking was to mirror apps across locations with a single deploy. I don't suppose you're plannning something similar with this one?

apitman 2y ago
Anycast is a really underrated feature and I'm surprised no major VPS providers offer it. It's one of my favorite things about fly.io

vander_elst 2y ago

How does it compare to ansible? I didn't immediately find a link

vickodin 2y ago

Very interesting. The similarity to the sidekiq is a bit confusing, but it doesn't really matter.

strzibny 2y ago

Given the choice of Docker and Traefik I would love to know what's the exact difference to Kamal? And btw Kamal will soon have a new improved version with a custom proxy.

udev4096 2y ago

I made something similar sometime ago (without using docker): http://github.com/thebigbone/deploypot http://github.com/thebigbone/deploypot

pmarreck 2y ago

I would love this, but with Nix.

worik 2y ago

Nice work. I wonder, though. Why Ubuntu? Why not Debian? With all due respect to Cannonical, Ubuntu is not really suitable. It is not aimed at developers Unless it has changed since I left it in a fury, it takes too much control away from you with the Snap system.

mutant 2y ago

I just tried this out on DO and Linode. Worked great! Very cool stack.

lenkite 2y ago

Gorgeous tool! Thanks for making this.