6 ms·

I have a cluster using terraform-hcloud-kube-hetzner [0] and I'm quite happy with it. It uses MicroOS for rolling releases and auto-upgrades for less maintenan

I have a cluster using terraform-hcloud-kube-hetzner [0] and I'm quite happy with it. It uses MicroOS for rolling releases and auto-upgrades for less maintenance. I'm quite happy with it so far, and I like that it integrates with terraform. [0] https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...

SkyLinx 4y ago
I had never heard of MicroOS before, sounds interesting. Before building this tool I actually used both Terraform and Pulumi. My tool is a ton faster setting things up and only requires a small YAML config file. Would be able to give it a try? I am looking for feedback :)
- stephankoelle 4y ago
  You might then be interested in Fedora CoreOS, we use it with Hetzner cloud and real Hetzner root server with good success. (Also with ovh)
  - SkyLinx 4y ago
    I can test it. What would be the advantage if I use it in my tool instead of Ubuntu? At the end of the day with Kubernetes I don't think it makes much difference which OS you use. I picked Ubuntu because it's standard and servers with this image build very quickly with Hetzner Cloud.
    - mdaniel 4y ago
      CoreOS (and its spiritual successor Flatcar) are immutable, which gravely limits the silliness that can be done to them post-launch. We use now use Bottlerocket since we're on EKS, but it's even more locked down since to even get an interactive shell is some major hoopjumpery. I believe Talos goes even further and is completely devoid of a shell So, yes, it absolutely matters which OS you use, of course depending on your threat model and tolerance for "no, you don't get to ssh onto a Node and do whateverthehell you want"
- evrflx 4y ago
  How did you like pulumi? Why did you use both terraform and pulumi at the same time, my understanding was that they serve the same purpose.
  - wise0wl 4y ago
    We use Pulumi at my current workplace and love it. Being able to use a real language (Typescript, in our case) allows us to create data structure abstractions to feed our Infrastructure-as-Code that make sense from a developer interaction point-of-view. Our abstractions are less leaky and more in-line with a well designed interface, ready for use by a team that doesn't need to know all the details. We can have developers write feature code and design services and just plug in simple details for new services in our stacks Yaml. However there is a gotcha. There are nearly infinite ways to setup your Pulumi codebase and there is no real prescribed structure outside of "create stacks". This can lead to a lot of time spent in architecting / implementing / testing different structures and abstractions which are not really a thing with Terraform. This can take time and can lead to refactors if you are not extremely intentional with your design.
  - SkyLinx 4y ago
    I used Terraform first and then switched to Pulumi. What I like of Pulumi is that you get to use a proper programming language rather than a DSL, so you have more control/power in a way.
incompleteCode 4y ago
How much does this cost you on a monthly basis?
- SkyLinx 4y ago
  I can give you an example of one cluster than I use for personal stuff: - 3 masters with 3 cores and 4 GB of RAM each - load balancer for the masters (for the Kubernetes API) - 3 workers with faster AMD CPUs and 4 cores/8GB of RAM - around 200GB of storage in persistent volumes I think it costs around 60-70 euros per month total! I can't remember the exact amount because I have many things with Hetzner and do tests creating and deleting stuff all the time so my bill is rather long lol. Hetzner is very cheap, but I can voice for the performance and reliability as well. It's a very good provider and I highly recommend them. Now they even have 2 locations in the USA!
  - dakiol 4y ago
    Nice price. Are those dedicated servers? Or just "cloud" (standard or dedicated vcpu?)?
    - SkyLinx 4y ago
      Cloud servers with standard shared cores. They perform really well
  - incompleteCode 4y ago
    That’s cost-effective! I’ve been toying with the idea of running a personal distributed system that’s not serverless to just hack around and Hetzner seems approachable going by your experience.
    - SkyLinx 4y ago
      Definitely. I have been using them for years and I still can't understand how they can offer such a good service at their prices, considering how much the competition charges.
unity1001 4y ago
Is your control plane high available? How hands off is it?
- SkyLinx 4y ago
  It's up to you. You can configure it with one master or HA multiple masters (e.g.). If you set the instance count for the masters pool >1 then my tool will create a load balancer in front of the API server. The tool simply requires a small YAML config file and creates and configures everything that you need to get a fully functional cluster in a few minutes. It creates all the infrastructure resources (servers etc), it deploys kubernetes on all the nodes, and then installs some software that allows you to provision both load balancers and persistent volumes. It also installs a controller that makes upgrading to a new version easier. What the tool does not install is other apps, because this would be too opinionated and depends from cluster to cluster.

SkyLinx 4y ago

I had never heard of MicroOS before, sounds interesting. Before building this tool I actually used both Terraform and Pulumi. My tool is a ton faster setting things up and only requires a small YAML config file. Would be able to give it a try? I am looking for feedback :)

stephankoelle 4y ago
You might then be interested in Fedora CoreOS, we use it with Hetzner cloud and real Hetzner root server with good success. (Also with ovh)
- SkyLinx 4y ago
  I can test it. What would be the advantage if I use it in my tool instead of Ubuntu? At the end of the day with Kubernetes I don't think it makes much difference which OS you use. I picked Ubuntu because it's standard and servers with this image build very quickly with Hetzner Cloud.
  - mdaniel 4y ago
    CoreOS (and its spiritual successor Flatcar) are immutable, which gravely limits the silliness that can be done to them post-launch. We use now use Bottlerocket since we're on EKS, but it's even more locked down since to even get an interactive shell is some major hoopjumpery. I believe Talos goes even further and is completely devoid of a shell So, yes, it absolutely matters which OS you use, of course depending on your threat model and tolerance for "no, you don't get to ssh onto a Node and do whateverthehell you want"
evrflx 4y ago
How did you like pulumi? Why did you use both terraform and pulumi at the same time, my understanding was that they serve the same purpose.
- wise0wl 4y ago
  We use Pulumi at my current workplace and love it. Being able to use a real language (Typescript, in our case) allows us to create data structure abstractions to feed our Infrastructure-as-Code that make sense from a developer interaction point-of-view. Our abstractions are less leaky and more in-line with a well designed interface, ready for use by a team that doesn't need to know all the details. We can have developers write feature code and design services and just plug in simple details for new services in our stacks Yaml. However there is a gotcha. There are nearly infinite ways to setup your Pulumi codebase and there is no real prescribed structure outside of "create stacks". This can lead to a lot of time spent in architecting / implementing / testing different structures and abstractions which are not really a thing with Terraform. This can take time and can lead to refactors if you are not extremely intentional with your design.
- SkyLinx 4y ago
  I used Terraform first and then switched to Pulumi. What I like of Pulumi is that you get to use a proper programming language rather than a DSL, so you have more control/power in a way.

incompleteCode 4y ago

How much does this cost you on a monthly basis?

SkyLinx 4y ago
I can give you an example of one cluster than I use for personal stuff: - 3 masters with 3 cores and 4 GB of RAM each - load balancer for the masters (for the Kubernetes API) - 3 workers with faster AMD CPUs and 4 cores/8GB of RAM - around 200GB of storage in persistent volumes I think it costs around 60-70 euros per month total! I can't remember the exact amount because I have many things with Hetzner and do tests creating and deleting stuff all the time so my bill is rather long lol. Hetzner is very cheap, but I can voice for the performance and reliability as well. It's a very good provider and I highly recommend them. Now they even have 2 locations in the USA!
- dakiol 4y ago
  Nice price. Are those dedicated servers? Or just "cloud" (standard or dedicated vcpu?)?
  - SkyLinx 4y ago
    Cloud servers with standard shared cores. They perform really well
- incompleteCode 4y ago
  That’s cost-effective! I’ve been toying with the idea of running a personal distributed system that’s not serverless to just hack around and Hetzner seems approachable going by your experience.
  - SkyLinx 4y ago
    Definitely. I have been using them for years and I still can't understand how they can offer such a good service at their prices, considering how much the competition charges.

unity1001 4y ago

Is your control plane high available? How hands off is it?

SkyLinx 4y ago
It's up to you. You can configure it with one master or HA multiple masters (e.g.). If you set the instance count for the masters pool >1 then my tool will create a load balancer in front of the API server. The tool simply requires a small YAML config file and creates and configures everything that you need to get a fully functional cluster in a few minutes. It creates all the infrastructure resources (servers etc), it deploys kubernetes on all the nodes, and then installs some software that allows you to provision both load balancers and persistent volumes. It also installs a controller that makes upgrading to a new version easier. What the tool does not install is other apps, because this would be too opinionated and depends from cluster to cluster.