5 ms·

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ https://microos.opensuse.org/ for my homelab for a while, and

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups. I even use it with Longhorn https://longhorn.io/ https://longhorn.io/ for shared block storage on the mini cluster. If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ https://kured.dev/ for the transactional-updates reboot method. I'd love to compare it against Talos https://www.talos.dev/ https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/issues/4041 https://github.com/siderolabs/talos/issues/4041 ) really hurts most small home / office usage I'd want to try.

organsnyder 3y ago
Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.
- davkan 3y ago
  Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?
  - organsnyder 3y ago
    I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.
    - davkan 3y ago
      Ah gotcha. Haven’t had to deal with that yet. Maybe possible to add a static pod via the machine config? But yeah it’s basically throwing out a bunch of linux admin muscle memory.
osigurdson 3y ago
Kudos to you. I feel like setting things up on real hardware is somehow needed in order to make things concrete enough to full understand. At least for me (I fully admit this may be a personal flaw) working with a vm on in the cloud is a little too abstract - even though eventually this is where things will land.
imiric 3y ago
Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nightmare I'd like to avoid. :)
- intangible 3y ago
  I wouldn't really treat it as a replacement for a NAS, mostly only for the container workloads running on kubernetes itself... Ideally, any apps you develop should use something more sane like object storage (Minio etc) for their data. I push it pretty minimally right now, so no great performance testing myself, and I do run it in synchronous mode, so that means its write performance is likely going to be limited to the 1gbps network it syncs over.
- iamdbtoo 3y ago
  I can't speak to performance because the workloads aren't really intense, but I run a small 3 node cluster using k3s and Longhorn and Longhorn has been really great. It was easy to setup and has been reliably running with very minimal maintanence since.
  - arcastroe 3y ago
    If you use longhorn, make sure to enable the network policies when installing the helm chart. For some odd reason, these are disabled by default, which means ANY pod running on your cluster has full access to the longhorn manager, API, and all your volumes https://github.com/longhorn/charts/blob/v1.5.x/charts/longhorn/values.yaml#L24 https://github.com/longhorn/charts/blob/v1.5.x/charts/longho...
- MPSimmons 3y ago
  I've run Rook/Ceph, and I run Longhorn right now. I wish I didn't, and I'm actively migrating to provider-managed PVs. My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money). If you must use some sort of in-cluster distributed storage solution, I would advise you to exclude members of your control plane from taking part, and I would also dedicate entirely separate drives and volumes for the storage distribution so that normal host workload doesn't impact latency and contention for the distributed storage.
c0wb0yc0d3r 3y ago
How did you go about deploying k3s to MicroOS? Did you go the route of installing it via combustion and a systemd unit?[0] To me it seems strange that a systemd unit is used, but I didn't know if I was missing something about the way MicroOS worked. [0]: https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroOS https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroO...
sgarland 3y ago
Re: Talos persistent storage, why not run it as a VM and pass in block devices from the hypervisor? You also then gain the benefit of templated VMs that you can easily recreate or scale as needed.

organsnyder 3y ago

Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.

davkan 3y ago
Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?
- organsnyder 3y ago
  I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.
  - davkan 3y ago
    Ah gotcha. Haven’t had to deal with that yet. Maybe possible to add a static pod via the machine config? But yeah it’s basically throwing out a bunch of linux admin muscle memory.

osigurdson 3y ago

Kudos to you. I feel like setting things up on real hardware is somehow needed in order to make things concrete enough to full understand. At least for me (I fully admit this may be a personal flaw) working with a vm on in the cloud is a little too abstract - even though eventually this is where things will land.

imiric 3y ago

Thanks for your perspective. How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nightmare I'd like to avoid. :)

intangible 3y ago
I wouldn't really treat it as a replacement for a NAS, mostly only for the container workloads running on kubernetes itself... Ideally, any apps you develop should use something more sane like object storage (Minio etc) for their data. I push it pretty minimally right now, so no great performance testing myself, and I do run it in synchronous mode, so that means its write performance is likely going to be limited to the 1gbps network it syncs over.
iamdbtoo 3y ago
I can't speak to performance because the workloads aren't really intense, but I run a small 3 node cluster using k3s and Longhorn and Longhorn has been really great. It was easy to setup and has been reliably running with very minimal maintanence since.
- arcastroe 3y ago
  If you use longhorn, make sure to enable the network policies when installing the helm chart. For some odd reason, these are disabled by default, which means ANY pod running on your cluster has full access to the longhorn manager, API, and all your volumes https://github.com/longhorn/charts/blob/v1.5.x/charts/longhorn/values.yaml#L24 https://github.com/longhorn/charts/blob/v1.5.x/charts/longho...
MPSimmons 3y ago
I've run Rook/Ceph, and I run Longhorn right now. I wish I didn't, and I'm actively migrating to provider-managed PVs. My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money). If you must use some sort of in-cluster distributed storage solution, I would advise you to exclude members of your control plane from taking part, and I would also dedicate entirely separate drives and volumes for the storage distribution so that normal host workload doesn't impact latency and contention for the distributed storage.

c0wb0yc0d3r 3y ago

How did you go about deploying k3s to MicroOS? Did you go the route of installing it via combustion and a systemd unit?[0] To me it seems strange that a systemd unit is used, but I didn't know if I was missing something about the way MicroOS worked. [0]: https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroOS https://en.opensuse.org/SDB:K3s_cluster_deployment_on_MicroO...

sgarland 3y ago

Re: Talos persistent storage, why not run it as a VM and pass in block devices from the hypervisor? You also then gain the benefit of templated VMs that you can easily recreate or scale as needed.