5 ms·

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ https://m

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ https://cilium.io/ (Networking) https://rook.io/ https://rook.io/ (Persistent Storage)

salamander014 7y ago
I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router (https://kube-router.io https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (currently running Canal). Not sure what we will decide on when we go to production.
- fyfy18 7y ago
  I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device. Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It hasn't lost data though, even though one of the SSDs died and was unrecoverable, so props for that. To be fair I haven't tried upgrading since I installed it so maybe my issues would be resolved by that. Overall Rook feels somewhat overkill for a homelab user, but I haven't seen any recommended alternatives.
  - Nullabillity 7y ago
    You need to set a memory target for your OSDs, or they will gobble up as much as possible for their cache. In Rook you do that by specifying the resource limits on your CephCluster[0]. Sadly, Rook enforces some pretty high minimum values (2GiB per OSD, 1 GiB per mon), which is fine for production but can make homelab experimentation annoying. [0]: https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster-wide-resources-configuration-settings https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster...
bogomipz 7y ago
Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is that correct?
- moondev 7y ago
  Not directly, rook provides a storage backend for in-cluster workload persistent volumes. In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.
  - ownagefool 7y ago
    I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume. You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.
    - moondev 7y ago
      The parent comment was asking about root volumes for kubernetes nodes > Does Rook give you the equivalent of EBS root volumes for your nodes then? I think we are saying the same thing if i'm not mistaken?
      - ownagefool 7y ago
        I doubt the parent really cares about whether or not it's a root volume, but for the record you can mount it to root. It'll just essentially be empty, which means you have to add a step of populating root with something useful. You'd have to either try and use an initContainer to copy in a filesystem at runtime, have ceph give you a pre-populated directory, probably via thin provisioning, or do something out of band (I've did this in k8s). This is usually more effort than it's worth though, as container runtimes populate the root for you with whatever you want anyway. Plus, if you start treating containers as blobs like VMs, you'll end up in the situation where you don't know where your important variable data is, which leads to situations where people forget to back it up and test it.

salamander014 7y ago

I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router (https://kube-router.io https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (currently running Canal). Not sure what we will decide on when we go to production.

fyfy18 7y ago
I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device. Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It hasn't lost data though, even though one of the SSDs died and was unrecoverable, so props for that. To be fair I haven't tried upgrading since I installed it so maybe my issues would be resolved by that. Overall Rook feels somewhat overkill for a homelab user, but I haven't seen any recommended alternatives.
- Nullabillity 7y ago
  You need to set a memory target for your OSDs, or they will gobble up as much as possible for their cache. In Rook you do that by specifying the resource limits on your CephCluster[0]. Sadly, Rook enforces some pretty high minimum values (2GiB per OSD, 1 GiB per mon), which is fine for production but can make homelab experimentation annoying. [0]: https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster-wide-resources-configuration-settings https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster...

bogomipz 7y ago

Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is that correct?

moondev 7y ago
Not directly, rook provides a storage backend for in-cluster workload persistent volumes. In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.
- ownagefool 7y ago
  I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume. You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.
  - moondev 7y ago
    The parent comment was asking about root volumes for kubernetes nodes > Does Rook give you the equivalent of EBS root volumes for your nodes then? I think we are saying the same thing if i'm not mistaken?
    - ownagefool 7y ago
      I doubt the parent really cares about whether or not it's a root volume, but for the record you can mount it to root. It'll just essentially be empty, which means you have to add a step of populating root with something useful. You'd have to either try and use an initContainer to copy in a filesystem at runtime, have ceph give you a pre-populated directory, probably via thin provisioning, or do something out of band (I've did this in k8s). This is usually more effort than it's worth though, as container runtimes populate the root for you with whatever you want anyway. Plus, if you start treating containers as blobs like VMs, you'll end up in the situation where you don't know where your important variable data is, which leads to situations where people forget to back it up and test it.