7 ms·

Building Burstables: CPU slicing with cgroups

130 points by msarnowicz 1y ago

msarnowicz 1y ago
Hey, author here. Please AMA. I came into the Linux world via Postgres, and this was an interesting project for me learning more about Linux internals. While cgroups v2 do offer basic support for CPU bursting, the bursts are short-lived, and credits don’t persist beyond sub-second intervals. If you’ve run into scenarios where more adaptive or sustained bursting would help, we’d love to hear about them. Knowing your use cases will help shape what we build next.
- parrit 1y ago
  Thanks! That was a pleasant read. I have been wanting to mess with cgroups for a while, in order to hack together a "docker" like many have done before to understand it better. This will help! Are there typical use cases where you reach for cgroups directly instead of using the container abstraction?
  - msarnowicz 1y ago
    Thanks for the kind words. Even if you are not building a cloud service, I think it is good to understand how the underlying layer works and what are the knobs and the limits of the platform. I could see a use case where two or more processes need to run on one VM or a container, maybe for cost-saving reasons or specific architecture/security reasons, but need to be guaranteed a certain amount of resources and a certain isolation from each other.
- motrm 1y ago
  Echoing parrit's comment, this was indeed a very nice read and very well written. I particularly enjoyed the gentle exposition into the world of cgroups and how they work, the levers available, and finally how Ubicloud uses them. Looking forward to reading how you handle burst credits over longer periods, once you implement that :) Lovely work, Maciek!
  - msarnowicz 1y ago
    Thank you very much, I appreciate your comment.
- nighthawk454 1y ago
  Great article, thanks! I’ve been curious if there’s any scheduling optimizations for workloads that are extremely burst-y. Such as super low traffic websites or cron job type work - where you may want your database ‘provisioned’ all the time, but realistically it won’t get anywhere near even the 50% cpu minimum at any kind of sustained rate. Presumably those could be hosted at even a fraction of the burst cost. Is that a use case Ubicloud has considered?
  - msarnowicz 1y ago
    This is a very valid scenario, however, one that is not yet fully baked into this implementation. But, as mentioned, this is a starting point. We want to hear feedback and see customers' workloads on Burstables first. The main challenge here is that cpu.max.burst can be set no higher than the limit set in cpu.max. This limits our options to some extent. But we can still look at some possible implementation choices here: - Pack more VMs into the same slice/group, and with that lower the minimum CPU guaranteed, and at the same time lower the price point. This would increase the chance of running into a "noisy neighbor", but we expect it would not be used for any critical workload. - Implement calculation of CPU credits outside of the kernel and change the CPU max and burst limits dynamically over an extended period of time (hours and days, instead of sub-second).
    - nighthawk454 1y ago
      Gotcha, thanks for the reply. Makes sense to target burstables first - that seems to be the most common feature set. That’s interesting that it’s not readily available in the kernel. I once spoke to some AWS folks dealing with Batch/ECS scheduling of docker container tasks and they hit similar limitations. As a result their CPU max/burst settings work like the underlying cgroups too. I imagine writing a custom scheduler would be quite an undertaking!
      - msarnowicz 1y ago
        I think so, too!
- phrotoma 1y ago
  I don't have a question but I really wanted to say thanks for the blog post. Extremely clear and cogent writing on a tricky topic. Well done!
jauntywundrkind 1y ago
I'd also strongly recommend this view of how Kubernetes uses cgroups, showing similar drill downs for how everything gets managed. Lovely view of what's really happening! https://martinheinz.dev/blog/91 https://martinheinz.dev/blog/91 I've been a bit apoplectic in the past that cgroups seemed not super helpful in Kubernetes, but this really showed me how the different Kubernetes QoS levels are driven by similar juggling of different cgroups. I'm not sure if this makes use of cpu.max.burst or not. There's a fun article that monkeys with these cgroups directly, which is neat to see. It also links to an ask that Kubernetes get support for the new (5.14) CFS Burst system. Which is a whole nother fun rabbit hole of fair share bursting to go down! https://medium.com/@christian.cadieux/kubernetes-throttling-and-bursting-with-cgroup-v2-57d98cf49d72 https://medium.com/@christian.cadieux/kubernetes-throttling-... https://github.com/kubernetes/kubernetes/issues/104516 https://github.com/kubernetes/kubernetes/issues/104516
- msarnowicz 1y ago
  Thank you, that is a good perspective, too!
- __turbobrew__ 1y ago
  cpu.max.burst increases the chances of noisy neighbours stealing CPU from other tenants. I run multi-tenant k8s clusters with hundreds of tenants and it fundamentally is a hard problem to balance workload performance with efficiency. Sharing resources increases efficiency but in most cases increases tail latencies.
  - jeffbee 1y ago
    If you use k8s qos levels "guaranteed" cpu resources will be distinct — via cpu sets — from the ones used by the riff-raff. This is a good way to segregate latency-sensitive apps where you care about latency from throughtput-oriented stuff where you don't.
    - __turbobrew__ 1y ago
      Guaranteed QoS isn’t perfect: 1. Neighbours can be noisy to the other hyperthread on the same CPU. For example, heavy usage of avx-512 and other vectorized instructions can affect a tenant running on the same core but different hyperthread. You can disable hyperthreading, but now you are making the same tradeoff where you are sacrificing efficiency for low tail latencies. 2. There are certain locks in the kernel which can be exhausted by certain behaviour of a single tenant. For example, on kernel 5.15 there was one global kernel lock for cgroup resource accounting. If you have a tenant which is constantly hitting cgroup limits it increases lock contention in the kernel which slows down other tenants on the system which also use the same locks. This particular issue with cgroups accounting has been improved in later kernels. 3. If your latency sensitive service runs on the same cores which service IRQs, the tail latency can greatly increase when there are heavy IRQ load, for example high speed NIC IRQs. You can isolate those CPUs from the pool of CPUs offered to pods, but now you are dedicating 4-8 CPUs to just process interrupts. Ideally you could run the non-guaranteed pods on the CPUs which service IRQs, but that is not supported by kubernetes. 4. During full node memory pressure, the kernel does not respect memory.min and will reclaim pages of guaranteed QoS workloads. 5. The current implementation of memory QoS does not adjust memory.max of the burstable pod slice, so bursable pods can take up the entire free memory of the kubepods slice which starves new memory allocations from guaranteed pods. Dont even get me started on NUMA issues.
iluvcommunism 1y ago
[dead]
solarkraft 1y ago
My main takeaway from this is that you can control KVM VMs with cgroups just like normal processes. I didn’t expect that.
- msarnowicz 1y ago
  I am glad you found this useful!

130 points by msarnowicz 1y ago

Hey, author here. Please AMA. I came into the Linux world via Postgres, and this was an interesting project for me learning more about Linux internals. While cgroups v2 do offer basic support for CPU bursting, the bursts are short-lived, and credits don’t persist beyond sub-second intervals. If you’ve run into scenarios where more adaptive or sustained bursting would help, we’d love to hear about them. Knowing your use cases will help shape what we build next.

parrit 1y ago
Thanks! That was a pleasant read. I have been wanting to mess with cgroups for a while, in order to hack together a "docker" like many have done before to understand it better. This will help! Are there typical use cases where you reach for cgroups directly instead of using the container abstraction?
- msarnowicz 1y ago
  Thanks for the kind words. Even if you are not building a cloud service, I think it is good to understand how the underlying layer works and what are the knobs and the limits of the platform. I could see a use case where two or more processes need to run on one VM or a container, maybe for cost-saving reasons or specific architecture/security reasons, but need to be guaranteed a certain amount of resources and a certain isolation from each other.
motrm 1y ago
Echoing parrit's comment, this was indeed a very nice read and very well written. I particularly enjoyed the gentle exposition into the world of cgroups and how they work, the levers available, and finally how Ubicloud uses them. Looking forward to reading how you handle burst credits over longer periods, once you implement that :) Lovely work, Maciek!
- msarnowicz 1y ago
  Thank you very much, I appreciate your comment.
nighthawk454 1y ago
Great article, thanks! I’ve been curious if there’s any scheduling optimizations for workloads that are extremely burst-y. Such as super low traffic websites or cron job type work - where you may want your database ‘provisioned’ all the time, but realistically it won’t get anywhere near even the 50% cpu minimum at any kind of sustained rate. Presumably those could be hosted at even a fraction of the burst cost. Is that a use case Ubicloud has considered?
- msarnowicz 1y ago
  This is a very valid scenario, however, one that is not yet fully baked into this implementation. But, as mentioned, this is a starting point. We want to hear feedback and see customers' workloads on Burstables first. The main challenge here is that cpu.max.burst can be set no higher than the limit set in cpu.max. This limits our options to some extent. But we can still look at some possible implementation choices here: - Pack more VMs into the same slice/group, and with that lower the minimum CPU guaranteed, and at the same time lower the price point. This would increase the chance of running into a "noisy neighbor", but we expect it would not be used for any critical workload. - Implement calculation of CPU credits outside of the kernel and change the CPU max and burst limits dynamically over an extended period of time (hours and days, instead of sub-second).
  - nighthawk454 1y ago
    Gotcha, thanks for the reply. Makes sense to target burstables first - that seems to be the most common feature set. That’s interesting that it’s not readily available in the kernel. I once spoke to some AWS folks dealing with Batch/ECS scheduling of docker container tasks and they hit similar limitations. As a result their CPU max/burst settings work like the underlying cgroups too. I imagine writing a custom scheduler would be quite an undertaking!
    - msarnowicz 1y ago
      I think so, too!
phrotoma 1y ago
I don't have a question but I really wanted to say thanks for the blog post. Extremely clear and cogent writing on a tricky topic. Well done!

jauntywundrkind 1y ago

I'd also strongly recommend this view of how Kubernetes uses cgroups, showing similar drill downs for how everything gets managed. Lovely view of what's really happening! https://martinheinz.dev/blog/91 https://martinheinz.dev/blog/91 I've been a bit apoplectic in the past that cgroups seemed not super helpful in Kubernetes, but this really showed me how the different Kubernetes QoS levels are driven by similar juggling of different cgroups. I'm not sure if this makes use of cpu.max.burst or not. There's a fun article that monkeys with these cgroups directly, which is neat to see. It also links to an ask that Kubernetes get support for the new (5.14) CFS Burst system. Which is a whole nother fun rabbit hole of fair share bursting to go down! https://medium.com/@christian.cadieux/kubernetes-throttling-and-bursting-with-cgroup-v2-57d98cf49d72 https://medium.com/@christian.cadieux/kubernetes-throttling-... https://github.com/kubernetes/kubernetes/issues/104516 https://github.com/kubernetes/kubernetes/issues/104516

msarnowicz 1y ago
Thank you, that is a good perspective, too!
__turbobrew__ 1y ago
cpu.max.burst increases the chances of noisy neighbours stealing CPU from other tenants. I run multi-tenant k8s clusters with hundreds of tenants and it fundamentally is a hard problem to balance workload performance with efficiency. Sharing resources increases efficiency but in most cases increases tail latencies.
- jeffbee 1y ago
  If you use k8s qos levels "guaranteed" cpu resources will be distinct — via cpu sets — from the ones used by the riff-raff. This is a good way to segregate latency-sensitive apps where you care about latency from throughtput-oriented stuff where you don't.
  - __turbobrew__ 1y ago
    Guaranteed QoS isn’t perfect: 1. Neighbours can be noisy to the other hyperthread on the same CPU. For example, heavy usage of avx-512 and other vectorized instructions can affect a tenant running on the same core but different hyperthread. You can disable hyperthreading, but now you are making the same tradeoff where you are sacrificing efficiency for low tail latencies. 2. There are certain locks in the kernel which can be exhausted by certain behaviour of a single tenant. For example, on kernel 5.15 there was one global kernel lock for cgroup resource accounting. If you have a tenant which is constantly hitting cgroup limits it increases lock contention in the kernel which slows down other tenants on the system which also use the same locks. This particular issue with cgroups accounting has been improved in later kernels. 3. If your latency sensitive service runs on the same cores which service IRQs, the tail latency can greatly increase when there are heavy IRQ load, for example high speed NIC IRQs. You can isolate those CPUs from the pool of CPUs offered to pods, but now you are dedicating 4-8 CPUs to just process interrupts. Ideally you could run the non-guaranteed pods on the CPUs which service IRQs, but that is not supported by kubernetes. 4. During full node memory pressure, the kernel does not respect memory.min and will reclaim pages of guaranteed QoS workloads. 5. The current implementation of memory QoS does not adjust memory.max of the burstable pod slice, so bursable pods can take up the entire free memory of the kubepods slice which starves new memory allocations from guaranteed pods. Dont even get me started on NUMA issues.

iluvcommunism 1y ago

[dead]

solarkraft 1y ago

My main takeaway from this is that you can control KVM VMs with cgroups just like normal processes. I didn’t expect that.

msarnowicz 1y ago
I am glad you found this useful!