7 ms·

Yea. I realize I ought to dig into things more to understand how to push past into 90%-95% utilization territory. Thanks for the resource to read through.

colechristensen 9mo ago
One way to think about it is 80% IS full utilization. The engineering time, the risks of decreased performance, and the fragility of pushing the limit at some point become not worth the benefits of reaching some higher metric of utilization. If it's not where you are, that optimum trade off point is somewhere.
mpyne 9mo ago
You absolutely do not want 90-95% utilization. At that level of utilitization random variability alone is enough to cause massive whiplash in average queue lengths. The cycle time impact of variability of a single-server/single-queue system at 95% load is nearly 25x the impact on the same system at 75% load, and there are similar measures for other process queues. As the other comment notes, you should really work from an assumption that 80% is max loading, just as you'd never aim to have a swap file or swap partition of exactly the amount of memory overcommit you expect.
- rcxdude 9mo ago
  Man, if there's one idea I wish I could jam into the head of anyone running an organization, it would be queuing theory. So many people can't understand that slack is necessary to have quick turnaround.
  - sovietmudkipz 9mo ago
    Mmmm, I remember reading this in Systems Performance Brendan Gregg. I should revisit what was written…
- sovietmudkipz 9mo ago
  I target 80% utilization because I’ve seen that figure multiple times. I suppose I should rephrase: I’d like to understand the constraints and systems involved that make 80% considered full utilization. There’s obviously something that limits a OS; is it tunable? Questions I imagine a thorough multiplayer solutions engineer would be curious of, the kind of person whose trying to squeeze as much juice out of the hardware specs as possible.
  - btschaegg 9mo ago
    It might not be the OS, but just statistical inevitability. If you're talking about CPU utilization on Linux, for example, it's not all that unlikely that the number you're staring at isn't "time spent by CPU doing things" but "average CPU run queue length". "100%" then doesn't only mean the CPU gets no rest, but "there's always someone waiting for a CPU to become free". It likely pays off to understand where the load numbers in your tooling actually come from. Even if that weren't the case, lead times for tasks will always increase with more utilization; see e.g. [1]: If you push a system from 80% to 95% utilization, you have to expect a ~4.75x increase in lead time for each task _on average_: (0.95/0.05) / (0.8/0.2) Note that all except the term containing ρ in the formula are defined by your system/software/clientele, so you can drop them for a purely relative comparison. [1]: https://en.wikipedia.org/wiki/Kingman%27s_formula https://en.wikipedia.org/wiki/Kingman%27s_formula Edit: Or, to try to picture the issue more intuitively: If you're on a highway nearing 100% utilization, you're likely standing in a traffic jam. And if that's not (yet) strictly the case, the probabilty of a small hiccup creating one increases exponentially.
  - mpyne 9mo ago
    > I’d like to understand the constraints and systems involved that make 80% considered full utilization. There’s obviously something that limits a OS; is it tunable? There are OS tunables, and these tunables will have some measure of impact on the overall system performance. But the things that make high-utilization systems so bad for cycle time are inherent aspects of a queue-based system that you cannot escape through better tuning, because the issues these cause to cycle time were not due to a lack of tuning. If you can tune a system so that what previously would have been 95% loading is instead 82% loading that will show significant performance improvements, but you'd erase all those improvements if you just allowed the system to go back up to 95% loaded.

colechristensen 9mo ago

One way to think about it is 80% IS full utilization. The engineering time, the risks of decreased performance, and the fragility of pushing the limit at some point become not worth the benefits of reaching some higher metric of utilization. If it's not where you are, that optimum trade off point is somewhere.

mpyne 9mo ago

You absolutely do not want 90-95% utilization. At that level of utilitization random variability alone is enough to cause massive whiplash in average queue lengths. The cycle time impact of variability of a single-server/single-queue system at 95% load is nearly 25x the impact on the same system at 75% load, and there are similar measures for other process queues. As the other comment notes, you should really work from an assumption that 80% is max loading, just as you'd never aim to have a swap file or swap partition of exactly the amount of memory overcommit you expect.

rcxdude 9mo ago
Man, if there's one idea I wish I could jam into the head of anyone running an organization, it would be queuing theory. So many people can't understand that slack is necessary to have quick turnaround.
- sovietmudkipz 9mo ago
  Mmmm, I remember reading this in Systems Performance Brendan Gregg. I should revisit what was written…
sovietmudkipz 9mo ago
I target 80% utilization because I’ve seen that figure multiple times. I suppose I should rephrase: I’d like to understand the constraints and systems involved that make 80% considered full utilization. There’s obviously something that limits a OS; is it tunable? Questions I imagine a thorough multiplayer solutions engineer would be curious of, the kind of person whose trying to squeeze as much juice out of the hardware specs as possible.
- btschaegg 9mo ago
  It might not be the OS, but just statistical inevitability. If you're talking about CPU utilization on Linux, for example, it's not all that unlikely that the number you're staring at isn't "time spent by CPU doing things" but "average CPU run queue length". "100%" then doesn't only mean the CPU gets no rest, but "there's always someone waiting for a CPU to become free". It likely pays off to understand where the load numbers in your tooling actually come from. Even if that weren't the case, lead times for tasks will always increase with more utilization; see e.g. [1]: If you push a system from 80% to 95% utilization, you have to expect a ~4.75x increase in lead time for each task _on average_: (0.95/0.05) / (0.8/0.2) Note that all except the term containing ρ in the formula are defined by your system/software/clientele, so you can drop them for a purely relative comparison. [1]: https://en.wikipedia.org/wiki/Kingman%27s_formula https://en.wikipedia.org/wiki/Kingman%27s_formula Edit: Or, to try to picture the issue more intuitively: If you're on a highway nearing 100% utilization, you're likely standing in a traffic jam. And if that's not (yet) strictly the case, the probabilty of a small hiccup creating one increases exponentially.
- mpyne 9mo ago
  > I’d like to understand the constraints and systems involved that make 80% considered full utilization. There’s obviously something that limits a OS; is it tunable? There are OS tunables, and these tunables will have some measure of impact on the overall system performance. But the things that make high-utilization systems so bad for cycle time are inherent aspects of a queue-based system that you cannot escape through better tuning, because the issues these cause to cycle time were not due to a lack of tuning. If you can tune a system so that what previously would have been 95% loading is instead 82% loading that will show significant performance improvements, but you'd erase all those improvements if you just allowed the system to go back up to 95% loaded.