8 ms·

%CPU utilization is a lie

437 points by BrendanLong 1y ago

aaa_2006 1y ago
CPU utilization alone is misleading. Pair it with per core load average or runqueue length to see how threads are actually queuing. That view often reveals the real bottleneck, whether it is I/O, memory, or scheduling delays.
tgma 1y ago
The way they refer to cores in their system is confusing and non-standard. The author talks about a 5900X as a 24 core machine and discusses as if there are 24 cores, 12 of which are piggybacking on the other 12. In reality, there are 24 hyperthreads that are pretty much pairwise symmetric that execute on top of 12 cores with two sets of instruction pipeline sharing same underlying functional units.
- bboreham 1y ago
  Worth noting that the major clouds will sell this as 24 "vcpus".
- saghm 1y ago
  Years ago, when trying to explain hyper threading to my brother, who doesn't have any specialized technical knowledge, he came up with the analogy that it's like 2-ply toilet paper. You don't quite have 24 distinct things, but you have 12 that are roughly twice as useful as the individual ones, although you can't really separate them and expect them to work right.
  - nayuki 1y ago
    Nah, it's easier than that. Putting two chefs in the same kitchen doesn't let you cook twice the amount of food in the same amount of time, because sometimes the two chefs need to use the same resource at the same time - e.g. sink, counter space, oven. But, the additional chef does improve the utilization of the kitchen equipment, leaving fewer things unused.
    - whizzter 1y ago
      Maybe simplify more to make the concept of shared resource explicit. 2 chefs with one stove. As long as they're doing other things than frying it's ok and speeding things up but once they both need the stove you're down to 1 working and 1 waiting.
  - BobbyTables2 1y ago
    That’s perfect! Especially when it come to those advertisements “6 large rolls == 18 normal rolls”. Sure it might be thicker but nobody wipes their butt with 1/3 a square…
  - skeezyboy 1y ago
    > he came up with the analogy that it's like 2-ply toilet paper. as in youd only use it to wipe excrement from around your sphincter
- BrendanLong 1y ago
  Thanks for the feedback. I think you're right, so I changed a bunch of references and updated the description of the processor to 12 core / 24 thread. In some cases, I still think "cores" is the right terminology though, since my OS (confusingly) reports utilization as-if I had 24 cores.
  - sroussey 1y ago
    Eh, what’s a thread really? It’s a term for us humans. The difference between two threads and one core or two cores with shared resources? Nothing is really all that neat and clean. It more of a 2 level NUMA type architecture with 2 sets of 6 SMP sets of 2. The scheduler may look at it that way (depending), but to the end user? Or even to most of the system? Nah.
    - tgma 1y ago
      There are observable differences. For example, under HT, TLB flush or context switch will likely be observable by a neighboring thread whereas for in a full dedicated core, you won't observe such things.
- sroussey 1y ago
  Will be interesting when (if?) Intel ships software defined cores which are the logical inverse of hyper threading. Instead of having a big core with two instruction pipelines sharing big ALUs etc, they have two (or more) cores that combine resources and become one core. Almost the same, yet quite different. https://patents.google.com/patent/EP4579444A1/en https://patents.google.com/patent/EP4579444A1/en
  - tgma 1y ago
    There was the dreaded AMD FX chip which was advertised as 8 core, but shared functional units. Got sued, etc.
    - hedora 1y ago
      That patent seems to be describing a dumb way to implement pipelining / speculative execution. Am I missing something? Anyway, by my reading, it’s also similar to the Itanic, er, Itanium, where the “cores” that got combined were pipeline stages.
      - tgma 1y ago
        I did not read the patent (do not read patents as a matter of policy.) Was simply responding to the second paragraph that kind of reminded me of FX Bulldozer chips.
- Neil44 1y ago
  If both SMT cores are being asked to do the same workload they will likely contend for the same resource and execution units internally so the boost from SMT will be less. If they have different workloads the boost will be more. Now throw in P and E cores on newer CPU's, turbo and non-turbo, everything gets very complicated. I did see a study that adding SMT got a much better performance per watt boost than adding turbo which was interesting/useful.
hinkley 1y ago
How many times has hyperthreading been an actual performance benefit in processors? I cannot count how many times an article has come out saying you'll get better performance out of your <insert processor here> by turning off hyperthreading in the BIOS. It's gotta be at least 2 out of every 3 chip generations going back to the original implementation, where you're better off without it than with.
- FpUser 1y ago
  In the old days it had made the difference between my multimedia game like application not working at all with hyperthreading off to working just fine with it on.
  - hinkley 1y ago
    Yeah when it was one core versus 1.3 cores that's fair. But 3 core machines often did better (or at least more consistently run to run) with HT disabled.
- tgma 1y ago
  It has a lot to do with your workload as well as if not moreso than the chip architecture. The primary trade-off is the cache utilization when executing two sets of instruction streams.
  - hinkley 1y ago
    That's likely the primary factor, but then there's thermal throttling as well. You can't run all of the logic units flat out on a bunch of models of CPU.
    - gruez 1y ago
      >but then there's thermal throttling as well. You can't run all of the logic units flat out on a bunch of models of CPU. That doesn't make any sense. Disabling SMT likely saves negligible amount of power, but disables any performance to be gained from the other thread. If there's thermal budget available, it's better to spend it by shoving more work onto the second thread than to leave it disabled. If anything, due to voltage/frequency curves, it might even be better to run your CPU at lower clocks but with SMT enabled to make up for it (assuming it's amenable to your workloads), than it is to run with SMT disabled.
ot 1y ago
Utilization is not a lie, it is a measurement of a well-defined quantity, but people make assumptions to extrapolate capacity models from it, and that is where reality diverges from expectations. Hyperthreading (SMT) and Turbo (clock scaling) are only a part of the variables causing non-linearity, there are a number of other resources that are shared across cores and "run out" as load increases, like memory bandwidth, interconnect capacity, processor caches. Some bottlenecks might come even from the software, like spinlocks, which have non-linear impact on utilization. Furthermore, most CPU utilization metrics average over very long windows, from several seconds to a minute, but what really matters for the performance of a latency-sensitive server happens in the time-scale of tens to hundreds of milliseconds, and a multi-second average will not distinguish a bursty behavior from a smooth one. The latter has likely much more capacity to scale up. Unfortunately, the suggested approach is not that accurate either, because it hinges on two inherently unstable concepts > Benchmark how much work your server can do before having errors or unacceptable latency. The measurement of this is extremely noisy, as you want to detect the point where the server starts becoming unstable. Even if you look at a very simple queueing theory model, the derivatives close to saturation explode, so any nondeterministic noise is extremely amplified. > Report how much work your server is currently doing. There is rarely a stable definition of "work". Is it RPS? Request cost can vary even throughout the day. Is it instructions? Same, the typical IPC can vary. Ultimately, the confidence intervals you get from the load testing approach might be as large as what you can get from building an empirical model from utilization measurement, as long as you measure your utilization correctly.
- deleted 1y ago
  [deleted]
- SirMaster 1y ago
  What about 2 workloads that both register 100% CPU usage, but one workload draws significantly more power and heats the CPU up way more? Seems like that workload is utilizing more of the CPU, more of the transistors or something.
  - inetknght 1y ago
    Indeed, and there's a thing called "race to sleep". That is, you want to light up as much of the core as possible as fast as possible so you can get the CPU back to idle as soon as possible to save on battery power, because having the CPU active for more time (but not using as many circuits as it "could") draws a lot more power.
judge123 1y ago
This hits so close to home. I once tried to explain to a manager that a server at 60% utilization had zero room left, and they looked at me like I had two heads. I wish I had this article back then!
- hinkley 1y ago
  You also want to hit him with queueing theory. Up to a hair over 60% utilization the queuing delays on any work queue remain essentially negligible. At 70 they become noticeable, and at 80% they've doubled. And then it just turns into a shitshow from there on. The rule of thumb is 60% is zero, and 80% is the inflection point where delays go exponential. The biggest cluster I ran, we hit about 65% CPU at our target P95 time, which is pretty much right on the theoretical mark.
  - BrendanLong 1y ago
    A big part of this is that CPU utilization metrics are frequently averaged over a long period of time (like a minute), but if your SLO is 100 ms, what you care about is whether there's any ~100 ms period where CPU utilization is at 100%. Measuring p99 (or even p100) CPU utilization can make this a lot more visible.
    - hinkley 1y ago
      The vertical for this company was one where the daily traffic was oddly regular. That the two lines matched expectations likely has to do with the smoothness of the load. The biggest problem was not variance in request rate it was variance in request cost, which is usually where queuing kicks in, unless you're being dumb about things. I think for a lot of apps p98 is probably a better metric to chase, p99 and p100 are useful for understanding your application better, but I'm not sure you want your bosses to fixate on them. But our contracts were for p95, which was fortunate given the workload, or at least whoever made the contracts got good advice from the engineering team.
    - kccqzy 1y ago
      If your SLO is 100 ms you need far more granular measurement periods than that. You should measure the p99 or p100 utilization for every 5-ms interval or so.
deleted 1y ago
[deleted]
PaulKeeble 1y ago
This is bang on, you can't count the hyperthreads as double the performance, typically they are actually in practice only going to bring 15-30% if the job works well with it and their use will double the latency. Failing to account for loss in clockspeed as the core utilisation climbs is another way its not linear and in modern software for the desktop its really something to pay careful attention to. It should be possible from the information you can get on a CPU from the OS to better estimate utilisation involving at the very least these two factors. It becomes a bit more tricky to start to account for significantly going past the cache or available memory bandwidth and the potential drop in performance to existing threads that occurs from the increased pipeline stalls. But it can definitely be done better than it is currently.
- c2h5oh 1y ago
  To complicate things more HT performance varies wildly between CPU architectures and workloads. e.g. AMD implementation, especially in later Zen cores, is closer to a performance of a full thread than you'd see in Intel CPUs. Provided you are not memory bandwidth starved.
  - shim__ 1y ago
    Whats the difference between Intels and AMDs approach?
    - richardwhiuk 1y ago
      Basically it comes down to how much shared vs dedicated resources each core has.
  - RaftPeople 1y ago
    > To complicate things more HT performance varies wildly between CPU architectures and workloads. IBM's Power cpu's have also traditionally done a great job with SMT compared to Intel's implementation.
- magicalhippo 1y ago
  For memory-bound applications the scaling can be much better. A renderer I worked on was primarily memory-bound walking the accelerator structure, and saw 60-70% increase from hyperthreads. But overall yeah.
deleted 1y ago
[deleted]
pama 1y ago
Wait until you encounter GPU utilization. You could have two codes listing 100% utilization and have well over 100x performance difference from each other. The name of these metrics creates natural assumptions that are just wrong. Luckily it is relatively easy to estimate the FLOP/s throughput for most GPU codes and then simply compare to the theoretical peak performance of the hardware.
- spindump8930 1y ago
  Don't forget that theoretical peak performance is (probably) half the performance listed on the nvidia datasheet because they used the "with sparsity" numbers! I've seen this bite folks who miss the * on the figure or aren't used to reading those spec sheets.
- BrendanLong 1y ago
  Yeah, the obvious thing with processors is to do something similar: (1) Measure MIPS with perf (2) Compare that to max MIPS for your processor Unfortunately, MIPS is too vague since the amount of work done depends on the instruction, and there's no good way to measure max MIPS for most processors. (╯°□°)╯︵ ┻━┻
- saagarjha 1y ago
  If your workload is compute bound, of course. Sometimes you want to look at bandwidth instead.
  - pama 1y ago
    Of course. Lots of useful metrics exist to help tweak code performance without always needing to go into detailed profiler traces. GPU utilization is a particularly poor metric in helping much, except for making sure the code made it to the GPU somehow :-)
kristopolous 1y ago
Tried to explain this in a job interview 5 years ago. They thought I was a bullshitter
- bionsystem 1y ago
  Happened to me on a different topic, felt bad for way too long ; in hindsight I'm pretty sure I dodged a bullet.
  - kristopolous 1y ago
    This was the same interview where some guy was asking me about "big-o" - like the thing that you teach 19 year olds and I was saying that parallelization matters, i/o matters, quantization matters, whether you can run it on the GPU, these all matter. The simple "big-o" number doesn't account for whether you need to pass terabytes over the bus for every operation - and on actual computers moving around terabytes, I know, shockingly, this affects performance. And if you have a dual epyc board with 1,024 threads, being able to parallelize a solution and design things for cache optimization, this isn't meaningless. It's a weak classifier - if you really think I'm going to be doing a lexical sort in like O(n^3) like some kind of clown, I don't know what you're hiring here. Found out later he scored me "2/5". Alright, cool.
    - kiitos 1y ago
      "big o" usually refers to algorithmic complexity, which is something entirely orthogonal to all of the dimensions you mentioned obviously all of this stuff matters in the end but big-o comes before all of those other things
      - nomel 1y ago
        > but big-o comes before all of those other things If you're attempting to quantify algorithmic scalability with big-o, without those in mind, you'll often be wrong. There was a great post here a few years ago going into this, and how memory access "complexity" is what usually matters, and what dominantly shapes the scalability curve. It had nice examples showing how the expected big-o scalability curves were often completely wrong, outside of toys. If you're not trying to quantify algorithmic scalability with big-o, then have fun coming up with a fun collection of symbols to put next to your code, and petting your spherical cow!
1gn15 1y ago
Love that this website is public domain. Thank you, Brendan!
N_Lens 1y ago
This has been my experience running production workloads as well. Anytime CPU% goes over 50-60% suddenly it'll spike to 100% rather quickly, and the app/service is unusable. Learned to scale earlier than first thought.
CCs 1y ago
Uses stress-ng for benchmarking, even though the stress-ng documentation says it is not suitable for benchmarking. It was written to max out one component until it burns. Using a real app, like Memcached or Postgres would show more realistic numbers, closer to what people use in production. The difference is not major, 50% utilization is closer to 80% in real load, but it breaks down faster. Stress-ng is nicely linear until 100%, memcached will have a hockey stick curve at the end.
- BrendanLong 1y ago
  The advantage of stress-ng is that it's easy to make it run with specific CPU utilization numbers. The tests where I run some number of workers at 100% utilization are interesting since they give such perfect graphs, but I think the version where I have 24 workers and increase their utilization slowly is more realistic for showing how production CPU utilization changes.
  - BrendanLong 1y ago
    Fun data point though, I just ran three data points of the Phoronix nginx benchmark and got these results: - Pinned to 6 cores: 28k QPS - Pinned to 12 cores: 56k QPS - All 24 cores: 62k QPS I'm not sure how this applies to realistic workloads where you're using all of the cores but not maxing them out, but it looks like hyperthreading only adds ~10% performance in this case.
    - BrendanLong 1y ago
      Here's results of the Nginx benchmark pinned to 1-24 cores: https://docs.google.com/spreadsheets/d/1d_OK_ckLT1zTA_fG4vkq0NUSu585pLC9Wvcep6STVDQ/edit?usp=sharing https://docs.google.com/spreadsheets/d/1d_OK_ckLT1zTA_fG4vkq... At 51% reported CPU utilization, it's doing about 80% of the maximum requests per second, and it can't get above 80% utilization. I also added a section: https://www.brendanlong.com/cpu-utilization-is-a-lie.html#bonus-nginx https://www.brendanlong.com/cpu-utilization-is-a-lie.html#bo...
    - justsomehnguy 1y ago
gbin 1y ago
Yeah and those tests don't even trigger some memory or cache contention ...
throwmeaway222 1y ago
Yeah, this is what we all talked about when hyperthreading was first invented in 2000 era.
0xbadcafebee 1y ago
The benchmark is basically application performance testing, which is the most accurate representation you can get. Test the specific app(s) your server is running, with real-world data/scenarios, and keep cranking up the requests, until the server falls over. Nothing else will give you as accurate an indication of your server's actual maximum performance with that app. Do that for every variable that's relevant (# requests/s, payload size, # parameters, etc), so you have multiple real-world maximum-performance indicators to configure your observability monitors for. One way to get closer to reliable performance is to apply cpu scheduler limits to what runs your applications to keep them below a given threshold. This way you can better ensure you can sustain a given amount of performance. You don't want to run at 100% cpu for long, especially if disk i/o becomes hampered, system load skyrockets, and availability starts to plummet. Two thousand servers with 5000ms ping times due to system load is not a fun day at the office. (And actually you'll never get a completely accurate view, as performance can change per-server. Rack two identical servers in two different racks, run the same app on each, and you may see different real-world performance. One rack may be hotter than the other, there could be hidden hardware or firmware differences, etc. Even within a server, if one CPU is just nearer a hotter component than on another server, for reasons)
dragontamer 1y ago
There's many ways CPU utilization fails to work as expected. I didn't expect an article on this style. I was expecting the normal Linux/Windows utilization but wtf it's all RAM bottlenecked and the CPU is actually quiet and possibly down clocking thing. CPU Utilization is only how many cores are given threads to run by the OS (be it Windows or Linux). Those threads could be 100% blocked on memcpy but that's still CPU utilization. ------- Hyperthreads help: if one thread is truly CPU bound (or even more specifically: AVX / Vector unit bound), while a 2nd thread is hyperthreaded together that's memcpy / RAM bound, you'll magically get more performance due to higher utilization of resources. (Load/store units are separate from AVX compute units). In any case, this is a perennial subject with always new discoveries about how CPU Utilization is far less intuitive than many think. Still kinda fun to learn about new perspectives on this matter in any case.
smallstepforman 1y ago
Read kernel code to see how CPU utilisation is calculated. In essence, count scheduled threads to execute and divide by number of cores. Any latency (eg. wait for memory) is still calculated as busy core.
kqr 1y ago
It might be a lie, but it surely is a practical one. In my brief foray into site reliability engineering I used CPU utilisation (of CPU-bofund tasks) with queueing theory to choose how to scale servers before big events. The %CPU suggestions ran contrary to (and were much more conservative than) the "old wisdom" that would otherwise have been used. It worked out great at much lower cost than otherwise. What I'm trying to say is you shouldn't be afraid of using semi-crappy indicators just because they're semi-crappy. If it's the best you got it might be good enough anyway. In the case of CPU utilisation, though, the number in production shouldn't go above 40 % for many reasons. At 40 % there's usually still a little headroom. The mistake of the author was not using fundamentals of queueing theory to avoid high utilisation!
- mayama 1y ago
  Combination of CPU% and loadavg would generally tell how system is doing. I had systems where loadavg is high, waiting on network/io, but little cpu%. Tracing high load is not always straightforward as cpu% though, you have to go through io%, net%, syscalls etc.
- zekrioca 1y ago
  I noticed exactly the same thing. The author is saying something that has been repeatedly written in queueing theory books for decades, still they are noticing this only now.
- therealdrag0 1y ago
  > semi-crappy indicator … good enough. Agree. Another example of this is for metrics as percentiles per host that you have to average, vs histograms per host that get percentile calculated at aggregation time among hosts. Sure an avg/max of a percentile is technically not a percentile, but in practice switching between one or the other hasn’t affected my operations at all. Yet I know some people are adamant about mathematical correctness as if that translates to operations.
  - arccy 1y ago
    That works ok when you have evenly distributed load (which you want / would hope to have), much less so when your workload is highly unbalanced.
timzaman 1y ago
What's become of hacker news that this is #2 post ? This is basic knowledge any programmer gets in their first few years..
- therealdrag0 1y ago
  It’s a big industry with a wide range of knowledge levels.
- saagarjha 1y ago
  I encounter very few programmers who learn this.
steventhedev 1y ago
%cpu is misleading at best, and should largely be considered harmful. System load is well defined, matches user expectations, and covers several edge cases (auditd going crazy, broken CPU timers, etc).
mustache_kimono 1y ago
Reminds me of Brendan Gregg's "CPU Utilization is Wrong" but this blog fails to discuss that blog's key point that CPU utilization is a measure of whether or not the CPU is busy, including whether the CPU is waiting [0]. That blog also explains that the IPC (instructions per cycle) metric actually measures useful work hidden within that busy state. [0]: https://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html https://www.brendangregg.com/blog/2017-05-09/cpu-utilization...
- 4gotunameagain 1y ago
  What's up with Brendans and CPU utilisation concerns, any Brendan to shine some light ?
  - BrendanLong 1y ago
    I'd love to explain, but you'd need to change your name to Brendan first.
ChaoPrayaWave 1y ago
These days I treat CPU usage as just a hint, not a conclusion. I also look at response times, queue lengths, and try to figure out what the app is actually doing when it looks idle.
kunley 1y ago
tl;dr: guy vibecodes a thing to measure something he doesn't fully understand and then realizes his methodology is wrong. Ends up with a catchy "X is a lie" title, which itself can be considered a lie.
swiftcoder 1y ago
I remember being stuck in a discussion with management one time, that went something like this: Manager: CPU utilisation is 100% under load! We have to migrate to bigger instances. Me: but is the CPU actually doing useful work? (chat, it was not. busy waiting is CPU utilisation too)
- kristianp 1y ago
  How do you measure the amount of busy waiting?
  - swiftcoder 1y ago
    I don't think there is a good general tool for this. In this specific case, I went spelunking for all the points where we had thread contention over resources, and discovered that for several resources quite a lot of CPU cycles were being expended to no use. The goal is really to eliminate the underlying resource contention - we added per-thread caches I various places, swapped out the logging system, and were able to ~double the system throughput during times when top showed the system to be "fully loaded"
HPsquared 1y ago
GPU utilisation as reported in Task Manager also seems quite a big lie, it bears little relation to Watts / TDP.
Aissen 1y ago
Funny that it talks about matrixprod, which I think is not that relevant as benchmark — unless you care about x87 performance specifically. I recently sent a pull request to try to address that in a generic manner: https://github.com/ColinIanKing/stress-ng/pull/561 https://github.com/ColinIanKing/stress-ng/pull/561 Yet I'm still surprised by this benchmark. On both Zen2 and Zen4 in my tests (5900X from the article is Zen3), matrixprod still benefits from hyperthreading and scales a bit after all the physical cores are filled, unlike what the article results show. All of this is tangential of course, as I'd tend to agree that CPU utilization% is just an imprecise metric and should only be used as a measure of "is something running".
bob1029 1y ago
I think looking at power consumption is potentially a more interesting canary when using very high core count parts. I've ran some ML experiments on my 5950x and I can tell that the CPU utilization figure is entirely decoupled from physical reality by observing the amount of flicker induced in my office lighting by the PWM noise in the machine. There are some code paths that show 10% utilization across all cores but make the cicadas outside my office window stop buzzing because the semiconductors get so loud. Other code paths show all cores 100% maxed flatline and it's like the machine isn't even on.
fennecfoxy 1y ago
I think it's more for cores, right? % util is just % of idle cycles across all logical cores as far as I know. It wouldn't really make sense to include all parts of the CPU in the calculation.
fuzzfactor 1y ago
Windows users try this: Ctrl-Alt-Del then launch TaskManager. In TaskManager, click the "Performance" tab and see the simple stats. While on the Performance tab, then click the ellipsis (. . .) menu, so you can then open ResourceMonitor. Then close TaskManager. In ResourceMonitor, under the Overview tab, for the CPU click the column header for "Average CPU" so that the processes using the most CPU are shown top-down from most usage to least. In Overview, for Disk click the Write (B/sec) column header, for Network click Send (B/sec), and for Memory click Commit (KB). Then under the individual CPU, Memory, Disk, and Network tabs click on the similar column headers. Under any tab now you should be able to see the most prominent resource usages. Notice how your CPU settles down after a while of idling. Then click on the Disk tab to focus your attention on that one exclusively. Let it sit for 5 or 10 minutes then check your CPU usage. See if it's been climbing gradually higher while you weren't looking.
rollcat 1y ago
I'm surprised nobody has mentioned OpenBSD yet. They've been advocating against SMT for a long while, citing security risks and inconsistent performance gains. I don't know which HW/CPU bug in the long series of rowhammer, meltdown, spectre, etc prompted the action, but they've completely disabled SMT in the default installation at some point. The core idea by itself is fine: keep the ALUs busy. Maybe security-wise, the present trade-off is acceptable, if you can instruct the scheduler to put threads from the same security domain on the same physical core. (How to tell when two threads are not a threat to each other is left up as an exercise.)
- saagarjha 1y ago
  The security argument might make sense but OpenBSD is not really the place to take performance advice from
  - whizzter 1y ago
    Do people even use or mention OpenBSD out of performance concerns? We all know they prioritize security.
  - rollcat 1y ago
    My original point stands, also per TFA - performance gains from SMT are questionable for certain workloads. Whether OpenBSD prioritises absolute performance is besides the point - they benchmark against their own goals, not someone else's achievements.
freehorse 1y ago
Author discovers that performance does not scale proportionally to %CPU utilisation, and gets instead to the conclusion that %CPU utilisation is a lie. There are many reasons for the lack of a proportional relationship, even in the case where you do not have hyperthreading or downclocking (in which cases you just need to interpret %CPU utilisation in that context, rather than declare it "a lie"). Even in apple silicon where these are usually not an issue, you often do not get an exactly proportional scaling. There may be overheads when utilising multiple cores wrt how data is passed around, or resource bottlenecks other than CPU.
- saagarjha 1y ago
  Apple silicon downclocks quite a lot especially if you have a passively cooled machine
  - freehorse 1y ago
    With the exception of macbook air that has passive cooling nothing as aggressive as "turbo" modes, and ime it is relatively hard to get to thermal limits just with cpu in general for the devices I have used. Most other manufacturers nowadays officially advertise boosted single core clock speeds that are much higher and lower when more cores are used at the same time. Thermal limits, in contrast, are much more circumstantial.
PathOfEclipse 1y ago
I think it was always a mistake to pretend hyperthreading doubles your core count. I always assumed it was just due to laziness; the operating system treats a hyperthreaded core as two "virtual cores" and schedules as two cores, so then every other piece of tooling sees double the number of actual cores. There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages. But, maybe that's hard to do given how the OS implements hyperthreading.
- fluoridation 1y ago
  >There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages On AMD, threads may as well be cores. If you take a Ryzen and disable SMT, you're basically halving its parallelism, at least for some tasks. On Intel you're just turning off an extra 10-20%.
  - PathOfEclipse 1y ago
    Can you provide some links for this? A quick web search turns this up at near the top from 2024: https://www.techpowerup.com/review/amd-ryzen-9-9700x-performance-smt-disabled/22.html https://www.techpowerup.com/review/amd-ryzen-9-9700x-perform... The benchmarks show a 10% drop in "application" performance when SMT is disabled, but an overall 1-3% increase in performance for games. From a hardware perspective, I can't imagine how it could be physically possible to double performance by enabling SMT.
    - fluoridation 1y ago
      I don't. It's based off my own testing, not by disabling SMT, but by running either <core_count> or <thread_count> parallel threads. It was my own code, so it's possible code that uses SIMD more heavily will see a less-significant speed-up. It's also possible I just measured wrong; running Cargo on a directory with -j16 and -j32 takes 58 and 48 seconds respectively. >From a hardware perspective, I can't imagine how it could be physically possible to double performance by enabling SMT. It depends on which parts of the processor your code uses. SMT works by duplicating some but not all the components of each core, so a single core can work on multiple independent uops simultaneously. I don't know the specifics, but I can imagine ALU-type code (jumps, calls, movs, etc.) benefits more from SMT than very math-heavy code. That would explain why rustc saw a greater speedup than Cinebench, as compiler code is very twisty with not a lot of math.
tonymet 1y ago
I like his empirical approach to get to the root significance of the cpu %-age indicator. Software engineers and data analysts take discrete "data" measurements and statistics for granted. "data" / "stats" are only a report, and that report is often incorrect.
bdhcuidbebe 1y ago
Thats some strong words about not RTFM.
deleted 1y ago
[deleted]
morning-coffee 1y ago
The lie is that hyper thread "cores" are equal to real "cores". Maybe this is what happens when an over 20-year old technology (hack) becomes ubiquitous and gets forgotten about? (We have to rediscover why our performance measurements don't seem to make sense?) The other thing I think we have a hard time visualizing is that processor is only either executing (100%) or its waiting to execute (0%) and that happens over varying timescales... so trying to assign a % in between inherently means you're averaging over some arbitrary timescale...
codedokode 1y ago
A worse lie is memory usage reporting, I think in every major OS it is understated and misreported. In case with Linux, I wanted to know who is using memory, and tried to add PSS values for every process, I never got back the total memory usage. In case with Windows/Mac I judge by screenshot of their tools which show unrealistically small values. As for the article, the slowdown can be also caused by increased use of shared resources like caches, TLBs, branch predictors.
- biggusdickus69 1y ago
  The memory usage is interesting, where different kind of shared memory is obvious hard to visualize, just two values per process doesn’t say enough. Most users actually wants a list of ”what can I kill to make the computer faster”, I.e. they want an oracle (no pun) that knows how fast the computer will be if different processes are killed.

437 points by BrendanLong 1y ago

aaa_2006 1y ago

CPU utilization alone is misleading. Pair it with per core load average or runqueue length to see how threads are actually queuing. That view often reveals the real bottleneck, whether it is I/O, memory, or scheduling delays.

tgma 1y ago

The way they refer to cores in their system is confusing and non-standard. The author talks about a 5900X as a 24 core machine and discusses as if there are 24 cores, 12 of which are piggybacking on the other 12. In reality, there are 24 hyperthreads that are pretty much pairwise symmetric that execute on top of 12 cores with two sets of instruction pipeline sharing same underlying functional units.

bboreham 1y ago
Worth noting that the major clouds will sell this as 24 "vcpus".
saghm 1y ago
Years ago, when trying to explain hyper threading to my brother, who doesn't have any specialized technical knowledge, he came up with the analogy that it's like 2-ply toilet paper. You don't quite have 24 distinct things, but you have 12 that are roughly twice as useful as the individual ones, although you can't really separate them and expect them to work right.
- nayuki 1y ago
  Nah, it's easier than that. Putting two chefs in the same kitchen doesn't let you cook twice the amount of food in the same amount of time, because sometimes the two chefs need to use the same resource at the same time - e.g. sink, counter space, oven. But, the additional chef does improve the utilization of the kitchen equipment, leaving fewer things unused.
  - whizzter 1y ago
    Maybe simplify more to make the concept of shared resource explicit. 2 chefs with one stove. As long as they're doing other things than frying it's ok and speeding things up but once they both need the stove you're down to 1 working and 1 waiting.
- BobbyTables2 1y ago
  That’s perfect! Especially when it come to those advertisements “6 large rolls == 18 normal rolls”. Sure it might be thicker but nobody wipes their butt with 1/3 a square…
- skeezyboy 1y ago
  > he came up with the analogy that it's like 2-ply toilet paper. as in youd only use it to wipe excrement from around your sphincter
BrendanLong 1y ago
Thanks for the feedback. I think you're right, so I changed a bunch of references and updated the description of the processor to 12 core / 24 thread. In some cases, I still think "cores" is the right terminology though, since my OS (confusingly) reports utilization as-if I had 24 cores.
- sroussey 1y ago
  Eh, what’s a thread really? It’s a term for us humans. The difference between two threads and one core or two cores with shared resources? Nothing is really all that neat and clean. It more of a 2 level NUMA type architecture with 2 sets of 6 SMP sets of 2. The scheduler may look at it that way (depending), but to the end user? Or even to most of the system? Nah.
  - tgma 1y ago
    There are observable differences. For example, under HT, TLB flush or context switch will likely be observable by a neighboring thread whereas for in a full dedicated core, you won't observe such things.
sroussey 1y ago
Will be interesting when (if?) Intel ships software defined cores which are the logical inverse of hyper threading. Instead of having a big core with two instruction pipelines sharing big ALUs etc, they have two (or more) cores that combine resources and become one core. Almost the same, yet quite different. https://patents.google.com/patent/EP4579444A1/en https://patents.google.com/patent/EP4579444A1/en
- tgma 1y ago
  There was the dreaded AMD FX chip which was advertised as 8 core, but shared functional units. Got sued, etc.
  - hedora 1y ago
    That patent seems to be describing a dumb way to implement pipelining / speculative execution. Am I missing something? Anyway, by my reading, it’s also similar to the Itanic, er, Itanium, where the “cores” that got combined were pipeline stages.
    - tgma 1y ago
      I did not read the patent (do not read patents as a matter of policy.) Was simply responding to the second paragraph that kind of reminded me of FX Bulldozer chips.
Neil44 1y ago
If both SMT cores are being asked to do the same workload they will likely contend for the same resource and execution units internally so the boost from SMT will be less. If they have different workloads the boost will be more. Now throw in P and E cores on newer CPU's, turbo and non-turbo, everything gets very complicated. I did see a study that adding SMT got a much better performance per watt boost than adding turbo which was interesting/useful.

hinkley 1y ago

How many times has hyperthreading been an actual performance benefit in processors? I cannot count how many times an article has come out saying you'll get better performance out of your <insert processor here> by turning off hyperthreading in the BIOS. It's gotta be at least 2 out of every 3 chip generations going back to the original implementation, where you're better off without it than with.

FpUser 1y ago
In the old days it had made the difference between my multimedia game like application not working at all with hyperthreading off to working just fine with it on.
- hinkley 1y ago
  Yeah when it was one core versus 1.3 cores that's fair. But 3 core machines often did better (or at least more consistently run to run) with HT disabled.
tgma 1y ago
It has a lot to do with your workload as well as if not moreso than the chip architecture. The primary trade-off is the cache utilization when executing two sets of instruction streams.
- hinkley 1y ago
  That's likely the primary factor, but then there's thermal throttling as well. You can't run all of the logic units flat out on a bunch of models of CPU.
  - gruez 1y ago
    >but then there's thermal throttling as well. You can't run all of the logic units flat out on a bunch of models of CPU. That doesn't make any sense. Disabling SMT likely saves negligible amount of power, but disables any performance to be gained from the other thread. If there's thermal budget available, it's better to spend it by shoving more work onto the second thread than to leave it disabled. If anything, due to voltage/frequency curves, it might even be better to run your CPU at lower clocks but with SMT enabled to make up for it (assuming it's amenable to your workloads), than it is to run with SMT disabled.

ot 1y ago

Utilization is not a lie, it is a measurement of a well-defined quantity, but people make assumptions to extrapolate capacity models from it, and that is where reality diverges from expectations. Hyperthreading (SMT) and Turbo (clock scaling) are only a part of the variables causing non-linearity, there are a number of other resources that are shared across cores and "run out" as load increases, like memory bandwidth, interconnect capacity, processor caches. Some bottlenecks might come even from the software, like spinlocks, which have non-linear impact on utilization. Furthermore, most CPU utilization metrics average over very long windows, from several seconds to a minute, but what really matters for the performance of a latency-sensitive server happens in the time-scale of tens to hundreds of milliseconds, and a multi-second average will not distinguish a bursty behavior from a smooth one. The latter has likely much more capacity to scale up. Unfortunately, the suggested approach is not that accurate either, because it hinges on two inherently unstable concepts > Benchmark how much work your server can do before having errors or unacceptable latency. The measurement of this is extremely noisy, as you want to detect the point where the server starts becoming unstable. Even if you look at a very simple queueing theory model, the derivatives close to saturation explode, so any nondeterministic noise is extremely amplified. > Report how much work your server is currently doing. There is rarely a stable definition of "work". Is it RPS? Request cost can vary even throughout the day. Is it instructions? Same, the typical IPC can vary. Ultimately, the confidence intervals you get from the load testing approach might be as large as what you can get from building an empirical model from utilization measurement, as long as you measure your utilization correctly.

deleted 1y ago
[deleted]
SirMaster 1y ago
What about 2 workloads that both register 100% CPU usage, but one workload draws significantly more power and heats the CPU up way more? Seems like that workload is utilizing more of the CPU, more of the transistors or something.
- inetknght 1y ago
  Indeed, and there's a thing called "race to sleep". That is, you want to light up as much of the core as possible as fast as possible so you can get the CPU back to idle as soon as possible to save on battery power, because having the CPU active for more time (but not using as many circuits as it "could") draws a lot more power.

judge123 1y ago

This hits so close to home. I once tried to explain to a manager that a server at 60% utilization had zero room left, and they looked at me like I had two heads. I wish I had this article back then!

hinkley 1y ago
You also want to hit him with queueing theory. Up to a hair over 60% utilization the queuing delays on any work queue remain essentially negligible. At 70 they become noticeable, and at 80% they've doubled. And then it just turns into a shitshow from there on. The rule of thumb is 60% is zero, and 80% is the inflection point where delays go exponential. The biggest cluster I ran, we hit about 65% CPU at our target P95 time, which is pretty much right on the theoretical mark.
- BrendanLong 1y ago
  A big part of this is that CPU utilization metrics are frequently averaged over a long period of time (like a minute), but if your SLO is 100 ms, what you care about is whether there's any ~100 ms period where CPU utilization is at 100%. Measuring p99 (or even p100) CPU utilization can make this a lot more visible.
  - hinkley 1y ago
    The vertical for this company was one where the daily traffic was oddly regular. That the two lines matched expectations likely has to do with the smoothness of the load. The biggest problem was not variance in request rate it was variance in request cost, which is usually where queuing kicks in, unless you're being dumb about things. I think for a lot of apps p98 is probably a better metric to chase, p99 and p100 are useful for understanding your application better, but I'm not sure you want your bosses to fixate on them. But our contracts were for p95, which was fortunate given the workload, or at least whoever made the contracts got good advice from the engineering team.
  - kccqzy 1y ago
    If your SLO is 100 ms you need far more granular measurement periods than that. You should measure the p99 or p100 utilization for every 5-ms interval or so.

deleted 1y ago

[deleted]

PaulKeeble 1y ago

This is bang on, you can't count the hyperthreads as double the performance, typically they are actually in practice only going to bring 15-30% if the job works well with it and their use will double the latency. Failing to account for loss in clockspeed as the core utilisation climbs is another way its not linear and in modern software for the desktop its really something to pay careful attention to. It should be possible from the information you can get on a CPU from the OS to better estimate utilisation involving at the very least these two factors. It becomes a bit more tricky to start to account for significantly going past the cache or available memory bandwidth and the potential drop in performance to existing threads that occurs from the increased pipeline stalls. But it can definitely be done better than it is currently.

c2h5oh 1y ago
To complicate things more HT performance varies wildly between CPU architectures and workloads. e.g. AMD implementation, especially in later Zen cores, is closer to a performance of a full thread than you'd see in Intel CPUs. Provided you are not memory bandwidth starved.
- shim__ 1y ago
  Whats the difference between Intels and AMDs approach?
  - richardwhiuk 1y ago
    Basically it comes down to how much shared vs dedicated resources each core has.
- RaftPeople 1y ago
  > To complicate things more HT performance varies wildly between CPU architectures and workloads. IBM's Power cpu's have also traditionally done a great job with SMT compared to Intel's implementation.
magicalhippo 1y ago
For memory-bound applications the scaling can be much better. A renderer I worked on was primarily memory-bound walking the accelerator structure, and saw 60-70% increase from hyperthreads. But overall yeah.

deleted 1y ago

[deleted]

pama 1y ago

Wait until you encounter GPU utilization. You could have two codes listing 100% utilization and have well over 100x performance difference from each other. The name of these metrics creates natural assumptions that are just wrong. Luckily it is relatively easy to estimate the FLOP/s throughput for most GPU codes and then simply compare to the theoretical peak performance of the hardware.

spindump8930 1y ago
Don't forget that theoretical peak performance is (probably) half the performance listed on the nvidia datasheet because they used the "with sparsity" numbers! I've seen this bite folks who miss the * on the figure or aren't used to reading those spec sheets.
BrendanLong 1y ago
Yeah, the obvious thing with processors is to do something similar: (1) Measure MIPS with perf (2) Compare that to max MIPS for your processor Unfortunately, MIPS is too vague since the amount of work done depends on the instruction, and there's no good way to measure max MIPS for most processors. (╯°□°)╯︵ ┻━┻
saagarjha 1y ago
If your workload is compute bound, of course. Sometimes you want to look at bandwidth instead.
- pama 1y ago
  Of course. Lots of useful metrics exist to help tweak code performance without always needing to go into detailed profiler traces. GPU utilization is a particularly poor metric in helping much, except for making sure the code made it to the GPU somehow :-)

kristopolous 1y ago

Tried to explain this in a job interview 5 years ago. They thought I was a bullshitter

bionsystem 1y ago
Happened to me on a different topic, felt bad for way too long ; in hindsight I'm pretty sure I dodged a bullet.
- kristopolous 1y ago
  This was the same interview where some guy was asking me about "big-o" - like the thing that you teach 19 year olds and I was saying that parallelization matters, i/o matters, quantization matters, whether you can run it on the GPU, these all matter. The simple "big-o" number doesn't account for whether you need to pass terabytes over the bus for every operation - and on actual computers moving around terabytes, I know, shockingly, this affects performance. And if you have a dual epyc board with 1,024 threads, being able to parallelize a solution and design things for cache optimization, this isn't meaningless. It's a weak classifier - if you really think I'm going to be doing a lexical sort in like O(n^3) like some kind of clown, I don't know what you're hiring here. Found out later he scored me "2/5". Alright, cool.
  - kiitos 1y ago
    "big o" usually refers to algorithmic complexity, which is something entirely orthogonal to all of the dimensions you mentioned obviously all of this stuff matters in the end but big-o comes before all of those other things
    - nomel 1y ago
      > but big-o comes before all of those other things If you're attempting to quantify algorithmic scalability with big-o, without those in mind, you'll often be wrong. There was a great post here a few years ago going into this, and how memory access "complexity" is what usually matters, and what dominantly shapes the scalability curve. It had nice examples showing how the expected big-o scalability curves were often completely wrong, outside of toys. If you're not trying to quantify algorithmic scalability with big-o, then have fun coming up with a fun collection of symbols to put next to your code, and petting your spherical cow!

1gn15 1y ago

Love that this website is public domain. Thank you, Brendan!

N_Lens 1y ago

This has been my experience running production workloads as well. Anytime CPU% goes over 50-60% suddenly it'll spike to 100% rather quickly, and the app/service is unusable. Learned to scale earlier than first thought.

CCs 1y ago

Uses stress-ng for benchmarking, even though the stress-ng documentation says it is not suitable for benchmarking. It was written to max out one component until it burns. Using a real app, like Memcached or Postgres would show more realistic numbers, closer to what people use in production. The difference is not major, 50% utilization is closer to 80% in real load, but it breaks down faster. Stress-ng is nicely linear until 100%, memcached will have a hockey stick curve at the end.

BrendanLong 1y ago
The advantage of stress-ng is that it's easy to make it run with specific CPU utilization numbers. The tests where I run some number of workers at 100% utilization are interesting since they give such perfect graphs, but I think the version where I have 24 workers and increase their utilization slowly is more realistic for showing how production CPU utilization changes.
- BrendanLong 1y ago
  Fun data point though, I just ran three data points of the Phoronix nginx benchmark and got these results: - Pinned to 6 cores: 28k QPS - Pinned to 12 cores: 56k QPS - All 24 cores: 62k QPS I'm not sure how this applies to realistic workloads where you're using all of the cores but not maxing them out, but it looks like hyperthreading only adds ~10% performance in this case.
  - BrendanLong 1y ago
    Here's results of the Nginx benchmark pinned to 1-24 cores: https://docs.google.com/spreadsheets/d/1d_OK_ckLT1zTA_fG4vkq0NUSu585pLC9Wvcep6STVDQ/edit?usp=sharing https://docs.google.com/spreadsheets/d/1d_OK_ckLT1zTA_fG4vkq... At 51% reported CPU utilization, it's doing about 80% of the maximum requests per second, and it can't get above 80% utilization. I also added a section: https://www.brendanlong.com/cpu-utilization-is-a-lie.html#bonus-nginx https://www.brendanlong.com/cpu-utilization-is-a-lie.html#bo...
  - justsomehnguy 1y ago

gbin 1y ago

Yeah and those tests don't even trigger some memory or cache contention ...

throwmeaway222 1y ago

Yeah, this is what we all talked about when hyperthreading was first invented in 2000 era.

0xbadcafebee 1y ago

The benchmark is basically application performance testing, which is the most accurate representation you can get. Test the specific app(s) your server is running, with real-world data/scenarios, and keep cranking up the requests, until the server falls over. Nothing else will give you as accurate an indication of your server's actual maximum performance with that app. Do that for every variable that's relevant (# requests/s, payload size, # parameters, etc), so you have multiple real-world maximum-performance indicators to configure your observability monitors for. One way to get closer to reliable performance is to apply cpu scheduler limits to what runs your applications to keep them below a given threshold. This way you can better ensure you can sustain a given amount of performance. You don't want to run at 100% cpu for long, especially if disk i/o becomes hampered, system load skyrockets, and availability starts to plummet. Two thousand servers with 5000ms ping times due to system load is not a fun day at the office. (And actually you'll never get a completely accurate view, as performance can change per-server. Rack two identical servers in two different racks, run the same app on each, and you may see different real-world performance. One rack may be hotter than the other, there could be hidden hardware or firmware differences, etc. Even within a server, if one CPU is just nearer a hotter component than on another server, for reasons)

dragontamer 1y ago

There's many ways CPU utilization fails to work as expected. I didn't expect an article on this style. I was expecting the normal Linux/Windows utilization but wtf it's all RAM bottlenecked and the CPU is actually quiet and possibly down clocking thing. CPU Utilization is only how many cores are given threads to run by the OS (be it Windows or Linux). Those threads could be 100% blocked on memcpy but that's still CPU utilization. ------- Hyperthreads help: if one thread is truly CPU bound (or even more specifically: AVX / Vector unit bound), while a 2nd thread is hyperthreaded together that's memcpy / RAM bound, you'll magically get more performance due to higher utilization of resources. (Load/store units are separate from AVX compute units). In any case, this is a perennial subject with always new discoveries about how CPU Utilization is far less intuitive than many think. Still kinda fun to learn about new perspectives on this matter in any case.

smallstepforman 1y ago

Read kernel code to see how CPU utilisation is calculated. In essence, count scheduled threads to execute and divide by number of cores. Any latency (eg. wait for memory) is still calculated as busy core.

kqr 1y ago

It might be a lie, but it surely is a practical one. In my brief foray into site reliability engineering I used CPU utilisation (of CPU-bofund tasks) with queueing theory to choose how to scale servers before big events. The %CPU suggestions ran contrary to (and were much more conservative than) the "old wisdom" that would otherwise have been used. It worked out great at much lower cost than otherwise. What I'm trying to say is you shouldn't be afraid of using semi-crappy indicators just because they're semi-crappy. If it's the best you got it might be good enough anyway. In the case of CPU utilisation, though, the number in production shouldn't go above 40 % for many reasons. At 40 % there's usually still a little headroom. The mistake of the author was not using fundamentals of queueing theory to avoid high utilisation!

mayama 1y ago
Combination of CPU% and loadavg would generally tell how system is doing. I had systems where loadavg is high, waiting on network/io, but little cpu%. Tracing high load is not always straightforward as cpu% though, you have to go through io%, net%, syscalls etc.
zekrioca 1y ago
I noticed exactly the same thing. The author is saying something that has been repeatedly written in queueing theory books for decades, still they are noticing this only now.
therealdrag0 1y ago
> semi-crappy indicator … good enough. Agree. Another example of this is for metrics as percentiles per host that you have to average, vs histograms per host that get percentile calculated at aggregation time among hosts. Sure an avg/max of a percentile is technically not a percentile, but in practice switching between one or the other hasn’t affected my operations at all. Yet I know some people are adamant about mathematical correctness as if that translates to operations.
- arccy 1y ago
  That works ok when you have evenly distributed load (which you want / would hope to have), much less so when your workload is highly unbalanced.

timzaman 1y ago

What's become of hacker news that this is #2 post ? This is basic knowledge any programmer gets in their first few years..

therealdrag0 1y ago
It’s a big industry with a wide range of knowledge levels.
saagarjha 1y ago
I encounter very few programmers who learn this.

steventhedev 1y ago

%cpu is misleading at best, and should largely be considered harmful. System load is well defined, matches user expectations, and covers several edge cases (auditd going crazy, broken CPU timers, etc).

mustache_kimono 1y ago

Reminds me of Brendan Gregg's "CPU Utilization is Wrong" but this blog fails to discuss that blog's key point that CPU utilization is a measure of whether or not the CPU is busy, including whether the CPU is waiting [0]. That blog also explains that the IPC (instructions per cycle) metric actually measures useful work hidden within that busy state. [0]: https://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html https://www.brendangregg.com/blog/2017-05-09/cpu-utilization...

4gotunameagain 1y ago
What's up with Brendans and CPU utilisation concerns, any Brendan to shine some light ?
- BrendanLong 1y ago
  I'd love to explain, but you'd need to change your name to Brendan first.

ChaoPrayaWave 1y ago

These days I treat CPU usage as just a hint, not a conclusion. I also look at response times, queue lengths, and try to figure out what the app is actually doing when it looks idle.

kunley 1y ago

tl;dr: guy vibecodes a thing to measure something he doesn't fully understand and then realizes his methodology is wrong. Ends up with a catchy "X is a lie" title, which itself can be considered a lie.

swiftcoder 1y ago

I remember being stuck in a discussion with management one time, that went something like this: Manager: CPU utilisation is 100% under load! We have to migrate to bigger instances. Me: but is the CPU actually doing useful work? (chat, it was not. busy waiting is CPU utilisation too)

kristianp 1y ago
How do you measure the amount of busy waiting?
- swiftcoder 1y ago
  I don't think there is a good general tool for this. In this specific case, I went spelunking for all the points where we had thread contention over resources, and discovered that for several resources quite a lot of CPU cycles were being expended to no use. The goal is really to eliminate the underlying resource contention - we added per-thread caches I various places, swapped out the logging system, and were able to ~double the system throughput during times when top showed the system to be "fully loaded"

HPsquared 1y ago

GPU utilisation as reported in Task Manager also seems quite a big lie, it bears little relation to Watts / TDP.

Aissen 1y ago

Funny that it talks about matrixprod, which I think is not that relevant as benchmark — unless you care about x87 performance specifically. I recently sent a pull request to try to address that in a generic manner: https://github.com/ColinIanKing/stress-ng/pull/561 https://github.com/ColinIanKing/stress-ng/pull/561 Yet I'm still surprised by this benchmark. On both Zen2 and Zen4 in my tests (5900X from the article is Zen3), matrixprod still benefits from hyperthreading and scales a bit after all the physical cores are filled, unlike what the article results show. All of this is tangential of course, as I'd tend to agree that CPU utilization% is just an imprecise metric and should only be used as a measure of "is something running".

bob1029 1y ago

I think looking at power consumption is potentially a more interesting canary when using very high core count parts. I've ran some ML experiments on my 5950x and I can tell that the CPU utilization figure is entirely decoupled from physical reality by observing the amount of flicker induced in my office lighting by the PWM noise in the machine. There are some code paths that show 10% utilization across all cores but make the cicadas outside my office window stop buzzing because the semiconductors get so loud. Other code paths show all cores 100% maxed flatline and it's like the machine isn't even on.

fennecfoxy 1y ago

I think it's more for cores, right? % util is just % of idle cycles across all logical cores as far as I know. It wouldn't really make sense to include all parts of the CPU in the calculation.

fuzzfactor 1y ago

Windows users try this: Ctrl-Alt-Del then launch TaskManager. In TaskManager, click the "Performance" tab and see the simple stats. While on the Performance tab, then click the ellipsis (. . .) menu, so you can then open ResourceMonitor. Then close TaskManager. In ResourceMonitor, under the Overview tab, for the CPU click the column header for "Average CPU" so that the processes using the most CPU are shown top-down from most usage to least. In Overview, for Disk click the Write (B/sec) column header, for Network click Send (B/sec), and for Memory click Commit (KB). Then under the individual CPU, Memory, Disk, and Network tabs click on the similar column headers. Under any tab now you should be able to see the most prominent resource usages. Notice how your CPU settles down after a while of idling. Then click on the Disk tab to focus your attention on that one exclusively. Let it sit for 5 or 10 minutes then check your CPU usage. See if it's been climbing gradually higher while you weren't looking.

rollcat 1y ago

I'm surprised nobody has mentioned OpenBSD yet. They've been advocating against SMT for a long while, citing security risks and inconsistent performance gains. I don't know which HW/CPU bug in the long series of rowhammer, meltdown, spectre, etc prompted the action, but they've completely disabled SMT in the default installation at some point. The core idea by itself is fine: keep the ALUs busy. Maybe security-wise, the present trade-off is acceptable, if you can instruct the scheduler to put threads from the same security domain on the same physical core. (How to tell when two threads are not a threat to each other is left up as an exercise.)

saagarjha 1y ago
The security argument might make sense but OpenBSD is not really the place to take performance advice from
- whizzter 1y ago
  Do people even use or mention OpenBSD out of performance concerns? We all know they prioritize security.
- rollcat 1y ago
  My original point stands, also per TFA - performance gains from SMT are questionable for certain workloads. Whether OpenBSD prioritises absolute performance is besides the point - they benchmark against their own goals, not someone else's achievements.

freehorse 1y ago

Author discovers that performance does not scale proportionally to %CPU utilisation, and gets instead to the conclusion that %CPU utilisation is a lie. There are many reasons for the lack of a proportional relationship, even in the case where you do not have hyperthreading or downclocking (in which cases you just need to interpret %CPU utilisation in that context, rather than declare it "a lie"). Even in apple silicon where these are usually not an issue, you often do not get an exactly proportional scaling. There may be overheads when utilising multiple cores wrt how data is passed around, or resource bottlenecks other than CPU.

saagarjha 1y ago
Apple silicon downclocks quite a lot especially if you have a passively cooled machine
- freehorse 1y ago
  With the exception of macbook air that has passive cooling nothing as aggressive as "turbo" modes, and ime it is relatively hard to get to thermal limits just with cpu in general for the devices I have used. Most other manufacturers nowadays officially advertise boosted single core clock speeds that are much higher and lower when more cores are used at the same time. Thermal limits, in contrast, are much more circumstantial.

PathOfEclipse 1y ago

I think it was always a mistake to pretend hyperthreading doubles your core count. I always assumed it was just due to laziness; the operating system treats a hyperthreaded core as two "virtual cores" and schedules as two cores, so then every other piece of tooling sees double the number of actual cores. There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages. But, maybe that's hard to do given how the OS implements hyperthreading.

fluoridation 1y ago
>There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages On AMD, threads may as well be cores. If you take a Ryzen and disable SMT, you're basically halving its parallelism, at least for some tasks. On Intel you're just turning off an extra 10-20%.
- PathOfEclipse 1y ago
  Can you provide some links for this? A quick web search turns this up at near the top from 2024: https://www.techpowerup.com/review/amd-ryzen-9-9700x-performance-smt-disabled/22.html https://www.techpowerup.com/review/amd-ryzen-9-9700x-perform... The benchmarks show a 10% drop in "application" performance when SMT is disabled, but an overall 1-3% increase in performance for games. From a hardware perspective, I can't imagine how it could be physically possible to double performance by enabling SMT.
  - fluoridation 1y ago
    I don't. It's based off my own testing, not by disabling SMT, but by running either <core_count> or <thread_count> parallel threads. It was my own code, so it's possible code that uses SIMD more heavily will see a less-significant speed-up. It's also possible I just measured wrong; running Cargo on a directory with -j16 and -j32 takes 58 and 48 seconds respectively. >From a hardware perspective, I can't imagine how it could be physically possible to double performance by enabling SMT. It depends on which parts of the processor your code uses. SMT works by duplicating some but not all the components of each core, so a single core can work on multiple independent uops simultaneously. I don't know the specifics, but I can imagine ALU-type code (jumps, calls, movs, etc.) benefits more from SMT than very math-heavy code. That would explain why rustc saw a greater speedup than Cinebench, as compiler code is very twisty with not a lot of math.

tonymet 1y ago

I like his empirical approach to get to the root significance of the cpu %-age indicator. Software engineers and data analysts take discrete "data" measurements and statistics for granted. "data" / "stats" are only a report, and that report is often incorrect.

bdhcuidbebe 1y ago

Thats some strong words about not RTFM.

deleted 1y ago

[deleted]

morning-coffee 1y ago

The lie is that hyper thread "cores" are equal to real "cores". Maybe this is what happens when an over 20-year old technology (hack) becomes ubiquitous and gets forgotten about? (We have to rediscover why our performance measurements don't seem to make sense?) The other thing I think we have a hard time visualizing is that processor is only either executing (100%) or its waiting to execute (0%) and that happens over varying timescales... so trying to assign a % in between inherently means you're averaging over some arbitrary timescale...

codedokode 1y ago

A worse lie is memory usage reporting, I think in every major OS it is understated and misreported. In case with Linux, I wanted to know who is using memory, and tried to add PSS values for every process, I never got back the total memory usage. In case with Windows/Mac I judge by screenshot of their tools which show unrealistically small values. As for the article, the slowdown can be also caused by increased use of shared resources like caches, TLBs, branch predictors.

biggusdickus69 1y ago
The memory usage is interesting, where different kind of shared memory is obvious hard to visualize, just two values per process doesn’t say enough. Most users actually wants a list of ”what can I kill to make the computer faster”, I.e. they want an oracle (no pun) that knows how fast the computer will be if different processes are killed.