7 ms·

Author here. After my last post about kernel bugs, I spent some time looking at how the JVM reports its own thread activity. It turns out that "What is the CPU

Author here. After my last post about kernel bugs, I spent some time looking at how the JVM reports its own thread activity. It turns out that "What is the CPU time of this thread?" is/was a much more expensive question than it should be.

Neywiny 8mo ago
Did you look into the large spread on your distributions? Some of these span multiple orders of magnitude which is interesting
- jerrinot 8mo ago
  Fair point. These were run on a standard dev workstation under load, which may account for the noise. I haven't done a deep dive into the outliers yet, but the distribution definitely warrants a more isolated look.
jacquesm 8mo ago
I don't think it is possible to talk about fractions of nanoseconds without having an extremely good idea of the stability and accuracy of your clock. At best I think you could claim there is some kind of reduction but it is super hard to make such claims in the absolute without doing a massive amount of prep work to ensure that the measured times themselves are indeed accurate. You could be off by a large fraction and never know the difference. So unless there is a hidden atomic clock involved somewhere in these measurements I think they should be qualified somehow.
- rcxdude 8mo ago
  Stability and accuracy, when applied to clocks, are generally about dynamic range, i.e. how good is the scale with which you are measuring time. So if you're talking about nanoseconds across a long time period, seconds or longer, then yeah, you probably should care about your clock. But when you're measuring nanoseconds out of a millisecond or microsecond, it really doesn't matter that much and you're going to be OK with the average crystal oscillator in a PC. (and if you're measuring a 10% difference like in the article, you're going to be fine with a mechanical clock as your reference if you can do the operation a billion times in a row).
  - jacquesm 8mo ago
    This setup is a user space program on a machine that is not exclusively dedicated to the test running all kinds of interrupts (and other tasks) left, right and center through the software under test.
    - loeg 8mo ago
      For something like this, you can just take several trials and look at the minimum observed time, which is when there will have been ~no interruptions. https://github.com/facebook/folly/blob/main/folly/docs/Benchmark.md#a-look-under-the-hood https://github.com/facebook/folly/blob/main/folly/docs/Bench...
      - jacquesm 8mo ago
        You don't actually know that for sure. You have only placed a new upper bound.
        loeg 8mo ago
        This seems like more of a philosophical argument than a practical one.
        jacquesm 8mo ago
        No, it is a very practical one and I'm actually surprised that you don't see it that way. Benchmarking is hard, and if you don't understand the basics then you can easily measure nonsense.
        jerrinot 8mo ago
        You raise a fair point about the percentiles. Those are reported as point estimates without confidence intervals and the implied precision overstates what system clock can deliver. The mean does get proper statistical treatment (t-distribution confidence interval), but you're right that JMH doesn't compute confidence intervals for percentiles. Reporting p0.00 with three significant figures is ... optimistic. That said I think the core finding survives this critique. The improvement shows up consistently across ~11 million samples at every percentile from p0.50 through p0.999.
6r17 8mo ago
Very thankful for the 1liner tldr edit : I had an afterthought about this because it ended up being a low quality comment ; Bringing up such TLDR give a lot of value to reading content, especially on HN, as it provides way more inertia and let focus on - reading this short form felt like that cool friend who gave you a heads up.
- jerrinot 8mo ago
  I was unsure whether to post it or not so I am glad you found it useful!
  - 6r17 8mo ago
    I have that 10-30s time window to fill when claude might be loading some stuff ; the 1 liner is exactly what fits in that window - it makes me wonder about the original idea of twitter now that I think of it - but since it's not the same kind of content I don't bother with it.It really feels like "here is the stuff, here's more about it if you want to" - really really appreciate that form and will definitely do the same format myself
abicklefitch 8mo ago
[dead]

Neywiny 8mo ago

Did you look into the large spread on your distributions? Some of these span multiple orders of magnitude which is interesting

jerrinot 8mo ago
Fair point. These were run on a standard dev workstation under load, which may account for the noise. I haven't done a deep dive into the outliers yet, but the distribution definitely warrants a more isolated look.

jacquesm 8mo ago

I don't think it is possible to talk about fractions of nanoseconds without having an extremely good idea of the stability and accuracy of your clock. At best I think you could claim there is some kind of reduction but it is super hard to make such claims in the absolute without doing a massive amount of prep work to ensure that the measured times themselves are indeed accurate. You could be off by a large fraction and never know the difference. So unless there is a hidden atomic clock involved somewhere in these measurements I think they should be qualified somehow.

rcxdude 8mo ago
Stability and accuracy, when applied to clocks, are generally about dynamic range, i.e. how good is the scale with which you are measuring time. So if you're talking about nanoseconds across a long time period, seconds or longer, then yeah, you probably should care about your clock. But when you're measuring nanoseconds out of a millisecond or microsecond, it really doesn't matter that much and you're going to be OK with the average crystal oscillator in a PC. (and if you're measuring a 10% difference like in the article, you're going to be fine with a mechanical clock as your reference if you can do the operation a billion times in a row).
- jacquesm 8mo ago
  This setup is a user space program on a machine that is not exclusively dedicated to the test running all kinds of interrupts (and other tasks) left, right and center through the software under test.
  - loeg 8mo ago
    For something like this, you can just take several trials and look at the minimum observed time, which is when there will have been ~no interruptions. https://github.com/facebook/folly/blob/main/folly/docs/Benchmark.md#a-look-under-the-hood https://github.com/facebook/folly/blob/main/folly/docs/Bench...
    - jacquesm 8mo ago
      You don't actually know that for sure. You have only placed a new upper bound.
      - loeg 8mo ago
        This seems like more of a philosophical argument than a practical one.
        jacquesm 8mo ago
        No, it is a very practical one and I'm actually surprised that you don't see it that way. Benchmarking is hard, and if you don't understand the basics then you can easily measure nonsense.
        jerrinot 8mo ago
        You raise a fair point about the percentiles. Those are reported as point estimates without confidence intervals and the implied precision overstates what system clock can deliver. The mean does get proper statistical treatment (t-distribution confidence interval), but you're right that JMH doesn't compute confidence intervals for percentiles. Reporting p0.00 with three significant figures is ... optimistic. That said I think the core finding survives this critique. The improvement shows up consistently across ~11 million samples at every percentile from p0.50 through p0.999.

6r17 8mo ago

Very thankful for the 1liner tldr edit : I had an afterthought about this because it ended up being a low quality comment ; Bringing up such TLDR give a lot of value to reading content, especially on HN, as it provides way more inertia and let focus on - reading this short form felt like that cool friend who gave you a heads up.

jerrinot 8mo ago
I was unsure whether to post it or not so I am glad you found it useful!
- 6r17 8mo ago
  I have that 10-30s time window to fill when claude might be loading some stuff ; the 1 liner is exactly what fits in that window - it makes me wonder about the original idea of twitter now that I think of it - but since it's not the same kind of content I don't bother with it.It really feels like "here is the stuff, here's more about it if you want to" - really really appreciate that form and will definitely do the same format myself

abicklefitch 8mo ago

[dead]