6 ms·

Isn't gettimeofday implemented with vDSO to avoid kernel context switching (and therefore, most of the overhead)? My understanding is that using tsc directly i

by sa46 1y ago

Isn't gettimeofday implemented with vDSO to avoid kernel context switching (and therefore, most of the overhead)? My understanding is that using tsc directly is tricky. The rate might not be constant, and the rate differs across cores. [1] [1]: https://www.pingcap.com/blog/how-we-trace-a-kv-database-with-less-than-5-percent-performance-impact/ https://www.pingcap.com/blog/how-we-trace-a-kv-database-with...

Dylan16807 1y ago
If you have something newer than a pentium 4 the rate will be constant. I'm not sure of the details for when cores end up with different numbers.
quotemstr 1y ago
Wizardly workarounds for broken APIs persist long after those APIs are fixed. People still avoid things like flock(2) because at one time NFS didn't handle file locking well. CLOCK_MONOTONIC_RAW is fine these days with the vDSO.
- denotational 1y ago
  Sadly GPFS still doesn’t support flock(2), so I still avoid it.
  - quotemstr 1y ago
    Doesn't it? https://sambaxp.org/archive-data-samba/sxp09/SambaXP2009-DATA/Henning_Henkel.pdf https://sambaxp.org/archive-data-samba/sxp09/SambaXP2009-DAT... It would be weird, even for AIX, to support POSIX byte range locks and not the much simpler flock.
    - denotational 1y ago
      It doesn't, at least on the version I have access to, as it is configured on that cluster. I’m using Linux rather than AIX. fcntl(2) locks are supported (as long as they aren't OFD), but flock(2) locks don't work across nodes.
toast0 1y ago
I think most current systems have invariant tsc, I skimmed your article and was surprised to see an offset (but not totally shocked), but the rate looked the same. You could cpu pin the thread that's reading the tsc, except you can't pin threads in OpenBSD :p
- wahern 1y ago
  But just to be clear (for others), you don't need to do that because using RDTSC/RDTSCP is exactly how gettimeofday and clock_gettime work these days, even on OpenBSD. Where using the TSC is practical and reliable, the optimization is already there. OpenBSD actually only implemented this optimization relatively recently. Though most TSCs will be invariant, they still need to be trained across cores, and there are other minutiae (sleeping states?) that made it a PITA to implement in a reliable way, and OpenBSD doesn't have as much manpower as Linux. Some of those non-obvious issues would be relevant to someone trying to do this manually, unless they could rely on their specific hardware behavior.
  - RossBencina 1y ago
    Out of interest, does training across cores result in any residual offset? If so, is the offset nondeterministic?
    - wahern 1y ago
      I was curious myself, poked around, and found some references. But I'm still woefully incapable of answering that with any confidence and don't want to risk saying anything misleading, so here's the code and some other breadcrumbs: 1. Apparently OpenBSD gave up on trying to fix desync'd TSCs. See https://github.com/openbsd/src/commit/78156938567f79506a923cf635bd525907207a76 https://github.com/openbsd/src/commit/78156938567f79506a923c... 2. Relevant OpenBSD kernel code: https://github.com/openbsd/src/blob/master/sys/arch/amd64/amd64/tsc.c#L346 https://github.com/openbsd/src/blob/master/sys/arch/amd64/am... 3. Relevant Linux kernel code: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc_sync.c https://github.com/torvalds/linux/blob/master/arch/x86/kerne..., https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc.c https://github.com/torvalds/linux/blob/master/arch/x86/kerne... 4. Linux kernel doc (out-of-date?): https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt https://www.kernel.org/doc/Documentation/virtual/kvm/timekee... 5. Detailed SUSE blog post with many links: https://www.suse.com/c/cpu-isolation-nohz_full-troubleshooting-tsc-clocksource-by-suse-labs-part-6/ https://www.suse.com/c/cpu-isolation-nohz_full-troubleshooti... 6. Linux patch (uncommitted?) to attempt to directly sync TSCs: https://lkml.rescloud.iu.edu/2208.1/00313.html https://lkml.rescloud.iu.edu/2208.1/00313.html
triknomeister 1y ago
TSC is about cycles consumed by a core. Not about actual time. And so for microbenchmarking, it actually makes sense, because you are often much more interested in CPU benchmarks than network benchmarks in microbenchmarking.
ainiriand 1y ago
You have to benchmark tsc against a fixed CPU speed, say 1000Mhz, then you have a reliable comparison.
- deleted 1y ago
  [deleted]
tonyarkles 1y ago
It was a while ago (2009-10ish) but I ran into an exceptionally interesting performance issue that was partly identified with RDTSC. For a course project in grad school I was measuring the effects of the Python GIL when running multi-threaded Python code on multi-core processors. I expected the overhead/lock contention to get worse as I added threads/cores but the performance fell off a cliff in a way that I hadn't expected. Great outcome for a course project, it made the presentation way more interesting. The issue ended up being that my multi-threaded code when running on a single core pinned that core at 100% CPU usage, as expected, but when running it across 4 cores it was running 4 cores at 25% usage each. This resulted in the clock governor turning down the frequency on the cores from ~2GHz to 900MHz and causing the execution speed to drop even worse than just the expected lock contention. It was a fun mystery to dig into for a while.

by sa46 1y ago

Dylan16807 1y ago

If you have something newer than a pentium 4 the rate will be constant. I'm not sure of the details for when cores end up with different numbers.

quotemstr 1y ago

Wizardly workarounds for broken APIs persist long after those APIs are fixed. People still avoid things like flock(2) because at one time NFS didn't handle file locking well. CLOCK_MONOTONIC_RAW is fine these days with the vDSO.

denotational 1y ago
Sadly GPFS still doesn’t support flock(2), so I still avoid it.
- quotemstr 1y ago
  Doesn't it? https://sambaxp.org/archive-data-samba/sxp09/SambaXP2009-DATA/Henning_Henkel.pdf https://sambaxp.org/archive-data-samba/sxp09/SambaXP2009-DAT... It would be weird, even for AIX, to support POSIX byte range locks and not the much simpler flock.
  - denotational 1y ago
    It doesn't, at least on the version I have access to, as it is configured on that cluster. I’m using Linux rather than AIX. fcntl(2) locks are supported (as long as they aren't OFD), but flock(2) locks don't work across nodes.

toast0 1y ago

I think most current systems have invariant tsc, I skimmed your article and was surprised to see an offset (but not totally shocked), but the rate looked the same. You could cpu pin the thread that's reading the tsc, except you can't pin threads in OpenBSD :p

wahern 1y ago
But just to be clear (for others), you don't need to do that because using RDTSC/RDTSCP is exactly how gettimeofday and clock_gettime work these days, even on OpenBSD. Where using the TSC is practical and reliable, the optimization is already there. OpenBSD actually only implemented this optimization relatively recently. Though most TSCs will be invariant, they still need to be trained across cores, and there are other minutiae (sleeping states?) that made it a PITA to implement in a reliable way, and OpenBSD doesn't have as much manpower as Linux. Some of those non-obvious issues would be relevant to someone trying to do this manually, unless they could rely on their specific hardware behavior.
- RossBencina 1y ago
  Out of interest, does training across cores result in any residual offset? If so, is the offset nondeterministic?
  - wahern 1y ago
    I was curious myself, poked around, and found some references. But I'm still woefully incapable of answering that with any confidence and don't want to risk saying anything misleading, so here's the code and some other breadcrumbs: 1. Apparently OpenBSD gave up on trying to fix desync'd TSCs. See https://github.com/openbsd/src/commit/78156938567f79506a923cf635bd525907207a76 https://github.com/openbsd/src/commit/78156938567f79506a923c... 2. Relevant OpenBSD kernel code: https://github.com/openbsd/src/blob/master/sys/arch/amd64/amd64/tsc.c#L346 https://github.com/openbsd/src/blob/master/sys/arch/amd64/am... 3. Relevant Linux kernel code: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc_sync.c https://github.com/torvalds/linux/blob/master/arch/x86/kerne..., https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc.c https://github.com/torvalds/linux/blob/master/arch/x86/kerne... 4. Linux kernel doc (out-of-date?): https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt https://www.kernel.org/doc/Documentation/virtual/kvm/timekee... 5. Detailed SUSE blog post with many links: https://www.suse.com/c/cpu-isolation-nohz_full-troubleshooting-tsc-clocksource-by-suse-labs-part-6/ https://www.suse.com/c/cpu-isolation-nohz_full-troubleshooti... 6. Linux patch (uncommitted?) to attempt to directly sync TSCs: https://lkml.rescloud.iu.edu/2208.1/00313.html https://lkml.rescloud.iu.edu/2208.1/00313.html

triknomeister 1y ago

TSC is about cycles consumed by a core. Not about actual time. And so for microbenchmarking, it actually makes sense, because you are often much more interested in CPU benchmarks than network benchmarks in microbenchmarking.

ainiriand 1y ago

You have to benchmark tsc against a fixed CPU speed, say 1000Mhz, then you have a reliable comparison.

deleted 1y ago
[deleted]

tonyarkles 1y ago

It was a while ago (2009-10ish) but I ran into an exceptionally interesting performance issue that was partly identified with RDTSC. For a course project in grad school I was measuring the effects of the Python GIL when running multi-threaded Python code on multi-core processors. I expected the overhead/lock contention to get worse as I added threads/cores but the performance fell off a cliff in a way that I hadn't expected. Great outcome for a course project, it made the presentation way more interesting. The issue ended up being that my multi-threaded code when running on a single core pinned that core at 100% CPU usage, as expected, but when running it across 4 cores it was running 4 cores at 25% usage each. This resulted in the clock governor turning down the frequency on the cores from ~2GHz to 900MHz and causing the execution speed to drop even worse than just the expected lock contention. It was a fun mystery to dig into for a while.