This paper is cited by "Taming Performance Variability" but is probably worth a read if you haven't read it before and find this paper interesting: https://www.

This paper is cited by "Taming Performance Variability" but is probably worth a read if you haven't read it before and find this paper interesting: https://www.seas.upenn.edu/~cis501/papers/producing-wrong-data.pdf https://www.seas.upenn.edu/~cis501/papers/producing-wrong-da... The authors of "Producing Wrong Data Without Doing Anything Obviously Wrong" find a bunch of "optimizations" that don't do anything but change performance by some percentage points. I think about this every time someone says big companies care about "sub percentage points" optimizations - how much of that is in the noise? Work like these papers ask these questions, which I think is good. Computer scientists seem to have a worse time with empirical evaluation, statistics, and experimental design. Not that any field of science does super good at this all the time, but CS seems to have this as an "unknown unknown" deficiency.