7 ms·
Nowadays, all major platforms follow IEEE754, which specifies exactly the results of elementary operations (+, -, x, /). You must get bit-for-bit reproducible r
by thxg 5y ago
Nowadays, all major platforms follow IEEE754, which specifies exactly the results of elementary operations (+, -, x, /). You must get bit-for-bit reproducible results across different runs, compiles, libraries, OSes and hardware. Any difference is a bug.
However:
1. As the paper emphasizes, math library functions (sin, cos, log, etc.) are not covered by that standardization, at least not mandatorily and not in practice. This is the problem that the paper tackles.
2. In the bad old days of the x87, the FPU carried out computations on eight 80-bit registers. The problem is that whenever those registers spilled out to the stack, precision was truncated down to whatever type the language actually meant (64-bit double or 32-bit float). Because the register-spilling decision is mostly taken by the compiler, it was largely unpredictable. This resulted in code that was not reproducible, even across compiles! Since x87 was deprecated in favor of SSE/AVX, the precision of operations is always that of the type used in the code. Lower than 80-bits but much better in practice because of consistency and reproducibility.
3. The bad old days are making a comeback with FMA instructions (fused multiply-add). This is a single operation (a x b + c) and so it is correctly rounded to the closest representable number. This number can be different from ((a x b) + c) in which two roundings occur. If we give the compiler freedom to decide whether or not to fuse such expressions, we will again see non-reproducibility across compiles. Unfortunately, gcc and icc both do that by default at -O3 (see -ffast-math, and more specifically -fexcess-precision). Specifying a proper language standard (for example -std=c99, -std=c++14, etc.) restores sane behavior.
- adgjlsfhk1 5y agoOn the one hand, the ambiguity of FMA is kind of annoying, but on the other, FMA is a godsend for writing accurate arithmatic. One reason for this, is that the error of `ab` is `fma(a, b, -ab)` which makes compensated algorithms much faster to write. Secondly, polynomial evaluations using horner's method converge to .5 ULP (units in last place) with FMA, but to 1.5 ULP without. This is incredibly useful since polynomial approximations are at the heart of pretty much everything else. I think Julia takes an interesting approach here of never reordering floating point computations unless you explecitly give permission using @fastmath (which acts locally instead of globally like a --fast-math in C/C++). This makes it much easier to use ieee specific tricks and be confident that the compiler won't mess with you, while still making it easy to tell the compiler to do what it wants.
- thxg 5y ago> FMA is a godsend for writing accurate arithmatic. True, and the small performance boost (on some platforms) is nice too > never reordering floating point computations unless you explecitly give permission Yes, requiring explicit code for FMA and arithmetic grouping/reordering in general seems like the sane approach.
- titzer 5y ago> I think Julia takes an interesting approach here of never reordering floating point computations WebAssembly as well. IEEE-754 is mandated and there are no unsafe compiler optimizations allowed. This is the only sane choice. It's just too hard to reason about programs otherwise. Nothing about a super-intelligent machine completely reorganizing your code in a way that subtly breaks it is good, IMHO.
- pklausler 5y agox87 can store and load full 80-bit register values. The truncation you mention is optional.
- jcelerier 5y ago> 2. In the bad old days of the x87, the FPU carried out computations on eight 80-bit registers. The problem is that whenever those registers spilled out to the stack, precision was truncated down to whatever type the language actually meant (64-bit double or 32-bit float). Because the register-spilling decision is mostly taken by the compiler, it was largely unpredictable. This resulted in code that was not reproducible, even across compiles! Since x87 was deprecated in favor of SSE/AVX, the precision of operations is always that of the type used in the code. Lower than 80-bits but much better in practice because of consistency and reproducibility. even today it's still possible to end up with things in the x87 registers, I don't know of a compiler toggle to entirely and unequivocally disable it.
- ant6n 5y agoThat seems like a fault of the language/compiler, not the 80but float. There are operations for storing the 80bit float in full precision, and I think Pascal even used real=80bit.
- thxg 5y agoCode using "long double" on x86_64 will compile to x87 instructions (at least with GCC on Linux). This can be triggered unintentionally. For example, string-to-double conversions in Boost use long doubles. (I have been bitten by that when using valgrind: Valgrind emulates x87 80-bit registers using 64-bit doubles. As a result, some Boost-using code will have a different behavior under valgrind.)
- camgunz 5y agoYup, exactly. You can imagine that a lot of collision code uses tan for example.