8 ms·
On gcc 13, the difference in assembly between the min(max()) version and std::clamp is eliminated when I add the -ffast-math flag. I suspect that the two implem
by celegans25 3y ago
On gcc 13, the difference in assembly between the min(max()) version and std::clamp is eliminated when I add the -ffast-math flag. I suspect that the two implementations handle one of the arguments being NaN a bit differently.
https://gcc.godbolt.org/z/fGaP6roe9 https://gcc.godbolt.org/z/fGaP6roe9
I see the same behavior on clang 17 as well
https://gcc.godbolt.org/z/6jvnoxWhb https://gcc.godbolt.org/z/6jvnoxWhb
- gumby 3y agoYou (celegans25) probably know this but here is a PSA that -ffast-math is really -finaccurate-math. The knowledgeable developer will know when to use it (almost never) while the naive user will have bugs.
- cogman10 3y agoEhh, not so much inaccurate, more of a "floating point numbers are tricky, let's act like they aren't". Compilers are pretty skittish about changing the order of floating point operations (for good reason) and ffast-math is the thing that lets them transform equations to try and generate faster code. IE, instead of doing "n / 10" doing "n * 0.1". The issue, of course, being that things like 0.1 can't be perfectly represented with floats but 100 / 10 can be. So now you've introduced a tiny bit of error where it might not have existed.
- phkahler 3y agoI've never understood why generating exceptions is preferable to just using higher precision.
- gumby 3y agoHigher precision isn’t always available. IEEE 754 is an unusually well-thought-through standard (thanks to some smart people with a lot of painful experience) and is pretty good at justifying its decisions, some of which are surprising (far from obvious) to anyone not steeped in it.
- cogman10 3y agoThe main problem with floats in general is they are designed primarily for scientific computing. We are fortunately starting to see newer (well, not that new now) CPU instructions like FMA that make more accurate decimal representations not take such huge performance hits.
- adgjlsfhk1 3y agowhat does fma have to do with decimal numbers?
- cogman10 3y agoOh shoot, nvm, I thought it was an optimization for integers. Really it'll be the SIMD style instructions that speeds things up.
- dahart 3y agoOn a GPU, higher precision can cost between 2 and 64 times more than single precision, with typical ratios for consumer cards being 16 or 32. Even on the CPU, fp64 workloads tend to run at half the speed on real data due to the extra bandwidth needed for higher precision.
- rwmj 3y agoIt isn't just that. -ffast-math also allows the compiler to ignore infinites. In fact for GCC with -ffast-math, isinf always returns false. Something similar happens for NaNs/isnan.
- cogman10 3y agoI lump this into "floating points are tricky". NaNs and inf are definitely legitimate floating point values. They are also things that a lot of applications will break on they ever encounter them.
- alexey-salmin 3y agoIf your code ventures into the domain where fast-math matters and you're not a mathematician trying to solve a lyapunov-unstable problem with very tricky numeric methods, then most likely your code is already broken.
- dahart 3y agoWhy do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all. I deal with a lot of floating point professionally day to day, and I use fast math all the time, since the tradeoff for higher performance and the relatively small loss of accuracy are acceptable. Maybe the biggest issue I run into is lack of denorms with CUDA fast-math, and it’s pretty rare for me to care about numbers smaller than 10^-38. Heck, I’d say I can tolerate 8 or 16 bits of mantissa most of the time, and fast-math floats are way more accurate than that. And we know a lot of neural network training these days can tolerate less than 8 bits of mantissa.
- mort96 3y agoThe scary thing IMO is: your code might be fine with unsafe math optimisations, but maybe you're using a library which is written to do operations in a certain order to minimise numerical error, and unsafe math operations changes the code which are mathematically equivalent but which results in many orders of magnitude more numerical error. It's probably fine most of the time, but it's kinda scary.
- deleted 3y ago[deleted]
- dahart 3y agoIt shouldn’t be scary. Any library that is sensitive to order of operations will hopefully have a big fat warning on it. And it can be compiled separately with fast-math disabled. I don’t know of any such libraries off the top of my head, and it’s quite rare to find situations that result in orders of magnitude more error, though I grant you it can happen, and it can be contrived pretty easily.
- planede 3y agoYou can't fully disable fast-math per-library, moreover a library compiled with fast-math might also introduce inaccuracies in a seemingly unrelated library or application code in the same executable. The reason is that fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.
- mort96 3y agoWhat you really should enable is the fun and safe math optimizations, with -funsafe-math-optimizations.
- aqfamnzc 3y agoI know almost nothing about compiler flags but I got a laugh out of this even though I still don't know if you're joking or not. Edit: Just read it again and now I understand the joke. Haha
- kevincox 3y agoTo others `-f` is a common prefix for GCC flags. You can think of this as "enable feature". So -funsafe-math-operations should be read as (-f) (unsafe-math-operations). Not (-)(funsafe-math-operations).
- arcticbull 3y agoI kind of like the idea the flag is sarcastically calling them very fun and very safe.
- dekhn 3y agodon't forget libiberty which is linked in using -liberty (and freedom for all)
- planede 3y agoAnother PSA is that dynamic libraries compiled with fast-math will also introduce inaccuracies in unrelated libraries in the same executable, as they introduce dynamic initialization that globally changes the floating point environment.
- pavlov 3y agoThis would only affect code that uses the old-school x87 floating point instructions, though? The x87 FPU unit indeed has scary global state that can make your doubles behave like floats in secret and silence. I would think practically all modern FPU code on x86-64 would be using the SIMD registers which have explicit widths.
- borodi 3y agoSo it was a bit more pervasive than this, the issue was that flushing subnormals (values very close to 0) to 0 is a register that gets set, so if a library is built with the fastmath flags and it gets loaded, it sets the register, causing the whole process to flush it's subnormals. i.e https://github.com/llvm/llvm-project/issues/57589 https://github.com/llvm/llvm-project/issues/57589
- jcranmer 3y ago> This would only affect code that uses the old-school x87 floating point instructions, though? Actually, no, the x87 FPU instructions are the only ones that won't be affected. It sets the FTZ/DAZ bits, which exist for SSE instructions but not x87 instructions.
- mhh__ 3y agoYou're mistaking something else for the rounding mode and subnormal handling flags.
- deleted 3y ago[deleted]
- mhh__ 3y agoOne of the things that you can do with D and as far as I know Julia is enable specific optimizations locally e.g. allow FMAs here and there, not globally. fast-math is one of the dumbest things we have as an industry IMO.
- ChrisRackauckas 3y agoTotally agreed. In Julia we use https://github.com/SciML/MuladdMacro.jl https://github.com/SciML/MuladdMacro.jl all over the place so that way it's contextual and does not bleed into other functions. fast-math changing everything is just... dangerous.
- deleted 3y ago[deleted]