5 ms·

Maybe we don't need denormals in most cases. Denormals are extremely tiny numbers (on order of 2^-149). It would be nice if RISC-V which aims for simpliciy, got

Maybe we don't need denormals in most cases. Denormals are extremely tiny numbers (on order of 2^-149). It would be nice if RISC-V which aims for simpliciy, got rid of them. If you need such small numbers, I don't know, just use doubles or write your own floats.

adrian_b 4mo ago
Denormals are not needed if you are willing to handle underflow exceptions. Before Intel 8087 and the IEEE 754 standard, any decent floating-point unit generated overflow exceptions and underflow exceptions, which had to be handled by the programmer, unless the default behavior of crashing the program was acceptable. Intel 8087 and the standard based on it have offered to the lazy programmers the option to not handle the exceptions, in which case overflow exceptions generate infinities and the underflow exceptions generate denormals. When the exceptions are not handled, it is supposed that the programmer will check the final results of a long computation, and if infinities and denormals are not desired, but they exist nonetheless in the results, the programmer will investigate the reason and then the bug will be fixed. So anyone is free to ensure that no denormals will ever appear in an application , by enabling the underflow exception. If it is desired that the program must not crash, then the program must be written carefully, so that underflows are impossible. There is no correct way of eliminating denormals, except throwing exceptions on underflows. The flush denormals to zero on output and interpret denormals as zero on input behaviors are not permissible in any program that must produce correct results. Anyone who uses -ffast-math or similar options for compiling a program that is not intended for graphics or ML/AI, where errors supposedly do not matter, makes an unforgivable mistake. Unfortunately, "-ffast-math" enables a very large number of compilation options. A part of them are safe and they can cause a great increase in performance, like using fused multiply-add instructions. Others not only are dangerous, like flushing denormals to zero, but they also provide negligible performance increase on many processors. Therefore, instead of aggregate options like "-ffast-math" one must enable only some of the component options, for maximum performance, without affecting result accuracy. For example, in gcc and clang one must use "-ffp-contract=fast", for enabling FMA instructions.
- rwallace 4mo ago
  The one way I know of for denormals to arise in real calculations is when you have a process that converges on zero. In which case, values will pass through the denormal range on the way from the normal range down to zero. And converting denormals to zero is obviously correct. What other cases have you seen of denormals arising in real calculations?
  - adrian_b 4mo ago
    If you compute a limit numerically, the decision that you have reached the limit must normally be taken long before there is any possibility of underflow, i.e. of generating denormals. Typically you compute the difference between 2 adjacent terms of the convergent sequence and you decide that you have reached the limit when adding the difference to the current term to get the next does not change it (or when the relative error is smaller than some threshold). At this time the difference will still be many orders of magnitude greater than a denormal. If the limit happens to be zero, then what you describe can happen. The programs where this can happen normally combine two different criteria to decide that the limit has been reached, i.e. that either the relative error is small enough, i.e. like I said that adding the difference to the previous term does not change it, or that the absolute error is small enough, i.e. that the difference is smaller than some threshold. The absolute error used as threshold is normally chosen based on what is physically meaningful in that problem, i.e. depending on what kind of physical quantity corresponds to the values of the terms of the convergent series. An example of applications where the users must configure both relative errors and absolute errors, to be used as criteria of convergence, are the SPICE-like programs used to simulate electronic circuits, where the user must configure both a generally-applicable relative error, and absolute errors for different kinds of physical quantities, e.g. an absolute error for voltages and an absolute error for electric currents, which will be used, respectively, when a sequence of voltages converges towards zero or a sequence of currents converges towards zero, so that the absolute error criterion will be satisfied before the relative error criterion is satisfied. In any case, in a correct program the attempt to find the limit should always stop before underflows become possible, so denormals should never be generated if the underflow exceptions are masked. Denormals can appear in a lot of cases when almost equal values are subtracted, e.g. in the solution of many kinds of badly conditioned equations, but in most such cases there may exist alternative formulae that avoid underflows, i.e. the generation of denormals. In general, when denormals appear, this signals bugs in the program, which must be investigated and fixed. The purpose of denormals is to allow the programmer to not fix the bugs, without causing catastrophic errors, like those that can happen when underflows generate null results, i.e. when "denormals are flushed to zero". Careful programmers should nonetheless fix any bugs that generate denormals.

adrian_b 4mo ago

Denormals are not needed if you are willing to handle underflow exceptions. Before Intel 8087 and the IEEE 754 standard, any decent floating-point unit generated overflow exceptions and underflow exceptions, which had to be handled by the programmer, unless the default behavior of crashing the program was acceptable. Intel 8087 and the standard based on it have offered to the lazy programmers the option to not handle the exceptions, in which case overflow exceptions generate infinities and the underflow exceptions generate denormals. When the exceptions are not handled, it is supposed that the programmer will check the final results of a long computation, and if infinities and denormals are not desired, but they exist nonetheless in the results, the programmer will investigate the reason and then the bug will be fixed. So anyone is free to ensure that no denormals will ever appear in an application , by enabling the underflow exception. If it is desired that the program must not crash, then the program must be written carefully, so that underflows are impossible. There is no correct way of eliminating denormals, except throwing exceptions on underflows. The flush denormals to zero on output and interpret denormals as zero on input behaviors are not permissible in any program that must produce correct results. Anyone who uses -ffast-math or similar options for compiling a program that is not intended for graphics or ML/AI, where errors supposedly do not matter, makes an unforgivable mistake. Unfortunately, "-ffast-math" enables a very large number of compilation options. A part of them are safe and they can cause a great increase in performance, like using fused multiply-add instructions. Others not only are dangerous, like flushing denormals to zero, but they also provide negligible performance increase on many processors. Therefore, instead of aggregate options like "-ffast-math" one must enable only some of the component options, for maximum performance, without affecting result accuracy. For example, in gcc and clang one must use "-ffp-contract=fast", for enabling FMA instructions.

rwallace 4mo ago
The one way I know of for denormals to arise in real calculations is when you have a process that converges on zero. In which case, values will pass through the denormal range on the way from the normal range down to zero. And converting denormals to zero is obviously correct. What other cases have you seen of denormals arising in real calculations?
- adrian_b 4mo ago
  If you compute a limit numerically, the decision that you have reached the limit must normally be taken long before there is any possibility of underflow, i.e. of generating denormals. Typically you compute the difference between 2 adjacent terms of the convergent sequence and you decide that you have reached the limit when adding the difference to the current term to get the next does not change it (or when the relative error is smaller than some threshold). At this time the difference will still be many orders of magnitude greater than a denormal. If the limit happens to be zero, then what you describe can happen. The programs where this can happen normally combine two different criteria to decide that the limit has been reached, i.e. that either the relative error is small enough, i.e. like I said that adding the difference to the previous term does not change it, or that the absolute error is small enough, i.e. that the difference is smaller than some threshold. The absolute error used as threshold is normally chosen based on what is physically meaningful in that problem, i.e. depending on what kind of physical quantity corresponds to the values of the terms of the convergent series. An example of applications where the users must configure both relative errors and absolute errors, to be used as criteria of convergence, are the SPICE-like programs used to simulate electronic circuits, where the user must configure both a generally-applicable relative error, and absolute errors for different kinds of physical quantities, e.g. an absolute error for voltages and an absolute error for electric currents, which will be used, respectively, when a sequence of voltages converges towards zero or a sequence of currents converges towards zero, so that the absolute error criterion will be satisfied before the relative error criterion is satisfied. In any case, in a correct program the attempt to find the limit should always stop before underflows become possible, so denormals should never be generated if the underflow exceptions are masked. Denormals can appear in a lot of cases when almost equal values are subtracted, e.g. in the solution of many kinds of badly conditioned equations, but in most such cases there may exist alternative formulae that avoid underflows, i.e. the generation of denormals. In general, when denormals appear, this signals bugs in the program, which must be investigated and fixed. The purpose of denormals is to allow the programmer to not fix the bugs, without causing catastrophic errors, like those that can happen when underflows generate null results, i.e. when "denormals are flushed to zero". Careful programmers should nonetheless fix any bugs that generate denormals.