6 ms·
Fast inverse square root lookalike.
by dogboat 2y ago
Fast inverse square root lookalike.
- shoo 2y agoYou can hand that GELU definition to a mathematician and they can interpret it as a function of a real number b. The definition does not depend upon b being a floating-point number with a particular bit representation. In contrast, the fast inverse square root really exploits the bit representation of a floating point input to cheaply compute an initial guess.