5 ms·
GELU really is like magic: UNARY(GELU, b / 2 * (1 + tanh(.7978845 * (b + .044715 * b * b * b))))
by go_prodev 2y ago
GELU really is like magic:
UNARY(GELU, b / 2 * (1 + tanh(.7978845 * (b + .044715 * b * b * b))))
- dogboat 2y agoFast inverse square root lookalike.
- shoo 2y agoYou can hand that GELU definition to a mathematician and they can interpret it as a function of a real number b. The definition does not depend upon b being a floating-point number with a particular bit representation. In contrast, the fast inverse square root really exploits the bit representation of a floating point input to cheaply compute an initial guess.
- jey 2y agoThis is just a practical approximation to the actual mathematical definition of GELU, which is `GELU(x) := x * Φ(x)` where Φ(x) is the CDF of the Gaussian distribution.
- fluoridation 2y agoIsn't that just erf()?
- kgwgk 2y agoThey are related but the error function approaches -1 for large negative numbers. Φ(x) approaches 0 and so does x * Φ(x).
- deleted 2y ago[deleted]