6 ms·

> Most autodiff packages (such as Pytorch) use something not much more advanced than this pytorch absolutely does not use the dual number formulation - there a

> Most autodiff packages (such as Pytorch) use something not much more advanced than this pytorch absolutely does not use the dual number formulation - there are absolutely no magic epsilons anywhere in pytorch's (or tensorflow's) code base. what you're calling duals are the adjoints where are indeed stored/cached on every node in pytorch graphs. there's a reason no one uses dual numbers (non-standard analysis) for anything (neither autodiff nor calculus itself): because manipulating infinitesmals like this is fraught formal manipulation (it's algebra...) where as limits are much more rigorous (bounds, inequalities, convergence, etc.). my favorite question to ask the non-standard analysis n00bs is: please tell me under what conditions this is true (dx/dy)(dy/dz)(dz/dx) = 1 edit: anyone that thinks i'm wrong and this other guy is right should go and do some reading, eg where this guy tried to make this same point and got shot down: https://math.stackexchange.com/a/341550 https://math.stackexchange.com/a/341550 spoiler alert: there's a reason you had to learn epsilon-delta proofs and limits and it's not because your math professors are mean. this is why i hate this kind of "TIL, gee whiz" math tidbits - they're full of exclamation marks and fancy sounding words ("non-archimedean rings" oooo fancy) but almost always come from a wikipedia level understanding, not actual research.

MikeBattaglia 2y ago
At the end of the day, if you are storing inputs and outputs to a function as a pair of numbers - one for the actual value, and one for the derivative - and if addition and multiplication work the way you expect and propagate derivatives correctly - then you are using dual numbers, regardless of if you notate it a + b*h or {"value": a, "derivative": b}. Pytorch does things slightly differently in that it is mostly focused on reverse-mode autodiff, and so it stores adjoints relative to the overall output rather than partial derivatives relative to the input, but this isn't really an entirely different thing, in the same way that the FFT isn't entirely different from the DFT. There seems to be some confusion about the relationship between dual numbers and smooth infinitesimal analysis. Both have nilpotent elements, but with dual numbers the background logic is classical, whereas it isn't with smooth infinitesimal analysis. EDIT: I see you've edited your post to try to get in some extra criticism after I've already responded. That's terrible form, so I'll just respond here. Dual numbers are a nice way to get started with forward-mode autodiff, to which it is so related that the two are essentially the same thing with different labels. Pytorch instead uses reverse-mode autodiff. Reverse-mode and forward-mode autodiff are different, but not so different that they are entirely different things. Reverse-mode is, as I put it in my OP, "not much more advanced" than forward-mode, even if not identical. What is entirely different, much more advanced, and what Pytorch really doesn't do, is anything like the "epsilon-delta proofs" you keep hanging your hat on. If Pytorch did that, it would be useless. The entire point of autodiff is to avoid such things. Beyond that, I would suggest slowing down a bit as you are mixing quite a few things up. Nonstandard analysis has nothing to do with dual numbers at all, for instance. And you're very much misinterpreting that MSE post of mine you linked to (thanks!).
- fpgamlirfanboy 2y ago
  > and if addition and multiplication work the way you expect and propagate derivatives correctly - then you are using dual numbers you literally started out your miraculous comment with > This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! not some observation about caching derivatives. so i'll repeat myself for the 3rd time: there are no magical numbers anywhere in pytorch or tensorflow or cafe or any other serious autodiff implementation that abide by the rules you so jubilantly exclaim about.
  - MikeBattaglia 2y ago
    Thank you for repeating yourself three times. It seems like you think that the dual number algebra involves "magic woo numbers." It seems like you haven't really worked through this stuff too much. I would suggest reading some of the resources above, such as the MIT lecture series. The rest of your points I think I have already addressed, though you ignored in your reply - I've said Pytorch does reverse mode diff several times at this point.
    - fpgamlirfanboy 2y ago
      > It seems like you haven't really worked through this stuff too much yup not at all - i just wandered in off the street and knew accidentally that you were talking about non-standard analysis. > The rest of your points I think I have already addressed please show me the source line number in pytorch or tensorflow that defines this number > we add one called "h" with h² = 0!
      - samatman 2y ago
        You seem somewhat obsessed with the idea that reverse-mode autodiff is not the same technique as forward-mode autodiff. It makes you,,, angry? Seems like such a trivial thing to act a complete fool over. What's up with that? Anyway, here's a forward differentiation package with a file that might interest you https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/dual.jl https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/...
bollu 2y ago
You're right on one count: no library implements forward mode. Hence, you're correct that no autodiff library (including pytorch) implement autodiff this way. However, *you're wrong* that forward mode cannot be written in terms of dual numbers. The point is that the addition and multiplication operation for dual numbers correspond exactly to rules of the derivative of addition and the derivative of the product.
- fpgamlirfanboy 2y ago
  i didn't say it cannot be, i said there's a natural reason it's not: the same reason the rest of analysis isn't. > You're right on one count: no library implements forward mode there are plenty of CFD type libraries that implement forward mode. they also do not use what i'm calling "magical" dual numbers i.e. these nilpotent ring elements.
SideQuark 2y ago
You've filled this page with comments on non-standard analysis, but the dual numbers have precisely zero to do with it. Calling people n00bs on a topic you apparently do no understand is silly. Non-standard analysis deals with fields only, and the dual numbers are not a field, there are infinitely many zero divisors. You should read the wiki pages on both, then maybe this mathoverflow post explaining it. The clearest way to maybe grasp the difference for you is that in any formulation of non-standard analysis, the square of any infinitesimal is another infinitesimal, and never 0. In the dual numbers, the square of any infinitesimal is always precisely, exactly zero. They are so fundamentally different that anyone (like you) that claims to be so cognizant of either would never repeat they are the same as loudly and frequently as you are. https://math.stackexchange.com/questions/341535/is-the-theory-of-dual-numbers-strong-enough-to-develop-real-analysis-and-does-i https://math.stackexchange.com/questions/341535/is-the-theor...

MikeBattaglia 2y ago

At the end of the day, if you are storing inputs and outputs to a function as a pair of numbers - one for the actual value, and one for the derivative - and if addition and multiplication work the way you expect and propagate derivatives correctly - then you are using dual numbers, regardless of if you notate it a + b*h or {"value": a, "derivative": b}. Pytorch does things slightly differently in that it is mostly focused on reverse-mode autodiff, and so it stores adjoints relative to the overall output rather than partial derivatives relative to the input, but this isn't really an entirely different thing, in the same way that the FFT isn't entirely different from the DFT. There seems to be some confusion about the relationship between dual numbers and smooth infinitesimal analysis. Both have nilpotent elements, but with dual numbers the background logic is classical, whereas it isn't with smooth infinitesimal analysis. EDIT: I see you've edited your post to try to get in some extra criticism after I've already responded. That's terrible form, so I'll just respond here. Dual numbers are a nice way to get started with forward-mode autodiff, to which it is so related that the two are essentially the same thing with different labels. Pytorch instead uses reverse-mode autodiff. Reverse-mode and forward-mode autodiff are different, but not so different that they are entirely different things. Reverse-mode is, as I put it in my OP, "not much more advanced" than forward-mode, even if not identical. What is entirely different, much more advanced, and what Pytorch really doesn't do, is anything like the "epsilon-delta proofs" you keep hanging your hat on. If Pytorch did that, it would be useless. The entire point of autodiff is to avoid such things. Beyond that, I would suggest slowing down a bit as you are mixing quite a few things up. Nonstandard analysis has nothing to do with dual numbers at all, for instance. And you're very much misinterpreting that MSE post of mine you linked to (thanks!).

fpgamlirfanboy 2y ago
> and if addition and multiplication work the way you expect and propagate derivatives correctly - then you are using dual numbers you literally started out your miraculous comment with > This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! not some observation about caching derivatives. so i'll repeat myself for the 3rd time: there are no magical numbers anywhere in pytorch or tensorflow or cafe or any other serious autodiff implementation that abide by the rules you so jubilantly exclaim about.
- MikeBattaglia 2y ago
  Thank you for repeating yourself three times. It seems like you think that the dual number algebra involves "magic woo numbers." It seems like you haven't really worked through this stuff too much. I would suggest reading some of the resources above, such as the MIT lecture series. The rest of your points I think I have already addressed, though you ignored in your reply - I've said Pytorch does reverse mode diff several times at this point.
  - fpgamlirfanboy 2y ago
    > It seems like you haven't really worked through this stuff too much yup not at all - i just wandered in off the street and knew accidentally that you were talking about non-standard analysis. > The rest of your points I think I have already addressed please show me the source line number in pytorch or tensorflow that defines this number > we add one called "h" with h² = 0!
    - samatman 2y ago
      You seem somewhat obsessed with the idea that reverse-mode autodiff is not the same technique as forward-mode autodiff. It makes you,,, angry? Seems like such a trivial thing to act a complete fool over. What's up with that? Anyway, here's a forward differentiation package with a file that might interest you https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/dual.jl https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/...

bollu 2y ago

You're right on one count: no library implements forward mode. Hence, you're correct that no autodiff library (including pytorch) implement autodiff this way. However, *you're wrong* that forward mode cannot be written in terms of dual numbers. The point is that the addition and multiplication operation for dual numbers correspond exactly to rules of the derivative of addition and the derivative of the product.

fpgamlirfanboy 2y ago
i didn't say it cannot be, i said there's a natural reason it's not: the same reason the rest of analysis isn't. > You're right on one count: no library implements forward mode there are plenty of CFD type libraries that implement forward mode. they also do not use what i'm calling "magical" dual numbers i.e. these nilpotent ring elements.

SideQuark 2y ago

You've filled this page with comments on non-standard analysis, but the dual numbers have precisely zero to do with it. Calling people n00bs on a topic you apparently do no understand is silly. Non-standard analysis deals with fields only, and the dual numbers are not a field, there are infinitely many zero divisors. You should read the wiki pages on both, then maybe this mathoverflow post explaining it. The clearest way to maybe grasp the difference for you is that in any formulation of non-standard analysis, the square of any infinitesimal is another infinitesimal, and never 0. In the dual numbers, the square of any infinitesimal is always precisely, exactly zero. They are so fundamentally different that anyone (like you) that claims to be so cognizant of either would never repeat they are the same as loudly and frequently as you are. https://math.stackexchange.com/questions/341535/is-the-theory-of-dual-numbers-strong-enough-to-develop-real-analysis-and-does-i https://math.stackexchange.com/questions/341535/is-the-theor...