8 ms·

This article is saying that it can be numerically unstable in certain situations, not that it's theoretically incorrect.

omnicognate 1y ago
Yeah, perhaps the actual title would be better: "The Numerical Analysis of Differentiable Simulation". (Rather than the subtitle, which is itself a poor rewording of the actual subtitle in the video.)
goosedragons 1y ago
It can be both. A mistake in AD primitives can lead to theoretically incorrect derivatives. With the system I use I have run into a few scenarios where edge cases aren't totally covered leading to the wrong result. I have also run into numerical instability too.
- froobius 1y ago
  > A mistake in AD primitives can lead to theoretically incorrect derivatives Ok but that's true of any program. A mistake in the implementation of the program can lead to mistakes in the result of the program...
  - goosedragons 1y ago
    That's true! But it's also true that any program dealing with floats can run into numerical instability if care isn't taken to avoid it, no? It's also not necessarily immediately obvious that the derivatives ARE wrong if the implementation is wrong.
    - srean 1y ago
      > It's also not necessarily immediately obvious that the derivatives ARE wrong if the implementation is wrong. It's neither full proof or fool proof but an absolute must is a check that the loss function is reducing. It quickly detects a common error that the sign came out wrong in my gradient call. Part of good practice one learns in grad school.
    - froobius 1y ago
      You can pretty concretely and easily check that the AD primatives are correct by comparing them to numerical differentiation.
  - godelski 1y ago
    I haven't watched the video but the text says they're getting like 60+% error on simple linear ODEs which is pretty problematic. You're right, but the scale of the problem seems to be the issue
adgjlsfhk1 1y ago
This is less about numerical instability and more that iterative algorithms with error control their error, but when you run AD on them you are ADing the approximation and a derivative of an approximation can be arbitrarily different from an approximation of a derivative.
- ogogmad 1y ago
  That makes more sense. The title is flat out wrong IMO.
  - adgjlsfhk1 1y ago
    I think it is correct. lots of people view AD as a black box that you can throw algorithms to and get derivatives out, and this shows that that isn't true
    - wakawaka28 1y ago
      If you wrote code that failed to compile, you wouldn't impulsively call your compiler incorrect. This title sounds like it puts the blame in the wrong place. You can get error accumulation from even a basic calculation in a loop. We could try to solve these problems but it's not the algorithm's fault you don't know what you're doing.
      - ChrisRackauckas 1y ago
        This has nothing to do with floating point error accumulation or numerical stability in the floating point sense. You can do this with arbitrary sized floating point values and you will still get the same non-convergence result.
        wakawaka28 1y ago
        Yes, it does. I admit that I have not tried to get deep into this, but it says right in the summary that this is due to error propagation and numerical instability. You can have such errors in ordinary hand-written code. The size of the float does not matter (much) for some examples, which wipe out a bunch of significant figures from the result. I'm not going to sit here and pretend I know exactly the details going on here but I studied numerical analysis and AD quite a bit back in the day. Even in the 2010s people knew that there was a chance of the resulting expressions having inherent instability. Even simple things like the quadratic formula have more stable and less stable forms. How much worse might it be for auto-generated expressions (or expression sequences, or equivalent; AD can be done a few ways) such as in AD? AD guarantees analytically correct logic (in infinite precision, for example) if you use it right but error analysis is not even attempted by most libraries.

omnicognate 1y ago

Yeah, perhaps the actual title would be better: "The Numerical Analysis of Differentiable Simulation". (Rather than the subtitle, which is itself a poor rewording of the actual subtitle in the video.)

goosedragons 1y ago

It can be both. A mistake in AD primitives can lead to theoretically incorrect derivatives. With the system I use I have run into a few scenarios where edge cases aren't totally covered leading to the wrong result. I have also run into numerical instability too.

froobius 1y ago
> A mistake in AD primitives can lead to theoretically incorrect derivatives Ok but that's true of any program. A mistake in the implementation of the program can lead to mistakes in the result of the program...
- goosedragons 1y ago
  That's true! But it's also true that any program dealing with floats can run into numerical instability if care isn't taken to avoid it, no? It's also not necessarily immediately obvious that the derivatives ARE wrong if the implementation is wrong.
  - srean 1y ago
    > It's also not necessarily immediately obvious that the derivatives ARE wrong if the implementation is wrong. It's neither full proof or fool proof but an absolute must is a check that the loss function is reducing. It quickly detects a common error that the sign came out wrong in my gradient call. Part of good practice one learns in grad school.
  - froobius 1y ago
    You can pretty concretely and easily check that the AD primatives are correct by comparing them to numerical differentiation.
- godelski 1y ago
  I haven't watched the video but the text says they're getting like 60+% error on simple linear ODEs which is pretty problematic. You're right, but the scale of the problem seems to be the issue

adgjlsfhk1 1y ago

This is less about numerical instability and more that iterative algorithms with error control their error, but when you run AD on them you are ADing the approximation and a derivative of an approximation can be arbitrarily different from an approximation of a derivative.

ogogmad 1y ago
That makes more sense. The title is flat out wrong IMO.
- adgjlsfhk1 1y ago
  I think it is correct. lots of people view AD as a black box that you can throw algorithms to and get derivatives out, and this shows that that isn't true
  - wakawaka28 1y ago
    If you wrote code that failed to compile, you wouldn't impulsively call your compiler incorrect. This title sounds like it puts the blame in the wrong place. You can get error accumulation from even a basic calculation in a loop. We could try to solve these problems but it's not the algorithm's fault you don't know what you're doing.
    - ChrisRackauckas 1y ago
      This has nothing to do with floating point error accumulation or numerical stability in the floating point sense. You can do this with arbitrary sized floating point values and you will still get the same non-convergence result.
      - wakawaka28 1y ago
        Yes, it does. I admit that I have not tried to get deep into this, but it says right in the summary that this is due to error propagation and numerical instability. You can have such errors in ordinary hand-written code. The size of the float does not matter (much) for some examples, which wipe out a bunch of significant figures from the result. I'm not going to sit here and pretend I know exactly the details going on here but I studied numerical analysis and AD quite a bit back in the day. Even in the 2010s people knew that there was a chance of the resulting expressions having inherent instability. Even simple things like the quadratic formula have more stable and less stable forms. How much worse might it be for auto-generated expressions (or expression sequences, or equivalent; AD can be done a few ways) such as in AD? AD guarantees analytically correct logic (in infinite precision, for example) if you use it right but error analysis is not even attempted by most libraries.