14 ms·

Yeah I think almost anyone will agree that these matrix formulations are more confusing than helpful. The subject itself isn't really that complicated, but the

Yeah I think almost anyone will agree that these matrix formulations are more confusing than helpful. The subject itself isn't really that complicated, but the jargon and notation makes it a lot less approachable than it is. Unwrapping the operations with explicit summation makes it a heck of a lot clearer what is happening. There are places where matrix expressions are informative and helpful, but at least in the context of teaching machine learning, this isn't one of them.

meken 4d ago
Hard disagree, to the point where I'm not sure there's not some confusion at play here. I find using indices and summations generally confusing and hard to read/follow/understand. To me, it's so much simpler to draw out the computational graph, derive the gradient with a single element (so I'm in agreement with you guys here?), see how that applies the same to every element the same, then scale up to the vectorized/matrix version. > and you need to memorize a bunch of new rules to apply it There's nothing to memorize though..? Is there something called "matrix calculus" that's different than just drawing the computational graph and deriving the gradient in the way I described..? Like, is this "matrix calculus"? - https://cs231n.github.io/optimization-2/ https://cs231n.github.io/optimization-2/ If so, what is there to memorize here?
- srean 4d ago
  Not memorize but one may need to derive the result. Consider differentiating the matrix logarithm of a complex matrix.
- pdhborges 3d ago
  OP was talking about this: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf Look into some examples of the section 2.4 "Derivatives of Matrices, Vectors and Scalar Forms"
srean 4d ago
Now that einsum syntax is supported in languages like Python and Julia this is more true. Before einsum syntax, it helped if the result could be written in terms of matrix operation because then one could utilise the primitives offered by matrix libraries. The alternative was to write the raw low level index loops, tedious and errorprone. For some cases though the matrix result is just more compact. For example the derivative of log determinant of matrix wrt the matrix.

meken 4d ago

Hard disagree, to the point where I'm not sure there's not some confusion at play here. I find using indices and summations generally confusing and hard to read/follow/understand. To me, it's so much simpler to draw out the computational graph, derive the gradient with a single element (so I'm in agreement with you guys here?), see how that applies the same to every element the same, then scale up to the vectorized/matrix version. > and you need to memorize a bunch of new rules to apply it There's nothing to memorize though..? Is there something called "matrix calculus" that's different than just drawing the computational graph and deriving the gradient in the way I described..? Like, is this "matrix calculus"? - https://cs231n.github.io/optimization-2/ https://cs231n.github.io/optimization-2/ If so, what is there to memorize here?

srean 4d ago
Not memorize but one may need to derive the result. Consider differentiating the matrix logarithm of a complex matrix.
pdhborges 3d ago
OP was talking about this: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf Look into some examples of the section 2.4 "Derivatives of Matrices, Vectors and Scalar Forms"

srean 4d ago

Now that einsum syntax is supported in languages like Python and Julia this is more true. Before einsum syntax, it helped if the result could be written in terms of matrix operation because then one could utilise the primitives offered by matrix libraries. The alternative was to write the raw low level index loops, tedious and errorprone. For some cases though the matrix result is just more compact. For example the derivative of log determinant of matrix wrt the matrix.