5 ms·

if quaternions, why not octonions?

• danharaj 9y ago
  The lack of associativity might suck.
  • mike_n 9y ago
    perhaps, but i would think that if you threw another tool (a different way to relate some inputs) at a neural net, it might figure out some way to exploit it, even if it's not clear to you or me. just like a RL agent sometimes finds + exploits bugs in the environment.
  • wolfgke 9y ago
    The lack of commutativity for quaternion multiplication might suck, too.
    • danharaj 9y ago
      It doesn't because neural networks are already (matrices + nonlinearity) over the ground field and therefore all but the simplest possible neural networks are non-commutative anyway. Non-associativity is much more troublesome because it sabotages the most important property of neural networks for deep learning, that you can compose them by feeding outputs of one as inputs into another. Take out the associativity and you've taken out the "deep" in "deep learning". Edit: On further reflection, the non-commutativity of neural networks is also a crucial component of machine learning. Without it, a neural network can't make a decision at one layer that depends on its decisions at a previous level!
      • repsilat 9y ago
        I'm not sure I buy this. If - your network structure is fixed, and - you always evaluate the matrix in a given order, and - you are careful/smart about how you train the weights (Ok, I haven't thought through the ramifications here...) then I'm not sure you care much about either commutativity or associativity. Maybe a lack of associativity makes backprop impossible, and maybe commutativity makes "Google deep dream" impossible (no idea), but I don't quite agree with the "composability" objection to a lack of associativity and I don't understand the objection to a lack of commutativity sorry.
        • AstralStorm 9y ago
          The problem is with gradient propagation which requires approximately associative mathematics. You will be training your network something else than you wanted to. Toy evaluate the network in two orders - forward (use) and backward (training).
          • varelse 9y ago
            Floating point math itself is commutative, but it's not associative as it is implemented in current processors so you've already lost associativity from the get-go.
            • danharaj 9y ago
              Floating point's non-associativity is certainly a problem for composability and stability of neural networks, you're right. However this non-associativity is well behaved in the sense that there's an ideal arithmetic we wish to approximate and we have techniques for mitigating the discrepancy between floating point and that arithmetic. The non-associativity of the octonions is fundamental to their structure, not something to be worked around. In particular, there's no way to consider an octonion-valued network as comprising several layers plugged in serial.
  • wespiser_2018 9y ago
    quaternions are associative! Anyway, you can clever code all the algebraic operations of quaternions into matrices and go with that!
    • gugagore 9y ago
      Yeah, but octonions are not associative. Which is what the poster was saying. (By the way, that also means that there isn't a matrix representation for octonions, since matrix multiplicative is associative).
• snissn 9y ago
  and to that - why not N-nions for large values of N?
  • jblow 9y ago
    Because after octonions there isn't anything more.
    • stephencanon 9y ago
      https://en.wikipedia.org/wiki/Sedenion https://en.wikipedia.org/wiki/Sedenion But they're pretty weird; very few of the rules you're used to for "numbers" apply. Ditto for the further constructions.
      • nabla9 9y ago
        Cayley–Dickson construction can be carried to infinity. https://en.wikipedia.org/wiki/Cayley%E2%80%93Dickson_construction https://en.wikipedia.org/wiki/Cayley%E2%80%93Dickson_constru...
      • aisofteng 9y ago
        For more on what's different after the octonions, see: https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composition_algebras) https://en.m.wikipedia.org/wiki/Hurwitz%27s_theorem_(composi... It isn't explicitly stated there, but the statement I recall from algebraic topology is that the octonions are the last normes division algebra.
      • osteele 9y ago
        Many references itemize what properties one gives up with each successive application of the Cayley-Dickson construction: Complex numbers - lose self-conjugate identity, but satisfies the fundamental theorem of algebra (and can represent 2D points or vectors) Quaternions - lose commutativity (but can represent 3D rotation, which isn’t commutative) Octonions - lose associativity, except for each of aab and abb Sedenions - lose associativity of aab and abb John Baez’s “This Week's Finds in Mathematical Physics (Week 59)” http://math.ucr.edu/home/baez/week59.html http://math.ucr.edu/home/baez/week59.html concludes with a letter by Toby Bartels explaining why. An excerpt: I will prove below that the 2^n onions are a division algebra only if the 2^(n-1) onions are associative. So, the question becomes: why aren't the octonions associative? Well, I've found a proof that 2^n onions are associative only if 2^(n-1) onions are commutative. So, why aren't the quaternions commutative? Again, I have a proof that 2^n onions are commutative only if 2^(n-1) onions equal their own conjugates. So, why don't the complex numbers equal their own conjugates? I have a proof that 2^n onions do equal their own conjugates, but it works only if the 2^(n-1) onions are of characteristic 2. The real numbers are not of characteristic 2, so the complex numbers don't equal their own conjugates, so the quaternions aren't commutative, so the octonions aren't associative, so the hexadecanions aren't a division algebra.
• wespiser_2018 9y ago
  Quaternions are known to represent spatial transforms, and there is a little bit of prior work that demonstrates quaternion filters 'make sense'. However, octonions are the obvious next step here: if you look at Appendix Figure 1, of "Deep Complex Networks" [1] , the authors authors used (Real + Complex), and Figure 1 of our paper[2] with quaternions uses (Real + Complex + Complex + Complex)! [1] https://arxiv.org/pdf/1705.09792.pdf https://arxiv.org/pdf/1705.09792.pdf [2] https://arxiv.org/pdf/1712.04604.pdf https://arxiv.org/pdf/1712.04604.pdf
  • joppy 9y ago
    I don't think that octonions are the obvious next step here. They are a non-associative algebra, and therefore incredibly difficult to deal with. Starting with real n-space, one can form the Clifford algebra, which essentially gives a method of multiplying vectors which "knows" something about the length and angle of vectors. The even subalgebra of the Clifford algebra gives a very convenient way of encoding rotations on real n-space. Furthermore, the Clifford algebra is always associative, and works for any n. If you apply this construction for n=1, 2, 3, you get back the real numbers, complex numbers, and quaternions respectively. If you apply this for n=4, you get back an 8-dimensional associative algebra encoding rotations in 4-space.
    • AstralStorm 9y ago
      Sounds quite quantum... negative time?
      • posterboy 9y ago
        phase shift, a shift in phase space.