6 ms·

Reading an article like this makes me realize I am too stupid to ever build a foundation model from scratch.

hoppp 1y ago
It takes a while to get into, just like with everything determination is key Also there are libraries that abstract away most if not all the things, so you don't have to know everything
- staticelf 1y ago
  That's the thing, it's too hard to learn so I rather do something else with the limited time I have left.
  - gregorygoc 1y ago
    I come from a country which had a strong Soviet influence, and in school basically we were taught that behind every hard formula lies an intuitive explanation. As otherwise, there’s no way to come up with the formula in the first place. This statement is not true, there are counter examples I encountered in my university studies but I would say that intuition will get you very far. Einstein was able to come up with special theory of relativity by just manipulating mental models after all. Only when he tried to generalize it, that’s when he hit the limit of the claim I learned in school. That being said after abandoning intuition, relying on pure mathematical reasoning drives you to a desired place and from there you usually can reason about the theorem in an intuitive way again. Math in this paper is not that hard to learn, you just need someone to present you the key idea.
    - aeonik 1y ago
      I wasn't taught this, but came to this conclusion after much struggle, and I think this mentality has served me very well. I hope anyone who is unsure will read your comment and at least try to follow it for a while.
    - godelski 1y ago
      > behind every hard formula lies an intuitive explanation Probably a good thing to teach people when starting out. Especially since I think one thing people have is converting the symbols to the abstract ideas they communicate. I agree you're definitely right that a lot of math (if not most) is really unintuitive. But I think I still like the sentiment behind that idea. Maybe it changes with translation, but I feel like that equations make a lot more sense when I break them down and think about what the symbols are doing and looking at their relationships with one another compared to when just looking at them as symbols to manipulate. Like seeing the form F=ma as more than mass and acceleration and but how in structure it is similar to F=-kx. Getting there isn't easy but once you do it is much more intuitive than it was before.
reactordev 1y ago
Haha, I was just going to say the same. I was hoping, I guess naively, that this would explain the math. Not just show me math. While I love a good figure, I like pseudocode just as much :)
oersted 1y ago
Paper authors (and this posts author apparently) like to throw in lots of scary-looking maths to signal that they are smart and that what they are doing has merit. The Reinforcement Learning field is particularly notorious for doing this, but it's all over ML. Often it is not on purpose, everyone is taught this is the proper "formal" way to express these things, and that any other representation is not precise or appropriate in a scientific context. In practice, when it comes down to code, even without higher-level libraries, it is surprisingly simple, concise and intuitive. Most of the math elements used have quite straightforward properties and utility, but of course if you combine them all together into big expressions with lots of single-character variables, it's really hard to understand for everyone. You kind of need to learn to squint your eyes and understand the basic building-blocks that the maths represent, but that shouldn't be necessary if it wasn't obfuscated like this.
- MattPalmer1086 1y ago
  Haha, recognise. I invented a fast search algorithm and worked with some academics to publish a paper on it last year. They threw in all the complex math to the paper. I could not initially understand it at all despite inventing the damn algorithm! Having said that, picking it apart and taking a little time with it, it actually wasn't that hard - but it sure looked scary and incomprehensible at first!
- catgary 1y ago
  I’m going to push back on this a bit. I think a simpler explanation (or at least one that doesn’t involve projecting one’s own insecurities onto the authors) is that the people who write these papers are generally comfortable enough with mathematics that they don’t believe anything has been obfuscated. ML is a mathematical science and many people in ML were trained as physicists or mathematicians (I’m one of them). People write things this way because it makes symbolic manipulations easier and you can keep the full expression in your head; what you’re proposing would actually make it significantly harder to verify results in papers.
  - voidhorse 1y ago
    Agreed. Also, fwiw, the mathematics involved in the paper are pretty simple as far as mathematical sophistication goes. Spend two to three months on one "higher level" maths course of your choosing and you'll be able to fully understand every equation in this paper relatively easily. Even a basic course in information theory coupled with some discrete maths should give you essentially all you need to comprehend the math in this post. The concepts being presented here are not mysterious and much of this math is banal. Mathematical notation can seem foreboding, but once you grasp it, you'll see, like Von Neumann said, that life is complicated but math is simple.
woah 1y ago
The big zig zaggy "E" is a for loop. That's all you really have to know
emseetech 1y ago
"You Are NOT Dumb, You Just Lack the Prerequisites" https://lelouch.dev/blog/you-are-probably-not-dumb/ https://lelouch.dev/blog/you-are-probably-not-dumb/

It takes a while to get into, just like with everything determination is key Also there are libraries that abstract away most if not all the things, so you don't have to know everything

staticelf 1y ago
That's the thing, it's too hard to learn so I rather do something else with the limited time I have left.
- gregorygoc 1y ago
  I come from a country which had a strong Soviet influence, and in school basically we were taught that behind every hard formula lies an intuitive explanation. As otherwise, there’s no way to come up with the formula in the first place. This statement is not true, there are counter examples I encountered in my university studies but I would say that intuition will get you very far. Einstein was able to come up with special theory of relativity by just manipulating mental models after all. Only when he tried to generalize it, that’s when he hit the limit of the claim I learned in school. That being said after abandoning intuition, relying on pure mathematical reasoning drives you to a desired place and from there you usually can reason about the theorem in an intuitive way again. Math in this paper is not that hard to learn, you just need someone to present you the key idea.
  - aeonik 1y ago
    I wasn't taught this, but came to this conclusion after much struggle, and I think this mentality has served me very well. I hope anyone who is unsure will read your comment and at least try to follow it for a while.
  - godelski 1y ago
    > behind every hard formula lies an intuitive explanation Probably a good thing to teach people when starting out. Especially since I think one thing people have is converting the symbols to the abstract ideas they communicate. I agree you're definitely right that a lot of math (if not most) is really unintuitive. But I think I still like the sentiment behind that idea. Maybe it changes with translation, but I feel like that equations make a lot more sense when I break them down and think about what the symbols are doing and looking at their relationships with one another compared to when just looking at them as symbols to manipulate. Like seeing the form F=ma as more than mass and acceleration and but how in structure it is similar to F=-kx. Getting there isn't easy but once you do it is much more intuitive than it was before.

reactordev 1y ago

Haha, I was just going to say the same. I was hoping, I guess naively, that this would explain the math. Not just show me math. While I love a good figure, I like pseudocode just as much :)

oersted 1y ago

Paper authors (and this posts author apparently) like to throw in lots of scary-looking maths to signal that they are smart and that what they are doing has merit. The Reinforcement Learning field is particularly notorious for doing this, but it's all over ML. Often it is not on purpose, everyone is taught this is the proper "formal" way to express these things, and that any other representation is not precise or appropriate in a scientific context. In practice, when it comes down to code, even without higher-level libraries, it is surprisingly simple, concise and intuitive. Most of the math elements used have quite straightforward properties and utility, but of course if you combine them all together into big expressions with lots of single-character variables, it's really hard to understand for everyone. You kind of need to learn to squint your eyes and understand the basic building-blocks that the maths represent, but that shouldn't be necessary if it wasn't obfuscated like this.

MattPalmer1086 1y ago
Haha, recognise. I invented a fast search algorithm and worked with some academics to publish a paper on it last year. They threw in all the complex math to the paper. I could not initially understand it at all despite inventing the damn algorithm! Having said that, picking it apart and taking a little time with it, it actually wasn't that hard - but it sure looked scary and incomprehensible at first!
catgary 1y ago
I’m going to push back on this a bit. I think a simpler explanation (or at least one that doesn’t involve projecting one’s own insecurities onto the authors) is that the people who write these papers are generally comfortable enough with mathematics that they don’t believe anything has been obfuscated. ML is a mathematical science and many people in ML were trained as physicists or mathematicians (I’m one of them). People write things this way because it makes symbolic manipulations easier and you can keep the full expression in your head; what you’re proposing would actually make it significantly harder to verify results in papers.
- voidhorse 1y ago
  Agreed. Also, fwiw, the mathematics involved in the paper are pretty simple as far as mathematical sophistication goes. Spend two to three months on one "higher level" maths course of your choosing and you'll be able to fully understand every equation in this paper relatively easily. Even a basic course in information theory coupled with some discrete maths should give you essentially all you need to comprehend the math in this post. The concepts being presented here are not mysterious and much of this math is banal. Mathematical notation can seem foreboding, but once you grasp it, you'll see, like Von Neumann said, that life is complicated but math is simple.

woah 1y ago

The big zig zaggy "E" is a for loop. That's all you really have to know

emseetech 1y ago

"You Are NOT Dumb, You Just Lack the Prerequisites" https://lelouch.dev/blog/you-are-probably-not-dumb/ https://lelouch.dev/blog/you-are-probably-not-dumb/