6 ms·

i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so i

i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so it's a pretty bad math book to actually learn math from (you can always spot a pedagogically unsound math book by its lack of exercises). the book doesn't even have any code in it which is surprising considering it has "programming" in the title. actually i know why people write these but i still don't know why they publish them: this is a phase everyone goes through in their "math" life where they look back on everything they've learned hastily between undergrad/phd/postdoc (or whatever) and they have the urge to formalize/crystallize. everyone has the urge - i had a late-career QFT prof tell me that he was excited to take his sabbatical so that he could finally do all of the exercises in peskin&schroeder for real real and type it all up neatly. i've done it too, in-the-small (some very nice notes that i'm proud of, on various things). you sit down, make your list of things, pull up all of the books/papers you're going to use as references and you start essentially transcribing - but you tell yourself you're putting your own spin on it (adding ample "motivation"). and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. ninja edit: before someone jumps down my throat about "what's the harm?". the harm is n00b/undergrads/young people/etc will look at this and think this is the right way to learn this material and some of them will even make an attempt to learn the material from this thing and they'll struggle and fail and be discouraged - i speak from experience! it's not a good thing for the community. sure maybe 1 in a 100 can learn this stuff from just reading a monograph (what these things used to be called...) but that's the exception that proves the rule.

deleted 2y ago
[deleted]
hyperbovine 2y ago
[flagged]
- fpgamlirfanboy 2y ago
  > graduate-level introduction to differentiable programming. go check out any real graduate textbook. what you will find is they all have exercises. > It's a book for ML researchers. I'm excited to read it. Calm down. just because the authors claim something doesn't make it true. i'm not wrong - this is not a good pedagogical resource and i would bet a year of my salary (as an ML researcher) that you will in fact not read more than 5% of this book.
  - smallnamespace 2y ago
    It's a good thing you're an ML researcher and not, say, a quant trader then ;)
  - thfuran 2y ago
    Really? I'd gladly read 6% of a book for a year of pay.
  - hyperbovine 2y ago
    You certainly have the demeanor of a certain type of ML researcher that I have encountered from time to time ...
bbor 2y ago
You’re getting hit on a bunch of different things here, but I’d like to focus on two: it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. Doesn’t this apply to all books of any kind? How do you know what if you can write a book before you try? I feel like “transcribe papers with a spin on them” is a perfect description of Russel and Norvig’s AI book, and many find it valuable. Is this just a math-specific criticism? Either way, thanks for the very interested contrarian comment! You’re well spoken and I do love a discussion more than “wow this looks useful”, which is all I was prepared to give.
- fpgamlirfanboy 2y ago
  > I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. that's not what this book is. like i said in the first sentence right there in the part right before the all caps calculus - this is book is a compendium of theorem/proofs/etc. very little actual software. > Doesn’t this apply to all books of any kind? again, i already covered this: a pedadogically sound textbook will have exercises and structure/themes/etc rather than just 120 "propositions".
nonagono 2y ago
It’s an introduction to a relatively niche new subfield. If I (an expert in the field but not the subfield) want to learn about differentiable programming, my only option before this monograph was to read through tens of random papers which use different presentation styles, terminology etc. Now I can read through the second half of this, around 100 pages, and jump back to the first half if there’s a prerequisite I don’t know. That’s how most subfields are born. Assorted papers -> monograph -> textbook. The first arrow is defining the subfield as a discrete topic, which is immensely valuable. Only after you have that you can start optimizing for presentation to nonexperts.
- fpgamlirfanboy 2y ago
  [flagged]
- bsdpufferfish 2y ago
  Differentiable programming is hardly a "subfield" it can be explained in a paragraph if you know calculus well. If there is any subfield, it's in researching specific compiler optimizations.
  - nonagono 2y ago
    Well clearly not, since at least 100 pages of content here are specifically about differentiable programming and not prerequisites :) More seriously, it's about doing the impossible. Formally, some functions are nondifferentiable, period. But it would be cool if we could actually "more or less" differentiate them. For that we'll necessarily need a bag of tricks which is now coalescing into "techniques" and "principles". Cf. numerical analysis. It takes a page or two to set up your definitions and show that many functions are badly conditioned, period. And yet we still want to compute them, so we've been building the bag of tricks for almost a century now.
    - bsdpufferfish 2y ago
      > not prerequisites Almost the entire document is undergrad numerical topics (finite difference methods, maximum likelihood, Jacobins, hessians, newtons method, etc). This is all well covered material that is soundly not research.
hansvm 2y ago
Taste-wise, I'd prefer these sorts of books to just briefly state facts and spend most of the time on building intuition, but just putting it all together in one place is nice in and of itself. I've had to re-derive all this shit on my own the last six months, and it would have been helpful to have a moderately complete compendium of the average practitioner's working knowledge available as a starting reference (admittedly, the first half is a hodge-podge that sort of backgrounds the second). It serves the same purpose as a good "survey" paper. Taking ordinary calculus knowledge and expanding it to differentiable programming is hard. Even if-else branches have nuances which, at first glance, it looks like the authors didn't handle. They confidently state, for example, that the derivatives of predicate variables are uninformative, but the veracity of that statement depends on the local properties of the branches in question. It's often true for "mathy" conditional statements and often not true for more computer-programming-esque if-statements. Separating a particular representation of a computation from what it actually computes and obtaining accurate partial derivatives is _hard_. Jax and Julia both have bugs on that front, and it's good that people are trying to make this knowledge more accessible (admittedly, not very well on that front IMO, but the other sections I spot-checked all look to be higher quality).

deleted 2y ago

[deleted]

hyperbovine 2y ago

[flagged]

fpgamlirfanboy 2y ago
> graduate-level introduction to differentiable programming. go check out any real graduate textbook. what you will find is they all have exercises. > It's a book for ML researchers. I'm excited to read it. Calm down. just because the authors claim something doesn't make it true. i'm not wrong - this is not a good pedagogical resource and i would bet a year of my salary (as an ML researcher) that you will in fact not read more than 5% of this book.
- smallnamespace 2y ago
  It's a good thing you're an ML researcher and not, say, a quant trader then ;)
- thfuran 2y ago
  Really? I'd gladly read 6% of a book for a year of pay.
- hyperbovine 2y ago
  You certainly have the demeanor of a certain type of ML researcher that I have encountered from time to time ...

bbor 2y ago

You’re getting hit on a bunch of different things here, but I’d like to focus on two: it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. Doesn’t this apply to all books of any kind? How do you know what if you can write a book before you try? I feel like “transcribe papers with a spin on them” is a perfect description of Russel and Norvig’s AI book, and many find it valuable. Is this just a math-specific criticism? Either way, thanks for the very interested contrarian comment! You’re well spoken and I do love a discussion more than “wow this looks useful”, which is all I was prepared to give.

fpgamlirfanboy 2y ago
> I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. that's not what this book is. like i said in the first sentence right there in the part right before the all caps calculus - this is book is a compendium of theorem/proofs/etc. very little actual software. > Doesn’t this apply to all books of any kind? again, i already covered this: a pedadogically sound textbook will have exercises and structure/themes/etc rather than just 120 "propositions".

nonagono 2y ago

It’s an introduction to a relatively niche new subfield. If I (an expert in the field but not the subfield) want to learn about differentiable programming, my only option before this monograph was to read through tens of random papers which use different presentation styles, terminology etc. Now I can read through the second half of this, around 100 pages, and jump back to the first half if there’s a prerequisite I don’t know. That’s how most subfields are born. Assorted papers -> monograph -> textbook. The first arrow is defining the subfield as a discrete topic, which is immensely valuable. Only after you have that you can start optimizing for presentation to nonexperts.

fpgamlirfanboy 2y ago
[flagged]
bsdpufferfish 2y ago
Differentiable programming is hardly a "subfield" it can be explained in a paragraph if you know calculus well. If there is any subfield, it's in researching specific compiler optimizations.
- nonagono 2y ago
  Well clearly not, since at least 100 pages of content here are specifically about differentiable programming and not prerequisites :) More seriously, it's about doing the impossible. Formally, some functions are nondifferentiable, period. But it would be cool if we could actually "more or less" differentiate them. For that we'll necessarily need a bag of tricks which is now coalescing into "techniques" and "principles". Cf. numerical analysis. It takes a page or two to set up your definitions and show that many functions are badly conditioned, period. And yet we still want to compute them, so we've been building the bag of tricks for almost a century now.
  - bsdpufferfish 2y ago
    > not prerequisites Almost the entire document is undergrad numerical topics (finite difference methods, maximum likelihood, Jacobins, hessians, newtons method, etc). This is all well covered material that is soundly not research.

hansvm 2y ago

Taste-wise, I'd prefer these sorts of books to just briefly state facts and spend most of the time on building intuition, but just putting it all together in one place is nice in and of itself. I've had to re-derive all this shit on my own the last six months, and it would have been helpful to have a moderately complete compendium of the average practitioner's working knowledge available as a starting reference (admittedly, the first half is a hodge-podge that sort of backgrounds the second). It serves the same purpose as a good "survey" paper. Taking ordinary calculus knowledge and expanding it to differentiable programming is hard. Even if-else branches have nuances which, at first glance, it looks like the authors didn't handle. They confidently state, for example, that the derivatives of predicate variables are uninformative, but the veracity of that statement depends on the local properties of the branches in question. It's often true for "mathy" conditional statements and often not true for more computer-programming-esque if-statements. Separating a particular representation of a computation from what it actually computes and obtaining accurate partial derivatives is _hard_. Jax and Julia both have bugs on that front, and it's good that people are trying to make this knowledge more accessible (admittedly, not very well on that front IMO, but the other sections I spot-checked all look to be higher quality).