9 ms·

Hacker's Guide to Neural Networks

450 points by bernatfp 12y ago

antimora 12y ago
I wanted to share this resource as well. I started this online free book "Neural Networks and Deep Learning" (http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/). I think it has a pretty good explanation and illustration.
- raphar 12y ago
  I've enjoyed the published chapters so far. When would you finish the missing chapters?
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
therobot24 12y ago
HN loves Neural Nets & Deep Learning - it's all i ever see in my RSS feed (with regard to ML methods)
- nojvek 12y ago
  For a guy who is not very good at Maths, this was a great explanation. Thanks.
ajtulloch 12y ago
This is a great piece of work - thanks @karpathy.
karpathy 12y ago
Thanks for the upvotes! I'm a little conflicted about this being linked around too much because it is still very much work in progress. I work on this guide on a side because I think there's a lot of interest in these models out there, and not enough explanations. On a related note, some of you might also be interested in a Stanford CS class I'll be co-teaching with my adviser next quarter. It will be focused primarily on Convolutional Networks (but a lot of the learning machinery is generic): CS231n: Convolutional Neural Networks for Visual Recognition http://vision.stanford.edu/teaching/cs231n/ http://vision.stanford.edu/teaching/cs231n/ I hope to make a lot of the materials/code freely available so everyone can follow along, and I will continue my work on this guide in parallel whenever I can squeeze in time. (And I'd be happy to hear any feedback on the content/style/presentation)
- miket 12y ago
  Thanks a lot for writing this tutorial, Andrej. Super clear, the next Feynman in the making here! Interesting choice of language, maybe you could add little run buttons to the code snippets to try all the examples in the browser :-)
  - karpathy 12y ago
    Thank you! I'm a huge fan of interactive demos (see e.g. my ConvNetJS demos: http://cs.stanford.edu/people/karpathy/convnetjs/ http://cs.stanford.edu/people/karpathy/convnetjs/ ) because they help develop intuitions that solidify the material. And yes, that's partly why I chose Javascript; I'm definitely planning to add little demos of the gradients flowing around the circuit and so on. Coming soon! :)
- Jailbird 12y ago
  A nice read - Just a minor nit: s/peaked/piqued/ -A
- wodenokoto 12y ago
  Thank you for publicizing an early draft. It is really good and very helpful. I think your explanations are great, but you seem to imply that the gradient 3 for f(x,y) = xy with x = -2 and y=3 derived w.r.t x is somehow the "best" gradient (you call it the correct or actual gradient) but I don't understand why this is better than any other positive number. Wouldn't we optimize faster of we tug with a force of 6 instead of 3? x = -2, y= 3, dx = 3, dy = -2 step = 0.0001 f(x,y) = -6 f(x+dx*step, y*dy*step) = -5.998... // Now if we double the gradient f(x+2*dx*step, y*2*dy*step) = -5.997...
xanderjanz 12y ago
https://class.coursera.org/ml-007 https://class.coursera.org/ml-007 Andrew Ng's Standford course has been a god send in laying out the mathmatics of Machine Learning. Would be a good next step for anybody who was intrigued by this article.
infinitone 12y ago
I was wondering the other day, since I have a CV project that could use neural networks to solve some problems. How big of a training dataset does one need? Is there some analysis on the relationship between the size of training data and accuracy?
- adam-_- 12y ago
  We've built a model to analyse the value of a CV. We had a dataset of 50,000 CVs. Feel free to drop me a line if you'd like to discuss any further. I'd be interested to hear what you're working on.
zackmorris 12y ago
This is the first explanation of neural nets that has really clicked for me. That's because it's written in the same explanatory tone that I would use in person while trying to convey the concepts. I wish more people would write like this. I think of these things like lectures in college. Assuming that a) you have all the context up to that point, b) you grok every step, and c) it doesn't run beyond the length of you attention span - call it an hour - then you can learn astonishing things in a matter of a few sessions. But if you miss a step, and don't raise your hand because you think your question is stupid, then you might as well pack up and go home. I find that even though the time element of articles is removed, I still trip over certain sections because the author was too lazy to anticipate my thinking process, which forces me to search other sites for the answers, and before you know it the impact of the writing has been completely diluted.
JoelHobson 12y ago
I haven't finished it yet, but I find this much easier to understand than any other article on neural networks that I've read.
jimmaswell 12y ago
"it is important to note that in the left-hand side of the equation above, the horizontal line does not indicate division." It pretty much is, though.
ilyaeck 12y ago
I saw the JS demos a few months ago and was blown away. It seems like the community is really thirsty for NN tools and materials, and the way you are going about this (interactive JS) is right on the money? Why not engage the community to keep building the site up, then? You may want to look into open-sourcing the entire site + the JS library, it may really pick up steam. Worse case, it breaks apart (can always reclaim it), but the exercise could be a very interesting one.
hoprocker 12y ago
This is great. Always appreciate different approaches to NN and ML. I'm amazed that a Stanford PhD candidate has the time to put this together (I won't tell your advisor :-), but still, thank you!
bribri 12y ago
Absolutely Amazing. I wish I could be taught more math in terms of programming. Hope some more people make "Hacker's Guide to _"
- wodenokoto 12y ago
  There's a series of books called "Thinking ..." that aims for exactly this. http://www.greenteapress.com/thinkbayes/ http://www.greenteapress.com/thinkbayes/ http://www.greenteapress.com/index.html http://www.greenteapress.com/index.html
jaza 12y ago
Great guide - the only material I've ever read on the subject, that hasn't completely made my head hurt and my brain not grok. I'm not a maths or ML guy, just a regular programmer. I've dabbled in NN before, but only to the extent of using some libraries as a "black box" to pass parameters / training data to. No doubt there are many more people like me (too many!). My understanding after reading this guide, is that a neural network is essentially just a formula for guessing the ideal parameter(s) of another formula, and for successively refining those parameters as more training data is passed in. I already knew this "in theory" before, but now I think most of the magic and mystery has been brushed off. Thanks karpathy!
- junto 12y ago
  I second this. I've always struggled with maths at this (relatively simple) level. However seeing the code and then the related equation is teaching me mathematics that I couldn't grok in school 20 years ago. This is (for me) a much better way to explain it! Thank you for sharing.
breflabb 12y ago
inspiring! i have always been interested in AI and neuralnets but i have been discuraged by the math, many educators that try to learn out these topics assume that their readers have the same math skills as them selfs and omits the boring parts for them. its actually the first time i enjoyed to read about AI and at the same time learn calculus! :) keepup the great work!
thewarrior 12y ago
This seems less like a biological neuron and more like a multi level table lookup.
sireat 12y ago
This was a very good read. My naive question are the gates are always so trivial or are they usually black boxes with an unknown function underneath(instead of clearly defined ,+, etc). In other words do we usually/always know what stands behind f(x,y)? Otherwise, if f(x,y)=xy then obviously you can hardcode a heuretic that just cranks up max x and max y and gets max f(x,y). That is the question is why should we tweak the input slightly not go all out.
midgetjones 12y ago
This was a fantastic read, thanks so much! Can anyone define 'pertubations' for me?
- nekopa 12y ago
  I think it means small changes, but there is also this: http://en.m.wikipedia.org/wiki/Perturbation_theory http://en.m.wikipedia.org/wiki/Perturbation_theory.
sathya_vj 12y ago
Thanks a lot for this!
jdminhbg 12y ago
I worked through the first few chapters on a flight yesterday and found this to be exactly what I need -- my ability to read code and sort of intuit alogrithms vastly exceeds my ability to read and understand math. Thanks for putting this together.
Elzair 12y ago
Thank you karpathy. This is a great introduction.

450 points by bernatfp 12y ago

I wanted to share this resource as well. I started this online free book "Neural Networks and Deep Learning" (http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/). I think it has a pretty good explanation and illustration.

raphar 12y ago
I've enjoyed the published chapters so far. When would you finish the missing chapters?

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]

therobot24 12y ago

HN loves Neural Nets & Deep Learning - it's all i ever see in my RSS feed (with regard to ML methods)

nojvek 12y ago
For a guy who is not very good at Maths, this was a great explanation. Thanks.

ajtulloch 12y ago

This is a great piece of work - thanks @karpathy.

karpathy 12y ago

Thanks for the upvotes! I'm a little conflicted about this being linked around too much because it is still very much work in progress. I work on this guide on a side because I think there's a lot of interest in these models out there, and not enough explanations. On a related note, some of you might also be interested in a Stanford CS class I'll be co-teaching with my adviser next quarter. It will be focused primarily on Convolutional Networks (but a lot of the learning machinery is generic): CS231n: Convolutional Neural Networks for Visual Recognition http://vision.stanford.edu/teaching/cs231n/ http://vision.stanford.edu/teaching/cs231n/ I hope to make a lot of the materials/code freely available so everyone can follow along, and I will continue my work on this guide in parallel whenever I can squeeze in time. (And I'd be happy to hear any feedback on the content/style/presentation)

miket 12y ago
Thanks a lot for writing this tutorial, Andrej. Super clear, the next Feynman in the making here! Interesting choice of language, maybe you could add little run buttons to the code snippets to try all the examples in the browser :-)
- karpathy 12y ago
  Thank you! I'm a huge fan of interactive demos (see e.g. my ConvNetJS demos: http://cs.stanford.edu/people/karpathy/convnetjs/ http://cs.stanford.edu/people/karpathy/convnetjs/ ) because they help develop intuitions that solidify the material. And yes, that's partly why I chose Javascript; I'm definitely planning to add little demos of the gradients flowing around the circuit and so on. Coming soon! :)
Jailbird 12y ago
A nice read - Just a minor nit: s/peaked/piqued/ -A
wodenokoto 12y ago
Thank you for publicizing an early draft. It is really good and very helpful. I think your explanations are great, but you seem to imply that the gradient 3 for f(x,y) = xy with x = -2 and y=3 derived w.r.t x is somehow the "best" gradient (you call it the correct or actual gradient) but I don't understand why this is better than any other positive number. Wouldn't we optimize faster of we tug with a force of 6 instead of 3? x = -2, y= 3, dx = 3, dy = -2 step = 0.0001 f(x,y) = -6 f(x+dx*step, y*dy*step) = -5.998... // Now if we double the gradient f(x+2*dx*step, y*2*dy*step) = -5.997...

xanderjanz 12y ago

https://class.coursera.org/ml-007 https://class.coursera.org/ml-007 Andrew Ng's Standford course has been a god send in laying out the mathmatics of Machine Learning. Would be a good next step for anybody who was intrigued by this article.

infinitone 12y ago

I was wondering the other day, since I have a CV project that could use neural networks to solve some problems. How big of a training dataset does one need? Is there some analysis on the relationship between the size of training data and accuracy?

adam-_- 12y ago
We've built a model to analyse the value of a CV. We had a dataset of 50,000 CVs. Feel free to drop me a line if you'd like to discuss any further. I'd be interested to hear what you're working on.

zackmorris 12y ago

This is the first explanation of neural nets that has really clicked for me. That's because it's written in the same explanatory tone that I would use in person while trying to convey the concepts. I wish more people would write like this. I think of these things like lectures in college. Assuming that a) you have all the context up to that point, b) you grok every step, and c) it doesn't run beyond the length of you attention span - call it an hour - then you can learn astonishing things in a matter of a few sessions. But if you miss a step, and don't raise your hand because you think your question is stupid, then you might as well pack up and go home. I find that even though the time element of articles is removed, I still trip over certain sections because the author was too lazy to anticipate my thinking process, which forces me to search other sites for the answers, and before you know it the impact of the writing has been completely diluted.

JoelHobson 12y ago

I haven't finished it yet, but I find this much easier to understand than any other article on neural networks that I've read.

jimmaswell 12y ago

"it is important to note that in the left-hand side of the equation above, the horizontal line does not indicate division." It pretty much is, though.

ilyaeck 12y ago

I saw the JS demos a few months ago and was blown away. It seems like the community is really thirsty for NN tools and materials, and the way you are going about this (interactive JS) is right on the money? Why not engage the community to keep building the site up, then? You may want to look into open-sourcing the entire site + the JS library, it may really pick up steam. Worse case, it breaks apart (can always reclaim it), but the exercise could be a very interesting one.

hoprocker 12y ago

This is great. Always appreciate different approaches to NN and ML. I'm amazed that a Stanford PhD candidate has the time to put this together (I won't tell your advisor :-), but still, thank you!

bribri 12y ago

Absolutely Amazing. I wish I could be taught more math in terms of programming. Hope some more people make "Hacker's Guide to _"

wodenokoto 12y ago
There's a series of books called "Thinking ..." that aims for exactly this. http://www.greenteapress.com/thinkbayes/ http://www.greenteapress.com/thinkbayes/ http://www.greenteapress.com/index.html http://www.greenteapress.com/index.html

jaza 12y ago

Great guide - the only material I've ever read on the subject, that hasn't completely made my head hurt and my brain not grok. I'm not a maths or ML guy, just a regular programmer. I've dabbled in NN before, but only to the extent of using some libraries as a "black box" to pass parameters / training data to. No doubt there are many more people like me (too many!). My understanding after reading this guide, is that a neural network is essentially just a formula for guessing the ideal parameter(s) of another formula, and for successively refining those parameters as more training data is passed in. I already knew this "in theory" before, but now I think most of the magic and mystery has been brushed off. Thanks karpathy!

junto 12y ago
I second this. I've always struggled with maths at this (relatively simple) level. However seeing the code and then the related equation is teaching me mathematics that I couldn't grok in school 20 years ago. This is (for me) a much better way to explain it! Thank you for sharing.

breflabb 12y ago

inspiring! i have always been interested in AI and neuralnets but i have been discuraged by the math, many educators that try to learn out these topics assume that their readers have the same math skills as them selfs and omits the boring parts for them. its actually the first time i enjoyed to read about AI and at the same time learn calculus! :) keepup the great work!

thewarrior 12y ago

This seems less like a biological neuron and more like a multi level table lookup.

sireat 12y ago

This was a very good read. My naive question are the gates are always so trivial or are they usually black boxes with an unknown function underneath(instead of clearly defined ,+, etc). In other words do we usually/always know what stands behind f(x,y)? Otherwise, if f(x,y)=xy then obviously you can hardcode a heuretic that just cranks up max x and max y and gets max f(x,y). That is the question is why should we tweak the input slightly not go all out.

midgetjones 12y ago

This was a fantastic read, thanks so much! Can anyone define 'pertubations' for me?

nekopa 12y ago
I think it means small changes, but there is also this: http://en.m.wikipedia.org/wiki/Perturbation_theory http://en.m.wikipedia.org/wiki/Perturbation_theory.

sathya_vj 12y ago

Thanks a lot for this!

jdminhbg 12y ago

I worked through the first few chapters on a flight yesterday and found this to be exactly what I need -- my ability to read code and sort of intuit alogrithms vastly exceeds my ability to read and understand math. Thanks for putting this together.

Elzair 12y ago

Thank you karpathy. This is a great introduction.