5 ms·

How I Became a Machine Learning Practitioner

290 points by sama 7y ago

minimaxir 7y ago
Given that Greg Brockman was the CTO of Stripe before OpenAI, that's a order of magnitude more technically/CS capable than the typical reader who might be looking into ML.
- copperx 7y ago
  Studying Math at Harvard/MIT certainly puts you in a different category than the average software engineer. And if ML was still challenging to Greg, it is honestly a bit discouraging.
  - gdb 7y ago
    (I wrote the post.) If it's helpful, I dropped out of both schools — the vast majority of my knowledge is self taught!
    - jcims 7y ago
      >I learn best when I have something specific in mind to build. This is so incredibly important for me and, based on my conversations, many others as well. The other thing I struggle with is the feeling that many of the problems I wish to solve are likely also solvable with simpler statistical methods and that I'm just being a poser by trying to pound them home with the ML hammer.
    - applecrazy 7y ago
      Question: will the OpenAI fellows curriculum ever be released? I need a nice, structured intro to deep learning research and feel like the curriculum modules your company has developed would have the highest quality. (For reference, I’m an undergrad looking to get into this field)
    - knbknb 7y ago
      How often do you go "back to basics" (read introductory book chapters, or classic papers, etc)?
  - streetcat1 7y ago
    Maybe if you want to write tensorflow, but not if you want to use it.
    - eanzenberg 7y ago
      If you want to write TF, then you are a ml engineer. If you want to use TF, then you are a scientist (data, research, whatever)
  - omarhaneef 7y ago
    I think "doing" ML math is a little different than "doing" math math. With math, on paper say, it is hard to tell if you are doing it right or wrong. You can trick yourself quite easily. A compelling proof can have a huge hole. You can still trick yourself programming -- in a sense, that is what a bug is -- but it is much harder. The upshot is, I think it is easier to teach yourself math that is applied to a computer program than math on a piece of paper.
  - dhairya 7y ago
    So there is a difference between ML research and application. Being a practitioner doesn't require deep math knowledge that perhaps research would. Jeremy Howard's fastai course is a great example of how someone with a solid programming background can effectively transition into being a deep learning practitioner. Given that production ml and deep learning is still the wild west, as a practioner, you can contribute also to the research around effective training, scaling, and application of these models. The math and intuition required are definitely acquirable. I think when you shift into pure research, yes a deep probability, information theory, linear algebra, and calculus background are needed. But at the level, you're rarely writing code and more likely working at theoretical level.
    - jph00 7y ago
      Also, note that Greg's goal was to contribute to OpenAI's flagship project. That's a rather ambitious goal! Also, most folks I know that are making practical deep learning contributions are doing so by combining their pre-existing domain expertise with their new deep learning skills. E.g. a journalist analyzing a large corpus of text for a story, or an oil&gas analyst building models from well plots, etc.
      - dhairya 7y ago
        as a side note, I love that you highlight regex in your new NLP course. There is an inherent tension between the probabilistic nature of models and the need for deterministic outputs in most production settings. Often if we can uncover linguistics rules or regex patterns that guarantee minimal precision (or as our VP puts it - don't look stupid), we'll eschew the model in the short term or use the model to augment the rules. Also I really appreciated that on of the training goals for ULMfit was to be trainable on a single gpu. With these large-capacity models, training is getting crazy expensive and out of hand. Any chance that your future work will still keep the single gpu training goal?
    - sevensor 7y ago
      I recently got the assignment to "do ML" on some data. I hadn't done anything in the area before, and a couple of things surprised me: 1. Most of your time is spent transforming data. Very little is spent building models. 2. Most of the eye-grabbing stuff that makes headlines is inapplicable. My application involves decisions that are expensive and can be safety critical. The models themselves have to be simple enough to be reasoned about, or they're no use. You might argue that this means what I'm actually doing is statistics.
- dhairya 7y ago
  Yes but the transition definitely doable and his advice is great. The key part of his advice is spending time experimenting, rapidly failing, and continuing to work on it with real world use case. Often the challenge is making the jump from the simple toy examples used in educational materials to the messiness of real-world data. I'm a senior data scientist at vc-back startup. I'm in a hybrid data scientist/ machine learning engineering role, where I build and train ml and deep learning models and also build the scaffolding them to support their production usage. But my previous roles included being a business analyst, project manager, and research analyst. My undergrad education was in Creative Writing and the social sciences. While I kind of accidentally transitioned into this career, how I got here is similar to most folks coming from a different background. Lot's of self-study and experimentation. I think one of the challenges to transitioning into ML and deep learning is that there are so many applications, domains, and input formats. It can be overwhelming to learn about vision, nlp, tabular, time-series and all other formats, applications and domains. Things solidified for me when I found a space I found compelling and I was able to dive deep into it. You kind of learn the fundamentals along the way through experimentations and reflection. My pattern was pick up a model or architecture. Learn to apply it first to get familiar with it, experiment with different data, and then go back to build it from scratch to learn the fundamentals. That and I read a lot of papers related to problems I was interested in. After a while, I started developing intuitions around classes of problems and how to engage them (in DS you rarely ever solve the problem, there's always room to improve the model ...)
  - streetcat1 7y ago
    Thanks for the info. I have a serious question (not for bashing) Can you please describe what part of your job CANNOT be automated?
    - dhairya 7y ago
      No worries, fair question. It worth noting that my job is not data analysis. So I do use data analysis to evaluate our metrics and model performance. Really none of it is really automatable. I'm working developing NLP features for our product (question answering, search, neural machine translation, dialog, etc). Our customer data is diverse, in different formats, and thier use cases are all distinct. So most of my work is novel applied research and development.
- sgt101 7y ago
  We know that someone with a good CS degree can do this because.. Ph.D students....
jorblumesea 7y ago
Does anyone not want to become a ML engineer? Is this the future, and will we even have a choice or else be out of a job?
- frenchman99 7y ago
  Well, you could save money and use it to feed yourself while learn new skills when you're out of a job. That works too.
- klipt 7y ago
  There's plenty of non ML software engineering to be done. Anecdotally a large proportion of interns want an "ML project", but only a small percentage of teams looking for interns are offering one. Too many people going into ML could skew the supply/demand into making it a worse job option (more work, less pay), like game programming or academia.
  - seanmcdirmid 7y ago
    There aren’t that many engineers with 10+ years of ML experience ATM, but there will be tons in 5 or so years. Chasing tomorrow’s trend is always more productive than chasing today’s, but of course the former requires predicting the future.
  - rhacker 7y ago
    The question is this: 10 years from now when the top job requirements list ML - are you going to be ready or out of the game?
    - carlmr 7y ago
      Alternatively the market is flooded with ML people and the value of ML on your CV is 0.
      - deleted 7y ago
        [deleted]
- 7y ago
mychael 7y ago
Congrats on your cool life, your ivy league education, your CTO role at OpenAI and all the access that provides. You've done it! Also thanks for telling us how you became a practitioner. It's definitely relatable and not a humble brag at all.
- strikelaserclaw 7y ago
  Jealousy is unbecoming. I had no idea who this guy was and i found the article nice to read. It is a story about a guy who feels out of his depth in a highly technical domain but instead of giving up, he devotes time to it and comes out the other end much more competent. The internet provides everyone with world class experts to help you if you are stuck on something, most people don't have the will to help themselves.
- ilyasut 7y ago
  Comments like this are what make HN great and are not at all toxic!
TrackerFF 7y ago
I think it's doable if you're at the right place and have enough opportunities around you, and something to show for. I feel that companies and startups around tech hubs are more willing to give someone a chance, and look through the formality, if you manage to convince / impress them. Where I live, far away from tech, it's almost impossible to land a job in ML / AI / DS unless you have a (minimum) Masters degree in something relevant. Preferably a Ph.D and solid experience to show for - I know because I work in the field, and lots of F500 dinosaurs are just now waking up. But are also unfortunately clinging to their old ways of hiring people. Schools all over are also picking up slack, starting to offer specialized graduate degrees in those domains. When I got my degree in ML, it was a sub-field at my schools engineering department, mixed up with signal processing and control theory groups. When first trying to get a job, the main problem was to explain what I actually could bring and do, and a lot of the recruiters or managers had no idea what Machine Learning was. Then you said "It's basically Artificial Intelligence" and, and they were instantly wooed.
- rolltiide 7y ago
  Your observation is correct. No degrees are needed there is a lot of stuff to build.
croh 7y ago
Indeed inspirational article.

290 points by sama 7y ago

Given that Greg Brockman was the CTO of Stripe before OpenAI, that's a order of magnitude more technically/CS capable than the typical reader who might be looking into ML.

copperx 7y ago
Studying Math at Harvard/MIT certainly puts you in a different category than the average software engineer. And if ML was still challenging to Greg, it is honestly a bit discouraging.
- gdb 7y ago
  (I wrote the post.) If it's helpful, I dropped out of both schools — the vast majority of my knowledge is self taught!
  - jcims 7y ago
    >I learn best when I have something specific in mind to build. This is so incredibly important for me and, based on my conversations, many others as well. The other thing I struggle with is the feeling that many of the problems I wish to solve are likely also solvable with simpler statistical methods and that I'm just being a poser by trying to pound them home with the ML hammer.
  - applecrazy 7y ago
    Question: will the OpenAI fellows curriculum ever be released? I need a nice, structured intro to deep learning research and feel like the curriculum modules your company has developed would have the highest quality. (For reference, I’m an undergrad looking to get into this field)
  - knbknb 7y ago
    How often do you go "back to basics" (read introductory book chapters, or classic papers, etc)?
- streetcat1 7y ago
  Maybe if you want to write tensorflow, but not if you want to use it.
  - eanzenberg 7y ago
    If you want to write TF, then you are a ml engineer. If you want to use TF, then you are a scientist (data, research, whatever)
- omarhaneef 7y ago
  I think "doing" ML math is a little different than "doing" math math. With math, on paper say, it is hard to tell if you are doing it right or wrong. You can trick yourself quite easily. A compelling proof can have a huge hole. You can still trick yourself programming -- in a sense, that is what a bug is -- but it is much harder. The upshot is, I think it is easier to teach yourself math that is applied to a computer program than math on a piece of paper.
- dhairya 7y ago
  So there is a difference between ML research and application. Being a practitioner doesn't require deep math knowledge that perhaps research would. Jeremy Howard's fastai course is a great example of how someone with a solid programming background can effectively transition into being a deep learning practitioner. Given that production ml and deep learning is still the wild west, as a practioner, you can contribute also to the research around effective training, scaling, and application of these models. The math and intuition required are definitely acquirable. I think when you shift into pure research, yes a deep probability, information theory, linear algebra, and calculus background are needed. But at the level, you're rarely writing code and more likely working at theoretical level.
  - jph00 7y ago
    Also, note that Greg's goal was to contribute to OpenAI's flagship project. That's a rather ambitious goal! Also, most folks I know that are making practical deep learning contributions are doing so by combining their pre-existing domain expertise with their new deep learning skills. E.g. a journalist analyzing a large corpus of text for a story, or an oil&gas analyst building models from well plots, etc.
    - dhairya 7y ago
      as a side note, I love that you highlight regex in your new NLP course. There is an inherent tension between the probabilistic nature of models and the need for deterministic outputs in most production settings. Often if we can uncover linguistics rules or regex patterns that guarantee minimal precision (or as our VP puts it - don't look stupid), we'll eschew the model in the short term or use the model to augment the rules. Also I really appreciated that on of the training goals for ULMfit was to be trainable on a single gpu. With these large-capacity models, training is getting crazy expensive and out of hand. Any chance that your future work will still keep the single gpu training goal?
  - sevensor 7y ago
    I recently got the assignment to "do ML" on some data. I hadn't done anything in the area before, and a couple of things surprised me: 1. Most of your time is spent transforming data. Very little is spent building models. 2. Most of the eye-grabbing stuff that makes headlines is inapplicable. My application involves decisions that are expensive and can be safety critical. The models themselves have to be simple enough to be reasoned about, or they're no use. You might argue that this means what I'm actually doing is statistics.
dhairya 7y ago
Yes but the transition definitely doable and his advice is great. The key part of his advice is spending time experimenting, rapidly failing, and continuing to work on it with real world use case. Often the challenge is making the jump from the simple toy examples used in educational materials to the messiness of real-world data. I'm a senior data scientist at vc-back startup. I'm in a hybrid data scientist/ machine learning engineering role, where I build and train ml and deep learning models and also build the scaffolding them to support their production usage. But my previous roles included being a business analyst, project manager, and research analyst. My undergrad education was in Creative Writing and the social sciences. While I kind of accidentally transitioned into this career, how I got here is similar to most folks coming from a different background. Lot's of self-study and experimentation. I think one of the challenges to transitioning into ML and deep learning is that there are so many applications, domains, and input formats. It can be overwhelming to learn about vision, nlp, tabular, time-series and all other formats, applications and domains. Things solidified for me when I found a space I found compelling and I was able to dive deep into it. You kind of learn the fundamentals along the way through experimentations and reflection. My pattern was pick up a model or architecture. Learn to apply it first to get familiar with it, experiment with different data, and then go back to build it from scratch to learn the fundamentals. That and I read a lot of papers related to problems I was interested in. After a while, I started developing intuitions around classes of problems and how to engage them (in DS you rarely ever solve the problem, there's always room to improve the model ...)
- streetcat1 7y ago
  Thanks for the info. I have a serious question (not for bashing) Can you please describe what part of your job CANNOT be automated?
  - dhairya 7y ago
    No worries, fair question. It worth noting that my job is not data analysis. So I do use data analysis to evaluate our metrics and model performance. Really none of it is really automatable. I'm working developing NLP features for our product (question answering, search, neural machine translation, dialog, etc). Our customer data is diverse, in different formats, and thier use cases are all distinct. So most of my work is novel applied research and development.
sgt101 7y ago
We know that someone with a good CS degree can do this because.. Ph.D students....

jorblumesea 7y ago

Does anyone not want to become a ML engineer? Is this the future, and will we even have a choice or else be out of a job?

frenchman99 7y ago
Well, you could save money and use it to feed yourself while learn new skills when you're out of a job. That works too.
klipt 7y ago
There's plenty of non ML software engineering to be done. Anecdotally a large proportion of interns want an "ML project", but only a small percentage of teams looking for interns are offering one. Too many people going into ML could skew the supply/demand into making it a worse job option (more work, less pay), like game programming or academia.
- seanmcdirmid 7y ago
  There aren’t that many engineers with 10+ years of ML experience ATM, but there will be tons in 5 or so years. Chasing tomorrow’s trend is always more productive than chasing today’s, but of course the former requires predicting the future.
- rhacker 7y ago
  The question is this: 10 years from now when the top job requirements list ML - are you going to be ready or out of the game?
  - carlmr 7y ago
    Alternatively the market is flooded with ML people and the value of ML on your CV is 0.
    - deleted 7y ago
      [deleted]
7y ago

mychael 7y ago

Congrats on your cool life, your ivy league education, your CTO role at OpenAI and all the access that provides. You've done it! Also thanks for telling us how you became a practitioner. It's definitely relatable and not a humble brag at all.

strikelaserclaw 7y ago
Jealousy is unbecoming. I had no idea who this guy was and i found the article nice to read. It is a story about a guy who feels out of his depth in a highly technical domain but instead of giving up, he devotes time to it and comes out the other end much more competent. The internet provides everyone with world class experts to help you if you are stuck on something, most people don't have the will to help themselves.
ilyasut 7y ago
Comments like this are what make HN great and are not at all toxic!

TrackerFF 7y ago

I think it's doable if you're at the right place and have enough opportunities around you, and something to show for. I feel that companies and startups around tech hubs are more willing to give someone a chance, and look through the formality, if you manage to convince / impress them. Where I live, far away from tech, it's almost impossible to land a job in ML / AI / DS unless you have a (minimum) Masters degree in something relevant. Preferably a Ph.D and solid experience to show for - I know because I work in the field, and lots of F500 dinosaurs are just now waking up. But are also unfortunately clinging to their old ways of hiring people. Schools all over are also picking up slack, starting to offer specialized graduate degrees in those domains. When I got my degree in ML, it was a sub-field at my schools engineering department, mixed up with signal processing and control theory groups. When first trying to get a job, the main problem was to explain what I actually could bring and do, and a lot of the recruiters or managers had no idea what Machine Learning was. Then you said "It's basically Artificial Intelligence" and, and they were instantly wooed.

rolltiide 7y ago
Your observation is correct. No degrees are needed there is a lot of stuff to build.

croh 7y ago

Indeed inspirational article.