6 ms·

This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. I

This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses. The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups). The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.

woah 7y ago
On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques
- SQueeeeeL 7y ago
  Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make. Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
  - ssivark 7y ago
    Agree with your first statement and disagree with your second; I don’t think the former implies the latter. I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
    - btrettel 7y ago
      These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables. As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
- TrainedMonkey 7y ago
  Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.
  - cygaril 7y ago
    Or our entire evolutionary history of data.
    - AlanSE 7y ago
      ...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.
      - eanzenberg 7y ago
        I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon
    - K0SM0S 7y ago
      It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us. The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself. It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
- mendeza 7y ago
  I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation. Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reality_gap.pdf http://josh-tobin.com/assets/pdf/randomization_and_the_reali... http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_Tobin_RSS19.pdf http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T... And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText https://github.com/ankush-me/SynthText
  - tasogare 7y ago
    How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.
- gdubs 7y ago
  This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.
andreyk 7y ago
wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.
- K0SM0S 7y ago
  So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough. Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
  - yorwba 7y ago
    It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone. On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
  - andreyk 7y ago
    yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/ http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.
z3t4 7y ago
> colorizing b&w photos You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.

woah 7y ago

On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

SQueeeeeL 7y ago
Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make. Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
- ssivark 7y ago
  Agree with your first statement and disagree with your second; I don’t think the former implies the latter. I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
  - btrettel 7y ago
    These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables. As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
TrainedMonkey 7y ago
Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.
- cygaril 7y ago
  Or our entire evolutionary history of data.
  - AlanSE 7y ago
    ...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.
    - eanzenberg 7y ago
      I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon
  - K0SM0S 7y ago
    It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us. The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself. It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
mendeza 7y ago
I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation. Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reality_gap.pdf http://josh-tobin.com/assets/pdf/randomization_and_the_reali... http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_Tobin_RSS19.pdf http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T... And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText https://github.com/ankush-me/SynthText
- tasogare 7y ago
  How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.
gdubs 7y ago
This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.

andreyk 7y ago

wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.

K0SM0S 7y ago
So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough. Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
- yorwba 7y ago
  It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone. On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
- andreyk 7y ago
  yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/ http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.

z3t4 7y ago

> colorizing b&w photos You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.