9 ms·

Generally speaking we have a handle on problems where you have lots of labeled data of the classes you want to identify (lots generally being quantified roughly

Generally speaking we have a handle on problems where you have lots of labeled data of the classes you want to identify (lots generally being quantified roughly by calculating N/(e^D) where N is number of entries and D is is the intrinsic dimensionality of the data. The larger that number is the more complex of a model you can train.). Some hard long term challenges revolve around cases where you don't have a lot of unlabeled data or examples of classes you care about. There's also the technical challenges of training large scale models without obscene computational resources. I also prefer to reserve the term AI for generalized AI, which we're still a ways off of, as opposed to modern classification problems etc. that I would call machine learning (though I know that nomenclature is uncommon). EDIT: jimfleming makes a great point about theory as well - we could likely be much more efficient with better theory for deep neural nets.

deleted 10y ago
[deleted]
tgflynn 10y ago
N/e^D would be vanishingly small for most problems on which deep learning is used. Image recognition for example may involve more than 1000 pixels and e^1000 is a number that makes the number of the atoms in the universe look tiny.
- emcq 10y ago
  The qualifier there is "intrinsic" dimensionality. There is smoothness to the image such that there is a lower dimensional embedding than 1m dimensions for a 1 megapixel image, particularly with any applied setting. That said there is a bigger problem with those bounds because it doesn't incorporate model complexity. The VC dimension is much more insightful because the complexity of your model and the hypothesis space it represents is important for proper training. As an example, add a regularization term to your model and you're no longer doing anything like N/e^D. Convolutions, dropout, etc all prevent NN models from becoming too complex to train.
  - tgflynn 10y ago
    Do you know of a way to measure "intrinsic dimensionality" ?
    - emcq 10y ago
      In general it's an abstract concept like Kolmogrov complexity but there are some practical approaches. People often try to intuit the intrinsic dimensionality of a dataset by using techniques like looking at singular values above some threshold or reconstruction error versus changing the output dimensionality of a dimensionality reduction/unsupervised technique like PCA, matrix factorization, or an autoencoder. An info theory person might argue entropy and compression ratios are also insightful.
      - TTPrograms 10y ago
        I'd also note that for the case of typical classification that there's an additional concept of "intrinsic boundary dimensionality" that should be low due to the presence of invariants like scale, rotation and translation which drastically lower the intrinsic boundary dimensionality.

deleted 10y ago

[deleted]

tgflynn 10y ago

N/e^D would be vanishingly small for most problems on which deep learning is used. Image recognition for example may involve more than 1000 pixels and e^1000 is a number that makes the number of the atoms in the universe look tiny.

emcq 10y ago
The qualifier there is "intrinsic" dimensionality. There is smoothness to the image such that there is a lower dimensional embedding than 1m dimensions for a 1 megapixel image, particularly with any applied setting. That said there is a bigger problem with those bounds because it doesn't incorporate model complexity. The VC dimension is much more insightful because the complexity of your model and the hypothesis space it represents is important for proper training. As an example, add a regularization term to your model and you're no longer doing anything like N/e^D. Convolutions, dropout, etc all prevent NN models from becoming too complex to train.
- tgflynn 10y ago
  Do you know of a way to measure "intrinsic dimensionality" ?
  - emcq 10y ago
    In general it's an abstract concept like Kolmogrov complexity but there are some practical approaches. People often try to intuit the intrinsic dimensionality of a dataset by using techniques like looking at singular values above some threshold or reconstruction error versus changing the output dimensionality of a dimensionality reduction/unsupervised technique like PCA, matrix factorization, or an autoencoder. An info theory person might argue entropy and compression ratios are also insightful.
    - TTPrograms 10y ago
      I'd also note that for the case of typical classification that there's an additional concept of "intrinsic boundary dimensionality" that should be low due to the presence of invariants like scale, rotation and translation which drastically lower the intrinsic boundary dimensionality.