5 ms·

I think it's super important to separate the following two situations, both of which I suppose are fair to call overfitting Situation A: Models are slightly ov

I think it's super important to separate the following two situations, both of which I suppose are fair to call overfitting Situation A: Models are slightly overfit to some portions of the test set. But the following holds IF PerformanceOnBenchmark(Model A) > PerformanceOnBenchmark(Model B) Then PerformanceOnSimilarDaset(Model A) > PerformanceOnSimilarDataset(Model B) Therefore progress on the benchmark is predictive of progress in general. Situation B: The relation does not hold, and therefore progress on the benchmark does not predict general progress. This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down. If you look at figure 2 of the first paper you sent, you will note that it shows we are in situation A and not situation B. Situation A overfitting = diminishing returns on improvements on benchmark, but the benchmark is still useful. Situation B overfitting = the benchmark is now useless

sdenton4 5y ago
>> "This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down." This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy. I personally think image classification is mostly a silly problem to judge new algos on as a result, and leads to all kinds of nonsense as people try to extrapolate meaning from new results.
- dontreact 5y ago
  Nearly all useful machine learning is supervised, still. And if you are using a neural network, it will eventually memorize. This is fine though, we have early stopping :)
  - sdenton4 5y ago
    Eh, I work in audio ml, where most of the interesting and useful work is in conditioned generative models. Compression, TTS, source separation. The bias towards classification is a side effect of people starting at imagenet too long; I really think it's holding the field back.

sdenton4 5y ago

>> "This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down." This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy. I personally think image classification is mostly a silly problem to judge new algos on as a result, and leads to all kinds of nonsense as people try to extrapolate meaning from new results.

dontreact 5y ago
Nearly all useful machine learning is supervised, still. And if you are using a neural network, it will eventually memorize. This is fine though, we have early stopping :)
- sdenton4 5y ago
  Eh, I work in audio ml, where most of the interesting and useful work is in conditioned generative models. Compression, TTS, source separation. The bias towards classification is a side effect of people starting at imagenet too long; I really think it's holding the field back.