12 ms·
A while ago a lot of the discussion about overparameterization was about explaining "double descent", the observation that test error doesn't descend monotonica
by getnormality 3mo ago
A while ago a lot of the discussion about overparameterization was about explaining "double descent", the observation that test error doesn't descend monotonically and actually hits a local maximum around the point where the model has just enough parameters to interpolate the data. My favorite article about double descent looks at this in terms of splines [1]. If I can try to summarize that article: when you are designing a parametrized model to fit to data, you have a choice. You can either:
1. Avoid overparameterization by design. Manually create or choose a space of functions that has limited degrees of freedom by construction.
2. Accept overparameterization and regularize.
The latter tends to be more robust, because of the bitter lesson. It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. The best way to approach that ideal, it turns out, is really to just let the computer figure it out via regularized optimization over an overparameterized space.
Statisticians started moving in favor of overparameterization long before deep learning got off the ground. This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001).
[1] https://mlu-explain.github.io/double-descent/ https://mlu-explain.github.io/double-descent/
- porridgeraisin 3mo agoHi, I work on RL, or as it is known today, "classical" RL. I'm interested in knowing the latest work that explains double descent and in general optimisation behaviour of overparameterized neural networks. Do you have a survey paper or blog post or anything else to recommend?
- schmuhblaster 3mo ago> This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001). Could you elaborate on this?
- disgruntledphd2 3mo agoI was also confused at first. I think what OP meant is that ESL talks a lot about regularization but doesn't mention over parametrization much. However, most of their example datasets are small, and they use complicated models, so it's sort of implicit. They definitely don't mention double descent though.
- skybrian 3mo ago> It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. What if that could be automated?
- rbanffy 3mo agoThe process would need to have some knowledge of the desired outcome, much like a human expert would have a hunch of the design decisions to make.
- Scene_Cast2 3mo agoNot necessarily. I'm a proponent of (admittedly not very popular) methodology of "train, do interpretability analysis, adjust model architecture". It's not more popular for a few reasons: 1) you first need to train a full general model anyhow 2) interpretability is nontrivial and not guaranteed 3) once you make the architectural changes, you can't commit to that architecture as you might miss out in the future with more advancements 4) with modern transformers, there is limited amount of architectural "play" happening.
- SiempreViernes 3mo ago> [Regularisation] tends to be more robust, because of the bitter lesson. This seems circular? Over-parametrisation and stochastic gradient decent provides automatic regularisation, and it turns out this regularisation is very good in practice. The bitter lesson is simply the a restatement of the empirical observation that this specific regularisation often works great, it doesn't come with any explanation.
- deleted 3mo ago[deleted]