6 ms·
You can make a model of any size with deep learning. If your concerns are about over fitting there are lots of regularization techniques used in practice like
by emcq 6y ago
You can make a model of any size with deep learning.
If your concerns are about over fitting there are lots of regularization techniques used in practice like dropout, weight decay, and data augmentation.
There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that.
- danieldk 6y agoThere's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well. E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.
- auraham 6y agoYou can also train a neural network with a single shared weight parameter: https://weightagnostic.github.io/ https://weightagnostic.github.io/