irodov_rg

10 ms·

1. 1.▲
   by irodov_rg 8y ago

   Training adjusts all whole net's weights. And you are right, we get the joint changes and decompression machinery to work together.
2. 2.▲
   by irodov_rg 8y ago

   This is shown in one of the plots, the less you compress the less you loose. While there is some analysis in the paper on how the computations reduce etc, the results are mostly emperical.
3. 3.▲
   by irodov_rg 8y ago

   Its mostly the lookup table which takes up the most space. This work is about breaking it into 2 layers and continuing to train to gain accuracy. The output model becomes 90% smaller compared to the original model.
4. 4.▲
   by irodov_rg 8y ago

   BERT is more computationally expensive. It might end up giving better results on the task mentioned in the paper but we don't know. At the time of writing this all of the contextual word embedding techniques were fairly new and were no