9 ms·
That could explain compute efficiency, but has nothing to do with the parameter efficiency pointed at in the paper.
by harles 2y ago
That could explain compute efficiency, but has nothing to do with the parameter efficiency pointed at in the paper.
- vlovich123 2y agoHaven’t read the paper but my guess around that is that the same reason sparse attention networks (where they 0 out many weights) just have the sparse tensors be larger.
- mayukhdeb 2y agoIn this paper, we don't zero out the weights. We remove them.
- vlovich123 2y agoThanks for the correction! Can it be retrofitted into existing models through distillation or do you have to train the model from scratch?