May I add another method: block fine-pruning of transformers (pruning while fine-tuning) ? https://arxiv.org/abs/2109.04838 https://arxiv.org/abs/2109.04838 U

May I add another method: block fine-pruning of transformers (pruning while fine-tuning) ? https://arxiv.org/abs/2109.04838 https://arxiv.org/abs/2109.04838 Using blocks allows to keep good performence on GPUS, while giving some flexibility in the pruning pattern. And when removing entirely empty rows and columns the pruned matrices are actually pretty dense, so competitive with structured pruning for speedup, but less "aggressive" on the network during the pruning process. Disclaimer: I am the main co-author.