I think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't wo

I think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't work too well... it definitely lost a lot of quality (for general language modeling) relative to the original model. See the paper here, Figure A28: https://kstatic.googleusercontent.com/files/b068c6c0e64d6f933068f7de30ea722359ef87c6c14d3065856b86d44fbdf2dea3ff373ed9eb751514f242d20df9d6a468622fad093f962563545e7d0cdb9dba https://kstatic.googleusercontent.com/files/b068c6c0e64d6f93... But if your downstream task is simple, like sequence classification, then it may be possible to compress the model without losing much quality.