As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplifie

As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning usually meant freezing most of a model and tuning a small layer on top of it. (ELMo also fits somewhere in here, being a kind of halfway step). ULMFiT was a relatively lesser known work but to my knowledge one of the first to do 1) LM pretraining and 2) fine-tuning all the layers, albeit with some complexity (gradual unfreezing/different learning rates for layers). GPT simplified this massively by simply tuning all weights: no nuance about it. (Also added a classifier on top). BERT took this a step further, benefiting from the larger size of BERT-large and bidirectional attention, which works really well on the NLU datasets of the time along with a classifier head. (It's not until T5 that seq2seq for general tasks became prominent again.) The idea that you tune all the weights of what was then considered a massive model was considered somewhat excessive at the time, but BERT so massively dominated every other architecture and approach at the time that everyone switched over to it. (Adapters (alongside PAL), one of the first parameter-efficient tuning methods in the Transformer era of NLP, came out shortly after.)