4 ms·
As I live next to EPFL, I'll give you example from them: their Meditron-70B model is adapted to the medical domain from Llama-2-70B through continued pretrainin
by mirekrusin 18d ago
As I live next to EPFL, I'll give you example from them: their Meditron-70B model is adapted to the medical domain from Llama-2-70B through continued pretraining. They took weights of Llama-2-70B and continued training on PubMed, medical guidelines and general data.
Weights aren't just executable artifact that's consumed by users. Third parties actually use released parameter state as the editable starting point for further training and produce new foundation models from it.
- frabcus 17d agoNice example - although it seems it ended up specialised in medical texts, so did indeed ("catastrophically") forget other knowledge?
- mirekrusin 17d agoNo, it didn't. It lost 69.2% -> 67.8% on MMLU while improving medical performance. If you're trying to argue that loss of ~1.4 points is "catastrophic forgetting" (it's not) then look at later work, ie. Me-LLaMA that clearly demonstrates continued pretraining that improved both general MMLU and medical performance. Not sure why you're fixating on catastrophic forgetting. How do you think model training works? Model training is just a sequence of checkpoints: pretraining produces it, training resumes from last, continued pretraining starts from last, supervised fine tuning starts from last, RL/post-training starts from last - it's just a sequence of checkpoints. There isn't some fundamental distinction where original author continuing training from checkpoint X is training but a third party downloading checkpoint X and continuing training from it suddenly isn't. ie. checkpoint doesn't somehow become a different kind of artifact when it's published.