5 ms·
Microsoft has been releasing LLMs for years.
by ComputerGuru 4mo ago
Microsoft has been releasing LLMs for years.
- ipsum2 4mo agoSort of. Phi models were just trained on GPT outputs though.
- not_a_bot_4sho 4mo agoBy design. The whole point of Phi is the "textbooks is all you need" theory on curated training data, as opposed to kitchen sinks.
- kingstnap 4mo agoFor those that don't know about this. Phi was announced with a paper called "Textbooks are all you need". What they did was use GPT 3.5 and created synthetic textbook chapters and exercises. They also did some more interesting work like showing very small models can be coherent as long as you have very simple children's book style training data (TinyStories is pretty famous). Lots of these ideas are still used. Learning facts at scale with active reading is an ICLR 2026 paper from Meta AI that does a lot of similar work.
- jwitthuhn 4mo agoAnd occasionally un-releasing them like with WizardLM.
- lemonish97 4mo agoThey were mostly distilled or fine-tuned OAI models.