5 ms·

GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still goin

by avarun 2mo ago

GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.

linkregister 2mo ago
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
zamadatix 2mo ago
The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.
Wowfunhappy 2mo ago
Where are they getting new data from at this point? Didn't they already read the entire internet?
- redanddead 2mo ago
  Custom sets and mining their own users, as every lab does
  - bevekspldnw 2mo ago
    Custom what though? And user queries are not a knowledge repository it’s an ignorance repo.
- krackers 2mo ago
  A lot of data these days is synthetically generated. As an example, to make a model good at understanding assembly you simply need to round-trip code through a compiler and disassembler, then train against the source of truth and the assembly. You can generate arbitrary algebra expressions and have it solve it. A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL
bevekspldnw 2mo ago
Where did they get a second copy of all books, written/audio/visual media, and the web? Couch cushions?

by avarun 2mo ago

GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.

linkregister 2mo ago

Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.

zamadatix 2mo ago

The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.

Wowfunhappy 2mo ago

Where are they getting new data from at this point? Didn't they already read the entire internet?

redanddead 2mo ago
Custom sets and mining their own users, as every lab does
- bevekspldnw 2mo ago
  Custom what though? And user queries are not a knowledge repository it’s an ignorance repo.
krackers 2mo ago
A lot of data these days is synthetically generated. As an example, to make a model good at understanding assembly you simply need to round-trip code through a compiler and disassembler, then train against the source of truth and the assembly. You can generate arbitrary algebra expressions and have it solve it. A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL

bevekspldnw 2mo ago

Where did they get a second copy of all books, written/audio/visual media, and the web? Couch cushions?