6 ms·
The weights inside a model interact with each other in a way that's a bit more complex than just saying "forget documentation from these 300 open source product
by macksd 3y ago
The weights inside a model interact with each other in a way that's a bit more complex than just saying "forget documentation from these 300 open source products you scanned last week and replace that knowledge with these updates". You're talking about doing a pretty big training job for each update that really ought to be done with all current training data.
- owenversteeg 3y agoSure, it's not trivial, but it is not hard in comparison to the work done to create GPT-4 itself. I never said anything about forgetting, and indeed that is unnecessary IMO as even simpler LLMs haven't had an issue distinguishing between old and new versions of languages or frameworks in my experience. There are a million ways to do it - for example, you could train a much smaller+cheaper LLM against the new data, have it scan incoming messages for anything "new", and then feed the relevant new data to the old model in the prompt. You could make the new data available to the old model as an API. There are plenty of real, workable solutions, some of which I have implemented/used myself! - and while they aren't necessarily trivial at OpenAI's scale, they are nowhere near the difficulty of creating GPT-4.