5 ms·

But they don't really have that option. They're trapped in a Red Queen's race. The world keeps moving on, and so the models need to be retrained so that they c

But they don't really have that option. They're trapped in a Red Queen's race. The world keeps moving on, and so the models need to be retrained so that they can keep up with new information. Otherwise you'll get stuck with a model that only works well with information that existed prior to a dataset horizon that's receding into the past at a constant rate. At the same time, they have to keep iterating on the training process itself. AI generated text and code is slowly spreading across the internet. Model collapse is a real concern; they wouldn't be spending quite so much energy on buying and scanning rare books if it weren't. But for coding in particular expanding their corpus of old text is not really a good option because of the previous problem - no good training your LLM to write 1980 vintage K&R C that won't even compile on a modern compiler.

cj 24d ago
> they don't really have that option I imagine it must somehow be possible to update a model's understanding of recent events without training a completely new model from scratch?
- anthonypasq 24d ago
  yeah its called web search
ninahaberl 23d ago
Models don't need to keep retraining just to stay current. Harnesses give them access to the internet, internal systems use RAGs, and so on. I think lower-cost models will get the largest piece of the pie, as with almost everything that has ever been sold. Just look at cars: US consumers buy the F-150, EU consumers buy the freaking Dacia Sandero the most :))) Ferrari/Lambo numbers are microscopic
- zdragnar 23d ago
  Even with a harness, models don't reach out for new information they don't know about. For some tech, I have to have a local model draft a plan, then I have to adjust the plan to update it with the new API and references for where to find it. Even if I include that updated information in the prompt for the plan, the model says "what the user says is wrong, they probably meant this instead" and goes off in its own direction with old APIs anyway.
  - ninahaberl 23d ago
    You are basically saying that some models (your local one, which is it?) in some setups (the API you mentioned) can fail to use fresh information if that conflicts with strong training priors. I agree:) BUT That's a bad model. My opinion is that for exactly this case we need to use RAGs/ APIs/ some retrieval mechanisms. It's silly to train them on stuff that changes every week/month I don't learn APIs by heart, I look them up. It's to expensive (my time) for me and (the compute) for the models
    - zdragnar 23d ago
      I most recently experienced this with Qwen 3.8 27b, though I've seen it on several other versions of their local models. It's also heavily biased towards digging into library source code rather than looking at API documentation. To get it to the point of being remotely useful, I've had it start to write condensed fact blurbs into the agents.md file. It doubts itself so much and questions its every decision to the point that it'll literally blow the entire context on thinking alone in anything but the most basic CRUD projects otherwise. What an earlier generation model would just start doing, it went out to research the source code in multiple libraries just to see if what it was thinking would work... then it said "Hey, I should really just do it" then went back and started researching more anyway, on and on (even on medium thinking level). If there's a better local model for writing code, I'm all ears.
    - bunderbunder 23d ago
      For starters, it's not just APIs. Like I pointed out with the C example, programming language syntax and semantics also evolve over time. But also, LLMs' use of RAG to keep track of API evolution is limited. You can see this if you watch an agent at work using a well-known library that has a high rate of breaking changes such as Polars or Guava. There's a huge amount of churn on repeatedly writing code that works with an older version of the API and then diagnosing and fixing the resulting compile- or run-time errors. It can burn through quite a lot of tokens, which drives up usage costs. I agree that, all else being equal, using language model training to bake knowledge that's easy to look up into the system is kind of silly and inefficient. That's actually been one of my top complaints about hawking these LLMs as a sort of general-purpose AI. But the fact of the matter is that's fairly fundamental to how they work, and RAG is arguably just a hack on top of the basic design to paper over this limitation. RAG's limits become pretty easy to see when working in knowledge domains that aren't very publicly accessible, and therefore produce little text that would have been incorporated into the models' training corpora. It can be a bit of a, "Ignore that man behind the curtain!" experience. And no I'm not just talking about local models. I've seen it happen with recent GPT-5 and Claude Opus series models, too.
cosmojg 23d ago
Newer models[1] are being trained in ways that prioritize coding and agentic performance over raw knowledge[2] such that they increasingly rely on external tools for accessing hard data and information. [1] https://artificialanalysis.ai/evaluations/omniscience?models=qwen3-6-27b%2Cqwen3-8-27b#omniscience-accuracy-tabs https://artificialanalysis.ai/evaluations/omniscience?models... [2] https://old.reddit.com/r/LocalLLaMA/comments/1vt7l3e/qwen3827b_took_a_serious_hit_to_knowledge_vs_36/ https://old.reddit.com/r/LocalLLaMA/comments/1vt7l3e/qwen382...

cj 24d ago

> they don't really have that option I imagine it must somehow be possible to update a model's understanding of recent events without training a completely new model from scratch?

anthonypasq 24d ago
yeah its called web search

ninahaberl 23d ago

Models don't need to keep retraining just to stay current. Harnesses give them access to the internet, internal systems use RAGs, and so on. I think lower-cost models will get the largest piece of the pie, as with almost everything that has ever been sold. Just look at cars: US consumers buy the F-150, EU consumers buy the freaking Dacia Sandero the most :))) Ferrari/Lambo numbers are microscopic

zdragnar 23d ago
Even with a harness, models don't reach out for new information they don't know about. For some tech, I have to have a local model draft a plan, then I have to adjust the plan to update it with the new API and references for where to find it. Even if I include that updated information in the prompt for the plan, the model says "what the user says is wrong, they probably meant this instead" and goes off in its own direction with old APIs anyway.
- ninahaberl 23d ago
  You are basically saying that some models (your local one, which is it?) in some setups (the API you mentioned) can fail to use fresh information if that conflicts with strong training priors. I agree:) BUT That's a bad model. My opinion is that for exactly this case we need to use RAGs/ APIs/ some retrieval mechanisms. It's silly to train them on stuff that changes every week/month I don't learn APIs by heart, I look them up. It's to expensive (my time) for me and (the compute) for the models
  - zdragnar 23d ago
    I most recently experienced this with Qwen 3.8 27b, though I've seen it on several other versions of their local models. It's also heavily biased towards digging into library source code rather than looking at API documentation. To get it to the point of being remotely useful, I've had it start to write condensed fact blurbs into the agents.md file. It doubts itself so much and questions its every decision to the point that it'll literally blow the entire context on thinking alone in anything but the most basic CRUD projects otherwise. What an earlier generation model would just start doing, it went out to research the source code in multiple libraries just to see if what it was thinking would work... then it said "Hey, I should really just do it" then went back and started researching more anyway, on and on (even on medium thinking level). If there's a better local model for writing code, I'm all ears.
  - bunderbunder 23d ago
    For starters, it's not just APIs. Like I pointed out with the C example, programming language syntax and semantics also evolve over time. But also, LLMs' use of RAG to keep track of API evolution is limited. You can see this if you watch an agent at work using a well-known library that has a high rate of breaking changes such as Polars or Guava. There's a huge amount of churn on repeatedly writing code that works with an older version of the API and then diagnosing and fixing the resulting compile- or run-time errors. It can burn through quite a lot of tokens, which drives up usage costs. I agree that, all else being equal, using language model training to bake knowledge that's easy to look up into the system is kind of silly and inefficient. That's actually been one of my top complaints about hawking these LLMs as a sort of general-purpose AI. But the fact of the matter is that's fairly fundamental to how they work, and RAG is arguably just a hack on top of the basic design to paper over this limitation. RAG's limits become pretty easy to see when working in knowledge domains that aren't very publicly accessible, and therefore produce little text that would have been incorporated into the models' training corpora. It can be a bit of a, "Ignore that man behind the curtain!" experience. And no I'm not just talking about local models. I've seen it happen with recent GPT-5 and Claude Opus series models, too.

cosmojg 23d ago

Newer models[1] are being trained in ways that prioritize coding and agentic performance over raw knowledge[2] such that they increasingly rely on external tools for accessing hard data and information. [1] https://artificialanalysis.ai/evaluations/omniscience?models=qwen3-6-27b%2Cqwen3-8-27b#omniscience-accuracy-tabs https://artificialanalysis.ai/evaluations/omniscience?models... [2] https://old.reddit.com/r/LocalLLaMA/comments/1vt7l3e/qwen3827b_took_a_serious_hit_to_knowledge_vs_36/ https://old.reddit.com/r/LocalLLaMA/comments/1vt7l3e/qwen382...