6 ms·

I seriously doubt that data set poisoning will be a real limiter in model performance. For one, if your website/book is poisoned, who is going to trust it for

I seriously doubt that data set poisoning will be a real limiter in model performance. For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models? For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc. Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.

microgpt 3mo ago
Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
- thfuran 3mo ago
  >Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
  - microgpt 3mo ago
    yeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't. and browsers forcibly auto-update
Analemma_ 3mo ago
I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.
- tayo42 3mo ago
  Who's doing llm seo right now? How does that work when you only gets feedback every few months when a new model is out?
  - natebc 3mo ago
    I'm pretty sure the Optimization part is just ... not present at all. This is how we get LLM summaries presenting something mentioned once by some nutjob in a reddit thread as bona fide FACT
  - DougN7 3mo ago
    Look at G2.com - they found their website is highly references by AIs and they are leaning into it hard.
- aspenmartin 3mo ago
  Data curation is important and expensive and frontier labs can afford to do it right. Natural data isn't the limitation, we are already literally out of tokens. It doesn't matter how much you poison things it's not going to stop the progress train.
chmod775 3mo ago
There's a post every other month where some dude who put nonsense information online celebrates because it actually ended up in some frontier models weights. If it's easy enough that some randos can do it for fun, what do you think happens when there's commercial interest behind it? Obviously companies are going try nudging AI towards recommending whatever they're selling. It's a logical extension of SEO - and that's a 100 billion USD industry. Additionally, if I believed myself to be in some sort of spending - err - AI race, I'd try to poison the data sets of my competitors by putting crap out there for others to ingest.
- aspenmartin 3mo ago
  It's not really a problem. We're out of natural tokens anyway. The future is synthetic verifiable traces (already the way we train coding agents).
  - maxnevermind 3mo ago
    > synthetic verifiable traces What does it mean, Is it like when somebody used some coding agent to develop a feature and later input prompts and a resulting PR can be used for training by a presumption that final PR was a correct implementation of a prompt?
    - aspenmartin 3mo ago
      Yea it’s rejection sampling, so you have an agent, you take a verifiable problem (people use lots of different verification signals but say unit tests etc) and have the agent attempt it K times. You accept the trajectories (all context, tool use etc, the entire log) that are positively verified and use these as training examples. The trick is to find the examples that are just in between too difficult and too easy for the existing agent, these have the strongest training signals
- jurgenaut23 3mo ago
  Do you have examples of such celebrations?
- 3mo ago

microgpt 3mo ago

Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.

thfuran 3mo ago
>Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
- microgpt 3mo ago
  yeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't. and browsers forcibly auto-update

Analemma_ 3mo ago

I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.

tayo42 3mo ago
Who's doing llm seo right now? How does that work when you only gets feedback every few months when a new model is out?
- natebc 3mo ago
  I'm pretty sure the Optimization part is just ... not present at all. This is how we get LLM summaries presenting something mentioned once by some nutjob in a reddit thread as bona fide FACT
- DougN7 3mo ago
  Look at G2.com - they found their website is highly references by AIs and they are leaning into it hard.
aspenmartin 3mo ago
Data curation is important and expensive and frontier labs can afford to do it right. Natural data isn't the limitation, we are already literally out of tokens. It doesn't matter how much you poison things it's not going to stop the progress train.

chmod775 3mo ago

There's a post every other month where some dude who put nonsense information online celebrates because it actually ended up in some frontier models weights. If it's easy enough that some randos can do it for fun, what do you think happens when there's commercial interest behind it? Obviously companies are going try nudging AI towards recommending whatever they're selling. It's a logical extension of SEO - and that's a 100 billion USD industry. Additionally, if I believed myself to be in some sort of spending - err - AI race, I'd try to poison the data sets of my competitors by putting crap out there for others to ingest.

aspenmartin 3mo ago
It's not really a problem. We're out of natural tokens anyway. The future is synthetic verifiable traces (already the way we train coding agents).
- maxnevermind 3mo ago
  > synthetic verifiable traces What does it mean, Is it like when somebody used some coding agent to develop a feature and later input prompts and a resulting PR can be used for training by a presumption that final PR was a correct implementation of a prompt?
  - aspenmartin 3mo ago
    Yea it’s rejection sampling, so you have an agent, you take a verifiable problem (people use lots of different verification signals but say unit tests etc) and have the agent attempt it K times. You accept the trajectories (all context, tool use etc, the entire log) that are positively verified and use these as training examples. The trick is to find the examples that are just in between too difficult and too easy for the existing agent, these have the strongest training signals
jurgenaut23 3mo ago
Do you have examples of such celebrations?
3mo ago