6 ms·

Yes, we can't block models from scraping the public repos and train on them, but we won't use your code (neither public nor private) to train models on them.

Yes, we can't block models from scraping the public repos and train on them, but *we* won't use your code (neither public nor private) to train models on them. We really only want to host your code and do that well.

msdz 12d ago
> we can't block models from scraping the public repos FYI, there is the nuke option, which is generating endless nonsense pages as a form of “bot sink” [1] that ends the scraping relatively quickly, but IIRC that also tanks your search rankings, since it likely affects benign crawlers too – not something you’d necessarily want to happen to a new domain, unless you really need to protect server resources against aggressive hostile crawlers like SourceHut and so many others had to combat. [1] https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrapers https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrape...
- NotHowItWorks 12d ago
  Facepalm
NotHowItWorks 12d ago
LLMs are read only. A developer would be the one using the LLM to evaluate page response text and scraping what they need. LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods.
- xienze 11d ago
  > LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods. Being a bit needlessly pedantic here. I think it's well understood in these circles that such models are being driven by agents, which absolutely can crawl and scrape. The only thing a human needs to do is write rules for the agent to follow.
  - YoureWrong23 11d ago
    No, it allows people to dodge accountability. It’s also not true. An LLM cannot visit the internet at all. There is no “agent” - that is a marketing term. It’s software and LLM that’s it.
    - 9763268964 11d ago
      There is no "software" - that is a marketing term. It's bits and that's it.
      - deleted 11d ago
        [deleted]
      - YoureWrong23 11d ago
        Not how words work, try calling it bits in any setting where you mean software. Wouldn’t matter in court anyway or anywhere you put it to the test even if you called it bits. Point is dodging accountability by saying some “AI did it” when it’s not reflective of reality. LLMs are read only and have no access to the internet :)

Yes, we can't block models from scraping the public repos and train on them, but *we* won't use your code (neither public nor private) to train models on them.

msdz 12d ago

> we can't block models from scraping the public repos FYI, there is the nuke option, which is generating endless nonsense pages as a form of “bot sink” [1] that ends the scraping relatively quickly, but IIRC that also tanks your search rankings, since it likely affects benign crawlers too – not something you’d necessarily want to happen to a new domain, unless you really need to protect server resources against aggressive hostile crawlers like SourceHut and so many others had to combat. [1] https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrapers https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrape...

NotHowItWorks 12d ago
Facepalm

NotHowItWorks 12d ago

LLMs are read only. A developer would be the one using the LLM to evaluate page response text and scraping what they need. LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods.

xienze 11d ago
> LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods. Being a bit needlessly pedantic here. I think it's well understood in these circles that such models are being driven by agents, which absolutely can crawl and scrape. The only thing a human needs to do is write rules for the agent to follow.
- YoureWrong23 11d ago
  No, it allows people to dodge accountability. It’s also not true. An LLM cannot visit the internet at all. There is no “agent” - that is a marketing term. It’s software and LLM that’s it.
  - 9763268964 11d ago
    There is no "software" - that is a marketing term. It's bits and that's it.
    - deleted 11d ago
      [deleted]
    - YoureWrong23 11d ago
      Not how words work, try calling it bits in any setting where you mean software. Wouldn’t matter in court anyway or anywhere you put it to the test even if you called it bits. Point is dodging accountability by saying some “AI did it” when it’s not reflective of reality. LLMs are read only and have no access to the internet :)