4 ms·
Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists. See https:
by simonw 7d ago
Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.
See https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
- kees99 7d agoThis is exactly the problem, unfortunately. For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0]. Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are. [0] https://dumps.wikimedia.org/ https://dumps.wikimedia.org/
- nubinetwork 7d agoYou can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.
- perching_aix 7d agoIs there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.