5 ms·

I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?

by tescreal 7d ago

SoftTalker 7d ago
Could be testing in preparation for attacking something more critical?
- lanyard-textile 7d ago
  Maybe just for the pleasure of doing it, too.
davidfischer 7d ago
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did. The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.
- SoftTalker 7d ago
  Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
  - davidfischer 7d ago
    It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate. It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo. [1] https://docs.readthedocs.com/platform/stable/reference/markdown-for-agents.html https://docs.readthedocs.com/platform/stable/reference/markd...
  - simonw 7d ago
    Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists. See https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
    - kees99 7d ago
      This is exactly the problem, unfortunately. For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0]. Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are. [0] https://dumps.wikimedia.org/ https://dumps.wikimedia.org/
      - nubinetwork 7d ago
        You can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.
    - perching_aix 7d ago
      Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.
- tescreal 7d ago
  Good to know. I use your site (with a manual transmission user-agent) often, and it's fantastic. Thanks for your work and the writeup!
  - davidfischer 7d ago
    I've never seen the phrase "manual transmission user-agent". Using your own browser yourself is the new stick shift. Love it.
    - gopher_space 7d ago
      It feels like everyone's rebuilding their own desktop experience. Kind of Minecraft with folders and text files. The really interesting part of this is how little people talk about what they're doing, and it doesn't feel secretive in any way.
      - tescreal 7d ago
        What do you mean exactly?
      - Cilvic 7d ago
        I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
        gopher_space 6d ago
        Just like you say. It's not a secret business plan, it just doesn't really feel useful to talk about.
kkapelon 7d ago
Either testing for something bigger OR demonstrating their power to a 3rd party with minimal real disruption
RobRivera 7d ago
Could have been a live-fire exercise by a nation state. Edit: why the down vote? That is literally in the realm of possibility!
- mitxela 6d ago
  Due to mandatory scheduler maintenance, this test has been replaced with a live-fire CORS designed for military Androids. If you are an iPhone user, please proceed quickly to the chamber lock.
gkoberger 7d ago
I run a similar service, and we get almost daily attacks like this. Sometimes it's a specific high-profile customer, other times it's broader. I can't speak for RTD, but I think it's less "documentation site" and more just that we sit on the domains of high-profile products and the tools are just looking for any hole they can find? Often it's even the company themselves, for whatever reason (security research, etc).

I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?

SoftTalker 7d ago

Could be testing in preparation for attacking something more critical?

lanyard-textile 7d ago
Maybe just for the pleasure of doing it, too.

davidfischer 7d ago

I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did. The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.

SoftTalker 7d ago
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
- davidfischer 7d ago
  It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate. It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo. [1] https://docs.readthedocs.com/platform/stable/reference/markdown-for-agents.html https://docs.readthedocs.com/platform/stable/reference/markd...
- simonw 7d ago
  Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists. See https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
  - kees99 7d ago
    This is exactly the problem, unfortunately. For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0]. Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are. [0] https://dumps.wikimedia.org/ https://dumps.wikimedia.org/
    - nubinetwork 7d ago
      You can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.
  - perching_aix 7d ago
    Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.
tescreal 7d ago
Good to know. I use your site (with a manual transmission user-agent) often, and it's fantastic. Thanks for your work and the writeup!
- davidfischer 7d ago
  I've never seen the phrase "manual transmission user-agent". Using your own browser yourself is the new stick shift. Love it.
  - gopher_space 7d ago
    It feels like everyone's rebuilding their own desktop experience. Kind of Minecraft with folders and text files. The really interesting part of this is how little people talk about what they're doing, and it doesn't feel secretive in any way.
    - tescreal 7d ago
      What do you mean exactly?
    - Cilvic 7d ago
      I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
      - gopher_space 6d ago
        Just like you say. It's not a secret business plan, it just doesn't really feel useful to talk about.

kkapelon 7d ago

Either testing for something bigger OR demonstrating their power to a 3rd party with minimal real disruption

RobRivera 7d ago

Could have been a live-fire exercise by a nation state. Edit: why the down vote? That is literally in the realm of possibility!

mitxela 6d ago
Due to mandatory scheduler maintenance, this test has been replaced with a live-fire CORS designed for military Androids. If you are an iPhone user, please proceed quickly to the chamber lock.

gkoberger 7d ago

I run a similar service, and we get almost daily attacks like this. Sometimes it's a specific high-profile customer, other times it's broader. I can't speak for RTD, but I think it's less "documentation site" and more just that we sit on the domains of high-profile products and the tools are just looking for any hole they can find? Often it's even the company themselves, for whatever reason (security research, etc).