5 ms·

codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.

by pluto_modadic 2mo ago

saghm 2mo ago
Couldn't they just rate limit them? Are they literally using a new IP for every request?
- anematode 2mo ago
  There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
  - sroussey 2mo ago
    We used to call this snow shoeing.
- kstrauser 2mo ago
  In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
kstrauser 2mo ago
You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it: * For every Git commit, fetch the version of every file in the repository at that commit. * See git blame for every file at every commit. * Attempt to download the archive of each repo at every commit. * Run every possible pull request search filter combination. * Run every possible issue search filter combination. * Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before. Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.
- andai 2mo ago
  That's crazy. So 600,000 someones are absolutely starved for data. Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?
  - kstrauser 2mo ago
    I think it was the latter. And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.
- grep_name 2mo ago
  What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start
  - kstrauser 2mo ago
    I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.
- lloydatkinson 2mo ago
  This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over. I can’t tell if it’s incompetence or malice.

codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.

saghm 2mo ago

Couldn't they just rate limit them? Are they literally using a new IP for every request?

anematode 2mo ago
There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
- sroussey 2mo ago
  We used to call this snow shoeing.
kstrauser 2mo ago
In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.

kstrauser 2mo ago

You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it: * For every Git commit, fetch the version of every file in the repository at that commit. * See git blame for every file at every commit. * Attempt to download the archive of each repo at every commit. * Run every possible pull request search filter combination. * Run every possible issue search filter combination. * Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before. Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.

andai 2mo ago
That's crazy. So 600,000 someones are absolutely starved for data. Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?
- kstrauser 2mo ago
  I think it was the latter. And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.
grep_name 2mo ago
What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start
- kstrauser 2mo ago
  I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.
lloydatkinson 2mo ago
This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over. I can’t tell if it’s incompetence or malice.