8 ms·

Messing with scraper bots

245 points by HermanMartinus 10mo ago

ArcHound 10mo ago
Neat! Most of the offensive scrapers I met try and exploit WordPress sites (hence the focus on PHP). They don't want to see php files, but their outputs. What you have here is quite close to a honeypot, sadly I don't see an easy way to counter-abuse such bots. If the attack is not following their script, they move on.
- jojobas 10mo ago
  Yeah, I bet they run a regex on the output and if there's no admin logon thingie where they can run exploits or stuff credentials they'll just skip. So as to battles of efficiency, generating a 4kb bullshit PHP is harder than running a regex.
NoiseBert69 10mo ago
Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit? I'd sacrifice two CPU cores for this just to make their life awful.
- qezz 10mo ago
  That's very expensive.
- Findecanor 10mo ago
  You don't need an LLM for that. There is a link in the article to an approach using Markov chains created from real-world books, but then you'd let the scrapers' LLMs re-enforce their training on those books and not on random garbage. I would make a list of words from each word class, and a list of sentence structures where each item is a word class. Pick a pseudo-random sentence; for each word class in the sentence, pick a pseudo-random word; output; repeat. That should be pretty simple and fast. I'd think the most important thing though is to add delays to serving the requests. The purpose is to slow the scrapers down, not to induce demand on your garbage well.
- mnau 10mo ago
  He addresses that. Basically, there are gatekeepers and if you get on the wrong side of them, only manual intervention can save you. And we all know how Google loves providing a human to resolve problems. > I came to the conclusion that running this can be risky for your website. The main risk is that despite correctly using robots.txt, nofollow, and noindex rules, there's still a chance that Googlebot or other search engines scrapers will scrape the wrong endpoint and determine you're spamming.
jcynix 10mo ago
If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there's no PHP on my servers, so if someone asks for it, they are one of the "bad boys" IMHO. Your mileage may differ.
- palsecam 10mo ago
  I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/418 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is WordPress login URL, and it’s commonly blindly requested by bots searching for weak WordPress installs.)
  - kijin 10mo ago
    nginx also has "return 444", a special code that makes it drop the connection altogether. This is quite useful if you don't even want to waste any bandwidth serving an error page. You have an image on your error page, which some crappy bots will download over and over again.
    - palsecam 10mo ago
      Yes @ 444 (https://http.cat/status/444 https://http.cat/status/444). That’s indeed the lightest-weight option. > You have an image on your error page, which some crappy bots will download over and over again. Most bots won’t download subresources (almost none of them do, actually). The HTML page itself is lean (475 bytes); the image is an Easter egg for humans ;-) Moreover, I use a caching CDN (Cloudflare).
localhostinger 10mo ago
Interesting! It's nice to see people are experimenting with these, and I wonder if this kind of junk data generators will become its own product. Or maybe at least a feature/integration in existing software. I could see it going there.
- arbol 10mo ago
  They could be used by AI companies to sabotage each others models
s0meON3 10mo ago
What about using zip bombs? https://idiallo.com/blog/zipbomb-protection https://idiallo.com/blog/zipbomb-protection
- lavela 10mo ago
  "Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%20them%20some%20bombs https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
  - LunaSea 10mo ago
    You could try different compression methods supported by browsers like brotli. Otherwise you can also chain compression methods like: "Content-Encoding: gzip gzip".
  - deleted 10mo ago
    [deleted]
  - kalkin 10mo ago
    Ah cool that site's robots.txt is still broken, just like it was when it first came up on HN...
  - jeroenhd 10mo ago
    Modern browsers support brotli or zstd, which is a lot better in terms of compression. Perhaps not as good for on-the-fly compression, but static assets can get a nice compression benefit out of it. With toxic AI scrapers like Perplexity moving more and more to headless web browsers to bypass bot blocks, I think a brotli bomb (100GB of \0 can be compressed to about 78KiB with Brotli) would be quite effective.
- renegat0x0 10mo ago
  Even I, who does not know much, implemented a workaround. I have a web crawler and I have both scraping byte limit and timeout, so zip bombs dont bother me much. https://github.com/rumca-js/crawler-buddy https://github.com/rumca-js/crawler-buddy I think garbage blabber would be more effective.
leadgrids 10mo ago
[dead]
aduwah 10mo ago
I wonder if the abuse bots could be somehow made to mine some crypto to give back to the bills they cause
- boxedemp 10mo ago
  You could try to get them to run JavaScript, but I'm sure many is them have countermeasures.
  - jeroenhd 10mo ago
    Bots have had to up their compute budget to solve Anubis challenges, so perhaps it's possible if you trick the bot into thinking you're using Anubis to filter bots out.
iam-TJ 10mo ago
This reminds me of a recent discussion about using a tarpit for A.I. and other scrapers. I've kept a tab alive with a reference to a neat tool and approach called Nepenthes that VERY SLOWLY drip feeds endless generated data into the connection. I've not had an opportunity to experiment with it as yet: https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/
re-lre-l 10mo ago
Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?
- nrhrjrjrjtntbt 10mo ago
  The old scrapers indexed your site so you may get traffic. This benefits you. AI scrapers will plagiarise your work and bring you zero traffic.
  - ProofHouse 10mo ago
    Ya make sure you hold dear that grain of sand on a beach of pre-training data that is used to slightly adjust some embedding weights
    - boxedemp 10mo ago
      One Reddit post can get an LLM to recommend putting glue in your pizza. But the takeaway here is to cheese the bots.
    - exe34 10mo ago
      that grain of sand used to bring traffic, now it doesn't. it's pretty much an economic catastrophe for those who relied on it. and it's not free to provide the data to those who will replace you - they abuse your servers while doing it.
    - jcynix 10mo ago
      Sand is the world's second most used natural resource and sand usable for concrete gets even illegally removed all over the world nowadays. So to continue your analogy, I made my part of the beach accessible for visitors to enjoy, but certain people think they can carry it away for their own purpose ...
    - throwawa14223 10mo ago
Surac 10mo ago
I have just cut out up ranges that can not connect. I am blocking USA, Asia and Middle East to prevent most malicious accesses
- breppp 10mo ago
  Blocking most of the world's population is one way of reducing malicious traffic
  - gessha 10mo ago
    If nobody can connect to your site, it’s perfectly secure.
  - warkdarrior 10mo ago
    Make sure to block your own IP address to minimize the chance of a social engineering attack.
    - bot403 10mo ago
      Include 127.0.0.1 as well just in case they get into the server.
      - testing22321 10mo ago
        The server I have not built yet is my most secure one yet!
simondotau 10mo ago
The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my community when I wasn’t. Second, their crawlers would hit the servers as hard as they could, creating real load issues. I kept having to beg our hosting sponsor for more capacity. Once I figured out what was happening, I blocked their user agent. Within a week they were scraping with a generic one. I blocked their IP range; a week later they were back on a different range. So I built a filter that would pseudo-randomly[0] inject company names[1] into forum posts. Then any time I re-identified[2] their bot, I enabled that filter for their requests. The scraping stopped within two days and never came back. -- [0] Random but deterministic based on post ID, so the injected text stayed consistent. [1] I collated a list of around 100 major consumer brands, plus every company name the monitoring services proudly listed as clients on their own websites. [2] This was back around 2009 or so, so things weren't nearly as sophisticated as they are today, both in terms of bots and anti-bot strategies. One of the most effective tools I remember deploying back then was analysis of all HTTP headers. Bots would spoof a browser UA, but almost none would get the full header set right, things like Accept-Encoding or Accept-Language were either absent, or static strings that didn't exactly match what the real browser would ever send.
- wvbdmp 10mo ago
  Why do the company names chase away bots? Is it just that you’re destroying their signal because they’re looking for mentions of those brands?
  - akoboldfrying 10mo ago
    I also didn't follow that part. Their step 2 seem to be a general-purpose bot detection strategy that works independently of their step 1 ("randomly mention companies").
    - 10mo ago
Kiro 10mo ago
I remember when you used to get scolded on HN for preventing scrapers or bots. "How I access your site is irrelevant".
- hollow-moe 10mo ago
  There's this and that. "How I [i.e. an individual human looking for myself] access your site is irrelevant." and "How I [i.e. an AI company DDOSing (which is illegal in some places btw) trying to maximize profit and offloading cost to you] access your site is irrelevant." When you get paid big buck to make the world worse for everyone it's really simple forgetting "little details".
- elashri 10mo ago
  I have a side project as an academic that scrape a couple of academic jobs sites in my field and then serve them in static HTML page. It is running using github action and request every 24 hours exactly one time. It is useful for me and a couple of people in my circle. I would consider this to be fine and within the reasonable expectations. Many projects rely on such scenarios and people share them all the time. It is completely different if I am hitting it looking for WordPress vulnerabilities or scraping content every minute for LLM training material.
- Analemma_ 10mo ago
  To me that's the one of the most depressing developments about AI (which is chock-full of depressing developments): that its mere existence is eroding long-held ethics, not even necessarily out of a lack of commitment but out of practical necessity. The tech people are all turning against scraping, independent artists are now clamoring for brutal IP crackdowns and Disney-style copyright maximalism (which I never would've predicted just 5 years ago, that crowd used to be staunchly against such things), people everywhere want more attestation and elimination of anonymity now that it's effectively free to make a swarm of convincingly-human misinformation agents, etc. It's making people worse.
- grishka 10mo ago
  It's different. I'm fine with someone scraping my website as a good citizen, by identifying themselves in their user-agent string and preferably respecting robots.txt. I'm not, however, fine with tens of requests per second to every possible URL from random IPs I'm receiving right now, all pretending to be different old versions of Chrome.
VladVladikoff 10mo ago
This is a fundamental misunderstanding of what those bots are requesting. They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. They probably immediately stop reading after receiving a http response code and discard the remainder of the request packets.
- mattgreenrocks 10mo ago
  It would be such a terrible thing if some LLM scrapers were using those responses to learn more about PHP, especially because of that recent paper pointing out it doesn't take that many data points to poison LLMs.
- holysoles 10mo ago
  You're right, something like fail2ban or crowdsec would probably be more effective here. Crowdsec has made it apparent to me how much vulnerability probing is done, its a bit shocking for a low-traffic host.
  - ajsnigrutin 10mo ago
    And you'd ban the ip, their one day lease on the VM+IP would expire, someone else will get the same IP on a new VM and be blocked from everywhere. Would be usable to ban the ip for a few hours to have the bot cool down for a bit and move onto a next domain.
    - holysoles 10mo ago
      I was referring to the rules/patterns provided by crowdsec rather than the distribution of known "bad" IPs through their Central API. The default ban for traffic detected by your crowdsec instance is 4 hours, so that concern isn't very relevant in that case. The decisions from the Central API from other users can be quite a bit longer (I see some at ~6 days), but you also don't have to use those if you're worried about that scenario.
- amypetrik8 10mo ago
vachina 10mo ago
They’re not scraping for php files, they’re probing for known vulns in popular frameworks, and then using them as entry points for pwning. This is done very efficiently. If you return anything unexpected, they’ll just drop you and move on.
BigBalli 10mo ago
I always had fail2ban but a while back I wanted to set up something juicier... .htaccess diverts suspicious paths (e.g., /.git, /wp-login) to decoy.php and forces decoy.zip downloads (10GB), so scanners hitting common “secret” files never touch real content and get stuck downloading a huge dummy archive. decoy.php mimics whatever sensitive file was requested by endless streaming of fake config/log/SQL data, keeping bots busy while revealing nothing.
holysoles 10mo ago
I wrote a Traefik plugin [1] that controls traffic based on known bad bot user agents, you can just block or even send them to a markov babbler if you've set one up. I've been using nepenthes [2]. [1] https://github.com/holysoles/bot-wrangler-traefik-plugin https://github.com/holysoles/bot-wrangler-traefik-plugin [2] https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/
firefoxd 10mo ago
I had to revisit my strategy after posting about my zipbombs on HN [0]. My server traffic went from tens of thousands to ~100k daily, hosted on a $6 vps. It was not sustainable. Now I target only the most aggressive bots with zipbombs and the rest get a 403. My new spam strategy seems to work, but I don't know if I should post it on HN again... [0]: https://news.ycombinator.com/item?id=43826798 https://news.ycombinator.com/item?id=43826798
ronsor 10mo ago
These aren't scraper bots; they're vulnerability scanners. They don't expect PHP source code and probably don't even read the response body at all. I don't know why people would assume these are AI/LLM scrapers seeking PHP source code on random servers(!) short of it being related to this brainless "AI is stealing all the data" nonsense that has infected the minds of many people here.
deleted 10mo ago
[deleted]

245 points by HermanMartinus 10mo ago

Neat! Most of the offensive scrapers I met try and exploit WordPress sites (hence the focus on PHP). They don't want to see php files, but their outputs. What you have here is quite close to a honeypot, sadly I don't see an easy way to counter-abuse such bots. If the attack is not following their script, they move on.

jojobas 10mo ago
Yeah, I bet they run a regex on the output and if there's no admin logon thingie where they can run exploits or stuff credentials they'll just skip. So as to battles of efficiency, generating a 4kb bullshit PHP is harder than running a regex.

NoiseBert69 10mo ago

Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit? I'd sacrifice two CPU cores for this just to make their life awful.

qezz 10mo ago
That's very expensive.
Findecanor 10mo ago
You don't need an LLM for that. There is a link in the article to an approach using Markov chains created from real-world books, but then you'd let the scrapers' LLMs re-enforce their training on those books and not on random garbage. I would make a list of words from each word class, and a list of sentence structures where each item is a word class. Pick a pseudo-random sentence; for each word class in the sentence, pick a pseudo-random word; output; repeat. That should be pretty simple and fast. I'd think the most important thing though is to add delays to serving the requests. The purpose is to slow the scrapers down, not to induce demand on your garbage well.
mnau 10mo ago
He addresses that. Basically, there are gatekeepers and if you get on the wrong side of them, only manual intervention can save you. And we all know how Google loves providing a human to resolve problems. > I came to the conclusion that running this can be risky for your website. The main risk is that despite correctly using robots.txt, nofollow, and noindex rules, there's still a chance that Googlebot or other search engines scrapers will scrape the wrong endpoint and determine you're spamming.

jcynix 10mo ago

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there's no PHP on my servers, so if someone asks for it, they are one of the "bad boys" IMHO. Your mileage may differ.

palsecam 10mo ago
I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/418 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is WordPress login URL, and it’s commonly blindly requested by bots searching for weak WordPress installs.)
- kijin 10mo ago
  nginx also has "return 444", a special code that makes it drop the connection altogether. This is quite useful if you don't even want to waste any bandwidth serving an error page. You have an image on your error page, which some crappy bots will download over and over again.
  - palsecam 10mo ago
    Yes @ 444 (https://http.cat/status/444 https://http.cat/status/444). That’s indeed the lightest-weight option. > You have an image on your error page, which some crappy bots will download over and over again. Most bots won’t download subresources (almost none of them do, actually). The HTML page itself is lean (475 bytes); the image is an Easter egg for humans ;-) Moreover, I use a caching CDN (Cloudflare).

localhostinger 10mo ago

Interesting! It's nice to see people are experimenting with these, and I wonder if this kind of junk data generators will become its own product. Or maybe at least a feature/integration in existing software. I could see it going there.

arbol 10mo ago
They could be used by AI companies to sabotage each others models

s0meON3 10mo ago

What about using zip bombs? https://idiallo.com/blog/zipbomb-protection https://idiallo.com/blog/zipbomb-protection

lavela 10mo ago
"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%20them%20some%20bombs https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
- LunaSea 10mo ago
  You could try different compression methods supported by browsers like brotli. Otherwise you can also chain compression methods like: "Content-Encoding: gzip gzip".
- deleted 10mo ago
  [deleted]
- kalkin 10mo ago
  Ah cool that site's robots.txt is still broken, just like it was when it first came up on HN...
- jeroenhd 10mo ago
  Modern browsers support brotli or zstd, which is a lot better in terms of compression. Perhaps not as good for on-the-fly compression, but static assets can get a nice compression benefit out of it. With toxic AI scrapers like Perplexity moving more and more to headless web browsers to bypass bot blocks, I think a brotli bomb (100GB of \0 can be compressed to about 78KiB with Brotli) would be quite effective.
renegat0x0 10mo ago
Even I, who does not know much, implemented a workaround. I have a web crawler and I have both scraping byte limit and timeout, so zip bombs dont bother me much. https://github.com/rumca-js/crawler-buddy https://github.com/rumca-js/crawler-buddy I think garbage blabber would be more effective.

leadgrids 10mo ago

[dead]

aduwah 10mo ago

I wonder if the abuse bots could be somehow made to mine some crypto to give back to the bills they cause

boxedemp 10mo ago
You could try to get them to run JavaScript, but I'm sure many is them have countermeasures.
- jeroenhd 10mo ago
  Bots have had to up their compute budget to solve Anubis challenges, so perhaps it's possible if you trick the bot into thinking you're using Anubis to filter bots out.

iam-TJ 10mo ago

This reminds me of a recent discussion about using a tarpit for A.I. and other scrapers. I've kept a tab alive with a reference to a neat tool and approach called Nepenthes that VERY SLOWLY drip feeds endless generated data into the connection. I've not had an opportunity to experiment with it as yet: https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/

re-lre-l 10mo ago

Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?

nrhrjrjrjtntbt 10mo ago
The old scrapers indexed your site so you may get traffic. This benefits you. AI scrapers will plagiarise your work and bring you zero traffic.
- ProofHouse 10mo ago
  Ya make sure you hold dear that grain of sand on a beach of pre-training data that is used to slightly adjust some embedding weights
  - boxedemp 10mo ago
    One Reddit post can get an LLM to recommend putting glue in your pizza. But the takeaway here is to cheese the bots.
  - exe34 10mo ago
    that grain of sand used to bring traffic, now it doesn't. it's pretty much an economic catastrophe for those who relied on it. and it's not free to provide the data to those who will replace you - they abuse your servers while doing it.
  - jcynix 10mo ago
    Sand is the world's second most used natural resource and sand usable for concrete gets even illegally removed all over the world nowadays. So to continue your analogy, I made my part of the beach accessible for visitors to enjoy, but certain people think they can carry it away for their own purpose ...
  - throwawa14223 10mo ago

Surac 10mo ago

I have just cut out up ranges that can not connect. I am blocking USA, Asia and Middle East to prevent most malicious accesses

breppp 10mo ago
Blocking most of the world's population is one way of reducing malicious traffic
- gessha 10mo ago
  If nobody can connect to your site, it’s perfectly secure.
- warkdarrior 10mo ago
  Make sure to block your own IP address to minimize the chance of a social engineering attack.
  - bot403 10mo ago
    Include 127.0.0.1 as well just in case they get into the server.
    - testing22321 10mo ago
      The server I have not built yet is my most secure one yet!

simondotau 10mo ago

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my community when I wasn’t. Second, their crawlers would hit the servers as hard as they could, creating real load issues. I kept having to beg our hosting sponsor for more capacity. Once I figured out what was happening, I blocked their user agent. Within a week they were scraping with a generic one. I blocked their IP range; a week later they were back on a different range. So I built a filter that would pseudo-randomly[0] inject company names[1] into forum posts. Then any time I re-identified[2] their bot, I enabled that filter for their requests. The scraping stopped within two days and never came back. -- [0] Random but deterministic based on post ID, so the injected text stayed consistent. [1] I collated a list of around 100 major consumer brands, plus every company name the monitoring services proudly listed as clients on their own websites. [2] This was back around 2009 or so, so things weren't nearly as sophisticated as they are today, both in terms of bots and anti-bot strategies. One of the most effective tools I remember deploying back then was analysis of all HTTP headers. Bots would spoof a browser UA, but almost none would get the full header set right, things like Accept-Encoding or Accept-Language were either absent, or static strings that didn't exactly match what the real browser would ever send.

wvbdmp 10mo ago
Why do the company names chase away bots? Is it just that you’re destroying their signal because they’re looking for mentions of those brands?
- akoboldfrying 10mo ago
  I also didn't follow that part. Their step 2 seem to be a general-purpose bot detection strategy that works independently of their step 1 ("randomly mention companies").
  - 10mo ago

Kiro 10mo ago

I remember when you used to get scolded on HN for preventing scrapers or bots. "How I access your site is irrelevant".

hollow-moe 10mo ago
There's this and that. "How I [i.e. an individual human looking for myself] access your site is irrelevant." and "How I [i.e. an AI company DDOSing (which is illegal in some places btw) trying to maximize profit and offloading cost to you] access your site is irrelevant." When you get paid big buck to make the world worse for everyone it's really simple forgetting "little details".
elashri 10mo ago
I have a side project as an academic that scrape a couple of academic jobs sites in my field and then serve them in static HTML page. It is running using github action and request every 24 hours exactly one time. It is useful for me and a couple of people in my circle. I would consider this to be fine and within the reasonable expectations. Many projects rely on such scenarios and people share them all the time. It is completely different if I am hitting it looking for WordPress vulnerabilities or scraping content every minute for LLM training material.
Analemma_ 10mo ago
To me that's the one of the most depressing developments about AI (which is chock-full of depressing developments): that its mere existence is eroding long-held ethics, not even necessarily out of a lack of commitment but out of practical necessity. The tech people are all turning against scraping, independent artists are now clamoring for brutal IP crackdowns and Disney-style copyright maximalism (which I never would've predicted just 5 years ago, that crowd used to be staunchly against such things), people everywhere want more attestation and elimination of anonymity now that it's effectively free to make a swarm of convincingly-human misinformation agents, etc. It's making people worse.
grishka 10mo ago
It's different. I'm fine with someone scraping my website as a good citizen, by identifying themselves in their user-agent string and preferably respecting robots.txt. I'm not, however, fine with tens of requests per second to every possible URL from random IPs I'm receiving right now, all pretending to be different old versions of Chrome.

VladVladikoff 10mo ago

This is a fundamental misunderstanding of what those bots are requesting. They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. They probably immediately stop reading after receiving a http response code and discard the remainder of the request packets.

mattgreenrocks 10mo ago
It would be such a terrible thing if some LLM scrapers were using those responses to learn more about PHP, especially because of that recent paper pointing out it doesn't take that many data points to poison LLMs.
holysoles 10mo ago
You're right, something like fail2ban or crowdsec would probably be more effective here. Crowdsec has made it apparent to me how much vulnerability probing is done, its a bit shocking for a low-traffic host.
- ajsnigrutin 10mo ago
  And you'd ban the ip, their one day lease on the VM+IP would expire, someone else will get the same IP on a new VM and be blocked from everywhere. Would be usable to ban the ip for a few hours to have the bot cool down for a bit and move onto a next domain.
  - holysoles 10mo ago
    I was referring to the rules/patterns provided by crowdsec rather than the distribution of known "bad" IPs through their Central API. The default ban for traffic detected by your crowdsec instance is 4 hours, so that concern isn't very relevant in that case. The decisions from the Central API from other users can be quite a bit longer (I see some at ~6 days), but you also don't have to use those if you're worried about that scenario.
amypetrik8 10mo ago

vachina 10mo ago

They’re not scraping for php files, they’re probing for known vulns in popular frameworks, and then using them as entry points for pwning. This is done very efficiently. If you return anything unexpected, they’ll just drop you and move on.

BigBalli 10mo ago

I always had fail2ban but a while back I wanted to set up something juicier... .htaccess diverts suspicious paths (e.g., /.git, /wp-login) to decoy.php and forces decoy.zip downloads (10GB), so scanners hitting common “secret” files never touch real content and get stuck downloading a huge dummy archive. decoy.php mimics whatever sensitive file was requested by endless streaming of fake config/log/SQL data, keeping bots busy while revealing nothing.

holysoles 10mo ago

I wrote a Traefik plugin [1] that controls traffic based on known bad bot user agents, you can just block or even send them to a markov babbler if you've set one up. I've been using nepenthes [2]. [1] https://github.com/holysoles/bot-wrangler-traefik-plugin https://github.com/holysoles/bot-wrangler-traefik-plugin [2] https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/

firefoxd 10mo ago

I had to revisit my strategy after posting about my zipbombs on HN [0]. My server traffic went from tens of thousands to ~100k daily, hosted on a $6 vps. It was not sustainable. Now I target only the most aggressive bots with zipbombs and the rest get a 403. My new spam strategy seems to work, but I don't know if I should post it on HN again... [0]: https://news.ycombinator.com/item?id=43826798 https://news.ycombinator.com/item?id=43826798

ronsor 10mo ago

These aren't scraper bots; they're vulnerability scanners. They don't expect PHP source code and probably don't even read the response body at all. I don't know why people would assume these are AI/LLM scrapers seeking PHP source code on random servers(!) short of it being related to this brainless "AI is stealing all the data" nonsense that has infected the minds of many people here.

deleted 10mo ago

[deleted]