5 ms·

I am not well versed in this problem but can't the web servers rate limit by known IP addresses of these crawler/scrapers?

ninja3925 10mo ago
Large cloud providers could offer that solution but then, crawlers can also change cycle IPs
strogonoff 10mo ago
You cannot block LLM crawlers by IP address, because some of them use residential proxies. Source: 1) a friend admins a slightly popular site and has decent bot detection heuristics, 2) just Google “residential proxy LLM”, they are not exactly hiding. Strip-mining original intellectual property for commercial usage is big business.
- skrebbel 10mo ago
  How does this work? Why would people let randos use their home internet connections? I googled it but the companies selling these services are not exactly forthcoming on how they obtained their "millions of residential IP addresses". Are these botnets? Are AI companies mass-funding criminal malware companies?
  - stackghost 10mo ago
    >Are these botnets? Are AI companies mass-funding criminal malware companies? Without a doubt some of them are botnets. AI companies got their initial foothold by violating copyright en masse with pirated textbook dumps for training data, and whatnot. Why should they suddenly develop scruples now?
  - joha4270 10mo ago
    I have seen it claimed that's a way of monetizing free phone apps. Just bundle a proxy and get paid for that.
    - cuu508 10mo ago
      A recent HN thread about this: https://news.ycombinator.com/item?id=45746156 https://news.ycombinator.com/item?id=45746156
  - fakwandi_priv 10mo ago
    It used to be Hola VPN which would let you use someone else’s connection and in the same way someone could use yours which was communicated transparently, that same hola client would also route business users. Im sure many other free VPN clients do the same thing nowadays.
- globalnode 10mo ago
  so user either has a malware proxy running requests without being noticed or voluntarily signed up as a proxy to make extra $ off their home connection. Either way I dont care if their IP is blocked. Only problem is if users behind CGNAT get their IP blocked then legitimate users may later be blocked. edit: ah yes another person above mentioned VPN's thats a good possibility, also another vector is users on mobile can sell their extra data that they dont use to 3rd parties. probably many more ways to acquire endpoints.
  - strogonoff 10mo ago
    “Known IP addresses” to me implies an infrequently changing list of large datacenter ranges. Maintaining a dynamic list (along with any metadata required for throttling purposes) of individual IPs is a different undertaking with higher level of effort. Of course, if you don’t care about affecting genuine users then it is much simpler. One could say it’s collateral damage and show a message suggesting to boycott companies and/or business practices that prompted these measures.
Yoric 10mo ago
Not the exact same problem, but a few months ago, I tried to block youtube traffic from my home (I was writing a parental app for my child) by IP. After a few hours of trying to collect IPs, I gave up, realizing that YouTube was dynamically load-balanced across millions of IPs, some of which also served traffic from other Google services I didn't want to block. I wouldn't be surprised if it was the same with LLMs. Millions of workers allocated dynamically on AWS, with varying IPs. In my specific case, as I was dealing with browser-initiated traffic, I wrote a Firefox add-on instead. No such shortcut for web servers, though.
- bonsai_spool 10mo ago
  Why not have local DNS at your router and do a block there? It can even be per-client with adguardhome
  - Yoric 10mo ago
    I did that, but my router doesn't offer a documented API (or even a ssh access) that I can use to reprogram DNS blocks dynamically. I wanted to stop YouTube only during homework hours, so enabling/disabling it a few times per day quickly became tiresome.
    - extra88 10mo ago
      Your router almost certainly lets you assign a DNS instead of using whatever your ISP sends down so you set it to an internal device running your DNS. Your DNS mostly passes lookup requests but during homework time, when there's a request for the ip for "www.youtube.com" it returns the ip of your choice instead of the actual one. The domain's TTL is 5 minutes. Or don't, technical solutions to social problems are of limited value.
      - Yoric 10mo ago
        Any solution based on this sounds monstruously more complicated than my browser addon. And technical bandaids to hyperactivity, however imperfect, are damn useful.
adobrawy 10mo ago
They rely on residential proxies powered by botnets — often built by compromising IoT devices (see: https://krebsonsecurity.com/2025/10/aisuru-botnet-shifts-from-ddos-to-residential-proxies/ https://krebsonsecurity.com/2025/10/aisuru-botnet-shifts-fro... ). In other words, many AI startups — along with the corporations and VC funds backing them — are indirectly financing criminal botnets.

ninja3925 10mo ago

Large cloud providers could offer that solution but then, crawlers can also change cycle IPs

strogonoff 10mo ago

You cannot block LLM crawlers by IP address, because some of them use residential proxies. Source: 1) a friend admins a slightly popular site and has decent bot detection heuristics, 2) just Google “residential proxy LLM”, they are not exactly hiding. Strip-mining original intellectual property for commercial usage is big business.

skrebbel 10mo ago
How does this work? Why would people let randos use their home internet connections? I googled it but the companies selling these services are not exactly forthcoming on how they obtained their "millions of residential IP addresses". Are these botnets? Are AI companies mass-funding criminal malware companies?
- stackghost 10mo ago
  >Are these botnets? Are AI companies mass-funding criminal malware companies? Without a doubt some of them are botnets. AI companies got their initial foothold by violating copyright en masse with pirated textbook dumps for training data, and whatnot. Why should they suddenly develop scruples now?
- joha4270 10mo ago
  I have seen it claimed that's a way of monetizing free phone apps. Just bundle a proxy and get paid for that.
  - cuu508 10mo ago
    A recent HN thread about this: https://news.ycombinator.com/item?id=45746156 https://news.ycombinator.com/item?id=45746156
- fakwandi_priv 10mo ago
  It used to be Hola VPN which would let you use someone else’s connection and in the same way someone could use yours which was communicated transparently, that same hola client would also route business users. Im sure many other free VPN clients do the same thing nowadays.
globalnode 10mo ago
so user either has a malware proxy running requests without being noticed or voluntarily signed up as a proxy to make extra $ off their home connection. Either way I dont care if their IP is blocked. Only problem is if users behind CGNAT get their IP blocked then legitimate users may later be blocked. edit: ah yes another person above mentioned VPN's thats a good possibility, also another vector is users on mobile can sell their extra data that they dont use to 3rd parties. probably many more ways to acquire endpoints.
- strogonoff 10mo ago
  “Known IP addresses” to me implies an infrequently changing list of large datacenter ranges. Maintaining a dynamic list (along with any metadata required for throttling purposes) of individual IPs is a different undertaking with higher level of effort. Of course, if you don’t care about affecting genuine users then it is much simpler. One could say it’s collateral damage and show a message suggesting to boycott companies and/or business practices that prompted these measures.

Yoric 10mo ago

Not the exact same problem, but a few months ago, I tried to block youtube traffic from my home (I was writing a parental app for my child) by IP. After a few hours of trying to collect IPs, I gave up, realizing that YouTube was dynamically load-balanced across millions of IPs, some of which also served traffic from other Google services I didn't want to block. I wouldn't be surprised if it was the same with LLMs. Millions of workers allocated dynamically on AWS, with varying IPs. In my specific case, as I was dealing with browser-initiated traffic, I wrote a Firefox add-on instead. No such shortcut for web servers, though.

bonsai_spool 10mo ago
Why not have local DNS at your router and do a block there? It can even be per-client with adguardhome
- Yoric 10mo ago
  I did that, but my router doesn't offer a documented API (or even a ssh access) that I can use to reprogram DNS blocks dynamically. I wanted to stop YouTube only during homework hours, so enabling/disabling it a few times per day quickly became tiresome.
  - extra88 10mo ago
    Your router almost certainly lets you assign a DNS instead of using whatever your ISP sends down so you set it to an internal device running your DNS. Your DNS mostly passes lookup requests but during homework time, when there's a request for the ip for "www.youtube.com" it returns the ip of your choice instead of the actual one. The domain's TTL is 5 minutes. Or don't, technical solutions to social problems are of limited value.
    - Yoric 10mo ago
      Any solution based on this sounds monstruously more complicated than my browser addon. And technical bandaids to hyperactivity, however imperfect, are damn useful.

adobrawy 10mo ago

They rely on residential proxies powered by botnets — often built by compromising IoT devices (see: https://krebsonsecurity.com/2025/10/aisuru-botnet-shifts-from-ddos-to-residential-proxies/ https://krebsonsecurity.com/2025/10/aisuru-botnet-shifts-fro... ). In other words, many AI startups — along with the corporations and VC funds backing them — are indirectly financing criminal botnets.