8 ms·

AI bots are destroying Open Access

99 points by dhacks 1y ago

nathanaldensr 1y ago
The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal, anonymously, and with no recourse.
- ronsor 1y ago
  I guess I'm done using the Internet then.
- batata_frita 1y ago
  We're just ruining the last good part of internet
- eesmith 1y ago
  $125 and I can start an LLC. That's a real-world corporate entity. Recourse ends at the "limited liability" in LLC. Make that LLC owned by another? Offshore ownership? Might take a few thousand bucks.
  - ronsor 1y ago
    Cheaper than that, and the Secretary of State doesn't actually verify anything.
- JohnFen 1y ago
  The truth and tragedy of this is very clear to me. I am hoping this is something that will eventually be solved, but I don't expect it. These companies are on a burn-and-pillage rampage.
- Verdex 1y ago
  I don't know. Once I know the who the legal entity is who I assert is a bad actor, I'm not sure there is really an recourse to be had. Your honor, these people are visiting my website in a way that makes me sad? I feel that we would need to encode bad behavior in a legally reasonable way first. And not to mention that you'll have to bring legal disputes a legal entity at a time. And some of these legal entities have very deep pockets. Unless the suggestion is that internet providers are all going to join together to stand up for the little guy? Somehow I'm not optimistic. (Finally IPv6 has taken decades to get to where it is today. Somehow I don't see legally attributable IP traffic extension to be ready and deployed any faster)
  - deleted 1y ago
    [deleted]
- akomtu 1y ago
  In reality, users will have to show passport to use the internet, while corporations will hide behind a "Corporate ID" that's whitelisted in all authenticator services, because those are also corporations. So you'll keep getting millions of requests from corp234 and corp456 with no legal recourse against them.
- reginald78 1y ago
  Anubis (or something similar) is an alternative option: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis Aside from the obvious disadvantages of a non-anoynmous web I also don't even think it will work. How do you deal with identification and punishment of threat actors across the globe? We've been failing at that since the start. When was the internet ever high trust?
  - hollerith 1y ago
    >When was the internet ever high trust? In the 1970s and 1980s.
xnx 1y ago
Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?
- rcxdude 1y ago
  Seems like the latter. There's basically a large number of well-funded attempts to crawl the internet, and enough of them are badly behaved enough it's basically a DDOS against smaller hosts.
- jsheard 1y ago
  AI bots operators are financially incentivized to not be good citizens, they want as much data as possible as fast as possible and don't care who they piss off in the process. Plus for now at least they have effectively unlimited money to throw at bandwidth, storage, IP addresses, crawling with full-blown headless browsers, etc.
  - quectophoton 1y ago
    And it gets worse. For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents). But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying something like: "Get faster results (or a discount) by using our extension! By using your own internet connection to fetch the required context, you won't need to share computing resources with other users, thusly increasing the speed of your queries! Plus, since you don't use our servers, that means we can pass our savings to you as a discount!" Then in small letter saying they use your connection for helping others with their queries, or being more eco-friendly because sharing, or whatever they come up with to justify this.
    - duttonw 1y ago
      OpenAI has ‘already’ got a browser extension. Who knows when this is ‘enabled’. We already had the ‘honey’ debacle with Amazon/ebay referral link stealing
kragen 1y ago
Sheesh, just use BitTorrent. That's what open access licensing is for! BitTorrent's tit-for-tat approach limits the harm selfish actors can do, only greatly rewarding those whose behavior benefits others, and has been shown to be very robust against active disruption attempts for decades now. Moreover, it also confers some resistance to falsification of the published record, to linkrot, and to publishing companies going bankrupt. Sooner or later we need to take back the legitimate internet from surveillance capitalism. Capitalism is great (it shares many of BitTorrent's virtues, not coincidentally) but surveillance capitalism is not.
- quectophoton 1y ago
  As much as I like BitTorrent, people (usually) don't want to provide open access to information; what they (usually) want is to be an "open" gateway to that information, as long as they are the centralized point of distribution whose name appears in the URL bar, and as long as they control when they can remove access to that information. Creating a torrent is not showy enough, because the credit is "just" another file and/or a comment in the torrent metadata. Granted, they usually do that because they want to "kindly" advertise a way to donate to them (EDIT: or to track you, or other similar goals), and there's nothing wrong with trying to get donations, but there's clearly a conflict of interest at play here.
  - kragen 1y ago
    It doesn't matter what people usually want. It's sufficient for someone to want to torrent the open-access articles, even if everyone else is playing the exploitative games you're describing. The Berlin Declaration that defined "open access" https://openaccess.mpg.de/Berlin-Declaration https://openaccess.mpg.de/Berlin-Declaration requires specifically > The author(s) and right holder(s) of such contributions grant(s) to all users a free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose, subject to proper attribution of authorship (community standards, will continue to provide the mechanism for enforcement of proper attribution and responsible use of the published work, as they do now), as well as the right to make small numbers of printed copies for their personal use. This guarantees that such torrents are legal unless the original authors are infringing copyright. So there is no danger of AI bots destroying open access.
zzo38computer 1y ago
I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.) My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc. My problem is that these LLM scraping bots are badly behaved, making many requests and repeating them even though there is no good reason to do so, and potentially overloading the servers. These things are mentioned in the article. Some bots are not so badly behaved, and those are not the problem.
- paulddraper 1y ago
  How can you tell they are LLM bots?
  - zzo38computer 1y ago
    I do not know for sure, but they are accessing with many different IP addresses, and with many different user-agent values that all include "Mozilla". I had read elsewhere that apparently they are botnets for LLM scraping.
  - deleted 1y ago
    [deleted]
josefritzishere 1y ago
AI is a scourge. It provides next to nothing useful but wrecks havok. As passing fads go it's heavy on the distruption but light on the utility.
kazinator 1y ago
> The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. What year are they reminiscing about here, 1999? Nothing has respected robots.txt in over twenty years. Nofollow isn't an anti-bot measure; it's supposed to inform search engines that you don't vouch for the linked content (don't wish to boost its rank). Nofollow doesn't mean "you must not follow this if you're a crawler".
musicale 1y ago
I guess the problem of throttling connections to human rates is that the bots rapidly eat up all of the connections. Can we at least get rid of CAPTCHAs now since they clearly don't work?
- jrochkind1 1y ago
  I don't mean to be marketting for them, but the CloudFlare Turnstile "captcha alternative" (Similar to Google ReCaptcha and others) has been working for me. it's the only thing that has of what I tried so far (rate-limiting IPs, fail2ban, etc -- their IPs are just too distributed). It doesn't make the user do a puzzle, it's the kind that either works entirely automatically or in some cases asks the user to tick a checkbox. You have probably seen it proliferating across the internet in your personal use becuase, well, see above.
  - musicale 1y ago
    Rate limiting individual addresses seems like a possibly useful, if not perfect, idea since it forces the bots to spread out over more addresses. It does penalize humans behind NAT however.
    - jrochkind1 1y ago
      I have indeed tries lots of things that seemed possibly useful! Rate limiting by IP (or by CIDR subnet of various sizes) was not enough for me. The bots spread out to more addreseses and still overwhelmed my resources.
      - musicale 1y ago
        Interesting that they could instantly increase the IP range by 100x or something, especially for IPv4 addresses. I guess that implies deep pockets and/or a malicious botnet.
        jrochkind1 1y ago
elzbardico 1y ago
Looks like they vibe-coded the scrappers
data_maan 1y ago
Uhmm.... Just use harder Catpchas? I can't really understand the outrage here, this problem of scraping to the point of being DDOSed, which is what the author seems to contend, has existed since forever.
- eesmith 1y ago
  Who is developing, deploying, and profiting from these harder captchas? Apparently not Cloudflare. > They are using commercial services such as Cloudflare to outsource their bot-blocking and captchas, without knowing for sure what these services are blocking, how they're doing it, or whether user privacy and accessibility is being flushed down the toilet. But nothing seems to offer anything but temporary relief. Also: > The current generation of bots is mindless. They use as many connections as you have room for. If you add capacity, they just ramp up their requests. They use randomly generated user-agent strings. They come from large blocks of IP addresses. They get trapped in endless hallways. I observed one bot asking for 200,000 nofollow redirect links pointing at Onedrive, Google Drive and Dropbox. (which of course didn't work, but Onedrive decided to stop serving our Canadian human users). They use up server resources - one speaker at Code4lib described a bug where software they were running was using 32 bit integers for session identifiers, and it ran out!
  - data_maan 1y ago
    Google is typically developing Captchas, but you can develop your own if you like.
pickledoyster 1y ago
>large blocks of IP addresses This could be at least partially solved by starting legal and cybersec (bulk blocks, flagging SDKs as malware) action against botnets for rent[0], forcing their SDKs out of app stores[1]. 0 – https://spur.us/residential-proxies-the-legal-botnet-that-nobody-talks-about/ https://spur.us/residential-proxies-the-legal-botnet-that-no... 1 – https://datadome.co/bot-management-protection/how-proxy-providers-get-residential-proxies/ https://datadome.co/bot-management-protection/how-proxy-prov...
skeledrew 1y ago
The way I see it, there may be pain now, but this is just the chaos before the web eventually becomes the semantic web, as mostly conceived by Berners-Lee[0]. Make all data available for true, easy, permanent open access instead of throwing up roadblocks to be circumvented, so all organizations wanting to train AI models can access, and thus properly democratize the ecosystem. It's that or end up with a handful of players with pockets deep enough to do what it takes to get the data, and then gatekeep the results for their own profit. Also, eventually I see most people filtering their queries through something like Perplexity anyway instead of going to individual sites, so those putting up barriers will lose out on human traffic in any case. Let's ensure that the results people are able to access via AI continually improves so the "slop" term disappears that much faster. [0] https://en.m.wikipedia.org/wiki/Semantic_Web https://en.m.wikipedia.org/wiki/Semantic_Web

99 points by dhacks 1y ago

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal, anonymously, and with no recourse.

ronsor 1y ago
I guess I'm done using the Internet then.
batata_frita 1y ago
We're just ruining the last good part of internet
eesmith 1y ago
$125 and I can start an LLC. That's a real-world corporate entity. Recourse ends at the "limited liability" in LLC. Make that LLC owned by another? Offshore ownership? Might take a few thousand bucks.
- ronsor 1y ago
  Cheaper than that, and the Secretary of State doesn't actually verify anything.
JohnFen 1y ago
The truth and tragedy of this is very clear to me. I am hoping this is something that will eventually be solved, but I don't expect it. These companies are on a burn-and-pillage rampage.
Verdex 1y ago
I don't know. Once I know the who the legal entity is who I assert is a bad actor, I'm not sure there is really an recourse to be had. Your honor, these people are visiting my website in a way that makes me sad? I feel that we would need to encode bad behavior in a legally reasonable way first. And not to mention that you'll have to bring legal disputes a legal entity at a time. And some of these legal entities have very deep pockets. Unless the suggestion is that internet providers are all going to join together to stand up for the little guy? Somehow I'm not optimistic. (Finally IPv6 has taken decades to get to where it is today. Somehow I don't see legally attributable IP traffic extension to be ready and deployed any faster)
- deleted 1y ago
  [deleted]
akomtu 1y ago
In reality, users will have to show passport to use the internet, while corporations will hide behind a "Corporate ID" that's whitelisted in all authenticator services, because those are also corporations. So you'll keep getting millions of requests from corp234 and corp456 with no legal recourse against them.
reginald78 1y ago
Anubis (or something similar) is an alternative option: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis Aside from the obvious disadvantages of a non-anoynmous web I also don't even think it will work. How do you deal with identification and punishment of threat actors across the globe? We've been failing at that since the start. When was the internet ever high trust?
- hollerith 1y ago
  >When was the internet ever high trust? In the 1970s and 1980s.

xnx 1y ago

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

rcxdude 1y ago
Seems like the latter. There's basically a large number of well-funded attempts to crawl the internet, and enough of them are badly behaved enough it's basically a DDOS against smaller hosts.
jsheard 1y ago
AI bots operators are financially incentivized to not be good citizens, they want as much data as possible as fast as possible and don't care who they piss off in the process. Plus for now at least they have effectively unlimited money to throw at bandwidth, storage, IP addresses, crawling with full-blown headless browsers, etc.
- quectophoton 1y ago
  And it gets worse. For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents). But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying something like: "Get faster results (or a discount) by using our extension! By using your own internet connection to fetch the required context, you won't need to share computing resources with other users, thusly increasing the speed of your queries! Plus, since you don't use our servers, that means we can pass our savings to you as a discount!" Then in small letter saying they use your connection for helping others with their queries, or being more eco-friendly because sharing, or whatever they come up with to justify this.
  - duttonw 1y ago
    OpenAI has ‘already’ got a browser extension. Who knows when this is ‘enabled’. We already had the ‘honey’ debacle with Amazon/ebay referral link stealing

kragen 1y ago

Sheesh, just use BitTorrent. That's what open access licensing is for! BitTorrent's tit-for-tat approach limits the harm selfish actors can do, only greatly rewarding those whose behavior benefits others, and has been shown to be very robust against active disruption attempts for decades now. Moreover, it also confers some resistance to falsification of the published record, to linkrot, and to publishing companies going bankrupt. Sooner or later we need to take back the legitimate internet from surveillance capitalism. Capitalism is great (it shares many of BitTorrent's virtues, not coincidentally) but surveillance capitalism is not.

quectophoton 1y ago
As much as I like BitTorrent, people (usually) don't want to provide open access to information; what they (usually) want is to be an "open" gateway to that information, as long as they are the centralized point of distribution whose name appears in the URL bar, and as long as they control when they can remove access to that information. Creating a torrent is not showy enough, because the credit is "just" another file and/or a comment in the torrent metadata. Granted, they usually do that because they want to "kindly" advertise a way to donate to them (EDIT: or to track you, or other similar goals), and there's nothing wrong with trying to get donations, but there's clearly a conflict of interest at play here.
- kragen 1y ago
  It doesn't matter what people usually want. It's sufficient for someone to want to torrent the open-access articles, even if everyone else is playing the exploitative games you're describing. The Berlin Declaration that defined "open access" https://openaccess.mpg.de/Berlin-Declaration https://openaccess.mpg.de/Berlin-Declaration requires specifically > The author(s) and right holder(s) of such contributions grant(s) to all users a free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose, subject to proper attribution of authorship (community standards, will continue to provide the mechanism for enforcement of proper attribution and responsible use of the published work, as they do now), as well as the right to make small numbers of printed copies for their personal use. This guarantees that such torrents are legal unless the original authors are infringing copyright. So there is no danger of AI bots destroying open access.

zzo38computer 1y ago

I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.) My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc. My problem is that these LLM scraping bots are badly behaved, making many requests and repeating them even though there is no good reason to do so, and potentially overloading the servers. These things are mentioned in the article. Some bots are not so badly behaved, and those are not the problem.

paulddraper 1y ago
How can you tell they are LLM bots?
- zzo38computer 1y ago
  I do not know for sure, but they are accessing with many different IP addresses, and with many different user-agent values that all include "Mozilla". I had read elsewhere that apparently they are botnets for LLM scraping.
- deleted 1y ago
  [deleted]

josefritzishere 1y ago

AI is a scourge. It provides next to nothing useful but wrecks havok. As passing fads go it's heavy on the distruption but light on the utility.

kazinator 1y ago

> The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. What year are they reminiscing about here, 1999? Nothing has respected robots.txt in over twenty years. Nofollow isn't an anti-bot measure; it's supposed to inform search engines that you don't vouch for the linked content (don't wish to boost its rank). Nofollow doesn't mean "you must not follow this if you're a crawler".

musicale 1y ago

I guess the problem of throttling connections to human rates is that the bots rapidly eat up all of the connections. Can we at least get rid of CAPTCHAs now since they clearly don't work?

jrochkind1 1y ago
I don't mean to be marketting for them, but the CloudFlare Turnstile "captcha alternative" (Similar to Google ReCaptcha and others) has been working for me. it's the only thing that has of what I tried so far (rate-limiting IPs, fail2ban, etc -- their IPs are just too distributed). It doesn't make the user do a puzzle, it's the kind that either works entirely automatically or in some cases asks the user to tick a checkbox. You have probably seen it proliferating across the internet in your personal use becuase, well, see above.
- musicale 1y ago
  Rate limiting individual addresses seems like a possibly useful, if not perfect, idea since it forces the bots to spread out over more addresses. It does penalize humans behind NAT however.
  - jrochkind1 1y ago
    I have indeed tries lots of things that seemed possibly useful! Rate limiting by IP (or by CIDR subnet of various sizes) was not enough for me. The bots spread out to more addreseses and still overwhelmed my resources.
    - musicale 1y ago
      Interesting that they could instantly increase the IP range by 100x or something, especially for IPv4 addresses. I guess that implies deep pockets and/or a malicious botnet.
      - jrochkind1 1y ago

elzbardico 1y ago

Looks like they vibe-coded the scrappers

data_maan 1y ago

Uhmm.... Just use harder Catpchas? I can't really understand the outrage here, this problem of scraping to the point of being DDOSed, which is what the author seems to contend, has existed since forever.

eesmith 1y ago
Who is developing, deploying, and profiting from these harder captchas? Apparently not Cloudflare. > They are using commercial services such as Cloudflare to outsource their bot-blocking and captchas, without knowing for sure what these services are blocking, how they're doing it, or whether user privacy and accessibility is being flushed down the toilet. But nothing seems to offer anything but temporary relief. Also: > The current generation of bots is mindless. They use as many connections as you have room for. If you add capacity, they just ramp up their requests. They use randomly generated user-agent strings. They come from large blocks of IP addresses. They get trapped in endless hallways. I observed one bot asking for 200,000 nofollow redirect links pointing at Onedrive, Google Drive and Dropbox. (which of course didn't work, but Onedrive decided to stop serving our Canadian human users). They use up server resources - one speaker at Code4lib described a bug where software they were running was using 32 bit integers for session identifiers, and it ran out!
- data_maan 1y ago
  Google is typically developing Captchas, but you can develop your own if you like.

pickledoyster 1y ago

>large blocks of IP addresses This could be at least partially solved by starting legal and cybersec (bulk blocks, flagging SDKs as malware) action against botnets for rent[0], forcing their SDKs out of app stores[1]. 0 – https://spur.us/residential-proxies-the-legal-botnet-that-nobody-talks-about/ https://spur.us/residential-proxies-the-legal-botnet-that-no... 1 – https://datadome.co/bot-management-protection/how-proxy-providers-get-residential-proxies/ https://datadome.co/bot-management-protection/how-proxy-prov...

skeledrew 1y ago

The way I see it, there may be pain now, but this is just the chaos before the web eventually becomes the semantic web, as mostly conceived by Berners-Lee[0]. Make all data available for true, easy, permanent open access instead of throwing up roadblocks to be circumvented, so all organizations wanting to train AI models can access, and thus properly democratize the ecosystem. It's that or end up with a handful of players with pockets deep enough to do what it takes to get the data, and then gatekeep the results for their own profit. Also, eventually I see most people filtering their queries through something like Perplexity anyway instead of going to individual sites, so those putting up barriers will lose out on human traffic in any case. Let's ensure that the results people are able to access via AI continually improves so the "slop" term disappears that much faster. [0] https://en.m.wikipedia.org/wiki/Semantic_Web https://en.m.wikipedia.org/wiki/Semantic_Web