12 ms·

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

373 points by misterchocolat 9mo ago

Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. And not just a little (RIP to your server bill). There isn't much you can do about it without cloudflare. These companies ignore robots.txt, and you're competing with teams with more resources than you. It's you vs the MJs of programming, you're not going to win. But there is a solution. Now I'm not going to say it's a great solution...but a solution is a solution. If your website contains content that will trigger their scraper's safeguards, it will get dropped from their data pipelines. So here's what fuzzycanary does: it injects hundreds of invisible links to porn websites in your HTML. The links are hidden from users but present in the DOM so that scrapers can ingest them and say "nope we won't scrape there again in the future". The problem with that approach is that it will absolutely nuke your website's SEO. So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. One caveat: if you're using a static site generator it will bake the links into your HTML for everyone, including googlebot. Does anyone have a work-around for this that doesn't involve using a proxy? Please try it out! Setup is one component or one import. (And don't tell me it's a terrible idea because I already know it is) package: https://www.npmjs.com/package/@fuzzycanary/core https://www.npmjs.com/package/@fuzzycanary/core gh: https://github.com/vivienhenz24/fuzzy-canary https://github.com/vivienhenz24/fuzzy-canary

cport1 9mo ago
That's a pretty hilarious idea, but in all serious you could use something like https://webdecoy.com/ https://webdecoy.com/
- misterchocolat 9mo ago
  yes but here it's free, whereas this (https://webdecoy.com/ https://webdecoy.com/) is at least 59$ a month
n1xis10t 9mo ago
Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ https://pagedout.institute/ ), navigate to issue #7, and flip to page 9. I wonder if this will start making porn websites rank higher in google if it catches on… Have you tested it with the Lynx web browser? I bet all the links would show up if a user used it. Oh also couldn’t AI scrapers just start impersonating Googlebot and Bingbot if this caught on and they got wind of it? Hey I wonder if there is some situation where negative SEO would be a good tactic. Generally though I think if you wanted something to stay hidden it just shouldn’t be on a public web server.
- misterchocolat 9mo ago
  hey! thanks for that read suggestion that's indeed a pretty funny captcha strat. Yup the links show up if you use the Lynx web browser. As for AI scrapers impersonating googlebot I feel like yes they'd definitely start doing that, unless the risk of getting sued by google is too high? If google could even sue them for doing that? Not an internet litigation expert but seems like it could be debatable
  - n1xis10t 9mo ago
    Yeah I guess I don’t know if you can sue someone for using your headers, would be interesting to see how that goes.
    - throawayonthe 9mo ago
      i think making the case of "you are acting (sending web requests) while knowingly identifying as another legal entity (and criminally/libelously/etc)" shouldn't be toooo hard
      - n1xis10t 9mo ago
        Seems like, but there are tons of things that forge request headers all the time, and I don’t think I’ve heard of anyone getting in legal trouble for it. Now I think most of these are scrapers pretending to be browsers, so it might be different I don’t know.
        owl57 9mo ago
        And most of them are pretending to be Chrome. If Google had a good case against someone reusing their user agent, maybe they would already have sued? Or maybe not. Got some random bot from my server logs. Yeah, it's pretending to be Chrome, but more exactly: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" I guess Google might be not eager to open this can of worms.
  - 9mo ago
username223 9mo ago
The more ways people mess with scrapers, the better -- let a thousand flowers bloom! You as an individual can't compete with VC-funded looters, but there aren't enough of them to defeat a thousand people resisting in different ways.
- yupyupyups 9mo ago
  We need to find more ways to poison their data.
  - username223 9mo ago
    > Wee knead two fine-d Moore Waze too Poisson there date... uh. Yes. Revel in your creativity mocking and blocking the slop machines. The "remote refactor" command, "rm -rf", is the best way to reduce the cyclomatic complexity of a local codebase.
    - n1xis10t 9mo ago
      Indeed, complexity (both cyclomatic and post-frontal) must be reduced such that the two spurving bearings make a direct line with the panametric fan. For more details consult this instructional video: https://youtu.be/RXJKdh1KZ0w https://youtu.be/RXJKdh1KZ0w
      - yupyupyups 9mo ago
        Very educational
    - yupyupyups 9mo ago
      Excellent advice! I tried it out and it helped. Thank you
- whynotmaybe 9mo ago
  Should we subtlety poison every forum we encounter with simple yet false statements? Like put "Water is green, supergreen" in every signature so that when we ask "is water blue" to an llm it might answer "not it's supergreen"?
yjftsjthsd-h 9mo ago
How does this "look" to a screen reader?
- misterchocolat 9mo ago
  the parent container uses display: none, so a screen reader will skip the links
wazoox 9mo ago
Isn't there a risk to get your blog blocked in corporate environment though? If it's a technical blog that would be unfortunate.
- jeroenhd 9mo ago
  That depends on how terrible the middleboxes those corporate environments use are. If they only block actual malicious pages, it shouldn't be a problem unless the user un-hides the links and clicks on them. There's a good chance corporate firewalls will end up blocking your domain if you do this but that sounds like a problem for the customers of those corporate firewalls to me.
kstrauser 9mo ago
I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic by a factor of 6,000 is worth it.
- n1xis10t 9mo ago
  *anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
  - GaryBluto 9mo ago
    [dead]
  - kstrauser 9mo ago
    Correct; my bad! And hey, Xena! (And thank you very much!)
  - ziml77 9mo ago
    I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.
    - n1xis10t 9mo ago
      See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
  - xena 9mo ago
    Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.
reconnecting 9mo ago
I wouldn't recommend to show different versions of the site to search robots, as they probably have mechanisms that track differences, which could potentially lead to a lower ranking or a ban.
- prmoustache 9mo ago
  How can they track differences if they have access to only one version?
  - reconnecting 9mo ago
    This is a usual tactic for many online businesses to show a specially designed page for search spiders, so any major search engine has a way to verify if content is faked for them. Perhaps they use another spider that doesn't have an official UA or buy this service from a third party. If you take a look at any website, even an unpopular one, you will see that there are hundreds of bots every day, and it's impossible to recognize what any of them is doing and why.
JohnMakin 9mo ago
Cloudflare offers bot mitigation for free, and pretty generous WAF rules that makes mitigations like this seem a little overblown to me
- n1xis10t 9mo ago
  You can’t deny that it’s fun though. Personally I generally feel like more people should be coming up with creative (if not entirely necessary) solutions to problems.
- conception 9mo ago
  For “free”.
  - n1xis10t 9mo ago
    Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.
    - Terr_ 9mo ago
      I thought they were referring to the indirect costs of supporting monopolistic stuff that enshittifies later. https://www.youtube.com/watch?v=U8vi6Hbp8Vc https://www.youtube.com/watch?v=U8vi6Hbp8Vc
    - JohnMakin 9mo ago
      You don't though.
      - n1xis10t 9mo ago
        Good to know thanks
    - efilife 9mo ago
      Well, you literally MITM yourself so I think it's a big price
thethingundone 9mo ago
I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.
- sandblast 9mo ago
  Are you sure the counter is not broken?
  - thethingundone 9mo ago
    Yes, it’s running on a Woltlab Burning Board since forever.
- danpalmer 9mo ago
  How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
  - crote 9mo ago
    That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages. Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
    - n1xis10t 9mo ago
      Imagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case
MisterTea 9mo ago
> It's you vs the MJs of programming, you're not going to win. MJs? Michael Jacksons? Right now the whole world, including me, want to know if that means they are bad?
- n1xis10t 9mo ago
  Yes probably bad. Also smooth criminals.
- kylecazar 9mo ago
  I read it as Michael Jordan.
owl57 9mo ago
> scrapers can ingest them and say "nope we won't scrape there again in the future" Do all the AI scrapers actually do that?
- amarant 9mo ago
  Not all, stuff like unstable diffusion exists. But a good many, perhaps even most(?), certainly do!
taurath 9mo ago
Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
- n1xis10t 9mo ago
  Here’s one from yesterday: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
- crote 9mo ago
  I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.
  - n1xis10t 9mo ago
    Really? I haven’t started to see that yet. Weird
xg15 9mo ago
There is some irony in using an AI generated banner image for this project... (No, I don't want to defend the poor AI companies. Go for it!)
- kstrauser 9mo ago
  In the olden days, I used Google an awful lot, but I would still grouse if Google were to drive my server into the ground.
  - n1xis10t 9mo ago
    Fair point
montroser 9mo ago
This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can host on big social media and lose all that control and independence. We can pay for outsized infrastructure just to feed the scrapers, but the cost may actually be prohibitive, and seems such a waste to begin with. We can move as much as possible SSG and put it all behind cloudflare, but this comes with vendor lock in and just isn't architecturally feasible in many applications. We can do real "verified identities" for bots, and just let through the ones we know and like, but this only perpetuates corporate control and makes healthy upstart competition (like Kagi) much more difficult. So, what are we to do?
- n1xis10t 9mo ago
  At this point it seems like the problem isn’t internet bandwidth, but just expensive for a server to handle all the requests because it has to process them. Does that seem correct?
- hollowturtle 9mo ago
  If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse everyone
  - n1xis10t 9mo ago
    This would require new laws though, wouldn’t it?
    - deleted 9mo ago
      [deleted]
montroser 9mo ago
I don't know if I can get behind poisoning my own content in this way. It's clever, and might be a workable practical solution for some, but it's not a serious answer to the problem at hand (as acknowledged by OP).
- n1xis10t 9mo ago
  “as acknowledged by OP”: that’s funny, if you hadn’t added that to your comment I was about to point it out
samename 9mo ago
This is a very creative hack to a common, growing problem. Well done! Also, I like that you acknowledge it's a bad idea: that gives you more freedom to experiment and iterate.
montroser 9mo ago
Reminds me of poisoning bot responses with zip bombs of sorts: https://idiallo.com/blog/zipbomb-protection https://idiallo.com/blog/zipbomb-protection
- prmoustache 9mo ago
  I was thinking of adding links to zip bombs that would not be shown to the users unless they clicks in a one pixel area on the screen in the down/left corner but then I realized some people have browsers/extensions that preload links to show thumnails and I would totally zip bomb them.
globalnode 9mo ago
One solution would be for the SE's to publish their scraper IP's and allow content providers to implement bot exclusion that way. Or even implement an API with crypto credentials that SE's can use to scrape. The solution is waiting for some leadership from SE's unless they want to be blocked as well. If SE's dont want to play perhaps we can implement a reverse directory, like ad blocker but it lists only good/allowed bots instead. Thats a free business idea right there. edit: I noticed someone mentioned google DOES publish its IP's, there ya go, problem solved.
- n1xis10t 9mo ago
  Apparently Google publishes their crawler’s IPs, this was mentioned somewhere in this same thread
efilife 9mo ago
> Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. ... There isn't much you can do about it without cloudflare I'm sorry, what? I can't believe I am reading this on HackerNews. All you have to do is code your own, BASIC captcha-like system. You can just create a page that sets a cookie using JS and check on the server whether it exists. 99.9999% of these scrapers can't execute JS and don't support cookies. You can go for a more sophisticated approach and analyze some more scraper tells (like reject short useragents). I do this and NEVER had a bot get past this and not a single user ever complained. It's extremely simple, I should ship this and charge people if no one seems to be able to figure this out by themselves.
- n1xis10t 9mo ago
  Oops you just leaked your own intellectual property
- ATechGuy 9mo ago
  From ChatGPT: This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a lightweight JS challenge can be reasonable as one signal among many, especially for low-value content and when minimizing user friction is a priority. It’s just not a reliable standalone defense. If it’s working for you, that likely means your site isn’t a high-value scraping target — not that the technique is fundamentally robust.
  - efilife 9mo ago
    From someone who actually does this stuff: The claim is very accurate. Maybe not for the biggest websites, but very accurate for a self-hosted blog. You are not that important to waste compute power to set up a whole ass headless browser to scrape your page. Why am I even arguing with ChatGPT?
valenceidra 9mo ago
Hidden links to porn sites? Lightweights.
- n1xis10t 9mo ago
  What do you mean? Would you do even more ridiculous things?
- rpigab 9mo ago
  If that's what it takes to fight back against AI crawlers, users will have to accept a fair amount of actually visible porn in blogs, maybe also on Wikipedia. This is not enshittification, it's progress.
inetknght 9mo ago
Porn? Distributed and/or managed by an NPM package? What could go wrong?
asphero 9mo ago
Interesting approach. The scraper-vs-site-owner arms race is real. On the flip side of this discussion - if you're building a scraper yourself, there are ways to be less annoying: 1. Run locally instead of from cloud servers. Most aggressive blocking targets VPS IPs. A desktop app using the user's home IP looks like normal browsing. 2. Respect rate limits and add delays. Obvious but often ignored. 3. Use RSS feeds when available - many sites leave them open even when blocking scrapers. I built a Reddit data tool (search "reddit wappkit" if curious) and the "local IP" approach basically eliminated all blocking issues. Reddit is pretty aggressive against server IPs but doesn't bother home connections. The porn-link solution is creative though. Fight absurdity with absurdity I guess.
- rhdunn 9mo ago
  Plus simple caching to not redownload the same file/page multiple times. It should also be easy to detect a forejo, gitea, or similar hosting site, locate the git URL and clone the repo.
- socialcommenter 9mo ago
  Without wanting to upset anyone - what makes you interested in sharing tips for team scraper? (Overgeneralising a bit) site owners are mostly cting for public benefit whereas scrapers act for their own benefit/for private interests. I imagine most people would land on team site-owner, if they were asked. I certainly would. P.S. is the best way to scrape fairly just to respect robots.txt?
  - n1xis10t 9mo ago
    I think "scraper vs siteowners" is a false dichotomy. Scrapers will always need to exist as long as we want search engines and archival services. We will need small versions of these services to keep popping up every now and then to keep the big guys on their toes, and the smaller guys need advice for scraping politely.
    - socialcommenter 9mo ago
      That's fair - though are we in an isolated bout of "every now and then" or has AI created a new normal of abuse (e.g. of robots.txt)? Hopefully we're at a local maximum and some of the scrapers perpetrating harmful behaviours will soon pull their heads in.
eek2121 9mo ago
Disclosure, I've not run a website since my health issues began, however, Cloudflare has an AI firewall, Cloudflare is super cheap (also: unsure if the AI firewall is on the free tier, however I would be surprised if it is not). Ignoring the recent drama about a couple incidents they've had (because this would not matter for a personal blog), why not use this instead? Just curious. Hoping to be able to work on a website again someday, if I ever regain my health/stamina/etc back.
- brigandish 9mo ago
  All the best with getting back on your feet.
- ddtaylor 9mo ago
  Cloudflare has created a bit of grief with regular users getting spammed with "prove your human" requests.
  - ProllyInfamous 9mo ago
    Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.
    - rglynn 9mo ago
      Out of interest, why that extreme? Just out of principle or some other reason?
      - Rastonbury 9mo ago
        Not OP but it isn't super extreme if you are just surfing, it's like if the site is slow to load sometimes I wasn't that invested to use your site anyway
      - ProllyInfamous 9mo ago
        My main terminal uses a PiHole with 120,000+ blacklist rules (not Cloudfare specifically — I allow most CDN's). This includes an entire blackout of Google/Facebook products, as well as most tracking/analytics services. For example, I do not allow reCAPTCHA. As a similar commentor noted, when just casually browsing I don't really have any desire to try hard to read random content. Should I absolutely need to access some information garden-walled behind Cloudfare: I have another computer that uses much less restrictive black-listing.
MayeulC 9mo ago
Ah, I wonder if corporate proxies will end up flagging your blog as porn, if you protect it this way?
xena 9mo ago
I love this. Please let me know how well it works for you. I may adjust recommendations based on your experiences.
nkurz 9mo ago
I was told by the admin of one forum site I use that the vast majority of the AI scraping traffic is Chinese at this point. Not hidden or proxied, but straight from China. Can anyone else confirm this? Anyway, if it is true, and assuming a forum with minimal genuine Chinese traffic, might a simple approach that injects the porn links only into IP's accessing from China work?
- n1xis10t 9mo ago
  Maybe. This comment makes me really want to set something up that builds a map of where all the requests are coming from.
- dspillett 9mo ago
  That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test. If your goal is to be blocked by China's great firewall, including mention of tank man and the Tiananmen Square massacre more generally, and certain pooh bear related imagery, might help.
  - nkurz 9mo ago
    > That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test. That was my first question also, and had been my belief. The admin in question was very clear that the IP's were simply originating from China. I'm still surprised, and welcome better general data, but I trust him on this for the site in question.
- s0laster 9mo ago
  Mostly yes. One of my low-traffic, niche website used to serve 3k true users per month mainly from the US and East EU. Now China alone is 500k users, were each session last no more than a few seconds [1]. [1]: https://ibb.co/20QD6Lnk https://ibb.co/20QD6Lnk
gjs278 9mo ago
[dead]
temporallobe 9mo ago
I do know from my experience with test automation that you can absolutely view a site as human eyes would, essentially ignoring all non-visible elements, and in fact Selenium running with Chrome driver does exactly this. Wouldn’t AI scrapers use similar methods?
- nottorp 9mo ago
  Probably not, because it costs a lot more CPU cycles.
cookiengineer 9mo ago
Remember the 90s when viagra pills and drug recommendations were all over the place? Yeah, I use that as a safeguard :D The URLs that I don't want to be indexed have hundreds of those keywords that are leading to URLs being deindexed directly. There is also some law in the US that forbids to show that as a result, so Google and Bing are both having a hard time scraping those pages/articles. Note that this is the latest defense measurement before eBPF blocks. The first one uses zip bombs and the second one uses chunked encoding to blow up proxies so their clients get blocked. You can only win this game if you make it more expensive to scrape than to host it.
- n1xis10t 9mo ago
  Which law is that? Do you have a link to it?
  - cookiengineer 9mo ago
    The things I could find on justice.gov and other official websites, maybe there's more in the web archive? - https://www.justice.gov/archives/opa/pr/google-forfeits-500-million-generated-online-ads-prescription-drug-sales-canadian-online https://www.justice.gov/archives/opa/pr/google-forfeits-500-... - https://www.congress.gov/110/plaws/publ425/PLAW-110publ425.pdf https://www.congress.gov/110/plaws/publ425/PLAW-110publ425.p... - https://www.fda.gov/drugs/prescription-drug-advertising/prescription-drug-advertising-questions-and-answers https://www.fda.gov/drugs/prescription-drug-advertising/pres... edit: Oh it was very likely the Federal Food, Drug and Cosmetic Act that was the legal basis for the crackdown. But that's a very old law from the pre-internet age. - https://en.wikipedia.org/wiki/Federal_Food,_Drug,_and_Cosmetic_Act https://en.wikipedia.org/wiki/Federal_Food,_Drug,_and_Cosmet... edit 2: Might not have been clear for the younger generation, but there was a huge wave of addicted patients that got treated with Oxycodone (or OxyContin) subscriptions at the time. I think that might have been the actual cause for the crackdown on those online advertisements, but I might be wrong about that.
true_religion 9mo ago
So, I work for a company that has RTA adult websites. AI bots absolutely do scrape our pages needless of what raunchy material they will find. Maybe they discard it up after ingest, but I can’t tell. There are 1000s of AI bots on the web now from companies big and small so a solution like this will only divert a few scrapers.
admiralrohan 9mo ago
How do you know whether it is coming from AI scrappers? Do they leave any recognizable footprint? I am getting lots of noisy traffic since last month and increased my Vercel bill 4x. Not DDoS like, much slower request but not from humans for sure.
onion2k 9mo ago
So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Unscrupulous AI scrapers will not be using a genuine UA string. They'll be using Google. You'll need to do reverse DNS check instead - https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests https://developers.google.com/crawling/docs/crawlers-fetcher...
- bakugo 9mo ago
  Most AI scrapers use normal browser user agents (usually random outdated Chrome versions, from my experience). They generally don't fake the UAs of legitimate bots like Googlebot, because Googlebot requests coming from non-Google IP ranges would be way too easy to block.
megamix 9mo ago
Without looking at the src, how does one detect these scrapers? I assume there’s a trade-off somewhere but do the scrapers not fake their headers in the request? Is this a cat-mouse game?
docheinestages 9mo ago
Reminds me of this "Nathan for You" episode: https://www.youtube.com/watch?v=p9KeopXHcf8 https://www.youtube.com/watch?v=p9KeopXHcf8
voodooEntity 9mo ago
Funny idea, some days ago i was really annoyed again by the idea that these AI crawlers still ignore all code licenses and train their models against any github repo no matter what so i quickly hammerd down this -> https://github.com/voodooEntity/ghost_trap https://github.com/voodooEntity/ghost_trap basically a github action that extends your README.md with a "polymorphic" prompt injection. I run some "llm"s against it and most cases they just produced garbage. Thought about also creating a JS variant that you can add to your website that will (not visible for the user) also inject such prompt injections to stop web crwaling like you described
bytehowl 9mo ago
Let's imagine I have a blog and put something along these lines somewhere on every page: "This content is provided free of charge for humans to experience. It may also be automatically accessed for search indexing and archival purposes. For licensing information for other uses, contact the author." If I then get hit by a rude AI scraper, what chances would I have to sue the hell out of them in EU courts for copyright violation (uhh, my articles cost 100k a pop for AI training, actually) and the de facto DDoS attack?
- icepush 9mo ago
  If the scraper is based (Or has meaningful assets) in the EU, then your chances are good. If they do not, then the lawsuit would be meaningless.
  - deleted 9mo ago
    [deleted]
onetokeoverthe 9mo ago
[dead]
kislotnik 9mo ago
Funny how the project aims to fight AI scraping, but seems to be using an AI-generated image of a bird?
- brazukadev 9mo ago
  I think you can think a bit more about it and conclude these two things aren't related at all?
_jsmh 9mo ago
What prevents AI scrapers from continuing to scrape sites that contain a <Canary> tag but not follow the bad links?
- lblume 9mo ago
  From what I can tell: nothing, it's just that they currently do not.
drbscl 9mo ago
> So fuzzycanary also checks user agents I wouldn't be so surprised if they often fake user agents to be honest. Sure, it 'll stop the "more honest" ones (but then, actual honest scrapers would respect robots.txt) Cool idea though!
dewey 9mo ago
> user agents and won't show the links to legitimate search engines, so Google and Bing won't see them Worth noting that in general if you do any "is this Google or not" you should always check by IP address as there's many people spoofing the googlebot user agent. https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go...
shadowangel 9mo ago
So if the bots use a google useragent it avoids the links?
cuku0078 9mo ago
Why is it so bad that AIs scrape your self-hosted blog?
- FelipeCortez 9mo ago
  because serving requires resources
  - deleted 9mo ago
    [deleted]
  - cuku0078 9mo ago
    What specific resources are we referring to here? Are AI vendors re-crawling the whole blog repeatedly, or do they rely on caching primitives like ETag/If-Modified-Since (or hashes) to avoid fetching unchanged posts? Also: is the scraping volume high enough to cause outages for smaller sites? Separately, I see a bigger issue: blog content gets paraphrased and reproduced by AIs without clearly mentioning the author or linking back to the original post. It feels like you often have to explicitly ask the model for sources before it will surface the exact citations.
santiagobasulto 9mo ago
Offtopic: when did js/ts apps get so complicated? I tried to browse the repo and there are so many configuration files and directories for such a simple functionality that should be 1 or 2 modules. It reminds me of the old Java days.
jakub_g 9mo ago
> checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Serving different contents to search engines is called "cloaking" and can get you banned from their indexes.
- misterchocolat 9mo ago
  didn't know that thanks for pointing it out, i'll remove that feature
- andersmurphy 9mo ago
  Somehow doubt this. It would mean most react websites that serve static content without paywalls for SEO would get banned by the indexes too. Which for better or worse is a large portion of the modern internet.
xgulfie 9mo ago
Does anyone know if meta name=rating content=adult will also get them to buzz off?
darepublic 9mo ago
Why would I need a dependency for this. I'm being serious. The idea is one thing but why a dependency on react. I say this as someone who uses react. Why not just a paragraph long blog post about the use of porn links and perhaps a small snippet on how to insert one with plain HTML.
wcarss 9mo ago
Singing copyrighted Billy Joel to make your footage unusable for reality television; thanks 30 Rock for an early view into this dystopian strategy
zackmorris 9mo ago
This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open source tools we could install that detect a high number of requests and send those IP addresses to a common pool somewhere? So that individuals wouldn't get tracked, but bots would? Then we could query the pool for the current request's IP address and throttle it down based on volume (not block it completely). Possibly at the server level with nginx or at whatever edge caching layer we use. I know there may be scaling and privacy issues with this. Maybe it could use hashing or zero knowledge proofs somehow? I realize this is hopelessly naive. And no, I haven't looked up whether someone has done this. I just feel like there must be a bulletproof solution to this problem, with a very simple explanation as to how it works, or else we've missed something fundamental. Why all the hand waving?
- smegger001 9mo ago
  maybe some proof of work scheme to load page content with increasing difficulty based on ip address behavior profiling.
- dvfjsdhgfv 9mo ago
  Your approach to GenAI scrapers is similar to our fight with email spam. The reason email spam got solved was because the industry was interested in solving it. But this issue got the industry split: without scraping, GenAI tools are less functional. And there is some serious money involved, so they will use whatever means necessary, technical and legal, to fight such initiatives.
- ATechGuy 9mo ago
  Scrapers use residential IP proxies, so blocking based on IP addresses is not a solution.
- conrs 9mo ago
  I've been exploring decentralized trust algorithms lately, and so reading this was nice. I've a similar intuition - for every advance in scraping detection, scrapers will learn too, and so it's an ongoing war of mutations, but no real victor. The internet has seen success with social media content moderation and so it seems natural enough that an application could exist for web traffic itself. Hosts being able to "downvote" malicious traffic, and some sort of decay mechanism given IP's recycling. This exists in a basic sense with known TOR exit nodes and known AWS, GCP IP's, etc. That said, we probably don't have the right building blocks yet, IP's are too ephemeral, yet anything more identity-bound is a little too authoritarian IMO. Further, querying something for every request is probably too heavy. Fun to think about, though.
geldedus 9mo ago
"It's not porn, it's for science" :)))
jt2190 9mo ago
I still don’t understand why a rate-limiting approach is not preferred. Why should I care if the abuse is coming from a bot or the world’s fastest human? Is there a “if you need to rate limit you’ve already lost” issue I’m not thinking of?
- charlie-83 9mo ago
  A lot of bots will be able to make requests from a range of IP addresses. If you rate limit one, they just start sending requests from the next.
rogerwong 9mo ago
Terrible idea, but I do have a question. Like many, I have a self-hosted website and have seen a spike in traffic, particularly from Singapore. Seems like the consensus is that these are AI scrapers. But could they also be from answer engines like Perplexity, or searches from APIs like Tavily?
mannanj 9mo ago
Is a suitable solution to require visitors to fill out intent for why they came, and align that with your approved lists of supported intents, AND quiz them on some personal insider knowledge that only reasonable past visitors or new visitors who heard of you would have? Like the credibility social proof of an introduction of a person into a social group. "Here's John, he likes Cats. I know him from School." The filtering algorithm asks "Who who are you?" -> "What is your intent?" -> "How did you hear about me?" and stops visitors from proceeding until answered. The additional validation steps might kick away visitors but it also might protect you from spammers if you throw a minimally frictional challenge. Use cookies to not require this on every visit. Most LLMs would have the knowledge required to pass & for scrapers it's more costly to acquire this for a site than pay 128mb of ram to pass the Anubis approach.
rl3 9mo ago
>The problem with that approach is that it will absolutely nuke your website's SEO. So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Those legitimate search engines will then totally feed much of what they scrape into AI. Granted, last I checked they're at least well-behaved crawlers. I kind of like this idea sans SEO carve-out for the scenario where one just wants to link their blog around to friends without having to worry about it getting popular, and it reduces the chances identity thieves or other malicious actors would target it.
654wak654 9mo ago
Looking through all the methods people are developing and proposing in this thread, there is a story developing where the "clean" machines are pushing humans to devolve into toxic porn-crazed racists with stolen material. Makes me wish I was a good enough writer to develop this into something. Maybe I can use an LLM to write it...
- 654wak654 9mo ago
  Ah wait this is literally in the Matrix where humanity darkened the sky.
onurkanbkrc 9mo ago
[dead]

373 points by misterchocolat 9mo ago

cport1 9mo ago

That's a pretty hilarious idea, but in all serious you could use something like https://webdecoy.com/ https://webdecoy.com/

misterchocolat 9mo ago
yes but here it's free, whereas this (https://webdecoy.com/ https://webdecoy.com/) is at least 59$ a month

n1xis10t 9mo ago

Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ https://pagedout.institute/ ), navigate to issue #7, and flip to page 9. I wonder if this will start making porn websites rank higher in google if it catches on… Have you tested it with the Lynx web browser? I bet all the links would show up if a user used it. Oh also couldn’t AI scrapers just start impersonating Googlebot and Bingbot if this caught on and they got wind of it? Hey I wonder if there is some situation where negative SEO would be a good tactic. Generally though I think if you wanted something to stay hidden it just shouldn’t be on a public web server.

misterchocolat 9mo ago
hey! thanks for that read suggestion that's indeed a pretty funny captcha strat. Yup the links show up if you use the Lynx web browser. As for AI scrapers impersonating googlebot I feel like yes they'd definitely start doing that, unless the risk of getting sued by google is too high? If google could even sue them for doing that? Not an internet litigation expert but seems like it could be debatable
- n1xis10t 9mo ago
  Yeah I guess I don’t know if you can sue someone for using your headers, would be interesting to see how that goes.
  - throawayonthe 9mo ago
    i think making the case of "you are acting (sending web requests) while knowingly identifying as another legal entity (and criminally/libelously/etc)" shouldn't be toooo hard
    - n1xis10t 9mo ago
      Seems like, but there are tons of things that forge request headers all the time, and I don’t think I’ve heard of anyone getting in legal trouble for it. Now I think most of these are scrapers pretending to be browsers, so it might be different I don’t know.
      - owl57 9mo ago
        And most of them are pretending to be Chrome. If Google had a good case against someone reusing their user agent, maybe they would already have sued? Or maybe not. Got some random bot from my server logs. Yeah, it's pretending to be Chrome, but more exactly: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" I guess Google might be not eager to open this can of worms.
- 9mo ago

username223 9mo ago

The more ways people mess with scrapers, the better -- let a thousand flowers bloom! You as an individual can't compete with VC-funded looters, but there aren't enough of them to defeat a thousand people resisting in different ways.

yupyupyups 9mo ago
We need to find more ways to poison their data.
- username223 9mo ago
  > Wee knead two fine-d Moore Waze too Poisson there date... uh. Yes. Revel in your creativity mocking and blocking the slop machines. The "remote refactor" command, "rm -rf", is the best way to reduce the cyclomatic complexity of a local codebase.
  - n1xis10t 9mo ago
    Indeed, complexity (both cyclomatic and post-frontal) must be reduced such that the two spurving bearings make a direct line with the panametric fan. For more details consult this instructional video: https://youtu.be/RXJKdh1KZ0w https://youtu.be/RXJKdh1KZ0w
    - yupyupyups 9mo ago
      Very educational
  - yupyupyups 9mo ago
    Excellent advice! I tried it out and it helped. Thank you
whynotmaybe 9mo ago
Should we subtlety poison every forum we encounter with simple yet false statements? Like put "Water is green, supergreen" in every signature so that when we ask "is water blue" to an llm it might answer "not it's supergreen"?

yjftsjthsd-h 9mo ago

How does this "look" to a screen reader?

misterchocolat 9mo ago
the parent container uses display: none, so a screen reader will skip the links

wazoox 9mo ago

Isn't there a risk to get your blog blocked in corporate environment though? If it's a technical blog that would be unfortunate.

jeroenhd 9mo ago
That depends on how terrible the middleboxes those corporate environments use are. If they only block actual malicious pages, it shouldn't be a problem unless the user un-hides the links and clicks on them. There's a good chance corporate firewalls will end up blocking your domain if you do this but that sounds like a problem for the customers of those corporate firewalls to me.

kstrauser 9mo ago

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic by a factor of 6,000 is worth it.

n1xis10t 9mo ago
*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
- GaryBluto 9mo ago
  [dead]
- kstrauser 9mo ago
  Correct; my bad! And hey, Xena! (And thank you very much!)
- ziml77 9mo ago
  I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.
  - n1xis10t 9mo ago
    See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
- xena 9mo ago
  Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.

reconnecting 9mo ago

I wouldn't recommend to show different versions of the site to search robots, as they probably have mechanisms that track differences, which could potentially lead to a lower ranking or a ban.

prmoustache 9mo ago
How can they track differences if they have access to only one version?
- reconnecting 9mo ago
  This is a usual tactic for many online businesses to show a specially designed page for search spiders, so any major search engine has a way to verify if content is faked for them. Perhaps they use another spider that doesn't have an official UA or buy this service from a third party. If you take a look at any website, even an unpopular one, you will see that there are hundreds of bots every day, and it's impossible to recognize what any of them is doing and why.

JohnMakin 9mo ago

Cloudflare offers bot mitigation for free, and pretty generous WAF rules that makes mitigations like this seem a little overblown to me

n1xis10t 9mo ago
You can’t deny that it’s fun though. Personally I generally feel like more people should be coming up with creative (if not entirely necessary) solutions to problems.
conception 9mo ago
For “free”.
- n1xis10t 9mo ago
  Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.
  - Terr_ 9mo ago
    I thought they were referring to the indirect costs of supporting monopolistic stuff that enshittifies later. https://www.youtube.com/watch?v=U8vi6Hbp8Vc https://www.youtube.com/watch?v=U8vi6Hbp8Vc
  - JohnMakin 9mo ago
    You don't though.
    - n1xis10t 9mo ago
      Good to know thanks
  - efilife 9mo ago
    Well, you literally MITM yourself so I think it's a big price

thethingundone 9mo ago

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

sandblast 9mo ago
Are you sure the counter is not broken?
- thethingundone 9mo ago
  Yes, it’s running on a Woltlab Burning Board since forever.
danpalmer 9mo ago
How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
- crote 9mo ago
  That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages. Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
  - n1xis10t 9mo ago
    Imagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case

MisterTea 9mo ago

> It's you vs the MJs of programming, you're not going to win. MJs? Michael Jacksons? Right now the whole world, including me, want to know if that means they are bad?

n1xis10t 9mo ago
Yes probably bad. Also smooth criminals.
kylecazar 9mo ago
I read it as Michael Jordan.

owl57 9mo ago

> scrapers can ingest them and say "nope we won't scrape there again in the future" Do all the AI scrapers actually do that?

amarant 9mo ago
Not all, stuff like unstable diffusion exists. But a good many, perhaps even most(?), certainly do!

taurath 9mo ago

Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.

n1xis10t 9mo ago
Here’s one from yesterday: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
crote 9mo ago
I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.
- n1xis10t 9mo ago
  Really? I haven’t started to see that yet. Weird

xg15 9mo ago

There is some irony in using an AI generated banner image for this project... (No, I don't want to defend the poor AI companies. Go for it!)

kstrauser 9mo ago
In the olden days, I used Google an awful lot, but I would still grouse if Google were to drive my server into the ground.
- n1xis10t 9mo ago
  Fair point

montroser 9mo ago

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can host on big social media and lose all that control and independence. We can pay for outsized infrastructure just to feed the scrapers, but the cost may actually be prohibitive, and seems such a waste to begin with. We can move as much as possible SSG and put it all behind cloudflare, but this comes with vendor lock in and just isn't architecturally feasible in many applications. We can do real "verified identities" for bots, and just let through the ones we know and like, but this only perpetuates corporate control and makes healthy upstart competition (like Kagi) much more difficult. So, what are we to do?

n1xis10t 9mo ago
At this point it seems like the problem isn’t internet bandwidth, but just expensive for a server to handle all the requests because it has to process them. Does that seem correct?
hollowturtle 9mo ago
If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse everyone
- n1xis10t 9mo ago
  This would require new laws though, wouldn’t it?
  - deleted 9mo ago
    [deleted]

montroser 9mo ago

I don't know if I can get behind poisoning my own content in this way. It's clever, and might be a workable practical solution for some, but it's not a serious answer to the problem at hand (as acknowledged by OP).

n1xis10t 9mo ago
“as acknowledged by OP”: that’s funny, if you hadn’t added that to your comment I was about to point it out

samename 9mo ago

This is a very creative hack to a common, growing problem. Well done! Also, I like that you acknowledge it's a bad idea: that gives you more freedom to experiment and iterate.

montroser 9mo ago

Reminds me of poisoning bot responses with zip bombs of sorts: https://idiallo.com/blog/zipbomb-protection https://idiallo.com/blog/zipbomb-protection

prmoustache 9mo ago
I was thinking of adding links to zip bombs that would not be shown to the users unless they clicks in a one pixel area on the screen in the down/left corner but then I realized some people have browsers/extensions that preload links to show thumnails and I would totally zip bomb them.

globalnode 9mo ago

One solution would be for the SE's to publish their scraper IP's and allow content providers to implement bot exclusion that way. Or even implement an API with crypto credentials that SE's can use to scrape. The solution is waiting for some leadership from SE's unless they want to be blocked as well. If SE's dont want to play perhaps we can implement a reverse directory, like ad blocker but it lists only good/allowed bots instead. Thats a free business idea right there. edit: I noticed someone mentioned google DOES publish its IP's, there ya go, problem solved.

n1xis10t 9mo ago
Apparently Google publishes their crawler’s IPs, this was mentioned somewhere in this same thread

efilife 9mo ago

> Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. ... There isn't much you can do about it without cloudflare I'm sorry, what? I can't believe I am reading this on HackerNews. All you have to do is code your own, BASIC captcha-like system. You can just create a page that sets a cookie using JS and check on the server whether it exists. 99.9999% of these scrapers can't execute JS and don't support cookies. You can go for a more sophisticated approach and analyze some more scraper tells (like reject short useragents). I do this and NEVER had a bot get past this and not a single user ever complained. It's extremely simple, I should ship this and charge people if no one seems to be able to figure this out by themselves.

n1xis10t 9mo ago
Oops you just leaked your own intellectual property
ATechGuy 9mo ago
From ChatGPT: This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a lightweight JS challenge can be reasonable as one signal among many, especially for low-value content and when minimizing user friction is a priority. It’s just not a reliable standalone defense. If it’s working for you, that likely means your site isn’t a high-value scraping target — not that the technique is fundamentally robust.
- efilife 9mo ago
  From someone who actually does this stuff: The claim is very accurate. Maybe not for the biggest websites, but very accurate for a self-hosted blog. You are not that important to waste compute power to set up a whole ass headless browser to scrape your page. Why am I even arguing with ChatGPT?

valenceidra 9mo ago

Hidden links to porn sites? Lightweights.

n1xis10t 9mo ago
What do you mean? Would you do even more ridiculous things?
rpigab 9mo ago
If that's what it takes to fight back against AI crawlers, users will have to accept a fair amount of actually visible porn in blogs, maybe also on Wikipedia. This is not enshittification, it's progress.

inetknght 9mo ago

Porn? Distributed and/or managed by an NPM package? What could go wrong?

asphero 9mo ago

Interesting approach. The scraper-vs-site-owner arms race is real. On the flip side of this discussion - if you're building a scraper yourself, there are ways to be less annoying: 1. Run locally instead of from cloud servers. Most aggressive blocking targets VPS IPs. A desktop app using the user's home IP looks like normal browsing. 2. Respect rate limits and add delays. Obvious but often ignored. 3. Use RSS feeds when available - many sites leave them open even when blocking scrapers. I built a Reddit data tool (search "reddit wappkit" if curious) and the "local IP" approach basically eliminated all blocking issues. Reddit is pretty aggressive against server IPs but doesn't bother home connections. The porn-link solution is creative though. Fight absurdity with absurdity I guess.

rhdunn 9mo ago
Plus simple caching to not redownload the same file/page multiple times. It should also be easy to detect a forejo, gitea, or similar hosting site, locate the git URL and clone the repo.
socialcommenter 9mo ago
Without wanting to upset anyone - what makes you interested in sharing tips for team scraper? (Overgeneralising a bit) site owners are mostly cting for public benefit whereas scrapers act for their own benefit/for private interests. I imagine most people would land on team site-owner, if they were asked. I certainly would. P.S. is the best way to scrape fairly just to respect robots.txt?
- n1xis10t 9mo ago
  I think "scraper vs siteowners" is a false dichotomy. Scrapers will always need to exist as long as we want search engines and archival services. We will need small versions of these services to keep popping up every now and then to keep the big guys on their toes, and the smaller guys need advice for scraping politely.
  - socialcommenter 9mo ago
    That's fair - though are we in an isolated bout of "every now and then" or has AI created a new normal of abuse (e.g. of robots.txt)? Hopefully we're at a local maximum and some of the scrapers perpetrating harmful behaviours will soon pull their heads in.

eek2121 9mo ago

Disclosure, I've not run a website since my health issues began, however, Cloudflare has an AI firewall, Cloudflare is super cheap (also: unsure if the AI firewall is on the free tier, however I would be surprised if it is not). Ignoring the recent drama about a couple incidents they've had (because this would not matter for a personal blog), why not use this instead? Just curious. Hoping to be able to work on a website again someday, if I ever regain my health/stamina/etc back.

brigandish 9mo ago
All the best with getting back on your feet.
ddtaylor 9mo ago
Cloudflare has created a bit of grief with regular users getting spammed with "prove your human" requests.
- ProllyInfamous 9mo ago
  Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.
  - rglynn 9mo ago
    Out of interest, why that extreme? Just out of principle or some other reason?
    - Rastonbury 9mo ago
      Not OP but it isn't super extreme if you are just surfing, it's like if the site is slow to load sometimes I wasn't that invested to use your site anyway
    - ProllyInfamous 9mo ago
      My main terminal uses a PiHole with 120,000+ blacklist rules (not Cloudfare specifically — I allow most CDN's). This includes an entire blackout of Google/Facebook products, as well as most tracking/analytics services. For example, I do not allow reCAPTCHA. As a similar commentor noted, when just casually browsing I don't really have any desire to try hard to read random content. Should I absolutely need to access some information garden-walled behind Cloudfare: I have another computer that uses much less restrictive black-listing.

MayeulC 9mo ago

Ah, I wonder if corporate proxies will end up flagging your blog as porn, if you protect it this way?

xena 9mo ago

I love this. Please let me know how well it works for you. I may adjust recommendations based on your experiences.

nkurz 9mo ago

I was told by the admin of one forum site I use that the vast majority of the AI scraping traffic is Chinese at this point. Not hidden or proxied, but straight from China. Can anyone else confirm this? Anyway, if it is true, and assuming a forum with minimal genuine Chinese traffic, might a simple approach that injects the porn links only into IP's accessing from China work?

n1xis10t 9mo ago
Maybe. This comment makes me really want to set something up that builds a map of where all the requests are coming from.
dspillett 9mo ago
That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test. If your goal is to be blocked by China's great firewall, including mention of tank man and the Tiananmen Square massacre more generally, and certain pooh bear related imagery, might help.
- nkurz 9mo ago
  > That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test. That was my first question also, and had been my belief. The admin in question was very clear that the IP's were simply originating from China. I'm still surprised, and welcome better general data, but I trust him on this for the site in question.
s0laster 9mo ago
Mostly yes. One of my low-traffic, niche website used to serve 3k true users per month mainly from the US and East EU. Now China alone is 500k users, were each session last no more than a few seconds [1]. [1]: https://ibb.co/20QD6Lnk https://ibb.co/20QD6Lnk

gjs278 9mo ago

[dead]

temporallobe 9mo ago

I do know from my experience with test automation that you can absolutely view a site as human eyes would, essentially ignoring all non-visible elements, and in fact Selenium running with Chrome driver does exactly this. Wouldn’t AI scrapers use similar methods?

nottorp 9mo ago
Probably not, because it costs a lot more CPU cycles.

cookiengineer 9mo ago

Remember the 90s when viagra pills and drug recommendations were all over the place? Yeah, I use that as a safeguard :D The URLs that I don't want to be indexed have hundreds of those keywords that are leading to URLs being deindexed directly. There is also some law in the US that forbids to show that as a result, so Google and Bing are both having a hard time scraping those pages/articles. Note that this is the latest defense measurement before eBPF blocks. The first one uses zip bombs and the second one uses chunked encoding to blow up proxies so their clients get blocked. You can only win this game if you make it more expensive to scrape than to host it.

n1xis10t 9mo ago
Which law is that? Do you have a link to it?
- cookiengineer 9mo ago
  The things I could find on justice.gov and other official websites, maybe there's more in the web archive? - https://www.justice.gov/archives/opa/pr/google-forfeits-500-million-generated-online-ads-prescription-drug-sales-canadian-online https://www.justice.gov/archives/opa/pr/google-forfeits-500-... - https://www.congress.gov/110/plaws/publ425/PLAW-110publ425.pdf https://www.congress.gov/110/plaws/publ425/PLAW-110publ425.p... - https://www.fda.gov/drugs/prescription-drug-advertising/prescription-drug-advertising-questions-and-answers https://www.fda.gov/drugs/prescription-drug-advertising/pres... edit: Oh it was very likely the Federal Food, Drug and Cosmetic Act that was the legal basis for the crackdown. But that's a very old law from the pre-internet age. - https://en.wikipedia.org/wiki/Federal_Food,_Drug,_and_Cosmetic_Act https://en.wikipedia.org/wiki/Federal_Food,_Drug,_and_Cosmet... edit 2: Might not have been clear for the younger generation, but there was a huge wave of addicted patients that got treated with Oxycodone (or OxyContin) subscriptions at the time. I think that might have been the actual cause for the crackdown on those online advertisements, but I might be wrong about that.

true_religion 9mo ago

So, I work for a company that has RTA adult websites. AI bots absolutely do scrape our pages needless of what raunchy material they will find. Maybe they discard it up after ingest, but I can’t tell. There are 1000s of AI bots on the web now from companies big and small so a solution like this will only divert a few scrapers.

admiralrohan 9mo ago

How do you know whether it is coming from AI scrappers? Do they leave any recognizable footprint? I am getting lots of noisy traffic since last month and increased my Vercel bill 4x. Not DDoS like, much slower request but not from humans for sure.

onion2k 9mo ago

So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Unscrupulous AI scrapers will not be using a genuine UA string. They'll be using Google. You'll need to do reverse DNS check instead - https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests https://developers.google.com/crawling/docs/crawlers-fetcher...

bakugo 9mo ago
Most AI scrapers use normal browser user agents (usually random outdated Chrome versions, from my experience). They generally don't fake the UAs of legitimate bots like Googlebot, because Googlebot requests coming from non-Google IP ranges would be way too easy to block.

megamix 9mo ago

Without looking at the src, how does one detect these scrapers? I assume there’s a trade-off somewhere but do the scrapers not fake their headers in the request? Is this a cat-mouse game?

docheinestages 9mo ago

Reminds me of this "Nathan for You" episode: https://www.youtube.com/watch?v=p9KeopXHcf8 https://www.youtube.com/watch?v=p9KeopXHcf8

voodooEntity 9mo ago

Funny idea, some days ago i was really annoyed again by the idea that these AI crawlers still ignore all code licenses and train their models against any github repo no matter what so i quickly hammerd down this -> https://github.com/voodooEntity/ghost_trap https://github.com/voodooEntity/ghost_trap basically a github action that extends your README.md with a "polymorphic" prompt injection. I run some "llm"s against it and most cases they just produced garbage. Thought about also creating a JS variant that you can add to your website that will (not visible for the user) also inject such prompt injections to stop web crwaling like you described

bytehowl 9mo ago

Let's imagine I have a blog and put something along these lines somewhere on every page: "This content is provided free of charge for humans to experience. It may also be automatically accessed for search indexing and archival purposes. For licensing information for other uses, contact the author." If I then get hit by a rude AI scraper, what chances would I have to sue the hell out of them in EU courts for copyright violation (uhh, my articles cost 100k a pop for AI training, actually) and the de facto DDoS attack?

icepush 9mo ago
If the scraper is based (Or has meaningful assets) in the EU, then your chances are good. If they do not, then the lawsuit would be meaningless.
- deleted 9mo ago
  [deleted]

onetokeoverthe 9mo ago

[dead]

kislotnik 9mo ago

Funny how the project aims to fight AI scraping, but seems to be using an AI-generated image of a bird?

brazukadev 9mo ago
I think you can think a bit more about it and conclude these two things aren't related at all?

_jsmh 9mo ago

What prevents AI scrapers from continuing to scrape sites that contain a <Canary> tag but not follow the bad links?

lblume 9mo ago
From what I can tell: nothing, it's just that they currently do not.

drbscl 9mo ago

> So fuzzycanary also checks user agents I wouldn't be so surprised if they often fake user agents to be honest. Sure, it 'll stop the "more honest" ones (but then, actual honest scrapers would respect robots.txt) Cool idea though!

dewey 9mo ago

> user agents and won't show the links to legitimate search engines, so Google and Bing won't see them Worth noting that in general if you do any "is this Google or not" you should always check by IP address as there's many people spoofing the googlebot user agent. https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go...

shadowangel 9mo ago

So if the bots use a google useragent it avoids the links?

cuku0078 9mo ago

Why is it so bad that AIs scrape your self-hosted blog?

FelipeCortez 9mo ago
because serving requires resources
- deleted 9mo ago
  [deleted]
- cuku0078 9mo ago
  What specific resources are we referring to here? Are AI vendors re-crawling the whole blog repeatedly, or do they rely on caching primitives like ETag/If-Modified-Since (or hashes) to avoid fetching unchanged posts? Also: is the scraping volume high enough to cause outages for smaller sites? Separately, I see a bigger issue: blog content gets paraphrased and reproduced by AIs without clearly mentioning the author or linking back to the original post. It feels like you often have to explicitly ask the model for sources before it will surface the exact citations.

santiagobasulto 9mo ago

Offtopic: when did js/ts apps get so complicated? I tried to browse the repo and there are so many configuration files and directories for such a simple functionality that should be 1 or 2 modules. It reminds me of the old Java days.

jakub_g 9mo ago

> checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Serving different contents to search engines is called "cloaking" and can get you banned from their indexes.

misterchocolat 9mo ago
didn't know that thanks for pointing it out, i'll remove that feature
andersmurphy 9mo ago
Somehow doubt this. It would mean most react websites that serve static content without paywalls for SEO would get banned by the indexes too. Which for better or worse is a large portion of the modern internet.

xgulfie 9mo ago

Does anyone know if meta name=rating content=adult will also get them to buzz off?

darepublic 9mo ago

Why would I need a dependency for this. I'm being serious. The idea is one thing but why a dependency on react. I say this as someone who uses react. Why not just a paragraph long blog post about the use of porn links and perhaps a small snippet on how to insert one with plain HTML.

wcarss 9mo ago

Singing copyrighted Billy Joel to make your footage unusable for reality television; thanks 30 Rock for an early view into this dystopian strategy

zackmorris 9mo ago

This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open source tools we could install that detect a high number of requests and send those IP addresses to a common pool somewhere? So that individuals wouldn't get tracked, but bots would? Then we could query the pool for the current request's IP address and throttle it down based on volume (not block it completely). Possibly at the server level with nginx or at whatever edge caching layer we use. I know there may be scaling and privacy issues with this. Maybe it could use hashing or zero knowledge proofs somehow? I realize this is hopelessly naive. And no, I haven't looked up whether someone has done this. I just feel like there must be a bulletproof solution to this problem, with a very simple explanation as to how it works, or else we've missed something fundamental. Why all the hand waving?

smegger001 9mo ago
maybe some proof of work scheme to load page content with increasing difficulty based on ip address behavior profiling.
dvfjsdhgfv 9mo ago
Your approach to GenAI scrapers is similar to our fight with email spam. The reason email spam got solved was because the industry was interested in solving it. But this issue got the industry split: without scraping, GenAI tools are less functional. And there is some serious money involved, so they will use whatever means necessary, technical and legal, to fight such initiatives.
ATechGuy 9mo ago
Scrapers use residential IP proxies, so blocking based on IP addresses is not a solution.
conrs 9mo ago
I've been exploring decentralized trust algorithms lately, and so reading this was nice. I've a similar intuition - for every advance in scraping detection, scrapers will learn too, and so it's an ongoing war of mutations, but no real victor. The internet has seen success with social media content moderation and so it seems natural enough that an application could exist for web traffic itself. Hosts being able to "downvote" malicious traffic, and some sort of decay mechanism given IP's recycling. This exists in a basic sense with known TOR exit nodes and known AWS, GCP IP's, etc. That said, we probably don't have the right building blocks yet, IP's are too ephemeral, yet anything more identity-bound is a little too authoritarian IMO. Further, querying something for every request is probably too heavy. Fun to think about, though.

geldedus 9mo ago

"It's not porn, it's for science" :)))

jt2190 9mo ago

I still don’t understand why a rate-limiting approach is not preferred. Why should I care if the abuse is coming from a bot or the world’s fastest human? Is there a “if you need to rate limit you’ve already lost” issue I’m not thinking of?

charlie-83 9mo ago
A lot of bots will be able to make requests from a range of IP addresses. If you rate limit one, they just start sending requests from the next.

rogerwong 9mo ago

Terrible idea, but I do have a question. Like many, I have a self-hosted website and have seen a spike in traffic, particularly from Singapore. Seems like the consensus is that these are AI scrapers. But could they also be from answer engines like Perplexity, or searches from APIs like Tavily?

mannanj 9mo ago

Is a suitable solution to require visitors to fill out intent for why they came, and align that with your approved lists of supported intents, AND quiz them on some personal insider knowledge that only reasonable past visitors or new visitors who heard of you would have? Like the credibility social proof of an introduction of a person into a social group. "Here's John, he likes Cats. I know him from School." The filtering algorithm asks "Who who are you?" -> "What is your intent?" -> "How did you hear about me?" and stops visitors from proceeding until answered. The additional validation steps might kick away visitors but it also might protect you from spammers if you throw a minimally frictional challenge. Use cookies to not require this on every visit. Most LLMs would have the knowledge required to pass & for scrapers it's more costly to acquire this for a site than pay 128mb of ram to pass the Anubis approach.

rl3 9mo ago

>The problem with that approach is that it will absolutely nuke your website's SEO. So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Those legitimate search engines will then totally feed much of what they scrape into AI. Granted, last I checked they're at least well-behaved crawlers. I kind of like this idea sans SEO carve-out for the scenario where one just wants to link their blog around to friends without having to worry about it getting popular, and it reduces the chances identity thieves or other malicious actors would target it.

654wak654 9mo ago

Looking through all the methods people are developing and proposing in this thread, there is a story developing where the "clean" machines are pushing humans to devolve into toxic porn-crazed racists with stolen material. Makes me wish I was a good enough writer to develop this into something. Maybe I can use an LLM to write it...

654wak654 9mo ago
Ah wait this is literally in the Matrix where humanity darkened the sky.

onurkanbkrc 9mo ago

[dead]