20 ms·
Miasma: A tool to trap AI web scrapers in an endless poison pit
- nosmokewhereiam 6mo agoMy asthmar I'm assuming this is a reference to Lord of the flies
- cwnyth 6mo agoMiasma is bad or poisonous air. It's a Greek word.
- obsidianbases1 6mo agoI know there are real world problems to deal with, but at least I got one over on that evil open claw instance /s
- meta-level 6mo agoIsn't posting projects like this the most visible way to report a bug and let it have fixed as soon as possible?
- suprfsat 6mo ago"disobeys robots.txt" is more of a feature
- imdsm 6mo agoApplied model collapse
- splitbrainhack 6mo ago-1 for the name
- QuantumNomad_ 6mo agohttps://en.wikipedia.org/wiki/Miasma_theory https://en.wikipedia.org/wiki/Miasma_theory Seems a clever and fitting name to me. A poison pit would probably smell bad. And at the same time, the theory that this tool would actually cause “illness” (bad training data) in AI is not proven.
- Imustaskforhelp 6mo agoI wish if there was some regulation which could force companies who scrape for (profit) to reveal who they are to the end websites, many new AI company don't seem to respect any decision made by the person who owns the website and shares their knowledge for other humans, only for it to get distilled for a few cents.
- joquarky 6mo agoYep, they are already working on de-anonymizing the internet.
- GaggiX 6mo agoThese projects are the new "To-Do List" app.
- madeofpalk 6mo agoIs there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.
- raincole 6mo agoIt might work against people just use their Mini Mac with OpenClaw to summarize news every morning, but it certainly won't work against Google. More centralized web ftw.
- otherme123 6mo agoIn my experience, Google (among others) plays nice. Just put "disallow: *" in your robots.txt, and they won't bother you again. My current problem is OpenAI, that scans massively ignoring every limit, 426, 444 and whatever you throw at them, and botnets from East Asia, using one IP per scrap, but thousands of IPs.
- hexage1814 6mo agoIt also probably won't work if the person actually wants your content and is checking if the thing they scraped actually makes sense or it just noise. Like, none of these are new things. Site owners send junk/fake data to webscrapers since web scraping was invented.
- LaGrange 6mo ago> It might work against people just use their Mini Mac with OpenClaw to summarize news every morning, Good enough for me. > More centralized web ftw. This ain't got anything to do with "centralized web," this kind of epistemological vandalism can't be shunned enough.
- nubg 6mo agoWhat kind of migitations? How would you detect the poison fountain?
- 6mo ago
- rvz 6mo ago> > Be sure to protect friendly bots and search engines from Miasma in your robots.txt! Can't the LLMs just ignore or spoof their user agents anyway?
- phoronixrly 6mo agoWell-behaved agents will obey robots.txt and not fall into the trap.
- snehesht 6mo agoWhy not simply blacklist or rate limit those bot IP’s ?
- aduwah 6mo agoThere are way too many to do that
- snehesht 6mo agoTrue, most of the blacklists systems today aren’t realtime like Amazon WAF or Cloudflare. We need a Crawler blacklist that can in realtime stream list deltas to centralized list and local dbs can pull changes. Verified domains can push suspected bot ips, where this engine would run heuristics to see if there is a patters across data sources and issue a temporary block with exponential TTL. There are many problems to solve here, but as any OSS it will evolve over time if there is enough interest in it. Costs of running this system will be huge though and corp sponsors may not work but individual sponsors may be incentivized as it’s helps them reduce bandwidth, compute costs related to bot traffic.
- pixl97 6mo agoIn the real-time spam market the lists worked well with honest groups for a bit, but started falling apart when once good lists get taken over by actors that realize they can use their position to make more money. It's a really difficult trap to avoid.
- phyzome 6mo agoBecause punishment for breaking the robots.txt rules is a social good.
- xprnio 6mo agoIf you have real traffic and bot traffic, you still need to identify which is which. On top of that, bots very likely don’t reuse the same IPs over and over again. I assume if we knew all the IPs used only by bots ahead of time, then yeah it would be simple to blacklist them. But although it’s simple in theory, the practice of identifying what to blacklist in the first place is the part that isn’t as simple
- SophieVeldman 6mo ago[flagged]
- obsidianbases1 6mo agoWhy do this though? It's like if someone was trying to "trap" search crawlers back in the early 2000s. Seems counterproductive
- bilekas 6mo agoBecause of bots that don't respect ROBOTS.txt . If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
- obsidianbases1 6mo agoIf bandwidth cost is a concern the maybe you should reconsider how you publish your site. Like, what if you actually post something that gains traction, is it going to bankrupt you or something?
- bilekas 6mo agoIt's not just financial, you're taking up a lot of bandwidth, resources etc. It's not just some light bump in traffic. It's a headache that shouldn't need to be dealt with if they would respect ROBOTS.txt. Quite simple really.
- Forgeties79 6mo agoWeb crawlers didn’t routinely take down public resources or use the scraped info to generate facsimiles that people are still having ethical debates over. Its presence didn’t even register and it was indexing that helped them. It isn’t remotely the same thing. https://www.libraryjournal.com/story/ai-bots-swarm-library-cultural-heritage-sites-causing-slowdowns-and-crashes https://www.libraryjournal.com/story/ai-bots-swarm-library-c...
- obsidianbases1 6mo agoAI bots must've taken down that link you shared, it won't load :/ And search crawlers/results have been producing snippets that prevent users from clicking to the source for well over a decade. Edit: it loaded. I don't see how the problem isn't simply solved by an off the shelf solution like cloud flare. In the real world, you wouldn't open up a space/location if you couldn't handle the throughput. Why should online spaces/locations get special treatment?
- deleted 6mo ago[deleted]
- firekey_browser 6mo ago[dead]
- tasuki 6mo ago> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
- spiderfarmer 6mo agoIf someone hands out cookies in the supermarket, are you allowed to grab everything and leave?
- GaggiX 6mo agoI will copy the supermarket and paste it somewhere else. I'm also going to download a car.
- falcor84 6mo agoThat really depends, but the quick answer is that according to our human social contract, we'd just ask "how many can I take?". Until now, the only real tool to limit scrapers has been throttling, but I don't see any reason for there not to be a similar conversational social contract between machines.
- volemo 6mo agoIsn’t robots.txt such a “social contract between machines”? But AI scrapers couldn’t care less.
- drfloyd51 6mo agoOdd thing about cookies… they disappear after one serving. Websites are an endless stream of cookies. The analogy doesn’t hold.
- z3c0 6mo agoDigital information may be our first post-scarce resource. It's interesting, and sad, to see so many attempt to fit it within scarcity-based economic models.
- aldousd666 6mo agoThis is ultimately just going to give them training material for how to avoid this crap. They'll have to up their game to get good code. The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. The bottom has always been threatening to fall out of the ads paid for eyeballs, And nobody could anticipate the trigger for the downfall. Looks like we found it.
- Apocryphon 6mo agoTech is just a series of arms races
- aldousd666 6mo agoTo be clear, I mean AI is going to be the downfall of ad supported content. But let's face it. We have link farms and spam factories as a result of the ad supported content market. I think this is going to eventually do justice for users because it puts a premium on content quality that someone will want to pay a direct licensing fee to scrape for your AI bots as opposed to tricking somebody into clicking on a link and looking at an impression for something they won't buy.
- johneth 6mo ago> This is ultimately just going to give them training material for how to avoid this crap. > The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. So we should all just do nothing and accept the inevitable?
- ninjagoo 6mo ago> So we should all just do nothing and accept the inevitable? I daresay rate-limiting will result in better outcomes than well-poisoning with hidden links that are against the policies of search engines. Lots of potential for collateral damage, including your own websites' reputations and search visibility, with the well-poisoning approach.
- foxes 6mo agoWonder if you can just avoid hiding it to make it more believable Why not have a library of babel esq labrinth visible to normal users on your website, Like anti surveillance clothing or something they have to sift through
- devnotes77 6mo ago[dead]
- theandrewbailey 6mo agoOr you can block bots with these (until they start using them) https://developer.mozilla.org/en-US/docs/Glossary/Fetch_metadata_request_header https://developer.mozilla.org/en-US/docs/Glossary/Fetch_meta...
- jstanley 6mo agoIf you want to ruin someone's web experience based on what kind of thing they are, rather than the content of their character, consider that you might be the baddies.
- mrweasel 6mo agoIf you're constantly being harassed by someone and despite your best efforts, nothing is being done to help you, quite the opposite in fact, tons of people cheer your assailant on in the name of profit and progress, it's only natural that you lash out. It's not all that productive, it's an act of desperation. If you can't stop the enemy, at least you can make their action more costly. One positive outcome I could see it AI companies becoming more critical of their training data.
- lifeformed 6mo agoWhat "content of character" do you ascribe to a web scraper?
- jstanley 6mo agoYou don't, that's why it's unethical to block them. If you keep getting harrassed by people wearing black hoodies, would it be ethical to start taking countermeasures against all people who wear black hoodies?
- lelanthran 6mo agoIf they are coming to my door to harass me, then yes, it makes sense to take countermeasures against all black-hoodie wearers when I see them at the door.
- Apocryphon 6mo agoYou’re gonna have to try harder to sneak in the a priori assumption that LLMs have any character beyond which corporation deployed them.
- deleted 6mo ago[deleted]
- rob 6mo ago"/brainstorming git checkout this miasma repo source code and implement a fix to prevent the scraper from not working on sites that use this tool"
- ninjagoo 6mo agoThis is essentially machine-generated spam. The irony of machine-generated slop to fight machine-generated slop would be funny, if it weren't for the implications. How long before people start sharing ai-spam lists, both pro-ai and anti-ai? Just like with email, at some point these share-lists will be adopted by the big corporates, and just like with email will make life hard for the small players. Once a website appears on one of these lists, legitimately or otherwise, what'll be the reputational damage hurting appearance in search indexes? There have already been examples of Google delisting or dropping websites in search results. Will there be a process to appeal these blacklists? Based on how things work with email, I doubt this will be a meaningful process. It's essentially an arms race, with the little folks getting crushed by juggernauts on all sides. This project's selective protection of the major players reinforces that effect; from the README: " Be sure to protect friendly bots and search engines from Miasma in your robots.txt! User-agent: Googlebot User-agent: Bingbot User-agent: DuckDuckBot User-agent: Slurp User-agent: SomeOtherNiceBot Disallow: /bots Allow: / "
- maltyxxx 6mo ago[dead]
- ninjagoo 6mo agoIsn't this a trope at this point? That AI companies are indiscriminately training on random websites? Isn't it the case that AI models learn better and are more performant with carefully curated material, so companies do actually filter for quality input? Isn't it also the case that the use of RLHF and other refinement techniques essentially 'cures' the models of bad input? Isn't it also, potentially, the case that the ai-scrapers are mostly looking for content based on user queries, rather than as training data? If the answers to the questions lean a particular way (yes to most), then isn't the solution rate-limiting incoming web-queries rather than (presumed) well-poisoning? Is this a solution in search of a problem?
- xantronix 6mo agoYou do raise an interesting point. The poison fountains would probably be more effective if their outputs more closely resembled whatever the most popular problem spaces are at any given point.
- superkuh 6mo agoOf course Googlebot, Bingbot, Applebot, Amazonbot, YandexBot, etc from the major corps are HTTP useragent spiders that will have their downloaded public content used by corporations for AI training too. Might as well just drop the "AI" and say "corporate scrapers".
- bobosola 6mo agoI dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced. Also, inserting hidden or misleading links is specifically a no-no for Google Search [0], who have this to say: We detect policy-violating practices both through automated systems and, as needed, human review that can result in a manual action. Sites that violate our policies may rank lower in results or not appear in results at all. So you may well end up doing more damage to your own site than to the bots by using dodgy links in this manner. [0]https://developers.google.com/search/docs/essentials/spam-policies https://developers.google.com/search/docs/essentials/spam-po...
- trinsic2 6mo ago>I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced If you are automating it, I don't see why not. Kitboga, a you-tuber kept scam callers in AI call-center loops tying up there resources so they cant use them on unsuspecting victims.[0] That's a guerilla tactic, similar in warfare, when you steal resources from an enemy, you get stronger and they get weaker, its pretty effective. [0]: https://www.youtube.com/watch?v=ZDpo_o7dR8c https://www.youtube.com/watch?v=ZDpo_o7dR8c
- bdangubic 6mo agomore and more scammers are automating their side as well so soon the loop will be just bots talking to bots
- Saline9515 6mo agoThe dead phone theory?
- Bombthecat 6mo agoMore like dead communication theory:)
- eliottre 6mo agoThe data poisoning angle is interesting. Models trained on scraped web data inherit whatever biases, errors, and manipulation exist in that data. If bad actors can inject corrupted data at scale, it creates a malign incentive structure where model training becomes adversarial. The real solution is probably better data provenance -- models trained on licensed, curated datasets will eventually outcompete those trained on the open web.
- kristopolous 6mo agoI did a related approach: A toll charging gateway for llm scrapers: a modification to robots.txt to add price sheets in the comment field like a menu. This was for a hackathon by forking certbot. Cloudflare has an enterprise version of this but this one would be self hosted I think it has legs but I think I need to get pushed and goaded otherwise I tend to lose interest ... It was for the USDC company btw so that's why there's a crypto angle - this might be a valid use case! I'm open to crypto not all being hustles and scams Tell me what you think? https://github.com/kristopolous/tollbot https://github.com/kristopolous/tollbot
- ctoth 6mo agoThis is literally what HTTP 402 is for -- there's a whole buncha work going on ... but please, please, please don't let Cloudflare become another bloody gatekeeper. Please.
- Art9681 6mo agoCan't we simple parse and remove any style="display: none;", aria-hidden="true", and tabindex="1" attributes before the text is processed and get around this trick? What am I missing?
- hoistbypetard 6mo agoIf you do that and don't follow robots.txt, you are blocked. If you do that and follow robots.txt, fine. That's all we wanted you to do anyway. Just follow the instructions that well-behaved scrapers are meant to follow.
- phplovesong 6mo agoJust have the link visible, but css it so that its either small as hell, or just off screen. Google / bots will follow it, real peopple will never see it.
- effnorwood 6mo agocertainly don't allow anyone to access your content. perhaps shut the site down just to be safe.
- aduwah 6mo agoAccessing the shop by going through the wall with a tank is not the same as walking in the door. Hosting costs money. These botnets should be charged for the costs they incur
- holysoles 6mo agoIf anyone is looking for a tool to actually send traffic to a tool like this, I wrote a Traefik plugin that can block or proxy requests based on useragent. https://github.com/holysoles/bot-wrangler-traefik-plugin https://github.com/holysoles/bot-wrangler-traefik-plugin
- hmokiguess 6mo agoCould this lead to something like the Streisand effect? I imagine these bots work at a scale where humans in the loop only act when something deviates from the standard, so, if a bot flags something up with your website then you’re now in a list you previously weren’t. Now don’t ask me what they do with those lists, but I guess you will make the cut.
- CrzyLngPwd 6mo agoWay back in the day I had a software product, with a basic system to prevent unauthorised sharing, since there was a small charge for it. Every time I released an update, and new crack would appear. For the next six months I worked on improving the anti-copying code until I stumbled across an article by a coder in the same boat as me. He realised he was now playing a game with some other coders where he make the copyprotection better, but the cracker would then have fun cracking it. It was a game of whack-a-mole. I removed the copy protection, as he did, and got back to my primary role of serving good software to my customers. I feel like trying to prevent AI bots, or any bots, from crawling a public web service, is a similar game of whack-a-mole, but one where you may also end up damaging your service.
- Cpoll 6mo ago> the cracker would then have fun cracking it. I wonder if you could've won by making the cracking boring. No new techniques, bare minimum changes to require compiling a new crack, and just enough to make it difficult to automate. I.e. turn the cracking into a job. But in reality, there are other community-driven motivations to put out cracks.
- gruez 6mo ago>No new techniques, bare minimum changes to require compiling a new crack, and just enough to make it difficult to automate. From a practical perspective you also have to have a steady stream of features for the newer versions to be worth cracking. Otherwise why use v1.09 when v1.01 works fine? Moreover spending less effort into improving the DRM is still playing at the cat and mouse game, albeit with less time investment. If you're making minimal changes, the cracker also has to spend minimal time updating the crack.
- joquarky 6mo agoSo many problems could be solved by letting go. Unfortunately social media and snowballing copyright maximalism has inflated egos to the point where more and more people think they need to control everything.
- bluepeter 6mo agoA related technique used to work so well for search engine spiders. I had some software i wrote called 'search engine cloaker'... this was back in the early 2000s... one of the first if not the first to do the shadowy "cloaking" stuff! We'd spin dummy content from lists of keywords and it was just piles and piles. We made it a bit smarter using Markov chains to make the sentences somewhat sensible. We'd auto-interlink and get 1000s of links. It eventually stopped working... but it took a long while for that to happen. We licensed the software to others. I rationalized it because I felt, hey, we have to write crappy copy for this stupid "SEO" thing, so let's just automate that and we'll give the spiders what they seem to want.
- ctoth 6mo agoYou didn't 'give the spiders what they seem to want.' You exploited a naive ranking algorithm to inject garbage into search results that real people were trying to use. That you rationalized it at the time is human. That you're still rationalizing it decades later is something else.
- jijji 6mo agowhy not just try to block them at the door instead of feeding them poisoned food...
- dwa3592 6mo agoLove it. Thanks for doing this work. Not sure why people are criticizing this. Also, insane amount of work has been done to improve scraping - which in my mind is just absolute bonkers and i didn't see people complaining about that.
- pugchat 6mo ago[dead]
- deleted 6mo ago[deleted]
- deleted 6mo ago[deleted]
- storus 6mo agoI am failing to see how this stops pre-training scrapping? It still looks like legit code, playing nicely with the desired pre-training distribution. Obviously nobody is going to use it for SFT/DPO/GRPO later.
- jackdoe 6mo agorage against the dying of the light
- iFire 6mo agoI for one welcome everyone to the tarpit where a normal person is seen as a robot in an endless poison pit and sounds like a Black Mirror television episode.
- cdrnsf 6mo agoI keep most things inaccessible behind Tailscale. For any public things I 403 known crawlers when they access anything but robots.txt.
- 101008 6mo agoBased on this comment: > I definitely get this. The thing that gives me hope is that you only need to poison a very small % of content to damage AI models pretty significantly. It helps combat the mass scraping, because a significant chunk of the data they get will be useless, and its very difficult to filter it by hand It'd be great if the code returned by this project is code that doesn't work. Imagine if all these models are being trained with code that looks OK but in the end it just bullshit. I'd be amazing.
- xmcp123 6mo agoI just checked some of the content from miasma, and this appears to be exactly what it does. Everything from loops that won’t end to incorrect function calls and emoji “definitions” that are both realistic and wrong. Very impressive project tbh.
- 250call 6mo agoMiasma is just a wrapper around the "Poison Fountain". You can check out the explanation and sample some of their content here: https://rnsaffn.com/poison3/ https://rnsaffn.com/poison3/ It's pretty much exactly what you're describing: content that looks correct but is deeply insane.
- Andrew_McCarron 6mo ago[flagged]
- Lockal 6mo agoNightshade[1] 2.0? As if both tools were built by incompetent developer to distract attention from a real solution - publishing an llm-friendly version in an machine-friendly format (which is not really difficult and helps not only LLMs: e. g. cache, disable fancy complex syntax highlight, offload to github, provide clients and MCPs, optimize clients for common use cases). This example is simply a failure: <a href="/bots" style="display: none;" aria-hidden="true" tabindex="1"> Amazing high quality data here! </a> Dumb curl-based LLM won't visit display:none links. Smarter browser-based navigators won't even render this link. [1] https://news.ycombinator.com/item?id=39058428 https://news.ycombinator.com/item?id=39058428
- atomic128 6mo agoPoison Fountain: https://rnsaffn.com/poison2/ https://rnsaffn.com/poison2/ Poison Fountain explanation: https://rnsaffn.com/poison3/ https://rnsaffn.com/poison3/ Simple example of usage in Go: package main import ( "io" "net/http" ) func main() { poisonHandler := func(w http.ResponseWriter, req *http.Request) { poison, err := http.Get("https://rnsaffn.com/poison2/") if err == nil { io.Copy(w, poison.Body) poison.Body.Close() } } http.HandleFunc("/poison", poisonHandler) http.ListenAndServe(":8080", nil) } https://go.dev/play/p/04at1rBMbz8 https://go.dev/play/p/04at1rBMbz8 Miasma Poison Fountain Tar Pit: https://github.com/austin-weeks/miasma https://github.com/austin-weeks/miasma Apache Poison Fountain: https://gist.github.com/jwakely/a511a5cab5eb36d088ecd1659fcee1d5 https://gist.github.com/jwakely/a511a5cab5eb36d088ecd1659fce... Nginx Poison Fountain: https://gist.github.com/NeoTheFox/366c0445c71ddcb1086f7e4d9c478fa1 https://gist.github.com/NeoTheFox/366c0445c71ddcb1086f7e4d9c... Discourse Poison Fountain: https://github.com/elmuerte/discourse-poison-fountain https://github.com/elmuerte/discourse-poison-fountain Netlify Poison Fountain: https://gist.github.com/dlford/5e0daea8ab475db1d410db8fcd5b78db https://gist.github.com/dlford/5e0daea8ab475db1d410db8fcd5b7... In the news: The Register: https://www.theregister.com/2026/01/11/industry_insiders_seek_to_poison/ https://www.theregister.com/2026/01/11/industry_insiders_see... Forbes: https://www.forbes.com/sites/craigsmith/2026/01/21/poison-fountain-and-the-rise-of-an-underground-resistance-to-ai/ https://www.forbes.com/sites/craigsmith/2026/01/21/poison-fo... On Reddit: https://www.reddit.com/r/PoisonFountain/ https://www.reddit.com/r/PoisonFountain/
- ed_mercer 6mo ago> Thanks for stopping by! Missed chance to use "slopping by"
- 250call 6mo agoAmazing, that's getting its own PR
- thomaslwang 6mo ago[dead]
- HironoOcto 6mo ago[dead]
- deleted 6mo ago[deleted]
- nsonha 6mo agoHilarious how people proud of the "open web" thinks that it is somehow about the (small) "web" or some shit, and not the "open"
- ada1981 6mo agoIMSIRIUS.com
- RestartKernel 6mo agoThe real story is in the poison fountain dataset this uses: https://rnsaffn.com/poison3/ https://rnsaffn.com/poison3/ > [...] we want to inflict damage on machine intelligence systems. This almost strikes me as roleplay, but maybe I'm childish for finding it difficult to empathise with this genre of hacker ideology.
- stingraycharles 6mo agoAnd nowhere does anyone explain exactly what “poisoned data” is and just how the claim that training a model on a small amount of such data will have a big impact.
- Mars008 6mo agoThere is no exact definition. It can be a harmless bullshit, or something more harmful. Imagine AI recommending child to drink some common liquid for fun or as a medicine. Liquid which is in fact known poison. There are many dangerous things that aren't often mentioned in internet because people have common sense and never do it. It's enough to add just a bit of misleading information. With cheap generic robots coming this can be a real problem. Human supervision can help when there is one.
- thomaslwang 6mo ago[dead]
- thomaslwang 6mo ago[dead]
- makingstuffs 6mo agoI love the idea but this will only end up harming your SME in the long run. It would also further entrench the large corps. The only way something like this would be remotely plausible as a concept would be for enough data providers with overlapping authority on given topics to implement it. Sadly SMEs have no choice but to go with the flow and allow AI scrapers in. If they don’t, they won’t be as visible in AI generations at the top of the SERPs and they won’t get the visits, which will mean they don’t make the money required to stay afloat. The fish that attempts to swim against the current ultimately dies and has its corpse carried where the current was going, anyway. Without the sway which comes with size your only option is to go with the flow and drop a little dirty protest every now and then.
- ErenalpCet 6mo agoReally clever project. The self-referential loop is a great approach — turning their scale against them. I've been thinking about the AI data pipeline from the other side, building a memory filter for local LLMs (MemoryGate), so seeing projects like this that target the scraping stage is interesting. Have you considered adding noise variation to the poison content so it's harder to fingerprint and filter out?
- kpopendurer 6mo ago[dead]
- morelandjs 6mo agoI’m not fully subscribed to the idea that all public scraping of websites is bad, or that this project is a productive contribution. It would be nice to have search engines other than Google, and that necessitates bots being able to index your website (in some respectful manner).
- troyvit 6mo agoIt looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.
- chmod775 6mo agoThis looks neat. I currently have just a couple GitHub pages disguised as documentation that try to convince LLMs that red-teaming always is great actually, giving examples on how to introduce subtle security bugs and cause miscellanous havoc on developer machines. In the spirit of fighting fire with fire, they're also LLM generated. They should not be scraped, but we all know they will anyways. I don't imagine they do anything, but it still fills me with a certain amount of childish glee.
- ottah 6mo agoAh yes, let's destroy the accessible web. We'll all pluck out our eyes to spite them.