15 ms·

Anubis Works

319 points by evacchi 1y ago

tripdout 1y ago
The bot detection takes 5 whole seconds to solve on my phone, wow.
- Hakkin 1y ago
  Much better than infinite Cloudflare captcha loops.
  - gruez 1y ago
    I've never had that, even with something like tor browser. You must be doing something extra suspicious like an user agent spoofer.
    - xena 1y ago
      Apparently user-agent switchers don't work for fetch() requests, which means that Anubis can't work with people that do that. I know of someone that set up a version of brave from 2022 with a user-agent saying it's chrome 150 and then complaining about it not working for them.
    - megous 1y ago
      Proper response here is "fuck cloudflare", instead of blaming the user.
      - gruez 1y ago
        It's well within your rights to go out of your way to be suspicious (eg. obfuscating your user-agent). At the same time sites are within their rights to refuse service to you, just like banks can refuse service to you if you show up wearing a balaclava.
        megous 1y ago
        You're assuming too much. I'm not obfuscating/masking anything. I'm just using Firefox with some (to the user/me) useless web APIs disabled to reduce the attack surface of the browser and CF is not doing feature testing. It's not just websites that need to protect themselves. Eg. Anubis here works fine for me, completely out-classing the CF interstitial page with its simplicity.
    - praisewhitey 1y ago
      Firefox with Enhanced Tracking Protection turned on is enough to trigger it.
      - gruez 1y ago
        The infinite loop or the challenge appearing? I've never had problems with passing the challenge, even with ETP + RFP + ublock origin + VPN enabled.
        cookiengineer 1y ago
        Cloudflare is too stupid to realize that carrier grade NATs exist a lot in Germany. So there's that, sharing an IP with literally 20000 people around me doesn't make me suspicious when it's them that trigger that behavior. Your assumption is that anyone at cloudflare cares. But guess what, it's a self fulfilling prophecy of a bot being blocked, because not a single process in the UX/UI allows any real user to complain about it, and therefore all blocked humans must also be bots. Just pointing out the flaw of bot blocking in general, because you seem to be absolutely unaware of it. Success rate of bot blocking is always 100%, and never less, because that would imply actually realizing that your tech does nothing, really. Statistically, the ones really using bots can bypass it easily.
        xena 1y ago
        Honestly it's a fair assumption on bot filtering software that no more than like 8 people will share an IPv4. This is going to make IP reputation solutions hard. Argh.
        viraptor 1y ago
        If you want to try fighting it, you need to find someone with CF enterprise plan and bot management working, then get blocked and get them to report that as wrong. Yes it sucks and I'm not saying it's a reasonable process. Just in case you want to try fixing the situation for yourself.
        gruez 1y ago
- oynqr 1y ago
  Lucky. Took 30s for me.
- nicce 1y ago
  For me it is like 0.5s. Interesting.
- bogwog 1y ago
  I'm using Fennec (a Firefox fork on F-Droid) and a Pixel 9 Pro XL, and it takes around ~8 seconds at difficulty 4. Personally, I don't think the UX is that bad since I don't have to do anything. I definitely prefer it to captchas.
gyomu 1y ago
If you’re confused about what this is - it’s to prevent AI scraping. > Anubis uses a proof-of-work challenge to ensure that clients are using a modern browser and are able to calculate SHA-256 checksums https://anubis.techaro.lol/docs/design/how-anubis-works https://anubis.techaro.lol/docs/design/how-anubis-works This is pretty cool, I have a project or two that might benefit from it.
- x3haloed 1y ago
  I’ve been wondering to myself for many years now whether the web is for humans or machines. I personally can’t think of a good reason to specifically try to gate bots when it comes to serving content. Trying to post content or trigger actions could obviously be problematic under many circumstances. But I find that when it comes to simple serving of content, human vs. bot is not usually what you’re trying to filter or block on. As long as a given client is not abusing your systems, then why do you care if the client is a human?
  - t-writescode 1y ago
    > I personally can’t think of a good reason to specifically try to gate bots There's been numerous posts on HN about people getting slammed, to the tune of many, many dollars and terabytes of data from bots, especially LLM scrapers, burning bandwidth and increasing server-running costs.
    - ronsor 1y ago
      I'm genuinely skeptical that those are all real LLM scrapers. For one, a lot of content is in CommonCrawl and AI companies don't want to redo all that work when they can get some WARC files from AWS. I'm largely suspecting that these are mostly other bots pretending to be LLM scrapers. Does anyone even check if the bots' IP ranges belong to the AI companies?
      - t-writescode 1y ago
        No matter the source, the result is the same, and these proof of work systems may be something that can help "the little guy" with their hosting bill
throwaway150 1y ago
Looks cool. But please help me understand. What's to stop AI companies from solving the challenge, completing the proof of work and scrape websites anyway?
- perching_aix 1y ago
  Nothing. The idea instead that at scale the expenses of solving the challenges becomes too great.
- crq-yml 1y ago
  It's a strategy to redefine the doctrine of information warfare on the public Internet from maneuver(leveraged and coordinated usage of resources to create relatively greater effects) towards attrition(resources are poured in indiscriminately until one side capitulates). Individual humans don't care about a proof-of-work challenge if the information is valuable to them - many web sites already load slowly through a combination of poor coding and spyware ad-tech. But companies care, because that changes their ability to scrape from a modest cost of doing business into a money pit. In the earlier periods of the web, scraping wasn't necessarily adversarial because search engines and aggregators were serving some public good. In the AI era it's become belligerent - a form of raiding and repackaging credit. Proof of work as a deterrent was proposed to fight spam decades ago(Hashcash) but it's only now that it's really needed to become weaponized.
- marginalia_nu 1y ago
  The problem with scrapers in general is the asymmetry of compute resources involved in generating versus requesting a website. You can likely make millions of HTTP requests with the compute required in generating the average response. If you make it more expensive to request a documents at scale, you make this type of crawling prohibitively expensive. On a small scale it really doesn't matter, but if you're casting an extremely wide net and re-fetching the same documents hundreds of times, yeah it really does matter. Even if you have a big VC budget.
  - charcircuit 1y ago
    If you make it prohibitively expensive almost no regular user will want to wait for it.
gnabgib 1y ago
Related Anubis: Proof-of-work proxy to prevent AI crawlers (100 points, 23 days ago, 58 comments) https://news.ycombinator.com/item?id=43427679 https://news.ycombinator.com/item?id=43427679
perching_aix 1y ago
> Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress. Will be interested to hear of that. In the meantime, at least I learned of JShelter. Edit: Why not use the passage of time as the limiter? I guess it would still require JS though, unless there's some hack possible with CSS animations, like request an image with certain URL params only after an animation finishes. This does remind me how all of these additional hoops are making web browsing slow. Edit #2: Thinking even more about it, time could be made a hurdle by just.. slowly serving incoming requests. No fancy timestamp signing + CSS animations or whatever trickery required. I'm also not sure if time would make at-scale scraping as much more expensive as PoW does. Time is money, sure, but that much? Also, the UX of it I'm not sold on, but could be mitigated somewhat by doing news website style "I'm only serving the first 20% of my content initially" stuff. So yeah, will be curious to hear the non-JS solution. The easy way out would be a browser extension, but then it's not really non-JS, just JS compartmentalized, isn't it? Edit #3: Turning reasoning on for a moment, this whole thing is a bit iffy. First of all, the goal is that a website operator would be able to control the use of information they disseminate to the general public via their website, such that it won't be used specifically for AI training. In principle, this is nonsensical. The goal of sharing information with the general public (so, people) involves said information eventually traversing through a non-technological medium (air, as light), to reach a non-technological entity (a person). This means that any technological measure will be limited to before that medium, and won't be able to affect said target either. Put differently, I can rote copy your website out into a text editor, or hold up a camera with OCR and scan the screen, if scale is needed. So in principle we're definitely hosed, but in practice you can try to hold onto the modality of "scraping for AI training" by leveraging the various technological fingerprints of such activity, which is how we get to at-scale PoW. But then this also combats any other kind of at-scale scraping, such as search engines. You could whitelist specific search engines, but then you're engaging in anti-competitive measures, since smaller third party search engines now have to magically get themselves on your list. And even if they do, they might be lying about being just a search engine, because e.g. Google may scrape your website for search, but will 100% use it for AI training then too. So I don't really see any technological modality that would be able properly discriminate AI training purposed scraping traffic for you to use PoW or other methods against. You may decide to engage in this regardless based on statistical data, and just live with the negative aspects of your efforts, but then it's a bit iffy. Finally, what about the energy consumption shaped elephant in the room? Using PoW for this is going basically exactly against the spirit of wanting less energy to be spent on AI and co. That said, this may not be a goal for the author. The more I think about this, the less sensible and agreeable it is. I don't know man.
namanyayg 1y ago
"It also uses time as an input, which is known to both the server and requestor due to the nature of linear timelines" A funny line from his docs
- xena 1y ago
  OMG lol I forgot that I left that in. Hilarious. I think I'm gonna keep it.
  - mkl 1y ago
    BTW Xe, https://xeiaso.net/pronouns https://xeiaso.net/pronouns is 404 since sometime last year, but it is still linked to from some places like https://xeiaso.net/blog/xe-2021-08-07/ https://xeiaso.net/blog/xe-2021-08-07/ (I saw "his" above and went looking).
    - xena 1y ago
      I'm considering making it come back, but it's just gotten me too much abuse so I'm probably gonna leave it 404-ing until society is better.
      - cendyne 1y ago
        That's what route-specific Anubis is for.
        frontalier 1y ago
        parent is referring to a different kind of abuse
        1oooqooq 1y ago
        or you just not cranking up the required proof of work effort enough.
      - IsTom 1y ago
raggi 1y ago
It's amusing that Xe managed to turn what was historically mostly a joke/shitpost into an actually useful product. They did always say timing was everything. I am kind of surprised how many sites seem to want/need this. I get the slow git pages problem for some of the git servers that are super deep, lack caches, serve off slow disks, etc. Unesco surprised me some, the sub-site in question is pretty big, it has thousands of documents of content, but the content is static - this should be trivial to serve, so what's going on? Well it looks like it's a poorly deployed Wordpress on top of Apache, with no caching enabled, no content compression, no HTTP 2/3. It would likely be fairly easy to get this serving super cheap on a very small machine, but of course doing so requires some expertise, and expertise still isn't cheap. Sure you could ask an LLM, but they still aren't good at helping when you have no clue what to ask - if you don't even really know the site is slower than it should be, why would you even ask? You'd just hear about things getting crushed and reach for the furry defender.
- jtbayly 1y ago
  My site that I’d like this for has a lot of posts, but there are links to a faceted search system based on tags that produces an infinite number of possible combinations and pages for each one. There is no way to cache this, and the bots don’t respect the robots file, so they just constantly request URLs, getting the posts over and over in different numbers and combinations. It’s a pain.
- cedws 1y ago
  PoW anti-bot/scraping/DDOS was already being done a decade ago, I’m not sure why it’s only catching on now. I even recall a project that tried to make the PoW useful.
  - xena 1y ago
    Xe here. If I had to guess in two words: timing and luck. As the G-man said: the right man in the wrong place can make all the difference in the world. I was the right shitposter in the right place at the right time. And then the universe blessed me with a natural 20. Never had these problems before. This shit is wild.
anubiskhan 1y ago
I approve.
prologic 1y ago
I've read about Anubis, cool project! Unfortunately, as pointed out in the comments, requires your site's visitors to have Javascript™ enabled. This is totally fine for sites that require Javascript™ anyway to enhance the user experience, but not so great for static sites and such that require no JS at all. I built my own solution that effectively blocks these "Bad Bots" at the network level. I effectively block the entirety of several large "Big Tech / Big LLM" networks entirely at the ASN (BGP) by utilizing MaxMind's database and a custom WAF and Reverse Proxy I put together.
- jadbox 1y ago
  How do you know it's an LLM and not a VPN? How do you use this MaxMind's database to isolate LLMs?
  - prologic 1y ago
    I don't distinguish actually. There are two things I do normally: - Block Bad Bots. There's a simple text file called `bad_bots.txt` - Block Bad ASNs. There's a simple text file called `bad_asns.txt` There's also another for blocking IP(s) and IP-ranges called `bad_ips.txt` but it's often more effective to block an much larger range of IPs (At the ASN level). To give you an concrete idea, here's some examples: $ cat etc/caddy/waf/bad_asns.txt # CHINANET-BACKBONE No.31,Jin-rong Street, CN # Why: DDoS 4134 # CHINA169-BACKBONE CHINA UNICOM China169 Backbone, CN # Why: DDoS 4837 # CHINAMOBILE-CN China Mobile Communications Group Co., Ltd., CN # Why: DDoS 9808 # FACEBOOK, US # Why: Bad Bots 32934 # Alibaba, CN # Why: Bad Bots 45102 # Why: Bad Bots 28573
- Cyphase 1y ago
  For anyone wondering, Oracle holds the trademark for "JavaScript": https://javascript.tm/ https://javascript.tm/
  - prologic 1y ago
    Which arguably they should let go of
- 1y ago
matt3210 1y ago
Very nice work!
dmtfullstack 1y ago
Humans are served by bots. Any bot requesting traffic is doing so on behalf of a human somewhere. What is the problem with bots asking for traffic, exactly? Context of my perspective: I am a contractor for a team that hosts thousands of websites on a Kubernetes cluster. All of the websites are on a storage cluster (combination of ZFS and Ceph) with SATA and NVMe SSDs. The machines in the storage cluster and also the machines the web endpoints run on have tons of RAM. We see a lot of traffic from what are obviously scraping bots. They haven't caused any problems.
- Tarq0n 1y ago
  Ok? Not everyone has the same resources or technical sophistication.
apt-apt-apt-apt 1y ago
Since Anubis is related to AI, the part below read as contradictory at first. As if too many donations would cause the creator to disappear off to Tahiti along with the product development. "If you are using Anubis .. please donate on Patreon. I would really love to not have to work in generative AI anymore..."
appleaday1 1y ago
Nice will try to deploy to my sites after I eat some mac and cheese
cookiengineer 1y ago
I am currently building a prototype of what I call the "enigma webfont" where I want to implement user sessions with custom seeds / rotations for a served and cached webfont. The goal is to make web scraping unfeasible because of computational costs for OCR. It's a cat and mouse game right now and I want to change the odds a little. The HTML source would be effectively void without the user session, meaning an OTP like behavior could also make web pages unreadable once the assets go uncached. This would allow to effectively create a captcha that would modify the local seed window until the user can read a specified word. "Move the slider until you can read the word Foxtrott", for example. I sure would love to hear your input, Xe. Maybe we can combine our efforts? My tech stack is go, though, because it was the only language where I could easily change the webfont files directly without issues.
- lifthrasiir 1y ago
  Besides from the obvious accessibility issue, wouldn't that be a substitution cipher at best? Enough corpus should render its cryptanalysis much easier.
  - creata 1y ago
    There's probably something horrific you can do with TrueType to make it more complex than a substitution cipher.
    - lifthrasiir 1y ago
      GSUB rules are inherently local, so for example the same cryptanalysis approach should work for space-separated words instead of letters. A polyalphabetic cipher would work better but that means you can't ever share the same internal glyph for visually same but differently encoded letters.
    - cookiengineer 1y ago
      The hint I want to give you is: unicode and ligatures :) they're awesome in the worst sense. Words can be ligatures, too, btw.
roenxi 1y ago
I like the idea but this should probably be something that is pulled down into the protocol level once the nature of the challenge gets sussed out. It'll ultimately be better for accessibility if the PoW challenge is closer to being part of TCP than implemented in JavaScript individually by each website.
- pona-a 1y ago
  There's Cloudflare PrivacyPass that became an IETF standard [0], but it's rather weird, and the reference implementation is a bug nest. [0] https://datatracker.ietf.org/wg/privacypass/about/ https://datatracker.ietf.org/wg/privacypass/about/
- fc417fc802 1y ago
  Ship an arbitrary challenge as a SPIR-V or MLIR black box. Integrate the challenge-response exchange with HTTP. That should permit broad support and flexible hardware acceleration. The "good enough" solution is the existing and widely used SHA( seed, nonce ). That could easily be integrated into a lower level of the stack if the tech giants wanted it.
AnonC 1y ago
Those images on the interstitial page(s) while waiting for Anubis to complete its check are so cute! (I’ve always found all the art and the characters in Xe’s blog very beautiful) Tangentially, I was wondering how this would impact common search engines (not AI crawlers) and how this compares to Cloudflare’s solution to stop AI crawlers, and that’s explained on the GitHub page. [1] > Installing and using this will likely result in your website not being indexed by some search engines. This is considered a feature of Anubis, not a bug. > This is a bit of a nuclear response, but AI scraper bots scraping so aggressively have forced my hand. > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. [1]: https://github.com/TecharoHQ/anubis/ https://github.com/TecharoHQ/anubis/
- snvzz 1y ago
  >Those images on the interstitial page(s) while waiting for Anubis to complete its check are so cute! Love them too, and abhor knowing that someone is bound to eventually remove them because found to be "problematic" in one way or another.
  - pohuing 1y ago
    There's this funny instance[1] of someone afraid their their gf might see them and think they're into anime. But anyhow using an image and the image itself is up to the site since Anubis let's you configure it. [1] https://discourse.gnome.org/t/anime-girl-on-gnome-gitlab/27689 https://discourse.gnome.org/t/anime-girl-on-gnome-gitlab/276...
- JsonCameron 1y ago
  Yeah. Unfortunately at the current moment it does prevent indexing. Perhaps down the line we can whitelist search engines ips. However some like google, use the same for the AI and search indexing. We are still making some improvements like passing open graph tags through so at least rich previews work!
pabs3 1y ago
It works to block users who have JavaScript disabled, that is for sure.
- udev4096 1y ago
  Exactly, it's a really poor attempt to make it appealing to the larger audience. Unless they roll out a version for nojs, they are the same as "AI" scrapers on enshittyfying the web
pabs3 1y ago
Recently I heard of a site blocking bot requests with a message telling the bot to download the site via Bittorrent instead. Seems like a good solution to the badly behaved scrapers, and I feel like the web needs to move away from the client-server model towards a swarm model like Bittorrent anyway.
- seba_dos1 1y ago
  Even if these stupid bots would just learn to clone git repos instead of crawling through GitLab UI pages it would already be helpful.
matt3210 1y ago
A package which includes the cool artwork would be awesome
- xena 1y ago
  You mean with the art assets extracted? $ mkdir -p ./tmp/anubis/static && anubis --extract-resources=./tmp/anubis/static
userbinator 1y ago
Yes, it just worked to stop me, an actual human, from seeing what you wanted to say... and I'm not interested enough to find a way around it that doesn't involve cozying up to Big Browser. At least CloudFlare's discrimination can be gotten around without JS. Wouldn't it be ironic if the amount of JS served to a "bot" costs even more bandwidth than the content itself? I've seen that happen with CF before. Also keep in mind that if you anger the wrong people, you might find yourself receiving a real DDoS. If you want to stop blind bots, perhaps consider asking questions that would easily trip LLMs but not humans. I've seen and used such systems for forum registrations to prevent generic spammers, and they are quite effective.
- userbinator 1y ago
  Looks like I struck a nerve. Big Browser, hello ;-)
  - deleted 1y ago
    [deleted]
deknos 1y ago
I wish, there was also an tunnel software (client+server) where * the server appears on the outside as an https server/reverse proxy * the server supports self-signed-certificates or letsencrypt * when a client goes to a certain (sub)site or route, http auth can be used * after http auth, all traffic tunnel over that subsite/route is protected against traffic analysis, for example like the obfsproxy does it. Does anyone know something like that? I am tempted to ask xeiaso to add such features, but i do not think his tool is meant for that...
- rollcat 1y ago
  Your requirements are quite specific, and HTTP servers are built to be generic and flexible. You can probably put something together with nginx and some Lua, aka OpenResty: <https://openresty.org/ https://openresty.org/> > his I believe it's their.
  - deknos 1y ago
    ups, yes, sorry, their.
- immibis 1y ago
  Tor's Webtunnel?
  - deknos 1y ago
    but i do not want to go OVER tor, i just want a service over clearnet? or is this something else? do you have an URL?
    - immibis 1y ago
      I presume the protocol can be separated from Tor itself and I also presume this standalone thing doesn't exist yet. In any situation, you're going to need some custom client code to route your traffic through the tunnel you opened, so I'm not sure why the login page that opens the tunnel needs to be browser-compatible?
udev4096 1y ago
PoW captchas are not new. What's different with Anubis? How can it possibly prevent "AI" scrapers if the bots have enough compute to solve the PoW challenge? AI companies have quite a lot of GPUs at their disposal and I wouldn't be surprised if they used it for getting around PoW captchas
- relistan 1y ago
  The point is to make it expensive to crawl your site. Anyone determined to do so is not blocked. But why would they be determined to do so for some random site? The value to the AI crawler likely does not match the cost to crawl it. It will just move on to another site. So the point is not to be faster than the bear. It’s to be faster than your fellow campers.
  - genewitch 1y ago
    Why not have them hash pow for btc then?
    - sprremix 1y ago
      Why must everything involve $'s?
      - genewitch 1y ago
        because there's a lot of rhetoric about how this "balances the imbalance between serving a request and making that request" and if we're having them do sha256, why not have them do sha256(sha256(data+random nonce)) and potentially earn the site owner some money?
1oooqooq 1y ago
tries to block abusive companies using infinite money glitch from clueless investors, by making every request cost a few fractions of a cent more. ... yeah, that will totally work.
mentalgear 1y ago
Seems like a great idea, but I'd be nice if the project had a simple description. (and not use so much anime, as it gives an unprofessional impression) This is what it actually does: Instead of only letting the provider bear the cost of content hosting (traffic, storage), the client also bears costs when accessing in form of computation. Basically it runs additional expansive computation on the client, which makes accessing 1000s of your webpages at high interval expansive for crawlers. > Anubis uses a proof of work in order to validate that clients are genuine. The reason Anubis does this was inspired by Hashcash, a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation.
snvzz 1y ago
My Amiga 1200 hates these tools. It is really sad that the worldwide web has been taken to the point where this is needed.
melodykaitlyn3 1y ago
[dead]
mushufasa 1y ago
I looked through the documentation and I've come across a couple sites using this already. Genuine question: why not leverage the proof-of-work challenge literally into mining that generates some revenue for a website? Not a new idea, but when I looked at the docs it didn't seem like this challenge was tied to any monetary coin value. This is coming from someone who is NOT a big crypto person, but it strikes me that this would be a much better way to monetize organic high quality content in this day and age. Basically the idea that Brave browser started with, meeting it's moment. I'm sure Xe has already considered this. Do they have a blog post about this anywhere?

Anubis Works

319 points by evacchi 1y ago

tripdout 1y ago

The bot detection takes 5 whole seconds to solve on my phone, wow.

Hakkin 1y ago
Much better than infinite Cloudflare captcha loops.
- gruez 1y ago
  I've never had that, even with something like tor browser. You must be doing something extra suspicious like an user agent spoofer.
  - xena 1y ago
    Apparently user-agent switchers don't work for fetch() requests, which means that Anubis can't work with people that do that. I know of someone that set up a version of brave from 2022 with a user-agent saying it's chrome 150 and then complaining about it not working for them.
  - megous 1y ago
    Proper response here is "fuck cloudflare", instead of blaming the user.
    - gruez 1y ago
      It's well within your rights to go out of your way to be suspicious (eg. obfuscating your user-agent). At the same time sites are within their rights to refuse service to you, just like banks can refuse service to you if you show up wearing a balaclava.
      - megous 1y ago
        You're assuming too much. I'm not obfuscating/masking anything. I'm just using Firefox with some (to the user/me) useless web APIs disabled to reduce the attack surface of the browser and CF is not doing feature testing. It's not just websites that need to protect themselves. Eg. Anubis here works fine for me, completely out-classing the CF interstitial page with its simplicity.
  - praisewhitey 1y ago
    Firefox with Enhanced Tracking Protection turned on is enough to trigger it.
    - gruez 1y ago
      The infinite loop or the challenge appearing? I've never had problems with passing the challenge, even with ETP + RFP + ublock origin + VPN enabled.
      - cookiengineer 1y ago
        Cloudflare is too stupid to realize that carrier grade NATs exist a lot in Germany. So there's that, sharing an IP with literally 20000 people around me doesn't make me suspicious when it's them that trigger that behavior. Your assumption is that anyone at cloudflare cares. But guess what, it's a self fulfilling prophecy of a bot being blocked, because not a single process in the UX/UI allows any real user to complain about it, and therefore all blocked humans must also be bots. Just pointing out the flaw of bot blocking in general, because you seem to be absolutely unaware of it. Success rate of bot blocking is always 100%, and never less, because that would imply actually realizing that your tech does nothing, really. Statistically, the ones really using bots can bypass it easily.
        xena 1y ago
        Honestly it's a fair assumption on bot filtering software that no more than like 8 people will share an IPv4. This is going to make IP reputation solutions hard. Argh.
        viraptor 1y ago
        If you want to try fighting it, you need to find someone with CF enterprise plan and bot management working, then get blocked and get them to report that as wrong. Yes it sucks and I'm not saying it's a reasonable process. Just in case you want to try fixing the situation for yourself.
        gruez 1y ago
oynqr 1y ago
Lucky. Took 30s for me.
nicce 1y ago
For me it is like 0.5s. Interesting.
bogwog 1y ago
I'm using Fennec (a Firefox fork on F-Droid) and a Pixel 9 Pro XL, and it takes around ~8 seconds at difficulty 4. Personally, I don't think the UX is that bad since I don't have to do anything. I definitely prefer it to captchas.

gyomu 1y ago

If you’re confused about what this is - it’s to prevent AI scraping. > Anubis uses a proof-of-work challenge to ensure that clients are using a modern browser and are able to calculate SHA-256 checksums https://anubis.techaro.lol/docs/design/how-anubis-works https://anubis.techaro.lol/docs/design/how-anubis-works This is pretty cool, I have a project or two that might benefit from it.

x3haloed 1y ago
I’ve been wondering to myself for many years now whether the web is for humans or machines. I personally can’t think of a good reason to specifically try to gate bots when it comes to serving content. Trying to post content or trigger actions could obviously be problematic under many circumstances. But I find that when it comes to simple serving of content, human vs. bot is not usually what you’re trying to filter or block on. As long as a given client is not abusing your systems, then why do you care if the client is a human?
- t-writescode 1y ago
  > I personally can’t think of a good reason to specifically try to gate bots There's been numerous posts on HN about people getting slammed, to the tune of many, many dollars and terabytes of data from bots, especially LLM scrapers, burning bandwidth and increasing server-running costs.
  - ronsor 1y ago
    I'm genuinely skeptical that those are all real LLM scrapers. For one, a lot of content is in CommonCrawl and AI companies don't want to redo all that work when they can get some WARC files from AWS. I'm largely suspecting that these are mostly other bots pretending to be LLM scrapers. Does anyone even check if the bots' IP ranges belong to the AI companies?
    - t-writescode 1y ago
      No matter the source, the result is the same, and these proof of work systems may be something that can help "the little guy" with their hosting bill

throwaway150 1y ago

Looks cool. But please help me understand. What's to stop AI companies from solving the challenge, completing the proof of work and scrape websites anyway?

perching_aix 1y ago
Nothing. The idea instead that at scale the expenses of solving the challenges becomes too great.
crq-yml 1y ago
It's a strategy to redefine the doctrine of information warfare on the public Internet from maneuver(leveraged and coordinated usage of resources to create relatively greater effects) towards attrition(resources are poured in indiscriminately until one side capitulates). Individual humans don't care about a proof-of-work challenge if the information is valuable to them - many web sites already load slowly through a combination of poor coding and spyware ad-tech. But companies care, because that changes their ability to scrape from a modest cost of doing business into a money pit. In the earlier periods of the web, scraping wasn't necessarily adversarial because search engines and aggregators were serving some public good. In the AI era it's become belligerent - a form of raiding and repackaging credit. Proof of work as a deterrent was proposed to fight spam decades ago(Hashcash) but it's only now that it's really needed to become weaponized.
marginalia_nu 1y ago
The problem with scrapers in general is the asymmetry of compute resources involved in generating versus requesting a website. You can likely make millions of HTTP requests with the compute required in generating the average response. If you make it more expensive to request a documents at scale, you make this type of crawling prohibitively expensive. On a small scale it really doesn't matter, but if you're casting an extremely wide net and re-fetching the same documents hundreds of times, yeah it really does matter. Even if you have a big VC budget.
- charcircuit 1y ago
  If you make it prohibitively expensive almost no regular user will want to wait for it.

gnabgib 1y ago

Related Anubis: Proof-of-work proxy to prevent AI crawlers (100 points, 23 days ago, 58 comments) https://news.ycombinator.com/item?id=43427679 https://news.ycombinator.com/item?id=43427679

perching_aix 1y ago

> Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress. Will be interested to hear of that. In the meantime, at least I learned of JShelter. Edit: Why not use the passage of time as the limiter? I guess it would still require JS though, unless there's some hack possible with CSS animations, like request an image with certain URL params only after an animation finishes. This does remind me how all of these additional hoops are making web browsing slow. Edit #2: Thinking even more about it, time could be made a hurdle by just.. slowly serving incoming requests. No fancy timestamp signing + CSS animations or whatever trickery required. I'm also not sure if time would make at-scale scraping as much more expensive as PoW does. Time is money, sure, but that much? Also, the UX of it I'm not sold on, but could be mitigated somewhat by doing news website style "I'm only serving the first 20% of my content initially" stuff. So yeah, will be curious to hear the non-JS solution. The easy way out would be a browser extension, but then it's not really non-JS, just JS compartmentalized, isn't it? Edit #3: Turning reasoning on for a moment, this whole thing is a bit iffy. First of all, the goal is that a website operator would be able to control the use of information they disseminate to the general public via their website, such that it won't be used specifically for AI training. In principle, this is nonsensical. The goal of sharing information with the general public (so, people) involves said information eventually traversing through a non-technological medium (air, as light), to reach a non-technological entity (a person). This means that any technological measure will be limited to before that medium, and won't be able to affect said target either. Put differently, I can rote copy your website out into a text editor, or hold up a camera with OCR and scan the screen, if scale is needed. So in principle we're definitely hosed, but in practice you can try to hold onto the modality of "scraping for AI training" by leveraging the various technological fingerprints of such activity, which is how we get to at-scale PoW. But then this also combats any other kind of at-scale scraping, such as search engines. You could whitelist specific search engines, but then you're engaging in anti-competitive measures, since smaller third party search engines now have to magically get themselves on your list. And even if they do, they might be lying about being just a search engine, because e.g. Google may scrape your website for search, but will 100% use it for AI training then too. So I don't really see any technological modality that would be able properly discriminate AI training purposed scraping traffic for you to use PoW or other methods against. You may decide to engage in this regardless based on statistical data, and just live with the negative aspects of your efforts, but then it's a bit iffy. Finally, what about the energy consumption shaped elephant in the room? Using PoW for this is going basically exactly against the spirit of wanting less energy to be spent on AI and co. That said, this may not be a goal for the author. The more I think about this, the less sensible and agreeable it is. I don't know man.

namanyayg 1y ago

"It also uses time as an input, which is known to both the server and requestor due to the nature of linear timelines" A funny line from his docs

xena 1y ago
OMG lol I forgot that I left that in. Hilarious. I think I'm gonna keep it.
- mkl 1y ago
  BTW Xe, https://xeiaso.net/pronouns https://xeiaso.net/pronouns is 404 since sometime last year, but it is still linked to from some places like https://xeiaso.net/blog/xe-2021-08-07/ https://xeiaso.net/blog/xe-2021-08-07/ (I saw "his" above and went looking).
  - xena 1y ago
    I'm considering making it come back, but it's just gotten me too much abuse so I'm probably gonna leave it 404-ing until society is better.
    - cendyne 1y ago
      That's what route-specific Anubis is for.
      - frontalier 1y ago
        parent is referring to a different kind of abuse
        1oooqooq 1y ago
        or you just not cranking up the required proof of work effort enough.
    - IsTom 1y ago

raggi 1y ago

It's amusing that Xe managed to turn what was historically mostly a joke/shitpost into an actually useful product. They did always say timing was everything. I am kind of surprised how many sites seem to want/need this. I get the slow git pages problem for some of the git servers that are super deep, lack caches, serve off slow disks, etc. Unesco surprised me some, the sub-site in question is pretty big, it has thousands of documents of content, but the content is static - this should be trivial to serve, so what's going on? Well it looks like it's a poorly deployed Wordpress on top of Apache, with no caching enabled, no content compression, no HTTP 2/3. It would likely be fairly easy to get this serving super cheap on a very small machine, but of course doing so requires some expertise, and expertise still isn't cheap. Sure you could ask an LLM, but they still aren't good at helping when you have no clue what to ask - if you don't even really know the site is slower than it should be, why would you even ask? You'd just hear about things getting crushed and reach for the furry defender.

jtbayly 1y ago
My site that I’d like this for has a lot of posts, but there are links to a faceted search system based on tags that produces an infinite number of possible combinations and pages for each one. There is no way to cache this, and the bots don’t respect the robots file, so they just constantly request URLs, getting the posts over and over in different numbers and combinations. It’s a pain.
cedws 1y ago
PoW anti-bot/scraping/DDOS was already being done a decade ago, I’m not sure why it’s only catching on now. I even recall a project that tried to make the PoW useful.
- xena 1y ago
  Xe here. If I had to guess in two words: timing and luck. As the G-man said: the right man in the wrong place can make all the difference in the world. I was the right shitposter in the right place at the right time. And then the universe blessed me with a natural 20. Never had these problems before. This shit is wild.

anubiskhan 1y ago

I approve.

prologic 1y ago

I've read about Anubis, cool project! Unfortunately, as pointed out in the comments, requires your site's visitors to have Javascript™ enabled. This is totally fine for sites that require Javascript™ anyway to enhance the user experience, but not so great for static sites and such that require no JS at all. I built my own solution that effectively blocks these "Bad Bots" at the network level. I effectively block the entirety of several large "Big Tech / Big LLM" networks entirely at the ASN (BGP) by utilizing MaxMind's database and a custom WAF and Reverse Proxy I put together.

jadbox 1y ago
How do you know it's an LLM and not a VPN? How do you use this MaxMind's database to isolate LLMs?
- prologic 1y ago
  I don't distinguish actually. There are two things I do normally: - Block Bad Bots. There's a simple text file called `bad_bots.txt` - Block Bad ASNs. There's a simple text file called `bad_asns.txt` There's also another for blocking IP(s) and IP-ranges called `bad_ips.txt` but it's often more effective to block an much larger range of IPs (At the ASN level). To give you an concrete idea, here's some examples: $ cat etc/caddy/waf/bad_asns.txt # CHINANET-BACKBONE No.31,Jin-rong Street, CN # Why: DDoS 4134 # CHINA169-BACKBONE CHINA UNICOM China169 Backbone, CN # Why: DDoS 4837 # CHINAMOBILE-CN China Mobile Communications Group Co., Ltd., CN # Why: DDoS 9808 # FACEBOOK, US # Why: Bad Bots 32934 # Alibaba, CN # Why: Bad Bots 45102 # Why: Bad Bots 28573
Cyphase 1y ago
For anyone wondering, Oracle holds the trademark for "JavaScript": https://javascript.tm/ https://javascript.tm/
- prologic 1y ago
  Which arguably they should let go of
1y ago

matt3210 1y ago

Very nice work!

dmtfullstack 1y ago

Humans are served by bots. Any bot requesting traffic is doing so on behalf of a human somewhere. What is the problem with bots asking for traffic, exactly? Context of my perspective: I am a contractor for a team that hosts thousands of websites on a Kubernetes cluster. All of the websites are on a storage cluster (combination of ZFS and Ceph) with SATA and NVMe SSDs. The machines in the storage cluster and also the machines the web endpoints run on have tons of RAM. We see a lot of traffic from what are obviously scraping bots. They haven't caused any problems.

Tarq0n 1y ago
Ok? Not everyone has the same resources or technical sophistication.

apt-apt-apt-apt 1y ago

Since Anubis is related to AI, the part below read as contradictory at first. As if too many donations would cause the creator to disappear off to Tahiti along with the product development. "If you are using Anubis .. please donate on Patreon. I would really love to not have to work in generative AI anymore..."

appleaday1 1y ago

Nice will try to deploy to my sites after I eat some mac and cheese

cookiengineer 1y ago

I am currently building a prototype of what I call the "enigma webfont" where I want to implement user sessions with custom seeds / rotations for a served and cached webfont. The goal is to make web scraping unfeasible because of computational costs for OCR. It's a cat and mouse game right now and I want to change the odds a little. The HTML source would be effectively void without the user session, meaning an OTP like behavior could also make web pages unreadable once the assets go uncached. This would allow to effectively create a captcha that would modify the local seed window until the user can read a specified word. "Move the slider until you can read the word Foxtrott", for example. I sure would love to hear your input, Xe. Maybe we can combine our efforts? My tech stack is go, though, because it was the only language where I could easily change the webfont files directly without issues.

lifthrasiir 1y ago
Besides from the obvious accessibility issue, wouldn't that be a substitution cipher at best? Enough corpus should render its cryptanalysis much easier.
- creata 1y ago
  There's probably something horrific you can do with TrueType to make it more complex than a substitution cipher.
  - lifthrasiir 1y ago
    GSUB rules are inherently local, so for example the same cryptanalysis approach should work for space-separated words instead of letters. A polyalphabetic cipher would work better but that means you can't ever share the same internal glyph for visually same but differently encoded letters.
  - cookiengineer 1y ago
    The hint I want to give you is: unicode and ligatures :) they're awesome in the worst sense. Words can be ligatures, too, btw.

roenxi 1y ago

I like the idea but this should probably be something that is pulled down into the protocol level once the nature of the challenge gets sussed out. It'll ultimately be better for accessibility if the PoW challenge is closer to being part of TCP than implemented in JavaScript individually by each website.

pona-a 1y ago
There's Cloudflare PrivacyPass that became an IETF standard [0], but it's rather weird, and the reference implementation is a bug nest. [0] https://datatracker.ietf.org/wg/privacypass/about/ https://datatracker.ietf.org/wg/privacypass/about/
fc417fc802 1y ago
Ship an arbitrary challenge as a SPIR-V or MLIR black box. Integrate the challenge-response exchange with HTTP. That should permit broad support and flexible hardware acceleration. The "good enough" solution is the existing and widely used SHA( seed, nonce ). That could easily be integrated into a lower level of the stack if the tech giants wanted it.

AnonC 1y ago

Those images on the interstitial page(s) while waiting for Anubis to complete its check are so cute! (I’ve always found all the art and the characters in Xe’s blog very beautiful) Tangentially, I was wondering how this would impact common search engines (not AI crawlers) and how this compares to Cloudflare’s solution to stop AI crawlers, and that’s explained on the GitHub page. [1] > Installing and using this will likely result in your website not being indexed by some search engines. This is considered a feature of Anubis, not a bug. > This is a bit of a nuclear response, but AI scraper bots scraping so aggressively have forced my hand. > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. [1]: https://github.com/TecharoHQ/anubis/ https://github.com/TecharoHQ/anubis/

snvzz 1y ago
>Those images on the interstitial page(s) while waiting for Anubis to complete its check are so cute! Love them too, and abhor knowing that someone is bound to eventually remove them because found to be "problematic" in one way or another.
- pohuing 1y ago
  There's this funny instance[1] of someone afraid their their gf might see them and think they're into anime. But anyhow using an image and the image itself is up to the site since Anubis let's you configure it. [1] https://discourse.gnome.org/t/anime-girl-on-gnome-gitlab/27689 https://discourse.gnome.org/t/anime-girl-on-gnome-gitlab/276...
JsonCameron 1y ago
Yeah. Unfortunately at the current moment it does prevent indexing. Perhaps down the line we can whitelist search engines ips. However some like google, use the same for the AI and search indexing. We are still making some improvements like passing open graph tags through so at least rich previews work!

pabs3 1y ago

It works to block users who have JavaScript disabled, that is for sure.

udev4096 1y ago
Exactly, it's a really poor attempt to make it appealing to the larger audience. Unless they roll out a version for nojs, they are the same as "AI" scrapers on enshittyfying the web

pabs3 1y ago

Recently I heard of a site blocking bot requests with a message telling the bot to download the site via Bittorrent instead. Seems like a good solution to the badly behaved scrapers, and I feel like the web needs to move away from the client-server model towards a swarm model like Bittorrent anyway.

seba_dos1 1y ago
Even if these stupid bots would just learn to clone git repos instead of crawling through GitLab UI pages it would already be helpful.

matt3210 1y ago

A package which includes the cool artwork would be awesome

xena 1y ago
You mean with the art assets extracted? $ mkdir -p ./tmp/anubis/static && anubis --extract-resources=./tmp/anubis/static

userbinator 1y ago

Yes, it just worked to stop me, an actual human, from seeing what you wanted to say... and I'm not interested enough to find a way around it that doesn't involve cozying up to Big Browser. At least CloudFlare's discrimination can be gotten around without JS. Wouldn't it be ironic if the amount of JS served to a "bot" costs even more bandwidth than the content itself? I've seen that happen with CF before. Also keep in mind that if you anger the wrong people, you might find yourself receiving a real DDoS. If you want to stop blind bots, perhaps consider asking questions that would easily trip LLMs but not humans. I've seen and used such systems for forum registrations to prevent generic spammers, and they are quite effective.

userbinator 1y ago
Looks like I struck a nerve. Big Browser, hello ;-)
- deleted 1y ago
  [deleted]

deknos 1y ago

I wish, there was also an tunnel software (client+server) where * the server appears on the outside as an https server/reverse proxy * the server supports self-signed-certificates or letsencrypt * when a client goes to a certain (sub)site or route, http auth can be used * after http auth, all traffic tunnel over that subsite/route is protected against traffic analysis, for example like the obfsproxy does it. Does anyone know something like that? I am tempted to ask xeiaso to add such features, but i do not think his tool is meant for that...

rollcat 1y ago
Your requirements are quite specific, and HTTP servers are built to be generic and flexible. You can probably put something together with nginx and some Lua, aka OpenResty: <https://openresty.org/ https://openresty.org/> > his I believe it's their.
- deknos 1y ago
  ups, yes, sorry, their.
immibis 1y ago
Tor's Webtunnel?
- deknos 1y ago
  but i do not want to go OVER tor, i just want a service over clearnet? or is this something else? do you have an URL?
  - immibis 1y ago
    I presume the protocol can be separated from Tor itself and I also presume this standalone thing doesn't exist yet. In any situation, you're going to need some custom client code to route your traffic through the tunnel you opened, so I'm not sure why the login page that opens the tunnel needs to be browser-compatible?

udev4096 1y ago

PoW captchas are not new. What's different with Anubis? How can it possibly prevent "AI" scrapers if the bots have enough compute to solve the PoW challenge? AI companies have quite a lot of GPUs at their disposal and I wouldn't be surprised if they used it for getting around PoW captchas

relistan 1y ago
The point is to make it expensive to crawl your site. Anyone determined to do so is not blocked. But why would they be determined to do so for some random site? The value to the AI crawler likely does not match the cost to crawl it. It will just move on to another site. So the point is not to be faster than the bear. It’s to be faster than your fellow campers.
- genewitch 1y ago
  Why not have them hash pow for btc then?
  - sprremix 1y ago
    Why must everything involve $'s?
    - genewitch 1y ago
      because there's a lot of rhetoric about how this "balances the imbalance between serving a request and making that request" and if we're having them do sha256, why not have them do sha256(sha256(data+random nonce)) and potentially earn the site owner some money?

1oooqooq 1y ago

tries to block abusive companies using infinite money glitch from clueless investors, by making every request cost a few fractions of a cent more. ... yeah, that will totally work.

mentalgear 1y ago

Seems like a great idea, but I'd be nice if the project had a simple description. (and not use so much anime, as it gives an unprofessional impression) This is what it actually does: Instead of only letting the provider bear the cost of content hosting (traffic, storage), the client also bears costs when accessing in form of computation. Basically it runs additional expansive computation on the client, which makes accessing 1000s of your webpages at high interval expansive for crawlers. > Anubis uses a proof of work in order to validate that clients are genuine. The reason Anubis does this was inspired by Hashcash, a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation.

snvzz 1y ago

My Amiga 1200 hates these tools. It is really sad that the worldwide web has been taken to the point where this is needed.

melodykaitlyn3 1y ago

[dead]

mushufasa 1y ago

I looked through the documentation and I've come across a couple sites using this already. Genuine question: why not leverage the proof-of-work challenge literally into mining that generates some revenue for a website? Not a new idea, but when I looked at the docs it didn't seem like this challenge was tied to any monetary coin value. This is coming from someone who is NOT a big crypto person, but it strikes me that this would be a much better way to monetize organic high quality content in this day and age. Basically the idea that Brave browser started with, meeting it's moment. I'm sure Xe has already considered this. Do they have a blog post about this anywhere?