11 ms·

This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be

This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be dismissive of Anubis. Sure, the people who make the AI scraper bots are going to figure out how to actually do the work. The point is that they hadn't, and this worked for quite a while. As the botmakers circumvent, new methods of proof-of-notbot will be made available. It's really as simple as that. If a new method comes out and your site is safe for a month or two, great! That's better than dealing with fifty requests a second, wondering if you can block whole netblocks, and if so, which. This is like those simple things on submission forms that ask you what 7 + 2 is. Of course everyone knows that a crawler can calculate that! But it takes a human some time and work to tell the crawler HOW.

psionides 1y ago
The problem is that 7 + 2 on a submission form only affects people who want to submit something, Anubis affects every user who wants to read something on your site
- account42 1y ago
  The question then is why read only users are consuming so much resources that serving them big chunks of JS instead reduces loads of the server. Maybe improve you rendering and/or caching before employing DRM solutions that are doomed to fail anyway.
  - Mateon1 1y ago
    The problem it's originally fixing is bad scrapers accessing dynamic site content that's expensive to produce, like trying to crawl all diffs in a git repo, or all mediawiki oldids. Now it's also used on mostly static content because it is effective vs scrapers that otherwise ignore robots.txt.
odo1242 1y ago
Also, it forces the crawler to gain code execution capabilities, which for many companies will just make them give up and scrape someone else.
- wredcoll 1y ago
  I don't know if you've noticed, but there's a few websites these days that use javascript as part of their display logic.
  - odo1242 1y ago
    Yes, and those sites take way more effort to crawl than other sites. They may still get crawled, but likely less often than the ones that don't use JavaScript for rendering (which is the main purpose of Anubis - saving bandwidth from crawlers who crawl sites way too often). (Also, note the difference between using JavaScript for display logic and requiring JavaScript to load any content at all. Most websites do the first, the second isn't quite as common.)
cakealert 1y ago
This arms race will have a terminus. The bots will eventually be indistinguishable from humans. Some already are.
- neumann 1y ago
  It will be hard to tune them to be just the right level of ignorant and slow as us though!
  - cwmoore 1y ago
    Soon enough there will be competing Unicode characters that can remove exclamation points.
- overfeed 1y ago
  > The bots will eventually be indistinguishable from humans Not until they get issued government IDs they won't! Extrapolating from current trends, some form of online ID attestation (likely based on government-issued ID[1]) will become normal in the next decade, and naturally, this will be included in the anti-bot arsenal. It will be up to the site operator to trust identities signed by the Russian government. 1. Despite what Sam Altman's eyeball company will try to sell you, government registers will always be the anchor of trust for proof-of-identity, they've been doing it for centuries and have become good at it and have earned the goodwill.
  - bhawks 1y ago
    Can't wait to sign into my web browser with my driver's license.
    - overfeed 1y ago
      In all likelihood, most people will do so via the Apple Wallet (or the equivalent on their non-Apple devices). It's going to be painful to use Open source OSes for a while, thanks to CloudFlare and Anubis. This is not the future I want, but we can't have nice things.
tptacek 1y ago
Respectfully, I think it's you missing the point here. None of this is to say you shouldn't use Anubis, but Tavis Ormandy is offering a computer science critique of how it purports to function. You don't have to care about computer science in this instance! But you can't dismiss it because it's computer science. Consider: An adaptive password hash like bcrypt or Argon2 uses a work function to apply asymmetric costs to adversaries (attackers who don't know the real password). Both users and attackers have to apply the work function, but the user gets ~constant value for it (they know the password, so to a first approx. they only have to call it once). Attackers have to iterate the function, potentially indefinitely, in the limit obtaining 0 reward for infinite cost. A blockchain cryptocurrency uses a work function principally as a synchronization mechanism. The work function itself doesn't have a meaningfully separate adversary. Everyone obtains the same value (the expected value of attempting to solve the next round of the block commitment puzzle) for each application of the work function. And note in this scenario most of the value returned from the work function goes to a small, centralized group of highly-capitalized specialists. A proof-of-work-based antiabuse system wants to function the way a password hash functions. You want to define an adversary and then find a way to incur asymmetric costs on them, so that the adversary gets minimal value compared to legitimate users. And this is in fact how proof-of-work-based antispam systems function: the value of sending a single spam message is so low that the EV of applying the work function is negative. But here we're talking about a system where legitimate users (human browsers) and scrapers get the same value for every application of the work function. The cost:value ratio is unchanged; it's just that everything is more expensive for everybody. You're getting the worst of both worlds: user-visible costs and a system that favors large centralized well-capitalized clients. There are antiabuse systems that do incur asymmetric costs on automated users. Youtube had (has?) one. Rather than simply attaching a constant extra cost for every request, it instead delivered a VM (through JS) to browsers, and programs for that VM. The VM and its programs were deliberately hard to reverse, and changed regularly. Part of their purpose was to verify, through a bunch of fussy side channels, that they were actually running on real browsers. Every time Youtube changed the VM, the bots had to do large amounts of new reversing work to keep up, but normal users didn't. This is also how the Blu-Ray BD+ system worked. The term of art for these systems is "content protection", which is what I think Anubis actually wants to be, but really isn't (yet?). The problem with "this is good because none of the scrapers even bother to do this POW yet" is that you don't need an annoying POW to get that value! You could just write a mildly complicated Javascript function, or do an automated captcha.
wat10000 1y ago
Technical people are prone to black-and-white thinking, which makes it hard to understand that making something more difficult will cause people to do it less even though it’s still possible.
- mattnewton 1y ago
  I think the argument on offer is more, this juice isn't worth the squeeze. Each user is being slowed down and annoyed for something that bots will trivially bypass if they become aware of it.
  - wat10000 1y ago
    If they become aware of it and actually think it’s worthwhile. Malicious bots work by scaling, and implementing special cases for every random web site doesn’t scale. And it’s likely they never even notice.
    - mattnewton 1y ago
      If this kind of security by not being noticed is the plan, why not just have a trivial (but unique) captcha that asks the user to click a button with no battery wasting computation?
      - account42 1y ago
        Because you can't sell that as a commercial solution that the open source software ecosystem provides free advertising to.
      - wat10000 1y ago
        That works too, but not quite as well so it decreases the unwanted activity somewhat less.
interstice 1y ago
The cost benefit calculus for workarounds changes based on popularity. Your custom lock might be easy to break by a professional, but the handful of people who might ever care to pick it are unlikely to be trying that hard. A lock which lets you into 5% of houses however might be worth learning to break.
Aurornis 1y ago
> This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be dismissive of Anubis. This is a confusing comment because it appears you don’t understand the well-written critique in the linked blog post. > This is like those simple things on submission forms that ask you what 7 + 2 is. Of course everyone knows that a crawler can calculate that! But it takes a human some time and work to tell the crawler HOW. The key point in the blog post is that it’s the inverse of a CAPTCHA: The proof of work requirement is solved by the computer automatically. You don’t have to teach a computer how to solve this proof of work because it’s designed for the computer to solve the proof of work. It makes the crawling process more expensive because it has to actually run scripts on the page (or hardcode a workaround for specific versions) but from a computational perspective that’s actually easier and far more deterministic than trying to have AI solve visual CAPTCHA challenges.
- necovek 1y ago
  But for actual live users who don't see anything but a transient screen, Anubis is a better experience than all those pesky CAPTCHAs (I am bored of trying to recognize bikes, pedestrian crossings, buses, hydrants). The question is if this is the sweet spot, and I can't find anyone doing the comparative study (how many annoyed human visitors, how many humans stopped and, obviously, how many bots stopped).
  - JimDabell 1y ago
    > Anubis is a better experience than all those pesky CAPTCHAs (I am bored of trying to recognize bikes, pedestrian crossings, buses, hydrants). Most CAPTCHAs are invisible these days, and Anubis is worse than them. Also, CAPTCHAs are not normally deployed just for visiting a site, they are mostly used when you want to submit something.
    - necovek 1y ago
      We are obviously living a different Internet reality, and that's the whole point — we need numbers to really establish baseline truth. FTR, I am mostly browsing from Serbia using Firefox browser on a Linux or MacOS machine.
deleted 1y ago
[deleted]
agwa 1y ago
It sounds like you're saying that it's not the proof-of-work that's stopping AI scrapers, but the fact that Anubis imposes an unusual flow to load the site. If that's true Anubis should just remove the proof-of-work part, so legitimate human visitors don't have to stare at a loading screen for several seconds while their device wastes electricity.
- kaszanka 1y ago
  This is basically what most of the challenge types in go-away (https://git.gammaspectra.live/git/go-away/wiki/Challenges https://git.gammaspectra.live/git/go-away/wiki/Challenges) do.
  - Tmpod 1y ago
    +1 for go-away. It's a bit more involved to configure, but worth the effort imo. It can be considerably more transparent to the user, triggering the nuclear PoW check less often, while being just as effective, in my experience.
- amarant 1y ago
  I feel like the future will have this, plus ads displayed while the work is done, so websites can profit while they profit.
  - silversmith 1y ago
    Every now and then I consider stepping away from the computer job, and becoming a lumberjack. This is one of those moments.
    - zxexz 1y ago
      Worth getting to know the in and outs of forest management now. I don’t think AI will take most tech jobs soon, but they sure as hell are already making them boring.
    - jones89176 1y ago
colordrops 1y ago
On a side note, is the anime girl image customizable? I did a quick Google search an it seems that only the commercial version offers rebranding.
- boomboomsubban 1y ago
  It's free software. The paid version includes an option to change it, and they ask politely that you don't change it otherwise.
technion 1y ago
It really should be recognised just how many people are watching Cloudflare interstitials on nearly every site these days (and I totally get why this happens) yet making a huge amount of noise about Anubis on a very small amount of sites.
- mlyle 1y ago
  I don't trip over CloudFlare except when in a weird VPN, and then it always gets out of my way after the challenge. Anubis screws with me a lot, and often doesn't work.
  - thayne 1y ago
    I get one basically every time I go to gitlab.com on Firefox. It is easy to pass the challange, but it isn't any better than Anubis.
  - necovek 1y ago
    It's the other way around for me sometimes — I've never had issue with Anubis, I frequently get it with CF-protected sites. (Not to mention all the sites which started putting country restrictions in on their generally useful instruction articles etc — argh)
  - Pinus 1y ago
    I’m planning a trip to France right now, and it seems like half the websites in that country (for example, ratp.fr for Paris public transport info) require me to check a CloudFlare checkbox to promise that I am a human. And of those that don’t, quite a few just plain lock me out...
    - ta988 1y ago
      And a lot of US sites don't work in France either, or they ban you after just a couple requests with no appeal...
    - Symbiote 1y ago
      I find the same when using some foreign sites. I think the operator must have configured that France is OK, maybe neighboring countries too, the rest of the world must be checked.
TylerE 1y ago
No, it’s exactly because I understand that it bothers me. I understand it will be effective against bots for a few months and best, and legitimate human users will be stuck dealing with the damn thing for years to come. Just like captchas.
hedora 1y ago
This was obviously dumb when it launched: 1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked. 2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and get stupid hot in the process. Note that (2) holds even if the AI is doing something smart like batch processing 10-ish articles at once.
- pilif 1y ago
  > This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources is even dumber. And so it all works out. And so the question remains: how dumb was it exactly, when it works so well and continues to work so well?
  - bananalychee 1y ago
    Does it actually? I don't think I've seen a case study with hard numbers.
    - pilif 1y ago
      Here’s one study https://dukespace.lib.duke.edu/server/api/core/bitstreams/816ef134-55cf-49f6-9a8b-1e8a2324b1ff/content https://dukespace.lib.duke.edu/server/api/core/bitstreams/81... And of all the high-profile projects implementing it, like the LKML archives, none have backed down yet, so I’m assuming the initial improvement in numbers must continue or it would have been removed since
      - bananalychee 1y ago
        Great, thanks for the link.
necovek 1y ago
> The point is that they hadn't, and this worked for quite a while. That's what I was hoping to get from the "Numbers" section. I generally don't look up the logs or numbers on my tiny, personal web spaces hosted on my server, and I imagine I could, at some point, become the victim of aggressive crawling (or maybe I have without noticing because I've got an oversized server on a dual link connection). But the numbers actually only show the performance of doing the PoW, not the effect it has had on any site — I am just curious, and I'd love it if someone has done the analysis, ideally grouped by the bot type ("OpenAI bot was responsible for 17% of all requests, this got reduced from 900k requests a day to 0 a day"...). Search, unfortunately, only gives me all the "Anubis is helping fight aggressive crawling" blog articles, nothing with substance (I haven't tried hard, I admit). Edit: from further down the thread there's https://dukespace.lib.duke.edu/server/api/core/bitstreams/816ef134-55cf-49f6-9a8b-1e8a2324b1ff/content https://dukespace.lib.duke.edu/server/api/core/bitstreams/81... but no analysis of how many real customers were denied — more data would be even better
raverbashing 1y ago
[flagged]
numpad0 1y ago
Wasn't there also weird behaviors reported by webadmins across the world, like crawlers used by LLM companies are fetching evergreen data ad nauseum or something along that? I thought the point of adding PoW than just blocking them was to convince them to at least do it right.
Kwpolska 1y ago
With all the SPAs out there, if you want to crawl the entire Web, you need a headless browser running JavaScript. Which will pass Anubis for free.
TZubiri 1y ago
As I understand it, this is Proof of Work, which is strictly not a mouse and cat situation.
- account42 1y ago
  It is because you are dealing with crawlers that already have a nontrivial cost per page, adding something relatively trivial that is still within the bounds regular users accept won't change the motivations of bad actors at all.
  - TZubiri 1y ago
    What is the existing cost per page? as far as I know an http request and some string parsing is somewhat trivial, say 14kb of bandwidth per page?
palata 1y ago
> they are just feigning a lack of understanding to be dismissive of Anubis. I actually find the featured article very interesting. It doesn't feel dismissive of Anubis, but rather it questions whether this particular solution makes sense or not in a constructive way.
- johnklos 1y ago
  I agree - the article is interesting and not dismissive. I was talking more about some of the people here ;)
  - dmesg 1y ago
    I still don't understand what Anubis solves if it can be bypassed too easily: If you use User-agent switcher (i emulate wget) as firefox addon on kernel.org or ffmpeg.org you save the entire check time and straight up skip Anubis. Apparently they use a whitelist for user-agents due to allowing legitimate wget usage on these domains. However if I (an honest human can) the scrapers and grifters can too. https://addons.mozilla.org/en-US/firefox/addon/uaswitcher/ https://addons.mozilla.org/en-US/firefox/addon/uaswitcher/ If anyone wants to try themselves. This is by no means against Anubis, but raising the question: Can you even protect a domain if you force yourself to whitelist (for a full bypass) easy to guess UAs?
    - hooverd 1y ago
      It's extra work for scrapers. They pretend to be upstanding citizens (Chrome UA from residential IPs). You can more easily block those.
      - 20after4 1y ago
        A lot of scrapers are actually utilizing some malware installed on residential user's machines, so the request is legitimately coming from a chrome UA on a residential ip.
        1y ago
dcow 1y ago
I deployed a proof of work based auth system once where every single request required hashing a new nonce. Compare with Anubis where only one request a week requires it. The math said doing it that frequently, and with variable argon params the server could tune if it suspected bots, would be impactful enough to deter bots. Would I do that again? Probably not. These days I’d require a weekly mDL or equivalent credential presentation. I have to disagree that an anti-bot measure that only works globally for a few weeks until bots trivially bypass it is effective. In an arms race against bots the bots win. You have to outsmart them by challenging them to do something that only a human can do or is actually prohibitively expensive for bots to do at scale. Anubis doesn't pass that test. And now it’s littered everywhere defunct and useless.
ehnto 1y ago
It's been going on for decades now too. It's a cat and mouse game that will be with us for as long as people try to exploit online resources with bots. Which will be until the internet is divided into nation nets, suffocated by commercial interests, and we all decide to go play outside instead.
- rob_c 1y ago
  No. This went into overdrive in the "AI" (crawlers for massive LLM for ML chatbot) era. Frankly it's something I'm sad we don't yet see a lawsuit for similar to the times v OpenAI. A lot of "new crawlers" claim to innocently forget about established standards like robots.txt I just wish people would name and shame the massive companies at the top stomping on the rest of the internet in an edge to "get a step up over the competition".
  - account42 1y ago
    Yes, this is not a problem that will be solved with technical measures. Trying to do so is only going to make the web worse for us humans.
  - ehnto 1y ago
    That doesn't really challenge what I said, there's not much "different this time" except the scale is commensurate to the era. Search engine crawlers used to take down websites as well. I understand and agree with what you are saying though, the cat and mouse is not necessarily technical. Part of solving the searchbot issue was also social, with things like robots.txt being a social contract between companies and websites, not a technical one.
topranks 1y ago
Sure. It might be a tool in the box. But it’s still cat and mouse. In my place we quickly concluded the scrapers have tons of compute and the “proof-of-work” aspect was meaningless to them. It’s simply the “response from site changed, need to change our scraping code” aspect that helps.
account42 1y ago
If you are going to rely on security through obscurity there are plenty of ways to do that that won't block actual humans because they dare use a non-mainstream browser. You can also do it without displaying cringeworthy art that is only there to get people to pay for the DRM solution you are peddling - that shit has no place in the open source ecosystem.
- ForHackernews 1y ago
  On the contrary: Making things look silly and unprofessional so that Big Serious Corporations With Money will pay thousands of dollars to whitelabel them is an OUTSTANDING solution for preserving software freedom while raising money for hardworking developers.
  - account42 1y ago
    I'd rather not raise money for "hardworking" developers if their work is spreading DRM on the web. And it's not just "Big Serious Corporations" that don't want to see your furry art.
    - ForHackernews 1y ago
      I'm not commenting on the value of this project (I wouldn't characterize captchas as DRM, but I see why you have that negative connotation) and I tend to agree with the OP that this is simply wasting energy, but the amount of seething over "anime catgirls" makes me want to write all the docs for my next projects in UwU text and charge for a whimsy-free version. (o˘◡˘o)
      - account42 1y ago
        Please do, it's better if people make their negative personality traits public so that you can avoid them before wasting your time. It will also be useful to show your hypocrisy when you inevitably complain about someone else doing something that you don't like.
rozab 1y ago
>But it takes a human some time and work to tell the crawler HOW. Yes, for these human-based challenges. But this challenge is defined in code. It's not like crawlers don't run JavaScript. It's 2025, they all use headless browsers, not curl.
sneak 1y ago
The fundamental failure of this is that you can’t publish data to the web and not publish data to the web. If you make things public, the public will use it. It’s ineffective. (And furry sex-subculture propaganda pushed by its author, which is out of place in such software.)
- sznio 1y ago
  >And furry sex-subculture propaganda pushed by its author if your first thought when seeing a catgirl is sex, i got bad news for you
  - deleted 1y ago
    [deleted]
- pferde 1y ago
  The misguided parenthetical aside, this is not about resources being public, this is about bad actors accessing those resources in a highly inefficient and resource-intensive manner, effectively DDOS-ing the source.
windward 1y ago
Many sufficiently technical people take to heart: - Everything is pwned - Security through obscurity is bad Without taking to heart: - What a threat model is And settle on a kind of permanent contrarian nihilist doomerism. Why eat greens? You'll die one day anyway.
monooso 1y ago
The author make it very clear that he understands the problem Anubis is attempting to solve. His issue is that the chosen approach doesn't solve that problem; it just inhibits access to humans, particularly those with limited access to compute resources. That's the opposite of being dismissive. The author has taken the time to deeply understand both the problem and the proposed solution, and has taken the time to construct a well-researched and well-considered argument.
raxxorraxor 1y ago
Everytime we need to deploy such mechanisms, you reward those that already crawled the data and you penalize newcomers and other honest crawlers. For some sites Anubis might be fitting, but it should be mindfully deployed.
ramblerman 1y ago
Did you read the article? OP doesn't care about bots figuring it out. It's about the compute needed to do the work. It's quite an interesting piece, I feel like you projected something completely different onto it. Your point is valid, but completely adjacent.
dwaite 1y ago
> As the botmakers circumvent, new methods of proof-of-notbot will be made available. Yes, but the fundamental problem is that the AI crawler does the same amount of work as a legitimate user, not more. So if you design the work such that it takes five seconds on a five year old smartphone, it could inconvenience a large portion of your user base. But once that scheme is understood by the crawler, it will delay the start of their aggressive crawling by... well-under five seconds. An open source javascript challenge as a crawler blocker may work until it gets large enough for crawlers to care, but then they just have an engineer subscribe to changes on GitHub and have new challenge algorithms implemented before the majority of the deployment base migrates.
casey2 1y ago
You don't even need to go there. If the damn thing didn't work the site admin wouldn't have added it and kept it. Sure the program itself is jank in multiple ways but it solves the problem well enough.

psionides 1y ago

The problem is that 7 + 2 on a submission form only affects people who want to submit something, Anubis affects every user who wants to read something on your site

account42 1y ago
The question then is why read only users are consuming so much resources that serving them big chunks of JS instead reduces loads of the server. Maybe improve you rendering and/or caching before employing DRM solutions that are doomed to fail anyway.
- Mateon1 1y ago
  The problem it's originally fixing is bad scrapers accessing dynamic site content that's expensive to produce, like trying to crawl all diffs in a git repo, or all mediawiki oldids. Now it's also used on mostly static content because it is effective vs scrapers that otherwise ignore robots.txt.

odo1242 1y ago

Also, it forces the crawler to gain code execution capabilities, which for many companies will just make them give up and scrape someone else.

wredcoll 1y ago
I don't know if you've noticed, but there's a few websites these days that use javascript as part of their display logic.
- odo1242 1y ago
  Yes, and those sites take way more effort to crawl than other sites. They may still get crawled, but likely less often than the ones that don't use JavaScript for rendering (which is the main purpose of Anubis - saving bandwidth from crawlers who crawl sites way too often). (Also, note the difference between using JavaScript for display logic and requiring JavaScript to load any content at all. Most websites do the first, the second isn't quite as common.)

cakealert 1y ago

This arms race will have a terminus. The bots will eventually be indistinguishable from humans. Some already are.

neumann 1y ago
It will be hard to tune them to be just the right level of ignorant and slow as us though!
- cwmoore 1y ago
  Soon enough there will be competing Unicode characters that can remove exclamation points.
overfeed 1y ago
> The bots will eventually be indistinguishable from humans Not until they get issued government IDs they won't! Extrapolating from current trends, some form of online ID attestation (likely based on government-issued ID[1]) will become normal in the next decade, and naturally, this will be included in the anti-bot arsenal. It will be up to the site operator to trust identities signed by the Russian government. 1. Despite what Sam Altman's eyeball company will try to sell you, government registers will always be the anchor of trust for proof-of-identity, they've been doing it for centuries and have become good at it and have earned the goodwill.
- bhawks 1y ago
  Can't wait to sign into my web browser with my driver's license.
  - overfeed 1y ago
    In all likelihood, most people will do so via the Apple Wallet (or the equivalent on their non-Apple devices). It's going to be painful to use Open source OSes for a while, thanks to CloudFlare and Anubis. This is not the future I want, but we can't have nice things.

tptacek 1y ago

Respectfully, I think it's you missing the point here. None of this is to say you shouldn't use Anubis, but Tavis Ormandy is offering a computer science critique of how it purports to function. You don't have to care about computer science in this instance! But you can't dismiss it because it's computer science. Consider: An adaptive password hash like bcrypt or Argon2 uses a work function to apply asymmetric costs to adversaries (attackers who don't know the real password). Both users and attackers have to apply the work function, but the user gets ~constant value for it (they know the password, so to a first approx. they only have to call it once). Attackers have to iterate the function, potentially indefinitely, in the limit obtaining 0 reward for infinite cost. A blockchain cryptocurrency uses a work function principally as a synchronization mechanism. The work function itself doesn't have a meaningfully separate adversary. Everyone obtains the same value (the expected value of attempting to solve the next round of the block commitment puzzle) for each application of the work function. And note in this scenario most of the value returned from the work function goes to a small, centralized group of highly-capitalized specialists. A proof-of-work-based antiabuse system wants to function the way a password hash functions. You want to define an adversary and then find a way to incur asymmetric costs on them, so that the adversary gets minimal value compared to legitimate users. And this is in fact how proof-of-work-based antispam systems function: the value of sending a single spam message is so low that the EV of applying the work function is negative. But here we're talking about a system where legitimate users (human browsers) and scrapers get the same value for every application of the work function. The cost:value ratio is unchanged; it's just that everything is more expensive for everybody. You're getting the worst of both worlds: user-visible costs and a system that favors large centralized well-capitalized clients. There are antiabuse systems that do incur asymmetric costs on automated users. Youtube had (has?) one. Rather than simply attaching a constant extra cost for every request, it instead delivered a VM (through JS) to browsers, and programs for that VM. The VM and its programs were deliberately hard to reverse, and changed regularly. Part of their purpose was to verify, through a bunch of fussy side channels, that they were actually running on real browsers. Every time Youtube changed the VM, the bots had to do large amounts of new reversing work to keep up, but normal users didn't. This is also how the Blu-Ray BD+ system worked. The term of art for these systems is "content protection", which is what I think Anubis actually wants to be, but really isn't (yet?). The problem with "this is good because none of the scrapers even bother to do this POW yet" is that you don't need an annoying POW to get that value! You could just write a mildly complicated Javascript function, or do an automated captcha.

wat10000 1y ago

Technical people are prone to black-and-white thinking, which makes it hard to understand that making something more difficult will cause people to do it less even though it’s still possible.

mattnewton 1y ago
I think the argument on offer is more, this juice isn't worth the squeeze. Each user is being slowed down and annoyed for something that bots will trivially bypass if they become aware of it.
- wat10000 1y ago
  If they become aware of it and actually think it’s worthwhile. Malicious bots work by scaling, and implementing special cases for every random web site doesn’t scale. And it’s likely they never even notice.
  - mattnewton 1y ago
    If this kind of security by not being noticed is the plan, why not just have a trivial (but unique) captcha that asks the user to click a button with no battery wasting computation?
    - account42 1y ago
      Because you can't sell that as a commercial solution that the open source software ecosystem provides free advertising to.
    - wat10000 1y ago
      That works too, but not quite as well so it decreases the unwanted activity somewhat less.

interstice 1y ago

The cost benefit calculus for workarounds changes based on popularity. Your custom lock might be easy to break by a professional, but the handful of people who might ever care to pick it are unlikely to be trying that hard. A lock which lets you into 5% of houses however might be worth learning to break.

Aurornis 1y ago

> This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be dismissive of Anubis. This is a confusing comment because it appears you don’t understand the well-written critique in the linked blog post. > This is like those simple things on submission forms that ask you what 7 + 2 is. Of course everyone knows that a crawler can calculate that! But it takes a human some time and work to tell the crawler HOW. The key point in the blog post is that it’s the inverse of a CAPTCHA: The proof of work requirement is solved by the computer automatically. You don’t have to teach a computer how to solve this proof of work because it’s designed for the computer to solve the proof of work. It makes the crawling process more expensive because it has to actually run scripts on the page (or hardcode a workaround for specific versions) but from a computational perspective that’s actually easier and far more deterministic than trying to have AI solve visual CAPTCHA challenges.

necovek 1y ago
But for actual live users who don't see anything but a transient screen, Anubis is a better experience than all those pesky CAPTCHAs (I am bored of trying to recognize bikes, pedestrian crossings, buses, hydrants). The question is if this is the sweet spot, and I can't find anyone doing the comparative study (how many annoyed human visitors, how many humans stopped and, obviously, how many bots stopped).
- JimDabell 1y ago
  > Anubis is a better experience than all those pesky CAPTCHAs (I am bored of trying to recognize bikes, pedestrian crossings, buses, hydrants). Most CAPTCHAs are invisible these days, and Anubis is worse than them. Also, CAPTCHAs are not normally deployed just for visiting a site, they are mostly used when you want to submit something.
  - necovek 1y ago
    We are obviously living a different Internet reality, and that's the whole point — we need numbers to really establish baseline truth. FTR, I am mostly browsing from Serbia using Firefox browser on a Linux or MacOS machine.

deleted 1y ago

[deleted]

agwa 1y ago

It sounds like you're saying that it's not the proof-of-work that's stopping AI scrapers, but the fact that Anubis imposes an unusual flow to load the site. If that's true Anubis should just remove the proof-of-work part, so legitimate human visitors don't have to stare at a loading screen for several seconds while their device wastes electricity.

kaszanka 1y ago
This is basically what most of the challenge types in go-away (https://git.gammaspectra.live/git/go-away/wiki/Challenges https://git.gammaspectra.live/git/go-away/wiki/Challenges) do.
- Tmpod 1y ago
  +1 for go-away. It's a bit more involved to configure, but worth the effort imo. It can be considerably more transparent to the user, triggering the nuclear PoW check less often, while being just as effective, in my experience.
amarant 1y ago
I feel like the future will have this, plus ads displayed while the work is done, so websites can profit while they profit.
- silversmith 1y ago
  Every now and then I consider stepping away from the computer job, and becoming a lumberjack. This is one of those moments.
  - zxexz 1y ago
    Worth getting to know the in and outs of forest management now. I don’t think AI will take most tech jobs soon, but they sure as hell are already making them boring.
  - jones89176 1y ago

colordrops 1y ago

On a side note, is the anime girl image customizable? I did a quick Google search an it seems that only the commercial version offers rebranding.

boomboomsubban 1y ago
It's free software. The paid version includes an option to change it, and they ask politely that you don't change it otherwise.

technion 1y ago

It really should be recognised just how many people are watching Cloudflare interstitials on nearly every site these days (and I totally get why this happens) yet making a huge amount of noise about Anubis on a very small amount of sites.

mlyle 1y ago
I don't trip over CloudFlare except when in a weird VPN, and then it always gets out of my way after the challenge. Anubis screws with me a lot, and often doesn't work.
- thayne 1y ago
  I get one basically every time I go to gitlab.com on Firefox. It is easy to pass the challange, but it isn't any better than Anubis.
- necovek 1y ago
  It's the other way around for me sometimes — I've never had issue with Anubis, I frequently get it with CF-protected sites. (Not to mention all the sites which started putting country restrictions in on their generally useful instruction articles etc — argh)
- Pinus 1y ago
  I’m planning a trip to France right now, and it seems like half the websites in that country (for example, ratp.fr for Paris public transport info) require me to check a CloudFlare checkbox to promise that I am a human. And of those that don’t, quite a few just plain lock me out...
  - ta988 1y ago
    And a lot of US sites don't work in France either, or they ban you after just a couple requests with no appeal...
  - Symbiote 1y ago
    I find the same when using some foreign sites. I think the operator must have configured that France is OK, maybe neighboring countries too, the rest of the world must be checked.

TylerE 1y ago

No, it’s exactly because I understand that it bothers me. I understand it will be effective against bots for a few months and best, and legitimate human users will be stuck dealing with the damn thing for years to come. Just like captchas.

hedora 1y ago

This was obviously dumb when it launched: 1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked. 2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and get stupid hot in the process. Note that (2) holds even if the AI is doing something smart like batch processing 10-ish articles at once.

pilif 1y ago
> This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources is even dumber. And so it all works out. And so the question remains: how dumb was it exactly, when it works so well and continues to work so well?
- bananalychee 1y ago
  Does it actually? I don't think I've seen a case study with hard numbers.
  - pilif 1y ago
    Here’s one study https://dukespace.lib.duke.edu/server/api/core/bitstreams/816ef134-55cf-49f6-9a8b-1e8a2324b1ff/content https://dukespace.lib.duke.edu/server/api/core/bitstreams/81... And of all the high-profile projects implementing it, like the LKML archives, none have backed down yet, so I’m assuming the initial improvement in numbers must continue or it would have been removed since
    - bananalychee 1y ago
      Great, thanks for the link.

necovek 1y ago

> The point is that they hadn't, and this worked for quite a while. That's what I was hoping to get from the "Numbers" section. I generally don't look up the logs or numbers on my tiny, personal web spaces hosted on my server, and I imagine I could, at some point, become the victim of aggressive crawling (or maybe I have without noticing because I've got an oversized server on a dual link connection). But the numbers actually only show the performance of doing the PoW, not the effect it has had on any site — I am just curious, and I'd love it if someone has done the analysis, ideally grouped by the bot type ("OpenAI bot was responsible for 17% of all requests, this got reduced from 900k requests a day to 0 a day"...). Search, unfortunately, only gives me all the "Anubis is helping fight aggressive crawling" blog articles, nothing with substance (I haven't tried hard, I admit). Edit: from further down the thread there's https://dukespace.lib.duke.edu/server/api/core/bitstreams/816ef134-55cf-49f6-9a8b-1e8a2324b1ff/content https://dukespace.lib.duke.edu/server/api/core/bitstreams/81... but no analysis of how many real customers were denied — more data would be even better

raverbashing 1y ago

[flagged]

numpad0 1y ago

Wasn't there also weird behaviors reported by webadmins across the world, like crawlers used by LLM companies are fetching evergreen data ad nauseum or something along that? I thought the point of adding PoW than just blocking them was to convince them to at least do it right.

Kwpolska 1y ago

With all the SPAs out there, if you want to crawl the entire Web, you need a headless browser running JavaScript. Which will pass Anubis for free.

TZubiri 1y ago

As I understand it, this is Proof of Work, which is strictly not a mouse and cat situation.

account42 1y ago
It is because you are dealing with crawlers that already have a nontrivial cost per page, adding something relatively trivial that is still within the bounds regular users accept won't change the motivations of bad actors at all.
- TZubiri 1y ago
  What is the existing cost per page? as far as I know an http request and some string parsing is somewhat trivial, say 14kb of bandwidth per page?

palata 1y ago

> they are just feigning a lack of understanding to be dismissive of Anubis. I actually find the featured article very interesting. It doesn't feel dismissive of Anubis, but rather it questions whether this particular solution makes sense or not in a constructive way.

johnklos 1y ago
I agree - the article is interesting and not dismissive. I was talking more about some of the people here ;)
- dmesg 1y ago
  I still don't understand what Anubis solves if it can be bypassed too easily: If you use User-agent switcher (i emulate wget) as firefox addon on kernel.org or ffmpeg.org you save the entire check time and straight up skip Anubis. Apparently they use a whitelist for user-agents due to allowing legitimate wget usage on these domains. However if I (an honest human can) the scrapers and grifters can too. https://addons.mozilla.org/en-US/firefox/addon/uaswitcher/ https://addons.mozilla.org/en-US/firefox/addon/uaswitcher/ If anyone wants to try themselves. This is by no means against Anubis, but raising the question: Can you even protect a domain if you force yourself to whitelist (for a full bypass) easy to guess UAs?
  - hooverd 1y ago
    It's extra work for scrapers. They pretend to be upstanding citizens (Chrome UA from residential IPs). You can more easily block those.
    - 20after4 1y ago
      A lot of scrapers are actually utilizing some malware installed on residential user's machines, so the request is legitimately coming from a chrome UA on a residential ip.
      - 1y ago

dcow 1y ago

I deployed a proof of work based auth system once where every single request required hashing a new nonce. Compare with Anubis where only one request a week requires it. The math said doing it that frequently, and with variable argon params the server could tune if it suspected bots, would be impactful enough to deter bots. Would I do that again? Probably not. These days I’d require a weekly mDL or equivalent credential presentation. I have to disagree that an anti-bot measure that only works globally for a few weeks until bots trivially bypass it is effective. In an arms race against bots the bots win. You have to outsmart them by challenging them to do something that only a human can do or is actually prohibitively expensive for bots to do at scale. Anubis doesn't pass that test. And now it’s littered everywhere defunct and useless.

ehnto 1y ago

It's been going on for decades now too. It's a cat and mouse game that will be with us for as long as people try to exploit online resources with bots. Which will be until the internet is divided into nation nets, suffocated by commercial interests, and we all decide to go play outside instead.

rob_c 1y ago
No. This went into overdrive in the "AI" (crawlers for massive LLM for ML chatbot) era. Frankly it's something I'm sad we don't yet see a lawsuit for similar to the times v OpenAI. A lot of "new crawlers" claim to innocently forget about established standards like robots.txt I just wish people would name and shame the massive companies at the top stomping on the rest of the internet in an edge to "get a step up over the competition".
- account42 1y ago
  Yes, this is not a problem that will be solved with technical measures. Trying to do so is only going to make the web worse for us humans.
- ehnto 1y ago
  That doesn't really challenge what I said, there's not much "different this time" except the scale is commensurate to the era. Search engine crawlers used to take down websites as well. I understand and agree with what you are saying though, the cat and mouse is not necessarily technical. Part of solving the searchbot issue was also social, with things like robots.txt being a social contract between companies and websites, not a technical one.

topranks 1y ago

Sure. It might be a tool in the box. But it’s still cat and mouse. In my place we quickly concluded the scrapers have tons of compute and the “proof-of-work” aspect was meaningless to them. It’s simply the “response from site changed, need to change our scraping code” aspect that helps.

account42 1y ago

If you are going to rely on security through obscurity there are plenty of ways to do that that won't block actual humans because they dare use a non-mainstream browser. You can also do it without displaying cringeworthy art that is only there to get people to pay for the DRM solution you are peddling - that shit has no place in the open source ecosystem.

ForHackernews 1y ago
On the contrary: Making things look silly and unprofessional so that Big Serious Corporations With Money will pay thousands of dollars to whitelabel them is an OUTSTANDING solution for preserving software freedom while raising money for hardworking developers.
- account42 1y ago
  I'd rather not raise money for "hardworking" developers if their work is spreading DRM on the web. And it's not just "Big Serious Corporations" that don't want to see your furry art.
  - ForHackernews 1y ago
    I'm not commenting on the value of this project (I wouldn't characterize captchas as DRM, but I see why you have that negative connotation) and I tend to agree with the OP that this is simply wasting energy, but the amount of seething over "anime catgirls" makes me want to write all the docs for my next projects in UwU text and charge for a whimsy-free version. (o˘◡˘o)
    - account42 1y ago
      Please do, it's better if people make their negative personality traits public so that you can avoid them before wasting your time. It will also be useful to show your hypocrisy when you inevitably complain about someone else doing something that you don't like.

rozab 1y ago

>But it takes a human some time and work to tell the crawler HOW. Yes, for these human-based challenges. But this challenge is defined in code. It's not like crawlers don't run JavaScript. It's 2025, they all use headless browsers, not curl.

sneak 1y ago

The fundamental failure of this is that you can’t publish data to the web and not publish data to the web. If you make things public, the public will use it. It’s ineffective. (And furry sex-subculture propaganda pushed by its author, which is out of place in such software.)

sznio 1y ago
>And furry sex-subculture propaganda pushed by its author if your first thought when seeing a catgirl is sex, i got bad news for you
- deleted 1y ago
  [deleted]
pferde 1y ago
The misguided parenthetical aside, this is not about resources being public, this is about bad actors accessing those resources in a highly inefficient and resource-intensive manner, effectively DDOS-ing the source.

windward 1y ago

Many sufficiently technical people take to heart: - Everything is pwned - Security through obscurity is bad Without taking to heart: - What a threat model is And settle on a kind of permanent contrarian nihilist doomerism. Why eat greens? You'll die one day anyway.

monooso 1y ago

The author make it very clear that he understands the problem Anubis is attempting to solve. His issue is that the chosen approach doesn't solve that problem; it just inhibits access to humans, particularly those with limited access to compute resources. That's the opposite of being dismissive. The author has taken the time to deeply understand both the problem and the proposed solution, and has taken the time to construct a well-researched and well-considered argument.

raxxorraxor 1y ago

Everytime we need to deploy such mechanisms, you reward those that already crawled the data and you penalize newcomers and other honest crawlers. For some sites Anubis might be fitting, but it should be mindfully deployed.

ramblerman 1y ago

Did you read the article? OP doesn't care about bots figuring it out. It's about the compute needed to do the work. It's quite an interesting piece, I feel like you projected something completely different onto it. Your point is valid, but completely adjacent.

dwaite 1y ago

> As the botmakers circumvent, new methods of proof-of-notbot will be made available. Yes, but the fundamental problem is that the AI crawler does the same amount of work as a legitimate user, not more. So if you design the work such that it takes five seconds on a five year old smartphone, it could inconvenience a large portion of your user base. But once that scheme is understood by the crawler, it will delay the start of their aggressive crawling by... well-under five seconds. An open source javascript challenge as a crawler blocker may work until it gets large enough for crawlers to care, but then they just have an engineer subscribe to changes on GitHub and have new challenge algorithms implemented before the majority of the deployment base migrates.

casey2 1y ago

You don't even need to go there. If the damn thing didn't work the site admin wouldn't have added it and kept it. Sure the program itself is jank in multiple ways but it solves the problem well enough.