10 ms·
Use proof-of-work captchas, many are private by default. Look into Private Captcha or Cap captcha.
by ribtoks 4mo ago
Use proof-of-work captchas, many are private by default. Look into Private Captcha or Cap captcha.
- matheusmoreira 4mo agoCan this be repurposed as some kind of distributed cryptocurrency mining mechanism? Pay websites by mining some monero in order to access them?
- phoronixrly 4mo agoHow does proof of work stop bots?
- stephantul 4mo agoBecause it destroys the economics of scraping. It’s too expensive with proof of work, or at least not as economically viable
- gruez 4mo agoDepends on what type of scraping you're trying to stop. For the dumb scrapers that would try to scrape every page on a git forge (for which there are a bazillion pages for a modest project, because of how the site works), yeah it might deter them enough to stop. For anything high value (eg. reddit comments or retail prices), 10s of cpu time isn't going to stop them.
- pmontra 4mo agoIt will not scare away bots but 10 seconds of wait (CPU or only a sleep) will turn away many real users. "This site is so slow, I'll use something else." A kind of reverse captcha.
- Hnrobert42 4mo agoMaybe, the proof of work can run in the background.
- btown 4mo agoOr it can run as part of a checkout wizard's "verifying your browser and processing your payment, don't close your tab" step.
- mattstir 4mo agoAt which point it exists solely to punish real human users? What scraper bot is going through checkout?
- BenjiWiebe 4mo agoThe credit card tester bots go through the checkout process. PoW wouldn't be a big issue for them though since their volume is much lower.
- deleted 4mo ago[deleted]
- stephantul 4mo agoSure, the whole premise is exactly that proof of work reduces the value of scraping, while having negligible impact on users. If the data is so valuable that bot operators are willing to pay 10s of cpu, then other measures are necessary. Nevertheless even for these high value cases, you can still argue that it disincentivizes the business model, it becomes less efficient.
- thayne 4mo agoIf it's high value, there isn't really much you can do that will be completely effective. Traditional captchas can often be beaten by AI, or by "captcha farms" where impoverished people are paid pennies to complete captchas. Fingerprinting can be beaten by using a full browser to make the requests. Basically anything you do is just a matter of making it more expensive for bots to access it.
- arbol 4mo agoBeating fingerprinting and beating traditional captcha is far more expensive than solving pow. Pow doesn't stop anyone, not even the most novice bot operators
- tobyhinloopen 4mo agoYou can just download all of Reddit from torrent sites
- arbol 4mo agoExcept it doesn't
- deleted 4mo ago[deleted]
- ranger_danger 4mo ago5W load for 2 seconds is 0.002Wh, I think we'll be fine
- ray_v 4mo agoIf it gets too expensive/time-consuming to scrape then it won't happen at scale (as much)?
- mootothemax 4mo agoSpeaking from the scraper’s perspective, I like proof of work; a ten year old 96-core server will cost a couple of quid to run for a few hours and will grab an absurd number of pages thanks to the access granted by repeatedly solving proofs of work. Small slick codebases too!
- tardedmeme 4mo agoThere's also the Anubis idea where your PoW is persistent until your IP address or session cookie changes, so you get to skip PoW in exchange for making yourself identifiable, which means the PoW can then be ramped up to take a couple of minutes. I don't use Anubis though. I just make my site not take five seconds to render a page so bots can overload it easily? It's not actually that hard?
- Velocifyer 4mo agoIt would be more profitable to mine bitcoin.
- arbol 4mo agoPoW doesn't stop bots.. It's an annoyance at most. A rate limiter and nothing more
- 0123456789ABCDE 4mo agoPoW difficulty can be scaled, eg: all cookies must work 1s, but 2nd cookie from the same ip, might have to do 2s of work ideally one would pick something a bit more forgiving than a linear function, to avoid penalizing too much users connecting from CGNAT
- arbol 4mo agoI think we're talking about 2 different things. PoW is annoying for basic scrapers but it really doesn't affect enterprise grade bot operations with access to unlimited residential proxies.
- 0123456789ABCDE 4mo agolook, if you have the kind of bank to pay $1/GB of web pages to get to whatever we're hosting, maybe just send an email to sales so we can give you discount on that.
- deleted 4mo ago[deleted]