15 ms·
I disagree, fingerprinting is necessary to track humans and it will be used regardless of scrapers being there or not.
by codedokode 2mo ago
I disagree, fingerprinting is necessary to track humans and it will be used regardless of scrapers being there or not.
- coldbrewed 2mo agoI work at a CDN that provides bot detection services. I agree that there's baseline necessity in terms of fraud detection, and if not necessity then definitely financial motivation to fingerprint. But these days, abusive scraping is far and way the the main driver for fingerprinting. We don't fingerprint for ad purposes, and we destroy PII for humans as fast as we can because PII should be treated as radioactive. But we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down. The current approach to scraping is strip mining the Internet and is having the corresponding pollution effects that you'd expect. I'm fine with individuals doing whatever weird automation they want, more power to you, but it's this industrial scale crawling and extraction that's degrading the Internet from all angles.
- tormeh 2mo agoDo you think this could drive a shift towards more efficient web tech? I.e. more static websites, or websites served by faster software?
- deleted 2mo ago[deleted]
- 0xbadcafebee 2mo ago> we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down So, I have two dumb questions: 1) Can't the customer rate-limit connections? If the "abuse" in scraping is the number of requests... limit the number of requests? 2) There is probably a market now for federated authentication where the provider gives legal guarantees of anonymized fingerprint in exchange for either payment or selling data. This would provide an identifier for rate-limiting requests, and be pretty much the same as "Sign in with Google", but without identifying the user to the customer site. Are there too many problems with this that make it unfeasible, or would your company/customers be open to such a solution?
- sunnybeetroot 2mo ago1) Scrapers often rotate ip per request, rate limiting doesn’t help
- codedokode 2mo agoRate limiting requires them to buy more IPs (which are not free) and slows down scraping.
- sunnybeetroot 2mo agoYes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.
- fn-mote 2mo ago> Can't the customer rate-limit connections? Bad guys use a botnet (compromised residential computers and routers) to defeat rate limiting and ip-based blocking.
- cute_boi 2mo agoRecently there was a article where LG smart tv were hacked and used for this entire thing.
- coldbrewed 2mo agoExactly this. Residential proxies are _the_ tool that makes stealth browsers and massive scraping work, and it's overwhelmingly done without informed consent.
- mvdtnz 2mo ago> If the "abuse" in scraping is the number of requests... limit the number of requests? Step one in doing this is identifying which requests are coming from the same agent. Guess what fingerprinting helps with.
- nradov 2mo agoWhy is it necessary to track humans? It might be more profitable for advertising but that's not the same as necessary.
- zdragnar 2mo agoTracking humans isn't necessary. Fingerprinting is necessary to tracking humans.