11 ms·
I'm puzzled as there seems to be a clear pattern on how a human user would look like vs a bot. High bot likely hood if: If a session jumps to a different ip. I
by schobi 17d ago
I'm puzzled as there seems to be a clear pattern on how a human user would look like vs a bot.
High bot likely hood if: If a session jumps to a different ip. If the session jumps IP after just a few requests. If a new blank session starts with a deep link. Maybe those are cases where some POW is better justified?
Assumption: the rendered HTML might be viewed by a legitimate developer, even via a deep link from outside. But rarely from a wget script without a session cookie .
The other nice idea from the comments - is this rendering effort something that can be pushed to the user? Instead of pointless POW work, can you offload the expensive rendering to the user side?
But certainly, this is just an armchair comment and the kernel guys certainly have tried everything in this arms race...
- dewey 17d ago> But rarely from a wget script without a session cookie If you do scraping on a large scale you emulate a human very well to bypass bot protection. Setting dynamic but accurate user agents, setting proper sessions and persisting it, emulating the TLS handshake (https://fingerprint.com/blog/what-is-tls-fingerprinting-transport-layer-security/ https://fingerprint.com/blog/what-is-tls-fingerprinting-tran...) and emulating mouse movements to be more "human like" (https://github.com/oxylabs/OxyMouse https://github.com/oxylabs/OxyMouse) are table stakes.