4 ms·

You could do this without effecting normal traffic depending on uniqueness of ip doing the scraping. Love the idea.

• swsieber 4y ago
  I think you missed the point - if people show up at $PROXY expect nice stuff but see junk, then they won't move over to $REAL and instead blame $REAL. E.g. you'd like some way to redirect people from $PROXY site to $REAL site, and disgusting content on $PROXY won't do that - it'll reflect poorly on $REAL
  • heelix 4y ago
    If you can identify the crawler - you can provide 'dynamic' content for that specific user context.
    • nomel 4y ago
      It's a proxy, so there's no "crawler". It's just an agent relaying to the user. Passing something to this proxy agent just passes it directly to the user.