6 ms·
Allow? Deep down, do you think you have a choice? Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of book
by swarnie 4mo ago
Allow? Deep down, do you think you have a choice?
Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of books being gobbled up by TheZuck but ultimately do you really have a choice?
Rules and laws don't exists for you.
- alt227 4mo agoYes, Google advertises its crawler IP ranges and it is quite easy to keep track of this and block them. But only if you control the infrastructure that your site runs on of course.
- rolph 4mo agostego your site, google sees the red herring version, intended users see the payload. this has been done before, quite often, but toward ends morally askew.
- account42 4mo agoYou think they'll stick to those IPs any more than robots.txt when it comes to that?
- alt227 4mo agoYes I do, I also think they will continue identifying themselves in the user-agent header. https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers https://developers.google.com/crawling/docs/crawlers-fetcher...