7 ms·
This is true but irrelevant to the parent's question -- in the article, it's made clear that Google's requests are happening over HTTP, which is the most obviou
by cmatthias 4y ago
This is true but irrelevant to the parent's question -- in the article, it's made clear that Google's requests are happening over HTTP, which is the most obvious reason why robots.txt should be respected.
- yamtaddle 4y agoIt's relevant because it attacks the premise of their objection.