6 ms·

Google at least claims that noindex will keep your site from getting crawled [1]. Do people think this is false? [1] https://developers.google.com/search/docs/

• eli 8mo ago
  Strictly speaking no, that doesn’t prevent crawling - at the least Googlebot has to fetch the page to see the meta tag or the robots.txt to see what’s allowed, and it will periodically recheck for changes. It doesn’t even prevent indexing. If a page is linked from elsewhere, Google will show it in search results even if noindex’d. And why does Google get to set these rules on my site anyway? I didn’t agree to them.