4 ms·

What if I put a link to this URL and google bot will follow? You will block google bot?

• Ded7xSEoPKYNsDd 11y ago
  Googlebot won't follow the link because it's listed in robots.txt.
  • ceequof 11y ago
    Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.
  • ryanlol 11y ago
    Yes, but it will index it.