Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
pipihu
searching Neon…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
6 ms
·
1.
▲
by
pipihu
12y ago
The main use for robots.txt is to prevent crawling of infinite URL spaces: http://googlewebmastercentral.blogspot.com.br/2008/08/to-inf... Alongside tagging links to such resources with nofollow.