The main use for robots.txt is to prevent crawling of infinite URL spaces: http://googlewebmastercentral.blogspot.com.br/2008/08/to-infinity-and-beyond-no.html

The main use for robots.txt is to prevent crawling of infinite URL spaces: http://googlewebmastercentral.blogspot.com.br/2008/08/to-infinity-and-beyond-no.html http://googlewebmastercentral.blogspot.com.br/2008/08/to-inf... Alongside tagging links to such resources with nofollow.