You should add some request-rate settings to your robots.txt file to prevent robots from crawling your web site to often. Here is an example: http://www.sparkle

You should add some request-rate settings to your robots.txt file to prevent robots from crawling your web site to often. Here is an example: http://www.sparkledb.net/robots.txt http://www.sparkledb.net/robots.txt