Or your user-agent is the same as one. Unless they are whitelisting IPs. I tested this with: wget --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +h

Or your user-agent is the same as one. Unless they are whitelisting IPs. I tested this with: wget --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://www.google.com/bot.html)" http://www.facebook.com/robots.txt http://www.facebook.com/robots.txt and diffed, but the results are the same.