Luckily, this question has been asked and answered fairly recently! Specifically https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item

Luckily, this question has been asked and answered fairly recently! Specifically https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item?id=5955043 from last week, complete with a response from Matt Cutts (https://news.ycombinator.com/item?id=5955374 https://news.ycombinator.com/item?id=5955374): > It's not that PG has a grudge against Google (or vice versa) or anything like that. I believe that search engine bots crawl Hacker News hard enough that PG blocks most crawling by bots. In the case of Google, he does allow us to crawl from some IP addresses, but it's true that Google isn't able to crawl/index every page on Hacker News. > Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this. > And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news http://www.bing.com/search?q=hacker+news > In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.