6 ms·
Hacker News isn't even listed on search engines because it disallows Googlebot.
by sillysaurus 13y ago
Hacker News isn't even listed on search engines because it disallows Googlebot.
- pearjuice 13y agoIt is definitely listed.
- sillysaurus 13y agoHm, that's a recent change, then. HN wasn't listed just a week or two ago.
- simonbrown 13y agoI've been using Google to find things on HN for a few years. The file from January 2013 appears just as permissive: http://web.archive.org/web/20130109093744/http://news.ycombinator.com/robots.txt http://web.archive.org/web/20130109093744/http://news.ycombi...
- sillysaurus 13y agoI have been too. I was surprised to find that HN didn't show up on Google a couple weeks ago. Yes, it looks like I was wrong. No, pg isn't changing titles because of SEO.
- 542458 13y agoMaybe I'm missing something but https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt shows no such thing...
- sp332 13y agoI dont' think that's true. https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /submitlink? Disallow: /threads? Crawl-delay: 30 Links in comments are marked nofollow but submissions are not.