6 ms·
Reddit blocks all search engines from crawling their comments, essentially blocking all user generated contents. See this directive in https://www.reddit.com/r
by lolpanda 3y ago
Reddit blocks all search engines from crawling their comments, essentially blocking all user generated contents.
See this directive in https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt
`Disallow: /r//comments////*`
But I can still search Reddit on Google. How does Google manage to get the data?
- gruez 3y agoI think you're misinterpreting the rule. The relevant robots.txt rules are: User-Agent: * Disallow: */comment/* Disallow: /r/*/comments/*/*/*/* The url for the comment sections looks like: https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/ https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/. This doesn't match the above rules because there's only 2 parts after "comments", not 4 parts as specified by the rules.
- hollerith 3y agoNow it says "Disallow: /"
- Ajay-p 3y agoDoes Reddit really block all search engines, or do "all" search engines abide by No Spiders? If it's freely available it's ripe for scraping, no matter what Reddit may say.