5 ms·
No, it doesn't. Your linked example contains: User-agent: * Disallow: / While the (new) digg.com/robots.txt contains: User-agent: * Disallow: Those
by dredge 14y ago
No, it doesn't. Your linked example contains:
User-agent: *
Disallow: /
While the (new) digg.com/robots.txt contains:
User-agent: *
Disallow:
Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.