6 ms·
craigslist didn't change robots.txt: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modifi
by sigmadelta 14y ago
craigslist didn't change robots.txt:
$ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified
Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT