6 ms·
Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that wer
by sigmadelta 14y ago
Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
- mooism2 14y agoGoogle, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
- sigmadelta 14y agoIf that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
- dangrossman 14y agoThey probably did it through the webmaster tools portals provided by the search engines themselves. 3Taps has gone from indexing ~2 million posts a day to virtually 0. I did a couple site:cityname.craigslist.org searches on Google, restricted to the past 24 hours, and got no results.