sigmadelta

5 ms·

1. 1.▲
   by sigmadelta 14y ago

   craigslist didn't change robots.txt: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
2. 2.▲
   by sigmadelta 14y ago

   http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs... "One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from including Craigslist pages in search results. But that repor
3. 3.▲
   by sigmadelta 14y ago

   Saying that "newyork.craigslist.org does not have any robots.txt and can be crawled as you like" is false. Search engines follow redirects until valid robots.txt files are found. From that same document you linked: 3xx (redirection) Redir
4. 4.▲
   by sigmadelta 14y ago

   http://www.seroundtable.com/archives/019533.html That article is from 2009, noting that CL had added the meta tag to posting pages. So I think this is old news. I think storborg has it right in http://news.ycombinator.com/item?id=435312
5. 5.▲
   by sigmadelta 14y ago

   https://twitter.com/markmilian/statuses/233015694432813057 Mark Milian ‏@markmilian 7 Aug Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still allowing indexing of pages." Still nothing from CL PR
6. 6.▲
   by sigmadelta 14y ago

   Just tried again, and I got a bunch of results: http://www.google.com/search?q=site%3Asfbay.craigslist.org+b... Of the 10 results on the first page, 7 are within the last hour. (1) Boat sfbay.craigslist.org/sby/boa/3191071012.html 2 hours
7. 7.▲
   by sigmadelta 14y ago

   Those tags aren't as obvious in interpretation as they first appear. A full write-up is a http://noarchive.net/meta/ for completeness, but in short: NOARCHIVE: ok to index this page, but don't cache the result permanently NOFOLLOW: don't
8. 8.▲
   by sigmadelta 14y ago

   http://www.sfgate.com/technology/businessinsider/article/Cra... Not sure I agree with most the conclusions drawn in that article. The article does say that "sure enough, Google displays recent listings from Craigslist right now," which do
9. 9.▲
   by sigmadelta 14y ago

   http://news.ycombinator.com/item?id=4348808
10. 10.▲
    by sigmadelta 14y ago

    Did these people do any fact checking? I just ran another test: visit http://sfbay.craigslist.org/bia/ spot ad for a kids Giant MTX 250 bike http://sfbay.craigslist.org/nby/bik/3189980073.html perform a google search for the same thing
11. 11.▲
    by sigmadelta 14y ago

    I just tried the following: visited http://sfbay.craigslist.org/bia/ saw ad for "Burning Man Beach Cruiser Bicycle" http://sfbay.craigslist.org/eby/bik/3189652975.html performed the following search on google http://www.google.com/sear
12. 12.▲
    by sigmadelta 14y ago

    http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3...
13. 13.▲
    by sigmadelta 14y ago

    If that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
14. 14.▲
    by sigmadelta 14y ago

    Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
15. 15.▲
    3taps says CL blocking "all general search engines"

    25 points by sigmadelta 14y ago|16 comments