5 ms·

3taps says CL blocking "all general search engines"

25 points by sigmadelta 14y ago

homepage pop-up: "At approximately noon on Sunday August 5th, Craigslist instructed all general search engines to stop indexing CL postings -- effectively blocking 3taps and other 3rd party use of that data from these public domain sources. We are sorry that CL has chosen this course of action and are exploring options to restore service but may be down for an extended period of time unless we or CL change practices. As soon as we know more, we will share it here and on our Twitter account."

true_religion 14y ago
Pretty brilliant, I don't think Craiglist ever needed Google traffic at all anymore. People know to go there to buy and sell.
- calbear81 14y ago
  Not to mention that local classifieds are not really meant to have any permanence so often times when I saw a CL listing on Google for an item I was looking for, it was already sold or the posting was deleted. I'm still looking forward to a viable Craigslist competitor though.
sigmadelta 14y ago
Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
- mooism2 14y ago
  Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
  - sigmadelta 14y ago
    If that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
- dangrossman 14y ago
  They probably did it through the webmaster tools portals provided by the search engines themselves. 3Taps has gone from indexing ~2 million posts a day to virtually 0. I did a couple site:cityname.craigslist.org searches on Google, restricted to the past 24 hours, and got no results.
storborg 14y ago
I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
- andrewcooke 14y ago
  i don't know, but if i understand http://3taps-statistics.qatro.com/craigslist/index.pl http://3taps-statistics.qatro.com/craigslist/index.pl correctly then they are missing lots of posts. the numbers seem to be percentages.
  - dangrossman 14y ago
    http://3taps.com/stats http://3taps.com/stats Scroll down to the "daily report" and they've dropped from almost 2 million posts a day to 153.
    - storborg 14y ago
      Yeah. This is all talking about 3taps, not "general search engines". 3taps seems to be claiming that Craigslist has cut off Google, but I think it's just that Craigslist has cut off 3taps. *Edit: Craigslist added nocache directives to their posts, which means that 3taps can't scrape the Google cached copies. They're not blocking anyone. Interestingly, this also reveals that 3taps was previously violating the Google TOS, which prohibits automated access of the Google cache.
      - dangrossman 14y ago
        Do some site:city.craigslist.org searches and limit to 24 hours. You'll get no results. You can't cut off 3taps without cutting off Google.
        14y ago
sigmadelta 14y ago
http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3taps-padmapper http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3...
sigmadelta 14y ago
http://www.sfgate.com/technology/businessinsider/article/Craigslist-Is-Definitely-Blocking-Search-3769297.php http://www.sfgate.com/technology/businessinsider/article/Cra... Not sure I agree with most the conclusions drawn in that article. The article does say that "sure enough, Google displays recent listings from Craigslist right now," which does seem to be true for me, too, when I try.
sigmadelta 14y ago
https://twitter.com/markmilian/statuses/233015694432813057 https://twitter.com/markmilian/statuses/233015694432813057 Mark Milian ‏@markmilian 7 Aug Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still allowing indexing of pages." Still nothing from CL PR
sigmadelta 14y ago
http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs-off-exclusive-rights-to-ads/ http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs... "One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from including Craigslist pages in search results. But that report was inaccurate. 3taps’ product and quality assurance leader, Meg Nakamura, acknowledged Wednesday in a chat with The Chronicle that something fishy was taking place, but developers there haven’t fully figured out what’s going on."

25 points by sigmadelta 14y ago

true_religion 14y ago

Pretty brilliant, I don't think Craiglist ever needed Google traffic at all anymore. People know to go there to buy and sell.

calbear81 14y ago
Not to mention that local classifieds are not really meant to have any permanence so often times when I saw a CL listing on Google for an item I was looking for, it was already sold or the posting was deleted. I'm still looking forward to a viable Craigslist competitor though.

sigmadelta 14y ago

Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.

mooism2 14y ago
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
- sigmadelta 14y ago
  If that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
dangrossman 14y ago
They probably did it through the webmaster tools portals provided by the search engines themselves. 3Taps has gone from indexing ~2 million posts a day to virtually 0. I did a couple site:cityname.craigslist.org searches on Google, restricted to the past 24 hours, and got no results.

storborg 14y ago

I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?

andrewcooke 14y ago
i don't know, but if i understand http://3taps-statistics.qatro.com/craigslist/index.pl http://3taps-statistics.qatro.com/craigslist/index.pl correctly then they are missing lots of posts. the numbers seem to be percentages.
- dangrossman 14y ago
  http://3taps.com/stats http://3taps.com/stats Scroll down to the "daily report" and they've dropped from almost 2 million posts a day to 153.
  - storborg 14y ago
    Yeah. This is all talking about 3taps, not "general search engines". 3taps seems to be claiming that Craigslist has cut off Google, but I think it's just that Craigslist has cut off 3taps. *Edit: Craigslist added nocache directives to their posts, which means that 3taps can't scrape the Google cached copies. They're not blocking anyone. Interestingly, this also reveals that 3taps was previously violating the Google TOS, which prohibits automated access of the Google cache.
    - dangrossman 14y ago
      Do some site:city.craigslist.org searches and limit to 24 hours. You'll get no results. You can't cut off 3taps without cutting off Google.
      - 14y ago

sigmadelta 14y ago

http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3taps-padmapper http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3...

sigmadelta 14y ago

http://www.sfgate.com/technology/businessinsider/article/Craigslist-Is-Definitely-Blocking-Search-3769297.php http://www.sfgate.com/technology/businessinsider/article/Cra... Not sure I agree with most the conclusions drawn in that article. The article does say that "sure enough, Google displays recent listings from Craigslist right now," which does seem to be true for me, too, when I try.

sigmadelta 14y ago

https://twitter.com/markmilian/statuses/233015694432813057 https://twitter.com/markmilian/statuses/233015694432813057 Mark Milian ‏@markmilian 7 Aug Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still allowing indexing of pages." Still nothing from CL PR

sigmadelta 14y ago

http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs-off-exclusive-rights-to-ads/ http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs... "One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from including Craigslist pages in search results. But that report was inaccurate. 3taps’ product and quality assurance leader, Meg Nakamura, acknowledged Wednesday in a chat with The Chronicle that something fishy was taking place, but developers there haven’t fully figured out what’s going on."