7 ms·

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably oth

by ok_coo 5y ago

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/ https://commoncrawl.org/

dewey 5y ago
I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.
mycall 5y ago
I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
- wumpus 5y ago
  web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
  - kevinsundar 5y ago
    Hey! I was using your tool a couple months ago. It was super helpful for my project.
    - wumpus 5y ago
      Thanks! I rarely hear from users, great to hear from you!
- kevinsundar 5y ago
  They do and its better than common crawl's by my testing.
joe_91 5y ago
That looks like a great resource! How often is the data set "updated"? I'd imagine most people's use cases need data which can change from day to day or week to week but I do think that this is fantastic if I was to have a project which was looking at data across a longer timeframe.
weird-eye-issue 5y ago
Have you used this in real world scenarios? Or is it just a nice hypothetical that sounds great in theory but almost never works in practice?
LunaSea 5y ago
Common Crawl is missing far too many URLs for it to be useful in a real world scenario.
- Chris2048 5y ago
  But can't you add to their index?
  - wumpus 5y ago
    No. You can add to the Wayback Machine at web.archive.org via their "save page now" interface... Common Crawl is attempting to be a sample of the web, and doesn't take url suggestions.
jimkri 5y ago
That is too much data to parse for a simple website scrape. I do think Common Crawl has a lot of potential for people to use instead of scraping, but I think its for larger projects. It gave me the idea to look at the links to ID if they are a business or non-business website

by ok_coo 5y ago

dewey 5y ago

I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.

mycall 5y ago

I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org

wumpus 5y ago
web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
- kevinsundar 5y ago
  Hey! I was using your tool a couple months ago. It was super helpful for my project.
  - wumpus 5y ago
    Thanks! I rarely hear from users, great to hear from you!
kevinsundar 5y ago
They do and its better than common crawl's by my testing.

joe_91 5y ago

That looks like a great resource! How often is the data set "updated"? I'd imagine most people's use cases need data which can change from day to day or week to week but I do think that this is fantastic if I was to have a project which was looking at data across a longer timeframe.

weird-eye-issue 5y ago

Have you used this in real world scenarios? Or is it just a nice hypothetical that sounds great in theory but almost never works in practice?

LunaSea 5y ago

Common Crawl is missing far too many URLs for it to be useful in a real world scenario.

Chris2048 5y ago
But can't you add to their index?
- wumpus 5y ago
  No. You can add to the Wayback Machine at web.archive.org via their "save page now" interface... Common Crawl is attempting to be a sample of the web, and doesn't take url suggestions.

jimkri 5y ago

That is too much data to parse for a simple website scrape. I do think Common Crawl has a lot of potential for people to use instead of scraping, but I think its for larger projects. It gave me the idea to look at the links to ID if they are a business or non-business website