5 ms·
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collecte
by ccgreg 2mo ago
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
- mips_avatar 2mo agoThanks for making common crawl as good as it is. It’s a really important part of making the Internet better
- ccgreg 2mo agoAppreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.