6 ms·
Honestly, what would be interesting is if there was an open source database of crawled webpages, available for anyone to search / use with their own algorithms.
by LeftHandPath 7y ago
Honestly, what would be interesting is if there was an open source database of crawled webpages, available for anyone to search / use with their own algorithms. That would make it possible for... a lot of things, really.
I feel like the web parsing / indexing, perhaps rather than the search algorithm itself, is the hardest part of rolling a new search engine (largely due to the associated computing and storage costs).
- jkaptur 7y agoThere is https://commoncrawl.org/ https://commoncrawl.org/, but it would be really cool if there were a more well-lit path towards building the rest of a simple search engine. For example, another commentator wanted “like Google, but without the spelling correction”, well, spin up one of these and just stub out the spelling module :)
- LeftHandPath 7y agoThanks for the tip! I'll have to play around with that sometime.