5 ms·
I've been developing web crawlers for the better half of a decade. They are used for various purposes such as cataloging sentiment/bias in news media, finding n
by ikinsey 5y ago
I've been developing web crawlers for the better half of a decade. They are used for various purposes such as cataloging sentiment/bias in news media, finding new tv shows to watch, or mapping out the Tor hidden service directory.
Currently, I am writing a web crawler application framework in golang. Always looking for help or new ideas on what to crawl next!
Emails welcome, check my profile.
- vivegi 5y agoI had posted this on Ask HN (https://news.ycombinator.com/item?id=30096235#30096410 https://news.ycombinator.com/item?id=30096235#30096410) a few days ago. Your crawler perhaps could be customized to crawl and publish an index of all available Progressive Web Apps. A naive way would be to check for sites that have a PWA App manifest file in their root folder. Let me know if you are interested in collaborating.
- ikinsey 5y agoThis looks very possible. It would only require a two modules for analysis and frontier management. It would be great to collaborate on something like this!
- vivegi 5y agoThat's awesome. Thank you. Please do think about it for the next couple of days and I will drop an email to you in the next 48 hours (I live in India). We can then decide how to take it forward. Just curious: How long does an end-to-end crawl take and how much resources does the process consume (in terms of hardware/mem etc.,)?
- fsflover 5y agoWhat's your opinion about YaCy: https://yacy.net https://yacy.net?
- ikinsey 5y agoYaCy is a great tool! Haven't used it all that extensively since 2012. Very good for setting up simple crawls with minimal configuration or for crawling intranets.
- datavirtue 5y agoTips for information on getting into web crawling?