6 ms·
I'm interested to know more about the tools/techniques you used to scrap the sites and clean the results. I am working on scraping contents from a couple of sit
by rprameshwor 9y ago
I'm interested to know more about the tools/techniques you used to scrap the sites and clean the results. I am working on scraping contents from a couple of sites and it seems to be a pain, probably because i don't have lot of experience around it.
- wasi0013 9y agoIf you are familiar with python then try scrapy[1]. You can also scrape websites using beautifulsoup4[2] [1] https://scrapy.org/ https://scrapy.org/ [2] https://pypi.python.org/pypi/beautifulsoup4 https://pypi.python.org/pypi/beautifulsoup4
- codefined 9y agoWe actually used Node.JS' request module, combined with some NLP (using natural) in order to pick out the main content. This worked pretty well, but for our purposes we didn't need it to be perfect because anything like headers would be removed when we processed the content (not being full sentences).