81 ms·
A Web Crawler with Asyncio Coroutines
- theVirginian 11y agoGreat tutorial, I would love to see this rewritten with the new async / await syntax in python 3.5
- justusw 11y agoI've created a similar example in order to try out the new Python 3.5 async syntax. While the async function bodies themselves do not change, there is some boilerplate necessary in order to call async functions. You can check it out right here https://github.com/justuswilhelm/kata/blob/master/python/coroutine.py#L35 https://github.com/justuswilhelm/kata/blob/master/python/cor...
- rgacote 11y agoAppreciate the in-depth description. Look forward to working through this in detail.
- Animats 11y agoIt would be interesting to compare this Python approach with a Go goroutine approach. The main question is whether Go's libraries handle massive numbers of connections well. Since Google wrote Go to be used internally, they probably do.
- fabiandesimone 11y agoI'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is doing about 120K pages per day which to our initial needs is not bad at all, but wonder if in the crawling world this is peanuts or a decent chunk of pages?
- Jake232 11y agoI have scrapers built in Python that do well over a million pages per day, but that's not really a benchmark you can use. It all depends on the amount of computation required to extract the page data among other things. You should be able to achieve > 120k per day for sure though. That's less than two per second.
- fabiandesimone 11y agoThank you. Well, I'm doing several things: 1) I check whether or not the page we just scrapped has any of the tags we are looking for. 2) We then extract any information within those tags (images, etc.) 3) We follow trough every link and if it's not in the seen/scrapped list, we add them to the queue. Not sure if this helps to narrow it down. Thanks!
- reinhardt 11y agoIt doesn't make much sense to give a number for speed without some specifics about the crawler environment, such as: - How many servers (if distributed)? - How many cores/server? - What kind of processing takes place for each page? Does it just download and save the pages somewhere (local filesystem, cloud storage, database) or it extracts (semi) structured data? And so on. Specifics aside, these days it's not hard to crawl millions of pages/day on commodity servers. Some related posts: http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/ http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil... http://blog.semantics3.com/how-we-built-our-almost-distributed-web-crawler/ http://blog.semantics3.com/how-we-built-our-almost-distribut... http://engineering.bloomreach.com/crawling-billions-of-pages-building-large-scale-crawling-cluster-part-1/ http://engineering.bloomreach.com/crawling-billions-of-pages... http://engineering.bloomreach.com/crawling-billions-of-pages-building-large-scale-crawling-cluster-part-2/ http://engineering.bloomreach.com/crawling-billions-of-pages...
- juddlyon 11y agoNode is well-suited for this type of thing and there are numerous libraries to help.
- Schwolop 11y agoThis article is way more important than the web crawler example used to motivate it. It's easily the single best thing I've ever read on asyncio, and I've been using it in anger for a year now. I've passed it around my team, and will be recommending it far and wide!
- potatosareok 11y agoOne question I have about this - and I might have missed in article is - I'm all for using asyncio to make HTTP requests. But I see they apparently also use asyncio for "parse_links". Since parselinks should be CPU op, would it make sense to use fibers to download links and pass them into a thread pool to actually parse them//add to queue? I'm messing around with some of the ParallelUniverse Java fiber implementation and what I do is spam fibers to download pages and send the String response over to another fiber over a channel that maintains a thread pool to parse response body as they come in//create new fibers to read these links. I'm really just doing this to get more familiar with async programming and specifically the paralleluniverse Java libs but one thing I'm struggling a bit with is how to best make it well behaved (e.g right now there's no bound on number of outstanding HTPT requests).