I have done some webscraping with twisted and BeautifulSoup. But it was just for a few small sites. That was when I found http://scrapy.org/ http://scrapy.org/

I have done some webscraping with twisted and BeautifulSoup. But it was just for a few small sites. That was when I found http://scrapy.org/ http://scrapy.org/ , a complete python framework for writing webcrawlers. It looks promissing, but I haven't used it so far.