5 ms·

If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a conf

If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a config file: https://gotripod.com/insights/super-simple-site-crawling-and-scraping/ https://gotripod.com/insights/super-simple-site-crawling-and... [1] http://go-colly.org/ http://go-colly.org/

IceWreck 5y ago
+1 for Go. Its easy concurrency makes it an awesome language for web scraping. The go-colly framework was a bit too restrictive for my needs, but its very easy to build something on top of the standard lib's net/http, its cookiejar, and a third party library called goquery (afaik go-colly uses this too). Fun Fact: We were scraping something from an apparantly zero rate limits azure blob container, and we had to enumerate around a million URLs daily (didn't know which URLs actually existed so we guessed an offset and enumerated from there, also we had to do it at a fixed time daily). We had proxys at our disposal but didn't need them cause the blob container did not rate-limit. I wrote the scraper in Go, but a friend wrote it in Rust. Using Go was fast enough, satisfying all our requirements, but it turned out that the Rust one was 3-5 times faster. I tried to improve the Go scraper's speed by tweaking net/http transport's parameters, increasing workers, removing all NOFILE limits from SystemD and tried to profile and remove the low hanging speed issues. Nothing reduced the gap. Then I replaced the net/http client with valyala/fasthttp (another http implementation in Go) , which made it as fast as or slightly faster than the Rust one which was using the reqwest crate as http client.
hivacruz 5y ago
I used this library to get familiar with Go. It is indeed very powerful and really easy to create a scraper. My main concerns though were about testing. What if you want to create tests to check if your scraper still gets the data we want? Colly allows nested scraping and it's easy to implement but you have all your logic into one big function, making it harder to test. Did you find a solution to this? I'm considering switching to net/http + GoQuery only to have more freedom.
- colinramsay 5y ago
  Not yet but my plan was to just have a static HTML site which the tests could run against.

IceWreck 5y ago

+1 for Go. Its easy concurrency makes it an awesome language for web scraping. The go-colly framework was a bit too restrictive for my needs, but its very easy to build something on top of the standard lib's net/http, its cookiejar, and a third party library called goquery (afaik go-colly uses this too). Fun Fact: We were scraping something from an apparantly zero rate limits azure blob container, and we had to enumerate around a million URLs daily (didn't know which URLs actually existed so we guessed an offset and enumerated from there, also we had to do it at a fixed time daily). We had proxys at our disposal but didn't need them cause the blob container did not rate-limit. I wrote the scraper in Go, but a friend wrote it in Rust. Using Go was fast enough, satisfying all our requirements, but it turned out that the Rust one was 3-5 times faster. I tried to improve the Go scraper's speed by tweaking net/http transport's parameters, increasing workers, removing all NOFILE limits from SystemD and tried to profile and remove the low hanging speed issues. Nothing reduced the gap. Then I replaced the net/http client with valyala/fasthttp (another http implementation in Go) , which made it as fast as or slightly faster than the Rust one which was using the reqwest crate as http client.

hivacruz 5y ago

I used this library to get familiar with Go. It is indeed very powerful and really easy to create a scraper. My main concerns though were about testing. What if you want to create tests to check if your scraper still gets the data we want? Colly allows nested scraping and it's easy to implement but you have all your logic into one big function, making it harder to test. Did you find a solution to this? I'm considering switching to net/http + GoQuery only to have more freedom.

colinramsay 5y ago
Not yet but my plan was to just have a static HTML site which the tests could run against.