This is true :) been a great tool to write a spider in, fast native code, a strict type system and very cheap threads has made it much faster than it otherwise

This is true :) been a great tool to write a spider in, fast native code, a strict type system and very cheap threads has made it much faster than it otherwise would have been. ZeroMQ has also been a great help in modularising it. I'll try to get a piece out on the technical architecture later. (I'm the CTO, btw.)