5 ms·
Golang custom transports and timeouts
- tptacek 11y agoI had a similar (though unrelated) problem early on with Golang writing a port scanner, which forced me to hoist my own event loop (build around an explicit select call) into my program: timeout or not, I needed lots and lots of OS file descriptors, and by relying on Golang's own event loop to close stale ones on timeouts, I'd invariably run out.
- FiloSottile 11y agoSimilar unrelated problem here, too, on the sole machine now running the Heartbleed test. After recompiling with Go 1.4 (from Go 1.2 - apparently the network poller was rewritten?) I'd start running out of file descriptors really fast, all leaking deep in net/http.ReadRequest. Turns out http.DefaultServer doesn't have any default ReadTimeout, so when a client goes away before finishing a request I'd lose the goroutine and the file descriptor with it. So yeah, you probably also want to explicitly set http.Server.ReadTimeout if you are not doing weird streaming/"comet" stuff.
- tptacek 11y agoThat's related to my problem, but not the same; what I needed was control over when an fd/goroutine would be destroyed. Explicit timeouts (at least in 1.0, but I think still) couldn't give me that. I needed an obscene number of fds, though.
- sebcat 11y ago> I needed lots and lots of OS file descriptors Why? I ask because I wrote a port scanner professionally once, and I got away with using one fd for network I/O. I can only think of one case where you end up using a lot of fds and that is if you're using TCP sockets and connect(2) where you'd end up with one fd/tested port, and that's not good.
- tptacek 11y agoIt had to work unprivileged in userland.
- jbox 11y agoIf you're going across the network, a slow timeout is often trickier than an explicit failure. I wrote about handling this in Python: http://www.mobify.com/blog/http-requests-are-hard/ http://www.mobify.com/blog/http-requests-are-hard/