5 ms·

Step 1, create a named pipe, and open it in non-blocking mode. ... Unless things have changed a lot in this domain in the last few years, that won't work reli

Step 1, create a named pipe, and open it in non-blocking mode. ... Unless things have changed a lot in this domain in the last few years, that won't work reliably _anywhere_.

harikb 3y ago
Go team does acknowledge [1] it as a bug, so there is some point here However, that said, I wonder if OP (duckdb) could have written their solution [2] differently. Shouldn't they be able to select from a Pipe as well as Error channel simultaneously? (similar to how they are doing it inside here [3]). If not, I would have create a go-routine that does blocking read on the Pipe and then pass it on to another channel to select on. [1] https://github.com/golang/go/issues/66239 https://github.com/golang/go/issues/66239 [2] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e202126c581924c3bee2b1b2b10335d/pkg/destinations/duckdb/query.go#L14 https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20... [3] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e202126c581924c3bee2b1b2b10335d/pkg/destinations/duckdb/query.go#L94 https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20...
- memset 3y ago
  I’m not sure you can. Both the Open() and Read() calls will block. You can select() on a pipe but not on open. I haven’t checked to see if you can select on a non blocking fd though. This is what the go wrappers do behind the scenes.
o11c 3y ago
Er, whyever wouldn't it? On Linux it certainly works as expected, and it seems that it works fine on OS X when avoiding the broken Go shims.
- chipaca 3y ago
  I seem to remember non-blocking io with named pipes was unreliable, in the sense that the test suite of whatever-it-was (memory is reeealy sketchy, this was tens of years ago) would fail randomly often enough that we had to throw the easy solution away and do actual networking. A bit like the .signature named pipe from fortune thing, where sometimes you got two signatures and sometimes none? Anybody remember that?
  - thwarted 3y ago
    Is this what you're referring to? signature file can't be a named pipe (or /proc/uptime) https://bugzilla.mozilla.org/show_bug.cgi?id=62993 https://bugzilla.mozilla.org/show_bug.cgi?id=62993 can't attach signature from pipe (fifo) https://bugzilla.mozilla.org/show_bug.cgi?id=163831 https://bugzilla.mozilla.org/show_bug.cgi?id=163831

Step 1, create a named pipe, and open it in non-blocking mode. ... Unless things have changed a lot in this domain in the last few years, that won't work reliably _anywhere_.

harikb 3y ago

Go team does acknowledge [1] it as a bug, so there is some point here However, that said, I wonder if OP (duckdb) could have written their solution [2] differently. Shouldn't they be able to select from a Pipe as well as Error channel simultaneously? (similar to how they are doing it inside here [3]). If not, I would have create a go-routine that does blocking read on the Pipe and then pass it on to another channel to select on. [1] https://github.com/golang/go/issues/66239 https://github.com/golang/go/issues/66239 [2] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e202126c581924c3bee2b1b2b10335d/pkg/destinations/duckdb/query.go#L14 https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20... [3] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e202126c581924c3bee2b1b2b10335d/pkg/destinations/duckdb/query.go#L94 https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20...

memset 3y ago
I’m not sure you can. Both the Open() and Read() calls will block. You can select() on a pipe but not on open. I haven’t checked to see if you can select on a non blocking fd though. This is what the go wrappers do behind the scenes.

o11c 3y ago

Er, whyever wouldn't it? On Linux it certainly works as expected, and it seems that it works fine on OS X when avoiding the broken Go shims.

chipaca 3y ago
I seem to remember non-blocking io with named pipes was unreliable, in the sense that the test suite of whatever-it-was (memory is reeealy sketchy, this was tens of years ago) would fail randomly often enough that we had to throw the easy solution away and do actual networking. A bit like the .signature named pipe from fortune thing, where sometimes you got two signatures and sometimes none? Anybody remember that?
- thwarted 3y ago
  Is this what you're referring to? signature file can't be a named pipe (or /proc/uptime) https://bugzilla.mozilla.org/show_bug.cgi?id=62993 https://bugzilla.mozilla.org/show_bug.cgi?id=62993 can't attach signature from pipe (fifo) https://bugzilla.mozilla.org/show_bug.cgi?id=163831 https://bugzilla.mozilla.org/show_bug.cgi?id=163831