6 ms·
This library is wild https://github.com/cvilsmeier/sqinn https://github.com/cvilsmeier/sqinn Sqlite over stdin, to a subprocess, and it's fast!
by maxmcd 1y ago
This library is wild https://github.com/cvilsmeier/sqinn https://github.com/cvilsmeier/sqinn
Sqlite over stdin, to a subprocess, and it's fast!
- Twirrim 1y agoIt's wild to me that stdin/stdout is apparently significantly faster than using the API in so many cases. That's the kind of result that makes me wonder if there is something odd with the benchmarking.
- kreelman 1y agoThat's an interesting thought. I wonder. I wonder if the following things make the C driven version slower... - prepare the send buffers (sqlite side) - prepare the receive buffers (go side) - do the call - get the received data into go buffers of some kind - free up the send buffers (happens automatically) - free up the receive buffers (semi automatically in Go). When using stdin/stdout, the system looks after send/receive buffers. It's simply reading/writing them. No allocation is needed. The stream can be as big or as little as wanted/needed. The OS will look after the integrity of the streams and these are probably fairly well tested subsystems on most operating systems. stdin/stdout becomes a "library" for "fast data transfer". Pretty neat.
- raggi 1y agofwiw, the tailscale fork of Crawshaw’s library has a good number of allocation removals and other optimizations, but cgo is still expensive.
- kitd 1y agoAnd presumably that implies there's OS context switching going on underneath. Still, I can see a few downsides. Though sqinn-go is pure Go, the forked process is pure C, so you'll need to either download a prebuilt one (Linux and Windows only atm), or build it yourself. This rather defeats the benefits of Go's killer feature of "single-binary distribution". Still, I agree it's wild it is so fast.
- karel-3d 1y agonote that the author of the benchmark is also author of this library.
- sureglymop 1y agoThat's an interesting approach but doesn't it mean that if you want multiple connections at the same time, you'd need multiple subprocesses? Perhaps I misunderstand though.
- cvilsmeier 1y agoYou are correct: Multiple connections means multiple sqinn subprocesses.
- wener 1y agoI used to use sqlite3 with stdio to read VoIP SQLite data. It's difficult or impossible to get a compatible SQLite version, and it's also hard to use cgo. I want to read the SQLite data on the server, and stdio is the only choice.