8 ms·
This is pretty similar to Sparkey[0] and bam[1]. Sparkey also comes from growing out of cdb's limitations. It supports block-level compression like Riffle does,
by jdp 12y ago
This is pretty similar to Sparkey[0] and bam[1]. Sparkey also comes from growing out of cdb's limitations. It supports block-level compression like Riffle does, and is optimized for accepting bulk writes. Riffle's linear-time merge behavior lifted from Sorted String Tables is a nice alternative to accepting writes at runtime. bam is cool in that it takes a plain separated values file as input, and builds an index file from a minimal perfect hash function over the input file.
[0]: https://github.com/spotify/sparkey https://github.com/spotify/sparkey
[1]: https://github.com/StefanKarpinski/bam https://github.com/StefanKarpinski/bam
- prospero 12y agoThere are a lot of variants on this design out there, I had seen 'bam' but not the Spotify implementation. An additional constraint we had that I didn't allude to in the post was avoiding JNI, which adds some nasty failure modes for remote installations that can be very hard to debug. This meant any C implementation was off-limits for us. It's unfortunate that using the JVM means that some wheels need to be reinvented, but those are the breaks, I guess.