6 ms·
Why not store the log files in an FM-index? (https://en.wikipedia.org/wiki/FM-index https://en.wikipedia.org/wiki/FM-index)
by dpkendal 11y ago
Why not store the log files in an FM-index? (https://en.wikipedia.org/wiki/FM-index https://en.wikipedia.org/wiki/FM-index)
- mattgodbolt 11y agoInteresting point, thanks for the link! For my use case I wanted to put stuff together without writing too much novel code; zindex uses a SQLite file to store indices and checkpoints. I'm hoping to add json support to zq/zindex, which means a general text search isn't that useful. Additionally the original log files I run against in my use case are generated upstream of me and are many many multi-gigabyte gzipped log files. I didn't want to add too much extra storage for my logs: the "key" I index on is a tiny part of my log, so the storage space for the index is considerably less than the original text (even accounting for the gzip checkpoints).