rorosen

10 ms·

1.
by rorosen 1y ago
If seeking to frames is good enough for random seeking, this can be done with zeekstd already. The cli sets a custom window log (ZSTD_c_windowLog) on the compression context when creating binary patches[1], I regularly use it with a window
2.
by rorosen 1y ago
It depends on the frame size you choose. Every frame requires a few bytes of additional metadata, how much exactly depends on other compression settings (e.g. frame checksums, which are 4 byte, are only present if enabled). I just tested wi
3.
by rorosen 1y ago
Zeekstd will just error when the seek table is corrupted. Scanning for frame boundaries should also be possible, though it isn't very efficient. If you don't need the seek table, you can just write it to /dev/null or not
4.
by rorosen 1y ago
You can decompress a complete file with "zeekstd d seekable.zst". Piping a seekable file for decompression via stdin isn't possible unfortunately. Decompression of seekable files requires to read the seek table first (which i
5.
by rorosen 1y ago
From what I can see zstd-seekable is more closely aligned to the C functions in the zstd repo. The decompress function in zstd-seekable starts decompression at the beginning of the frame to which the offset belongs and discards data until t
6.
by rorosen 1y ago
Writing the seek table to an external file is also possible with zeekstd, the initial spec of the seekable format doesn't allow this.
7.
by rorosen 1y ago
Yes, dictionaries should be totally possible. However, I've never tried them to be honest because I usually only compress big files. They can be set on the (de)compression contexts the same way as with regular zstd.
8.
by rorosen 1y ago
It's not standardized as far as I know.
9.
by rorosen 1y ago
Thanks! I was also surprised that there are very few tools to work with the seekable format. I could imagine that at least some people have a use-case for it. Yes, the name is a combination of zstd and seek. Funnily enough, I wanted to name
10.
Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format(github.com)
214 points by rorosen 1y ago|46 comments