5 ms·

Writeup on design and changes: https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c894ed5 https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c..

Writeup on design and changes: https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c894ed5 https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c... Block and Stream Format: https://raw.githubusercontent.com/minio/minlz/refs/heads/main/SPEC.md https://raw.githubusercontent.com/minio/minlz/refs/heads/mai...

ptman 2y ago
The blog post is quite nice: https://blog.min.io/minlz-compression-algorithm/ https://blog.min.io/minlz-compression-algorithm/ But comparisons with zstd & lz4 would be nice. Is someone planning on a wuffs/rust implementation with C API so that it wouldn't be Go only?
- klauspost 2y ago
  The main reason I took the effort to write the spec was to encourage ports - and commit to the format. I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C implementation. Having been 10+ years since I've done any serious C, I didn't want to assume I could write a good C API for it. If anyone are interested in picking it up, reach out. > comparisons with zstd & lz4 The repo has lz4 comparisons (Go version) for both blocks and streams. I see MinLZ more like a compliment to zstd, just like lz4 - where speed matter more than ultimate size. Here is the fully parallel speed of block decompressing with zstd or minlz - Go versions: * Protobuf - zstd: 31,597 MB/s - MinLZ 155,804 MB/s. * HTML - zstd: 25,157 MB/s - MinLZ 82,292 MB/s. * URL List - zstd: 16,869 MB/s - MinLZ 45,521 MB/s. * GEO data - zstd: 11,837 MB/s - MinLZ 36,566 MB/s. Of course zstd compresses to a smaller size - but for things like streams transfers or for fast readbacks from disk you probably want the fastest. Overall, I would say that MinLZ is about 2x faster decompressing, and compresses about 2x the speed compressing at max speed. But of course with less compression.

ptman 2y ago

The blog post is quite nice: https://blog.min.io/minlz-compression-algorithm/ https://blog.min.io/minlz-compression-algorithm/ But comparisons with zstd & lz4 would be nice. Is someone planning on a wuffs/rust implementation with C API so that it wouldn't be Go only?

klauspost 2y ago
The main reason I took the effort to write the spec was to encourage ports - and commit to the format. I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C implementation. Having been 10+ years since I've done any serious C, I didn't want to assume I could write a good C API for it. If anyone are interested in picking it up, reach out. > comparisons with zstd & lz4 The repo has lz4 comparisons (Go version) for both blocks and streams. I see MinLZ more like a compliment to zstd, just like lz4 - where speed matter more than ultimate size. Here is the fully parallel speed of block decompressing with zstd or minlz - Go versions: * Protobuf - zstd: 31,597 MB/s - MinLZ 155,804 MB/s. * HTML - zstd: 25,157 MB/s - MinLZ 82,292 MB/s. * URL List - zstd: 16,869 MB/s - MinLZ 45,521 MB/s. * GEO data - zstd: 11,837 MB/s - MinLZ 36,566 MB/s. Of course zstd compresses to a smaller size - but for things like streams transfers or for fast readbacks from disk you probably want the fastest. Overall, I would say that MinLZ is about 2x faster decompressing, and compresses about 2x the speed compressing at max speed. But of course with less compression.