klauspost

7 ms·

1.
Minlz v1.2: Search compressed data with without decompression(github.com)
1 points by klauspost 2mo ago|1 comments
2.
by klauspost 2mo ago
Available via cli and Go API
3.
by klauspost 6mo ago
https://github.com/klauspost/stdgozstd
4.
by klauspost 6mo ago
About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one. Also making good progress on getting a slimmer version of zstd into th
5.
by klauspost 2y ago
The main reason I took the effort to write the spec was to encourage ports - and commit to the format. I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C
6.
by klauspost 2y ago
Writeup on design and changes: https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c... Block and Stream Format: https://raw.githubusercontent.com/minio/minlz/refs/heads/mai..
7.
MinLZ: Efficient and fast Snappy/LZ4 style compressor in Go (Apache 2.0)(github.com)
18 points by klauspost 2y ago|4 comments
8.
by klauspost 2y ago
I just released about 2 years of work on improving compression with a fixed encoding LZ77 style compressor. Our goal was to improve compression by combining and tweaking the best aspects of LZ4 and Snappy. The package provides Block (up to
9.
AIStor: The Most Powerful Version of MinIO(blog.min.io)
12 points by klauspost 2y ago|5 comments
10.
by klauspost 2y ago
In my experience, the hybrid approach is practically better due to cache locality anyway, so I don't think even native GF16 or GF32 would be of much help. FWIW, I've ported rs-leopard to Go and found it very effective, except in t
11.
by klauspost 2y ago
Almost all (x86) CPUs sold have GFNI. That can pretty much saturate memory bandwidth on a single core or two. You can use SSSE3 pshufb for the rest which is about half the speed. ARM has NEON and SVE/SVE 2. They also operate very fast.
12.
by klauspost 2y ago
Looking at it quickly, it seems like you just moving entropy to the frequency table. If the frequency table isn't included in the count, I could just make an "infinite" compressor by storing the frequency and cut one byte off
13.
by klauspost 3y ago
> If I could change one thing about S3's API I would like an option to read the metadata with the listings. Agree. In MinIO (disclaimer: I work there) we added a "secret" parameter (metadata=true) to include metadata and t
14.
Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf](github.com)
4 points by klauspost 3y ago|2 comments
15.
by klauspost 3y ago
Repo: https://github.com/mxmlnkn/rapidgzip As someone who has dabbled quite a lot with deflate this was a very interesting find. It seems like the format that never wants to die. First of all I am surprised this is eve
16.
by klauspost 3y ago
Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.
17.
by klauspost 3y ago
.. for small payloads. QAT cannot compress across blocks. So as soon as your input is more than 128KB the compression ratio tanks. Usually well below what even level 1 of software zstd does. They choose the input very carefully.
18.
by klauspost 4y ago
This could be very useful for a project I am just starting on. No documentation, and example has no content makes the learning curve a bit steep. Does anyone have any pointers on how to use this?
19.
by klauspost 4y ago
Reading it, it seems very similar to the Playstation 3 Cell, with its fate mirroring it very much. A highly specialized processor that has very high computational throughput for specialized operations, but a quite limited scalar unit. In bo
20.
by klauspost 4y ago
You can use presigned urls for that.
21.
Go Implementation of Leopard-RS GF16(github.com)
1 points by klauspost 4y ago|1 comments
22.
by klauspost 4y ago
Leopard-RS GF16 support added to reedsolomon package, with SSE3, AVX2 and AVX512 assembly. This allows to encode up to 65536 data+parity shards at O(K Log M), where K is the original data size, and M is up to twice the size of the recovery
23.
by klauspost 4y ago
> If you see a match that would not be visible from given compression level[...] Yes, but you'd need to reconstruct the hash table and the link table. For that you'd need to decompress fully. You could compare matches, but you
24.
by klauspost 4y ago
There is no way to "rule some levels out". The levels only differ in the quality of the lz matches. Slower levels search further potential matches - you cannot see that from a compressed stream. You can compare the output from eac
25.
by klauspost 4y ago
A block can end at any time and another huffman block with another table can start. There is no way to know that without decoding all symbols. Decoding the huffman codes is by far the most expensive operation, so you don't save much. F
26.
by klauspost 4y ago
If relevant for you, we have an unadvertised feature in MinIO, that adds Zstandard, LZ4, Snappy support: https://github.com/minio/minio/blob/master/internal/s3select...
27.
by klauspost 4y ago
You only need a single part to calculate a specific offset, assuming you have part sizes stored in metadata already (a good idea). Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB. Even if you unpack to 8*2 bytes in memory when de
28.
by klauspost 4y ago
You would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly. This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLAT
29.
by klauspost 4y ago
For MinIO (an S3 compatible server), we add an index for each part, which contains uncompressed -> compressed offset pairs. Since we already used a Snappy-derived method, each 1MB block is stored without backreferences. With this we only
30.
by klauspost 4y ago
It doesn't really have to impact performance. The index is generated easily as a side-effect of compression. And the index is only needed if you need to seek. I implemented this as part of the MinIO server. See "Seeking Compressed

More ›