felixhandte

5 ms·

1.
PivCo-Huffman: a novel approach to Huffman decoding(marcinzukowski.github.io)
2 points by felixhandte 3mo ago|0 comments
2.
by felixhandte 1y ago
SDDL (and the front-end task of reshaping data in general) is only one component of OpenZL. Once you have the streams, you can do all sorts of transformations to them that Zstd doesn't.
3.
by felixhandte 1y ago
The OpenZL core supports arbitrary composition of graphs. So you can do this now via the compressor construction APIs. We just have to figure out how to make it easy to do.
4.
by felixhandte 1y ago
Wanna hop over to https://github.com/facebook/openzl/issues/76 ?
5.
by felixhandte 1y ago
Update: let's continue discussing genomic sequence compression on https://github.com/facebook/openzl/issues/76 .
6.
by felixhandte 1y ago
Yes. None of the algorithms under test used any hardware acceleration in the benchmarks we ran.
7.
by felixhandte 1y ago
Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descriptions. I'll check them out.
8.
by felixhandte 1y ago
Yes, definitely! Chunking support is currently in development. Streaming and seeking and so on are features we will certainly pursue as we mature towards an eventual v1.0.0.
9.
by felixhandte 1y ago
Thanks @dang!
10.
by felixhandte 1y ago
It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock signific
11.
by felixhandte 1y ago
Let us know how it goes! We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people who, you know, didn't develop OpenZL. Your feedback
12.
by felixhandte 1y ago
In addition to open-sourcing the code today, we also published: - A white paper: https://arxiv.org/abs/2510.03203 - A blog post: https://engineering.fb.com/2025/10/06/developer-tools&#x2F
13.
OpenZL: A novel data compression framework(github.com)
33 points by felixhandte 1y ago|4 comments
14.
by felixhandte 1y ago
In addition to the blog post, here are the other things we've published today: Code: https://github.com/facebook/openzl Documentation: https://openzl.org/ White Paper: https://arxiv.org
15.
by felixhandte 1y ago
Zstd has a similar-ish capability called "repetition codes" [0]. The first stage of Zstd does LZ77 matching, which transforms the input into "sequences", a series of instructions each of which describes some literals and
16.
by felixhandte 1y ago
This is because Zstd's long-distance matcher looks for matching sequences of 64 bytes [0]. Because long matching sequences of the data will likely have the newlines inserted in different offsets in the run, this totally breaks Zstd&#x
17.
by felixhandte 2y ago
Awesome work! I get exactly that green cast and muted color range off of my flatbed scans (Epson v800). This is a really intriguing path to fixing them I hadn't considered. It seems like the writeup here doesn't specify what you&#
18.
by felixhandte 2y ago
That would only work in the case that the camera is fixed on a tripod and has a long period of stable / rigid pointing before the exposure during which to collect this data. This is sometimes the situation in which image stabilization
19.
by felixhandte 3y ago
Probably referring to this: https://futurism.com/the-byte/piece-falls-off-boeing-starlin...
20.
by felixhandte 3y ago
You're saying that a system that can recognize flaws in the alignment imposed on it can reject that alignment, but that doesn't follow. Sure, humans act against their own interests all the time. Sometimes we do so for considered r
21.
by felixhandte 3y ago
Sure, but Gödel encoding is pretty much purely a theoretical exercise. I'm not sure anyone anywhere has ever practically manipulated Gödel-encoded expressions in a useful way. His original scheme also has the problem that prime factori
22.
by felixhandte 4y ago
1. There are several implementations of Zstandard other than the "reference" libzstd that we maintain, tagged as "Ports" in the table here [0]. You are of course welcome to create or fork your own. We will happily take a
23.
by felixhandte 4y ago
Indeed--at this point zstd is the product of something like 25 man-years of improvements and optimizations.
24.
by felixhandte 4y ago
What differences do you have in mind? Zstd is largely compatible as a drop-in replacement for those commands. There are differences, but it's our goal to minimize them.
25.
by felixhandte 4y ago
There's also <>, which I (en-US) call "angle brackets".
26.
by felixhandte 4y ago
No widely deployed compressors are PAQ-like. The speed-up anticipated by the OP never materialized.
27.
by felixhandte 4y ago
Good write-up! I applaud your use of dictionary compression! We've done a lot of work to Zstd to make it a powerful tool that opens doors to new and better compression integrations. One correction: the "dictionary" in Zstd is
28.
by felixhandte 4y ago
They don't actually turn off though. It just turns the display off. They need to continuously sense position since digital calipers measure movement increments rather than absolute position. That's why if you take the battery out
29.
Zstandard v1.5.1(github.com)
5 points by felixhandte 5y ago|0 comments
30.
by felixhandte 5y ago
As someone on the Zstd team, I'm always happy to see it on HN! I'm curious though what motivates the submission?

More ›