terrelln

6 ms·

1.
by terrelln 3mo ago
Fusion is so useful though. It is easy to set up on ships because it requires no consumables other than the fuel cells, and it has a small footprint. It is easy to set up power on all planets by stamping down a blueprint, and requires minim
2.
by terrelln 4mo ago
I ran into a fun crash a year or so ago in the interaction of clang’s profile guided speculative devirtualization and identical code folding (ICF) done by BOLT on the binary. Clang relied on checking the address of a function pointer in the
3.
by terrelln 6mo ago
> Also making good progress on getting a slimmer version of zstd into the stdlib Awesome! Please let me know if there is anything I can do to help
4.
by terrelln 8mo ago
Yeah, it isn't quite that simple. E.g. `/bin/ksh` reports 1.4MB, but it is actually 2.4MB. Initially, I thought it was because the file was sparse, but there are only 493KB of zeros. So something else is going on. Perhaps som
5.
by terrelln 8mo ago
I've figured out the issue. Use `wc -c` instead of `du`. I can repro on my Mac with these steps with either `zstd` or `gzip`: $ rm -f ksh.zst $ zstd < /bin/ksh > ksh.zst $ du -h ksh.zst 1.2M ksh.zst
6.
by terrelln 8mo ago
Ah I understand. In this benchmark, Zstd's decompression time is 284 MB/s, and Gzip's is 330 MB/s. This benchmark is likely dominated by file IO for the faster decompressors. On the incompressible files, I'd expect
7.
by terrelln 8mo ago
Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of data, zstd checks that the output size is < 128
8.
by terrelln 8mo ago
Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This seems like a bug, or somehow measuring the wrong thing, and not
9.
by terrelln 8mo ago
> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any co
10.
by terrelln 11mo ago
Its slang that has made its way from queer culture into mainstream. It is not meant to satisfy anyones ego, other than it just has vague positive connotations. It is used in a pretty equivalent way as "take that money, dude".
11.
by terrelln 11mo ago
Out of curiosity, what was the input file format? We actually worked on a demo WAV compressor a while back. We are currently missing codecs to run the types of predictors that FLAC runs. We expect to add this kind of functionality in the fu
12.
by terrelln 1y ago
Amazing, thank you! I will take a look as soon as I get a chance. Looking at the BAM format, it looks like the tokenization portion will be easy. Which means I can focus on the compression side, which is more interesting.
13.
by terrelln 1y ago
You could have an LLM generate the SDDL description [0] for you, or even have it write a C++ or Python tokenizer. If compression succeeds, then it is guaranteed to round trip, as the LLM-generated logic lives only on the compression side, a
14.
by terrelln 1y ago
The charts in the "Results With OpenZL" section compare against all levels of zstd, xz, and zlib. On highly structured data where OpenZL is able to understand the format, it blows Zstandard and Xz out of the water. However, not al
15.
by terrelln 1y ago
We left it out of the paper because it is an implementation detail that is absolutely going to change as we evolve the format. This is the function that actually does it [0], but there really isn't anything special here. There are some
16.
by terrelln 1y ago
Do you happen to have a pointer to a good open source dataset to look at? Naively and knowing little about CRAM, I would expect that OpenZL would beat Zstd handily out of the box, but need additional capabilities to match the performance of
17.
by terrelln 1y ago
You'd have to tell OpenZL what your format looks like by writing a tokenizer for it, and annotating which parts are which. We aim to make this easier with SDDL [0], but today is not powerful enough to parse JSON. However, you can do th
18.
by terrelln 1y ago
> Is openzl indexable Not today. However, we are considering this as we are continuing to evolve the frame format, and it is likely we will add this feature in the future.
19.
by terrelln 1y ago
Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in C++ or Python to parse your format. And this code is compres
20.
by terrelln 1y ago
There's a Quick Start guide here: https://openzl.org/getting-started/quick-start/ However, OpenZL is different in that you need to tell the compressor how to compress your data. The CLI tool has a few builtin
21.
by terrelln 1y ago
Yeah, backend compression in columnar data formats is a natural fit for OpenZL. Knowing the data it is compressing is numeric, e.g. a column of i64 or float, allows for immediate wins over Zstandard.
22.
OpenZL: An open source format-aware compression framework(engineering.fb.com)
434 points by terrelln 1y ago|107 comments
23.
by terrelln 2y ago
You can also use general purpose compressors like Zstandard to create a generic patch: zstd --patch-from old.json new.json -o patch.zst zstd --patch-from old.json -d patch.zst -o updated.json
24.
by terrelln 2y ago
Meta drove the Zstandard content encoding, but Google drove the adoption of Zstandard in Chrome. The faster Brotli levels could probably be made to match Zstandard’s compression speed. But we’ve invested a lot in optimizing these levels, so
25.
by terrelln 2y ago
The latest Zstandard exposes several parameters which are useful for reducing time to first byte latency in web compression. They make Zstandard cut the compressed data into smaller blocks, e.g. 4 KB, with the goal to fit a compressed block
26.
by terrelln 2y ago
I believe that the "more private" part is referencing the "Encrypted Client Hello (ECH)" section in the later part of the post.
27.
New standards for a faster and more private Internet(blog.cloudflare.com)
170 points by terrelln 2y ago|84 comments
28.
by terrelln 2y ago
Variable-length integer schemes are great when you are encoding one integer. But when you are encoding a list of integers, you should really consider a different scheme. Variable-length integer schemes generally interleave the control bits
29.
by terrelln 2y ago
I agree, this is a very exciting direction. We shouldn’t let existing formats stifle innovation, but there is a lot of value in back porting modern techniques to existing encoders.
30.
by terrelln 3y ago
I went through this course online a few summers ago, and learned a ton. I highly recommend it! It was very engaging to put up PRs for small issues in the Bril IR, and work with the professor to fix them.

More ›