6 ms·

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are simil

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks. There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!

a_t48 14d ago
I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.
Twirrim 14d ago
I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
- hinkley 14d ago
  Some compression libraries have an 'rsync compatibility mode', that plays some games with the block sizes to make it easier to rsync to not have to completely re-transmit a large compressed file because not the whole file changes every time. I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
colechristensen 14d ago
You may or may not be familiar with the content defined chunking family of algorithms like FastCDC. https://joshleeb.com/posts/chunking.html https://joshleeb.com/posts/chunking.html https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia https://www.usenix.org/conference/atc16/technical-sessions/p... On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
oefrha 14d ago
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.
- jmtulloss 14d ago
  I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
- edflsafoiewq 14d ago
  It was the second most reported problem on the 2025 Rust survey, after compilation times.
- tancop 14d ago
  Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big. It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help. The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
  - eviks 14d ago
    Don't they have nightly for imperfect experiments like this?
vbezhenar 13d ago
I did simpler trick with jars (basically zip archives). Java ecosystem loves huge directories full of jars. If one would just use good compressor like 7z over that directory, it won't compress that good. So I extracted every jar into a separate directory and then compressed them all with 7z. The results were very good. I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win. The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.

a_t48 14d ago

I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.

Twirrim 14d ago

I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.

hinkley 14d ago
Some compression libraries have an 'rsync compatibility mode', that plays some games with the block sizes to make it easier to rsync to not have to completely re-transmit a large compressed file because not the whole file changes every time. I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.

colechristensen 14d ago

You may or may not be familiar with the content defined chunking family of algorithms like FastCDC. https://joshleeb.com/posts/chunking.html https://joshleeb.com/posts/chunking.html https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia https://www.usenix.org/conference/atc16/technical-sessions/p... On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.

oefrha 14d ago

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.

jmtulloss 14d ago
I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
edflsafoiewq 14d ago
It was the second most reported problem on the 2025 Rust survey, after compilation times.
tancop 14d ago
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big. It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help. The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
- eviks 14d ago
  Don't they have nightly for imperfect experiments like this?

vbezhenar 13d ago

I did simpler trick with jars (basically zip archives). Java ecosystem loves huge directories full of jars. If one would just use good compressor like 7z over that directory, it won't compress that good. So I extracted every jar into a separate directory and then compressed them all with 7z. The results were very good. I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win. The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.