5 ms·
I would think that git would need a parallel storage scheme for binaries. Something that does binary chunking and deduplication between revisions, but keeps the
by digikata 1y ago
I would think that git would need a parallel storage scheme for binaries. Something that does binary chunking and deduplication between revisions, but keeps the same merkle referencing scheme as everything else.
- zigzag312 1y agoXet uses block level deduplication.
- tempay 1y ago> binary chunking and deduplication Are there many binaries that people would store in git where this would actually help? I assume most files end up with compression or some other form of randomization between revisions making deduplication futile.
- hinkley 1y agoIt would likely require more tooling.
- digikata 1y agoI don't know, it's all probability in the dataset that makes one optimization strategy better over another. Git annex iirc does file level dedupe. That would take care of most of the problem if you're storing binaries that are compressed or encrypted. It's a lot of work to go beyond that, and probably one reason no one has bothered with git yet. But borg and restic both do chunked dedupe I think.
- adastra22 1y agoA lot in the game and visual art industries.
- zigzag312 1y ago2-3x reduction in repository size compared to Git LFS in this test: https://xethub.com/blog/benchmarking-the-modern-development-experience https://xethub.com/blog/benchmarking-the-modern-development-...