6 ms·
One downside to encryption, is it prevents the server operator from doing any deduplication (file or block level) on their end. Maybe one reason why cloud prov
by eemil 2y ago
One downside to encryption, is it prevents the server operator from doing any deduplication (file or block level) on their end.
Maybe one reason why cloud providers aren't pushing it that heavily. Especially the big players, since more data = more duplication = more efficient deduplication.
- idle_zealot 2y agoIs that true? Couldn't you run dedupe on blocks of encrypted files? I assume there would be fewer duplicate blocks compared to the cleartext, but if you have a bunch of blocks full of random bits there are bound to be repeats with a large enough number of blocks.
- RcouF1uZ4gsC 2y agoEven 32 bytes of random data has an astronomically low chance to ever have a collision.
- hansvm 2y agoIf you can, you've effectively broken the encryption. Any scheme that takes random data and stores it in less space, when accounting for the overhead of the scheme itself, is astronomically unlikely to succeed by more than a few bits saved in any specific example (and on average across all such random streams cannot save space at all).
- mprime1 2y agoIndeed. Borg for example is e2e but able to dedupe. My bookmark archive is 10TB but deduped on-disk size is 100GB because most files are the same across backups! https://www.borgbackup.org/ https://www.borgbackup.org/
- orf 2y agoThat’s not the same thing at all.
- mprime1 2y agoSame thing as what? Parent was asking about deduping encrypted data. Someone said (wrongly) it’s impossible and I shared a popular project that does exactly that.
- orf 2y ago“Not backing up the same file twice” is not the same thing as deduplicating encrypted data, as encryption has no relevance there. You can do that with or without encryption.
- tjpnz 2y agoDouble edged sword. Mega Upload were doing it and it was argued (successfully) in court that they therefore had knowledge of what they were hosting.
- willis936 2y agoThat's fine. We pay for storage. I'll pay extra to not have the host spy, sell, etc. my data. Deduplication only really shines if most data is pirated copy data. In reality the vast majority of data is in fine details of high resolution photos and videos of completely uncorrelated images.