4 ms·

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g.

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly. Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.

amelius 16d ago
Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.
- Timon3 16d ago
  Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?
  - amelius 16d ago
    You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.
    - Timon3 15d ago
      Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox? Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.
      - amelius 15d ago
        Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?
        ecnahc515 15d ago
        Well uv runs on many systems and most don't have filesystem level dedupe configured, unfortunately. Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.
        Timon3 15d ago
        Because (going by scheme271's numbers) I might not want to waste lots of expensive RAM on something or mess around with offline deduplication, when a less resource-intensive alternative is entirely sufficient for whatever my use case is. Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.
nagaiaida 15d ago
> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level). there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.
- scheme271 14d ago
  Just curious but what would be a good way to design for it ahead of time?
  - nagaiaida 14d ago
    mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play. for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.

amelius 16d ago

Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.

Timon3 16d ago
Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?
- amelius 16d ago
  You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.
  - Timon3 15d ago
    Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox? Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.
    - amelius 15d ago
      Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?
      - ecnahc515 15d ago
        Well uv runs on many systems and most don't have filesystem level dedupe configured, unfortunately. Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.
      - Timon3 15d ago
        Because (going by scheme271's numbers) I might not want to waste lots of expensive RAM on something or mess around with offline deduplication, when a less resource-intensive alternative is entirely sufficient for whatever my use case is. Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.

nagaiaida 15d ago

> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level). there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.

scheme271 14d ago
Just curious but what would be a good way to design for it ahead of time?
- nagaiaida 14d ago
  mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play. for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.