6 ms·
Seems useful for s3 buckets. I suppose a script that watches for new objects, calculates its sha256 and stores that in a DB, then checks for duplicate hashes wo
by nodesocket 1mo ago
Seems useful for s3 buckets. I suppose a script that watches for new objects, calculates its sha256 and stores that in a DB, then checks for duplicate hashes would be a fairly trivial task. Though s3 doesn’t support symbolic links so accounting would need to be handled by server side code.
- dmarinus 1mo agoI don't think it's very useful but there's already a checksum in the meta data of an S3 object: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h... It might be better to keep an index of paths to checksums and use the checksum as object key in S3. I'm not sure what you like to achieve but if you use restic with S3 as a backend you can achieve much better deduplicate and compression.
- AceJohnny2 1mo agoWhile the idea of data-deduplication does apply everywhere, this specifically depends on the APFS CoW data feature, so it's not applicable outside of macOS (or APFS Volumes)
- rbanffy 1mo agoThe same concepts exist in ZFS, BtrFS and, IIRC, XFS.
- rf15 1mo agoalso, worst case you can do this with link features? Which is basically every system under the sun.
- msh 1mo agobut link is dangerous, as a edit made to one file will then be made to all copies. For example, If I have two files /home/m/todo.txt and /home/m/backup2025/todo.txt that are identical and you then replace /home/m/todo.txt with a link to /home/m/backup2025/todo.txt and I later edit /home/m/todo.txt then backup copy would also be updated. With apfs clones the backup would not be modified when I updated the file.
- rbanffy 1mo agoLink might be desirable for your e-book or mp3 collections. Or .iso images, MAME ROMs, and other files that are unlikely to be changed.
- AceJohnny2 1mo agoLinks are not CoW