8 ms·
JuiceFS relies on the object store to provide integrity for data. Besides that, JuiceFS stores the checksum of each object as tags in S3, and verifies that when
by daviesliu 3y ago
JuiceFS relies on the object store to provide integrity for data. Besides that, JuiceFS stores the checksum of each object as tags in S3, and verifies that when downloading the objects.
Inside the metadata service, it uses merkle tree (hash of hash) to verify the integrity of whole namespace (including id of data blocks) between RAFT replicas. Once we store the hash (4 bytes) of each objects into metadata, it should provide the integrity of the whole namespace.
- amluto 3y agoDoes JuiceFS allow the user to specify the hash of a file when uploaded? And then to read that hash back later? Otherwise there’s no end-to-end integrity check.
- markhahn 3y agosurely you mean that the FS should calculate the hash on file creation/update, not take some random value from the user. but I agree that a FS that maintains file-content hash should allow clients to query it.
- amluto 3y agoNo, the FS should verify the hash on creation/update. Otherwise corruption during creation/update would just cause the hash to match the corrupted data.
- deleted 3y ago[deleted]
- daviesliu 3y agoThe S3 API allow user to specify the hash of content as HTTP header, it will be verified by the JuiceFS gateway and persisted into JuiceFS as ETag. With POSIX API or HDFS, there is no such API to do that, unfortunately.