6 ms·

ZeroFS vs. Amazon S3 Files

92 points by cbrewster 2mo ago

Lucasoato 2mo ago
The article diagrams can’t be seen well if the device has the dark mode as default, just a suggestion for the author. Opened with Safari in iOS
- ranger_danger 2mo ago
  I only saw one diagram but it is perfectly visible for me on the desktop.
- smartbit 2mo ago
  I have the same in Brave on iOS: to see the the diagrams I need to turn of Night Mode.
- bayesnet 2mo ago
  The page appears to be dark themed even without dark mode enabled; if I use Noir to force-add dark mode, then the diagrams indeed become difficult to read but this seems like a Noir issue more than a site issue
pierrebarre 2mo ago
Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.
- nubg 2mo ago
  This blog post seems to have been written by an AI based on a prompt you gave it? Can you confirm or deny this?
- chillfox 2mo ago
  Thanks for building this, I am just about to give it ago with my self-hosted Garage cluster. Does running `stat` against a file require pulling the whole file from s3, or can that be handled by the metadata? Do you know what backup performance is like for something like borg/borgmatic or restic, especially on follow up runs where most files are just checked. Is there any particular Redis/Valkey config you recommend when using it for `conditional_put`, or just default config? Is there any chance for NFSv4 support?
  - pierrebarre 2mo ago
    Thanks! stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached. I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata. The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.
- tandr 2mo ago
  I'm thinking about using this and have a few questions: 1. How are hardlinks and duplicate files (same content, different paths) handled? 2. Does deduplication work on a block/chunk level for partially matching files, or does it only look at whole files? 3. Is there any specific integration or handling for Copy-on-Write (CoW)? Thank you!
  - pierrebarre 2mo ago
    Hardlinks work as you’d expect: multiple paths point to the same inode and data, so a write through one path is visible through the others. Two separate files with the same contents are stored separately. There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality. If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.
    - tandr 2mo ago
      Thank you for the quick answer.
- the_duke 2mo ago
  I was investigating the design a little. Two big questions: A) You notably don't write a recovery log (WAL/journal) for things not yet flushed, so data can be lost. Do you have plans to add this? I think it would be pretty crucial. B) the system is single writer. Do you have plans for adding horizontal scalability so a writer can be dynamically selected and routed to, transparent to the client? (Or with client cooperation, but without forcing sharding on the user)
  - pierrebarre 2mo ago
    [dead]
- selfhoster1312 2mo ago
  I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqlite durability (so NFS is out of the question) - inotify support for external script integration (did not find an issue/docs about this) - support for reading/writing small blocks without hammering the CPU/disks (benchmarked with JuiceFS using torrents and was catastrophic) Do you think ZeroFS currently makes a good candidate? If not, is the described usecase part of the ZeroFS roadmap? (1) https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
  - snapplebobapple 2mo ago
    Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.
    - selfhoster1312 2mo ago
      garage was working fine in other scenarios but it really had pathological behavior with the small read/write with torrenting benchmarks. (see my previous link for detailed benchmarking) The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend to extend available space, and potentially removing the need for backups for most of the non-critical data once it's replicated across different sites. I never got NFS/SMB to saturate a Gbit/s LAN link, and other solutions like cephfs/glusterfs seemed more complex and did not necessarily have all the features (inotify/fsync). I never tried rustfs but was particularly interested in garage for the "philosophy" of development: developed by academics with proper R&D, fully FLOSS, with low-end hardware and unstable links/sites in mind, aiming for technological degrowth. From the looks at the rustfs website, it looks like the opposite of that. You are right i was trying to overengineer things. For now i still run RAID0 (+ backups for important data).
- tony_francis 2mo ago
  How does the performance compare to juicefs?
hhthrowaway1230 2mo ago
I really want this, but it feels a little scary to trust all my files. I wouls like it if there was some contineous suite trying to corrupt the files and then see the failure cases!
- pierrebarre 2mo ago
  That’s a fair concern. The closest thing right now is a deterministic simulation suite that injects storage faults and crashes at arbitrary points, then checks the recovered data against reference models. It runs hourly with fresh seeds. CI also runs pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen crash/failover tests: https://github.com/Barre/ZeroFS#testing https://github.com/Barre/ZeroFS#testing ZeroFS is still pretty young as storage projects go, so I completely understand wanting to see it prove itself over time.
  - ahofmann 2mo ago
    Why the hell was this answer dead? I vouched for it, because I don't see what might be wrong with it.
    - the_duke 2mo ago
      No clue, I also vouched.
- deleted 2mo ago
  [deleted]
hhthrowaway1230 2mo ago
Also would love to know how was we could iterate over the files. ie a bit how duckdb allows for paruet fikes scanning and reading would be nice to see how fast we can query the fs for ai/ml training workloads
deleted 2mo ago
[deleted]
sieabahlpark 2mo ago
[dead]
jijji 2mo ago
s3 is expensive... there are a lot of cheap options. I think I pay $48/month for a linux vps with 8 cpus and 16tb of storage with interserver.net... the same storage on amazon s3 is $377/month lol
- Onavo 2mo ago
  Their bandwidth costs are overpriced yes but in terms of storage you are paying for RAID3 replicated across three data centers (configurable) and high availability. The engineering behind it is also formally verified with strict theoretical bounds on data loss. $48*3 = ~$150 is within the same ballpark if you factor in the managed services and cloud overhead.
deknos 2mo ago
how does encryption work with zerofs?
vasanthps 2mo ago
how does it compare with cloudflare r2

92 points by cbrewster 2mo ago

The article diagrams can’t be seen well if the device has the dark mode as default, just a suggestion for the author. Opened with Safari in iOS

ranger_danger 2mo ago
I only saw one diagram but it is perfectly visible for me on the desktop.
smartbit 2mo ago
I have the same in Brave on iOS: to see the the diagrams I need to turn of Night Mode.
bayesnet 2mo ago
The page appears to be dark themed even without dark mode enabled; if I use Noir to force-add dark mode, then the diagrams indeed become difficult to read but this seems like a Noir issue more than a site issue

pierrebarre 2mo ago

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

nubg 2mo ago
This blog post seems to have been written by an AI based on a prompt you gave it? Can you confirm or deny this?
chillfox 2mo ago
Thanks for building this, I am just about to give it ago with my self-hosted Garage cluster. Does running `stat` against a file require pulling the whole file from s3, or can that be handled by the metadata? Do you know what backup performance is like for something like borg/borgmatic or restic, especially on follow up runs where most files are just checked. Is there any particular Redis/Valkey config you recommend when using it for `conditional_put`, or just default config? Is there any chance for NFSv4 support?
- pierrebarre 2mo ago
  Thanks! stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached. I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata. The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.
tandr 2mo ago
I'm thinking about using this and have a few questions: 1. How are hardlinks and duplicate files (same content, different paths) handled? 2. Does deduplication work on a block/chunk level for partially matching files, or does it only look at whole files? 3. Is there any specific integration or handling for Copy-on-Write (CoW)? Thank you!
- pierrebarre 2mo ago
  Hardlinks work as you’d expect: multiple paths point to the same inode and data, so a write through one path is visible through the others. Two separate files with the same contents are stored separately. There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality. If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.
  - tandr 2mo ago
    Thank you for the quick answer.
the_duke 2mo ago
I was investigating the design a little. Two big questions: A) You notably don't write a recovery log (WAL/journal) for things not yet flushed, so data can be lost. Do you have plans to add this? I think it would be pretty crucial. B) the system is single writer. Do you have plans for adding horizontal scalability so a writer can be dynamically selected and routed to, transparent to the client? (Or with client cooperation, but without forcing sharding on the user)
- pierrebarre 2mo ago
  [dead]
selfhoster1312 2mo ago
I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqlite durability (so NFS is out of the question) - inotify support for external script integration (did not find an issue/docs about this) - support for reading/writing small blocks without hammering the CPU/disks (benchmarked with JuiceFS using torrents and was catastrophic) Do you think ZeroFS currently makes a good candidate? If not, is the described usecase part of the ZeroFS roadmap? (1) https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
- snapplebobapple 2mo ago
  Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.
  - selfhoster1312 2mo ago
    garage was working fine in other scenarios but it really had pathological behavior with the small read/write with torrenting benchmarks. (see my previous link for detailed benchmarking) The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend to extend available space, and potentially removing the need for backups for most of the non-critical data once it's replicated across different sites. I never got NFS/SMB to saturate a Gbit/s LAN link, and other solutions like cephfs/glusterfs seemed more complex and did not necessarily have all the features (inotify/fsync). I never tried rustfs but was particularly interested in garage for the "philosophy" of development: developed by academics with proper R&D, fully FLOSS, with low-end hardware and unstable links/sites in mind, aiming for technological degrowth. From the looks at the rustfs website, it looks like the opposite of that. You are right i was trying to overengineer things. For now i still run RAID0 (+ backups for important data).
tony_francis 2mo ago
How does the performance compare to juicefs?

hhthrowaway1230 2mo ago

I really want this, but it feels a little scary to trust all my files. I wouls like it if there was some contineous suite trying to corrupt the files and then see the failure cases!

pierrebarre 2mo ago
That’s a fair concern. The closest thing right now is a deterministic simulation suite that injects storage faults and crashes at arbitrary points, then checks the recovered data against reference models. It runs hourly with fresh seeds. CI also runs pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen crash/failover tests: https://github.com/Barre/ZeroFS#testing https://github.com/Barre/ZeroFS#testing ZeroFS is still pretty young as storage projects go, so I completely understand wanting to see it prove itself over time.
- ahofmann 2mo ago
  Why the hell was this answer dead? I vouched for it, because I don't see what might be wrong with it.
  - the_duke 2mo ago
    No clue, I also vouched.
deleted 2mo ago
[deleted]

hhthrowaway1230 2mo ago

Also would love to know how was we could iterate over the files. ie a bit how duckdb allows for paruet fikes scanning and reading would be nice to see how fast we can query the fs for ai/ml training workloads

deleted 2mo ago

[deleted]

sieabahlpark 2mo ago

[dead]

jijji 2mo ago

s3 is expensive... there are a lot of cheap options. I think I pay $48/month for a linux vps with 8 cpus and 16tb of storage with interserver.net... the same storage on amazon s3 is $377/month lol

Onavo 2mo ago
Their bandwidth costs are overpriced yes but in terms of storage you are paying for RAID3 replicated across three data centers (configurable) and high availability. The engineering behind it is also formally verified with strict theoretical bounds on data loss. $48*3 = ~$150 is within the same ballpark if you factor in the managed services and cloud overhead.

deknos 2mo ago

how does encryption work with zerofs?

vasanthps 2mo ago

how does it compare with cloudflare r2