6 ms·

TernFS – an exabyte scale, multi-region distributed filesystem

138 points by kirlev 11mo ago

cpach 11mo ago
Previously: https://news.ycombinator.com/item?id=45290245 https://news.ycombinator.com/item?id=45290245
semessier 11mo ago
should post again when having 5% of the features of the other parallel file systems starting with RDMA, whereby it's not clear if this FS does even stripe that is if it is even a parallel file system
poppafuze 11mo ago
Great default license.
- Joel_Mckay 11mo ago
  CephFS looks stable, and has diskprediction and Prometheus modules: https://docs.ceph.com/en/quincy/cephfs/index.html https://docs.ceph.com/en/quincy/cephfs/index.html https://github.com/ceph/ceph https://github.com/ceph/ceph Still not completely decoupled from host roles, but seems to work for some folks. =3
president_zippy 11mo ago
Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer scientists insist, we only need about 20Gb/s of throughput in either direction, so robustness and simplicity are the only concerns we have.
- mat_epice 11mo ago
  There are several other systems I would recommend before TernFS for your environment. If you're looking at Lustre versus this in particular, Lustre has been through the wringer, and ANL/DOE has plenty of people who understand it enough to run it well and fix it when it breaks. However, you are right. Your bandwidth needs don't really require Lustre.
  - president_zippy 11mo ago
    Seriously man, I'm asking because I don't know: which filesystems do you recommend instead? I dabbled in CephFS because our data is write-once, but helping computer illiterate research scientists at other universities and national labs retrieve their data is a lot simpler from Lustre because it's just plain-old POSIX filesystem semantics. I'm not joking, I didn't ask this as a way to namedrop my experience and credentials (common 'round this neck o' the woods), I honestly don't know what all the much more competent organizations are doing and would really like to find out.
    - huntaub 11mo ago
      I’d be happy to chat more about your needs and try to help recommend a path forward. Feel free to shoot me an email at the address in my profile.
      - x______________ 11mo ago
        Is this an ad? Why can't the topic continue here as a reply to op?
        Borg3 11mo ago
        Because its a consulting oportunity.
        nhanb 11mo ago
anon-3988 11mo ago
Isn't this literally what ZFS is designed for? What is ZFS lacking that this is needed.
- somat 11mo ago
  ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.
  - president_zippy 11mo ago
    Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZFS and their networking stack are separately. P.S. Glad someone pointed this out tactfully. A lot of people would have pounced on the chance to mock the poor commenter who just didn't know what he didn't know. The culture associated with software development falsely equates being opinionated with being knowledgeable, so hopefully we get a lot more people reducing the stigma of not knowing and reducing the stigma of saying "I don't know".
    - nh2 11mo ago
      Even if you were build a ZFS mega-machine with an Exabyte of storage with RDMA (the latencies of "normal" Ethernet in the datacenters would probably not be good enough), wouldn't you still have the problem that ZFS is fundamentally designed to be managed by and accessed on one machine? All data in and out of it would have to flow through that machine, which would be quite the bottleneck.
      - mgerdts 11mo ago
        If your entire system is connected via RDMA networks (rather common in HPC) I would not worry at all about latency. If you are buying NICs and switches that are capable of 100Gb or better, there’s a reasonable chance they support RoCE.
darkwater 11mo ago
To the OP: nice karma trick posting the URL with the anchor to bypass the HN duplicates detector. Dang & co, this is a bug, it should be fixed. I know because I stumbled on the same page following the links from the blog of the author of another post that made the frontpage yesterday (https://news.ycombinator.com/item?id=45589156 https://news.ycombinator.com/item?id=45589156), liked the TernFS concept, submitted it and got redirected to https://news.ycombinator.com/item?id=45290245 https://news.ycombinator.com/item?id=45290245
- dspillett 11mo ago
  > Dang & co, this is a bug, it should be fixed. Agreed, more or less, this would be easy to work around naively. Though an duplicate detection should not block reposts based on removing the anchor nor should the anchor portion automatically be removed generally. Some sites unnecessarily setup as SPAs use the anchor portion for context so it is needed for direct links to the right article in those¹, and going directly to a specific section in a long page can also be useful. > nice karma trick posting the URL with the anchor to bypass the HN duplicates detector. Karma jealously is as unbecoming as karma whoring, so perhaps wind the ol' neck in a little there. Laziness/ineptitude is more common than malice and this could have been accidental via quick copy+paste. A better way to respond in this situation is a more neutral “Already recently discussed at [link]”, as had been done some hours before your comment: https://news.ycombinator.com/item?id=45646691#45647047 https://news.ycombinator.com/item?id=45646691#45647047 ---- [1] Yes, those sites are badly designed, but they are unlikely to change because of our technical preferences and breaking the ability to deep link into them would add an issue for HN while not being noticed by those sites at all.
  - darkwater 11mo ago
    > Karma jealously is as unbecoming as karma whoring, so perhaps wind the ol' neck in a little there. Laziness/ineptitude is more common than malice and this could have been accidental via quick copy+paste. Agreed, and sorry for that (even though my gut and not-so-gut feeling is that it was done on purpose rather than by mistake, but I might be on the wrong side myself on this).
roadbuster 11mo ago
> all reads and writes go through the leader One of the pain points of scaling Zookeeper is that all writes must go to the leader (reads can be fulfilled by followers). I understand this is "leader of a shard" and not a "global leader," but it still means a skewed write load on a shard has to run through a single leader instance > given that horizontal scaling of metadata requires no rebalancing This means a skewed load cannot be addressed via horizontal scaling (provisioning additional shards). To their credit, they acknowledge this later in the (very well-written) article: > This design decision has downsides: TernFS assumes that the load will be > spread across the 256 logical shards naturally.

138 points by kirlev 11mo ago

cpach 11mo ago

Previously: https://news.ycombinator.com/item?id=45290245 https://news.ycombinator.com/item?id=45290245

should post again when having 5% of the features of the other parallel file systems starting with RDMA, whereby it's not clear if this FS does even stripe that is if it is even a parallel file system

poppafuze 11mo ago

Great default license.

Joel_Mckay 11mo ago
CephFS looks stable, and has diskprediction and Prometheus modules: https://docs.ceph.com/en/quincy/cephfs/index.html https://docs.ceph.com/en/quincy/cephfs/index.html https://github.com/ceph/ceph https://github.com/ceph/ceph Still not completely decoupled from host roles, but seems to work for some folks. =3

president_zippy 11mo ago

Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer scientists insist, we only need about 20Gb/s of throughput in either direction, so robustness and simplicity are the only concerns we have.

mat_epice 11mo ago
There are several other systems I would recommend before TernFS for your environment. If you're looking at Lustre versus this in particular, Lustre has been through the wringer, and ANL/DOE has plenty of people who understand it enough to run it well and fix it when it breaks. However, you are right. Your bandwidth needs don't really require Lustre.
- president_zippy 11mo ago
  Seriously man, I'm asking because I don't know: which filesystems do you recommend instead? I dabbled in CephFS because our data is write-once, but helping computer illiterate research scientists at other universities and national labs retrieve their data is a lot simpler from Lustre because it's just plain-old POSIX filesystem semantics. I'm not joking, I didn't ask this as a way to namedrop my experience and credentials (common 'round this neck o' the woods), I honestly don't know what all the much more competent organizations are doing and would really like to find out.
  - huntaub 11mo ago
    I’d be happy to chat more about your needs and try to help recommend a path forward. Feel free to shoot me an email at the address in my profile.
    - x______________ 11mo ago
      Is this an ad? Why can't the topic continue here as a reply to op?
      - Borg3 11mo ago
        Because its a consulting oportunity.
        nhanb 11mo ago

anon-3988 11mo ago

Isn't this literally what ZFS is designed for? What is ZFS lacking that this is needed.

somat 11mo ago
ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.
- president_zippy 11mo ago
  Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZFS and their networking stack are separately. P.S. Glad someone pointed this out tactfully. A lot of people would have pounced on the chance to mock the poor commenter who just didn't know what he didn't know. The culture associated with software development falsely equates being opinionated with being knowledgeable, so hopefully we get a lot more people reducing the stigma of not knowing and reducing the stigma of saying "I don't know".
  - nh2 11mo ago
    Even if you were build a ZFS mega-machine with an Exabyte of storage with RDMA (the latencies of "normal" Ethernet in the datacenters would probably not be good enough), wouldn't you still have the problem that ZFS is fundamentally designed to be managed by and accessed on one machine? All data in and out of it would have to flow through that machine, which would be quite the bottleneck.
    - mgerdts 11mo ago
      If your entire system is connected via RDMA networks (rather common in HPC) I would not worry at all about latency. If you are buying NICs and switches that are capable of 100Gb or better, there’s a reasonable chance they support RoCE.

darkwater 11mo ago

To the OP: nice karma trick posting the URL with the anchor to bypass the HN duplicates detector. Dang & co, this is a bug, it should be fixed. I know because I stumbled on the same page following the links from the blog of the author of another post that made the frontpage yesterday (https://news.ycombinator.com/item?id=45589156 https://news.ycombinator.com/item?id=45589156), liked the TernFS concept, submitted it and got redirected to https://news.ycombinator.com/item?id=45290245 https://news.ycombinator.com/item?id=45290245

dspillett 11mo ago
> Dang & co, this is a bug, it should be fixed. Agreed, more or less, this would be easy to work around naively. Though an duplicate detection should not block reposts based on removing the anchor nor should the anchor portion automatically be removed generally. Some sites unnecessarily setup as SPAs use the anchor portion for context so it is needed for direct links to the right article in those¹, and going directly to a specific section in a long page can also be useful. > nice karma trick posting the URL with the anchor to bypass the HN duplicates detector. Karma jealously is as unbecoming as karma whoring, so perhaps wind the ol' neck in a little there. Laziness/ineptitude is more common than malice and this could have been accidental via quick copy+paste. A better way to respond in this situation is a more neutral “Already recently discussed at [link]”, as had been done some hours before your comment: https://news.ycombinator.com/item?id=45646691#45647047 https://news.ycombinator.com/item?id=45646691#45647047 ---- [1] Yes, those sites are badly designed, but they are unlikely to change because of our technical preferences and breaking the ability to deep link into them would add an issue for HN while not being noticed by those sites at all.
- darkwater 11mo ago
  > Karma jealously is as unbecoming as karma whoring, so perhaps wind the ol' neck in a little there. Laziness/ineptitude is more common than malice and this could have been accidental via quick copy+paste. Agreed, and sorry for that (even though my gut and not-so-gut feeling is that it was done on purpose rather than by mistake, but I might be on the wrong side myself on this).

roadbuster 11mo ago

> all reads and writes go through the leader One of the pain points of scaling Zookeeper is that all writes must go to the leader (reads can be fulfilled by followers). I understand this is "leader of a shard" and not a "global leader," but it still means a skewed write load on a shard has to run through a single leader instance > given that horizontal scaling of metadata requires no rebalancing This means a skewed load cannot be addressed via horizontal scaling (provisioning additional shards). To their credit, they acknowledge this later in the (very well-written) article: > This design decision has downsides: TernFS assumes that the load will be > spread across the 256 logical shards naturally.