6 ms·

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find

by cuno 3y ago

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2. Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") you get massive speedups.

crabbone 3y ago
> find it faster than a local filesystem for many benchmarks. What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work... Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have? ---- The thing is: you should expect, in the best case, something like 5 ms latency on network calls over the Internet in an ideal case. Within the datacenter, maybe you can achieve sub-ms latency, but that's hard. AWS within region but different zones tends to be around 1 ms latency. This is while NVMe latency, even on consumer products, is 10-20 micro seconds. I.e. we are talking about roughly 100 times faster than anything going through the network can offer.
- cuno 3y ago
  For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-cost-performance-of-different-efs-options-and-alternatives/ https://cuno.io/blog/making-the-right-choice-comparing-the-c...
  - hnlmorg 3y ago
    EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.
    - dekhn 3y ago
      if you turn all the EFS performance knobs up (at a high cost), it's quite fast.
      - hnlmorg 3y ago
        Faster, sure. But I wouldn’t got so far as to say it is fast
    - harshaw 3y ago
      Have you tried it recently? Because we've made it a lot faster over the years.
      - hnlmorg 3y ago
        More recently and for more use cases and varied workflows than most people. But that’s as much as I can say without getting people to sign an NDA. Our AWS spend is high enough to warrant a very close working relationship with AWS so this is something we have worked with you guys on already.
    - geertj 3y ago
      > EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS. Would you care to elaborate on your experience or use case a bit more? We've made a lot of improvements over the last few years (and are actively working on more), and we have many happy customers. I'd be happy to give a perspective of how well your use case would work with EFS. Source: PMT turned engineer on EFS, with the team for over 6 years
      - 3y ago
supriyo-biswas 3y ago
> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.
- cuno 3y ago
  For example, you can do separate parallel ListObjectV2 for files starting a-f and g-k, etc.. covering the whole key space. You can parallelize recursively based on what is found in the first 1000 entries so that it matches the statistics of the keys. Yes there may be pathological cases, but in practice we find this works very well.
- radlad 3y ago
  You're right that it won't work for all use cases, but starting two threads with prefixes A and M, for example, is one way you might achieve this.
fijiaarone 3y ago
If you think s3 is fast, you should try FTP. It’s at least a hundred times faster. And combined with rsync, dozens of times more reliable.
- orf 3y ago
  Neither of those are true though? Not sure if this is sarcastic or not, if so make it more clear in the future
YZF 3y ago
It can't be a POSIX filesystem if it doesn't meet POSIX filesystem guarantees. I worked on an S3 compatible object store in a large storage company and we also had distributed filesystem products. Those are completely different animals due to the different semantics and requirements. We've also built compliant filesystems over object store and the other way around. Certain operations like, write-append, are tricky to simulate over object stores (S3 didn't use to support append, I haven't really stayed up to date, does it now?). At least when I worked on this it wasn't possible to simulate POSIX semantics over S3 at all without needing to add additional object store primitives.

by cuno 3y ago

crabbone 3y ago

> find it faster than a local filesystem for many benchmarks. What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work... Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have? ---- The thing is: you should expect, in the best case, something like 5 ms latency on network calls over the Internet in an ideal case. Within the datacenter, maybe you can achieve sub-ms latency, but that's hard. AWS within region but different zones tends to be around 1 ms latency. This is while NVMe latency, even on consumer products, is 10-20 micro seconds. I.e. we are talking about roughly 100 times faster than anything going through the network can offer.

cuno 3y ago
For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/ https://cuno.io/technology/ and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-cost-performance-of-different-efs-options-and-alternatives/ https://cuno.io/blog/making-the-right-choice-comparing-the-c...
- hnlmorg 3y ago
  EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.
  - dekhn 3y ago
    if you turn all the EFS performance knobs up (at a high cost), it's quite fast.
    - hnlmorg 3y ago
      Faster, sure. But I wouldn’t got so far as to say it is fast
  - harshaw 3y ago
    Have you tried it recently? Because we've made it a lot faster over the years.
    - hnlmorg 3y ago
      More recently and for more use cases and varied workflows than most people. But that’s as much as I can say without getting people to sign an NDA. Our AWS spend is high enough to warrant a very close working relationship with AWS so this is something we have worked with you guys on already.
  - geertj 3y ago
    > EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS. Would you care to elaborate on your experience or use case a bit more? We've made a lot of improvements over the last few years (and are actively working on more), and we have many happy customers. I'd be happy to give a perspective of how well your use case would work with EFS. Source: PMT turned engineer on EFS, with the team for over 6 years
    - 3y ago

supriyo-biswas 3y ago

> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.

cuno 3y ago
For example, you can do separate parallel ListObjectV2 for files starting a-f and g-k, etc.. covering the whole key space. You can parallelize recursively based on what is found in the first 1000 entries so that it matches the statistics of the keys. Yes there may be pathological cases, but in practice we find this works very well.
radlad 3y ago
You're right that it won't work for all use cases, but starting two threads with prefixes A and M, for example, is one way you might achieve this.

fijiaarone 3y ago

If you think s3 is fast, you should try FTP. It’s at least a hundred times faster. And combined with rsync, dozens of times more reliable.

orf 3y ago
Neither of those are true though? Not sure if this is sarcastic or not, if so make it more clear in the future

YZF 3y ago

It can't be a POSIX filesystem if it doesn't meet POSIX filesystem guarantees. I worked on an S3 compatible object store in a large storage company and we also had distributed filesystem products. Those are completely different animals due to the different semantics and requirements. We've also built compliant filesystems over object store and the other way around. Certain operations like, write-append, are tricky to simulate over object stores (S3 didn't use to support append, I haven't really stayed up to date, does it now?). At least when I worked on this it wasn't possible to simulate POSIX semantics over S3 at all without needing to add additional object store primitives.