daviesliu

5 ms·

1.
by daviesliu 8mo ago
JuiceFS can be scaled to hundreds of PB by design, also is verified by thousands of users in production [1]. [1] https://juicefs.com/en/blog/company/2025-recap-artificial-in...
2.
by daviesliu 2y ago
Founder of JuiceFS here, congrats to the Launch! I'm super excited to see more people doing creative things in the using-S3-as-file-system space. When we started JuiceFS back in 2017, applied YC for 2 times but no luck. We are still wo
3.
Exposing Purple Face Olympic Swimmers [video](youtube.com)
1 points by daviesliu 2y ago|0 comments
4.
by daviesliu 3y ago
This post said another topic: how to backup the metadata of JuiceFS in readable format (JSON) and restore it into an empty database.
5.
by daviesliu 3y ago
The recover process is similar to what a database does after a crash, it loads the recent snapshot in disk and apply any newer transaction logs.
6.
by daviesliu 3y ago
The S3 API allow user to specify the hash of content as HTTP header, it will be verified by the JuiceFS gateway and persisted into JuiceFS as ETag. With POSIX API or HDFS, there is no such API to do that, unfortunately.
7.
by daviesliu 3y ago
JuiceFS relies on the object store to provide integrity for data. Besides that, JuiceFS stores the checksum of each object as tags in S3, and verifies that when downloading the objects. Inside the metadata service, it uses merkle tree (hash
8.
by daviesliu 3y ago
The company may want the product to be difficult to use, so they can sell more support, that's not win-win for both sides.
9.
by daviesliu 3y ago
Agreed, the all-in-one solution (Ceph) should be better, if you have to setup all the components. If you already have the infra (databases and object stores), then JuiceFS is the easiest solution to have a distributed file system.
10.
by daviesliu 3y ago
JuiceFS is similar to HDFS/CephFS/Lustre, so it MUST has a component to manage metadata, similar to NameNode of HDFS or MDS of CephFS, this point of failure is the problem we have to address. The underlying blob store systems is s
11.
JuiceFS 1.1: Easier Cloud Storage for Billions of Files(juicefs.com)
3 points by daviesliu 3y ago|0 comments
12.
by daviesliu 3y ago
If you really expect a file system experience over GCS, please try JuiceFS [1], which scales to 10 billions of files pretty well with TiKV or FoundationDB as meta engine. PS, I'm founder of JuiceFS. [1] https://github.com&#x
13.
by daviesliu 4y ago
Usually the meta engine or object storage can scale horizontally by itself, JuiceFS is middleware to talk to these two services. To serve S3 request, you can setup multiple S3 gateway and put a load director in front of them.
14.
by daviesliu 4y ago
Agreed, it's very hard, that's why GFS and HDFS had give up some parts of POSIX compatibility. Per CAP, it's addressed by different meta engines (CP system, Redis, MySQL, TiKV) and also different object stores (AP system). Wh
15.
by daviesliu 4y ago
Atomic file/directory renames/moves is the fundamental feature of JuiceFS, which makes it truely a file system rather than a proxy to S3, please check the docs for all the compatibility details [1]. https://github.com
16.
by daviesliu 4y ago
This is an experimental feature to do this, still working on it.
17.
by daviesliu 4y ago
It's doable to run a MinIO gateway on top of CephFS mount point, but that will has performance issue, especially for multipart-upload and copy. That's why we put MinIO and JuiceFS client together and use some internal API to do ze
18.
by daviesliu 4y ago
Yes, JuiceFS uses the Apache 2 fork [1] directly (master branch), but also provide a full featured S3 gateway (gateway branch) under AGPL for people' choice. [1] https://github.com/juicedata/minio/tree/ma
19.
by daviesliu 4y ago
Yes, the data can be encrypted [1] by the client before sending to S3, but the metadata is not encrypted. [1] https://juicefs.com/docs/community/security/encrypt
20.
by daviesliu 4y ago
The button to switch language is at the bottom of right-top menu, we will fix that.
21.
by daviesliu 4y ago
Apache Ozone is not POSIX compatible, even with the File System Optimized format [1]. https://ozone.apache.org/docs/current/feature/prefixfso.html
22.
by daviesliu 4y ago
99.99999999% reliability means you will not loss more than one byte in every 10 GB in a year. JuiceFS uses S3 as the underlying data storage, so S3 provides this durability SLA.
23.
by daviesliu 4y ago
Yes, we picked the default one in docs, which should be NFS v3, will redo the test against NFS v4 and update the article, thanks!
24.
by daviesliu 4y ago
Azure has Azure Files and Azrue NetApp Files, the later one is provided from NetApp. Azure Files was used in the article, maybe you are using NetApp Files? We will update the article to make it clear, thanks!
25.
by daviesliu 4y ago
Juicedata Inc is a US company, was registered in Delaware. The founding team are Chinese. ps, I'm the founder of Juicedata.
26.
by daviesliu 4y ago
We had compared nfs of Azure in the article, I believe cifs/samba should be much worse.
27.
by daviesliu 4y ago
Yes, JuiceFS is not a good choice for PG, unless if you don't care the performance. One interesting use case is the backup of MySQL [1]. [1] https://juicefs.com/docs/cloud/backup_mysql_in_juicefs/
28.
by daviesliu 4y ago
The JuiceFS Cloud supports ACL, but open source one does not support it yet.
29.
by daviesliu 4y ago
JuiceFS supports create-if-not-existed by using the Java SDK (HDFS compatible), so I guess it should work well with Delta.io.
30.
by daviesliu 4y ago
We have a fork of MinIO at https://github.com/juicedata/minio , which will be maintained by us.

More ›