objectivefs

5 ms·

1.
by objectivefs 5mo ago
One advantage over S3FS would be that multiple filesystem mounts would see a consistent view of the filesystem, but it looks like this advantage disappears when mixing direct bucket access with filesystem mounts. Given the famously slow sma
2.
by objectivefs 1y ago
There is also ObjectiveFS that supports FUSE and uses S3 for both data and metadata storage, so there is no need to run any metadata nodes. Using S3 instead of a separate database also allows scaling both data and metadata with the performa
3.
by objectivefs 2y ago
Congratulations on your launch from ObjectiveFS! There is a lot of interest in 1-to-1 filesystems for mixed workloads, hope you can capture a nice share of that. Using NFS and being able to use an existing bucket is a nice way to make it ea
4.
by objectivefs 3y ago
For workloads with many small files, it usually is better to store many files in a single object. Filesystems with regular POSIX semantics, such as atomic directory renames etc, also makes it easier to integrate with existing software. We h
5.
by objectivefs 6y ago
ObjectiveFS takes a different approach to high-availability durable file system by using S3 as the storage backend and building a log structured file system on top. This leverages the durability and scalability of S3 while allowing e.g. git
6.
by objectivefs 6y ago
For a high performance shared file system on AWS, an alternative is ObjectiveFS[0]. It uses memory caching to achieve performance closer to local disk. [0]: https://objectivefs.com
7.
by objectivefs 7y ago
For TB-scale heavy read loads ObjectiveFS[0] is an alternative to EFS that don't have EFS's IO/bandwidth limits. It is often used for distributing data for machine learning and large scale simulations on EC2 and GCE. [0]: ht
8.
by objectivefs 8y ago
Another file system that works well with lots of small files and that doesn't require maintaining any extra servers is ObjectiveFS[0]. It is a FUSE based log structured file system (with snapshots) using GCS or S3 for log storage with
9.
by objectivefs 9y ago
We are looking into the best way to add native kubernetes support. Currently, you can add a mount on the host or directly mount the file system inside the container. Both approaches work well, so it mainly depends on your preferred architec
10.
by objectivefs 9y ago
Using a clustered/distributed filesystem definitively simplifies persisting the state between EC2 spot instances. It also makes it easier to scale out the work load when you need more instances accessing the same data. To add to your l
11.
by objectivefs 9y ago
When you have an object store available (Ceph, Cleversafe, S3, etc), and need a POSIX file system you can also use ObjectiveFS[1]. By using an existing object store for the backend storage, it separates the concerns of reliable storage from
12.
by objectivefs 10y ago
If you have access to an object store, you can get persistent file system storage for your containers using ObjectiveFS[0]. You can run it either on the hosts or inside the containers depending on your goal. [0]: https://objectiv
13.
by objectivefs 10y ago
Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache
14.
by objectivefs 10y ago
An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 you could try ObjectiveFS[1]. Using local SSD instance store and memory
15.
by objectivefs 10y ago
We have some notes on small file performance for EFS and ObjectiveFS at https://objectivefs.com/howto//performance-amazon-efs-vs-obj... that you might find interesting. Scaling out PHP on AWS is definitively easie
16.
by objectivefs 10y ago
If you need more performance than EFS for your shared filesystem storage, you could give our ObjectiveFS ( https://objectivefs.com ) a try. We see significantly higher read/write performance, especially for smaller files.
17.
by objectivefs 10y ago
Both GlusterFS and ObjectiveFS are shared filesystems instead of block storage, so they work with POSIX files/directories (like NFS) making them easier to scale out. If your application can work directly with an object store that is of
18.
by objectivefs 10y ago
One way to handle persistent container storage is to use a shared filesystem such as glusterfs or objectivefs. It's easy to set up and can be very helpful when moving your infrastructure to containers. Once created you can mount your s
19.
by objectivefs 10y ago
For stateful containers that need access to a shared folder you can also use a shared file system such as GlusterFS ( http://www.gluster.com ), EFS ( https://aws.amazon.com ), or ObjectiveFS ( https://objective
20.
by objectivefs 10y ago
Great that you have been wanting to use ObjectiveFS. We are happy to talk with you about your non-S3 storage use case and see if we can find a plan that works for you.
21.
by objectivefs 10y ago
Cool! If you are looking at shared filesystems on EC2, you might also want to take a look at ObjectiveFS[1]. Works in all regions and on GCS, and you can mount your filesystem securely (end-to-end encryption) between regions. [1]: https:&#
22.
by objectivefs 10y ago
ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers. [1] https://objectivefs.com
23.
by objectivefs 11y ago
If you are looking for a POSIX compatible file system for GCE or EC2, we think our ObjectiveFS[1] is the easiest way to get started and use. It is a log structured filesystem using GCS or S3 for backend storage and with a ZFS like interface
24.
by objectivefs 11y ago
You can use our ObjectiveFS[1] if you want a networked file system where multiple instances can simultaneously mount it read/write. It is backed by S3 and gives you a standard POSIX interface. [1] https://objectivefs.com
25.
by objectivefs 11y ago
ObjectiveFS does all the coordination among the clients through S3, so there is no extra coordination service needed. This is why we are really happy that Amazon recently moved the S3 us-east-1 region to read-after-write consistency (like a
26.
by objectivefs 11y ago
ObjectiveFS comes with fast compression that is always enabled, it works great for log files and other compressible data. Let us know what you think if you give it a try.
27.
by objectivefs 11y ago
It uses FUSE under the hood. Thanks for the feedback, we will make it clearer on the website.
28.
by objectivefs 11y ago
S3QL has a lot of great features. You are right that support for concurrent mounts is one of the main differences. ObjectiveFS also keeps the file system data consistent and stores it in S3 asap, for sharing and to handle EC2 unexpected ter
29.
by objectivefs 11y ago
We built a shared file system with an S3 backend. We've been running our own servers for many years, and wanted something low maintenance, reliable, scalable and secure. So, we built this POSIX-compliant shared filesystem for Linux and
30.
Show HN: ObjectiveFS – Shared file system for EC2 and laptop(objectivefs.com)
27 points by objectivefs 11y ago|21 comments

More ›