7 ms·
Google Cloud Storage FUSE
- rippercushions 3y agoIs this the same gcsfuse that's been around for years, only now with official Google support? https://github.com/GoogleCloudPlatform/gcsfuse https://github.com/GoogleCloudPlatform/gcsfuse
- beastman82 3y agoyes > export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
- scoobydoobydrew 3y agoLooking at change descriptions, looks like underlying changes were made to get to this like now using GO client library. I would expect a more stable product, and better performance which looks like the performance benchmarks located under docs has been updated as well. Happy to finally see Google standing behind this, and the official CSI driver is really cool to see.
- throwdbaaway 3y agoHeh, my old laptop has a git clone of this from September 1st 2016.
- trollied 3y agoBe aware that this is not free: "Cloud Storage FUSE is available free of charge, but the storage, metadata, and network I/O it generates to and from Cloud Storage are charged like any other Cloud Storage interface. In other words, all data transfer and operations performed by Cloud Storage FUSE map to Cloud Storage transfers and operations, and are charged accordingly."
- rippercushions 3y agoUsing FUSE doesn't cost you anything extra, but it doesn't make the underlying storage free.
- ElectricalUnion 3y agoYou will be doing storage operations silently and in a unoptimized fashion, more so if the underlying FUSE filesystem is implemented in a naive fashion. For example, Cloud Storage never moves or renames your objects; copying and deleting the original one instead. This can end up costing quite a lot if you're using data other that in "standard store" because of minimum storage duration.
- jefftk 3y agoCloud Storage FUSE does not support overwriting in the middle of a file. Only sequential writes are supported. This seems like a big limitation?
- throwawaaarrgh 3y agopretty standard limitation of object storage services iirc
- jefftk 3y agoDoesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?
- jsnell 3y agoI'd think non-appending writes are quite rare in practice, other than databases. Even when the application is logically overwriting data, in other kinds of programs it's almost always implemented as writing to a new file + an atomic rename, not in-place modification.
- dontlaugh 3y agoEven databases tend to do sequential writes, whether to a WAL or LSM tree.
- hawski 3y agoMost programs either write a full file every time and replace the old file by a single move or append to an old file. Writting in the middle could happen in a program writting to some kind of archive or disk image. There is probably a whole group of programs that do this I'm not familiar with, but I'm pretty sure of my first sentence.
- 3y ago
- MontyCarloHall 3y agoI’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesystem, this can be mitigated somewhat by the disk cache, but for applications that would thrash the cache, cloud buckets are simply not a good storage backend if you desire disk-like access. What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts.
- ashishbijlani 3y agoI'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready. 1. https://github.com/extfuse/extfuse https://github.com/extfuse/extfuse
- vamega 3y agoIs work on this continuing (or restarting)? I had heard of this a few years ago, but thought the project was shelved.
- ashishbijlani 3y agoThe project is active (just not merged in the kernel yet). Please DM me for questions.
- bitL 3y agoMoreover, there is no SLA on those FUSE adapters so putting it into any part of production is too risky.
- qsort 3y agoMy personal conspiracy theory: most "cloud services" are just... bad. VMs and disk space I understand completely, having machines on-prem is too much of an hassle and the price isn't that bad. But for stuff like this, managed services, databases especially, you're just getting scammed.
- askvictor 3y agoNow for official Google Drive support on Linux...
- curt15 3y agoHow do Googlers access Google Drive from their Linux workstations? Do they have an internal GDrive client?
- martius 3y agoNot that I know of, we have some virtual filesystems for specific things, but in general Drive is for shared docs, videos (recorded meetings/presentations) and things like this. We don't use drive to store other files. Actually, we don't really "store files" since almost everything we need is remote. See for instance this discussion: https://news.ycombinator.com/item?id=13561096 https://news.ycombinator.com/item?id=13561096
- plaidfuji 3y agoI wonder the same, but I also wonder what the actual use case is for the Drive app on Linux. For me, Drive is mostly for syncing office docs (namely MS-office docs), PDFs and images among teams. That type of work doesn’t lend itself well to a Linux env anyway. And for programming-heavy sync tasks, a user will more likely use a remote Git repo for code and GCS for data. Does google even use MS office internally?
- capableweb 3y ago> Does google even use MS office internally That'd be weird, considering they have their own suite of office tools. Kind of like if Microsoft would be using Google Cloud rather than Azure internally.
- olyjohn 3y agoKinda like if Microsoft ran Linux on their cloud...
- manigandham 3y agoObject storage is a higher-level abstraction than block-storage. FUSE and similar tech can do the job for basic requirements like read-only access by legacy applications but rarely works well for other scenarios. A more complex layer like https://objectivefs.com/ https://objectivefs.com/ (based on the S3 API) would be more useful, although I would've expected the cloud providers to scale their own block-store/SANs backed with object-stores by now.
- goodpoint 3y agoFUSE is really not suitable for this.
- ISL 3y agoCan this be used to mount Drive under linux?
- hobo_mark 3y agoGCS is not Drive, it's Google's equivalent of S3.
- lern_too_spel 3y agoUse rclone.
- speedster217 3y agohttps://github.com/astrada/google-drive-ocamlfuse https://github.com/astrada/google-drive-ocamlfuse is one option
- remram 3y agoSee also: JuiceFS: https://juicefs.com/ https://juicefs.com/ Adds a DBMS or key-value store for metadata, making the filesystem much faster (POSIX, small overwrites don't have to replace a full object in the GCS/S3 backend). Almost certainly a better solution if you want to turn your object storage into a mountable filesystem, with the (big) caveat that you can't access the files directly in the bucket (they are not stored transparently).
- e1g 3y agoJuiceFS is mostly POSIX compatible, but there are important caveats such as no extended ACL, copying files changes their mtime (impacts backup tools), it offers "close-to-open" consistency (dangerous for log appenders), etc. Choosing an appropriate solution in this space still depends on what you need to do with the storage, and a few other options are MooseFS (https://github.com/moosefs/moosefs https://github.com/moosefs/moosefs) SeaweedFS (https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs) Curve (https://github.com/opencurve/curve https://github.com/opencurve/curve) GeeseFS (https://github.com/yandex-cloud/geesefs https://github.com/yandex-cloud/geesefs)
- remram 3y agoThose other ones can't use GCS though, which is why I pointed to JuiceFS (at least the ones I recognize).
- jijji 3y agoI've been using rclone [0] to do the same under linux for years, how is this different? [0] https://rclone.org https://rclone.org
- dallbee 3y agoUnfortunately it's common to have a policy in place disallowing 3rd-party app api access to drive storage. This prevents apps like rclone from working, but the drive client works because it isn't 3rd-party.
- iamjk 3y agoI mean I get why everyone wants everything to be fuse-compatible but some things just aren't meant to be done.
- ggambetta 3y ago"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should!"
- nickcw 3y agoAs the author of rclone I thought I'd have a quick look through the docs to see what this is about. From reading the docs, it looks very similar to `rclone mount` with `--vfs-cache-mode off` (the default). The limitations are almost identical. * Metadata: Cloud Storage FUSE does not transfer object metadata when uploading files to Cloud Storage, with the exception of mtime and symlink targets. This means that you cannot set object metadata when you upload files using Cloud Storage FUSE. If you need to preserve object metadata, consider uploading files using gsutil, the JSON API, or the Google Cloud console. * Concurrency: Cloud Storage FUSE does not provide concurrency control for multiple writes to the same file. When multiple writes try to replace a file, the last write wins and all previous writes are lost. There is no merging, version control, or user notification of the subsequent overwrite. * Linking: Cloud Storage FUSE does not support hard links. * File locking and file patching: Cloud Storage FUSE does not support file locking or file patching. As such, you should not store version control system repositories in Cloud Storage FUSE mount points, as version control systems rely on file locking and patching. Additionally, you should not use Cloud Storage FUSE as a filer replacement. * Semantics: Semantics in Cloud Storage FUSE are different from semantics in a traditional file system. For example, metadata like last access time are not supported, and some metadata operations like directory renaming are not atomic. For a list of differences between Cloud Storage FUSE semantics and traditional file system semantics, see Semantics in the Cloud Storage FUSE GitHub documentation. * Overwriting in the middle of a file: Cloud Storage FUSE does not support overwriting in the middle of a file. Only sequential writes are supported. Access: Authorization for files is governed by Cloud Storage permissions. POSIX-style access control does not work. However rclone has `--vfs-cache-mode writes` which caches file writes to disk first to allow overwriting in the middle of a file and `--vfs-cache-mode full` to cache all objects on a LRU basis. They both make the file system a whole lot more POSIX compatible and most applications will run using `--vfs-cache-mode writes` unlike `--vfs-cache-mode off`. And of course rclone supports s3/azureblob/b2/r2/sftp/webdav/etc/etc also... I don't think it is possible to adapt something with cloud storage semantics to a file system without caching to disk, unless you are willing to leave behind the 1:1 mapping of files seen in the mount to object in the cloud storage.
- retrocryptid 3y agoThis has been a thing for a while; I remember using it (or something like it) several years ago. While it's great for random files you might want to place in the G-Cloud, what I really wanted was to access my google docs content from the Linux command line. And you can do that, it's just that they're in non-obvious, non-documented, frequently changing formats that will only ever be usable with Google Docs. But if you're using the google cloud like you might use Box.Net or DropBox, it seems fine for light usage.
- milesward 3y agoPlease, listen to me: use this only in extremely limited cases where performance, stability, and cost efficiency are not paramount. An object store is not a file system no matter how hard you bludgeon it.
- ofek 3y agoI do appreciate that Google is now officially supporting gcsfuse because it genuinely is a great project. However, their Kubernetes CSI driver seems to have in large part copied code from the one I and a co-maintainer have been working on for years: - https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver - https://github.com/ofek/csi-gcs https://github.com/ofek/csi-gcs Here is the initial commit: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/commit/04df4c7f0940a41530e94342e26ac2e65ef8cb77 https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/c... Notice for example not just the code but also the associated files. In the Dockerfile it blatantly copied the one from my repo, even the dual license I chose because I was very into Rust at the time. Or take a look at the deployment examples which use Kustomize which I like but is very uncommon and most Kubernetes projects provide Helm charts instead. They were most certainly aware of the project because Google reached out to discuss potential collaboration but never responded back: https://imgur.com/a/KDuf9mj https://imgur.com/a/KDuf9mj
- whuan 3y agoAre you accusing Google of "large part copied code" based on an old commit which is not even used in this official launch? Do you have any evidence from their recent commit? At least I don't see the current two repos are anywhere alike, except for that you both implement the same interface. Also, they did reach out to you and you just didn't respond, so why are you complaining now? It makes me sad that no one here cares about whether your blame is true. And I'd expect you can provide more convincing evidence. But looks like the accusation is not even true. It's not fair for those contributors man, I hope you can apologize.
- ofek 3y agoI'm not sure if you thoroughly read what I wrote but I did respond to them. This is not a false accusation as you are claiming, you can check the contents of the repo in the current state. Per the licenses they can copy but they must maintain attribution which has not been done.
- ofek 3y agoUpdate: attribution has been added to the readme file https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/commit/40dbfc5b736948f82051539ed9926609f399b792 https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/c...