7 ms·

I’d start with reading the article; it lists several.

by jgys 1y ago

remram 1y ago
The article lists other S3-compatible servers that also don't have UIs. That doesn't fix the problem.
- amluto 1y ago
  None of those support the use case I actually want: an S3-compatible server that stores data locally and replicates to a cloud provider. Ideally it should also store enough integrity-checking metadata what one can verify that everything is replicated correctly to the cloud after the fact.
  - deleted 1y ago
    [deleted]
  - haiku2077 1y ago
    I think you want rclone. Specifically a combination of: a local filesystem remote a cloud provider remote rclone serve s3 to serve a local s3 compatible API for the local remote something orchestating an rclone copy, rclone sync or rclone bisync command to sync the local and cloud remotes. Could be triggered on a timer, or using another tool to watch the local directory for changes and trigger a sync.
    - aspenmayer 1y ago
      I swear I suggested rclone on an old refresh of the page only to find you had posted this nearly an hour ago. I wasn't sure rclone could fit this use case, but I'm glad I wasn't totally off-base in assuming it was very close. I think you can run rclone as a service, they recommend you use bisync on crontab, for example. What's bisync? I don't really know, as it's new to me, but it's an experimental beta feature from rclone and sounds neat. https://rclone.org/bisync/ https://rclone.org/bisync/ https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-between-hosts https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-... > Sharing an encrypted folder tree between hosts > bisync can keep a local folder in sync with a cloud service, but what if you have some highly sensitive files to be synched? > Usage of a cloud service is for exchanging both routine and sensitive personal files between one's home network, one's personal notebook when on the road, and with one's work computer. The routine data is not sensitive. For the sensitive data, configure an rclone crypt remote to point to a subdirectory within the local disk tree that is bisync'd to Dropbox, and then set up an bisync for this local crypt directory to a directory outside of the main sync tree.
      - haiku2077 1y ago
        Yeah for my home server backups I run rclone on a systemd timer
      - remram 1y ago
        > https://rclone.org/bisync/ https://rclone.org/bisync/ This is quite an amazing man page, at no point does it say what the command does. It has a lot of details upfront about it being beta, and where its working directory is, and what the limitations are. But what does it do?
        aspenmayer 1y ago
        Bidirectional sync via rclone?
        haiku2077 1y ago
        That's not the manpage, this is the manpage: https://rclone.org/commands/rclone_bisync/ https://rclone.org/commands/rclone_bisync/
  - aspenmayer 1y ago
    I don't know if it is a server or client in this context, but have you looked into using rclone? I think due to the nature of the way it intermediates the S3 API, and involves (re)downloading files for some operations, rclone is probably either, or both, depending on what you're doing and how. https://rclone.org/ https://rclone.org/ https://github.com/rclone/rclone https://github.com/rclone/rclone Specifically, check out this new beta feature called bisync which has use cases that sound a lot like yours: https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-between-hosts https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-...
  - dardeaup 1y ago
    Out of curiosity, if using replication how many copies would you expect to be stored locally and how many in the cloud? If using erasure coding, which parameters locally and which in the cloud? How quickly would you expect the replication to cloud to occur?
    - amluto 1y ago
      One or more locally. One in the cloud. I want a local object store that archives, reliably, to the cloud. And I don’t mean some wildly inefficient periodic job that tries to find new objects and archive them — I mean something actually integrated in that reliably and verifiably replicates.
  - unsnap_biceps 1y ago
    Funny enough, MinIO supported that feature via bucket replication. You could set the bucket replication to asynchronous or synchronous, write to MinIO and it would sync away, including custom storage classes on the target bucket. I use it to sync to a aws S3 bucket.

I’d start with reading the article; it lists several.

by jgys 1y ago

I’d start with reading the article; it lists several.

remram 1y ago

The article lists other S3-compatible servers that also don't have UIs. That doesn't fix the problem.

amluto 1y ago
None of those support the use case I actually want: an S3-compatible server that stores data locally and replicates to a cloud provider. Ideally it should also store enough integrity-checking metadata what one can verify that everything is replicated correctly to the cloud after the fact.
- deleted 1y ago
  [deleted]
- haiku2077 1y ago
  I think you want rclone. Specifically a combination of: a local filesystem remote a cloud provider remote rclone serve s3 to serve a local s3 compatible API for the local remote something orchestating an rclone copy, rclone sync or rclone bisync command to sync the local and cloud remotes. Could be triggered on a timer, or using another tool to watch the local directory for changes and trigger a sync.
  - aspenmayer 1y ago
    I swear I suggested rclone on an old refresh of the page only to find you had posted this nearly an hour ago. I wasn't sure rclone could fit this use case, but I'm glad I wasn't totally off-base in assuming it was very close. I think you can run rclone as a service, they recommend you use bisync on crontab, for example. What's bisync? I don't really know, as it's new to me, but it's an experimental beta feature from rclone and sounds neat. https://rclone.org/bisync/ https://rclone.org/bisync/ https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-between-hosts https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-... > Sharing an encrypted folder tree between hosts > bisync can keep a local folder in sync with a cloud service, but what if you have some highly sensitive files to be synched? > Usage of a cloud service is for exchanging both routine and sensitive personal files between one's home network, one's personal notebook when on the road, and with one's work computer. The routine data is not sensitive. For the sensitive data, configure an rclone crypt remote to point to a subdirectory within the local disk tree that is bisync'd to Dropbox, and then set up an bisync for this local crypt directory to a directory outside of the main sync tree.
    - haiku2077 1y ago
      Yeah for my home server backups I run rclone on a systemd timer
    - remram 1y ago
      > https://rclone.org/bisync/ https://rclone.org/bisync/ This is quite an amazing man page, at no point does it say what the command does. It has a lot of details upfront about it being beta, and where its working directory is, and what the limitations are. But what does it do?
      - aspenmayer 1y ago
        Bidirectional sync via rclone?
      - haiku2077 1y ago
        That's not the manpage, this is the manpage: https://rclone.org/commands/rclone_bisync/ https://rclone.org/commands/rclone_bisync/
- aspenmayer 1y ago
  I don't know if it is a server or client in this context, but have you looked into using rclone? I think due to the nature of the way it intermediates the S3 API, and involves (re)downloading files for some operations, rclone is probably either, or both, depending on what you're doing and how. https://rclone.org/ https://rclone.org/ https://github.com/rclone/rclone https://github.com/rclone/rclone Specifically, check out this new beta feature called bisync which has use cases that sound a lot like yours: https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-between-hosts https://rclone.org/bisync/#sharing-an-encrypted-folder-tree-...
- dardeaup 1y ago
  Out of curiosity, if using replication how many copies would you expect to be stored locally and how many in the cloud? If using erasure coding, which parameters locally and which in the cloud? How quickly would you expect the replication to cloud to occur?
  - amluto 1y ago
    One or more locally. One in the cloud. I want a local object store that archives, reliably, to the cloud. And I don’t mean some wildly inefficient periodic job that tries to find new objects and archive them — I mean something actually integrated in that reliably and verifiably replicates.
- unsnap_biceps 1y ago
  Funny enough, MinIO supported that feature via bucket replication. You could set the bucket replication to asynchronous or synchronous, write to MinIO and it would sync away, including custom storage classes on the target bucket. I use it to sync to a aws S3 bucket.