Starofall

4 ms·

1.
by Starofall 2y ago
For a big sync this is definitly the way to go. But for a simple daily sync we found the tool more usefull. Especially when syncing to a non AWS S3 provider that does not offer a inventory feature.
2.
by Starofall 2y ago
File size is easy to compare, so you at least know that the full file got sycned. Hashes is a bigger issue with AWS in general, as you only get an ETag from S3 which is the MD5 of a file for Puts, but not for multiparts - and also dependent
3.
by Starofall 2y ago
Profile support might also make things less easy to use, as the tool has support for cross S3 provider syncing. Thats why it needs the key and secret of source and target seperatly. Local Sync is on the idea list, but not that simple - as l
4.
by Starofall 2y ago
I have not yet compared it against this tool, but the given numbers are for download and not for syncing files. For a S3->S3 sync using an C6gn.8xlarge instance, I got up to 800MB/sec using 64 workers, but the files were only in ave
5.
by Starofall 2y ago
The code is really extremly simple, just these few files: https://github.com/Starofall/S3HyperSync/tree/main/src/main/... 1) The underlying S3 Framework is already super fast https:/&#x2F
6.
by Starofall 2y ago
Yes! That was one usecase we had, that the AWS CLI was not able to do. You can sync any two S3 compatible services with each other, and it even supports legacy path-style if you e.g. want to sync to a local Minio S3 installation.
7.
Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s(github.com)
49 points by Starofall 2y ago|12 comments
8.
Show HN: QryGraph – Graphical editor for PigLatin(BigData/Hadoop) queries(github.com)
1 points by Starofall 10y ago|0 comments
9.
Show HN: DrawTogehter – Realtime-App-Prototype for Android,Scala,Scaloid,Play(github.com)
2 points by Starofall 10y ago|0 comments
10.
Show HN: Chakka – Akka, Akka.JS, typed websocket chat(github.com)
3 points by Starofall 10y ago|0 comments
11.
Show HN: Riskell – Risk Clone in Haskell Using Yesod(github.com)
1 points by Starofall 10y ago|0 comments