7 ms·
It turns out that rsync is slower than scp, unless the files are already present. It adds a lot of overhead that’s not needed most of the time. I often need to
by zeroimpl 6y ago
It turns out that rsync is slower than scp, unless the files are already present. It adds a lot of overhead that’s not needed most of the time.
I often need to transfer very large binary files. When stuck with standard Linux tools, the fastest process is first transfer via scp. If the scp gets interrupted, resume it using rsync with the —-append-verify option.
- sroussey 6y agonc is much faster if you don’t need encryption
- zeroimpl 6y agoYeah, in my case I do need it. Fortunately hpn-ssh has great performance: https://www.psc.edu/hpn-ssh https://www.psc.edu/hpn-ssh
- kbenson 6y agoWell, nc +tar +compression is likely faster. If you compress your SSH connections, scp or piping over SSH is likely much faster for files that aren't tiny, I would imagine. You aren't necessarily wrong, but there's a whole world of distinction you're glossing over. Netcat is part of a set of tools you can use to construct a somewhat equivalent tool to scp without encryption, but nc by itself without tar and gzip (or whatever) match only the simplest possibly use case of scp, which is copying one file, without compression, and ignoring all permissions and ownership info. This is also ignoring that you might not even have an open port on one side that the other can reach to have netcat listen on.
- m463 6y agoI started using rsync almost exclusively. mostly either: rsync -av src dest or: rsync -av --remove-source-files src dest