8 ms·

There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that

There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.

amelius 6d ago
There's something vastly superior over rsync still: "btrfs send" and "btrfs receive" (but require you to have btrfs on source and destination).
- greaber 6d ago
  Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
  - gcr 6d ago
    'btrfs send' is more like diff/patch than it is like rsync. It requires both source and destination to agree about the source snapshot.
  - amelius 6d ago
    btrfs send is useful for sending incremental updates; this is used a LOT for backups; and it saves a lot of time because it does not need to traverse files/directories that were not changed You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.
craftkiller 6d ago
One instance where rsync's delta merge comes up for me: For all of my virtual machines, I build a bootable live ISO so that way I can cleanly swap out the full ISO with a new one when there are software updates. This means that every time I want to change a config file inside the virtual machine, I end up building a new ISO and rsyncing it to my server. rsync speeds this up significantly since most (but not all) of the ISO is unchanged.
- dwedge 6d ago
  How do you handle updates this way? Do you mount the iso, update and keep it as an iso? Are all of the virtual machines mostly read only?
  - craftkiller 6d ago
    The iso itself is read-only. When I need to update software I build a whole new iso and replace the existing one. For any folders/files that need persistence across reboots, I store that data separately from the iso, either in a virtual nvme drive or a mounted 9pfs folder from the host. The iso mounts the persistence drive/9pfs via /etc/fstab. So in the end, the software and config files are read-only (until I replace them with a new iso) but my databases and data files are read-write. Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence. I build it all via NixOS and I use Impermanence to manage the bind mounts to the persist drive: https://wiki.nixos.org/wiki/Impermanence https://wiki.nixos.org/wiki/Impermanence
- greaber 6d ago
  Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)
Someone 6d ago
> Note that syq doesn't currently implement rsync's delta merge algorithm That, and still claiming it’s “better than rsync”? Seems like you shouldn’t compare it with rsync (yet, if doing delta merges is planned)
- greaber 6d ago
  Yeah, I kind of regret writing "better than rsync" in the tagline. Maybe I should have written "better than rsync in some respects".

amelius 6d ago

There's something vastly superior over rsync still: "btrfs send" and "btrfs receive" (but require you to have btrfs on source and destination).

greaber 6d ago
Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
- gcr 6d ago
  'btrfs send' is more like diff/patch than it is like rsync. It requires both source and destination to agree about the source snapshot.
- amelius 6d ago
  btrfs send is useful for sending incremental updates; this is used a LOT for backups; and it saves a lot of time because it does not need to traverse files/directories that were not changed You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.

craftkiller 6d ago

One instance where rsync's delta merge comes up for me: For all of my virtual machines, I build a bootable live ISO so that way I can cleanly swap out the full ISO with a new one when there are software updates. This means that every time I want to change a config file inside the virtual machine, I end up building a new ISO and rsyncing it to my server. rsync speeds this up significantly since most (but not all) of the ISO is unchanged.

dwedge 6d ago
How do you handle updates this way? Do you mount the iso, update and keep it as an iso? Are all of the virtual machines mostly read only?
- craftkiller 6d ago
  The iso itself is read-only. When I need to update software I build a whole new iso and replace the existing one. For any folders/files that need persistence across reboots, I store that data separately from the iso, either in a virtual nvme drive or a mounted 9pfs folder from the host. The iso mounts the persistence drive/9pfs via /etc/fstab. So in the end, the software and config files are read-only (until I replace them with a new iso) but my databases and data files are read-write. Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence. I build it all via NixOS and I use Impermanence to manage the bind mounts to the persist drive: https://wiki.nixos.org/wiki/Impermanence https://wiki.nixos.org/wiki/Impermanence
greaber 6d ago
Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)

Someone 6d ago

> Note that syq doesn't currently implement rsync's delta merge algorithm That, and still claiming it’s “better than rsync”? Seems like you shouldn’t compare it with rsync (yet, if doing delta merges is planned)

greaber 6d ago
Yeah, I kind of regret writing "better than rsync" in the tagline. Maybe I should have written "better than rsync in some respects".