7 ms·

I spent a good few weeks recently researching and testing different back up systems that I could install to a Raspberry Pi with a big USB hard drive attached, t

by da_n 10y ago

I spent a good few weeks recently researching and testing different back up systems that I could install to a Raspberry Pi with a big USB hard drive attached, to act as an extra offsite backup for some remote servers. rsnapshot was pretty much the only system I found that ticked every requirement, it has proven itself very robust in the month or so I have had it doing nightly snapshots of around 5 servers. Previously I had used a Ruby back up library which was actually really good but needed to be run from the server rather than the back up endpoint.

stevekemp 10y ago
I wonder what you included in your requirements? For me things I value are encryption on the backup-host, and de-duplication. That lead me to obnam, and attic. rsnapshot is a nice tool, because it only needs to transmit things that have changed, but without encryption it isn't something I'd personally want to run again.
- prohor 10y ago
  The de-duplication in this case is achieved by hardlinks of unchanged files between snapshots. This means, deduplication is only on file level, so even an append makes a new copy. I've been looking for an rsync-based tool with encryption, but really couldn't find any. There is that uses rsync way of finding differences & keeps change log, but this is in fact normal full-snapshot & incremental backups: http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
  - extra88 10y ago
    My understanding of what people looking for de-dup want is for their backup store to contain zero files with identical contents. With rsnapshot, if you rename a file, rename a parent directory of it, or move the file to another directory, you will get another copy of that file in your backup. I've used rsnapshot for about ten years, mainly to make remote backups of data on SMB/AFP file servers, and something I had to actively discourage was staff using the moving of files/folders between higher level folders as a method of project management. One group in particular does video production so a project between yesterday and today might have only a change of a few bytes but because the project's folder was moved to a different parent folder (e.g. Completed/), rsnapshot would make a new complete backup of that project which could be tens of gigs. Filesystem snapshots like ZFS's avoid this problem but I haven't worked with how those are copied remotely.
- prohor 10y ago
  You might like BoxBackup, but unfortunately not very active project now: https://www.boxbackup.org/ https://www.boxbackup.org/
- kijin 10y ago
  The backup tool itself doesn't need to know about encryption at all. I have full control over the backup host, so I would use either full-disk encryption (luks) or filesystem-level encryption (encfs/ecryptfs) on the backup host. The former works better if you want de-duplication. Regular ssh takes care of encryption in transit, so rsync just pipes unencrypted bits through this encrypted tunnel. You could probably add GPG in there somewhere if you wanted. Just a bunch of Unix tools doing Unixy things in the Unix way.
- da_n 10y ago
  Indeed as kijin states, the drive itself is full disk encrypted using LUKS, and SSH takes care of encrypting everything during transport. I did not require de-dupe but did only want to transmit only the diffs since last back up. I also wanted a something that would be of minimal resource usage on remote end (for rsnapshot was just running an rsync daemon if I recall correctly). In the case of one Ruby library I looked at, the entire system needed to be installed on each remote host to work correctly, which was unacceptable in my scenario. Primarily, it just feels like a solid tool and seems like it can be adapted to suit a lot of use cases (I have seen a config to allow backup of AWS RDS instances for instance).

by da_n 10y ago

stevekemp 10y ago

I wonder what you included in your requirements? For me things I value are encryption on the backup-host, and de-duplication. That lead me to obnam, and attic. rsnapshot is a nice tool, because it only needs to transmit things that have changed, but without encryption it isn't something I'd personally want to run again.

prohor 10y ago
The de-duplication in this case is achieved by hardlinks of unchanged files between snapshots. This means, deduplication is only on file level, so even an append makes a new copy. I've been looking for an rsync-based tool with encryption, but really couldn't find any. There is that uses rsync way of finding differences & keeps change log, but this is in fact normal full-snapshot & incremental backups: http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
- extra88 10y ago
  My understanding of what people looking for de-dup want is for their backup store to contain zero files with identical contents. With rsnapshot, if you rename a file, rename a parent directory of it, or move the file to another directory, you will get another copy of that file in your backup. I've used rsnapshot for about ten years, mainly to make remote backups of data on SMB/AFP file servers, and something I had to actively discourage was staff using the moving of files/folders between higher level folders as a method of project management. One group in particular does video production so a project between yesterday and today might have only a change of a few bytes but because the project's folder was moved to a different parent folder (e.g. Completed/), rsnapshot would make a new complete backup of that project which could be tens of gigs. Filesystem snapshots like ZFS's avoid this problem but I haven't worked with how those are copied remotely.
prohor 10y ago
You might like BoxBackup, but unfortunately not very active project now: https://www.boxbackup.org/ https://www.boxbackup.org/
kijin 10y ago
The backup tool itself doesn't need to know about encryption at all. I have full control over the backup host, so I would use either full-disk encryption (luks) or filesystem-level encryption (encfs/ecryptfs) on the backup host. The former works better if you want de-duplication. Regular ssh takes care of encryption in transit, so rsync just pipes unencrypted bits through this encrypted tunnel. You could probably add GPG in there somewhere if you wanted. Just a bunch of Unix tools doing Unixy things in the Unix way.
da_n 10y ago
Indeed as kijin states, the drive itself is full disk encrypted using LUKS, and SSH takes care of encrypting everything during transport. I did not require de-dupe but did only want to transmit only the diffs since last back up. I also wanted a something that would be of minimal resource usage on remote end (for rsnapshot was just running an rsync daemon if I recall correctly). In the case of one Ruby library I looked at, the entire system needed to be installed on each remote host to work correctly, which was unacceptable in my scenario. Primarily, it just feels like a solid tool and seems like it can be adapted to suit a lot of use cases (I have seen a config to allow backup of AWS RDS instances for instance).