5 ms·

What the author describes here is hard because it's "simple." What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS

What the author describes here is hard because it's "simple." What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢

smarmelling 14d ago
I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!
- xbar 14d ago
  I liked the post because it tells a true story about one of the remaining problems that are hard to solve well without a 3rd party.
- XorNot 14d ago
  Fair warning with ZFS: do not turn on deduplication at the moment. There is a straight up data loss bug in 2.4.3 (zeroed out files, totally silent). https://github.com/openzfs/zfs/issues/18366 https://github.com/openzfs/zfs/issues/18366 This caused all sorts of grief at day job where dedupe was useful for a big cache. Conversely I've run ZFS at home for like 15 years at this point without trouble. But this one is an absolute nightmare.
fc417fc802 14d ago
> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity. So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline. That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often. He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life. Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity. Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.
- nolist_policy 14d ago
  > Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity. Not if you use git-annex.
  - fc417fc802 14d ago
    Huh, I'd heard the name before but I hadn't realized how capable it was. Unfortunately when it comes to a data hoarder such as myself: https://git-annex.branchable.com/scalability/ https://git-annex.branchable.com/scalability/ > Scaling to hundreds of thousands of files is not a problem, scaling beyond that and git will start to get slow. So that's probably insufficient for me by at least a couple orders of magnitude. I'm able to maintain my sanity because snapshots simply capture device state, the NAS collects all snapshots while maintaining their independence, and (so far) borg has been sufficiently scalable to deduplicate any collection I've thrown at it.
cm2187 14d ago
Less hard these days. AI is a game changer for learning new technologies. It's like having a highly paid expert available to answer all your questions about your little USB backup. Makes learning how to use properly a new software trivial. And priceless when troubleshooting.

smarmelling 14d ago

I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!

xbar 14d ago
I liked the post because it tells a true story about one of the remaining problems that are hard to solve well without a 3rd party.
XorNot 14d ago
Fair warning with ZFS: do not turn on deduplication at the moment. There is a straight up data loss bug in 2.4.3 (zeroed out files, totally silent). https://github.com/openzfs/zfs/issues/18366 https://github.com/openzfs/zfs/issues/18366 This caused all sorts of grief at day job where dedupe was useful for a big cache. Conversely I've run ZFS at home for like 15 years at this point without trouble. But this one is an absolute nightmare.

fc417fc802 14d ago

> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity. So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline. That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often. He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life. Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity. Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.

nolist_policy 14d ago
> Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity. Not if you use git-annex.
- fc417fc802 14d ago
  Huh, I'd heard the name before but I hadn't realized how capable it was. Unfortunately when it comes to a data hoarder such as myself: https://git-annex.branchable.com/scalability/ https://git-annex.branchable.com/scalability/ > Scaling to hundreds of thousands of files is not a problem, scaling beyond that and git will start to get slow. So that's probably insufficient for me by at least a couple orders of magnitude. I'm able to maintain my sanity because snapshots simply capture device state, the NAS collects all snapshots while maintaining their independence, and (so far) borg has been sufficiently scalable to deduplicate any collection I've thrown at it.

cm2187 14d ago

Less hard these days. AI is a game changer for learning new technologies. It's like having a highly paid expert available to answer all your questions about your little USB backup. Makes learning how to use properly a new software trivial. And priceless when troubleshooting.