7 ms·

I wonder about this: how do people archive their information, especially photo and video data. I use an array of mirrored HDDs in different locations with every

I wonder about this: how do people archive their information, especially photo and video data. I use an array of mirrored HDDs in different locations with everything on it, plus parts of it on a cloud hosting service. It isn't very convenient though and it seems there should be a better solution. How do people in here do it? Most solutions (for me) fall short on quick retrieval and possibility to visually browse photos.

WalterBright 7y ago
I buy new hard drives every year and copy all the data to them. Edit: It's not just the longevity of the drive (though after 20 years they often don't work at all). Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer, and my old computers failed due to capacitor problems. For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. Just keep copying it all forward every year. Multi-terabyte hard drives are cheap.
- westpfelia 7y ago
  So this isnt a way to actively use a 5.25" floppy for storage... but if you have data on one you want to save this is an option: http://www.cbmstuff.com/proddetail.php?prod=SCP http://www.cbmstuff.com/proddetail.php?prod=SCP
  - WalterBright 7y ago
    Thanks for the tip! But it doesn't seem to present the floppy data as a file system :-(
- walrus01 7y ago
  > For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. It's really not that hard, assuming some older PC format. Motherboards up to the 486DX4/120 generation had onboard controllers for the cable to interface to a standard AT type, half height 5.25" drive. I had one in my PC at the time. They read 360KB disks fine too. From that sort of computer, which can run DOS 6.22, it's not too hard to use it to transfer the data to a modern machine. https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.ebay.com%2Fulk%2Fitm%2F173920548126 https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.eb... If you mean good luck trying to read a 5.25" disk due to magnetic degradation on the media,that's another problem.
  - WalterBright 7y ago
    > From that sort of computer, which can run DOS 6.22, it's not too hard to use it to transfer the data to a modern machine. Well, yes, if you have an old computer with 3.5 and 5.25 drives, you can transfer to a 3.5 and read that on a modern computer. The trouble is, my old computers all failed due to capacitor problems. I can buy a 3.5 floppy usb drive, plug it into any computer, and it works. Not so for 5.25. > If you mean good luck trying to read a 5.25" disk due to magnetic degradation on the media,that's another problem. I haven't had much issue with that. About 20 years ago, I transferred my thousand or so floppies one by one to CD ROMs, and later to hard disks. Only a couple were unreadable. I've since found a handful of others, but no way to read them any more.
- tialaramex 7y ago
  The philosophy of just copying data makes sense, but your justification not so much. > Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer 20+ year old PC drives will have IDE interfaces, which you can of course get a USB adaptor for very cheaply and I have several laying about. Really old PCs or non-PC hardware (e.g. the Amstrad CPC 6128 in the 1980s had an after market hard disk option) will need a different adaptor, but it's still just not that hard, and certainly shouldn't be categorised as "there is no way". > For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. You can easily buy refurbished 5.25" drives, and you can convert the data signal (which for a PC was basically the same as for 3.5" drives, but with a different physical connector) to USB. They're bulky and they need a lot of power at 12 volts, which is probably why off-the-shelf 5.25" USB floppy drives aren't a thing unlike for 3.5", but it won't require that you "build your own hardware" in any real sense. You're only going to need to go writing device drivers if you've got some crazy disk format, and it'd have to be pretty crazy because the Catweasel already covered a lot of that territory (a Catweasel was a thing for Amigans, whose disks are a crazy format, to let them use PC disk drives to read their disks years ago, the last ones have an FPGA in them so that the uploadable firmware can decide how to control the drive).
  - WalterBright 7y ago
    I have several 5.25 drives laying around. The connectors still fit on them to the motherboard, but the BIOS no longer recognizes them, though the BIOS will recognize 3.5 floppy drives. I can't find any device that connects the drive with the computer that doesn't require writing a device driver. It's not a crazy disk format, it's DOS format. As for hard disk drive interfaces, I have a couple off the shelf USB interfaces to IDE drives. The connectors fit, but they don't recognize the older drives.
    - tialaramex 7y ago
      I can't speak to what's up with the BIOS versus 5.25" drives, beyond to say that I'm confident somebody out there has this working without writing device drivers. For the USB to IDE thing, my guess would be that the USB interfaces you tried want to do some newer incarnation of ATA than the drive supports and so they can't communicate, at the extreme that could be LBA48 (48-bit logical block addressing, which is from this century and so obviously isn't going to work on say a 20 year old hard disk) but it could even be DMA, which was optional in early ATA specifications and might have been skipped by cheap manufacturers back in the day. I don't think logical addressing itself was ever optional, but I guess it's possible that a _really_ old IDE drive might not implement it and yet work anyway in DOS which didn't originally use LBA mode. Doing CHS mode really would be asking a lot, compared to PIO (the alternative to DMA) which really a competent implementation of that USB adaptor ought to be able to do, slowly like a real PC.
- WalterBright 7y ago
  In the early 80's, I had a PDP-11 with an 8" floppy drive. When I bought an IBM PC, I used Kermit to transfer my files to the PC over the serial cable. Fast forward 30 years, and I had put my PDP-11 Empire game up on github. Someone emailed me that it was missing a file. I was dead in the water, having gotten rid of the 11 long ago. I recalled my roommate in college might have one. Called him up, he said he was about to discard it, I called him just in time! He said it was unlikely to work (been a long time since it was powered up), and unlikely to be able to read those ancient floppies, but was willing to try. I had kept the floppies in the attic, and mailed them to him. They all read perfectly! He sent me all the files on them, and bit images of the disks. Thank you Shal, and thanks DEC for making machines and floppies that still worked. https://github.com/DigitalMars/Empire-for-PDP-11 https://github.com/DigitalMars/Empire-for-PDP-11 I wish I'd kept my 11.
dharmab 7y ago
A few of my co-workers have cross-site backups arranged with siblings or parents. Each house runs a few local servers and mirrors the backups to the other house. This relies on having a trusted party outside of your natural disasters zone, but it's pretty cost effective if both parties wanted to run home servers anyway (e.g. Plex, homelab)
enriquto 7y ago
Once a year, before christmas, you chose about a hundred of the best photos, print them in a beautiful album, send copies to your grandmothers, and delete the rest.
- Fnoord 7y ago
  I love this approach. It features (with some overlaps): 1) Less is more. 2) Data hygiene. 3) Present for grandfather/grandmother/family. The problem is that the more photos you made, the more difficult it becomes to pick a hundred. We leave behind so much data trails every day, including data which is no longer relevant. We constantly have to decide which data we keep or 'consume'. The problem is only becoming worse throughout the years (in history). Data is more easily available, and more easily able to be made (or fabricated!) now than 50 years ago.
- tonyedgecombe 7y ago
  I've had a policy of keeping one photo a day and discarding the rest for some time now. It's quite freeing to not have to worry about keeping a huge digital archive.
  - dewey 7y ago
    In times where storage space is cheap is there really a difference between keeping one picture safe or keeping 100 safe? If you can keep one safe, you can keep 1000s safe with the same kind of process.
    - enriquto 7y ago
      But how can you keep even a single picture safe? Are you able to read today any of your files from 20 years ago? Not in a theoretical sense.
      - dewey 7y ago
        I was mostly referring to the previous comment about only keeping one picture and not a huge archive. But I don’t see a problem opening images from 20 years ago. If you are worried just keep a jpeg or raw viewer app in the archive too. Worst case there will probably be emulators to run an ancient OS where your image viewer works and you can convert your images.
KozmoNau7 7y ago
I store my important documents, music collection etc. locally on my PC, regularly synced to a NAS, automatically synced to pCloud, and every week I sync to an external hard drive that I keep in my locker at work. Maybe it's overkill, but it really isn't much work to bring the disk home, fire off an rsync script and bring it to work again the next morning. And external hard drives are ridiculously cheap. For photos, my phone uploads them to Google Photos automatically. I've been considering moving that to pCloud as well.
lrem 7y ago
Processed photos and videos: Google Photos and YouTube. As an insider I know that I can't afford to match the availability of those and having the live display copy indestructible is soooo convenient. Downside: I don't think YouTube will give you back verbatim input material. Photos will, but it can be expensive. Raw media material after processing: single local HDD and a remote raid. Home directory: two local time machine HDDs and a remote raid copy of head. Unpublished code: home directory + a private git server. Published code: as unpublished + GitHub (which I assume to be indestructible too).
- y4mi 7y ago
  I agree that it's hard to beat the availability of Google consumer services... But that is not backup. A backup serves as a fallback if for any reason the primary source has become unavailable. In Google's case such an even can and does occur when your account gets banned because of some random event which you had hardly anything to do with or because your user was compromised. A backup let's you recover from that and if you don't have that, I hope for your sake you don't mind losing the data too much, because the previous situations happen quite frequently if you actually listen on Twitter, Reddit or whatever social network you frequent.
  - lrem 7y ago
    As I wrote: Google services are the display copy. Conveniently it's overwhelmingly likely to survive for years upon years without any effort. But it can disappear, for reasons you mentioned, in which case I can recover the input media from the options in the second paragraph.
- mceachen 7y ago
  Google Photos is most decidedly not a viable backup for photos and videos. Google Takeout is broken or flakey, and had been for at least a year. Good luck downloading any archive before it falls with "transfer failed: bad authentication" 30 minutes into the transfer. Of 30 tarballs, only one actually completed successfully. And the images in the tarballs aren't the same bits that I uploaded (via my pixel, so it's supposed to be "original quality," not "high quality"). Pulling your files via the API results in corrupted metadata. Time and GPS location tags are stripped. YouTube videos are, by design, reencoded with very low bitrates.
Cthulhu_ 7y ago
It would be neat to have a service that converts digital photos to the equivalent of film or microfilm, an analog medium that can be converted back to full size pictures in a hundred years. It would have to be scalable and cheap though, given how easy it is to take pictures nowadays. But yeah, spend $100 / year and get a big film roll of your photo archive, that would be neat.
ahje 7y ago
I have a small laptop running a few services for home use, incl. a DLNA server and network storage. Media dropped on a specific network drive will be available over DLNA. Makes it easy to look at photos from computers and DLNA capable TV's, but it doesn't work for archival purposes as the laptop used as a server isn't really redundant. Our own laptops are backed up using a cloud backup solution.

WalterBright 7y ago

I buy new hard drives every year and copy all the data to them. Edit: It's not just the longevity of the drive (though after 20 years they often don't work at all). Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer, and my old computers failed due to capacitor problems. For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. Just keep copying it all forward every year. Multi-terabyte hard drives are cheap.

westpfelia 7y ago
So this isnt a way to actively use a 5.25" floppy for storage... but if you have data on one you want to save this is an option: http://www.cbmstuff.com/proddetail.php?prod=SCP http://www.cbmstuff.com/proddetail.php?prod=SCP
- WalterBright 7y ago
  Thanks for the tip! But it doesn't seem to present the floppy data as a file system :-(
walrus01 7y ago
> For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. It's really not that hard, assuming some older PC format. Motherboards up to the 486DX4/120 generation had onboard controllers for the cable to interface to a standard AT type, half height 5.25" drive. I had one in my PC at the time. They read 360KB disks fine too. From that sort of computer, which can run DOS 6.22, it's not too hard to use it to transfer the data to a modern machine. https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.ebay.com%2Fulk%2Fitm%2F173920548126 https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.eb... If you mean good luck trying to read a 5.25" disk due to magnetic degradation on the media,that's another problem.
- WalterBright 7y ago
  > From that sort of computer, which can run DOS 6.22, it's not too hard to use it to transfer the data to a modern machine. Well, yes, if you have an old computer with 3.5 and 5.25 drives, you can transfer to a 3.5 and read that on a modern computer. The trouble is, my old computers all failed due to capacitor problems. I can buy a 3.5 floppy usb drive, plug it into any computer, and it works. Not so for 5.25. > If you mean good luck trying to read a 5.25" disk due to magnetic degradation on the media,that's another problem. I haven't had much issue with that. About 20 years ago, I transferred my thousand or so floppies one by one to CD ROMs, and later to hard disks. Only a couple were unreadable. I've since found a handful of others, but no way to read them any more.
tialaramex 7y ago
The philosophy of just copying data makes sense, but your justification not so much. > Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer 20+ year old PC drives will have IDE interfaces, which you can of course get a USB adaptor for very cheaply and I have several laying about. Really old PCs or non-PC hardware (e.g. the Amstrad CPC 6128 in the 1980s had an after market hard disk option) will need a different adaptor, but it's still just not that hard, and certainly shouldn't be categorised as "there is no way". > For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it. You can easily buy refurbished 5.25" drives, and you can convert the data signal (which for a PC was basically the same as for 3.5" drives, but with a different physical connector) to USB. They're bulky and they need a lot of power at 12 volts, which is probably why off-the-shelf 5.25" USB floppy drives aren't a thing unlike for 3.5", but it won't require that you "build your own hardware" in any real sense. You're only going to need to go writing device drivers if you've got some crazy disk format, and it'd have to be pretty crazy because the Catweasel already covered a lot of that territory (a Catweasel was a thing for Amigans, whose disks are a crazy format, to let them use PC disk drives to read their disks years ago, the last ones have an FPGA in them so that the uploadable firmware can decide how to control the drive).
- WalterBright 7y ago
  I have several 5.25 drives laying around. The connectors still fit on them to the motherboard, but the BIOS no longer recognizes them, though the BIOS will recognize 3.5 floppy drives. I can't find any device that connects the drive with the computer that doesn't require writing a device driver. It's not a crazy disk format, it's DOS format. As for hard disk drive interfaces, I have a couple off the shelf USB interfaces to IDE drives. The connectors fit, but they don't recognize the older drives.
  - tialaramex 7y ago
    I can't speak to what's up with the BIOS versus 5.25" drives, beyond to say that I'm confident somebody out there has this working without writing device drivers. For the USB to IDE thing, my guess would be that the USB interfaces you tried want to do some newer incarnation of ATA than the drive supports and so they can't communicate, at the extreme that could be LBA48 (48-bit logical block addressing, which is from this century and so obviously isn't going to work on say a 20 year old hard disk) but it could even be DMA, which was optional in early ATA specifications and might have been skipped by cheap manufacturers back in the day. I don't think logical addressing itself was ever optional, but I guess it's possible that a _really_ old IDE drive might not implement it and yet work anyway in DOS which didn't originally use LBA mode. Doing CHS mode really would be asking a lot, compared to PIO (the alternative to DMA) which really a competent implementation of that USB adaptor ought to be able to do, slowly like a real PC.
WalterBright 7y ago
In the early 80's, I had a PDP-11 with an 8" floppy drive. When I bought an IBM PC, I used Kermit to transfer my files to the PC over the serial cable. Fast forward 30 years, and I had put my PDP-11 Empire game up on github. Someone emailed me that it was missing a file. I was dead in the water, having gotten rid of the 11 long ago. I recalled my roommate in college might have one. Called him up, he said he was about to discard it, I called him just in time! He said it was unlikely to work (been a long time since it was powered up), and unlikely to be able to read those ancient floppies, but was willing to try. I had kept the floppies in the attic, and mailed them to him. They all read perfectly! He sent me all the files on them, and bit images of the disks. Thank you Shal, and thanks DEC for making machines and floppies that still worked. https://github.com/DigitalMars/Empire-for-PDP-11 https://github.com/DigitalMars/Empire-for-PDP-11 I wish I'd kept my 11.

dharmab 7y ago

A few of my co-workers have cross-site backups arranged with siblings or parents. Each house runs a few local servers and mirrors the backups to the other house. This relies on having a trusted party outside of your natural disasters zone, but it's pretty cost effective if both parties wanted to run home servers anyway (e.g. Plex, homelab)

enriquto 7y ago

Once a year, before christmas, you chose about a hundred of the best photos, print them in a beautiful album, send copies to your grandmothers, and delete the rest.

Fnoord 7y ago
I love this approach. It features (with some overlaps): 1) Less is more. 2) Data hygiene. 3) Present for grandfather/grandmother/family. The problem is that the more photos you made, the more difficult it becomes to pick a hundred. We leave behind so much data trails every day, including data which is no longer relevant. We constantly have to decide which data we keep or 'consume'. The problem is only becoming worse throughout the years (in history). Data is more easily available, and more easily able to be made (or fabricated!) now than 50 years ago.
tonyedgecombe 7y ago
I've had a policy of keeping one photo a day and discarding the rest for some time now. It's quite freeing to not have to worry about keeping a huge digital archive.
- dewey 7y ago
  In times where storage space is cheap is there really a difference between keeping one picture safe or keeping 100 safe? If you can keep one safe, you can keep 1000s safe with the same kind of process.
  - enriquto 7y ago
    But how can you keep even a single picture safe? Are you able to read today any of your files from 20 years ago? Not in a theoretical sense.
    - dewey 7y ago
      I was mostly referring to the previous comment about only keeping one picture and not a huge archive. But I don’t see a problem opening images from 20 years ago. If you are worried just keep a jpeg or raw viewer app in the archive too. Worst case there will probably be emulators to run an ancient OS where your image viewer works and you can convert your images.

KozmoNau7 7y ago

I store my important documents, music collection etc. locally on my PC, regularly synced to a NAS, automatically synced to pCloud, and every week I sync to an external hard drive that I keep in my locker at work. Maybe it's overkill, but it really isn't much work to bring the disk home, fire off an rsync script and bring it to work again the next morning. And external hard drives are ridiculously cheap. For photos, my phone uploads them to Google Photos automatically. I've been considering moving that to pCloud as well.

lrem 7y ago

Processed photos and videos: Google Photos and YouTube. As an insider I know that I can't afford to match the availability of those and having the live display copy indestructible is soooo convenient. Downside: I don't think YouTube will give you back verbatim input material. Photos will, but it can be expensive. Raw media material after processing: single local HDD and a remote raid. Home directory: two local time machine HDDs and a remote raid copy of head. Unpublished code: home directory + a private git server. Published code: as unpublished + GitHub (which I assume to be indestructible too).

y4mi 7y ago
I agree that it's hard to beat the availability of Google consumer services... But that is not backup. A backup serves as a fallback if for any reason the primary source has become unavailable. In Google's case such an even can and does occur when your account gets banned because of some random event which you had hardly anything to do with or because your user was compromised. A backup let's you recover from that and if you don't have that, I hope for your sake you don't mind losing the data too much, because the previous situations happen quite frequently if you actually listen on Twitter, Reddit or whatever social network you frequent.
- lrem 7y ago
  As I wrote: Google services are the display copy. Conveniently it's overwhelmingly likely to survive for years upon years without any effort. But it can disappear, for reasons you mentioned, in which case I can recover the input media from the options in the second paragraph.
mceachen 7y ago
Google Photos is most decidedly not a viable backup for photos and videos. Google Takeout is broken or flakey, and had been for at least a year. Good luck downloading any archive before it falls with "transfer failed: bad authentication" 30 minutes into the transfer. Of 30 tarballs, only one actually completed successfully. And the images in the tarballs aren't the same bits that I uploaded (via my pixel, so it's supposed to be "original quality," not "high quality"). Pulling your files via the API results in corrupted metadata. Time and GPS location tags are stripped. YouTube videos are, by design, reencoded with very low bitrates.

Cthulhu_ 7y ago

It would be neat to have a service that converts digital photos to the equivalent of film or microfilm, an analog medium that can be converted back to full size pictures in a hundred years. It would have to be scalable and cheap though, given how easy it is to take pictures nowadays. But yeah, spend $100 / year and get a big film roll of your photo archive, that would be neat.

ahje 7y ago

I have a small laptop running a few services for home use, incl. a DLNA server and network storage. Media dropped on a specific network drive will be available over DLNA. Makes it easy to look at photos from computers and DLNA capable TV's, but it doesn't work for archival purposes as the laptop used as a server isn't really redundant. Our own laptops are backed up using a cloud backup solution.