11 ms·

The long road to recover Frogger 2 source from tape drives

510 points by WhiteDawn 3y ago

h2odragon 3y ago
Truly noble effort. Hopefully the writeup and the tools will save others much heartbreak.
db48x 3y ago
Wow, that backup software sounds like garbage. Why not just use tar? Why would anyone reinvent that wheel?
- robotnikman 3y ago
  The company that made it probably was hoping for vendor lock-in
  - cosmotic 3y ago
    Vendor lock in for backup and archival products is so ridiculous. It increases R&D to ensure the lock-in, and the company won't exist by the time the lock-in takes effect.
    - fifteen1506 3y ago
      Well yes, but the boss probably is willing to invest more money (meaning higher salaries, more people, better tools) expecting a future return than when using reasonable formats.
- giantrobot 3y ago
  IIRC tar has some Unixisms that don't necessarily work for Windows/NTFS. Not saying reinventing tar is appropriate but there's Windows/NTFS that a Windows based tape backup need to support.
  - ilyt 3y ago
    The format is extensible enough that it could be added
  - cosmotic 3y ago
    Most of what makes NTFS different than FAT probably doesn't need to be backed up. Complex ACLs, alternative data streams, shadow copies, etc, are largely irrelevant when it comes to making a backup. Just a simple warning "The data being backed up includes alternative data streams. These aren't supported and won't be included in the backup" would suffice.
    - jandrese 3y ago
      All of that stuff matters when you're using the backup for its intended purpose: to restore a system after hardware failure. Unix tar is obviously not the right solution, but a Windows tar seems like it shouldn't be that hard to do and yet we are in the situation we are today. I've been using dump/restore for decades now on Unix, including to actually recover from loss, but I admit that it's not that pleasant to use. I like that it is very simple and reliable however, unlike the mess that is Time Machine (recovering from a hardware loss on a Mac is a roll of the dice, and I've gotten snakes) or worse Deja Dup. I'm not sure I've ever successfully recovered a system from a Deja Dup backup.
      - cosmotic 3y ago
        I think the use case for disaster recovery is a bit different than long-term archival.
        mrguyorama 3y ago
        Does anyone buying tape storage actually use it for archival rather than disaster recovery?
        kjs3 3y ago
        We do. We have vast amounts of data that we merely have to keep around for a decade or so for compliance purposes that would rarely be accessed, so it goes on tape and off to Iron Mountain. Then we have backups where we need to be able to recover a running system from some 'known good' state, which is somewhat complicated. The former is conceptually tape drive+tar/cpio/etc.; the latter is an expensive setup that includes some proprietary solutions.
- bombcar 3y ago
  The world of tape backup was (is?) absolutely filled with all sorts of vendor-lock in projects and tools. It's a complete mess. And even various versions of tar aren't compatible, and that's not even starting with star and friends.
  - stepupmakeup 3y ago
    It's not just limited to tape, most archiving and backup software is proprietary. It's impossible to open Acronis or Macrium Reflect images without their Windows software. In Acronis's case they even make it impossible to use offline or on a server OS without paying for a license. NTBackup is awfully slow and doesn't work past Vista, and it's not even part of XP POSReady for whatever reason, so I had to rip the exe from a XP ISO and unpack it (NTBACKUP._EX... I forgot microsoft's term for that) because the Vista version available on Microsoft's site specifically checks for longhorn or vista. Then there's slightly more obscure formats that didn't take off in the western world, and the physical mediums too. Not many people had the pleasure of having to extract hundreds of "GCA" files off of MO disks using obscure Japanese freeware from 2002. The English version of the software even has a bunch of flags on virustotal that the standard one doesn't. And there's obscure LZH compression algorithms that no tool available now can handle. I've found myself setting up one-time Windows 2000/XP VMs just to access backups made after 2000.
    - Dylan16807 3y ago
      I can only speak for macrium but they have good reasons to use their own format, so that you can have differential mountable backups. That's very different from someone inventing tar-but-worse.
  - jandrese 3y ago
    I have at various times considered a tape backup solution for my home, but always give up when it seems every tape vendor is only interested in business clients. It was a race to stay ahead of hard drives and oftentimes they seemed to be losing. The price points were clearly aimed at business customers, especially on the larger capacity tapes. In the end I do backup to hard drives instead because it's much cheaper and faster.
- EvanAnderson 3y ago
  ARCServe was a Computer Associates product. That's all you need to know. It had a great reputation on Novell Netware but the Windows product was a mess. I never had a piece of backup management software cause blue screens (e.g. kernel panics) before an unfortunate Customer introduced me to ARCServe on Windows.
  - nycdotnet 3y ago
    My favorite ArcServe bug which they released a patch for (and which didn’t actually fix the issue, as I recall) had a KB article called something along the lines of “The Open Database Backup Agent for Lotus Notes Cannot Backup Open Databases”.
masto 3y ago
This brings back (unpleasant) memories. I remember trying to get those tape drives working with FreeBSD back in 1999, and it going nowhere.
ilamont 3y ago
In The Singularity Is Near (2005) Ray Kurzweil discussed an idea for the “Document Image and Storage Invention”, or DAISI for short, but concluded it wouldn't work out. I interviewed him a few years later about this and here's what he said: The big challenge, which I think is actually important almost philosophical challenge — it might sound like a dull issue, like how do you format a database, so you can retrieve information, that sounds pretty technical. The real key issue is that software formats are constantly changing. People say, “well, gee, if we could backup our brains,” and I talk about how that will be feasible some decades from now. Then the digital version of you could be immortal, but software doesn’t live forever, in fact it doesn’t live very long at all if you don’t care about it if you don’t continually update it to new formats. Try going back 20 years to some old formats, some old programming language. Try resuscitating some information on some PDP1 magnetic tapes. I mean even if you could get the hardware to work, the software formats are completely alien and [using] a different operating system and nobody is there to support these formats anymore. And that continues. There is this continual change in how that information is formatted. I think this is actually fundamentally a philosophical issue. I don’t think there’s any technical solution to it. Information actually will die if you don’t continually update it. Which means, it will die if you don’t care about it. ... We do use standard formats, and the standard formats are continually changed, and the formats are not always backwards compatible. It’s a nice goal, but it actually doesn’t work. I have in fact electronic information that in fact goes back through many different computer systems. Some of it now I cannot access. In theory I could, or with enough effort, find people to decipher it, but it’s not readily accessible. The more backwards you go, the more of a challenge it becomes. And despite the goal of maintaining standards, or maintaining forward compatibility, or backwards compatibility, it doesn’t really work out that way. Maybe we will improve that. Hard documents are actually the easiest to access. Fairly crude technologies like microfilm or microfiche which basically has documents are very easy to access. So ironically, the most primitive formats are the ones that are easiest.
- krapp 3y ago
  I'm certain that 100 years from now, when the collapse really gets rolling, we'll still have cuneiform clay tablets complaining about Ea-Nassir's shitty copper but most of the digital information and culture we've created and tried to archive will be lost forever. Eventually, we're going to lose the infrastructure and knowledge base we need to keep updating everything, people will be too busy just trying to find food and fighting off mutants from the badlands to care.
huehehue 3y ago
Fascinating read that unlocked some childhood memories. I'm secondhand pissed at the recovery company, I have a couple of ancient SD cards laying around and this just reinforces my fear that if I send them away for recovery they'll be destroyed (the cards aren't recognized/readable by the readers built into MacBooks, at least)
- ryanjshaw 3y ago
  Painful lesson I've learned myself the hard way - don't rush something that doesn't need to be rushed.
- somat 3y ago
  My understanding is that flash memory does not do very well at all for long term unpowered data retention. flash memory is basically a capacitor(it is not really a capacitor but close to one) and will loose it's charge after a few years. And magnetic drives will seize up. and optical disks get oxidized, and tapes stick together. long term archiving is a tricky endeavor. It is however an interesting challenge. I think I would get acquainted with the low level protocol used by sd cards. then modify a microcontroller sdmmc driver to get me an image of the card(errors and all). that is, without all the scsi disk layers doing their best to give you normalized access to the device. Or more realistically, hope someone more talented than me does the above.
  - User23 3y ago
    Tapes hold up really well if they're not in absolutely awful storage conditions. And the claim at least was that the early CD-ROMs were quite durable, being a straight up laser carved diffraction grating. CDRs on the other hand rely on dye which will degrade rapidly.
    - somat 3y ago
      That is true about mag tape, I suspect tape to be one of the better choices for archival storage. In fact the biggest problem you will have with mag tape is making sure you have a working drive 10 - 20 years in the future when you want to look at your archives. to make things worse tape drives are getting more and more flimsy and fragile as tape tech advances. my first job we had a vault of ibm reel to reel tapes of old business data. our attitude if we were ever asked to pull any of the data was that we would probably only get one chance at it as the ferric material on on the tape had a disturbing tendency to flake off. note that there are techniques to reduce this, but we did not have the means or motivation to apply them. And a pedantic observation on your correct point about optical disks. you can't make backups on pressed disks, only recordable ones.
ogurechny 3y ago
Modern backup would simply state “API keys and settings are here:”, and a link to collaboration platform closed after 3 years of existence.
- jandrese 3y ago
  Hey, it's the cloud. Backups are "someone else's problem". That is until they are your problem, then you're up a creek.
  - tivert 3y ago
    > Hey, it's the cloud. Backups are "someone else's problem". That is until they are your problem, then you're up a creek. The FSF used to sell these wonderful stickers that said "There is not cloud. It's just someone else's computer."
    - isaidthis 3y ago
      The sticker: https://static.fsf.org/nosvn/stickers/thereisnocloud.svg https://static.fsf.org/nosvn/stickers/thereisnocloud.svg "Stickers from various FSF campaigns - Print out copies of our stickers for your own uses, local conferences and more." https://www.fsf.org/resources/stickers https://www.fsf.org/resources/stickers
- ilyt 3y ago
  Honestly backup space is weirdly sparse for anything on enterprise scale. For anything more than few machines there is bacula/bareos (that pretends everything is tape with mostly miserable results), backuppc (that pretends tapes are not a thing, with miserable results), and that's about it, everything else seems to be point-to-point backups only with no real central management.
  - somat 3y ago
    You can add amanda to the "pretends everything is tape with mostly miserable results" list.
LeoPanthera 3y ago
I really wish they would name the data recovery company so that I can never darken their door with my business.
- nickt 3y ago
  It’s not too hard to find with the following search, “we can recover data from tape formats including onstream”
  - stepupmakeup 3y ago
    The OP explicity didn't name them (despite many people recommending to, even preservationists in this field on Reddit and Discord) but it's easy to find just by googling the text on the screenshots
    - ddtaylor 3y ago
      Name them and we can setup a thread or site to publicly shame them
      - stepupmakeup 3y ago
        the comment I replied to edited the link out https://www.datarecovery.net/tape-data-recovery.aspx https://www.datarecovery.net/tape-data-recovery.aspx
    - omoikane 3y ago
      Reddit thread: https://www.reddit.com/r/DataHoarder/comments/13q1pv7/playstation_game_frogger_2_source_code_recovered/ https://www.reddit.com/r/DataHoarder/comments/13q1pv7/playst...
- bluedino 3y ago
  > Over the span of about a month, I received very infrequent and vague communications from the company despite me providing extremely detailed technical information and questions. Ahh the business model of "just tell them to send us the tape and we'll buy the drive on eBay"
bsder 3y ago
Is there way to read magnetic tapes like these in such a way as to get the raw magnetic flux at high resolution? It seems like it would be easier to process old magnetic tapes by imaging them and then applying signal processing rather than finding working tape drives with functioning rollers. Most of the time, you're not worried about tape speed since you're just doing recovery read rather than read/write operations. So, a slow but accurate operation seems like it would be a boon for these kinds of things.
- fifteen1506 3y ago
  You still need to know where to look, the format, and using specialized equipment which cost wasn't driven down by mass manufacturing, so, in theory yes, in practice not. (Completely guessing here with absolute no knowledge of the real state of things)
- iforgotpassword 3y ago
  Sounds like at least in this case that ASIC in the drive was doing some (non trivial) signal processing. Would be interesting to know how hard it would be to get from the flux pattern back to zeros and ones. I guess with a working drive you can at least write as many test patterns as you want until you maybe figure it out.
  - jandrese 3y ago
    At the very least the drive needs to be able to lock onto the signal. It's probably encoded in a helix on the drive and if the head isn't synchronized properly you won't get anything useful, even with a high sampling rate.
    - Clamchop 3y ago
      I would be surprised if it used helical recording. Data tape recorders rarely do because it's much more complex, increases tape wear, and the use cases don't usually demand that kind of linear bandwidth.
robotnikman 3y ago
I've always admired the tenacity of people who reverse engineer stuff. To be able to spend multiple months figuring out barely documented technologies with no promise of success takes a lot a willpower and discipline. It's something I wish I could improve more in myself.
- detrites 3y ago
  I think you could. In some sense "easily". It may be about finding that thing you're naturally so interested in or otherwise drawn to, that the months figuring out become a type of driven joy, and so the willpower kinda automatic. And if you find it, don't judge what it is or worry what others might think - or even necessarily tell anyone. Sometimes the most motivating things are highly personal, as with the OP; a significant part of their childhood.
  - robotnikman 3y ago
    You definitely have a point there, looking at some of my previous work I was able to stick to projects for many months if I found the work interesting. I'll have to admit in the past 5 or so years any time I've tried to start a project there was always the thought in the back of my mind of 'will this benefit my career' or 'how can I make money on this in the future'. It seems having such thoughts adds additional anxiety to whenever I try and start to work on something for fun. Looks like that is what I need to start looking for again, projects which I find interesting or fun to do in my spare time, without thinking about how it would affect my career or trying to find ways to monetize it.
    - detrites 3y ago
      I totally get this. Something I'm learning - slowly - as it is so counter to sound ethic, is that making some work into pure fun helps the other work by preventing burn-out, which for me at least is an ongoing risk. Certainly feels better!
smokel 3y ago
Heh, I remember playing .mp3 files directly from QIC-80 tapes, somewhere around 1996. One tape could store about 120 MB, which is equal to about two compact discs' worth of audio. The noise of the tape drive was slightly annoying, though. And it made me appreciate what the 't' in 'tar' stands for.
- mjaniczek 3y ago
  Did you mean 1200 MB? That would make sense wrt. 2x CD capacity.
  - smokel 3y ago
    No, it was really only 120 MB. I was referring to the length of an audio compact disc, not the capacity of a CD-ROM. At 128 kbps, you'd get about 2 hours of play time. Of course it didn't really make sense to use digital tapes for that use case, even back then. It was just for fun, and the article sparked some nostalgic joy, which felt worth sharing :)
  - deleted 3y ago
    [deleted]
  - stewarts 3y ago
    They reference MP3, and a CD ripped down to MP3 probably fits in the 50-100MB envelope for size. It has been a very long time since I last ripped an album, but that size jives with my memory.
  - deleted 3y ago
    [deleted]
jimbob45 3y ago
F2 was a really neat game. It almost invented Crypt of the Necrodancer’s genre decades early. It’s a little sad that it took such a monumental effort to bring the source code back from the brink of loss. It’s times like that that should inspire lawmakers to void copyright in the case that the copyright holders can’t produce the thing they’re claiming copyright over.
xigency 3y ago
As a kid, I got this game as a gift and really, really wanted to play it. But after beating the second level, the game would always crash on my computer with an Illegal Operation exception. I remember sending a crash report to the developer, and even updating the computer, but I never got it working.
- jakeinspace 3y ago
  I adored this game as a kid, and I think I do have a faint memory of some stability issues, but I believe I was able to beat the game.
tombert 3y ago
This is giving me some anxiety about my tape backups. I have backed up my blu-ray collection to a dozen or so LTO-6 tapes, and it's worked great, but I have no idea how long the drives are going to last for, and how easy it will be to repair them either. Granted, the LTO format is probably one of the more popular formats, but articles like this still keep me up at night.
- bombcar 3y ago
  Do test restores. LTO is very good but without verification some will fail at some point. But your original bluray disk are also a backup.
- EvanAnderson 3y ago
  The only surefire method to keep the bits readable is to continue moving them onto new media every few years. Data has a built-in recurring cost. I'd love to see a solution to that problem but I think it's unlikely. It's a least possible, though, that we'll come up with a storage medium with sufficient density and durability that'll it'll be good enough. I don't even want to think about the hairy issues associated with keeping the bits able to be interpreted. That's a human behavior problem more than a technology problem.
- wazoox 3y ago
  LTO-7 drives read LTO-6, and will be available for quite a while. In 2016 I've used an LTO-3 drive to restore a bunch (150 or 200) of LTO-1/2 tapes from 2000-2003, and almost all but one or two worked fine.
- antod 3y ago
  LTO is one of the best choices for compatibility. I remember just how awful DDS (same sort of media as DAT) tape backups were - due to differences in head alignments, it was a real lottery as to whether any given tape could be read on a different drive than the one that wrote it.
crazygringo 3y ago
Wow, this part makes my blood boil, emphasis mine: > This issue doesn't affect tapes written with the ADR-50 drive, but all the tapes I have tested written with the OnStream SC-50 do NOT restore from tape unless the PC which wrote the tape is the PC which restores the tape. This is because the PC which writes the tape stores a catalog of tape information such as tape file listing locally, which the ARCserve is supposed to be able to restore without the catalog because it's something which only the PC which wrote the backup has, defeating the purpose of a backup. Holy crap. A tape backup solution that doesn't allow the tape to be read by any other PC? That's madness. Companies do shitty things and programmers write bad code, but this one really takes the prize. I can only imagine someone inexperienced wrote the code, nobody ever did code review, and then the company only ever tested reading tapes from the same computer that wrote them, because it never occured to them to do otherwise? But yikes.
- Neil44 3y ago
  I'm familiar with needing to re-index a backup if it's accessed from a 'foreign' machine and sometimes the procedure is non-obvious but just not having that option seems pretty bad.
  - bluedino 3y ago
    I worked for an MSP a million years ago and we had a customer that thought they had lost everything. They had backup tapes but the backup server itself had died, after showing them the 'catalog tape' operation, and keeping their fingers crossed for a few hours, they bought me many beers.
    - EvanAnderson 3y ago
      I always had the Customer keep a written log of which tapes were used on which days. It helped for accountability but also prevented the "Oh, shit, we have to catalog all the tapes because the log of which tapes were used on which day are on the now-failed server."
- Nextgrid 3y ago
omnibrain 3y ago
Is anyone else calling it “froggering/to frogger” if they have to cross a bigger street by foot without a dedicated crossing?
bluedino 3y ago
This will be fun in 20 years, trying recover 'cloud' backups from servers found in some warehouse.
- Gigachad 3y ago
  Those drives will all be encrypted and most likely shredded.
- ilyt 3y ago
  Nah it will be very simple: ....What do you mean "nobody paid for the bucket for last 5 years" ? There is some chance someone might stash old hard drive or tape with backup somewhere in the closet. There is no chance there will be anything left when someone stops paying for cloud.
dllthomas 3y ago
On the topic of Froggers, I enjoyed https://www.youtube.com/watch?v=FCnjMWhCOcA https://www.youtube.com/watch?v=FCnjMWhCOcA
readyplayernull 3y ago
A few months ago I was looking for an external backup drive and thought that SSD would be great because it's fast and shock resistant. Years ago I killed a Macbook Pro HD by throwing it on my bed from few inches high. Then I read a comment on Amazon about SSD losing information when unpowered for a long time. I couldn't find any quick confirmation in the product page, took me a few hours of research to find some paper about this phenomenon. If I remember correctly it takes a few weeks for the stored SSD to start losing its data. So I bought a mechanical HD. Another tech tip is not buying 2 backup devices from the same batch or even the same model. Chances being these will fail in the same way.
- vidarh 3y ago
  To the last bit, I've seen this first hand. Had a whole RAID array of the infamous IBM DeathStar drives fail one after the other while we frantically copied data off. Last time I ever had the same model drives in an array.
  - smcameron 3y ago
    Heh, I remember in the early 1990s having a RAID array with a bunch of 4Gb IBM drives come up dead after a weekend powerdown for a physical move due to "stiction". I was on the phone with IBM, and they were telling me to physically bang the drives on the edge of desk to loosen them up. Didn't seem to be working, so their advice was "hit it harder!" When I protested, they said, "hey, it already doesn't work, what have you got to lose?" So I hit it harder. Eventually got enough drives to start up to get the array on line, and you better believe the first thing I did after that was create a fresh backup (not that we didn't have a recent backup anyway), and the 2nd thing I did was replace those drives, iirc, with Seagate Barracudas.
    - vidarh 3y ago
      Ouch. I knew someone who claimed to have dealt with that or a similar effect after their cleaning person had pulled the plug on their servers by putting the drive in an oven while connected and heating it slowly. Personally my most nailbiting period was when I got my first (20MB!) drive as a kid and it was too big an investment to replace even when it refused to spin up without me opening up the drive(!) and nudging the platter with my finger to help the motor spin it up... I backed everything up (on floppies), and stored everything important straight to floppies, but it was still more convenient to hold on to the HD for the next 6 months or so until I'd saved up enough to replace it... It's remarkable what drives can survive if you're lucky... Also remarkable how quickly that luck can run out, though. > "hey, it already doesn't work, what have you got to lose?" This attitude has saved me more than once. Recognising when you can afford to do things that seems ridiculous helps surprisingly often.
hlandau 3y ago
Absolutely amazing story. Fantastic! I've actually long been stunned by the propensity of proprietary backup software to use undocumented, proprietary formats. I've always found this quite stunning, in fact. It seems to me like the first thing one should make sure to solve when designing a backup format is to ensure it can be read in the future even if all copies of the backup software are lost. I may be wrong but I think some open source tape backup software (Amanda, I think?) does the right thing and actually starts its backup format with emergency restoration instructions in ASCII. I really like this kind of "Dear future civilization, if you are reading this..." approach. Frankly nobody should agree to use a backup system which generates output in a proprietary and undocumented format, but also I want a pony... It's interesting to note that the suitability of file formats for archiving is also a specialised field of consideration. I recall some article by someone investigating this very issue who argued formats like .xz or similar weren't very suited to archiving. Relevant concerns include, how screwed you are if the archive is partly corrupted, for example. The more sophisticated your compression algorithm (and thus the more state it records from longer before a given block), the more a single bit flip can result in massive amounts of run-on data corruption, so better compression essentially makes things worse if you assume some amount of data might be damaged. You also have the option of adding parity data to allow for some recovery from damage, of course. Though as this article shows, it seems like all of this is nothing compared to the challenge of ensuring you'll even be able to read the media at all in the future. At some point the design lifespan of the proprietary ASICs in these tape drives will presumably just expire(?). I don't know what will happen then. Maybe people will start using advanced FPGAs to reverse engineer the tape format and read the signals off, but the amount of effort to do that would be astronomical, far more even than the amazing effort the author here went to.
- hlandau 3y ago
  To add, thinking a bit more about it: Designing formats to be understandable by future civilizations actually reduces to a surprising degree to the same set of problems which METI has to face. As in, sending signals designed to be intelligible to extraterrestrials - Carl Sagan's Contact, etc. Even if you write an ASCII message directly to a tape, that data is obviously going to be encoded before being written to the tape, and you have no idea if anyone will be able to figure out that encoding in future. Trouble. What makes this particularly pernicious is the fact that LTO nowadays is a proprietary format(!!). I believe the spec for the first generation or two of LTO might be available, but last I checked, it's been proprietary for some time. The spec is only available to the (very small) consortium of companies which make the drives and media. And the number of companies which make the drives is now... two, I think? (They're often rebadged.) Wouldn't surprise me to see it drop to one in the future. This seems to make LTO a very untrustworthy format for archiving, which is deeply unfortunate.
PicassoCTs 3y ago
The author has fantastic endurance, what a marathon to get the files of the tape.
phkahler 3y ago
>> The tape was the only backup for those things, and it completes Frogger 2's development archives, which will be released publicly. In cases like this can imagine some company yelling "copyright infringement" even though they don't possess a copy themselves. It's a really odd situation.
FearNotDaniel 3y ago
> the ADR-50e drive was advertised as compatible, but there was a cave-at I'm assuming the use of "cave-at" means the author has inferred an etymology of "caveat" being made up of "cave" and "at", as in: this guarantee has a limit beyond which we cannot keep our promises, if we ever find ourselves AT that point then we're going to CAVE. (As in cave in, meaning give up.) I can't think of any other explanation of the odd punctuation. Really quite charming, I'm sure I've made similar inferences in the past and ended up spelling or pronouncing a word completely wrong until I found out where it really comes from. There's an introverted cosiness to this kind of usage, like someone who has gained a whole load of knowledge and vocabulary from quietly reading books without having someone else around to speak things out loud.
- nocoiner 3y ago
  I thought it might have been a transcription error of “carve out,” but your theory is more logical.
- Kneesnap 3y ago
  Dang it. OP here, I saw this typo and swear I fixed this typo before posting it!!
chrisstanchak 3y ago
I’ve been suffering through something similar with a DLT IV tape from 1999. Luckily I didn’t send out to the data recovery company. But still unsuccessful.
rootsudo 3y ago
Name and shame the company, you had a personal experience, you have proof. Name and shame. It helps nobody if you don't publicize it. Let them defend it, let them say whatever excuse, but your review will stand.
- Kneesnap 3y ago
  I don't want to even remotely tempt them to sue. They have no grounds, but I'm not taking risks-- companies are notorious for suing when they know they'll lose. Others who have posted it here have identified the right company though.
aidenn0 3y ago
TIL there are three completely different games named "Frogger 2" I assumed this was for the 1984 game, but this is for the 2000 game (there is also a 2008 game).
- pimlottc 3y ago
  Thanks for that, it seems like a surprisingly modern format for such an old game. Links for the games referenced: - Frogger II: ThreeeDeep! (1984) https://www.mobygames.com/game/7265/frogger-ii-threeedeep/ https://www.mobygames.com/game/7265/frogger-ii-threeedeep/ - Frogger 2: Swampy's Revenge (2000) [1] https://www.mobygames.com/game/2492/frogger-2-swampys-revenge/ https://www.mobygames.com/game/2492/frogger-2-swampys-reveng... - Frogger 2 (2008) [2] https://www.mobygames.com/game/47641/frogger-2/ https://www.mobygames.com/game/47641/frogger-2/
caycep 3y ago
At some point, I feel as if it may be easier just to rewrite the code from the ground up vs. going through all that computational archaeology.... Or in a few years, just have an AI write the code...
dabiged 3y ago
I work in the tape restoration space. My biggest piece of advice is never NEVER encrypt your tapes. If you think restoring data from an unknown format tape is hard, trying to do it when the drive will not let you read the blocks off the tape without a long lost decryption key is impossible.
ThorsBane 3y ago
> This is where the story should probably have stopped. Given up and called it a day, right? Maybe, but I care about this data, and I happen to know a thing or two about computers. Hahaha awwwww yeah :muscle:
userbinator 3y ago
As the other comment here says, any company claiming to do data recovery, and damaging the original media to that extent, should be named and shamed. I can believe that DR companies have generic drives and heads to read tapes of any format they come across, but even if they couldn't figure out how the data was encoded, there was absolutely no need to cut and splice the tape. I suspect they did that just out of anger at not likely being able to recover anything (and thus having spent a bunch of time for no profit.) Melted pinch rollers are not uncommon and there are plenty of other (mostly audio) equipment with similar problems and solutions --- dimensions are not absolutely critical and suitable replacements/substitutes are available. As an aside, I think that prominent "50 Gigabytes" capacity on the tape cartridge, with a small asterisk-note at the bottom saying "Assumes 2:1 compression", should be outlawed as a deceptive marketing practice. It's a good thing HDD and other storage media didn't go down that route.
kookamamie 3y ago
Someone was wise enough to erase the evidence in Party.
- Kneesnap 3y ago
  Nice catch! I think it was a little less juvenile than it might sound. I believe this was for a different game, Fusion Frenzy, which was a party minigame collection.
  - MayeulC 3y ago
    While I didn't understand the parent you are replying to (not your answer), your mention of Fusion Frenzy caught my eye. I've had a soft spot for that game since spending hours playing the "xbox magazine" demo with a childhood friend. Could you clarify? Is there any history gem about that one? I'd dig a PC port!
    - Kneesnap 3y ago
      In one of the screenshots was an empty folder called "Party", with the commenter suggesting they deleted it before the backup as some way to hide it, but most likely it was just because it was for another game. I don't have much knowledge about Fusion Frenzy, but I am looking for options for archiving its development too, if that's even a possibility, which I'm not certain of yet.
      - MayeulC 3y ago
        Ah, that makes more sense, thank you for explaining. I would love to see some development archives, and source code for it one day.
sydbarrett74 3y ago
This is a masterful recovery effort. The README should be shared as an object lesson far and wide to every data restoration and archival service around.
dark-star 3y ago
I'm pretty sure that even with the substantial damage done by the recovery company, a professional team like Kroll Ontrack can still recover the complete tape data, although it probably won't be cheap.
Const-me 3y ago
CD-R drives were already common in 2001: https://en.wikipedia.org/wiki/CD-R https://en.wikipedia.org/wiki/CD-R I wonder would a CD-R disk retain data for these 22 years?
- Tangurena2 3y ago
  Only if you kept the disk in a refrigerator. Bits are stored by melting the plastic slightly and the dye seeping in. Over time, the warmth of "room temperature" will cause the pits to become less well-defined so the decoder has to spend more time calculating "well, is that really a 1 or is it a sloppy 0". There's a lot of error detection/correction built into the CD specs, but eventually, there will be more error than can be corrected for. If you've ever heard the term "annealing" when used in machine learning, this is equivalent. Living in South Florida, ambient temperatures were enough to erase CD-Rs - typically in less than a year. I quickly started buying the much more expensive "archival" discs, but that wasn't enough. One fascinating "garage band" sold their music on CD-Rs and all of my discs died (it was a surfer band from Alabama).
  - Clamchop 3y ago
    The recording is made in the dye layer, a chemical change, and the dye degrades (particularly in sunlight) so the discs have a limited shelf life. Checking Wikipedia, it appears azo dye formulations can be good for tens of years. Melting polycarbonate would call for an absurdly powerful laser, a glacial pace, or both, and you wouldn't have to use dye at all. I'd guess such a scheme would be extremely durable, though.
- hakfoo 3y ago
  I tried a couple of CD-Rs that were stored in a dry closed drawer for most of the last 20 years recently, and they seemed to at least initially come up. Now I can reinstall Windows 2000 with Service Pack 4 slipstreamed!
jasomill 3y ago
Yet, despite ARCserve showing a popup which says "Restoration Successful", it restores up to the first 32KB of every file on the tape, but NO MORE." From 10,000 feet, this sounds suspiciously like ARCserve is reading a single tape block or transfer buffer's worth of data for each file, writing out the result, then failing and proceeding to the next file. Success popup notwithstanding, I'd expect to find errors in either the ARCserve or Windows event logs in this case — were there none? While it's been decades since I've dealt with ARCserve specifically, I've seen similar behavior caused by any number of things. Off the top of my head, (1) Incompatibilities between OS / backup software / HBA driver / tape driver. In particular, if you're using a version of Windows much newer than Windows 2000, try a newer version of ARCserve. In the absence of specific guidance, I'd probably start with the second* ARCserve version that officially supports Windows Server 2003: (a) Server 2003 made changes to the SCSI driver architecture that may not be 100% compatible with older software. (b) The second release will likely fix any serious Server 2003 feature-related bugs the first compatible version may have shipped with, without needing to install post-release patches that may be hard to find today. (b) Significantly newer ARCserve versions are more likely to introduce tape drive / tape format incompatibilities of their own. (2) Backup software or HBA driver settings incompatible with the hardware configuration (e.g., if ARCserve allows it, try reducing the tape drive transfer buffer size or switching from fixed block (= multiple tape blocks per transfer) to variable block (= single tape block per transfer) mode; if using an Adaptec HBA, try increasing the value of /MAXIMUMSGLIST[1]). (3) Shitty modern HBA driver support for tape (and, more generally, non-disk) devices. For example, modern Adaptec Windows HBA drivers have trouble with large tape block sizes that AFAIK cannot be resolved with configuration changes (though 32 kB blocks, as likely seen here, should be fine). In my experience with PCIe SCSI HBAs, LSI adapters are more likely to work with arbitrary non-disk devices and software out-of-the-box, whereas Adaptec HBAs often require registry tweaks for "unusual" circumstances (large transfer sizes; concurrent I/O to >>2 tape devices; using passthrough to support devices that lack Windows drivers, especially older, pre-SCSI 2 devices), assuming they can be made to work at all. LSI20320IE PCIe adapters are readily available for $50 or less on eBay and, in my experience, work well for most "legacy" applications. (To be fair to Adaptec, I've had nothing but good experiences using their adapters for "typical" applications: arbitrary disk I/O, tape backup to popular drive types, CD/DVD-R applications not involving concurrent I/O to many targets, etc.) (4) Misconfigured or otherwise flaky SCSI bus. In particular, if you're connecting a tape drive with a narrow (50-pin) SCSI interface to a wide (68-pin) port on the HBA, make sure the entire bus, including the unused pins, are properly terminated. The easiest way to ensure this is to use a standard 68-pin Ultra320 cable with built-in active LVD/SE termination, make sure termination is enabled on the HBA, disabled on the drive, that the opposite end of the cable from the built-in terminator is connected to the HBA, and, ideally, that the 68-to-50-pin adapter you're using to connect the drive to the cable is unterminated. You can also use a 50-pin cable connected to the HBA through a 68-to-50-pin adapter, but then you're either relying on the drive properly terminating the bus — which it may or may not do — or else you need an additional (50-pin) terminator for the drive end, which will probably cost as much as a Ultra320 cable with built-in termination (because the latter is a bog-standard part that was commonly bundled with both systems and retail HBA kits). Note that I have seen cases where an incorrect SCSI cable configuration works fine in one application, but fails spectacularly in another, seemingly similar application, or even the same application if the HBA manages to negotiate a faster transfer mode. While this should be far less likely to occur with a modern Ultra160 or Ultra320 HBA, assume nothing until you're certain the bus configuration is to spec (and if you're using an Ultra2 or lower HBA, consider replacing it). With all that said, reversing the tape format may well be easier than finding a compatible OS / ARCserve / driver / HBA combination. In any case, good job with that, and thanks for publishing source code! [1] http://download.adaptec.com/pdfs/readme/relnotes_29320lpe.pdf#page=4 http://download.adaptec.com/pdfs/readme/relnotes_29320lpe.pd...
GnarfGnarf 3y ago
DVDs should not be overlooked for backup. The Millennium type have been simulated to withstand 1,000 years.
ddingus 3y ago
This is just random, but reading this and the backup discussion made me think about SGI IRIX and how it could do incremental backups. One option was to specify a set of files, and that spec could just be a directory. Once done, the system built a mini filesystem and would write that to tape. XFS was the filesystem in use at the time I was doing systems level archival. On restores, each tape, each record was a complete filesystem. One could do it in place and literally see the whole filesystem build up and change as each record was added. Or, restore to an empty directory and you get whatever was in that record. That decision was not as information dense as others could be, but it was nice and as easy as it was robust. What our team did to back up some data managed engineering software was perform a full system backup every week, maybe two. Then incrementals every day, written twice to the tape. Over time, full backups were made and sent off site. One made on a fresh tape, another made on a tape that needed to be cycled out of the system before it aged out. New, fresh tapes entered the cycle every time one aged out. Restores were done to temp storage and rather than try and get a specific file, it was almost always easier to just restore the whole filesystem and then copy the desired file from there into its home location. The incrementals were not huge usually. Once in a while they got really big due to some maintenance type operation touching a ton of files. The nifty thing was no real need for a catalog. All one needed was the date to know which tapes were needed. Given the date, grab the tapes, run a script and go get coffee and then talk to the user needing data recovery to better understand what might be needed. Most of the time the tapes were read and the partial filesystem was sitting there ready to go right about the time those processes completed. Having each archive, even if it were a single file, contain a filesystem data set was really easy to use and manage. Loved it.

510 points by WhiteDawn 3y ago

h2odragon 3y ago

Truly noble effort. Hopefully the writeup and the tools will save others much heartbreak.

db48x 3y ago

Wow, that backup software sounds like garbage. Why not just use tar? Why would anyone reinvent that wheel?

robotnikman 3y ago
The company that made it probably was hoping for vendor lock-in
- cosmotic 3y ago
  Vendor lock in for backup and archival products is so ridiculous. It increases R&D to ensure the lock-in, and the company won't exist by the time the lock-in takes effect.
  - fifteen1506 3y ago
    Well yes, but the boss probably is willing to invest more money (meaning higher salaries, more people, better tools) expecting a future return than when using reasonable formats.
giantrobot 3y ago
IIRC tar has some Unixisms that don't necessarily work for Windows/NTFS. Not saying reinventing tar is appropriate but there's Windows/NTFS that a Windows based tape backup need to support.
- ilyt 3y ago
  The format is extensible enough that it could be added
- cosmotic 3y ago
  Most of what makes NTFS different than FAT probably doesn't need to be backed up. Complex ACLs, alternative data streams, shadow copies, etc, are largely irrelevant when it comes to making a backup. Just a simple warning "The data being backed up includes alternative data streams. These aren't supported and won't be included in the backup" would suffice.
  - jandrese 3y ago
    All of that stuff matters when you're using the backup for its intended purpose: to restore a system after hardware failure. Unix tar is obviously not the right solution, but a Windows tar seems like it shouldn't be that hard to do and yet we are in the situation we are today. I've been using dump/restore for decades now on Unix, including to actually recover from loss, but I admit that it's not that pleasant to use. I like that it is very simple and reliable however, unlike the mess that is Time Machine (recovering from a hardware loss on a Mac is a roll of the dice, and I've gotten snakes) or worse Deja Dup. I'm not sure I've ever successfully recovered a system from a Deja Dup backup.
    - cosmotic 3y ago
      I think the use case for disaster recovery is a bit different than long-term archival.
      - mrguyorama 3y ago
        Does anyone buying tape storage actually use it for archival rather than disaster recovery?
        kjs3 3y ago
        We do. We have vast amounts of data that we merely have to keep around for a decade or so for compliance purposes that would rarely be accessed, so it goes on tape and off to Iron Mountain. Then we have backups where we need to be able to recover a running system from some 'known good' state, which is somewhat complicated. The former is conceptually tape drive+tar/cpio/etc.; the latter is an expensive setup that includes some proprietary solutions.
bombcar 3y ago
The world of tape backup was (is?) absolutely filled with all sorts of vendor-lock in projects and tools. It's a complete mess. And even various versions of tar aren't compatible, and that's not even starting with star and friends.
- stepupmakeup 3y ago
  It's not just limited to tape, most archiving and backup software is proprietary. It's impossible to open Acronis or Macrium Reflect images without their Windows software. In Acronis's case they even make it impossible to use offline or on a server OS without paying for a license. NTBackup is awfully slow and doesn't work past Vista, and it's not even part of XP POSReady for whatever reason, so I had to rip the exe from a XP ISO and unpack it (NTBACKUP._EX... I forgot microsoft's term for that) because the Vista version available on Microsoft's site specifically checks for longhorn or vista. Then there's slightly more obscure formats that didn't take off in the western world, and the physical mediums too. Not many people had the pleasure of having to extract hundreds of "GCA" files off of MO disks using obscure Japanese freeware from 2002. The English version of the software even has a bunch of flags on virustotal that the standard one doesn't. And there's obscure LZH compression algorithms that no tool available now can handle. I've found myself setting up one-time Windows 2000/XP VMs just to access backups made after 2000.
  - Dylan16807 3y ago
    I can only speak for macrium but they have good reasons to use their own format, so that you can have differential mountable backups. That's very different from someone inventing tar-but-worse.
- jandrese 3y ago
  I have at various times considered a tape backup solution for my home, but always give up when it seems every tape vendor is only interested in business clients. It was a race to stay ahead of hard drives and oftentimes they seemed to be losing. The price points were clearly aimed at business customers, especially on the larger capacity tapes. In the end I do backup to hard drives instead because it's much cheaper and faster.
EvanAnderson 3y ago
ARCServe was a Computer Associates product. That's all you need to know. It had a great reputation on Novell Netware but the Windows product was a mess. I never had a piece of backup management software cause blue screens (e.g. kernel panics) before an unfortunate Customer introduced me to ARCServe on Windows.
- nycdotnet 3y ago
  My favorite ArcServe bug which they released a patch for (and which didn’t actually fix the issue, as I recall) had a KB article called something along the lines of “The Open Database Backup Agent for Lotus Notes Cannot Backup Open Databases”.

masto 3y ago

This brings back (unpleasant) memories. I remember trying to get those tape drives working with FreeBSD back in 1999, and it going nowhere.

ilamont 3y ago

In The Singularity Is Near (2005) Ray Kurzweil discussed an idea for the “Document Image and Storage Invention”, or DAISI for short, but concluded it wouldn't work out. I interviewed him a few years later about this and here's what he said: The big challenge, which I think is actually important almost philosophical challenge — it might sound like a dull issue, like how do you format a database, so you can retrieve information, that sounds pretty technical. The real key issue is that software formats are constantly changing. People say, “well, gee, if we could backup our brains,” and I talk about how that will be feasible some decades from now. Then the digital version of you could be immortal, but software doesn’t live forever, in fact it doesn’t live very long at all if you don’t care about it if you don’t continually update it to new formats. Try going back 20 years to some old formats, some old programming language. Try resuscitating some information on some PDP1 magnetic tapes. I mean even if you could get the hardware to work, the software formats are completely alien and [using] a different operating system and nobody is there to support these formats anymore. And that continues. There is this continual change in how that information is formatted. I think this is actually fundamentally a philosophical issue. I don’t think there’s any technical solution to it. Information actually will die if you don’t continually update it. Which means, it will die if you don’t care about it. ... We do use standard formats, and the standard formats are continually changed, and the formats are not always backwards compatible. It’s a nice goal, but it actually doesn’t work. I have in fact electronic information that in fact goes back through many different computer systems. Some of it now I cannot access. In theory I could, or with enough effort, find people to decipher it, but it’s not readily accessible. The more backwards you go, the more of a challenge it becomes. And despite the goal of maintaining standards, or maintaining forward compatibility, or backwards compatibility, it doesn’t really work out that way. Maybe we will improve that. Hard documents are actually the easiest to access. Fairly crude technologies like microfilm or microfiche which basically has documents are very easy to access. So ironically, the most primitive formats are the ones that are easiest.

krapp 3y ago
I'm certain that 100 years from now, when the collapse really gets rolling, we'll still have cuneiform clay tablets complaining about Ea-Nassir's shitty copper but most of the digital information and culture we've created and tried to archive will be lost forever. Eventually, we're going to lose the infrastructure and knowledge base we need to keep updating everything, people will be too busy just trying to find food and fighting off mutants from the badlands to care.

huehehue 3y ago

Fascinating read that unlocked some childhood memories. I'm secondhand pissed at the recovery company, I have a couple of ancient SD cards laying around and this just reinforces my fear that if I send them away for recovery they'll be destroyed (the cards aren't recognized/readable by the readers built into MacBooks, at least)

ryanjshaw 3y ago
Painful lesson I've learned myself the hard way - don't rush something that doesn't need to be rushed.
somat 3y ago
My understanding is that flash memory does not do very well at all for long term unpowered data retention. flash memory is basically a capacitor(it is not really a capacitor but close to one) and will loose it's charge after a few years. And magnetic drives will seize up. and optical disks get oxidized, and tapes stick together. long term archiving is a tricky endeavor. It is however an interesting challenge. I think I would get acquainted with the low level protocol used by sd cards. then modify a microcontroller sdmmc driver to get me an image of the card(errors and all). that is, without all the scsi disk layers doing their best to give you normalized access to the device. Or more realistically, hope someone more talented than me does the above.
- User23 3y ago
  Tapes hold up really well if they're not in absolutely awful storage conditions. And the claim at least was that the early CD-ROMs were quite durable, being a straight up laser carved diffraction grating. CDRs on the other hand rely on dye which will degrade rapidly.
  - somat 3y ago
    That is true about mag tape, I suspect tape to be one of the better choices for archival storage. In fact the biggest problem you will have with mag tape is making sure you have a working drive 10 - 20 years in the future when you want to look at your archives. to make things worse tape drives are getting more and more flimsy and fragile as tape tech advances. my first job we had a vault of ibm reel to reel tapes of old business data. our attitude if we were ever asked to pull any of the data was that we would probably only get one chance at it as the ferric material on on the tape had a disturbing tendency to flake off. note that there are techniques to reduce this, but we did not have the means or motivation to apply them. And a pedantic observation on your correct point about optical disks. you can't make backups on pressed disks, only recordable ones.

ogurechny 3y ago

Modern backup would simply state “API keys and settings are here:”, and a link to collaboration platform closed after 3 years of existence.

jandrese 3y ago
Hey, it's the cloud. Backups are "someone else's problem". That is until they are your problem, then you're up a creek.
- tivert 3y ago
  > Hey, it's the cloud. Backups are "someone else's problem". That is until they are your problem, then you're up a creek. The FSF used to sell these wonderful stickers that said "There is not cloud. It's just someone else's computer."
  - isaidthis 3y ago
    The sticker: https://static.fsf.org/nosvn/stickers/thereisnocloud.svg https://static.fsf.org/nosvn/stickers/thereisnocloud.svg "Stickers from various FSF campaigns - Print out copies of our stickers for your own uses, local conferences and more." https://www.fsf.org/resources/stickers https://www.fsf.org/resources/stickers
ilyt 3y ago
Honestly backup space is weirdly sparse for anything on enterprise scale. For anything more than few machines there is bacula/bareos (that pretends everything is tape with mostly miserable results), backuppc (that pretends tapes are not a thing, with miserable results), and that's about it, everything else seems to be point-to-point backups only with no real central management.
- somat 3y ago
  You can add amanda to the "pretends everything is tape with mostly miserable results" list.

LeoPanthera 3y ago

I really wish they would name the data recovery company so that I can never darken their door with my business.

nickt 3y ago
It’s not too hard to find with the following search, “we can recover data from tape formats including onstream”
- stepupmakeup 3y ago
  The OP explicity didn't name them (despite many people recommending to, even preservationists in this field on Reddit and Discord) but it's easy to find just by googling the text on the screenshots
  - ddtaylor 3y ago
    Name them and we can setup a thread or site to publicly shame them
    - stepupmakeup 3y ago
      the comment I replied to edited the link out https://www.datarecovery.net/tape-data-recovery.aspx https://www.datarecovery.net/tape-data-recovery.aspx
  - omoikane 3y ago
    Reddit thread: https://www.reddit.com/r/DataHoarder/comments/13q1pv7/playstation_game_frogger_2_source_code_recovered/ https://www.reddit.com/r/DataHoarder/comments/13q1pv7/playst...
bluedino 3y ago
> Over the span of about a month, I received very infrequent and vague communications from the company despite me providing extremely detailed technical information and questions. Ahh the business model of "just tell them to send us the tape and we'll buy the drive on eBay"

bsder 3y ago

Is there way to read magnetic tapes like these in such a way as to get the raw magnetic flux at high resolution? It seems like it would be easier to process old magnetic tapes by imaging them and then applying signal processing rather than finding working tape drives with functioning rollers. Most of the time, you're not worried about tape speed since you're just doing recovery read rather than read/write operations. So, a slow but accurate operation seems like it would be a boon for these kinds of things.

fifteen1506 3y ago
You still need to know where to look, the format, and using specialized equipment which cost wasn't driven down by mass manufacturing, so, in theory yes, in practice not. (Completely guessing here with absolute no knowledge of the real state of things)
iforgotpassword 3y ago
Sounds like at least in this case that ASIC in the drive was doing some (non trivial) signal processing. Would be interesting to know how hard it would be to get from the flux pattern back to zeros and ones. I guess with a working drive you can at least write as many test patterns as you want until you maybe figure it out.
- jandrese 3y ago
  At the very least the drive needs to be able to lock onto the signal. It's probably encoded in a helix on the drive and if the head isn't synchronized properly you won't get anything useful, even with a high sampling rate.
  - Clamchop 3y ago
    I would be surprised if it used helical recording. Data tape recorders rarely do because it's much more complex, increases tape wear, and the use cases don't usually demand that kind of linear bandwidth.

robotnikman 3y ago

I've always admired the tenacity of people who reverse engineer stuff. To be able to spend multiple months figuring out barely documented technologies with no promise of success takes a lot a willpower and discipline. It's something I wish I could improve more in myself.

detrites 3y ago
I think you could. In some sense "easily". It may be about finding that thing you're naturally so interested in or otherwise drawn to, that the months figuring out become a type of driven joy, and so the willpower kinda automatic. And if you find it, don't judge what it is or worry what others might think - or even necessarily tell anyone. Sometimes the most motivating things are highly personal, as with the OP; a significant part of their childhood.
- robotnikman 3y ago
  You definitely have a point there, looking at some of my previous work I was able to stick to projects for many months if I found the work interesting. I'll have to admit in the past 5 or so years any time I've tried to start a project there was always the thought in the back of my mind of 'will this benefit my career' or 'how can I make money on this in the future'. It seems having such thoughts adds additional anxiety to whenever I try and start to work on something for fun. Looks like that is what I need to start looking for again, projects which I find interesting or fun to do in my spare time, without thinking about how it would affect my career or trying to find ways to monetize it.
  - detrites 3y ago
    I totally get this. Something I'm learning - slowly - as it is so counter to sound ethic, is that making some work into pure fun helps the other work by preventing burn-out, which for me at least is an ongoing risk. Certainly feels better!

smokel 3y ago

Heh, I remember playing .mp3 files directly from QIC-80 tapes, somewhere around 1996. One tape could store about 120 MB, which is equal to about two compact discs' worth of audio. The noise of the tape drive was slightly annoying, though. And it made me appreciate what the 't' in 'tar' stands for.

mjaniczek 3y ago
Did you mean 1200 MB? That would make sense wrt. 2x CD capacity.
- smokel 3y ago
  No, it was really only 120 MB. I was referring to the length of an audio compact disc, not the capacity of a CD-ROM. At 128 kbps, you'd get about 2 hours of play time. Of course it didn't really make sense to use digital tapes for that use case, even back then. It was just for fun, and the article sparked some nostalgic joy, which felt worth sharing :)
- deleted 3y ago
  [deleted]
- stewarts 3y ago
  They reference MP3, and a CD ripped down to MP3 probably fits in the 50-100MB envelope for size. It has been a very long time since I last ripped an album, but that size jives with my memory.
- deleted 3y ago
  [deleted]

jimbob45 3y ago

F2 was a really neat game. It almost invented Crypt of the Necrodancer’s genre decades early. It’s a little sad that it took such a monumental effort to bring the source code back from the brink of loss. It’s times like that that should inspire lawmakers to void copyright in the case that the copyright holders can’t produce the thing they’re claiming copyright over.

xigency 3y ago

As a kid, I got this game as a gift and really, really wanted to play it. But after beating the second level, the game would always crash on my computer with an Illegal Operation exception. I remember sending a crash report to the developer, and even updating the computer, but I never got it working.

jakeinspace 3y ago
I adored this game as a kid, and I think I do have a faint memory of some stability issues, but I believe I was able to beat the game.

tombert 3y ago

This is giving me some anxiety about my tape backups. I have backed up my blu-ray collection to a dozen or so LTO-6 tapes, and it's worked great, but I have no idea how long the drives are going to last for, and how easy it will be to repair them either. Granted, the LTO format is probably one of the more popular formats, but articles like this still keep me up at night.

bombcar 3y ago
Do test restores. LTO is very good but without verification some will fail at some point. But your original bluray disk are also a backup.
EvanAnderson 3y ago
The only surefire method to keep the bits readable is to continue moving them onto new media every few years. Data has a built-in recurring cost. I'd love to see a solution to that problem but I think it's unlikely. It's a least possible, though, that we'll come up with a storage medium with sufficient density and durability that'll it'll be good enough. I don't even want to think about the hairy issues associated with keeping the bits able to be interpreted. That's a human behavior problem more than a technology problem.
wazoox 3y ago
LTO-7 drives read LTO-6, and will be available for quite a while. In 2016 I've used an LTO-3 drive to restore a bunch (150 or 200) of LTO-1/2 tapes from 2000-2003, and almost all but one or two worked fine.
antod 3y ago
LTO is one of the best choices for compatibility. I remember just how awful DDS (same sort of media as DAT) tape backups were - due to differences in head alignments, it was a real lottery as to whether any given tape could be read on a different drive than the one that wrote it.

crazygringo 3y ago

Wow, this part makes my blood boil, emphasis mine: > This issue doesn't affect tapes written with the ADR-50 drive, but all the tapes I have tested written with the OnStream SC-50 do NOT restore from tape unless the PC which wrote the tape is the PC which restores the tape. This is because the PC which writes the tape stores a catalog of tape information such as tape file listing locally, which the ARCserve is supposed to be able to restore without the catalog because it's something which only the PC which wrote the backup has, defeating the purpose of a backup. Holy crap. A tape backup solution that doesn't allow the tape to be read by any other PC? That's madness. Companies do shitty things and programmers write bad code, but this one really takes the prize. I can only imagine someone inexperienced wrote the code, nobody ever did code review, and then the company only ever tested reading tapes from the same computer that wrote them, because it never occured to them to do otherwise? But yikes.

Neil44 3y ago
I'm familiar with needing to re-index a backup if it's accessed from a 'foreign' machine and sometimes the procedure is non-obvious but just not having that option seems pretty bad.
- bluedino 3y ago
  I worked for an MSP a million years ago and we had a customer that thought they had lost everything. They had backup tapes but the backup server itself had died, after showing them the 'catalog tape' operation, and keeping their fingers crossed for a few hours, they bought me many beers.
  - EvanAnderson 3y ago
    I always had the Customer keep a written log of which tapes were used on which days. It helped for accountability but also prevented the "Oh, shit, we have to catalog all the tapes because the log of which tapes were used on which day are on the now-failed server."
Nextgrid 3y ago

omnibrain 3y ago

Is anyone else calling it “froggering/to frogger” if they have to cross a bigger street by foot without a dedicated crossing?

bluedino 3y ago

This will be fun in 20 years, trying recover 'cloud' backups from servers found in some warehouse.

Gigachad 3y ago
Those drives will all be encrypted and most likely shredded.
ilyt 3y ago
Nah it will be very simple: ....What do you mean "nobody paid for the bucket for last 5 years" ? There is some chance someone might stash old hard drive or tape with backup somewhere in the closet. There is no chance there will be anything left when someone stops paying for cloud.

dllthomas 3y ago

On the topic of Froggers, I enjoyed https://www.youtube.com/watch?v=FCnjMWhCOcA https://www.youtube.com/watch?v=FCnjMWhCOcA

readyplayernull 3y ago

A few months ago I was looking for an external backup drive and thought that SSD would be great because it's fast and shock resistant. Years ago I killed a Macbook Pro HD by throwing it on my bed from few inches high. Then I read a comment on Amazon about SSD losing information when unpowered for a long time. I couldn't find any quick confirmation in the product page, took me a few hours of research to find some paper about this phenomenon. If I remember correctly it takes a few weeks for the stored SSD to start losing its data. So I bought a mechanical HD. Another tech tip is not buying 2 backup devices from the same batch or even the same model. Chances being these will fail in the same way.

vidarh 3y ago
To the last bit, I've seen this first hand. Had a whole RAID array of the infamous IBM DeathStar drives fail one after the other while we frantically copied data off. Last time I ever had the same model drives in an array.
- smcameron 3y ago
  Heh, I remember in the early 1990s having a RAID array with a bunch of 4Gb IBM drives come up dead after a weekend powerdown for a physical move due to "stiction". I was on the phone with IBM, and they were telling me to physically bang the drives on the edge of desk to loosen them up. Didn't seem to be working, so their advice was "hit it harder!" When I protested, they said, "hey, it already doesn't work, what have you got to lose?" So I hit it harder. Eventually got enough drives to start up to get the array on line, and you better believe the first thing I did after that was create a fresh backup (not that we didn't have a recent backup anyway), and the 2nd thing I did was replace those drives, iirc, with Seagate Barracudas.
  - vidarh 3y ago
    Ouch. I knew someone who claimed to have dealt with that or a similar effect after their cleaning person had pulled the plug on their servers by putting the drive in an oven while connected and heating it slowly. Personally my most nailbiting period was when I got my first (20MB!) drive as a kid and it was too big an investment to replace even when it refused to spin up without me opening up the drive(!) and nudging the platter with my finger to help the motor spin it up... I backed everything up (on floppies), and stored everything important straight to floppies, but it was still more convenient to hold on to the HD for the next 6 months or so until I'd saved up enough to replace it... It's remarkable what drives can survive if you're lucky... Also remarkable how quickly that luck can run out, though. > "hey, it already doesn't work, what have you got to lose?" This attitude has saved me more than once. Recognising when you can afford to do things that seems ridiculous helps surprisingly often.

hlandau 3y ago

Absolutely amazing story. Fantastic! I've actually long been stunned by the propensity of proprietary backup software to use undocumented, proprietary formats. I've always found this quite stunning, in fact. It seems to me like the first thing one should make sure to solve when designing a backup format is to ensure it can be read in the future even if all copies of the backup software are lost. I may be wrong but I think some open source tape backup software (Amanda, I think?) does the right thing and actually starts its backup format with emergency restoration instructions in ASCII. I really like this kind of "Dear future civilization, if you are reading this..." approach. Frankly nobody should agree to use a backup system which generates output in a proprietary and undocumented format, but also I want a pony... It's interesting to note that the suitability of file formats for archiving is also a specialised field of consideration. I recall some article by someone investigating this very issue who argued formats like .xz or similar weren't very suited to archiving. Relevant concerns include, how screwed you are if the archive is partly corrupted, for example. The more sophisticated your compression algorithm (and thus the more state it records from longer before a given block), the more a single bit flip can result in massive amounts of run-on data corruption, so better compression essentially makes things worse if you assume some amount of data might be damaged. You also have the option of adding parity data to allow for some recovery from damage, of course. Though as this article shows, it seems like all of this is nothing compared to the challenge of ensuring you'll even be able to read the media at all in the future. At some point the design lifespan of the proprietary ASICs in these tape drives will presumably just expire(?). I don't know what will happen then. Maybe people will start using advanced FPGAs to reverse engineer the tape format and read the signals off, but the amount of effort to do that would be astronomical, far more even than the amazing effort the author here went to.

hlandau 3y ago
To add, thinking a bit more about it: Designing formats to be understandable by future civilizations actually reduces to a surprising degree to the same set of problems which METI has to face. As in, sending signals designed to be intelligible to extraterrestrials - Carl Sagan's Contact, etc. Even if you write an ASCII message directly to a tape, that data is obviously going to be encoded before being written to the tape, and you have no idea if anyone will be able to figure out that encoding in future. Trouble. What makes this particularly pernicious is the fact that LTO nowadays is a proprietary format(!!). I believe the spec for the first generation or two of LTO might be available, but last I checked, it's been proprietary for some time. The spec is only available to the (very small) consortium of companies which make the drives and media. And the number of companies which make the drives is now... two, I think? (They're often rebadged.) Wouldn't surprise me to see it drop to one in the future. This seems to make LTO a very untrustworthy format for archiving, which is deeply unfortunate.

PicassoCTs 3y ago

The author has fantastic endurance, what a marathon to get the files of the tape.

phkahler 3y ago

>> The tape was the only backup for those things, and it completes Frogger 2's development archives, which will be released publicly. In cases like this can imagine some company yelling "copyright infringement" even though they don't possess a copy themselves. It's a really odd situation.

FearNotDaniel 3y ago

> the ADR-50e drive was advertised as compatible, but there was a cave-at I'm assuming the use of "cave-at" means the author has inferred an etymology of "caveat" being made up of "cave" and "at", as in: this guarantee has a limit beyond which we cannot keep our promises, if we ever find ourselves AT that point then we're going to CAVE. (As in cave in, meaning give up.) I can't think of any other explanation of the odd punctuation. Really quite charming, I'm sure I've made similar inferences in the past and ended up spelling or pronouncing a word completely wrong until I found out where it really comes from. There's an introverted cosiness to this kind of usage, like someone who has gained a whole load of knowledge and vocabulary from quietly reading books without having someone else around to speak things out loud.

nocoiner 3y ago
I thought it might have been a transcription error of “carve out,” but your theory is more logical.
Kneesnap 3y ago
Dang it. OP here, I saw this typo and swear I fixed this typo before posting it!!

chrisstanchak 3y ago

I’ve been suffering through something similar with a DLT IV tape from 1999. Luckily I didn’t send out to the data recovery company. But still unsuccessful.

rootsudo 3y ago

Name and shame the company, you had a personal experience, you have proof. Name and shame. It helps nobody if you don't publicize it. Let them defend it, let them say whatever excuse, but your review will stand.

Kneesnap 3y ago
I don't want to even remotely tempt them to sue. They have no grounds, but I'm not taking risks-- companies are notorious for suing when they know they'll lose. Others who have posted it here have identified the right company though.

aidenn0 3y ago

TIL there are three completely different games named "Frogger 2" I assumed this was for the 1984 game, but this is for the 2000 game (there is also a 2008 game).

pimlottc 3y ago
Thanks for that, it seems like a surprisingly modern format for such an old game. Links for the games referenced: - Frogger II: ThreeeDeep! (1984) https://www.mobygames.com/game/7265/frogger-ii-threeedeep/ https://www.mobygames.com/game/7265/frogger-ii-threeedeep/ - Frogger 2: Swampy's Revenge (2000) [1] https://www.mobygames.com/game/2492/frogger-2-swampys-revenge/ https://www.mobygames.com/game/2492/frogger-2-swampys-reveng... - Frogger 2 (2008) [2] https://www.mobygames.com/game/47641/frogger-2/ https://www.mobygames.com/game/47641/frogger-2/

caycep 3y ago

At some point, I feel as if it may be easier just to rewrite the code from the ground up vs. going through all that computational archaeology.... Or in a few years, just have an AI write the code...

dabiged 3y ago

I work in the tape restoration space. My biggest piece of advice is never NEVER encrypt your tapes. If you think restoring data from an unknown format tape is hard, trying to do it when the drive will not let you read the blocks off the tape without a long lost decryption key is impossible.

ThorsBane 3y ago

> This is where the story should probably have stopped. Given up and called it a day, right? Maybe, but I care about this data, and I happen to know a thing or two about computers. Hahaha awwwww yeah :muscle:

userbinator 3y ago

As the other comment here says, any company claiming to do data recovery, and damaging the original media to that extent, should be named and shamed. I can believe that DR companies have generic drives and heads to read tapes of any format they come across, but even if they couldn't figure out how the data was encoded, there was absolutely no need to cut and splice the tape. I suspect they did that just out of anger at not likely being able to recover anything (and thus having spent a bunch of time for no profit.) Melted pinch rollers are not uncommon and there are plenty of other (mostly audio) equipment with similar problems and solutions --- dimensions are not absolutely critical and suitable replacements/substitutes are available. As an aside, I think that prominent "50 Gigabytes" capacity on the tape cartridge, with a small asterisk-note at the bottom saying "Assumes 2:1 compression", should be outlawed as a deceptive marketing practice. It's a good thing HDD and other storage media didn't go down that route.

kookamamie 3y ago

Someone was wise enough to erase the evidence in Party.

Kneesnap 3y ago
Nice catch! I think it was a little less juvenile than it might sound. I believe this was for a different game, Fusion Frenzy, which was a party minigame collection.
- MayeulC 3y ago
  While I didn't understand the parent you are replying to (not your answer), your mention of Fusion Frenzy caught my eye. I've had a soft spot for that game since spending hours playing the "xbox magazine" demo with a childhood friend. Could you clarify? Is there any history gem about that one? I'd dig a PC port!
  - Kneesnap 3y ago
    In one of the screenshots was an empty folder called "Party", with the commenter suggesting they deleted it before the backup as some way to hide it, but most likely it was just because it was for another game. I don't have much knowledge about Fusion Frenzy, but I am looking for options for archiving its development too, if that's even a possibility, which I'm not certain of yet.
    - MayeulC 3y ago
      Ah, that makes more sense, thank you for explaining. I would love to see some development archives, and source code for it one day.

sydbarrett74 3y ago

This is a masterful recovery effort. The README should be shared as an object lesson far and wide to every data restoration and archival service around.

dark-star 3y ago

I'm pretty sure that even with the substantial damage done by the recovery company, a professional team like Kroll Ontrack can still recover the complete tape data, although it probably won't be cheap.

Const-me 3y ago

CD-R drives were already common in 2001: https://en.wikipedia.org/wiki/CD-R https://en.wikipedia.org/wiki/CD-R I wonder would a CD-R disk retain data for these 22 years?

Tangurena2 3y ago
Only if you kept the disk in a refrigerator. Bits are stored by melting the plastic slightly and the dye seeping in. Over time, the warmth of "room temperature" will cause the pits to become less well-defined so the decoder has to spend more time calculating "well, is that really a 1 or is it a sloppy 0". There's a lot of error detection/correction built into the CD specs, but eventually, there will be more error than can be corrected for. If you've ever heard the term "annealing" when used in machine learning, this is equivalent. Living in South Florida, ambient temperatures were enough to erase CD-Rs - typically in less than a year. I quickly started buying the much more expensive "archival" discs, but that wasn't enough. One fascinating "garage band" sold their music on CD-Rs and all of my discs died (it was a surfer band from Alabama).
- Clamchop 3y ago
  The recording is made in the dye layer, a chemical change, and the dye degrades (particularly in sunlight) so the discs have a limited shelf life. Checking Wikipedia, it appears azo dye formulations can be good for tens of years. Melting polycarbonate would call for an absurdly powerful laser, a glacial pace, or both, and you wouldn't have to use dye at all. I'd guess such a scheme would be extremely durable, though.
hakfoo 3y ago
I tried a couple of CD-Rs that were stored in a dry closed drawer for most of the last 20 years recently, and they seemed to at least initially come up. Now I can reinstall Windows 2000 with Service Pack 4 slipstreamed!

jasomill 3y ago

Yet, despite ARCserve showing a popup which says "Restoration Successful", it restores up to the first 32KB of every file on the tape, but NO MORE." From 10,000 feet, this sounds suspiciously like ARCserve is reading a single tape block or transfer buffer's worth of data for each file, writing out the result, then failing and proceeding to the next file. Success popup notwithstanding, I'd expect to find errors in either the ARCserve or Windows event logs in this case — were there none? While it's been decades since I've dealt with ARCserve specifically, I've seen similar behavior caused by any number of things. Off the top of my head, (1) Incompatibilities between OS / backup software / HBA driver / tape driver. In particular, if you're using a version of Windows much newer than Windows 2000, try a newer version of ARCserve. In the absence of specific guidance, I'd probably start with the second* ARCserve version that officially supports Windows Server 2003: (a) Server 2003 made changes to the SCSI driver architecture that may not be 100% compatible with older software. (b) The second release will likely fix any serious Server 2003 feature-related bugs the first compatible version may have shipped with, without needing to install post-release patches that may be hard to find today. (b) Significantly newer ARCserve versions are more likely to introduce tape drive / tape format incompatibilities of their own. (2) Backup software or HBA driver settings incompatible with the hardware configuration (e.g., if ARCserve allows it, try reducing the tape drive transfer buffer size or switching from fixed block (= multiple tape blocks per transfer) to variable block (= single tape block per transfer) mode; if using an Adaptec HBA, try increasing the value of /MAXIMUMSGLIST[1]). (3) Shitty modern HBA driver support for tape (and, more generally, non-disk) devices. For example, modern Adaptec Windows HBA drivers have trouble with large tape block sizes that AFAIK cannot be resolved with configuration changes (though 32 kB blocks, as likely seen here, should be fine). In my experience with PCIe SCSI HBAs, LSI adapters are more likely to work with arbitrary non-disk devices and software out-of-the-box, whereas Adaptec HBAs often require registry tweaks for "unusual" circumstances (large transfer sizes; concurrent I/O to >>2 tape devices; using passthrough to support devices that lack Windows drivers, especially older, pre-SCSI 2 devices), assuming they can be made to work at all. LSI20320IE PCIe adapters are readily available for $50 or less on eBay and, in my experience, work well for most "legacy" applications. (To be fair to Adaptec, I've had nothing but good experiences using their adapters for "typical" applications: arbitrary disk I/O, tape backup to popular drive types, CD/DVD-R applications not involving concurrent I/O to many targets, etc.) (4) Misconfigured or otherwise flaky SCSI bus. In particular, if you're connecting a tape drive with a narrow (50-pin) SCSI interface to a wide (68-pin) port on the HBA, make sure the entire bus, including the unused pins, are properly terminated. The easiest way to ensure this is to use a standard 68-pin Ultra320 cable with built-in active LVD/SE termination, make sure termination is enabled on the HBA, disabled on the drive, that the opposite end of the cable from the built-in terminator is connected to the HBA, and, ideally, that the 68-to-50-pin adapter you're using to connect the drive to the cable is unterminated. You can also use a 50-pin cable connected to the HBA through a 68-to-50-pin adapter, but then you're either relying on the drive properly terminating the bus — which it may or may not do — or else you need an additional (50-pin) terminator for the drive end, which will probably cost as much as a Ultra320 cable with built-in termination (because the latter is a bog-standard part that was commonly bundled with both systems and retail HBA kits). Note that I have seen cases where an incorrect SCSI cable configuration works fine in one application, but fails spectacularly in another, seemingly similar application, or even the same application if the HBA manages to negotiate a faster transfer mode. While this should be far less likely to occur with a modern Ultra160 or Ultra320 HBA, assume nothing until you're certain the bus configuration is to spec (and if you're using an Ultra2 or lower HBA, consider replacing it). With all that said, reversing the tape format may well be easier than finding a compatible OS / ARCserve / driver / HBA combination. In any case, good job with that, and thanks for publishing source code! [1] http://download.adaptec.com/pdfs/readme/relnotes_29320lpe.pdf#page=4 http://download.adaptec.com/pdfs/readme/relnotes_29320lpe.pd...

GnarfGnarf 3y ago

DVDs should not be overlooked for backup. The Millennium type have been simulated to withstand 1,000 years.

ddingus 3y ago

This is just random, but reading this and the backup discussion made me think about SGI IRIX and how it could do incremental backups. One option was to specify a set of files, and that spec could just be a directory. Once done, the system built a mini filesystem and would write that to tape. XFS was the filesystem in use at the time I was doing systems level archival. On restores, each tape, each record was a complete filesystem. One could do it in place and literally see the whole filesystem build up and change as each record was added. Or, restore to an empty directory and you get whatever was in that record. That decision was not as information dense as others could be, but it was nice and as easy as it was robust. What our team did to back up some data managed engineering software was perform a full system backup every week, maybe two. Then incrementals every day, written twice to the tape. Over time, full backups were made and sent off site. One made on a fresh tape, another made on a tape that needed to be cycled out of the system before it aged out. New, fresh tapes entered the cycle every time one aged out. Restores were done to temp storage and rather than try and get a specific file, it was almost always easier to just restore the whole filesystem and then copy the desired file from there into its home location. The incrementals were not huge usually. Once in a while they got really big due to some maintenance type operation touching a ton of files. The nifty thing was no real need for a catalog. All one needed was the date to know which tapes were needed. Given the date, grab the tapes, run a script and go get coffee and then talk to the user needing data recovery to better understand what might be needed. Most of the time the tapes were read and the partial filesystem was sitting there ready to go right about the time those processes completed. Having each archive, even if it were a single file, contain a filesystem data set was really easy to use and manage. Loved it.