5 ms·

> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restore

> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with data?

deleted 3y ago
[deleted]
depereo 3y ago
Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.
- VectorLock 3y ago
  Sp they were manually doing a backup and then testing that backup, rather than testing their automated backups? If thats what they were doing that just makes me wonder... why?
  - fbdab103 3y ago
    I am not sure about the pricing structure of their provider, but maybe it is a Hotel California situation? Data egress price of the provider was such that they did not want to pay to retrieve the full backup "just for a test"? Instead, repeat the backup command, but swap it out for a local location.
  - depereo 3y ago
    It's possible that the backup restore/test process was still 'automated' to some degree, like a manually-run pipeline or playbook etc. Obviously there were mistakes made in process design and tools deployment but I don't think this is 'baffling incompetence' more than it is a couple of small mistakes that compounded each other to have a pretty spectacular impact.
healsdata 3y ago
During their tests, they were creating new backups and testing a restore with them. They're weren't actually testing their real backups.
upon_drumhead 3y ago
When I started at a large company a few years back, the company specified restoration test was just that the archive restored successfully onto a server, not that there was anything actually in the archive. Digging into it, the archives were all empty due to a commit a few years previous that added an incorrect exclude option that ended up excluding all files. They were running for years on the cusp of total failure and had automated restoration tests that caused a false sense of security in the tooling. The second thing I did was adjust the restoration tooling to validate data existence and over time added validation tests (percent of data matched current live systems, specific fields and values were there, etc). It's just too easy to screw up, doubly so when time constrained and alone doing the best you can without any oversight.
- andruby 3y ago
  That’s one of the reasons I started tracking the filezise of the archive, and monitoring that over time. Datadog will trigger an alert if our backups are suddenly X% smaller.
- anonacct37 3y ago
  Empty backups or exponentially I creasing in size backups of backups. The yin and yang of backup bugs.

deleted 3y ago

[deleted]

depereo 3y ago

Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.

VectorLock 3y ago
Sp they were manually doing a backup and then testing that backup, rather than testing their automated backups? If thats what they were doing that just makes me wonder... why?
- fbdab103 3y ago
  I am not sure about the pricing structure of their provider, but maybe it is a Hotel California situation? Data egress price of the provider was such that they did not want to pay to retrieve the full backup "just for a test"? Instead, repeat the backup command, but swap it out for a local location.
- depereo 3y ago
  It's possible that the backup restore/test process was still 'automated' to some degree, like a manually-run pipeline or playbook etc. Obviously there were mistakes made in process design and tools deployment but I don't think this is 'baffling incompetence' more than it is a couple of small mistakes that compounded each other to have a pretty spectacular impact.

healsdata 3y ago

During their tests, they were creating new backups and testing a restore with them. They're weren't actually testing their real backups.

upon_drumhead 3y ago

When I started at a large company a few years back, the company specified restoration test was just that the archive restored successfully onto a server, not that there was anything actually in the archive. Digging into it, the archives were all empty due to a commit a few years previous that added an incorrect exclude option that ended up excluding all files. They were running for years on the cusp of total failure and had automated restoration tests that caused a false sense of security in the tooling. The second thing I did was adjust the restoration tooling to validate data existence and over time added validation tests (percent of data matched current live systems, specific fields and values were there, etc). It's just too easy to screw up, doubly so when time constrained and alone doing the best you can without any oversight.

andruby 3y ago
That’s one of the reasons I started tracking the filezise of the archive, and monitoring that over time. Datadog will trigger an alert if our backups are suddenly X% smaller.
anonacct37 3y ago
Empty backups or exponentially I creasing in size backups of backups. The yin and yang of backup bugs.