---
title: "The restore nobody tested"
description: "Backups fail quietly and restores fail loudly, always at the worst possible moment. Here is the short list of things that break, and the one property that makes them survivable."
url: "https://saved.sh/blog/the-restore-nobody-tested"
date: "2026-08-04"
author: "saved.sh"
tag: "Engineering"
---

A backup that has never been restored is a hypothesis. It is usually a
reasonable hypothesis, and occasionally it is a very expensive one.

The failures are boring, which is exactly why they survive so long. Nobody is
outwitted by a backup system. They are let down by one, in one of about five
ways, and every one of them is discovered on the day it matters.

## The five [#the-five]

<Stats>
  <Stat value="1" label="The job stopped running and nothing said so" />

  <Stat value="2" label="It ran, and wrote an empty file every night" />

  <Stat value="3" label="The copy was reachable by whatever took out production" />
</Stats>

<Stats>
  <Stat value="4" label="Retention deleted it before anyone noticed the problem" />

  <Stat value="5" label="The file exists, and nothing on hand can open it" />
</Stats>

The first four are operational and mostly solvable with care. The fifth is
architectural, and it is the one worth designing against, because it is the only
one where doing everything else right does not save you.

## Number five is the interesting one [#number-five-is-the-interesting-one]

A backup you can only open through a vendor is not a backup of your data. It is a
backup of your relationship with that vendor. If they are acquired, change their
pricing, have an outage on the day you need them, or simply go away, the bytes
may still exist and be of no use to you at all.

<Aside title="The test">
  Ask any backup vendor you are evaluating to show you the documented procedure
  for opening one of their artifacts without their software. If they cannot, you
  have learned something more useful than anything on their pricing page.
</Aside>

## What we do about it [#what-we-do-about-it]

An artifact records its own format, compression and encryption. Opening one by
hand uses tools that are already on your machine:

```bash
gpg --decrypt prod-db-2026-08-04.sql.gz.gpg \
  | gunzip \
  > prod-db.sql
```

That is the whole procedure. There is no saved.sh container, no custom codec, and
no metadata sidecar you need from our API to make sense of the bytes.

<Figure caption="Two routes out. The lower one does not involve us at any point, and it is documented rather than implied.">
  <RecoveryPath />
</Figure>

Our CLI has a restore command and it is genuinely more convenient. It is also
entirely optional, and that is the point. Convenience that becomes a dependency
is not convenience.

## The first four, briefly [#the-first-four-briefly]

None of them are clever, and all of them need a system rather than a script.

* **Silent stoppage.** A durable engine records that a run was expected. A cron
  entry that stops firing produces no signal at all, because there is nothing
  watching for the absence of one.
* **Empty files.** Every run is checksummed and sized, and a backup that
  suddenly produces 4 KB where it produced 4 GB yesterday is visible in the run
  history rather than discovered in a year.
* **Shared blast radius.** Covered at length elsewhere: if the copy answers to
  the same credentials as the original, it is not a second copy.
* **Premature deletion.** A hold is stamped at write time and can be extended but
  never shortened, so loosening the policy cannot reach back to what is already
  stored.

<Aside title="What to actually do this week" tone="accent">
  Pick your most important database. Restore last night's backup into a scratch
  environment and diff the row counts. Whatever you find, you will know something
  you did not know on Monday.
</Aside>

## The property that matters [#the-property-that-matters]

Every one of these is really the same question asked five ways: &#x2A;*when this
system is gone, or broken, or compromised, is the data still recoverable by
someone who has only the file?**

If the answer is yes, the other four failures are inconveniences. If the answer
is no, then every other feature is decoration on a hypothesis.