A backup that has never been restored is a hypothesis. It is usually a reasonable hypothesis, and occasionally it is a very expensive one.
The failures are boring, which is exactly why they survive so long. Nobody is outwitted by a backup system. They are let down by one, in one of about five ways, and every one of them is discovered on the day it matters.
The five
1
The job stopped running and nothing said so
2
It ran, and wrote an empty file every night
3
The copy was reachable by whatever took out production
4
Retention deleted it before anyone noticed the problem
5
The file exists, and nothing on hand can open it
The first four are operational and mostly solvable with care. The fifth is architectural, and it is the one worth designing against, because it is the only one where doing everything else right does not save you.
Number five is the interesting one
A backup you can only open through a vendor is not a backup of your data. It is a backup of your relationship with that vendor. If they are acquired, change their pricing, have an outage on the day you need them, or simply go away, the bytes may still exist and be of no use to you at all.
What we do about it
An artifact records its own format, compression and encryption. Opening one by hand uses tools that are already on your machine:
gpg --decrypt prod-db-2026-08-04.sql.gz.gpg \
| gunzip \
> prod-db.sqlThat is the whole procedure. There is no saved.sh container, no custom codec, and no metadata sidecar you need from our API to make sense of the bytes.
Our CLI has a restore command and it is genuinely more convenient. It is also entirely optional, and that is the point. Convenience that becomes a dependency is not convenience.
The first four, briefly
None of them are clever, and all of them need a system rather than a script.
- Silent stoppage. A durable engine records that a run was expected. A cron entry that stops firing produces no signal at all, because there is nothing watching for the absence of one.
- Empty files. Every run is checksummed and sized, and a backup that suddenly produces 4 KB where it produced 4 GB yesterday is visible in the run history rather than discovered in a year.
- Shared blast radius. Covered at length elsewhere: if the copy answers to the same credentials as the original, it is not a second copy.
- Premature deletion. A hold is stamped at write time and can be extended but never shortened, so loosening the policy cannot reach back to what is already stored.
The property that matters
Every one of these is really the same question asked five ways: when this system is gone, or broken, or compromised, is the data still recoverable by someone who has only the file?
If the answer is yes, the other four failures are inconveniences. If the answer is no, then every other feature is decoration on a hypothesis.