Most homegrown backup setups produce a directory. Files with dates in the names, sorted newest first, and a rough sense that things are fine because the directory is not empty.
A directory answers one question: did something get written. It cannot answer the three that matter when you are staring at it during an incident.
1
Is this file complete, or did the dump die halfway?
2
Is it the size it should be, or the size of an error message?
3
Which run produced it, and did that run succeed?
The empty-file failure
This is the one that gets people, and it is worth describing precisely because it is so undramatic.
pg_dump "$DATABASE_URL" | gzip > backup.sql.gzIf pg_dump fails, it writes an error to stderr and exits non-zero. But the
pipeline's exit status is gzip's, and gzip succeeded: it compressed zero
bytes into a small, perfectly valid archive. The file exists. It has today's
date. It is 20 bytes.
Nothing about the directory listing looks wrong. It will not look wrong tomorrow either, or in eight months when you need it.
What we record instead
Every run produces a record, and the artifact is measured rather than assumed.
The record carries what the run did, how long it took, the bytes that arrived, the checksum of what was sealed, and which worker executed it. That turns the three unanswerable questions into a table lookup.
A backup that produced 4 KB where it produced 4.2 GB yesterday is visible in the run history the next morning, not in a year. Not because anything clever is inspecting the contents, but because the number is written down next to the number from last time.
Absence is a signal too
The subtler thing a run record gives you is the ability to notice a run that did not happen.
A cron entry that stops firing produces nothing: no file, no error, no log line, because the thing that would have written them never started. There is no artifact of its absence. A schedule that expects a run can tell the difference between "ran and failed" and "never ran", and those need different responses.
What it is not
A run record is not verification that the backup restores. Nothing short of restoring it is that, and we would not claim otherwise. What it does is remove the failures that are detectable without a restore, which is most of them, and leave you with the one honest remaining question.
The directory was never lying to you. It just was not saying anything.