S3-compatible storage

Lists a bucket or prefix and streams every object into a single tar. Available as both a local and a cloud backup, and it needs no external tool: it speaks S3 directly.

Any S3-compatible store works: AWS S3, Cloudflare R2, MinIO, Backblaze B2, Wasabi, DigitalOcean Spaces.

What is taken

Every object under the prefix, with its key as the tar entry name and its last-modified time preserved.

Keys ending in / are skipped: they are directory placeholders with no bytes.

The archive is streamed rather than staged, so backing up a 400 GB bucket does not need 400 GB of scratch space for the listing pass. Only the object being copied is in flight.

Fields

The same keys for both kinds. Local, in the worker's config.yaml:

Cloud, on the backup definition. Every field goes to the vault together and none is returned by any endpoint:

backups:
  "<backup-uuid>":
    source_type: s3
    source:
      bucket: uploads
      endpoint: minio.internal:9000
      access_key_id: "<key>"
      secret_access_key: "<secret>"
      use_path_style: true

The credential needs ListBucket on the bucket and GetObject on the prefix. It never needs write access, so do not grant any.

Objects that change mid-copy

A tar header declares the size before the bytes are written, and it cannot be corrected afterwards. If an object is rewritten between the listing and the copy, its real size no longer matches the header.

The run fails rather than truncating the file. The alternative would be an archive that looks fine and contains a corrupt member, discovered during a restore.

uploads/report.pdf changed while being copied: listed 481920 bytes, read 502133

If this happens regularly, the prefix is actively written and a backup of it will never be internally consistent. Back up a prefix that is append-only, or accept the failure rate and retry.

An empty bucket or prefix fails with EmptySource, rather than producing an empty archive you would mistake for a working backup.

This is not delivery

Two different features look similar and are easy to confuse.

You can use both at once: back up bucket A as a source, and deliver the resulting artifact to bucket B as a destination. Do not point a backup's source and its destination at the same bucket and prefix.

See Delivery.

Sizing

The tar is the sum of the objects plus a small per-entry overhead. Compression depends entirely on the contents: a bucket of JSON or CSV compresses well, a bucket of JPEGs and MP4s does not compress at all and costs CPU to try.

Large buckets are the case where a local backup is worth preferring even when the store is reachable from anywhere: the copy happens next to the data instead of across the internet.

Restoring

sctl restore file <artifact-id> --path ./uploads.tar
tar -tvf ./uploads.tar          # inspect
tar -xf ./uploads.tar -C ./out  # extract

Then put the objects back with whatever tooling you normally use:

aws s3 sync ./out s3://uploads/

Restoring is deliberately not automatic. Writing several hundred thousand objects back into a live bucket is not something a backup tool should do on your behalf.