Skip to main content

MinIO R2 Backup DR

The inspected evidence shows a design-correction sequence on April 15, 2026 rather than a single outage record. The end state is an immutable, timestamped R2 snapshot flow for MinIO buckets plus a separately scheduled CNPG base backup for infra-db.

Source paths​

  • lumie-infra/bootstrap/minio/manifests/r2-snapshot-cronjob.yaml
  • lumie-infra/bootstrap/minio/manifests/r2-barman-vss.yaml
  • lumie-infra/storage/infra-db/manifests/scheduled-backup.yaml
  • lumie-infra/storage/infra-db/manifests/r2-barman-vss.yaml
  • lumie-infra/observability/prometheus/helm-values.yaml

What changed​

CommitChangeWhy it did not remain the final design
a920ac08MinIO server-side replication to R2R2 did not satisfy the full S3 replication contract expected by MinIO
112bf594Per-bucket R2 targets for replicationPrefix targets were not supported, and the replication path was still blocked by R2 semantics
e2e842eamc mirror CronJob every 15 minutesBetter than replication, but still wrote back into mutable paths
3385be4eRemoved vault, kept zot, raised memoryzot listing was expensive and vault needed different durability semantics
851e2e6fRemoved zot, kept vault, lowered memoryCost and restore-value tradeoff changed again
06253107Removed vault from mirror scopeVault rewrites objects during normal operation, which conflicts with Compliance Lock
945cd24aSwitched to timestamped snapshots and aligned 04:00 KST backup intentThis is the shape that matches the current repo
34d35077Converted CNPG CRON_TZ to plain UTCCNPG admission rejected the time-zone-prefixed schedule

Final repo-managed contract​

The current MinIO DR path is a CronJob named minio-r2-snapshot:

spec:
  timeZone: "Asia/Seoul"
  schedule: "0 4,16 * * *"
...
  BUCKETS="lumie lumie-dev vault"
...
    TARGET="r2/lumie-dr/snapshots/$b/$TS/"
    mc mirror --quiet "local/$b" "$TARGET"

Source: lumie-infra/bootstrap/minio/manifests/r2-snapshot-cronjob.yaml

That design avoids overwriting existing R2 objects. Every run writes a fresh timestamped prefix, which is compatible with Compliance Lock even for mutable stores such as Vault.

The infra-db side is separate and intentionally uses CNPG ScheduledBackup:

spec:
  # 04:00 KST = 19:00 UTC (CNPG ScheduledBackup does not accept CRON_TZ prefix)
  schedule: "0 0 19 * * *"

Source: lumie-infra/storage/infra-db/manifests/scheduled-backup.yaml

Both paths depend on the same Vault path, rendered into namespace-local r2-barman-creds secrets:

destination:
  create: true
  name: r2-barman-creds

Source: lumie-infra/storage/infra-db/manifests/r2-barman-vss.yaml

Lessons that survived into the current repo​

  • Treat Cloudflare R2 as an immutable snapshot target here, not as a drop-in MinIO replication peer.
  • Keep Vault object storage out of overwrite-based mirror designs. Timestamped snapshots are the compatible pattern in the checked-in manifests.
  • CNPG ScheduledBackup needs a plain UTC cron expression. The checked-in KST intent lives only in the comment.
  • Backups for MinIO buckets and PostgreSQL are parallel DR surfaces, not one shared controller.

Observability drift to keep in mind​

The current Prometheus rules still describe the old design:

- name: minio-replication.rules
...
  - alert: MinIOReplicationLag
...
  - alert: MinIOReplicationFailed

Source: lumie-infra/observability/prometheus/helm-values.yaml

That no longer matches the active minio-r2-snapshot CronJob. The inspected repo still carries replication-oriented rule names and descriptions even though the backup mechanism is now timestamped snapshots using r2-barman-creds.

Verification​

kubectl get cronjob -n minio minio-r2-snapshot
kubectl get jobs -n minio | rg minio-r2-snapshot
kubectl get scheduledbackup.postgresql.cnpg.io -n infra-db infra-db-daily -o yaml
kubectl get secret -n minio r2-barman-creds
kubectl get secret -n infra-db r2-barman-creds

Success means the snapshot CronJob exists, recent Jobs are present, infra-db-daily renders the UTC 19:00 schedule, and the shared R2 credential secret exists in the namespaces that consume it.

  • MinIO operations reference
  • Infra DB operations reference
  • Prometheus operations reference