Skip to main content

MinIO

MinIO is Lumie's shared object store for product uploads, generated artifacts, Vault backend storage, and several backup flows. It is bootstrap-owned because later GitOps applications depend on it existing first.

Source paths​

  • lumie-infra/bootstrap/minio/argocd.yaml
  • lumie-infra/bootstrap/minio/helm-values.yaml
  • lumie-infra/bootstrap/minio/common-values.yaml
  • lumie-infra/bootstrap/minio/manifests/r2-barman-vss.yaml
  • lumie-infra/bootstrap/minio/manifests/r2-snapshot-cronjob.yaml
  • lumie-infra/bootstrap/minio/manifests/upload-proxy-ingress.yaml
  • lumie-infra/provision/ansible/roles/storage-setup/tasks/main.yml
  • lumie-infra/security/teleport/agent/helm-values.yaml

Runtime shape​

What is deployed​

  • Distributed MinIO with replicas: 4 and drivesPerNode: 1
  • One local PV per node, all using the minio-local StorageClass
  • Separate minio-ui console deployment rendered from charts/common
  • VaultStaticSecret refresh for the root credential secret
  • A twice-daily snapshot CronJob that mirrors selected buckets to R2
  • A Traefik IngressRoute that only allows PUT and OPTIONS to presigned upload paths

Key contracts​

  • Local disk paths come from common.persistentVolumes in lumie-infra/bootstrap/minio/common-values.yaml.
  • Bucket lifecycle is enforced with this post-install command:
customCommands:
  - command: ilm rule add myminio/lumie --prefix "tmp/" --expire-days 1

Source path: lumie-infra/bootstrap/minio/helm-values.yaml

  • Direct browser uploads are constrained to temporary prefixes:
- match: Host(`lumie-edu.com`) && PathPrefix(`/lumie/tmp/`) && (Method(`PUT`) || Method(`OPTIONS`))
- match: Host(`dev.lumie-infra.com`) && PathPrefix(`/lumie-dev/tmp/`) && (Method(`PUT`) || Method(`OPTIONS`))

Source path: lumie-infra/bootstrap/minio/manifests/upload-proxy-ingress.yaml

Access boundaries​

  • Application traffic uses the internal S3-compatible endpoint http://minio.minio.svc:9000.
  • The console is exposed through the Teleport app named minio, which proxies http://minio-ui.minio.svc.cluster.local:9090 and rewrites the host to minio.lumie-infra.com.
  • Standard Kubernetes ingress is disabled for both the API and console; the upload proxy is the only public HTTP route defined in the repo.

Backup and replication flow​

  • CNPG-related and other object consumers write into MinIO buckets.
  • minio-r2-snapshot runs at 0 4,16 * * * in the Asia/Seoul time zone.
  • The snapshot job mirrors the lumie, lumie-dev, and vault buckets into r2/lumie-dr/snapshots/<bucket>/<timestamp>/.

Failure modes​

  • Local PV loss or a missing node breaks distributed erasure-coded capacity and may trigger long healing or reduced durability.
  • If the minio-root-password secret stops refreshing, the MinIO StatefulSet and the snapshot CronJob drift from the expected credentials.
  • The upload proxy only allows tmp/ prefixes and PUT or OPTIONS; using permanent object keys or other HTTP verbs fails by design.
  • The tmp/ ILM rule automatically deletes temporary objects after one day, so workflows that expect those objects to be durable will fail later.

Verification​

kubectl get applications.argoproj.io -n argocd minio
kubectl get pods -n minio
kubectl get pvc,pv -n minio
kubectl get cronjobs -n minio
kubectl get jobs -n minio | rg minio-r2-snapshot
kubectl get ingressroutes.traefik.io -n minio

Success means the minio Application is Synced/Healthy, StatefulSet pods are Ready, PVCs are Bound, the snapshot CronJob exists, recent snapshot Jobs are visible, and Traefik exposes the repo-managed upload proxy route.

Observability​

  • The MinIO Helm values enable a ServiceMonitor with the release: prometheus label.
  • Grafana keeps a MinIO dashboard JSON in lumie-infra/observability/grafana/dashboards/minio.json.
  • Prometheus rule files also include MinIO replication alerts in lumie-infra/observability/prometheus/helm-values.yaml.