Skip to main content

Vault Circular Dependency Bootstrap

This page is an incident retrospective for the January 13, 2026 Vault recovery sequence. It records the bootstrap failure modes that mattered, the recovery order that worked, and the places where the current repo has already moved on to a different steady state.

What failed​

The primary outage was a closed bootstrap dependency:

Representative shape:

Vault -> PostgreSQL -> ExternalSecrets -> Vault

Vault needed PostgreSQL, PostgreSQL credentials were being sourced through ExternalSecrets, and ExternalSecrets could not reconcile until Vault was healthy. Once Vault stopped starting cleanly, the stack had no cold-start path.

The same work window also exposed four related failures:

  • PostgreSQL-backed Vault HA migration was attempted without the required Vault storage tables.
  • vault operator init was re-run, which made the old unseal keys and root token unusable and forced manual secret restoration from backups.
  • Vault OIDC login against Authelia failed because the configured user_claim expected preferred_username, which was no longer present in the ID token.
  • A broad master-node DNAT rule for ports 80 and 443 intercepted in-cluster Kubernetes API traffic and caused ClusterSecretStore to report InvalidProviderConfig.

Confirmed causes​

  • The bootstrap graph was cyclic. Vault could not come up without credentials that ultimately depended on Vault.
  • PostgreSQL storage required pre-created vault_kv_store and vault_ha_locks tables. Vault did not create that schema automatically during the failed migration attempt.
  • Re-running vault operator init was destructive in practice because it generated a new initialization state and severed access to previously stored data.
  • Authelia's claim behavior changed, so a Vault role that depended on preferred_username stopped matching.
  • Host-level DNAT on the master node was too broad and redirected traffic that should have stayed inside the cluster service network.

Resolution sequence​

  1. Break the bootstrap cycle by moving Vault onto a non-cyclic storage path so Vault could start without waiting on ExternalSecrets.
  2. Recreate the Vault auth and KV mounts needed by downstream secret consumers.
  3. Restore more than twenty secrets manually from backup material.
  4. Change the Vault OIDC role mapping so login no longer depended on the missing Authelia claim.
  5. Remove the broad NAT behavior so Kubernetes API calls from in-cluster clients stopped being redirected away from the API server.

The important lesson was order, not only correctness. Until Vault could start independently, every downstream secret consumer remained blocked.

What the current repo still proves​

The checked-in Vault contract is no longer the temporary remediation from the incident. The current repo shows standalone Vault with an external configuration secret mounted into the chart:

server:
  standalone:
    enabled: true
  ha:
    enabled: false
  extraArgs: "-config=/vault/userconfig/extraconfig-from-values.hcl"

Source: lumie-infra/bootstrap/vault/helm-values.yaml

The current repo also shows that Vault OIDC remains a runtime procedure rather than a Helm value:

  • lumie-infra/bootstrap/vault/OIDC-SETUP.md
  • lumie-infra/security/keycloak/common-values.yaml
  • lumie-infra/security/teleport/agent/helm-values.yaml

Drift to note​

The January 2026 incident note records a temporary file-backend recovery to break the cycle. The current repo no longer preserves that exact state. It now documents standalone Vault with config injected from vault-config-secret, and the OIDC setup guide is written for the later Keycloak-based flow rather than the Authelia-era claim workaround.

That drift is real and should be called out explicitly:

  • the incident page is the source of truth for what failed and how recovery was ordered
  • the checked-in manifests are the source of truth for the later steady-state contract

Lasting lessons​

  • Treat bootstrap dependencies between secret stores, databases, and secret renderers as a directed graph. If the graph contains a cycle, cold start is already broken.
  • Treat vault operator init as a one-time operation with backup and peer review around it. It is not a safe retry button.
  • Keep host-level DNAT rules narrow enough that cluster-internal traffic cannot match them accidentally.
  • Re-check OIDC claim mappings after every IDP upgrade, even when the version bump looks minor.
  • Backup and restore must both be automated. Manual recovery only proved the data model could be reconstructed, not that the recovery path was safe.

Verification​

cd Lumie
rg -n "standalone:|ha:|vault-config-secret|extraArgs" \
  lumie-infra/bootstrap/vault/helm-values.yaml
rg -n "vault-backend|ClusterSecretStore|vault.vault.svc.cluster.local" \
  lumie-infra/bootstrap/vault/common-values.yaml
rg -n "auth/oidc|preferred_username|groups_claim|default_role" \
  lumie-infra/bootstrap/vault/OIDC-SETUP.md

Success means the repo still shows standalone Vault, a repo-managed Vault auth surface, and a runtime OIDC procedure separate from Helm values. That verifies the current contract and also makes the historical drift visible.