Keycloak OIDC SSO Journey

Symptom

Use this guide when onboarding an infrastructure service to Keycloak OIDC and you hit one of these classes of failure:

• invalid_client during the OAuth callback
• invalid_scope for standard OIDC scopes
• realm sync succeeds but secrets remain unresolved placeholders
• RabbitMQ login works partway but fails on audience, TLS, or management-role checks
• a service technically supports OIDC but the operational cost is higher than the value

What the current repo expects from Keycloak GitOps

The checked-in contract is now based on keycloak-config-cli, not kc.sh import --override:

• lumie-infra/security/keycloak/common-values.yaml
• lumie-infra/security/keycloak/manifests/realm-sync-job.yaml

The current realm-sync Job explains the design directly:

# kc.sh import writes directly to the DB and bypasses Keycloak's
# realm-creation hooks, so it never creates default scopes.
#
# keycloak-config-cli substitutes secrets on the CLIENT side before
# calling REST, so the DB always contains real values.

Source: lumie-infra/security/keycloak/manifests/realm-sync-job.yaml

The current infra realm definition also preserves the surviving client-scope decisions from the rollout, including a custom permissions claim for RabbitMQ and explicit groups support:

• lumie-infra/security/keycloak/common-values.yaml

The architecture decision that resolved most of the failures

The April 11, 2026 rollout only became stable after switching tools:

• kc.sh import --override was good at one-shot import, but it bypassed realm-creation hooks and left client-secret placeholder handling in the wrong place
• keycloak-config-cli performs client-side substitution and idempotent REST sync, which matches GitOps much better

That single tool change explained several earlier symptoms at once:

• missing default scopes such as profile and email
• unresolved ${KEY} placeholders stored literally in the database
• repeated loss of manual fixes after each re-import

Known failure classes from the rollout

Failure class	What it meant	Lasting fix
invalid_client	the stored client secret was a literal placeholder, not a resolved secret	use keycloak-config-cli with ${KEY} substitution from keycloak-oauth-secrets
invalid_scope for standard scopes	the realm had been imported without the normal Keycloak realm-creation hooks	manage the realm through the REST-backed sync job
strict client-scope errors	openid was treated like a Keycloak client-scope object when it is actually the OIDC protocol marker	remove openid from defaultClientScopes lists
RabbitMQ audience failure	Keycloak did not add rabbitmq to aud automatically	use an explicit audience mapper
RabbitMQ management-role failure	scope is reserved and was rebuilt by Keycloak	emit a custom permissions claim instead
service-specific dead ends	Headlamp and MinIO both had known but high-cost follow-up work	keep those as intentional non-integrations unless their value changes

What the current repo still proves

The inspected repo preserves the successful parts of the rollout:

• the infra realm contains repo-managed clients for Coder, Vault, Zot, and RabbitMQ
• the RabbitMQ scope design now uses permissions rather than trying to overload scope
• the realm-sync Job documents why placeholder substitution must happen on the client side
• Vault still needs a runtime OIDC setup step after realm sync, documented in lumie-infra/bootstrap/vault/OIDC-SETUP.md

It also preserves two important boundaries:

• Headlamp is not part of the active app graph in lumie-infra/applications/kustomization.yaml
• the repo does not document a surviving MinIO OIDC contract here, which matches the decision to stop short of a long-term integration

Troubleshooting path for the next integration

1. Confirm the service should be a confidential client or a public client before creating the Keycloak client.
2. Verify that the realm is being applied by keycloak-config-cli, not by a DB import shortcut.
3. Check whether the service needs claims in the ID token, access token, or both.
4. If the service is RabbitMQ-like, design a dedicated claim instead of trying to encode authorization into scope.
5. If the integration requires cluster-wide auth model changes or depends on a deprecated upstream console path, decide explicitly whether the SSO value justifies that cost.

Verification

cd Lumie
rg -n "keycloak-config-cli|IMPORT_VAR_SUBSTITUTION|kc.sh import|invalid_client" \
  lumie-infra/security/keycloak/manifests/realm-sync-job.yaml \
  lumie-infra/security/keycloak/helm-values.yaml
rg -n "permissions|rabbitmq|defaultClientScopes|clientId" \
  lumie-infra/security/keycloak/common-values.yaml

kubectl get job -n keycloak keycloak-realm-sync
kubectl logs -n keycloak job/keycloak-realm-sync --tail=200

Success means the repo still uses the REST-based realm-sync path, the realm definitions still carry the RabbitMQ-specific claim design, and the latest sync Job converges without placeholder or scope errors.

Source Incident Detail

The April 11, 2026 source writeup was a one-day SSO integration retrospective, not an outage. The architectural decision was to stop treating Teleport Community Edition as an SSO hub and instead register individual services as Keycloak OIDC clients.

Detail	Value
Scope	Coder, Vault, Zot, Headlamp, RabbitMQ, MinIO
Successful paths	Coder, Vault, RabbitMQ, and the Coder-to-Zot chain
Intentionally deferred paths	Headlamp and MinIO
Most important tooling decision	replace kc.sh import --override with keycloak-config-cli
Main RabbitMQ lesson	do not overload scope; use a dedicated authorization claim such as permissions

The rollout found 11 separate walls. The ones worth preserving for future integrations are:

Failure class	Why it mattered
file-vault placeholder limits	Keycloak can resolve some config fields but not every OAuth secret comparison path
kc.sh import --override	reimport wiped manual changes and made GitOps unsafe
client secret substitution	unresolved placeholders produced misleading invalid_client failures
standard scope creation	service integrations expected claims that did not exist yet
RabbitMQ audience	azp alone was not enough for the broker's resource-server check
RabbitMQ TLS	Erlang TLS behavior did not match assumptions from other stacks
reserved scope behavior	protocol mappers could not safely overwrite final OAuth scope
Headlamp / K3s OIDC	solving it required cluster auth-model changes
MinIO console / STS	the value did not justify the complexity at that point

The durable operating rule is to decide explicitly when not to integrate a service. Headlamp and MinIO were not "forgotten"; the source record says their root causes were understood and the cost/value tradeoff did not justify finishing them at that time.