Headlamp OIDC Claim Breaking Change

Symptom

Use this guide when Headlamp reaches the identity provider successfully but still fails in one of these ways:

• browser login loops back to the sign-in page
• Kubernetes API calls return 401 Unauthorized
• login succeeds but every view returns 403 Forbidden
• the IDP rejects the request with invalid_request
• Headlamp shows a successful sign-in but no Kubernetes permissions

What the current repo manages today

The inspected repo does not currently preserve an active Headlamp OIDC integration:

• lumie-infra/applications/kustomization.yaml comments out headlamp/argocd.yaml
• lumie-infra/applications/headlamp/helm-values.yaml contains only the basic chart contract and no repo-managed OIDC client settings

That means this page is a historical troubleshooting guide for clusters that still use the January 2026 Headlamp plus K3s OIDC pattern. The exact K3s API server flags from that incident are not present in the inspected repo.

Failure ladder from the incident

The January 13, 2026 failure sequence had five distinct gates:

Gate	Symptom	Confirmed cause
Client auth	login loop	Headlamp and the IDP disagreed on token_endpoint_auth_method
Username claim	401 Unauthorized	the configured username claim was missing from the ID token
Username prefix	403 Forbidden	K3s prefixed the resolved username with the issuer URL
Scope request	invalid_request	Headlamp requested a scope the IDP client did not allow
RBAC	"no permissions"	the resolved Kubernetes user had no ClusterRoleBinding

The most important trigger was the claim change: after the IDP upgrade, preferred_username was no longer present where K3s expected it, so authentication failed even though browser login succeeded.

Diagnostic path

1. Check client authentication first

If the browser never settles into a session, compare the Headlamp client configuration with the IDP client registration. A mismatch between client_secret_basic and client_secret_post is enough to create a loop before Kubernetes is involved.

2. Decode the ID token, not only the userinfo response

K3s validates the ID token it receives. If the configured username claim is not present there, the request fails even if the IDP exposes the claim through userinfo.

The historical failure was this exact class:

• configured claim: preferred_username
• claim available in the ID token after the upgrade: no
• result: Kubernetes API 401

3. Check the resolved Kubernetes username

Even when the claim exists, K3s may prepend the issuer URL to the username. If RBAC was authored for bluemayne but Kubernetes resolves the user as https://issuer.example#bluemayne, authorization fails.

4. Check requested scopes against the IDP client

Headlamp can request more scopes than the IDP client currently allows. In the incident, an unsupported groups scope produced an OAuth error before RBAC was even evaluated.

5. Check RBAC last

If authentication succeeds and the token claim is correct, confirm that the final resolved Kubernetes username has the expected ClusterRoleBinding.

Fix pattern

The incident was resolved by correcting each layer in order:

1. align the token endpoint authentication method
2. switch the Kubernetes username claim to one that is guaranteed to exist in the ID token
3. remove or account for the K3s username prefix
4. request only scopes that the IDP client actually allows
5. bind the final resolved username to the intended Kubernetes role

Prevention

• Put IDP upgrades that can change OIDC claim behavior into manual review, even for minor-version bumps.
• Prefer durable claims such as sub when the relying party only validates the ID token.
• Keep a note of the exact Kubernetes username format that RBAC expects.

Verification

cd Lumie
sed -n '1,80p' lumie-infra/applications/headlamp/helm-values.yaml
sed -n '1,40p' lumie-infra/applications/kustomization.yaml

kubectl logs -n headlamp deploy/headlamp --tail=200
kubectl auth can-i --as='<resolved-oidc-username>' get pods -A

Success means the repo state is understood first, the Headlamp deployment logs no longer show auth churn, and the exact Kubernetes username resolved from the token has the permissions you expect.

Source Incident Detail

The source incident happened on January 13, 2026 after Renovate upgraded Authelia from 4.37.x to 4.38.0. The upgrade changed how profile claims were exposed. K3s validated only the ID token, while the claim Headlamp/K3s expected had moved to userinfo.

Detail	Value
Affected path	Headlamp -> Authelia OIDC -> K3s API
User impact	no product-user outage; cluster UI operations were blocked
Primary breaking change	preferred_username no longer reliably present in the ID token
Additional failures	token endpoint auth method, username prefix, scope, and RBAC drift
Durable lesson	IDP minor upgrades can be relying-party breaking changes

The failure ladder had five layers:

Layer	Symptom	Fix direction
claim location	K3s could not find the username claim in the ID token	use a claim K3s can actually validate
client auth method	token exchange failed before Kubernetes auth	align token_endpoint_auth_method
username format	resolved Kubernetes username did not match RBAC	preserve the exact prefix/claim shape
scopes	expected claims were not requested or emitted	align scopes with relying-party needs
RBAC	authentication succeeded but authorization failed	bind the resolved Kubernetes username or group

When this reappears, do not start with Headlamp UI symptoms. Decode the token first, compare ID token claims with K3s OIDC flags, then verify RBAC with the exact resolved username.