Coder Teleport WebSocket Bootstrap

Symptom

Use this guide when either of these happens:

• the Coder UI loads, but WebSocket-backed endpoints such as /api/v2/notifications/inbox/watch return 403
• a new workspace stays stuck at Agent state is timeout

What the current repo expects from Coder

The Coder chart values still assume public access through https://coder.lumie-infra.com and trust forwarded headers from an in-cluster reverse proxy:

- name: CODER_ACCESS_URL
  value: "https://coder.lumie-infra.com"
- name: CODER_PROXY_TRUSTED_HEADERS
  value: "X-Forwarded-For,X-Forwarded-Proto,X-Forwarded-Host"
- name: CODER_PROXY_TRUSTED_ORIGINS
  value: "10.42.0.0/16"

Source: lumie-infra/applications/coder/helm-values.yaml

The workspace template also replaces the default agent bootstrap so workspace pods download and connect through the in-cluster Coder service instead of the Teleport-fronted public URL:

command = ["sh", "-c", <<-EOT
  ...
  if curl -fsSL --compressed "$CODER_AGENT_URL/bin/coder-linux-arm64" -o coder; then
    break
  fi
  ...
  exec ./coder agent
EOT
]

env {
  name  = "CODER_AGENT_URL"
  value = "http://coder.coder.svc.cluster.local"
}

Source: lumie-infra/applications/coder/template/main.tf

Why this failure mode happened

Commit	Change	What it fixed
47d3465b	Trusted X-Forwarded-Host and pod-CIDR origins in Coder	Same-origin checks were comparing browser Origin to the wrong upstream Host
557a073e	Added a Teleport-side Host: coder.lumie-infra.com rewrite	WebSocket requests needed the forwarded Host to match the browser origin
76673143	Replaced the default agent bootstrap with an in-cluster download path	Workspace pods cannot pass Teleport SSO just to fetch the Coder agent binary

Diagnostic path

If WebSocket requests return 403

kubectl logs -n coder deploy/coder --tail=200
kubectl get configmap,secret -n coder

Look for origin-check failures rather than generic network errors. The historical failure was a strict same-origin rejection, not a broken TCP path.

If workspaces stay on Agent state is timeout

Check whether the workspace template still uses the in-cluster bootstrap:

rg -n "CODER_AGENT_URL|coder-linux-arm64|exec \\./coder agent" \
  applications/coder/template/main.tf

If those lines are missing, the workspace may have fallen back to the default Coder init script, which downloads from CODER_ACCESS_URL and therefore hits the Teleport login wall.

Current drift to verify before blaming Coder

The April 11, 2026 WebSocket fix sequence depended on a Teleport app registration for coder with a Host rewrite in security/teleport/agent/helm-values.yaml. The current inspected file no longer contains a coder app entry at all.

That means the present repo does not fully prove the old Teleport-side fix is still declared in Git, even though:

• applications/coder/helm-values.yaml still points at https://coder.lumie-infra.com
• commit 557a073e added the Teleport Host rewrite
• the existing Teleport reference page still lists coder as an app

Verify the live Teleport registration before assuming the repo still describes the full access path.

Fix

1. Keep the Coder-side trusted header settings in applications/coder/helm-values.yaml.
2. Keep the workspace-agent bootstrap on http://coder.coder.svc.cluster.local, not on CODER_ACCESS_URL.
3. If WebSocket 403 returns, verify that Teleport is still forwarding Host: coder.lumie-infra.com upstream. The current repo no longer proves that part.
4. If the repo-managed Teleport app registration is missing, restore that contract before debugging Coder runtime behavior in isolation.

Verification

kubectl get applications.argoproj.io -n argocd coder
kubectl logs -n coder deploy/coder --tail=200
kubectl get pods -n lumie-dev
rg -n "CODER_ACCESS_URL|CODER_PROXY_TRUSTED|CODER_AGENT_URL" \
  applications/coder/helm-values.yaml applications/coder/template/main.tf

Success means the Coder deployment still trusts forwarded headers, workspace pods still bootstrap against the in-cluster service URL, and the live access path no longer shows WebSocket 403 or agent timeout behavior.

Related pages

• Teleport operations reference

Operational Detail

This failure mode crosses three ownership boundaries: browser origin checks, Teleport app forwarding, and workspace pod bootstrap. A fix that only touches one layer can make the UI look healthier while workspaces still fail to connect.

Boundary	Required contract	Common broken state
Browser to Teleport	Browser origin is https://coder.lumie-infra.com.	WebSocket request reaches Coder with a mismatched host.
Teleport to Coder	Forwarded host and trusted headers preserve the public URL.	Coder rejects watch endpoints with 403.
Workspace pod to Coder	Agent downloads through the in-cluster service URL.	Pod tries to fetch the agent through Teleport SSO and times out.

Treat Agent state is timeout as a bootstrap-path problem until proven otherwise. It often means the workspace pod cannot fetch or connect the agent, not that the Coder control plane is down.

Recovery Playbook

1. Split the report into UI WebSocket failure or workspace agent timeout.
2. For WebSocket 403, capture Coder logs around the request and compare Origin, forwarded host, and trusted header settings.
3. For agent timeout, inspect the rendered workspace template and confirm CODER_AGENT_URL points to the service DNS name.
4. Verify whether the live Teleport app registration still contains the required host rewrite, because the current repo no longer fully proves it.
5. Restart only the affected workspace after the path is corrected; do not recycle all workspaces until the template is validated.

Verification Evidence

Close the incident only after recording:

• Coder UI watch endpoints no longer return 403,
• a newly created workspace downloads the agent binary without crossing Teleport SSO,
• the workspace reaches a connected agent state,
• Coder values still include trusted forwarded headers and origins,
• the live Teleport registration, if managed outside the inspected repo path, is captured in the follow-up notes.

Anti-Patterns

• Increasing workspace startup timeouts without proving the agent download path.
• Removing Coder origin checks instead of fixing the forwarded host contract.
• Pointing CODER_AGENT_URL at the public Teleport URL from inside workspace pods.
• Assuming the repo and live Teleport app list are identical when the repo no longer contains the old coder registration.