Troubleshooting
This section is for known failure modes where the useful shape is symptom, likely cause, diagnosis, fix, and prevention.
Longer incident narratives and architecture retrospectives live in Incidents & Retrospectives.
How to Use This Section
Start here when the system is already in a known failure pattern and the priority is restoring or validating operation. Each guide should help an engineer answer four questions without re-reading the full incident history:
- What exact symptom identifies this failure mode?
- Which source files, Kubernetes objects, or runtime signals prove the diagnosis?
- What is the smallest safe fix?
- What evidence shows the fix has actually held?
These pages are not replacement design documents. If a fix changes a durable product or platform contract, update the relevant reference page or retrospective as well.
Sections
| Section | Use for |
|---|---|
| Backend DB | Flyway, RLS, migration, transaction, and database runtime failures. |
| Backend Runtime | Tenant context, backend-to-worker calls, async job lifecycle, and provider callback failures. |
| Frontend Runtime | Next.js runtime, local proxy, image optimization, and dependency migration failures. |
| Worker Runtime | FastAPI, async lifecycle, report rendering, and worker service runtime failures. |
| Worker CI | Generated requirements and worker build or CI drift failures. |
| Infra Scaling | KEDA, autoscaling, probes, and workload scaling failures. |
| Infra Observability | Tempo, Loki, Grafana, tracing, metrics, and log pipeline failures. |
| Infra Access | Coder, Teleport, SSO, and WebSocket/origin access failures. |
| Infra Certificates | cert-manager, DNS-01, and certificate secret projection failures. |
| Infra Messaging | RabbitMQ, broker topology, credentials, and queue delivery failures. |
| Infra Networking | Traefik, registry upload, ingress, and proxy timeout failures. |
| Product Workflows | Repeatable product workflow failures such as imports and bulk operations. |
Documentation Quality Bar
A troubleshooting page is complete only when it preserves enough operational context for someone who did not handle the original incident. Prefer concrete source paths, exact config keys, commands, expected outputs, and negative checks over broad advice.
For each page, keep these sections explicit when the topic supports them:
- Symptom: user-visible or operator-visible behavior, including the affected surface.
- Likely Cause or Root Cause: the smallest causal explanation that distinguishes this issue from adjacent failures.
- Diagnosis: ordered checks that can confirm or reject the hypothesis.
- Fix: the intended remediation path and changes that must not be reintroduced.
- Verification: command or runtime evidence that proves recovery.
- Prevention: guardrails that keep the same failure from returning.
If a page is sourced from a longer incident note, preserve the incident-specific details in the Docusaurus page before deleting the source note. If live state is required but the repository cannot prove it, say that explicitly and list the runtime check.
Triage Order
Use this order during an active incident:
- Match the symptom to a guide in the table above.
- Run the repository checks first. They are fast and catch drift before cluster access is required.
- Run live Kubernetes, CI, or browser checks only after the repo contract still looks correct.
- Apply the smallest fix that restores the documented contract.
- Capture the verification evidence in the incident note or follow-up PR.
When symptoms overlap, prefer the guide closest to the failing boundary. For example, a failed Zot blob push belongs under Infra Networking if Traefik terminates the upload, but under the registry reference if Zot accepts the request and fails on storage.