Traefik Zot Blob Push Timeout

This page is a troubleshooting document for large OCI uploads to Zot failing through Traefik. The steady-state ingress and registry contracts live in the infrastructure networking Traefik and infrastructure GitOps Zot reference pages.

Symptom

Large pushes to https://zot.lumie-infra.com fail partway through, especially on multi-gigabyte layers, and Traefik or the client reports a disconnect such as HTTP 499.

The incident that produced the fix came from pushing a roughly 1.2GB Chromium layer in report-svc.

Likely Cause

The k3s-bundled Traefik addon is still using short entrypoint timeouts, so Traefik stops waiting before Zot finishes streaming the blob to MinIO.

This is not a Zot-only knob. The web and websecure entrypoint timeouts apply cluster-wide.

Fix

ca6c511f added platform/traefik-config/argocd.yaml and platform/traefik-config/helmchartconfig.yaml so the repo patches the bundled addon with bounded upload-friendly timeouts:

ports:
  web:
    transport:
      respondingTimeouts:
        readTimeout: 1200s
        writeTimeout: 1200s
        idleTimeout: 600s

The same values are applied to websecure.

0 was intentionally not used for readTimeout, because disabling the timeout entirely would also remove slow-body and slowloris protection for every ingress on those entrypoints.

Diagnostic Path

1. Inspect platform/kustomization.yaml and confirm it registers traefik-config/argocd.yaml.
2. Inspect platform/traefik-config/helmchartconfig.yaml and confirm both web and websecure set readTimeout and writeTimeout to 1200s plus idleTimeout to 600s.
3. If cluster access is available, inspect the live HelmChartConfig in kube-system and check Traefik logs while reproducing the push.

Prevention

• Keep entrypoint timeouts large enough for the biggest expected blob and the slowest expected MinIO-backed upload path.
• Keep the values bounded rather than disabled while Traefik still serves general web traffic on the same entrypoints.
• If Zot needs looser limits than the rest of the cluster, split it onto a dedicated registry-only entrypoint instead of widening the default web entrypoints further.

Verification

cd lumie-infra
rg -n "traefik-config/argocd.yaml|readTimeout: 1200s|writeTimeout: 1200s|idleTimeout: 600s" \
  platform/kustomization.yaml \
  platform/traefik-config/argocd.yaml \
  platform/traefik-config/helmchartconfig.yaml

kubectl get helmchartconfig -n kube-system traefik -o yaml
kubectl logs -n kube-system deploy/traefik

Success means the repo and live HelmChartConfig both show the bounded timeout patch, and large Zot uploads no longer terminate early at the Traefik layer.

Operational Detail

The distinguishing symptom is that the upload path fails while the registry itself can still be healthy. Zot can accept the request, but Traefik may terminate the long body stream before Zot and MinIO finish the blob write.

Layer	What to check	Failure clue
Client	Push output and layer size.	Disconnect during a large layer, not auth failure.
Traefik	Access logs and entrypoint timeout config.	499, timeout, or early close around the same timestamp.
Zot	Registry logs.	Upload starts but does not receive complete blob.
MinIO	Object write latency or errors.	Slow backing write extends the Traefik wait time.

Because the timeout applies to shared entrypoints, every change is a platform-wide ingress decision.

Recovery Playbook

1. Capture the failing layer size and elapsed time before the disconnect.
2. Confirm the failure is not an authentication or repository permission error.
3. Verify the live HelmChartConfig has the bounded timeout values for both web and websecure.
4. Reproduce with a representative large layer, not a small manifest push.
5. If the bounded values are still insufficient, evaluate a dedicated registry entrypoint before widening the shared entrypoints again.

Verification Evidence

Close the incident with:

• a successful push of the previously failing layer size,
• Traefik logs showing no early timeout for that request,
• Zot logs showing blob completion,
• live HelmChartConfig matching Git,
• a note on the largest layer size used for validation.

Anti-Patterns

• Setting readTimeout: 0 on shared entrypoints without a security review.
• Debugging Zot storage first when Traefik terminates the request earlier.
• Validating only small image pushes after changing timeout behavior.
• Increasing every ingress timeout to solve a registry-specific workload without considering a dedicated entrypoint.