Skip to main content

Custom Domain White-Label Routing Incident

This retrospective covers the July 12, 2026 custom-domain routing incident. The core invariant was simple: a tenant white-label domain must never render canonical lumie-edu.com marketing content. The original middleware path violated that invariant whenever custom-domain resolution timed out or returned null.

The host-native follow-up that removed tenant-path leakage entirely is documented in Host-Native Custom Domain Routing.

Context​

Lumie serves the same tenant landing through two URL models:

  • canonical host plus tenant path, such as https://lumie-edu.com/joossameng;
  • tenant-owned host, such as https://joossameng.com/.

The routing decision lives in the frontend:

  • lumie-frontend/proxy.ts
  • lumie-frontend/src/shared/lib/customDomain.ts

The initial custom-domain path depended on runtime tenant lookup. If the lookup missed, the request could continue down the normal public-route path instead of stopping at the host boundary.

What Broke Or Changed​

The incident played out in phases:

  • https://joossameng.com/ could rarely show the canonical Lumie landing page.
  • The first fix changed misses to fail closed with 503 and Cache-Control: no-store, which was safer but still too dependent on runtime lookup.
  • Valid auth and app URLs then surfaced gaps in the custom-domain allowlist, especially /auth/refresh?returnTo=%2Fadmin and /${customId}?auth=login.
  • One part of the debugging time was deployment-state ambiguity: the allowlist fix existed in git before production had rolled out the new image.

Root Cause​

The confirmed root cause was a fail-open routing decision in the frontend proxy.

lumie-frontend/src/shared/lib/customDomain.ts still shows the original moving parts:

  • configured host-to-tenant mapping with CUSTOM_DOMAIN_TENANT_MAP;
  • fallback runtime lookup through resolveCustomDomainTenant(host);
  • a bounded lookup timeout via CUSTOM_DOMAIN_RESOLVE_TIMEOUT_MS.

The durable lesson was not "increase the timeout." It was "do not let an unresolved custom-domain host fall through to canonical public routes."

Resolution​

The final design is visible in current repo state.

Deterministic host mapping​

lumie-infra/applications/lumie/frontend/common-values.yaml sets:

  • CUSTOM_DOMAIN_RESOLUTION_ENABLED
  • CUSTOM_DOMAIN_RESOLVE_TIMEOUT_MS
  • CUSTOM_DOMAIN_TENANT_MAP

That makes operator-managed custom domains deterministic before the fallback lookup is even considered.

Fail-closed proxy behavior​

lumie-frontend/proxy.ts now:

  • strips spoofable tenant and user headers before forwarding;
  • passes configured custom-domain root requests with trusted tenant headers;
  • allows /api/*, /auth/refresh, /admin*, and /dashboard* on configured custom domains;
  • redirects /login, /register, and /signup to root query-based auth URLs;
  • returns 404 for canonical marketing or tenant-path routes that should not exist on a configured custom domain;
  • returns 503 for unresolved bootstrap requests instead of serving canonical Lumie content.

Test coverage for the routing boundary​

The routing invariants are pinned in:

  • lumie-frontend/src/shared/lib/middlewareCustomDomain.test.ts
  • lumie-frontend/src/shared/lib/apiProxyCustomDomain.test.ts
  • lumie-frontend/src/shared/lib/apiProxyRoute.test.ts

Those tests cover the specific behaviors that failed during the incident: fail-closed root handling, trusted-header injection, 404 on canonical-page leakage, and allowed auth-refresh flow on configured domains.

Prevention​

  • Treat the host itself as a security and branding boundary for white-label domains.
  • Prefer configured host maps for operator-managed custom domains; keep runtime by-domain lookup as a transitional fallback, not the hot-path source of truth.
  • Fail closed with no-store responses whenever a custom-domain request cannot be resolved safely.
  • Verify production image rollout, not just merged code, when a routing fix depends on middleware behavior.