Async-First Worker Recovery

Symptom

Use this guide when a worker service starts drifting away from the intended async-first pattern:

• FastAPI handlers use def as a workaround for blocking I/O.
• live clients or containers are created at import time instead of inside FastAPI lifespan
• shared httpx.AsyncClient instances are not closed on shutdown
• best-effort failures disappear without a log trail

This recovery pattern comes directly from the analysis, report, and grading changes made between February 20, 2026 and May 19, 2026.

Why This Failure Mode Happened

The analysis service first shipped in c96370c7 with async def routes but a synchronous Anthropic client in services/analysis/main.py. The immediate workaround in 5c80a4d1 changed the route handlers to plain def to avoid blocking the event loop.

That fixed the symptom, not the architecture. The later recovery sequence was:

Date	Commit	Recovery step
2026-05-11	3710695f	analysis switched to AsyncOpenAI; report started logging best-effort rank failures instead of swallowing them silently
2026-05-11	622f1149	grading and report moved to lifespan-scoped shared httpx.AsyncClient wiring
2026-05-18	aa770cd7	analysis moved behind a DI container and src/ layering
2026-05-19	0cd5feaa	worker services stopped building containers at import time and moved them into FastAPI lifespan

Diagnostic Path

Check for import-time containers or clients

The current recovery target is the pattern used in services/analysis/main.py and services/report/main.py.

Representative shape:

_container: Container | None = None

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    global _container
    _container = build_container()
    try:
        yield
    finally:
        ...
        _container = None

If a worker instead constructs its container at module import, startup failures become harder to recover and tests become less isolated.

Check that async clients are shared and closed

services/grading/src/wiring.py is the reference pattern for shared backend HTTP access:

def build_http_client() -> httpx.AsyncClient:
    return httpx.AsyncClient(
        timeout=httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0),
        limits=httpx.Limits(max_connections=20, max_keepalive_connections=10),
        transport=_http_transport(),
    )

The container carries that client, and the entrypoint closes it during shutdown. services/report/main.py now closes callbacker, report_renderer, and http_client in its lifespan finally block. services/analysis/main.py closes joossameng_cache, http_client, and llm_adapter.

Check whether a sync route is masking a blocking adapter

If a route became def only to hide blocking network I/O, that is a regression against the current design. The fixed analysis entrypoint is back to async def for both /api/analysis/exam-commentary and /api/analysis/student-feedback, and it relies on async adapters behind the container instead of sync handler workarounds.

Check for silent best-effort failures

3710695f fixed a report-side blind spot by logging the student-rank fallback path in services/report/src/usecase.py. If a worker still has best-effort branches that return fallback values without logging, repeated failures can look like intermittent data issues instead of runtime faults.

Recovery Steps

1. Move container construction into FastAPI lifespan and keep only a nullable module-level reference.
2. Replace sync network clients with async adapters instead of converting handlers to def.
3. Put shared httpx.AsyncClient construction in the service wiring layer and close it from lifespan shutdown.
4. Keep CPU-bound work off the event loop with asyncio.to_thread(...) rather than with sync route handlers.
5. Add logger.exception(...) on best-effort fallback paths so repeated soft failures show up in logs.

Prevention

• Treat services/analysis/main.py and services/report/main.py as the entrypoint reference for lifespan-scoped containers.
• Treat services/grading/src/wiring.py as the reference for shared httpx.AsyncClient ownership.
• If a service needs a temporary sync workaround, document it as debt. The end state should still be async handlers backed by async adapters.

For CI failures that appear after dependency or runtime refactors, see Generated Requirements Drift.

Operational Detail

The async-first pattern is primarily about resource ownership. Worker services should make runtime dependencies explicit during FastAPI lifespan startup and release them during shutdown.

Resource	Expected owner	Failure when misplaced
DI container	FastAPI lifespan.	Import-time side effects and hard-to-isolate tests.
httpx.AsyncClient	Service wiring plus lifespan cleanup.	Connection leaks and noisy shutdowns.
LLM or backend adapters	Async adapter behind the container.	Event loop blocking or sync handler workarounds.
Best-effort fallbacks	Use case layer with logging.	Silent degraded behavior.

The goal is not to make every operation non-blocking at all costs. CPU-bound work can still run safely through asyncio.to_thread(...) when the ownership and logging boundaries are clear.

Recovery Playbook

1. Search for module-level client or container construction in the affected service.
2. Move construction into lifespan() and keep a nullable module-level handle only for request access.
3. Replace sync network adapters before changing route handlers back to async def.
4. Add shutdown cleanup for every shared async resource.
5. Add logger.exception(...) around best-effort fallbacks before relying on fallback behavior in production.
6. Re-run service tests that cover startup, shutdown, and at least one request path using the container.

Verification Evidence

Record:

• import no longer opens live network clients,
• startup builds the container exactly once per app lifespan,
• shutdown closes shared clients without unclosed-resource warnings,
• route handlers that perform I/O are async def and call async adapters,
• best-effort failure paths produce logs with enough context to diagnose repetition.

Anti-Patterns

• Converting a route to def as the permanent fix for blocking I/O.
• Creating global clients during module import because it is convenient for handlers.
• Swallowing ranking, callback, or report-render fallback errors without logs.
• Adding per-request httpx.AsyncClient instances instead of a shared lifespan-scoped client.