Chatbot Checkpointer Single Connection Behind PgBouncer

Symptom

Use this page when chatbot-svc accepts the SSE request but immediately emits the generic chatbot error after a long idle period, and the worker log shows psycopg.OperationalError: the connection is closed while entering graph.astream(...).

This failure mode affected the LangGraph checkpointer, not the backend's conversation tables. User-facing chat history could still be intact while the worker was unable to start or resume a graph run.

Why It Happens

The original scaffold used AsyncPostgresSaver.from_conn_string(dsn), which holds one psycopg.AsyncConnection for the lifetime of the pod.

That pattern is brittle behind the CNPG Pooler because the upstream pooler and network are allowed to rotate or close idle connections. Once that singleton handle goes stale, the next aget_tuple(...) fails before the graph can stream or resume anything.

The current entrypoint in lumie-worker/services/chatbot/main.py exists specifically to prevent that regression.

Diagnostic Path

Check these signals together:

1. the browser or backend opened /api/chatbot/stream successfully,
2. lumie-worker/services/chatbot/src/usecase.py logged graph run failed,
3. the traceback bottoms out in langgraph.checkpoint.postgres.aio or psycopg with the connection is closed,
4. the affected image or local experiment is not using the pooled checkpointer from main.py.

Representative shape from lumie-worker/services/chatbot/main.py:

pool = AsyncConnectionPool(
    conninfo=settings.langgraph_db_dsn,
    min_size=1,
    max_size=5,
    max_idle=180.0,
    check=AsyncConnectionPool.check_connection,
    kwargs={
        "autocommit": True,
        "prepare_threshold": 0,
        "row_factory": dict_row,
    },
)
checkpointer = AsyncPostgresSaver(pool)

The important guardrails are:

Setting	Why it matters
check=AsyncConnectionPool.check_connection	validates a handle before handout so dead connections are replaced before the graph uses them
max_idle=180.0	lets the worker retire idle connections before the upstream pooler surprises it
prepare_threshold=0	keeps psycopg compatible with transaction-pooled PgBouncer behavior
row_factory=dict_row	matches AsyncPostgresSaver's row-by-name access

Fix

Do not hold a singleton checkpointer connection for a long-lived worker.

The permanent fix is the current pattern in lumie-worker/services/chatbot/main.py:

1. open an AsyncConnectionPool during FastAPI lifespan,
2. wait for the pool before compiling the graph,
3. pass the pool into AsyncPostgresSaver,
4. close the pool only after usecase.aclose() so in-flight checkpoint writes can finish.

If you see this error on an older image, a rollout restart can temporarily restore service by forcing a fresh connection:

kubectl rollout restart deployment -n lumie-worker chatbot-svc

That is mitigation only. Without the pooled pattern, the same pod can fail again after the next idle window.

Prevention

• Treat lumie-worker/services/chatbot/main.py as the reference for any worker that keeps LangGraph or psycopg state open across requests.
• Avoid README quickstart patterns such as from_conn_string(...) in long-running services unless the library docs explicitly cover reconnection behavior.
• Keep the checkpointer path separate from user-message persistence when debugging so you do not mistake a graph-state outage for data loss in the monolith.

For the broader architecture choice behind the detached SSE worker, see LangGraph Chatbot Streaming Architecture.

Source Incident Detail

The source incident was detected on May 26, 2026 after a three-hour idle window. A user opened a fresh production chat, the browser received an SSE response, and the worker immediately emitted the generic Korean error frame from _run_graph.

Detail	Value
Affected service	lumie-worker/services/chatbot
Persistence path that failed	LangGraph checkpointer in the langgraph schema
Persistence path that stayed safe	backend-owned ai_conversations and ai_messages rows
Immediate error	psycopg.OperationalError: the connection is closed
Mitigation	kubectl rollout restart deployment -n lumie-worker chatbot-svc
Durable fix	replace one lifetime connection with AsyncConnectionPool and check_connection

The key timeline was phase-based:

Phase	Event
Phase 1	Phase 4 cutover smoke passed, then the chatbot sat idle during later cleanup work.
Phase 2	About 3h24m later, the next graph run entered AsyncPregelLoop and called checkpointer.aget_tuple(...).
Phase 3	The singleton psycopg connection had already been closed by the pooler/network idle path.
Phase 4	Rollout restart created a fresh pod and restored service quickly.
Phase 5	The permanent fix moved the checkpointer onto a pool with pre-handoff health checks.

The important operational distinction is that this did not corrupt user-visible message persistence. The dead handle was in the graph checkpoint path, not in the backend JPA persistence path that stores the conversation rows.

Keep these follow-ups attached to future worker scaffolds:

Follow-up	Status
Use pooled DB clients for long-running worker services.	Required pattern
Set prepare_threshold=0 behind PgBouncer transaction pooling.	Required for psycopg
Close the pool after usecase.aclose() during shutdown.	Required to let in-flight checkpoint writes finish