LangGraph Chatbot Streaming Architecture

On May 25, 2026, Lumie chose a worker-local LangGraph architecture for chatbot streaming. The decision rejected a queue-driven chat path, Redis resumable streams, and LangGraph Server for the first production version.

The core requirement was narrow but important: if a director closes the browser tab, the analysis should not be lost.

Context

The chatbot had to preserve several existing boundaries:

• the frontend SSE contract stayed fixed at token, pending, done, and error events;
• the monolith remained the system of record for conversations, assistant messages, and pending actions;
• tenant-scoped SQL stayed in lumie-backend under RLS;
• the Python worker owned only LangGraph execution and checkpoint state in the langgraph schema;
• write tools required HITL confirmation before execution.

The current code still reflects that split:

• lumie-worker/services/chatbot/src/usecase.py maps graph output to SSE events and persists final text through backend callbacks;
• lumie-worker/services/chatbot/src/graph/agent.py pauses write tools with interrupt() and resumes with Command(resume=...);
• lumie-backend/modules/ai/src/main/java/com/lumie/ai/application/service/ChatService.java owns conversation persistence under @Transactional;
• lumie-worker/services/chatbot/main.py wires the compiled graph and checkpointer in FastAPI lifespan.

Options Considered

Option	What it solved	Why it was not the final choice
request-bound SSE only	simplest live token delivery	client disconnect would cancel the run
detached SSE producer plus persisted result	tab-close safety without changing the frontend contract	chosen
Redis resumable streams	replay and reattach for in-progress tokens	extra infrastructure and frontend reconnect work for a requirement not yet requested
MQ-driven chat execution	stronger job durability	poor fit for low-latency token streaming and risky for duplicate write-tool execution
LangGraph Server	managed durable runs and resumable streams	rejected for Lumie's self-hosted posture and constraints captured in the source decision record

The decision separated three concerns that often get merged too early:

Concern	Chosen owner
execution while the pod is alive	detached background task in usecase.py
live delivery	SSE proxy through ChatbotClient and backend stream endpoint
durable conversation result	backend save_message and save_pending_action callbacks

Chosen Architecture

The representative runtime shape in usecase.py is:

task = asyncio.create_task(
    self._run_graph(graph, inputs, config, conversation_id, req, queue, consumer_gone)
)

try:
    while True:
        item = await asyncio.wait_for(queue.get(), timeout=1.0)
        ...
finally:
    consumer_gone.set()

The HTTP response drains a queue. The graph run itself is detached, so a client disconnect stops live delivery but does not automatically stop execution or final persistence.

The graph boundary stays explicit:

1. scope_check rejects obvious out-of-scope or prompt-injection requests.
2. agent decides whether to answer directly or call tools.
3. tools executes read tools through backend APIs or pauses write tools with interrupt().
4. confirm_tool() resumes the checkpointed run after confirm or reject.

Why This Fit Lumie

The chosen design was intentionally smaller than the available platform options:

• it preserved the frontend and backend streaming contract;
• it kept tenant-table access inside the monolith, where RLS and validators already existed;
• it avoided Redis or MQ behavior for an interaction that primarily needs live request-response semantics;
• it kept LangGraph portable because the graph is still a regular compiled StateGraph;
• it allowed durability for final results without promising resumable token replay.

The later PgBouncer checkpointer issue changed the connection-management implementation, not the architecture decision. That follow-up is covered in Chatbot Checkpointer Single Connection Behind PgBouncer.

What Went Well

• The decision focused on the actual requirement, not the most powerful runtime.
• Keeping SQL ownership in the backend avoided duplicating tenant and authorization rules in Python.
• The graph and transport boundary remained simple enough to harden incrementally.
• The architecture left room for Redis or another orchestrator later if resumable stream replay becomes a real product requirement.

What Went Poorly

• The initial decision still relied on later runtime hardening for production-grade connection management.
• Redis resumability and execution durability had to be explained carefully because they solve different problems.
• The source investigation did not preserve exact external research links, so this page documents the decision outcome rather than citing every evaluated doc.

Revisit Triggers

Reconsider this architecture when one of these requirements becomes real:

Trigger	Implication
users must reconnect to in-progress token streams	add resumable stream storage or an equivalent replay layer
graph execution must survive worker pod death mid-run	move execution ownership to a stronger durable runtime
write-tool execution needs exactly-once semantics across restarts	introduce stronger idempotency and execution records before queueing or replay
multiple workers need to attach to the same live run	evaluate external orchestration or shared stream ownership

Until then, detached SSE plus backend-owned persistence remains the smallest architecture that satisfies the recorded requirement.

Lessons

• "Do not lose the work" is a persistence requirement before it is a resumable-stream requirement.
• Redis can replay events, but it does not by itself decide who owns execution.
• MQ is appropriate for batch and callback workers in this stack, but not automatically for live token UX.
• A clean graph boundary lets the team defer heavier runtime choices without locking the product into a dead end.