LangGraph Chatbot Production Cutover

On May 26, 2026, Lumie moved the LangGraph chatbot from "deployed but not proven end to end" to a working production path behind the existing chat feature flag. The first real smoke test exposed integration bugs that isolated unit tests had not exercised together.

The cutover is useful as a retrospective because every failure happened at a boundary: backend transaction/RLS, worker tracing instrumentation, SSE message filtering, backend-to-worker HTTP versioning, or CI/deployment verification.

Context

The production chat path crosses the browser, monolith, worker, backend callback APIs, and database:

browser
  -> lumie-backend /v1/chat/stream
  -> ChatbotClient
  -> chatbot-svc LangGraph run
  -> backend internal callbacks
  -> backend persistence under RLS
  -> SSE frames back to browser

The important source surfaces are:

• lumie-backend/modules/ai/src/main/java/com/lumie/ai/adapter/out/external/ChatbotClient.java
• lumie-backend/modules/ai/src/main/java/com/lumie/ai/application/service/ChatService.java
• lumie-backend/libs/common/src/main/java/com/lumie/common/tenant/RlsTenantContextAspect.java
• lumie-worker/services/chatbot/src/usecase.py
• lumie-worker/services/chatbot/src/observability/tracing.py
• lumie-worker/services/chatbot/main.py

Impact

The cutover delayed confidence in the new production path rather than causing a broad outage. The risk was high because the new path would become the only chat surface after Spring AI removal.

The smoke test showed that "deployed" was not enough. The system needed proof that:

• tenant-scoped conversation writes worked under RLS;
• tracing could be enabled without breaking worker startup;
• only assistant text reached the frontend token stream;
• backend-to-worker transport matched the worker's HTTP capabilities;
• feature-flag fallback remained available until the new path was proven.

What Broke

Symptom during cutover	Confirmed cause	Fix kept in source
ai_conversations insert failed RLS checks	writes were not consistently inside the transactional service boundary that binds tenant context	ChatService owns create/save/pending writes under @Transactional
worker crashed when OTEL_ENABLED=true	global httpx instrumentation conflicted with langchain-openai client construction	tracing uses OpenInference LangChain instrumentation plus FastAPI instrumentation and skips httpx
first token exposed raw tool output	LangGraph stream_mode="messages" includes tool messages as well as assistant tokens	usecase.py forwards only AIMessageChunk content
backend-to-worker request failed against uvicorn	JDK HTTP client negotiated HTTP/2 while the worker speaks HTTP/1.1	ChatbotClient pins HttpClient.Version.HTTP_1_1

Representative shape from ChatbotClient.java:

HttpClient http11 = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build();
this.restClient = restClientBuilder
        .requestFactory(new JdkClientHttpRequestFactory(http11))
        .baseUrl(chatbotUrl)
        .build();

Diagnosis Path

The failures were diagnosed by walking the runtime path rather than treating the chatbot as one component.

Layer	Probe	Result	Lesson
persistence	inspect failed conversation insert	RLS denied the write	callback/persistence work needs the same tenant-bound transaction discipline as normal backend writes
worker startup	enable tracing and start the service	httpx instrumentation broke client construction	instrumentation should be selective and verified with the real startup path
SSE stream	inspect first emitted frames	tool messages reached the client	LangGraph stream modes need filtering before becoming product events
transport	compare backend client and uvicorn capability	HTTP/2 negotiation failed	generated/default clients are not always safe across internal service boundaries

Resolution

The cutover became stable only after these boundaries were aligned:

• ChatService became the transaction owner for conversation, message, and pending-action writes.
• RlsTenantContextAspect remained the tenant binding point for backend transactional data access.
• usecase.py filtered LangGraph output to preserve the frontend SSE contract.
• tracing.py avoided global httpx patching and used instrumentation that matched the LangChain/LangGraph path.
• ChatbotClient.java forced HTTP/1.1 for worker calls.
• application.yaml continued to point lumie.services.chatbot.url at CHATBOT_SVC_URL.

The shared-cluster CI failures in the same window mattered operationally, but they did not become chatbot runtime contract. They belong to rollout verification, not the steady-state chatbot design.

Verification

Future chatbot cutovers or rewrites should verify the complete chain:

Gate	What it proves
browser to backend to worker smoke	the public chat endpoint reaches the worker and streams frames
backend callback persistence	assistant messages and pending actions are saved under RLS
tracing-enabled startup	observability config does not break worker runtime
SSE frame inspection	only product-safe assistant tokens reach token events
HTTP-version check	backend client uses a protocol the worker actually speaks
feature flag fallback	old path remains available until the new path is proven

The later checkpointer pooling incident is a separate post-cutover hardening item, covered in Chatbot Checkpointer Single Connection Behind PgBouncer.

What Went Well

• The feature flag kept the old Spring AI path available while the new path was being proven.
• The failures were localized by runtime boundary, which made fixes small and source-owned.
• The final cutover left useful invariants in tests and source structure rather than only in operator memory.

What Went Poorly

• The first "real" test happened after several boundaries had already changed.
• Tracing was not validated as part of normal worker startup before production smoke.
• LangGraph message-stream semantics were treated as obvious until tool messages leaked into the product stream.
• Internal HTTP defaults were trusted too long.

Lasting Invariants

• Chat persistence belongs to the backend transaction boundary, not to ad hoc worker-side SQL.
• Worker traces should be enabled with instrumentation that matches the libraries actually used.
• LangGraph internal events are not the same thing as user-facing SSE frames.
• Backend-to-worker transport needs an explicit HTTP version when the server capability is known.

Related Follow-Ups

• Chatbot HITL Reject Deterministic Cancel
• Chatbot Checkpointer Single Connection Behind PgBouncer