Skip to main content

Billing Subscribe Cascading 409s

This retrospective covers the May 27-28, 2026 billing subscribe incident. The failure started as a demo blocker, but it exposed three lasting backend contract problems: the subscribe boundary lived on the frontend, database integrity failures all looked like the same 409, and a same-transaction billing-key upsert still violated the active-key unique index.

Context​

Before the fix, the paid-plan subscribe flow was coordinated by the frontend success page after the Toss billing callback:

  • register the billing key,
  • change the tenant subscription,
  • charge the first invoice.

That sequence assumed two invariants that were not always true:

  • every tenant already had a FREE subscription row;
  • each step could be retried independently without stranding the tenant in a half-finished billing state.

The current repo still shows the final boundary that replaced that design:

  • lumie-backend/modules/billing/src/main/java/com/lumie/billing/application/service/BillingSubscribeService.java
  • lumie-frontend/src/widgets/billing-home/ui/BillingSetupSuccess.tsx

What Broke Or Changed​

The visible symptom was a series of near-identical 409 Conflict responses, but the causes were different:

  • legacy tenants without a subscription row kept returning 404 until V35__backfill_free_subscriptions.sql restored the invariant;
  • Toss billing-key persistence hit nullable-card metadata drift until V36__billing_keys_nullable_card_metadata.sql widened the schema;
  • re-running the billing-key step after a partial failure collided with the tenant's existing active key;
  • the frontend was coordinating a distributed transaction across three backend calls;
  • DataIntegrityViolationException mapped to the same generic 409 even when the real cause was a server-side NOT NULL or foreign-key bug.

Because the wire response hid the difference between user conflicts and server defects, each fix only revealed the next failure class.

Root Cause​

Three confirmed defects survived the incident and explain the final design.

1. The business-operation boundary was on the wrong layer​

The durable fix was to move "subscribe to a paid plan with a new card" into one backend orchestration step. The current source of truth is BillingSubscribeService, which owns billing-key issuance, active-key replacement, subscription attachment, and the first charge under POST /v1/billing/subscribe.

The frontend success page now reflects that split:

  • planId === CHANGE_METHOD_SENTINEL keeps the card-swap path on useRegisterBillingKey();
  • every paid subscribe path calls useSubscribe() once and waits for the backend to finish the operation.

2. The global exception mapper hid the real failure class​

lumie-backend/libs/common/src/main/java/com/lumie/common/exception/GlobalExceptionHandler.java now branches on JDBC SQLSTATE instead of flattening every DataIntegrityViolationException into 409 RESOURCE_CONFLICT.

That change turned the debugging contract into something honest:

  • 23505 stays a real conflict;
  • 23514 becomes 422 BUSINESS_RULE_VIOLATION;
  • 23502, 23503, and unknown integrity failures surface as 500 INTERNAL_ERROR.

3. Billing-key replacement needed explicit flush ordering​

lumie-backend/modules/billing/src/main/java/com/lumie/billing/application/service/BillingKeyPersistenceHelper.java now calls saveAndFlush(existing) before inserting the new key. That ordering matters because the partial unique index only allows one ACTIVE billing key per tenant, and JPA can otherwise defer the revoke update until commit while flushing the insert immediately.

Resolution​

The lasting repository state is:

  • V35__backfill_free_subscriptions.sql restores the "every tenant has a subscription row" invariant for legacy tenants.
  • V36__billing_keys_nullable_card_metadata.sql allows nullable card metadata that Toss sandbox responses can omit.
  • BillingSubscribeService.java is the single backend subscribe boundary.
  • GlobalExceptionHandler.java preserves SQLSTATE-level cause visibility at the HTTP layer.
  • BillingKeyPersistenceHelper.java flushes revoke-before-insert ordering explicitly.
  • BillingSetupSuccess.tsx delegates paid subscription setup to one backend mutation instead of coordinating three.

Regression coverage also survived in current source:

  • lumie-backend/libs/common/src/test/java/com/lumie/common/exception/GlobalExceptionHandlerTest.java
  • lumie-backend/modules/billing/src/test/java/com/lumie/billing/application/service/BillingKeyPersistenceHelperTest.java
  • lumie-backend/modules/billing/src/test/java/com/lumie/billing/application/service/BillingSubscribeServiceTest.java

Prevention​

  • Put the business operation on the backend when the user action spans billing-key registration, subscription state, and a charge.
  • Treat exception-to-HTTP mapping as observability infrastructure, not just API polish.
  • Force write order explicitly when same-transaction updates and inserts share a partial unique index.
  • Keep legacy-invariant repair in migrations, not in frontend retries or ad hoc operator steps.