Skip to main content

SOLAPI SMS Rollout

Context​

The notification module started with queued tenant-scoped SMS history and scheduling, then expanded into a live provider rollout on June 28, 2026. The main steps were:

  • 3289196c: add provider dispatch, scheduler flow, webhook intake, and sms_messages provider-tracking columns.
  • afab0183, e73277f2, and 04aae48c: harden the schema rollout after index and migration-order issues.
  • 4de6a6c7: repair the transaction boundary so post-commit dispatch and webhook writes still bind app.tenant_id under RLS.
  • 03981d38, 01ad30b2, and 13852f8d: remove a SOLAPI-specific LMS subject mismatch, accept webhook payloads that omit groupId, and switch live sends from global config to tenant-owned sender numbers.

The steady-state contract now lives in the Notification Service reference page.

What Broke Or Changed​

The rollout was not just "turn on SOLAPI." It changed three contracts at once:

  • outbound send execution moved from a single in-transaction queue write to after-commit provider dispatch in SmsDispatchService;
  • provider state became part of the persisted message contract through V76__sms_provider_dispatch_tracking.sql;
  • live sender identity moved from app config to tenant-scoped sms_sender_settings in V78__sms_sender_settings.sql.

That exposed failure modes that the queued MVP did not have:

  • two dispatch workers could race the same PENDING row unless claim state was persisted first;
  • a REQUIRES_NEW transaction could lose the original tenant binding and read an empty RLS view;
  • SOLAPI webhook payloads were not perfectly stable, especially around groupId and LMS-only fields;
  • a global SOLAPI_FROM_NUMBER was not enough once one backend served many tenants.

Diagnosis And Decision Ladder​

  1. The first live-provider pass added provider columns and a scheduler, but it still assumed the same transaction and tenant context would naturally survive into post-commit work.
  2. Hardening moved dispatch from "load row, then mutate" to "claim first, then load" so retries and concurrent schedulers could not double-send the same message.
  3. The migration plan briefly split the provider indexes into V77 with CREATE INDEX CONCURRENTLY, then backed away after rollout pain. Current source keeps V76 as the real index-bearing migration and leaves V77 as a transactional no-op.
  4. The decisive fix was to stop relying on the declarative RLS aspect for programmatic transactions. SmsDispatchService now runs claim, result, failure, and webhook writes through a dedicated @RequiresNewTransaction template and explicitly binds the tenant before repository access.
  5. Once tenant-owned sender numbers arrived, live SOLAPI dispatch stopped reading a single app-level from-number and instead resolved the saved message sender or the tenant's approved fallback.

Current dispatch writes depend on this explicit binder path:

// lumie-backend/modules/notification/src/main/java/com/lumie/notification/application/service/SmsDispatchService.java
private <T> T executeDispatchTransaction(TransactionCallback<T> callback) {
    return dispatchTransactionTemplate.execute(status -> {
        rlsTenantTransactionBinder.bindRequiredTenantToCurrentTransaction();
        return callback.doInTransaction(status);
    });
}

Resolution​

The current rollout shape is stable because the code now separates concerns clearly:

  • SmsDispatchService claims work, performs provider I/O outside the transaction, then persists the outcome in a fresh tenant-bound transaction.
  • RlsTenantTransactionBinder binds app.tenant_id directly onto the JDBC connection used by that transaction.
  • SolapiSmsWebhookController and SmsWebhookService rehydrate tenant context from callback customFields, then SmsDispatchService validates message id, provider message id, and provider group id before mutating history.
  • SolapiSmsProviderAdapter no longer sends the LMS subject field and rejects live sends that do not have a resolved sender number.
  • V78__sms_sender_settings.sql adds the tenant RLS table that owns approved sender numbers, while SmsMessageResponse exposes the saved senderNumber so history reflects the real dispatch identity.

The current source-of-truth files are:

  • lumie-backend/modules/notification/src/main/java/com/lumie/notification/application/service/SmsDispatchService.java
  • lumie-backend/modules/notification/src/main/java/com/lumie/notification/application/service/SmsWebhookService.java
  • lumie-backend/modules/notification/src/main/java/com/lumie/notification/adapter/out/external/SolapiSmsProviderAdapter.java
  • lumie-backend/libs/common/src/main/java/com/lumie/common/tenant/RlsTenantTransactionBinder.java
  • lumie-backend/app/src/main/resources/db/migration/public/V76__sms_provider_dispatch_tracking.sql
  • lumie-backend/app/src/main/resources/db/migration/public/V77__sms_provider_dispatch_tracking_indexes.sql
  • lumie-backend/app/src/main/resources/db/migration/public/V78__sms_sender_settings.sql

Prevention​

  • Treat provider rollout work as a cross-cutting change across queueing, tenancy, schema, and callback contracts, not as an adapter-only feature.
  • Keep live-provider writes on explicit transaction infrastructure. Programmatic TransactionTemplate flows do not get the RLS aspect for free.
  • Preserve message-level provider state in the database before trusting asynchronous callbacks.
  • Prefer tenant-owned sender settings over global SMS sender config. Current application.yaml exposes lumie.sms.solapi.{api-key,api-secret,base-url,webhook-token} only; SOLAPI_FROM_NUMBER is no longer part of the runtime contract.
  • When webhook providers send inconsistent payloads, validate bindings defensively and allow the minimum backward-compatible fallback the stored row can prove. The current code accepts missing providerGroupId only when the message identity still matches.