Skip to main content

Solapi Dispatch Webhook State

Use this guide when an SMS send crosses the provider boundary and the failure is in dispatch claiming, provider result persistence, webhook binding, or tenant-scoped history updates.

The broader rollout narrative lives in the Solapi SMS retrospective. This page is the operational runbook.

Symptoms​

  • SMS appears to send through Solapi, but Lumie history remains PENDING or DISPATCHED.
  • Webhook requests return 200 but do not update recipient delivery state.
  • Webhook events are logged as ignored.
  • Retry attempts keep claiming the same message without a durable terminal state.
  • A live-provider send fails only after switching from mock provider behavior.

Known Failure History​

CommitFailure modeDurable fix
afab0183Live provider dispatch needed provider ids, recipient binding, and webhook hardening.Provider group/message tracking, binding validation, and tests.
4de6a6c7REQUIRES_NEW dispatch transactions lost RLS tenant binding.RlsTenantTransactionBinder is called inside dispatch transactions.
01ad30b2Solapi webhook reports did not fully apply provider delivery state.Webhook application updates dispatch state and recipient statuses.
e73277f2An already-applied SMS migration was accidentally edited.Restore historical migration and move forward only.
04aae48cFollow-up SMS index migration needed to be safe for already deployed environments.V77 became a transactional no-op; V76 owns provider columns and indexes.

Current Runtime Contract​

SurfaceSource pathRequired behavior
Dispatch claimmodules/notification/src/main/java/com/lumie/notification/application/service/SmsDispatchService.javaClaim pending messages in a short tenant-bound transaction.
Provider sendSolapiSmsProviderAdapterSend outside the database transaction and return provider group/message ids.
Webhook entrySolapiSmsWebhookControllerVerify X-Solapi-Secret, parse Lumie custom fields, and ignore only known invalid events.
Webhook applicationSmsWebhookService, SmsDispatchService.applyWebhookReport(...)Rehydrate tenant context and validate message, provider group, recipient index, and provider message id.
Migration stateV76__sms_provider_dispatch_tracking.sql, V77__sms_provider_dispatch_tracking_indexes.sqlProvider tracking lives in V76; V77 is intentionally no-op.

Diagnosis​

  1. Identify whether the failure is before provider send, after provider result, or during webhook application.
  2. Check the SmsMessage aggregate state: provider, provider group id, dispatch attempts, recipients, and per-recipient provider message ids.
  3. Check logs for SMS after-commit dispatch failed, provider send errors, or Ignored SOLAPI webhook event.
  4. Verify the webhook contains lumieTenant, lumieSmsId, and lumieRecipient custom fields.
  5. Verify the webhook messageId matches the recipient's provider message id and, when present, the group id matches the message provider group id.
  6. Confirm dispatch transactions call RlsTenantTransactionBinder.bindRequiredTenantToCurrentTransaction().
  7. If schema drift is suspected, verify V76 and V77 match the current forward-only contract.

Database fields worth inspecting include provider_group_id, dispatch_attempt_count, last_provider_status, last_provider_error, dispatched_at, confirmed_at, and the recipient-level provider ids in the recipients payload.

Useful source checks:

cd lumie-backend
rg -n "applyWebhookReport|bindRequiredTenantToCurrentTransaction|lumieTenant|providerGroupId|claimDispatchAttempt" \
  modules/notification app/src/main/resources/db/migration/public

Fix​

  • If dispatch runs after commit, preserve TenantContextHolder.withinContext(...) around dispatchMessage(...).
  • If using TransactionTemplate, bind the required tenant inside the transaction before repository reads or writes.
  • If webhook events are ignored, fix the custom fields or provider id binding rather than accepting unbound provider reports. A missing provider group id can fall back to Lumie's message id, but conflicting ids must still be rejected.
  • If provider send returns partial recipient results, merge them by recipient index and provider message id.
  • If migration drift appears, restore historical migrations exactly and add a new forward migration.

Verification Evidence​

  • a pending message is claimed once and leaves retry state after provider response,
  • provider group id and per-recipient provider message ids are stored,
  • a valid Solapi webhook updates the intended recipient only once,
  • mismatched tenant, group id, message id, or recipient index is ignored without mutating history,
  • dispatch and webhook writes pass under RLS for the correct tenant,
  • SMS history exposes the sender number and final delivery status expected by operators.

Prevention​

  • Keep provider callbacks bound to Lumie ids through custom fields and provider ids; do not trust provider status alone.
  • Keep live-provider writes in explicit tenant-bound transactions.
  • Do not treat webhook 200 as proof of state mutation; check the aggregate.
  • Keep migration fixes forward-only, especially for provider tracking columns and indexes.
  • Maintain mock-provider behavior separately from live Solapi binding rules.