Skip to main content

Grade Service Optimistic Lock Incident

Context​

On July 10, 2026, grade-svc finished grading all 100 images for jobId=135, but the backend job never moved past processed_images=99, success_count=99, and saved_count=99. The failed callback was imageIndex=91 for examId=79, and the recorded exception was ObjectOptimisticLockingFailureException on ExamResult#1168.

The confusing part was that worker-side processing looked healthy:

  • grading logs showed 100/100 work complete;
  • primary RabbitMQ queues were empty;
  • the failed callback was sitting in DLQ;
  • replaying that payload was skipped as a duplicate.

What Broke​

The incident was not just "an optimistic lock happened." The structural failure was that callback deduplication became visible before the durable database work completed.

The failure pattern was:

  1. Backend accepted the callback and reserved its dedup key.
  2. The transaction updating ExamResult and the OMR job counters hit an optimistic-lock failure.
  3. The callback became replayable only through DLQ or manual intervention, but replay immediately saw the dedup marker and skipped the payload.
  4. The backend job stayed stuck at 99/100 even though grading had already finished upstream.

That broke the required invariant: a callback should count as a duplicate only after its database side effects are durably complete.

Root Cause​

The incident note ties the race to a rollout window that delayed callbacks and then released them in a burst, but the deeper problem was normal callback concurrency against shared exam-result rows. Multiple callbacks touching the same ExamResult can legitimately race during ordinary production traffic, so the backend callback path had to tolerate optimistic locking instead of treating it as terminal.

The current backend source shows that this incident drove a durable fix in lumie-backend/modules/exam/src/main/java/com/lumie/exam/application/service/OmrGradingJobProcessor.java:

  • callbacks retry OptimisticLockingFailureException up to three times;
  • the dedup key is released through callbackDeduplication.forget(...) when the transaction fails before completion;
  • finalization happens only after the callback has been counted successfully.

The regression coverage is also present today in lumie-backend/modules/exam/src/test/java/com/lumie/exam/application/service/OmrGradingJobProcessorTest.java, including retry and dedup-release cases.

Resolution​

The durable recovery pattern has three parts:

  • retry optimistic-lock failures with a short backoff instead of DLQ-ing immediately;
  • clear the dedup reservation when the callback transaction fails before the counters are committed;
  • keep job counters and finalization tied to successful callback state transitions.

Those protections now exist in the OMR callback processor, so the July 2026 failure mode should replay safely instead of requiring direct database edits.

Prevention​

  • Treat callback concurrency as a steady-state property, not a rollout-only edge case.
  • Keep callback deduplication and durable completion aligned. If the database work fails, replay must stay possible.
  • Prefer tests that simulate retry, duplicate replay, and terminal-state finalization on the same job id.