Skip to main content

OMR Report Job Lifecycle

Use this guide when OMR grading or report generation has already left the request path and the failure is in backend job state, callback counting, or artifact finalization.

This is distinct from worker CPU scaling. If the worker is simply slow or undersized, start with OMR Grading Scaling.

Symptoms​

  • An OMR or report job stays PENDING or PROCESSING after worker callbacks were expected.
  • A job reaches terminal state but success/fail counters do not add up to the requested item count.
  • Duplicate callbacks are skipped but the job never finalizes.
  • Report ZIP download returns not found even though individual PDFs exist.
  • A retry or late callback mutates a terminal job.

Known Failure History​

CommitFailure modeDurable fix
2b20846eJob row persisted but RabbitMQ publish could be lost between commit and broker send.Spring Modulith transactional outbox via JobRequestForwarder.
bde8f625Concurrent callbacks hit optimistic locks and dedup keys could outlive failed transactions.Retry callback updates and forget dedup keys after uncommitted failures.
5023d873OMR results were not tied to per-job lifecycle strongly enough, and missing job versioning could break callback persistence.Added V40 versioning, V41 result-to-job linkage, V42 lookup index, and status endpoints.
4820197fStale async jobs could remain open indefinitely after worker loss.ExamJobTimeoutSweeper marks old PENDING/PROCESSING OMR and report jobs as FAILED.
a98cc026Batch report data used the wrong context for chunked jobs.ReportBatchDataQueryService validates job, exam, tenant, and student chunk membership.

Current Runtime Contract​

SurfaceSource pathRequired behavior
Job creationResultCommandService.confirmBatchOmrGrading(...), ReportJobService.createBatchJob(...)Persist job and publish request events in the same transaction.
Outbox forwardingmodules/exam/src/main/java/com/lumie/exam/adapter/out/messaging/JobRequestForwarder.javaForward committed outbox events to RabbitMQ and retry outstanding publications on restart.
OMR callbackOmrGradingJobProcessor.processGradingCallback(...)Deduplicate per job/image, retry optimistic locks, finalize only when counters reach total.
Report callbackReportJobService.processCallback(...)Deduplicate per job/report, validate PDF artifact, finalize and create ZIP when done.
Timeout sweepExamJobTimeoutSweeper.sweepStaleJobs(...)Fail stale open jobs after a generous timeout so polling resolves and users can retry.
Batch dataReportBatchDataQueryService.getBatchData(...)Validate job ownership, exam id, tenant, and chunk membership.

The outbox retry behavior is configured in app/src/main/resources/application.yaml with spring.modulith.events.republish-outstanding-events-on-restart=true and completion-mode=delete.

Diagnosis​

  1. Determine whether the job is OMR grading or report generation.
  2. Read the job row status, total count, processed count, success count, and fail count.
  3. Check backend logs for Background job enqueued, enqueue_failed, duplicate callback skipped, optimistic lock retry, or terminal-state logs.
  4. If no worker message was sent, inspect the Spring Modulith event_publication state before blaming RabbitMQ.
  5. If callbacks arrived but counters are wrong, inspect dedup keys and optimistic lock retry logs.
  6. If a worker disappeared, check whether ExamJobTimeoutSweeper marked stale jobs FAILED after the timeout window.
  7. If reports exist but ZIP download fails, inspect report fileKey validation and ZIP zipFileKey.
  8. If batch report data is wrong for a subset, verify the jobId, examId, studentIds, and tenant slug all belong to the same job.

Useful source check:

cd lumie-backend
rg -n "event_publication|tryFinalize|markIfAbsent|finalizeIfDone|zipFileKey|close.*stale|existsByIdAndStatusIn" \
  modules/exam app/src/main/resources

Fix​

  • If the job row exists but no worker request was sent, preserve the outbox contract and replay outstanding events rather than manually publishing ad hoc messages.
  • If duplicate callback handling blocked progress after an uncommitted failure, release the dedup key and let the callback retry path run again.
  • If counters are below total and no more callbacks will arrive, use the stale-job close path rather than leaving the job PROCESSING.
  • If report ZIP generation failed after terminal completion, rebuild the ZIP from validated per-student PDFs through ReportJobService.getZipBytes(...).
  • If batch report data is using the wrong chunk, fix the jobId/studentIds request path rather than widening the query.
  • If callback processing fails with missing column or result-to-job lookup errors, verify V40, V41, and V42 have applied in order before changing Java code.

Verification Evidence​

  • job status reaches COMPLETED or FAILED with counters matching total,
  • outbox publications for the job are complete,
  • event_publication has no stuck publication for the job request,
  • duplicate callbacks do not increment counters twice,
  • late callbacks do not mutate terminal jobs,
  • report ZIP downloads successfully after finalization,
  • tenant/job/exam/student validation rejects mismatched batch data requests.

Prevention​

  • Keep job persistence and enqueue intent atomic through the outbox.
  • Keep worker callbacks idempotent per item index.
  • Keep storage I/O outside long database transactions, but persist counters in short transactions.
  • Do not remove optimistic-lock retries without replacing the concurrency protection.
  • Treat job state, callback dedup, and artifact creation as one lifecycle, not separate features.