OMR Grading Accuracy Validation

On June 20-21, 2026, the OMR grading work shifted from speed to correctness. The question was whether Lumie should keep its own grade-svc, replace it with OMRChecker, or use OMRChecker only as a reference implementation.

The decision was to keep Lumie's engine, build a repeatable real-scan review loop, and tighten the decision rule so blank or ambiguous rows become invalid instead of silently selected.

Context

Lumie's grading worker extracts:

• student phone-number bubbles;
• 45 multiple-choice answers;
• invalid answers when marks are absent or ambiguous.

The implementation lives in lumie-worker/services/grading. It uses Lumie's fixed answer-sheet layout, registration-marker alignment, OpenCV preprocessing, and density-based bubble selection.

OMRChecker was worth testing because it is a template-driven open-source engine. But a replacement had to work on Lumie's real scanned forms, including weak pen marks, correction fluid, printed bubble artifacts, phone-number regions, damaged barcodes, and marker alignment issues.

Impact

The product risk was score correctness. A fast recognizer is unacceptable if it silently chooses answers from blank or ambiguous rows.

The validation work found two important classes of risk:

Risk	Why it matters
false positive selected answers	can change a score without an obvious failure signal
false negative real marks	can mark a real answer invalid or blank
wrong phone digits	can attach a sheet to the wrong student or fail matching
damaged forms	should be rejected or excluded, not tuned into false confidence

The resulting product bias was explicit: returning invalid is safer than inventing a selected answer.

What Failed

The comparison exposed several distinct failure modes:

Failure mode	Observed behavior	Product risk
weak real marks	OMRChecker sometimes returned empty for visible but light marks	false negatives for real answers
correction fluid	OMRChecker often treated residual printed bubbles as multiple marks	wrong or invalid answers after correction
blank answer false positives	Lumie's previous rule sometimes selected the highest-density bubble in a blank row	silent score changes
phone-number false positives	blank digit columns leaned toward dense printed digits such as 8 or 9	wrong student matching
damaged barcode or marker areas	a few scans could not be trusted as valid input	reject or exclude instead of tuning around the damage

The key observation was that every blank row still has a "winner" if the algorithm only chooses the highest-density bubble. Printed ink, compression artifacts, dust, and scan shading create relative differences even when the student did not mark an answer.

Diagnosis

The old Lumie rule was too absolute:

1. calculate density for each bubble;
2. choose the highest-density candidate;
3. accept it if it crosses a minimum threshold.

The recovered rule adds a margin requirement:

Area	Previous rule	Recovered rule
multiple-choice answers	accept highest density above minimum	accept only when highest density also beats the second candidate by a margin
phone digits	accept highest digit above minimum	accept only with a larger digit-specific margin

That margin requirement captures the difference between a real mark and a noisy blank row. It also keeps the implementation small enough to reason about while moving the confidence test closer to the actual product risk.

Validation Loop

The validation workflow made the decision reviewable:

1. deduplicate the scan corpus;
2. run the grading engine over every unique sheet;
3. generate overlays showing selected bubbles;
4. review sheets in a local browser tool;
5. record good, bad, and unsure states;
6. rerun after every meaningful threshold or pipeline change.

Final local review state from the source record:

Item	Count
unique reviewed images	706
good	705
bad	1
unsure	0
damaged barcode unique images removed	2

The remaining bad item was kept visible rather than hidden behind a perfect-success claim.

Decision

Lumie did not replace grade-svc with OMRChecker.

OMRChecker remained useful as a reference and comparison target, but direct replacement would have traded one class of errors for another. The lower-risk path was:

1. keep Lumie's fixed-template engine;
2. reject non-Lumie or physically damaged forms;
3. add corpus-backed regression checks;
4. borrow only ideas that survive real Lumie scans;
5. preserve invalid/ambiguous answers instead of forcing selection.

Verification

The correctness gate became the prerequisite for later performance work:

Level	Check	Result
unit	uv run pytest tests/test_omr_density_thresholds.py	passed in the source record
service suite	uv run pytest in lumie-worker/services/grading	passed in the source record
corpus review	local review UI over deduplicated scans	705 good / 706 total
damaged-form handling	damaged barcode images excluded or rejected	2 unique damaged barcode images removed

Future OMR recognition changes should be blocked unless they preserve the golden corpus or clearly document intentional behavior changes.

What Went Well

• The evaluation used real Lumie scans instead of synthetic confidence.
• Visual overlays made recognition mistakes easier to review than CSV diffs.
• The remaining bad/unknown cases were kept explicit.
• The work produced a correctness gate that later performance optimization could depend on.

What Went Poorly

• The earlier highest-density rule could choose a blank row's strongest artifact.
• Phone-number regions needed stricter margin handling than ordinary answers.
• Damaged input handling had to be separated from algorithm tuning.
• The source record does not preserve every review artifact in a publishable location.

Lasting Invariants

• Prefer invalid output over invented answers when confidence is weak.
• Do not swap recognition engines without testing the real scan distribution.
• Keep a golden corpus for any change that affects scoring.
• Treat physically damaged or non-Lumie sheets as input-validity problems, not threshold-tuning targets.

Related Page

• OMR Grading Performance Optimization