OMR Grading Performance Optimization

On June 21, 2026, OMR grading performance work focused on one constraint: make the worker faster without changing answers.

The final result combined code-level OpenCV cleanup, native thread caps, and an honest Kubernetes CPU request. In the 100-image production command path, wall time dropped from 49.3 seconds in the original request setting to 21.5 seconds after request tuning plus recognizer and thread optimization.

Context

lumie-worker/services/grading runs the OMR path:

1. download the scanned sheet from MinIO;
2. fetch exam and template context from the backend;
3. run OpenCV-based OMR grading;
4. publish the callback result.

The previous accuracy validation produced the guardrail: 706 deduplicated real scans with golden output. That meant performance changes could be rejected mechanically if they changed grading answers.

The runtime had a hidden concurrency multiplier. RabbitMQ prefetch allowed multiple in-flight grading jobs, while OpenCV used a native thread pool inside each job. With the Linux container defaulting to 10 OpenCV threads, effective runnable work became much larger than the visible queue concurrency.

Impact

The problem was user-visible latency and inefficient CPU use:

• large OMR batches felt slow;
• the pod CPU request did not honestly represent actual demand;
• OpenCV native threads multiplied with queue concurrency;
• tail latency worsened under contention even without a Kubernetes CPU limit.

The goal was not to lower resource settings blindly. The goal was to reduce CPU work per sheet and prevent native-thread oversubscription while preserving output on the real scan corpus.

Diagnosis

Stage timing showed that the bottleneck was CPU-heavy image recognition, not MinIO, RabbitMQ, or backend fetches.

For an 89-image production-like batch, the source record captured:

Stage	Observed result
batch wall time	36.5s
whole usecase latency avg	3.34s
whole usecase latency p95	7.09s
omr_grade avg	3.21s
omr_grade p95	6.94s
image download avg	102ms
exam fetch avg	12ms
callback publish avg	30ms

There was no Kubernetes CPU limit on the pod, so classic CFS throttling was not the direct root cause. The issue was inefficient CPU work plus native-thread oversubscription.

Changes Kept

The retained changes reduced unnecessary OpenCV work while preserving output:

Area	Change	Reason
image decode	decode directly as grayscale	OMR grading only needs intensity information
preprocessing	keep the pipeline grayscale-first	avoid redundant channel conversions
gamma correction	cache the lookup table	avoid rebuilding the same LUT per image
morphology	cache the reusable kernel	avoid repeated kernel allocation
deskew	detect horizontal lines from the top 15% region	relevant form lines are near the top
marker detection	search known top, left, and right marker regions	registration markers live in known regions
density calculation	use count_nonzero	faster for binary masks
answer selection	track top two densities without full sorting	the decision only needs best and second-best
runtime	set OPENCV_NUM_THREADS=4 and cap native numeric pools	reduce oversubscription while preserving throughput

Rejected experiments were as important as accepted ones:

Experiment	Result	Decision
resize with INTER_LINEAR	caused mismatches on the sample set	rejected
resize with INTER_AREA	did not provide useful speed improvement	rejected
remove CLAHE or gamma	caused mismatches	rejected
remove unsharp processing	passed a small sample but failed the full 706-image run	rejected
integral-image density	passed correctness but was slower	reverted
OpenCV threads 1 or 2	lower CPU, but too much throughput loss	not chosen

Benchmarks

Local old-vs-current benchmark

The local comparison over 200 images showed code-level improvement:

Metric	Previous	Current	Change
avg per image	144.27ms	78.49ms	45.6% lower
p50	128.66ms	71.67ms	44.3% lower
p95	259.38ms	123.95ms	52.2% lower
max	697.20ms	231.10ms	66.9% lower

This was useful evidence for the recognizer changes, but Linux container behavior mattered more for thread settings.

Linux thread benchmark

The Linux container benchmark over 706 images with grading concurrency 3 selected four OpenCV threads:

OpenCV threads	Worker concurrency	Wall time	Throughput	CPU time	Avg image	p95 image	Mismatches
10	3	44.278s	15.94 img/s	262.634s	187.63ms	282.45ms	0
4	3	40.785s	17.31 img/s	188.357s	172.93ms	266.37ms	0
2	3	45.037s	15.68 img/s	162.814s	190.91ms	265.23ms	0
4	2	47.158s	14.97 img/s	178.710s	133.33ms	186.76ms	0

Compared with the 10-thread default at concurrency 3, the four-thread setting used about 28.3% less CPU time while improving throughput by about 8.6%.

Production command path

The 100-image production command-path benchmark showed the combined result:

State	Wall time	Usecase avg	omr_grade avg	omr_grade p95
original 50m request	49.3s	3970ms	3828ms	9375ms
request raised to 250m	38.3s	3176ms	2971ms	5266ms
optimized recognizer + 250m + thread env	21.5s	1743ms	1327ms	2011ms

The result preserved the resource posture from the earlier scaling work:

• CPU request set to 250m;
• no CPU limit;
• OPENCV_NUM_THREADS=4;
• native numeric pool caps such as OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1, and NUMEXPR_NUM_THREADS=1.

Verification

Correctness owned the release gate:

Check	Result
uv run pytest in lumie-worker/services/grading	passed in the source record
golden runner over deduplicated real OMR scans	706 checked, 0 mismatches
local old-vs-current benchmark	about 45.6% lower average per-image latency
Linux thread benchmark	lower CPU time and higher throughput at four OpenCV threads
production command-path benchmark	wall time reduced from 49.3s to 21.5s

The optimization was acceptable because the 706-image golden corpus stayed identical.

What Went Well

• Stage timing isolated omr_grade before resource tuning started.
• The golden corpus prevented unsafe speedups from shipping.
• Rejected experiments were recorded, not hidden.
• Linux container benchmarks, not local macOS behavior, drove OpenCV thread selection.

What Went Poorly

• OpenCV's native thread pool was initially invisible compared with RabbitMQ prefetch.
• CPU request was too low for the real workload before measurement.
• Some observability/logging had to be added during the performance work rather than already existing.
• The production benchmark also surfaced a separate Redis routing issue where dedupe SET calls could hit a read-only replica.

Action Items

Action	Owner / surface	Status	Evidence
Keep the golden OMR corpus as a release gate for recognizer changes.	lumie-worker/services/grading	Required for future changes	706-image gate caught unsafe experiments.
Keep OpenCV and numeric thread caps explicit in runtime config.	grading worker deployment values	Done / maintain	OPENCV_NUM_THREADS=4 selected by container benchmark.
Investigate Redis primary routing for dedupe writes.	worker/backend cache routing	Follow-up	Production benchmark surfaced read-only replica writes.

Lessons

• Do not tune Kubernetes resources before isolating the expensive stage.
• Native thread pools multiply with queue concurrency.
• Performance changes in recognition code need an accuracy gate first.
• The best wins came from reducing unnecessary work, not deleting core recognition steps.
• Local benchmarks are useful, but Linux container benchmarks should decide runtime thread settings.