Background
Manual audit of 2-vs-2 content disagreements (see docs/experiments/2026-06-29-surya-spell-checker-bias.md) showed that 4-model ROVER ties are currently broken by insertion order (Surya wins). This works well in practice (35/36 correct) but is brittle: all 4 VLMs carry language-model priors that push toward standard spellings, so on non-standard dialect or rare compound words a tie can go the wrong way.
The fix: add a 5th model that reads glyphs without language-model priors, turning 2-vs-2 into 3-vs-2.
Why docTR
- Two-stage CNN pipeline (detection + CRNN recognition) — no language model, no spell-checking bias
- Fast enough to run on the full book (1–2 s/page on GPU, also works on CPU)
- Gives word-level bboxes + confidence scores, compatible with our block structure
- Pure Python, pip-installable, no
trust_remote_code
Plan
-
Write scripts/run_doctr.py following the same run layout as run_surya.py / run_unlimited.py — one manifest.json + per-page page_NNN.json with text, blocks (label/bbox/text per word), format: "doctr".
-
Run on full book with --label full (same suffix as the other models so consensus.py picks it up automatically).
-
Integrate into consensus.py as a tie-breaker: docTR participates in the full alignment (its tokens go into every column), but in vote_column() its vote only counts when the 4 VLMs are split 2-vs-2. Concretely: compute the 4-VLM majority first; if agreement == 0.5, use docTR to break the tie; otherwise ignore its vote. This preserves the 4-VLM consensus on unambiguous columns while giving an independent signal on contested ones.
-
Evaluate: re-run consensus, compare content_agreement_rate and check the former 2-vs-2 tie cases (especially Don'/Don't, creds/credits, raied/raised).
Acceptance criteria
Background
Manual audit of 2-vs-2 content disagreements (see
docs/experiments/2026-06-29-surya-spell-checker-bias.md) showed that 4-model ROVER ties are currently broken by insertion order (Surya wins). This works well in practice (35/36 correct) but is brittle: all 4 VLMs carry language-model priors that push toward standard spellings, so on non-standard dialect or rare compound words a tie can go the wrong way.The fix: add a 5th model that reads glyphs without language-model priors, turning 2-vs-2 into 3-vs-2.
Why docTR
trust_remote_codePlan
Write
scripts/run_doctr.pyfollowing the same run layout asrun_surya.py/run_unlimited.py— onemanifest.json+ per-pagepage_NNN.jsonwithtext,blocks(label/bbox/text per word),format: "doctr".Run on full book with
--label full(same suffix as the other models soconsensus.pypicks it up automatically).Integrate into
consensus.pyas a tie-breaker: docTR participates in the full alignment (its tokens go into every column), but invote_column()its vote only counts when the 4 VLMs are split 2-vs-2. Concretely: compute the 4-VLM majority first; ifagreement == 0.5, use docTR to break the tie; otherwise ignore its vote. This preserves the 4-VLM consensus on unambiguous columns while giving an independent signal on contested ones.Evaluate: re-run consensus, compare
content_agreement_rateand check the former 2-vs-2 tie cases (especiallyDon'/Don't,creds/credits,raied/raised).Acceptance criteria
run_doctr.pyproduces a valid run directory consumable byconsensus.pyvote_column()accepts an optionaltiebreakermodel name and uses it only on 2-vs-2 splitsdocs/experiments/content_agreement_ratedoes not regress; the known bad tie (p221Don't) is resolved