Skip to content

Add docTR as 5th model and tie-breaker in ROVER consensus #28

Description

@Arkoniak

Background

Manual audit of 2-vs-2 content disagreements (see docs/experiments/2026-06-29-surya-spell-checker-bias.md) showed that 4-model ROVER ties are currently broken by insertion order (Surya wins). This works well in practice (35/36 correct) but is brittle: all 4 VLMs carry language-model priors that push toward standard spellings, so on non-standard dialect or rare compound words a tie can go the wrong way.

The fix: add a 5th model that reads glyphs without language-model priors, turning 2-vs-2 into 3-vs-2.

Why docTR

  • Two-stage CNN pipeline (detection + CRNN recognition) — no language model, no spell-checking bias
  • Fast enough to run on the full book (1–2 s/page on GPU, also works on CPU)
  • Gives word-level bboxes + confidence scores, compatible with our block structure
  • Pure Python, pip-installable, no trust_remote_code

Plan

  1. Write scripts/run_doctr.py following the same run layout as run_surya.py / run_unlimited.py — one manifest.json + per-page page_NNN.json with text, blocks (label/bbox/text per word), format: "doctr".

  2. Run on full book with --label full (same suffix as the other models so consensus.py picks it up automatically).

  3. Integrate into consensus.py as a tie-breaker: docTR participates in the full alignment (its tokens go into every column), but in vote_column() its vote only counts when the 4 VLMs are split 2-vs-2. Concretely: compute the 4-VLM majority first; if agreement == 0.5, use docTR to break the tie; otherwise ignore its vote. This preserves the 4-VLM consensus on unambiguous columns while giving an independent signal on contested ones.

  4. Evaluate: re-run consensus, compare content_agreement_rate and check the former 2-vs-2 tie cases (especially Don'/Don't, creds/credits, raied/raised).

Acceptance criteria

  • run_doctr.py produces a valid run directory consumable by consensus.py
  • vote_column() accepts an optional tiebreaker model name and uses it only on 2-vs-2 splits
  • Full-book consensus re-run with docTR as tie-breaker, results recorded in docs/experiments/
  • content_agreement_rate does not regress; the known bad tie (p221 Don't) is resolved

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions