Hybrid PDF extraction: add DoclingTableRefiner step in ExtractionPipeline

[smodee]

Background

The Docling vs in-tree PDF-extraction evaluation in data/docling_eval/FINDINGS.md tested both stacks against 5 real biosecurity PDFs. Headline:

• In-tree (PdfParser, PyMuPDF + pdfplumber) is ~240× faster, gets publication dates and titles from PDF metadata, fails fast on scanned PDFs.
• Docling (TableFormer) is the only path that recovers two table-extraction failure modes observed in just 5 docs:
  1. Borderless tables (e.g. CDC MMWR demographic breakdowns): in-tree returned 6×13 of empty cells; Docling returned a clean 17×2 with every row populated. PyMuPDF and pdfplumber both fail on this layout — there is nothing to post-process.
  2. Dense merged-cell tables (e.g. WHO cholera country/cases/deaths matrix): in-tree over-segmented to 23×20 with values in shifting columns; Docling produced clean 21×8. Patchable in-tree with ~30 lines of heuristic post-processing, but parsing remains brittle.

For prose, headings, dates, and the common case, in-tree is the right default. For table extraction on hard layouts, only a learned-model approach (Docling, gmft, table-transformer/TATR — same class of solution) works reliably.

Goal

Introduce a hybrid PDF extraction path: in-tree as the default, Docling as a targeted post-pass that refines table sections when triggered. Synchronous (blocking) is fine — extraction is not on a latency-critical path.

Architecture

Add a new DoclingTableRefiner step that sits in ExtractionPipeline after parsing, before chunking:

ExtractionPipeline:
  1. Fetch
  2. Select parser (PdfParser, HtmlParser, …)
  3. Parse → ParsedContent
  4. [NEW] DoclingTableRefiner.refine(parsed) → ParsedContent (possibly with refined tables)
  5. Chunk

Properties of the new step:

• Pluggable: a single class (DoclingTableRefiner) in bioscancast/extraction/docling_refiner.py. The pipeline calls refiner.refine(parsed, source_url, content_bytes) and gets back a possibly-modified ParsedContent.
• Feature-flagged: a config flag toggles the whole step. When disabled, behaviour is identical to today.
• Lazy-import Docling: the heavyweight imports (docling, transformers, torch, …) happen only when the refiner is actually instantiated, so disabling the flag means zero runtime cost.
• Per-process singleton: DocumentConverter is expensive to construct (model load ~10-30 s, ~1.5 GB RAM); reuse across calls.
• HTML is untouched: refiner only runs for ParsedContent produced by PdfParser. HTML continues through the existing trafilatura + BS4 path.

Trigger policy (hybrid)

Two complementary triggers — first that fires wins:

Trigger 1 — source-family allowlist (fast path)

A configured allowlist of source URL patterns that are known to contain hard tables. Match against source_url; if any pattern matches → run Docling unconditionally.

Initial seed list (based on the eval; add more over time):

Pattern	Source family	Reason
cdc.gov/mmwr/	CDC MMWR	Borderless tables, observed total in-tree failure
cdn.who.int/.../situation-reports/	WHO outbreak situation reports	Dense merged-cell tables (cholera)
cdn.who.int/media/docs/default-source/_sage-	WHO sitreps (mpox-family)	Dense merged-cell tables

Lives in bioscancast/extraction/config.py as a list of regex patterns (or substrings — keep it simple unless we hit a false-match).

Trigger 2 — heuristic "looks broken" check (safety net)

For any PDF not on the allowlist, inspect each SectionContent with chunk_type == "table":

• Sparse-cell heuristic: count non-empty cells. If non_empty / total_cells < 0.5 AND there are at least 3 rows and 2 columns, flag the table as suspect.
• Single-cell-per-row heuristic: if more than half the rows have exactly one non-empty cell, flag (catches the cholera-style over-segmentation where each numeric column got 3 sub-columns).

If any table section in the doc is flagged → run Docling on the whole PDF.

Trigger logging

Log (info level) whenever the refiner triggers, with reason: "docling refiner triggered: source-allowlist hit cdc.gov/mmwr/" or "docling refiner triggered: suspect table on page 4 (empty-cell ratio 0.08)". Helps us tune the heuristic and grow the allowlist.

Implementation steps

1. Add Docling deps to requirements.txt:

   • docling[chunking]>=2.90,<3.0
   • First-run download of layout/tableformer models (~40 MB) lands in HuggingFace cache; document this in the README.
   • Verify Python version compatibility (eval ran on 3.13; pin minimum supported version).

2. Create bioscancast/extraction/docling_refiner.py with:

   • DoclingTableRefiner class
     • Constructor builds the DocumentConverter with PdfPipelineOptions(do_ocr=False, do_table_structure=True, table_structure_options.mode=TableFormerMode.FAST).
     • refine(self, parsed: ParsedContent, *, source_url: str, content: bytes) -> ParsedContent
   • Helper functions: _should_refine_by_url(source_url), _should_refine_by_heuristic(parsed), _merge_docling_tables_into_parsed(parsed, docling_doc).

3. Table-merge strategy: when Docling produces a DoclingDocument, replace in-tree's table sections by page number. For each SectionContent with chunk_type=="table", find the Docling table whose first occurrence is on the same page; if found, replace table_rows with Docling's rendering. If multiple tables on a page, match in order of appearance.

   • Add provenance: extend SectionContent with extractor: Optional[str] = None field (values: "pymupdf", "pdfplumber", "docling"). Default None for back-compat.

4. Plug into ExtractionPipeline (bioscancast/extraction/pipeline.py):

   • After parser.parse(...), before normalize_chunks(...), call self._docling_refiner.refine(...) if the refiner is enabled.
   • Construct DoclingTableRefiner lazily on first use, store as instance attribute.

5. Config knob: add to bioscancast/extraction/config.py:

   @dataclass
   class ExtractionConfig:
       ...
       enable_docling_refiner: bool = True
       docling_source_allowlist: list[str] = field(default_factory=lambda: [
           "cdc.gov/mmwr/",
           "cdn.who.int/media/docs/default-source/_sage-",
           "cdn.who.int/media/docs/default-source/documents/emergencies/situation-reports/",
       ])
       docling_sparse_cell_threshold: float = 0.5

6. Logging: use the existing module logger; INFO when triggered, DEBUG for skip reasons (already on allowlist mismatch + no broken-table flag).

7. Tests under bioscancast/extraction/tests/test_docling_refiner.py:

   • _should_refine_by_url: matches allowlist, doesn't match unrelated.
   • _should_refine_by_heuristic: flags a synthetic ParsedContent with mostly-empty table cells; passes through a healthy table.
   • refine end-to-end with a mocked Docling converter (return a stub DoclingDocument-like object with a known table). Verify provenance is set correctly.
   • Do not load the real Docling model in CI. Real-model integration tests can be run locally against PDFs pulled fresh from the publisher URLs (we keep those URLs documented in data/docling_eval/FINDINGS.md).

Definition of done

• DoclingTableRefiner lives in bioscancast/extraction/docling_refiner.py, has unit tests with mocked Docling.
• ExtractionPipeline calls the refiner between parse and chunk when enabled.
• Config flag enable_docling_refiner works (off ⇒ no Docling imports, no behaviour change).
• Allowlist + heuristic trigger both implemented and tested.
• SectionContent has an extractor field; in-tree paths set it to "pymupdf"/"pdfplumber", refiner sets it to "docling".
• When run locally against the 5 eval PDFs:
  • MMWR (mm7509a1-H.pdf): table is populated by Docling (matches data/docling_eval/cdc_mmwr_nm_measles.md's 17×2 table, not in-tree's 6×13 empties).
  • WHO cholera: table is populated by Docling (21×8 clean), not in-tree's 23×20.
  • WHO mpox: table is in-tree's output (no refinement triggered for a doc not on allowlist with clean tables).
  • Africa CDC: refiner is not triggered (parsed content carries is_partial=True, partial_reason="requires_ocr"; refiner short-circuits when is_partial indicates OCR is needed).
  • Reuters/CIDRAP/ProMED HTML: untouched by refiner.
• Logs make it clear which docs triggered the refiner and why.
• README mentions the first-run model download.

Out of scope (separate follow-up issues)

• Per-page Docling re-run: this issue runs Docling on the whole PDF. If we see unacceptable wall-clock on large docs, optimise to convert(page_range=(N,N)) only on pages with suspect tables. Tracked in Optimise DoclingTableRefiner: per-page Docling re-run for large PDFs #17.
• curl_cffi for bot-protected HTML sources (Reuters 401s on Cloudflare). Tracked in Add curl_cffi to fetcher to handle Cloudflare-fronted HTML sources #18.
• External OCR routing for fully-scanned PDFs (Africa CDC). Current behaviour: in-tree marks is_partial=True, partial_reason="requires_ocr"; refiner skips them; downstream code decides. No new path in this issue.

References

• Full eval write-up: data/docling_eval/FINDINGS.md
• Eval scripts: scripts/eval_docling.py, scripts/eval_intree_pdf.py, scripts/eval_docling_ocr_cost.py
• In-tree PDF parser: bioscancast/extraction/parsers/pdf_parser.py