What and why
The current consensus output (body_text) is a flat word stream — all structural information
(paragraph boundaries, heading labels, inline emphasis) is discarded after voting. To hand useful
content to Pandoc we need a richer intermediate: a per-token list with voted attributes that an
HTML assembler can turn into proper markup.
Changes to consensus.py
Token enrichment:
emphasis: bool → str | None — distinguish 'italic' / 'bold' / 'bold_italic' / None
(Surya: <i>/<b> HTML; Qwen3-VL: *…* italic, **…** bold)- Add
paragraph_start: bool — True for the first token of each source block
Voting additions (same participation rules as existing emphasis voting):
paragraph_start — only block-capable models vote (Surya + Unlimited grounding); naïve majorityemphasis type — only emphasis-capable models vote (Surya + Qwen3-VL); naïve majority- Conflicts flagged in
disagreements entries (same pattern as existing disagreement log)
Output: add voted_tokens to per-page JSON alongside existing body_text (backward compat):
"voted_tokens": [
{"text": "Chapter", "label": "title", "emphasis": null, "paragraph_start": true},
{"text": "One", "label": "title", "emphasis": null, "paragraph_start": false},
{"text": "He", "label": "text", "emphasis": null, "paragraph_start": true},
{"text": "said", "label": "text", "emphasis": null, "paragraph_start": false},
{"text": "nothing", "label": "text", "emphasis": "italic", "paragraph_start": false}
]New script: build_html.py
Reads voted_tokens from all consensus pages in order; outputs book.html (or chapters/*.html).
Label normalization (Surya/Unlimited → canonical):
| Surya |
Unlimited |
Canonical |
| Title / SectionHeader |
title |
title / heading |
| Text |
text |
text |
| PageHeader / PageFooter |
header / footer |
(skipped) |
| Picture |
image |
(raster placeholder) |
| Caption |
— |
caption |
HTML mapping:
title → <h1>, heading → <h2>, text → <p>caption → <figcaption> inside <figure>picture → <figure><img src="…"></figure> (path from Unlimited's extracted raster)emphasis: 'italic' → <em>, 'bold' → <strong>, 'bold_italic' → <strong><em>
Cross-page stitching (replaces the plain-text stitch.py approach from #20): detect open
paragraphs at page N/N+1 boundary using the same word-split / paragraph-continuation heuristics,
but now operating on the structured token stream — much cleaner because headers are already
gone and we know the label of the last/first block.
Naïve-first policy
For paragraph_start and label voting, ship the simplest working implementation and mark
disagreements in the output. Audit visually (same process as the content-disagreement audit).
Refine only after seeing real failure modes.
Downstream dependencies
This issue is a prerequisite for:
What and why
The current consensus output (
body_text) is a flat word stream — all structural information(paragraph boundaries, heading labels, inline emphasis) is discarded after voting. To hand useful
content to Pandoc we need a richer intermediate: a per-token list with voted attributes that an
HTML assembler can turn into proper markup.
Changes to consensus.py
Token enrichment:
emphasis: bool → str | None— distinguish'italic'/'bold'/'bold_italic'/None(Surya:
<i>/<b>HTML; Qwen3-VL:*…*italic,**…**bold)paragraph_start: bool—Truefor the first token of each source blockVoting additions (same participation rules as existing emphasis voting):
paragraph_start— only block-capable models vote (Surya + Unlimited grounding); naïve majorityemphasistype — only emphasis-capable models vote (Surya + Qwen3-VL); naïve majoritydisagreementsentries (same pattern as existing disagreement log)Output: add
voted_tokensto per-page JSON alongside existingbody_text(backward compat):New script: build_html.py
Reads
voted_tokensfrom all consensus pages in order; outputsbook.html(orchapters/*.html).Label normalization (Surya/Unlimited → canonical):
HTML mapping:
title→<h1>,heading→<h2>,text→<p>caption→<figcaption>inside<figure>picture→<figure><img src="…"></figure>(path from Unlimited's extracted raster)emphasis: 'italic'→<em>,'bold'→<strong>,'bold_italic'→<strong><em>Cross-page stitching (replaces the plain-text stitch.py approach from #20): detect open
paragraphs at page N/N+1 boundary using the same word-split / paragraph-continuation heuristics,
but now operating on the structured token stream — much cleaner because headers are already
gone and we know the label of the last/first block.
Naïve-first policy
For paragraph_start and label voting, ship the simplest working implementation and mark
disagreements in the output. Audit visually (same process as the content-disagreement audit).
Refine only after seeing real failure modes.
Downstream dependencies
This issue is a prerequisite for: