Join paragraphs and words split across page boundaries — now at HTML level, inside build_html.py.
Depends on: #30 (enriched voted_tokens per page; headers already stripped from body_text
and label = page_header/page_footer is already voted — no pre-processing needed here).
The original plain-text stitch.py approach is superseded: operating on the structured token stream
(with paragraph_start and label attributes) is cleaner and avoids the header-wedging problem
described in brainstorm Layer 4.
Logic (unchanged):
- Word split: last token ends with ASCII
- + first token of next page is lowercase → join without hyphen
- Paragraph continuation: no terminal punctuation (
.!?") + first token lowercase → join with space
- Otherwise: new paragraph break (
</p><p> boundary)
Distinguish — (em-dash, not a word-break) from - (ASCII hyphen at line end).
Unlimited multi-page mode as optional cross-check signal for flagging uncertain boundaries.
See brainstorm Layer 4.
Join paragraphs and words split across page boundaries — now at HTML level, inside
build_html.py.Depends on: #30 (enriched
voted_tokensper page; headers already stripped frombody_textand
label=page_header/page_footeris already voted — no pre-processing needed here).The original plain-text stitch.py approach is superseded: operating on the structured token stream
(with
paragraph_startandlabelattributes) is cleaner and avoids the header-wedging problemdescribed in brainstorm Layer 4.
Logic (unchanged):
-+ first token of next page is lowercase → join without hyphen.!?") + first token lowercase → join with space</p><p>boundary)Distinguish
—(em-dash, not a word-break) from-(ASCII hyphen at line end).Unlimited multi-page mode as optional cross-check signal for flagging uncertain boundaries.
See brainstorm Layer 4.