Skip to content

Cross-page paragraph/word stitching (deterministic, Path B) #20

Description

@Arkoniak

Join paragraphs and words split across page boundaries — now at HTML level, inside build_html.py.

Depends on: #30 (enriched voted_tokens per page; headers already stripped from body_text
and label = page_header/page_footer is already voted — no pre-processing needed here).

The original plain-text stitch.py approach is superseded: operating on the structured token stream
(with paragraph_start and label attributes) is cleaner and avoids the header-wedging problem
described in brainstorm Layer 4.

Logic (unchanged):

  • Word split: last token ends with ASCII - + first token of next page is lowercase → join without hyphen
  • Paragraph continuation: no terminal punctuation (.!?") + first token lowercase → join with space
  • Otherwise: new paragraph break (</p><p> boundary)

Distinguish (em-dash, not a word-break) from - (ASCII hyphen at line end).
Unlimited multi-page mode as optional cross-check signal for flagging uncertain boundaries.

See brainstorm Layer 4.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions