S1 — Decide the no-fabricated-zeros guard for FAO feature columns (C-26)

[Polichinel]

Epic: #51 · risk-register C-26 (Tier 1)

Problem

pipeline-core's data fetch applies df.fillna(0.0) unconditionally to all features (views-pipeline-core modules/dataloaders/dataloaders.py:~1208), consumed by this repo at unfao.py:_read_historical_data (L45-60). A datafactory assembly gap (unharvested month, failed source) becomes lr_ged_sb/ns/os = 0.0 — "zero fatalities" — and _validate (unfao.py:147-172) checks only the 9 metadata columns, never the features, so the fabricated zeros reach FAO with no error signal.

Why this is a decision, not a straight implementation

By the time this repo receives the DataFrame, the NaNs are already 0.0 — a filled zero is indistinguishable from a real zero. The only bounding signal — the zarr's last_valid_month_id attribute — is not reachable in this repo (verified: zero references in views_postprocessing/; it lives in the upstream zarr/dataloader). So the fix is mostly upstream, and this story decides the split.

Decision to make (and record in C-26 + here)

• (a) pipeline-core companion (preferred): stop the unconditional fill — bounded fill using last_valid_month_id (fill only outside the declared valid range, fail on NaN inside it) + log a fill count. Open the companion in views-pipeline-core.
• (b) this-repo guard (if the signal is surfaced): have the loader expose last_valid_month_id/valid range to the manager; _read/_validate drop or fail on (cell, month) inside the valid range that arrived as fill.
• (c) interim this-repo signal: a feature-column anomaly check (per-target zero-fraction log / not-all-zero-per-month) — weak, but better than nothing until (a) lands.

Work

• Confirm the current un_fao fetch path (datafactory vs viewser) and whether fillna(0.0) is still unconditional in the installed pipeline-core.
• Decide (a)/(b)/(c); write the decision into register C-26 and this issue.
• If (a)/(b): open the views-pipeline-core companion issue (bounded fill + expose last_valid_month_id) and link it here.

Acceptance criteria

• A written decision on how fabricated zeros are prevented/detected (in C-26 + this issue).
• The pipeline-core companion exists and is linked (if upstream work is needed).
• The this-repo portion (if any) is scoped into a follow-up implementation note.

Dependencies

Likely spawns a views-pipeline-core companion. No code in this repo until the decision is made.

[Polichinel]

DECISION (2026-06-26): splint now, cure later — "missing is not zero"

The problem in one line

Missing upstream data silently becomes 0 ("no conflict") and reaches FAO with no signal. It is structural, not occasional: the historical range is requested up to _current_month_id() - 1 (a clock boundary, views-models config_partitions.py), but UCDP's real data ends earlier (reporting lag) at last_valid_month_id. So every run ships the most-recent months as fabricated zeros — exactly the months FAO cares about most.

Root cause (why this isn't just "add a check")

The FAO delivery-of-record is produced by model-training machinery whose missing-data rule is the opposite of delivery's: a training loader is supposed to fill blanks with 0 (pipeline-core dataloaders.py:1208 — correct for fitting models); a record-of-truth must never invent a value. The manager even is-a ForecastingModelManager (unfao.py:26) and reads with validate=False (unfao.py:49-54). This is the same root as C-40 (delivery fused to the modeling framework by inheritance), seen from the data-semantics side.

Decision — two tracks

1. Splint — now, this repo (stops the Tier-1 bleed):
Make delivery enforce one invariant — do not ship "observed" months beyond the real data boundary (last_valid_month_id), and treat missing as missing (fail, don't zero). Concretely: clip/validate the historical delivery to ≤ last_valid_month_id; if asked to deliver beyond it, fail loud. This covers Case A (requested range beyond observed) — the common, every-run case.

• Dependency — small pipeline-core companion: we need last_valid_month_id surfaced to us. datafactory's loader already returns it (dataset.py:217) and pipeline-core receives and discards it (dataloaders.py). So the companion is "stop discarding last_valid_month_id; pass it through" (+ ideally log a fill count). To be opened when the splint starts.

2. Cure — later, deliberate (NOT delivery week) = C-40:
Give delivery its own data read that treats missing as missing, instead of inheriting the model-training loader (compose a thin observed-record reader + forecast reader + FAO writer; stop being a ForecastingModelManager). This dissolves the root for FAO and the 2–3 UN agencies coming next, so we stop re-patching per partner. The splint is removed by the cure.

Explicitly NOT doing

• Not the big datafactory "use NaN everywhere" refactor — correct for delivery but breaks every modeling consumer relying on dense 0-padding. Wrong blast radius.
• Not a per-cell coverage mask yet. That addresses Case B (a single unharvested cell inside the valid range) — undetectable downstream today (no mask exists), but rare by how datafactory assembles. Defer behind a cheap tripwire (log per-month zero-fraction) to gather evidence before building mask infrastructure.

Status

• ✅ Strategy decided (splint + cure) — needs-decision cleared.
• ◻ Splint implementation + the pipeline-core companion: start when the FAO input-integrity sprint (Epic: FAO delivery input-integrity — complete-coverage contract + no silent bad data #51) kicks off; this issue tracks it.
• ◻ Cure tracked under C-40 (register) — scheduled deliberately, with this fabricated-zeros problem as its explicit goal.

Grounded in a cross-repo survey + two expert-code-reviews (2026-06-25/26). Same root as C-40; both are the symptom of delivery built on modeling machinery.