Blocker: high-severity data correctness fixes before first model

[gabrool]

Blocker before first model training.

Please fix the five high-severity audit items as stacked PRs:

1. Correct B3 DI curve math: DI1 settlement must be treated as PU/discounted notional and converted to 252-business-day annual effective rates/discount factors before interpolation.
2. Make FRED point-in-time safe: use realtime/vintage handling for revised series or exclude them from model-usable outputs.
3. Correct Tesouro Direto sales/redemptions availability: sales must use the official two-business-day lag; redemptions must be verified and conservatively lagged if timing is not documented.
4. Add cross-source revision/first-seen policy: revision-prone datasets must expose vintage/first-seen/availability-basis fields and cannot silently use current-snapshot revised values in historical as-of panels.
5. Expand BCB SGS coverage beyond Selic/IPCA to broad rates, activity, credit, fiscal/debt, external, reserves, and monetary/liquidity series with documented availability timing.

A detailed implementation plan will follow in comments.

[gabrool]

Detailed plan, PR 1: fix DI futures / DI curve financial interpretation

Problem

src/bralpha/derived/b3/di_curve.py currently treats DI1 settlement as the curve value and interpolates it linearly by calendar days. For DI1 this is financially wrong: the settlement/adjustment value should be treated as PU on BRL 100,000 notional, and the annual effective DI rate is on a 252-business-day basis.

Affected files:

• src/bralpha/derived/b3/di_curve.py
• src/bralpha/derived/b3/futures_contract_panel.py
• src/bralpha/derived/b3/schemas.py
• src/bralpha/derived/b3/targets.py
• configs/derived/b3.yaml
• B3 docs/tests

Required implementation

1. Add src/bralpha/domain/di_futures.py with tested helpers:

DI_NOTIONAL = 100_000.0
DI_BDAY_BASIS = 252

def discount_factor_from_pu(pu): ...              # pu / 100000

def annual_rate_from_pu(pu, business_days): ...  # (100000 / pu) ** (252 / business_days) - 1

def pu_from_annual_rate(rate, business_days): ...

def log_discount_factor_from_pu(pu): ...

Return null/None for invalid PU or business_days <= 0.

2. Replace Mon-Fri maturity counts with B3-calendar-aware business-day counts wherever DI uses business_days_to_maturity. If the holiday set is unavailable, fallback only with an explicit documented flag/lineage note.

3. Preserve raw PU and derive economic fields. Suggested fields:

• raw_settlement_pu
• discount_factor
• log_discount_factor
• implied_annual_rate
• implied_annual_rate_bp
• curve_value_kind

If curve_value remains for compatibility, set it to implied_annual_rate, not raw PU.

4. Interpolate fixed-tenor DI grid by business-day tenor in log-discount-factor space. Convert interpolated DF back to annual effective rate. Do not extrapolate outside bracketing contracts unless explicitly flagged.

5. Change or alias di_curve.tenor_days to tenor_business_days in configs/derived/b3.yaml.

6. Add DI-specific targets: at least rate_bp_change and discount_factor_log_change. Do not use percent change of PU/rate as the primary DI target.

Tests

• PU/rate round trip.
• Example: PU = 100000 / (1 + 0.13) ** (252/252) maps to 13% for 252 business days.
• Invalid PU and zero/negative business days return null.
• Fixed-tenor grid interpolates log DF using business-day tenors.
• Grid available_date is max of contributing contracts.
• DI targets are bp/log-DF based.

[gabrool]

Detailed plan, PR 2: make FRED point-in-time / vintage safe

Problem

src/bralpha/normalization/fred.py preserves realtime_start and realtime_end, but computes available_date from the observation date. That leaks revisions for any revised FRED/ALFRED-style series.

Affected files:

• configs/datasets/fred.yaml
• configs/series/fred.yaml
• src/bralpha/ingestion/fred/*
• src/bralpha/parsing/fred_observations.py
• src/bralpha/normalization/fred.py
• src/bralpha/derived/fred/*
• FRED tests/docs

Required implementation

1. Extend configs/series/fred.yaml with explicit vintage metadata:

series_kind: market_daily | revised_macro | unrevised_reference
vintage_policy: latest_snapshot_allowed | fred_realtime_vintages_required
model_usable_without_vintage: true | false

2. For vintage-required series, store one row per vintage where possible. Suggested key/fields:

• series_id
• ref_date
• vintage_date
• realtime_start
• realtime_end
• value
• raw_value
• available_date
• vintage_id
• vintage_policy
• availability_basis

3. Compute available_date from the vintage/realtime publication date, not the observation date. If only a vintage date is known, apply the existing date-only next-business-day rule.

4. Update silver primary keys to include vintage_date or vintage_id for vintage rows. Current [series_id, ref_date] is insufficient for revised data.

5. Update build_fred_asof_daily() to select the latest vintage whose available_date <= asof_date. Latest-snapshot behavior is allowed only for series explicitly marked latest_snapshot_allowed.

6. Safety default: if a series requires vintages but only current-snapshot data are available, set model_usable = false and exclude it from model-ready daily-long outputs.

Tests

• Fixture with one observation and two vintages: as-of before the second vintage sees first value; as-of after second vintage sees revised value.
• A revised macro series without vintage rows is not model-usable.
• A market-like latest_snapshot_allowed series still uses date-only next-business-day availability.
• Daily-long has no duplicate (ref_date, feature_id, value_name) rows and never includes unavailable vintages.

[gabrool]

Detailed plan, PR 3: correct Tesouro Direto sales/redemptions availability

Problem

src/bralpha/normalization/tesouro_market.py gives Tesouro Direto sales next-business-day availability. Official Tesouro metadata says the sales dataset is available with a two-business-day lag, so current sales features can leak by one business day.

Affected files:

• configs/datasets/tesouro.yaml
• src/bralpha/normalization/tesouro_market.py
• src/bralpha/derived/tesouro/*
• Tesouro tests/docs

Required implementation

1. Add a reusable business-day helper, or reuse a common one if already available:

def add_business_days(ref_date: date, days: int, holidays: set[date] | None = None) -> date:
    ...

Use Brazil/B3 holiday logic where configured. If no holiday set is present, weekday fallback is acceptable only with a documented caveat.

2. For tesouro_direto_sales, set:

available_date = add_business_days(ref_date, 2)
availability_policy = "tesouro_direto_sales_official_2bd"

3. Verify official timing for tesouro_direto_redemptions resources. If documented as 2BD, use that. If not documented, use conservative 2BD and document:

availability_policy = "tesouro_direto_redemptions_conservative_2bd"

Do not leave redemptions at 1BD unless official docs explicitly support that.

4. Keep prices/rates timing unchanged unless official metadata says otherwise.

Tests

• Sales availability is exactly two business days after ref_date.
• Weekend/holiday boundary cases are covered.
• Redemptions follow documented or conservative 2BD policy.
• Derived Tesouro panels preserve corrected available_date.
• Docs note the sales/redemptions timing policy.

[gabrool]

Detailed plan, PR 4: add cross-source revision / first-seen policy

Problem

Several sources can be revised or republished: CVM, ONS, ANP, Novo CAGED, Receita, FRED macro, and some IBGE/BCB series. Many silver/research layers currently infer available_date from observation date plus heuristic lag. That can leak current-snapshot revised values into historical as-of panels.

Required implementation

1. Add docs/REVISION_AND_VINTAGE_POLICY.md defining:

• release_date
• available_datetime_local
• available_datetime_utc
• available_date
• download_timestamp_utc
• first_seen_timestamp_utc
• source_publication_datetime_utc
• vintage_id
• revision_sequence
• availability_basis
• revision_policy

2. Add common helpers, e.g. src/bralpha/timing/vintages.py, for:

• deriving usable dates from exact timestamp, date-only release date, source last-modified, or first-seen timestamp;
• generating stable vintage_id from source, dataset, resource identity, publication/first-seen timestamp, and content hash;
• classifying availability_basis: exact_source_timestamp, source_date_only, source_last_modified, first_seen_download_timestamp, official_release_calendar, official_lag_policy, conservative_heuristic, current_snapshot_no_vintage;
• determining whether a row is model_usable under the revision policy.

3. Add dataset config fields for revision-prone datasets:

revision_policy: unrevised | official_lag_no_revisions | revised_use_vintages | current_snapshot_reference_only
availability_basis: official_release_calendar | official_lag_policy | source_last_modified | conservative_heuristic | first_seen_download_timestamp
model_usable_requires_vintage: true | false

4. Add silver/research fields where applicable:

• release_date
• source_publication_datetime_utc
• first_seen_timestamp_utc
• available_date
• availability_basis
• availability_policy
• revision_policy
• vintage_id
• model_usable

5. Current-snapshot-only revised data must not silently enter model-ready daily-long panels. Either reconstruct vintages/first-seen snapshots or mark model_usable=false.

Family guidance

• CVM: monthly ZIPs can be refreshed; use resource metadata/download first-seen where possible.
• ONS: open-data files can receive consistency updates; preserve source last-modified or first-seen.
• ANP: prefer resource publication/update metadata for weekly/monthly files; otherwise official lag plus first-seen.
• Novo CAGED: official release calendar for first monthly releases; late declarations/exclusions should be separate revision-aware datasets.
• Receita: use dados.gov.br/resource metadata where possible; otherwise conservative lag flagged as heuristic.
• IBGE: release-calendar matched rows are strong; unmatched rows stay non-model-usable.
• BCB SGS: new series require documented timing before model_usable=true.

Tests

• Unit tests for vintage ID generation and availability-basis classification.
• Fixture where a later revision cannot appear in an earlier as-of date.
• Test that current_snapshot_no_vintage rows are excluded from model-ready panels unless explicitly allowed.
• Add a reusable PIT audit helper for daily-long/as-of panels.

[gabrool]

Detailed plan, PR 5: expand BCB SGS coverage

Problem

configs/series/bcb_sgs.yaml currently has only Selic overnight, Selic target, and IPCA as model-usable. The first Brazil RV model needs broader BCB coverage across rates, activity, credit, fiscal/debt, external sector, reserves, and monetary/liquidity.

Affected files:

• configs/series/bcb_sgs.yaml
• configs/datasets/bcb.yaml
• src/bralpha/ingestion/bcb/sgs.py
• src/bralpha/normalization/bcb_sgs.py
• src/bralpha/derived/bcb/sgs.py
• src/bralpha/derived/bcb/daily_long.py
• BCB docs/tests

Required implementation

1. Expand configs/series/bcb_sgs.yaml with verified official SGS IDs. Do not guess IDs. Verify each through official BCB SGS/Open Data documentation or BCB catalog/API before committing.

2. Minimum categories:

• rates / policy: Selic, CDI/DI-like rates if available, market/swap rates where official SGS coverage exists;
• activity: IBC-Br total and SA if available;
• inflation: IPCA plus relevant BCB-tracked price/inflation components where available;
• credit: total credit stock, household/corporate credit, lending rates, spreads, delinquency;
• fiscal/debt: gross debt, net public-sector debt, primary result, nominal result;
• external: reserves, current account, trade balance, FDI/direct investment, portfolio flows if available;
• monetary/liquidity: monetary base and monetary aggregates.

3. Every new series must include:

series_id:
slug:
name:
category:
frequency:
unit:
availability_policy:
availability_lag_days:
model_usable:
notes:
source_reference_url:

4. Timing rule: unknown timing means model_usable=false. Do not use same-day availability unless exact publication time is documented and before the model cutoff.

5. Keep this PR structural: raw -> silver -> as-of/daily-long coverage plus metadata. Bespoke preprocessing should be a follow-up PR.

Tests

• Config tests for unique IDs, required metadata, documented availability for every model-usable series, and exclusion of unknown-timing series.
• Fixture tests covering at least one daily, one monthly, and one stock/level macro series.
• Research as-of tests ensuring expanded series enter daily-long only after availability.

---

Cross-PR validation and final definition of done

Each PR must pass:

python -m pytest
python -m ruff check .

No PR should commit downloaded data, add paid data dependencies, silently change units, or make current-snapshot revised data model-usable without vintage/first-seen handling.

This issue is complete when:

1. DI curve features are PU -> discount-factor/rate -> fixed-tenor curve using B3 business days.
2. FRED revised-data handling is PIT safe, or revised series are excluded from model-ready outputs.
3. Tesouro sales/redemptions no longer use premature one-business-day availability.
4. Revision-heavy sources have a common documented vintage/first-seen policy.
5. BCB SGS coverage is broad enough for a first Brazil RV model and every model-usable series has documented timing.
6. Docs and tests make all guarantees explicit.

[gabrool]

Implemented the issue #48 plan as a five-PR draft stack:

1. [codex] Fix DI curve PU math #49 [codex] Fix DI curve PU math (base main)
2. [codex] Make FRED vintages point-in-time safe #50 [codex] Make FRED vintages point-in-time safe (base codex/issue-48-di-curve)
3. [codex] Correct Tesouro flow availability #51 [codex] Correct Tesouro flow availability (base codex/issue-48-fred-vintages)
4. [codex] Add revision and vintage policy #52 [codex] Add revision and vintage policy (base codex/issue-48-tesouro-availability)
5. [codex] Expand BCB SGS reference coverage metadata #53 [codex] Expand BCB SGS coverage metadata (base codex/issue-48-revision-policy)

Validation run on the top of the stack (f53dd87):

• python -m pytest -> 517 passed
• python -m ruff check . -> all checks passed