Design contract: harden non-ANBIMA source maps before feature/timing fixes

[gabrool]

Context

Implement the source-map hardening pass from the pipeline audit. ANBIMA is intentionally excluded: do not implement, revive, or re-prioritize ANBIMA in this issue.

This issue is only about source maps, discovery/download metadata, docs, and tests. Do not start feature redesign, preprocessing changes, timing-policy changes, model work, NLP, PDF parsing, OCR, or portfolio/backtest work.

Read first: AGENTS.md, docs/PROJECT_PLAN.md, docs/ENGINEERING_GUIDELINES.md, docs/B3_INGESTION_SETUP.md, docs/TIMING_AND_AVAILABILITY_POLICY.md, existing docs/*_SOURCE_MAP.md, and configs/datasets/*.yaml.

Design principles

1. Do not fake stable endpoints. If an official page is stable but a backing JSON/XHR endpoint is not formally documented, treat that backing endpoint as a candidate until fixture-verified.
2. No misleading blank P0 maps. A non-ANBIMA P0 dataset must not have source_urls: [] unless it is explicitly upstream-derived, demoted, manual/static, or source-map-only with a documented reason.
3. Prefer official page/API discovery over brittle hardcoded file names when official metadata pages exist.
4. Preserve resource metadata for later PIT work: resource_url, resource_name, HTTP/resource last-modified/update timestamps, and first-seen timestamps when available.
5. Keep layers clean: downloaders download, parsers parse, normalizers normalize. This issue should mostly touch configs, source-specific discovery/download helpers, docs, and tests.
6. No live-network tests. Use fixtures and mocked HTTP.

Use consistent statuses such as live_download, live_download_after_fixture_verified, raw_page_snapshot, implemented_via_<dataset_id>, source_map_only_pending_direct_url, source_map_only_feature_gated_large_files, and not_implemented_pending_endpoint.

Required changes

B3

Fix unresolved P0/P1 source maps in configs/datasets/b3.yaml.

1. b3_indexes_current_portfolio / b3_indexes_composition

   • Add official B3 index and portfolio page URLs.
   • First target should be IBOV.
   • Known public-web candidate for fixture verification: sistemaswebb3-listados.b3.com.br/indexProxy/indexCall/GetPortfolioDay/{index_id} with language=pt-br.
   • Treat the candidate as live_download_after_fixture_verified, not as a blindly trusted official API.
   • Store raw HTML snapshots separately from candidate JSON.
   • Do not backfill current portfolio historically unless official portfolio/effective dates prove it.

2. b3_indexes_historical_data

   • Add the official B3 historic-statistics page as a raw page snapshot source.
   • Add any backing resource only after fixture verification.
   • If no stable direct machine-readable resource is verified, status must be raw_page_snapshot or source_map_only_pending_direct_url, not live_download.

3. b3_traded_securities

   • Do not leave this as an independent blank source map.
   • Source it through b3_daily_bulletin_chapters using the existing sections for BVBG.028.02 Instruments File, BVBG.029.02 Instruments File, and Tradable Security List.
   • Use source_map_status: implemented_via_b3_daily_bulletin_chapters and an explicit upstream-dataset field if natural.

4. b3_reference_rates

   • Either source it explicitly via b3_daily_bulletin_chapters sections such as primitive risk factors, economic indicators, swap market rates, and scenario/risk files, or demote it with a documented reason because DI curves are currently built from futures settlements.
   • Do not leave it as P0 with no URL/upstream/demotion explanation.

5. b3_daily_bulletin_chapters

   • Add the official B3 daily bulletin/source page if stable.
   • If a direct report download mechanism is discovered, implement only after fixture verification.
   • Do not OCR PDFs.

6. Keep product/spec pages as raw snapshots only. Update stale URLs only if fixture-tested.

BCB

Add metadata-only/raw-page source maps for Copom calendar, decisions, and documents if stable official BCB pages/endpoints are identifiable. No statement/minute NLP, tone scores, or PDF text parsing. Existing SGS/PTAX/Focus source maps should remain unless minor docs cleanup is needed.

Update docs/BCB_SOURCE_MAP.md.

IBGE

Implement ibge_errata_revisions_metadata as metadata-only using the official IBGE news API. Use product IDs already configured in configs/series/ibge_sidra.yaml where available. Search terms should include errata, revisão, and revisao. Do not parse article bodies.

Update configs/datasets/ibge.yaml, src/bralpha/ingestion/ibge/news.py if needed, and docs/IBGE_SOURCE_MAP.md.

Tesouro

Move tesouro_rtn_series out of ambiguous pending status using Tesouro Transparente CKAN package_show discovery for resultado-do-tesouro-nacional. Select the resource matching API Séries Temporais or the configured equivalent. Store raw CKAN package JSON and resource metadata. Do not parse XLSX in this issue.

Update configs/datasets/tesouro.yaml, src/bralpha/ingestion/tesouro/ckan.py, and docs/TESOURO_SOURCE_MAP.md.

ONS

For ONS datasets currently using hardcoded annual S3 CSV templates, add CKAN package/resource discovery before direct-template fallback. Preserve resource URL/name and modified/update metadata in manifests. Keep direct S3 templates as fallback only.

Representative datasets requiring mocked fixture coverage: EAR, ENA, load, CMO, and energy balance.

Update configs/datasets/ons.yaml, src/bralpha/ingestion/ons/resources.py, and docs/ONS_SOURCE_MAP.md.

ANP

For ANP fuel prices, add official page link discovery as the primary source and keep fixed filename families as fallbacks. Accept .csv and .zip; reject overlap/latest-only files such as últimas quatro semanas / ultimas quatro semanas unless explicitly requested for validation.

For ANP fuel sales and oil/gas production, route resource selection through deterministic official-page/link-text discovery rather than hidden assumptions.

Update configs/datasets/anp.yaml, src/bralpha/ingestion/anp/resources.py, and docs/ANP_SOURCE_MAP.md.

CVM

Keep existing fund daily report and registry URLs, but clarify official dataset-page source URLs and docs. Keep CDA as source-map-only / feature-gated large files unless a later issue requests implementation. If adding CDA source maps, include the official dataset page but do not implement parsing. Clearly label current registry as current reference snapshot, not historical model data.

Update configs/datasets/cvm.yaml if needed and docs/CVM_SOURCE_MAP.md.

Novo CAGED / PDET

Keep the movement FTP template. Strengthen the release-calendar source map by documenting that the official calendar page covers the currently published calendar unless historical calendars are discovered and fixture-backed. Do not claim complete historical calendar coverage.

Update configs/datasets/novo_caged.yaml and docs/NOVO_CAGED_SOURCE_MAP.md.

Receita

Prefer dados.gov metadata API discovery for resultado-da-arrecadacao; fall back to the Receita category page only when metadata discovery fails. Preserve resource update/modified metadata. Keep PDFs rejected for ingestion.

Update configs/datasets/receita.yaml, src/bralpha/ingestion/receita/resources.py, and docs/RECEITA_SOURCE_MAP.md.

FRED

Add optional vintage-ready params to the FRED observations source map: realtime_start and realtime_end. These must be optional; existing non-vintage requests must keep working. Keep FRED_API_KEY behavior.

Update configs/datasets/fred.yaml, request rendering if needed, and docs/FRED_SOURCE_MAP.md.

Tests required

Add mocked/fixture tests only:

• All changed configs parse.
• Non-ANBIMA P0 source-map audit: no blank source_urls unless upstream/demoted/manual/source-map-only status is documented.
• B3 index portfolio/history page request rendering.
• B3 candidate portfolio JSON fixture if candidate endpoint is included.
• IBGE news/errata query rendering and pagination.
• Tesouro CKAN package/resource discovery.
• ONS CKAN resource discovery for representative datasets.
• ANP page link discovery and rejection filters.
• Receita dados.gov metadata discovery and fallback page discovery.
• FRED request rendering with and without vintage params.
• Manifest metadata carries discovered resource URL/name and modified/update timestamps when available.
• Failure paths for missing resource, changed schema, no matching link, and non-2xx responses.

Documentation required

Update source-map docs for every touched source. Each doc should state: official page/API, implemented status, raw format, expected frequency, primary keys if a downstream table exists, known limitations, and whether the map is live, raw-page-only, candidate fixture-verified, upstream-derived, or intentionally deferred.

Acceptance criteria

This issue is complete only when:

• ANBIMA remains intentionally excluded.
• All non-ANBIMA P0 source maps are honest and tested.
• B3 index composition/history/traded securities/reference-rates have explicit source-map paths or documented upstream/demotion decisions.
• BCB Copom metadata maps exist without NLP.
• IBGE errata/revision metadata uses official metadata APIs.
• Tesouro RTN uses CKAN discovery.
• ONS uses package/resource discovery before S3 fallback.
• ANP uses official page discovery before fixed filename fallback.
• CVM docs clarify daily reports, current registry, and deferred CDA.
• Novo CAGED release-calendar limitations are explicit.
• Receita discovery prefers dados.gov metadata and documents fallback.
• FRED source map supports optional vintage params.
• Tests pass without live network calls and without committing downloaded data.