Design contract: make priority B3, BCB Copom, and CVM positioning/event sources live

[gabrool]

Context

This issue is the next source-ingestion contract after the source-map hardening work in PR #79 / issue #78. The goal is to make the high-value datasets we agreed are economically important for a Brazil low-frequency relative-value model actually live, or to implement the best practical equivalent when a fully live official source is not yet deterministic.

This is not a broad data grab. Implement only the sources listed here. ANBIMA remains intentionally out of scope.

Read first

Before coding, read:

• AGENTS.md
• docs/PROJECT_PLAN.md
• docs/ENGINEERING_GUIDELINES.md
• docs/TIMING_AND_AVAILABILITY_POLICY.md
• docs/B3_INGESTION_SETUP.md
• docs/B3_P1_P2_SOURCE_MAP.md
• docs/BCB_SOURCE_MAP.md
• docs/CVM_SOURCE_MAP.md
• configs/datasets/b3.yaml
• configs/datasets/bcb.yaml
• configs/datasets/cvm.yaml
• Existing raw/bronze/silver ingestion patterns under src/bralpha/ingestion/*, src/bralpha/normalization/*, and src/bralpha/derived/*

This issue should be implemented after PR #79 is merged or rebased, because it relies on the source-map statuses and documentation introduced there.

Official documentation / source pages to use

Use official public source pages only:

B3

• Daily bulletin search / report source page: https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/historico/boletins-diarios/pesquisa-por-pregao/pesquisa-por-pregao/
• Historical quote data / COTAHIST documentation page: https://www.b3.com.br/en_us/market-data-and-indices/data-services/market-data/historical-data/equities/historical-quote-data/
• Ibovespa official page: https://www.b3.com.br/en_us/market-data-and-indices/indices/broad-indices/ibovespa.htm
• Ibovespa historic statistics page: https://www.b3.com.br/en_us/market-data-and-indices/indices/broad-indices/indice-ibovespa-ibovespa-historic-statistics.htm

BCB

• Copom calendar page: https://www.bcb.gov.br/controleinflacao/calendariocopom
• Copom interest-rate history page: https://www.bcb.gov.br/controleinflacao/historicotaxasjuros
• BCB SGS API already configured in repo: https://api.bcb.gov.br/dados/serie/bcdata.sgs.{series_id}/dados
• BCB Focus/PTAX OData already configured in repo.

CVM

• Daily fund reports: https://dados.cvm.gov.br/dataset/fi-doc-inf_diario
• Fund registry: https://dados.cvm.gov.br/dataset/fi-cad
• CDA fund portfolio holdings: https://dados.cvm.gov.br/dataset/fi-doc-cda
• IPE metadata: https://dados.cvm.gov.br/dataset/cia_aberta-doc-ipe
• FII monthly structured reports: https://dados.cvm.gov.br/dataset/fii-doc-inf_mensal
• FIDC monthly structured reports: https://dados.cvm.gov.br/dataset/fidc-doc-inf_mensal

Non-goals

Do not implement:

• ANBIMA.
• NLP / tone scoring / LLM text extraction.
• PDF OCR.
• Browser automation.
• Unofficial scraping when an official resource cannot be fixture-verified.
• Model training, labels, backtesting, or portfolio construction.
• Large raw-data commits.
• Any live-network tests.

Design principles

1. Make sources live only when deterministic. Official page snapshots are fine, but a live downloader for a data table/file requires fixture-verified URL/resource discovery and schema tests.
2. Fail closed. Missing links, changed schemas, non-matching resource names, or unknown file structures should fail clearly and/or write failed manifest records. Do not silently use an unrelated resource.
3. Keep raw/bronze/silver layering. Downloaders write immutable raw files and manifests; parsers create bronze; normalizers create source-specific silver; derived/model features come later unless explicitly requested below as simple aggregate source panels.
4. Point-in-time metadata is required. Preserve source URL, source page, resource name/id, first-seen/download timestamp, HTTP last-modified/resource modified metadata when available, and official reference dates.
5. Prefer low-dimensional, model-usable aggregates from high-dimensional sources. CDA, IPE, FII, and FIDC can be huge. Implement raw/bronze auditability, but also source-specific aggregated silver panels when appropriate so the first model has usable inputs without exploding the feature universe.

---

A. B3 daily bulletin live downloader and selected report parsers

Motivation

B3 daily bulletin files are the cleanest route to important sources that are not replaceable by current live data:

• b3_reference_rates: not just another DI curve; it can include B3 primitive risk factors, swap market rates, government-security reference prices, FX traded rates, and risk/scenario files.
• b3_traded_securities: COTAHIST gives historical quotes, not a true point-in-time security/tradability master.
• b3_indexes_historical_data: IND/WIN futures are tradable equity beta, but spot index data are required for futures basis and clean equity-state features.
• daily price/index updates: annual COTAHIST is fine for historical backfill, but daily bulletin PriceReport / IndexReport can support current incremental updates.

Required implementation

Create a source-specific B3 daily bulletin discovery/downloader. Do not build a generic web-scraping framework.

Suggested modules:

• src/bralpha/ingestion/b3/daily_bulletin.py
• src/bralpha/parsing/b3_daily_bulletin.py
• src/bralpha/normalization/b3_daily_bulletin.py or source-specific normalizers such as normalization/b3_index_report.py, normalization/b3_traded_securities.py, etc.

The downloader should:

1. Fetch/store the official daily bulletin search page for a requested ref_date.
2. Discover report resources by exact configured report-section names.
3. Store raw files separately by dataset_id, report_section, and ref_date.
4. Preserve in the manifest:
   • report_section
   • ref_date
   • official source page URL
   • discovered resource URL
   • discovered resource file name
   • content type
   • HTTP last-modified if available
   • download timestamp
   • discovery mode
5. Fail closed if the requested report section is absent or ambiguous.
6. Avoid OCR and avoid PDF parsing. If a report is only PDF, record source-map metadata but do not parse it in this issue.

B3 report sections to target first

Implement live download/discovery and fixture-backed parsing for as many of these as deterministic. Prioritize in this order:

1. BVBG.087.01 IndexReport

   • Purpose: make spot B3 index data live for Ibovespa and eventually other indices.
   • Output target: b3_index_daily or an equivalent index_daily silver table with ref_date, index_id, open/high/low/close or official index value fields if present, volume/liquidity fields if present, available_date, and source provenance.
   • Do not treat IND/WIN futures as a replacement for spot index data after this is live.

2. BVBG.086.01 PriceReport and/or BVBG.186.01 Simplified Price Report - Equities

   • Purpose: daily current update path for listed-market prices, complementing annual COTAHIST.
   • Output target: either append-compatible with b3_cotahist_yearly normalized market-daily schema or a separate b3_price_report_daily table with explicit source family.
   • Must not silently mix with annual COTAHIST if fields differ.

3. BVBG.028.02 Instruments File, BVBG.029.02 Instruments File, and Tradable Security List

   • Purpose: make b3_traded_securities live.
   • Output target: security/reference table with at least ref_date, symbol, isin if present, market/type/group identifiers, tradability/status fields if present, and source provenance.
   • Use this to improve universe hygiene; do not backfill security status prior to the file’s actual availability.

4. Primitive Risk Factors, Derivatives Market - Swap Market Rates, Derivatives Market - Economic Indicators, Securities Market - Government Securities Reference Prices, FX Market - Traded Rates, Opening Parameters and Contracted Transactions, and Scenario and risk-matrix files

   • Purpose: make b3_reference_rates and B3 official risk/reference controls live.
   • Output target: one or more source-specific long tables, e.g. b3_reference_rate_daily, b3_risk_factor_daily, b3_fx_reference_daily, with ref_date, source_report_section, instrument_or_curve_id, tenor_or_bucket, value_name, value, unit, available_date, and provenance.
   • Keep these separate from settlement-derived DI curves. They are controls/reference data, not replacements.

B3 investor / foreign-flow data

Foreign/investor-flow data is required for the model and is not replaceable by COTAHIST total volume, CVM fund flows, or BCB balance-of-payments flows.

Create a dedicated B3 investor-flow discovery implementation path:

• Search/verify official B3 public pages or daily-bulletin reports for:
  • foreign investor cash-equities flow / balance if available;
  • investor participation by investor class;
  • market-segment buy/sell/net values if available;
  • derivatives participation/positioning if a stable public file exists.
• Do not invent endpoints.
• If only an official page is stable, store raw-page snapshots and leave parser disabled until a direct resource is fixture-verified.
• If a file/resource is deterministic, implement live download and a normalized long panel.

Suggested dataset IDs if deterministic official sources are found:

• b3_equities_investor_participation
• b3_foreign_investor_movement
• b3_derivatives_investor_participation only if official structured data exists.

Minimum useful output schema:

• ref_date
• market_segment
• investor_type or residency_type
• buy_value if present
• sell_value if present
• net_flow if present or derivable from buy/sell
• trading_value_share if present or derivable with same-source denominators
• unit
• available_date
• source_report_section
• source_version / provenance

If no deterministic official source is found in this issue, document that explicitly and create source-map-only dataset entries with official page URLs and a recommended manual official-source workflow. Do not mark it live without fixture-backed discovery.

B3 tests required

• Mocked official page fixture with multiple report links and exact report-section matching.
• Ambiguous/missing report link failures.
• Raw file manifest metadata includes discovered URL/resource/report section.
• Parser fixture for each report section promoted to live.
• Quality tests for primary keys and required columns.
• A test proving daily bulletin-derived b3_reference_rates does not overwrite settlement-derived DI curves.
• A test proving current portfolio/security-master files are not backfilled before their ref_date/availability.

---

B. BCB Copom structured calendar and decisions

Motivation

SGS Selic target gives the policy-rate time series, but it does not provide clean event metadata. The model needs event-time awareness around Copom meetings and decisions.

Required implementation

Implement structured Copom event tables from official BCB pages or, if the official pages are not machine-readable enough, from a project-maintained official-source manual YAML file.

Suggested datasets:

• bcb_copom_calendar
• bcb_copom_decisions

Acceptable implementation modes:

1. Preferred: fixture-verified parser for official BCB calendar and rate-history pages or an official underlying endpoint discovered from those pages.
2. Fallback: manual YAML under configs/manual/bcb_copom_calendar.yaml and configs/manual/bcb_copom_decisions.yaml, with source URL, source access date, and explicit provenance for each year/decision.

Do not use NLP and do not parse PDFs.

Minimum output schemas

bcb_copom_calendar

• meeting_id
• meeting_number if available
• meeting_year
• meeting_start_date if available
• decision_date
• event_type
• source_url
• available_date
• source_version

bcb_copom_decisions

• decision_date
• target_selic_percent_pa
• previous_target_selic_percent_pa
• target_change_bp
• decision_direction = hike / cut / hold
• meeting_id if joinable
• source_url
• available_date
• source_version

Cross-check decisions against configured SGS selic_target where possible, but do not use the SGS series as the only event source.

Derived helper panel allowed in this issue

It is acceptable to add a simple source-specific event feature panel, not a full modeling feature set:

• days_to_next_copom
• days_since_last_copom
• is_copom_decision_day
• is_copom_window_t_minus_1, t_minus_2, t_minus_5, t_plus_1

Keep this in source/derived code, not model preprocessing.

BCB tests required

• Parser/manual YAML validation fixtures.
• No duplicate meeting IDs or decision dates.
• Decision target changes reconcile with consecutive decision rows.
• Event windows do not leak future decision outcomes; pre-decision days may know the scheduled decision date, not the decision value.

---

C. CVM CDA fund portfolio holdings

Motivation

CDA is the best free source for fund holdings/crowding. It is critical for Brazil RV because it can identify crowding into public bonds, private credit, fund quotas, swaps, foreign assets, and other asset blocks, especially when combined with CVM daily fund flows.

Official page: https://dados.cvm.gov.br/dataset/fi-doc-cda

Required implementation

Make cvm_fund_portfolio_cda live, but implement it carefully.

Suggested modules:

• src/bralpha/ingestion/cvm/cda.py
• src/bralpha/parsing/cvm_cda.py
• src/bralpha/normalization/cvm_cda.py
• src/bralpha/derived/cvm/cda_aggregates.py

Downloader requirements:

1. Discover CDA resources from the official CVM dataset page / stable CVM data directory.
2. Support current rolling monthly files and historical files if official deterministic paths exist.
3. Preserve file period, resource name, resource URL, confidentiality indicator if inferable, download timestamp, and HTTP/resource metadata.
4. Do not parse or expose confidential individual holdings before the official confidentiality restriction expires.
5. Fail closed on unknown file naming/layout.

Parser requirements:

CVM CDA files have multiple asset blocks. Parse blocks separately and normalize to a common long shape where possible. At minimum, detect and preserve:

• public bonds / SELIC block;
• fund quota block;
• swaps block;
• coded assets block;
• deposits / financial-institution titles block;
• agro/private-credit titles block;
• foreign-investment block;
• uncoded assets block;
• confidential/consolidated block if present.

Do not force all fields into one lossy schema. It is acceptable to emit:

• raw block-specific bronze tables;
• a common normalized long table with core fields and extra_json for block-specific fields.

Minimum normalized core fields:

• ref_date
• fund_id
• fund_class_id if available or joinable later
• asset_block
• asset_id / isin / ticker / issuer_id where available
• asset_description
• quantity
• market_value
• cost_value if available
• maturity_date if available
• indexer if available
• asset_type
• confidentiality_status
• available_date
• source_file
• source_version

Aggregate source-specific panel required:

Create practical model-ready source-specific aggregates without making a huge feature explosion:

• by ref_date, fund_class, asset_block;
• by public bond indexer/maturity bucket when fields permit;
• by private-credit type when fields permit;
• by foreign-asset bucket when fields permit;
• shares of total portfolio value;
• rolling changes in aggregate exposure;
• count of funds reporting by bucket;
• concentration/crowding measures such as top-N holder share if tractable.

Join with cvm_fund_daily_reports and registry/class data when available, but do not require perfect joins for raw ingestion.

CVM CDA tests required

• Resource discovery fixture for monthly/current and historical file names.
• Parser fixtures for at least two asset blocks plus confidential/consolidated behavior.
• Primary key / duplicate tests.
• Confidentiality tests proving restricted details are not marked available before their official availability.
• Aggregate tests for fund-class/block exposure shares.
• Memory/streaming test or chunked parser design for large files.

---

D. CVM IPE metadata, FII reports, and FIDC reports

D1. CVM IPE metadata

Official page: https://dados.cvm.gov.br/dataset/cia_aberta-doc-ipe

Implement metadata-only ingestion. Do not parse document bodies.

Minimum output fields:

• ref_date / delivery date
• company_id
• company_name
• document_id
• document_category
• document_type
• subject / title if structured metadata provides it
• version
• source_url
• available_date
• source_version

Useful aggregate/event panel:

• event counts by category and company/sector;
• flags for relevant facts, market announcements, public offerings, recovery/liquidation, debenture deeds/amendments, corporate events, insider/securities-held reports if categories are present;
• rolling 5/21/63 business-day event counts.

Keep all text-body/NLP work out of scope.

D2. FII monthly reports

Official page: https://dados.cvm.gov.br/dataset/fii-doc-inf_mensal

Implement structured monthly report ingestion if deterministic CSV/ZIP resources are available.

Use cases:

• real-estate / yield-product stress;
• FII NAV/equity value and income/distribution proxies if structured fields exist;
• fund-type exposure and market stress after joining with B3 listed FII tickers.

Minimum requirements:

• raw/bronze download and manifest;
• normalized monthly source-specific table;
• no PDF parsing;
• parser fixtures for official CSV layouts.

D3. FIDC monthly reports

Official page: https://dados.cvm.gov.br/dataset/fidc-doc-inf_mensal

Implement structured monthly report ingestion if deterministic CSV/ZIP resources are available.

Use cases:

• private-credit stress;
• FIDC NAV/equity/subordination/tranche metrics if present;
• delinquency/default/overdue metrics if present;
• growth/crowding in private credit.

Minimum requirements:

• raw/bronze download and manifest;
• normalized monthly source-specific table;
• no PDF parsing;
• parser fixtures for current and at least one historical layout if layout changed over time.

CVM IPE/FII/FIDC tests required

• Official resource discovery fixtures.
• Parser fixtures for each live resource.
• Metadata-only guarantee for IPE: no document-body parsing.
• Quality checks and duplicate primary-key tests.
• Source docs updated with known file-history/layout limitations.

---

Config/docs updates required

Update the relevant YAML configs:

• configs/datasets/b3.yaml
• configs/datasets/bcb.yaml
• configs/datasets/cvm.yaml
• configs/sources/b3.yaml if present or create only if it naturally fits existing source config patterns.
• configs/sources/bcb.yaml if present or create only if natural.
• configs/sources/cvm.yaml

Update docs:

• docs/B3_P1_P2_SOURCE_MAP.md
• docs/BCB_SOURCE_MAP.md
• docs/CVM_SOURCE_MAP.md
• docs/TIMING_AND_AVAILABILITY_POLICY.md only if you add source-specific availability rules.

Each live dataset must document:

• official source page/API URL;
• raw format;
• discovery mode;
• expected frequency;
• primary keys;
• known file-layout limitations;
• availability policy;
• whether it is model-usable immediately or reference-only.

CLI / pipeline integration

Add or extend pipeline entry points naturally. Do not overload b3_ingest.py with all unrelated sources unless existing style suggests it.

Expected additions may include:

• src/bralpha/pipelines/b3_daily_bulletin_ingest.py
• src/bralpha/pipelines/bcb_copom_ingest.py
• src/bralpha/pipelines/cvm_cda_ingest.py
• src/bralpha/pipelines/cvm_reports_ingest.py

Each pipeline should support:

• --dataset
• --start / --end when relevant
• --report-section or equivalent where relevant
• --repo-root
• no live network calls in tests.

Acceptance criteria

This issue is complete when:

1. At least one B3 daily-bulletin report family is live with fixture-backed discovery and parsing. Preferred first target is BVBG.087.01 IndexReport; if unavailable, implement the most deterministic of the prioritized sections and document why.
2. B3 investor/foreign-flow official-source investigation is complete and either live with fixtures or explicitly source-map-only with official pages and a manual-source workflow. Do not leave it ambiguous.
3. b3_traded_securities is live if the Instruments/Tradable Security List reports can be fixture-verified; otherwise it remains explicitly source-map-only with documented blocker.
4. b3_reference_rates has at least one live fixture-backed B3 reference/risk/rate report if deterministic; otherwise the blocker is documented and settlement-derived DI remains the canonical curve.
5. BCB Copom calendar and decisions produce structured source-specific tables, via official parser or manual official-source YAML.
6. CVM CDA is live or, if the file layout/resource discovery proves too large for one PR, the PR must at least implement deterministic discovery and one fixture-backed parser for a representative CDA block, with the remaining block parser backlog documented.
7. CVM IPE metadata is live metadata-only if deterministic official resources are available.
8. FII and FIDC monthly report ingestion are either live with structured parser fixtures or explicitly source-map-only with exact official blocker documented.
9. All new live datasets have raw manifests, bronze outputs where appropriate, silver/source-specific outputs where appropriate, and quality tests.
10. No ANBIMA work is included.
11. Full test suite passes with mocked fixtures and no live-network tests.

Suggested implementation order

1. BCB Copom manual/parser tables. This is small and immediately model-useful.
2. B3 daily-bulletin discovery framework plus IndexReport or Instruments File as first live report.
3. B3 investor/foreign-flow source investigation and implementation if deterministic.
4. CVM CDA discovery and parser foundation.
5. CVM IPE metadata.
6. FII/FIDC structured reports.

Follow-up issues expected

This issue may create follow-up issues for:

• Additional B3 daily-bulletin report parsers after the first deterministic parser lands.
• CDA block parsers not finished in the first CDA PR.
• Text/NLP over BCB/CVM documents, explicitly out of scope here.
• Model feature engineering from the newly live source-specific panels.