Skip to content

Design contract: close remaining priority source-map gaps after PR #81 #82

Open
#83
Open
Design contract: close remaining priority source-map gaps after PR #81#82
#83

Description

@gabrool
gabrool
opened on Jul 3, 2026

Context

This issue is the follow-up design contract to close the remaining source-map / live-ingestion gaps after PR #81. Codex should treat this issue as an implementation contract, not as a planning prompt.

The decisions below are intentional:

  1. B3 daily-bulletin report files are not automatically live yet. The official B3 page is known, but the real date-specific report-resource request is not fixture-proven. Do not promote B3 daily-bulletin download to live_download by scraping static labels or guessing endpoints.
  2. The correct B3 fallback is controlled official-file import. Until the date-specific workflow is proven, make B3 priority reports usable through a manual official-file ingestion path with strict provenance and parser fixtures.
  3. BCB Copom should be manual-official data now. The official pages are JS-rendered for plain HTTP clients; populate and validate a project-maintained manual YAML sourced from BCB official pages.
  4. CVM FII and FIDC can be made live from official direct directories. These should move from source-map-only to live ingestion with directory/resource discovery, parsers, and source-specific silver/gold panels.

This issue intentionally leaves ANBIMA out of scope.

Read first

  • AGENTS.md
  • docs/PROJECT_PLAN.md
  • docs/ENGINEERING_GUIDELINES.md
  • docs/TIMING_AND_AVAILABILITY_POLICY.md
  • docs/B3_INGESTION_SETUP.md
  • docs/B3_P1_P2_SOURCE_MAP.md
  • docs/BCB_SOURCE_MAP.md
  • docs/CVM_SOURCE_MAP.md
  • configs/datasets/b3.yaml
  • configs/datasets/bcb.yaml
  • configs/datasets/cvm.yaml
  • PR Implement priority source ingestion for issue 80 #81 changes around CVM CDA/IPE, B3 daily-bulletin status, and Copom manual schema.

Official source references

B3

  • Daily bulletin/report search page: https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/historico/boletins-diarios/pesquisa-por-pregao/pesquisa-por-pregao/
  • COTAHIST official page: https://www.b3.com.br/en_us/market-data-and-indices/data-services/market-data/historical-data/equities/historical-quote-data/
  • Ibovespa official page: https://www.b3.com.br/en_us/market-data-and-indices/indices/broad-indices/ibovespa.htm

BCB

  • Copom calendar page: https://www.bcb.gov.br/controleinflacao/calendariocopom
  • Copom target-rate history page: https://www.bcb.gov.br/controleinflacao/historicotaxasjuros

CVM

  • FII monthly structured reports: https://dados.cvm.gov.br/dataset/fii-doc-inf_mensal
  • FII direct directory: https://dados.cvm.gov.br/dados/FII/DOC/INF_MENSAL/DADOS/
  • FIDC monthly reports: https://dados.cvm.gov.br/dataset/fidc-doc-inf_mensal
  • FIDC current direct directory: https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/
  • FIDC historical direct directory: https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/HIST/

Non-goals

Do not implement:

  • ANBIMA.
  • Browser automation in production code or tests.
  • OCR.
  • NLP / tone scoring / LLM parsing.
  • Unofficial or guessed endpoints.
  • Live-network tests.
  • Model training/backtesting/portfolio logic.
  • Raw market-data commits.

1. B3 daily-bulletin and B3 flow reports: implement controlled official-file import

Decision

The B3 daily-bulletin page and report families are official, but automatic live download remains blocked until the actual date-specific report-resource request is captured and fixture-proven. This issue should not attempt to make B3 daily-bulletin reports live_download unless a deterministic official HTTP request is already known and can be represented by committed mocked fixtures.

Instead, implement a manual official-file import path. This is the best practical alternative and should be treated as first-class ingestion, not an ad hoc notebook workflow.

Required B3 config statuses

Add a new explicit status and use it consistently:

  • manual_official_file_ingest: official file must be manually downloaded from the official page, then imported through a controlled pipeline with manifest provenance.

Use this for B3 priority datasets that can be parsed from manually provided official files:

  • b3_index_daily
  • b3_traded_securities
  • b3_reference_rates
  • b3_equities_investor_participation
  • b3_foreign_investor_movement

Keep b3_daily_bulletin_chapters itself as source_map_only_official_page until automatic date-specific download is proven.

Keep any dataset without parser fixtures as source_map_only_blocked_daily_bulletin_workflow rather than marking it manual-ingest-ready.

Required B3 manual file importer

Add a pipeline such as:

python -m bralpha.pipelines.b3_official_report_import \
  --dataset b3_index_daily \
  --report-section "BVBG.087.01 IndexReport" \
  --ref-date 2026-06-30 \
  --raw-file /path/to/manually_downloaded_official_file.csv \
  --source-page-url "https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/historico/boletins-diarios/pesquisa-por-pregao/pesquisa-por-pregao/" \
  --manual-download-timestamp-utc 2026-07-01T12:00:00Z \
  --source-file-url <optional_exact_download_url_if_visible> \
  --operator-note "Downloaded manually from official B3 daily bulletin page."

The importer must:

  1. Verify the local file exists and compute SHA-256 before import.
  2. Copy bytes into immutable raw storage under the correct source/dataset/date path.
  3. Write a normal manifest record with:
    • dataset_id
    • source = b3
    • source_url = source_page_url
    • resource_url = source_file_url if supplied, otherwise null or source page URL
    • resource_name = report_section
    • ref_date
    • report_section
    • discovery_mode = manual_official_file
    • manual_download_timestamp_utc
    • operator_note
    • file size/hash/raw path
  4. Never mutate or overwrite the original file.
  5. Fail closed if the dataset/report-section pair is not explicitly allowed by config.
  6. Fail closed if the parser for that dataset/report-section is not implemented.

Required B3 report catalog

Add a config file, for example configs/manual/b3_official_report_catalog.yaml, mapping dataset IDs to allowed official report sections and parser versions:

reports:
  - dataset_id: b3_index_daily
    report_section: "BVBG.087.01 IndexReport"
    official_page_url: "https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/historico/boletins-diarios/pesquisa-por-pregao/pesquisa-por-pregao/"
    ingest_mode: manual_official_file
    parser: b3_index_report_v1
    model_usable: true
  - dataset_id: b3_traded_securities
    report_section: "Tradable Security List"
    ingest_mode: manual_official_file
    parser: b3_tradable_security_list_v1
    model_usable: true
  - dataset_id: b3_reference_rates
    report_section: "Derivatives Market - Swap Market Rates"
    ingest_mode: manual_official_file
    parser: b3_reference_rate_long_v1
    model_usable: true
  - dataset_id: b3_equities_investor_participation
    report_section: "<exact official B3 report label once manually confirmed>"
    ingest_mode: manual_official_file
    parser: b3_investor_participation_v1
    model_usable: true
  - dataset_id: b3_foreign_investor_movement
    report_section: "<exact official B3 report label once manually confirmed>"
    ingest_mode: manual_official_file
    parser: b3_foreign_investor_movement_v1
    model_usable: true

Important: for investor/foreign-flow rows, if the exact official report labels cannot be confirmed from the official page or official files, keep the entries as source_map_only_manual_official_workflow and do not mark parser/model usability true.

Required B3 parsers and schemas

1.1 b3_index_daily

Use the existing IndexReport fixture foundation from PR #81, but wire it to the manual importer rather than an automatic live downloader.

Minimum silver columns:

  • ref_date
  • available_date
  • availability_policy
  • index_id
  • index_name
  • index_value
  • open, high, low, close if present
  • variation_pct if present
  • financial_volume if present
  • trade_count if present
  • source_report_section
  • source_url
  • resource_url
  • raw_path
  • sha256
  • source_version

1.2 b3_traded_securities

Implement manual official-file import for at least one of:

  • Tradable Security List
  • BVBG.028.02 Instruments File
  • BVBG.029.02 Instruments File

Minimum silver columns:

  • ref_date
  • available_date
  • symbol
  • isin if present
  • instrument_id if present
  • market_segment / market_type if present
  • instrument_group if present
  • tradability_status if present
  • security_name if present
  • source_report_section
  • provenance columns

Do not backfill security status before the imported file’s ref_date / availability.

1.3 b3_reference_rates

Implement manual official-file import for at least one deterministic reference/risk/rate report if a parser fixture exists. Preferred first targets:

  • Derivatives Market - Swap Market Rates
  • Primitive Risk Factors
  • Securities Market - Government Securities Reference Prices
  • FX Market - Traded Rates, Opening Parameters and Contracted Transactions
  • Scenario and risk-matrix files

Minimum long silver columns:

  • ref_date
  • available_date
  • source_report_section
  • curve_or_instrument_id
  • tenor_or_bucket
  • value_name
  • value
  • unit
  • source_version
  • provenance columns

Keep settlement-derived DI curves as canonical tradable DI curves. B3 reference-rate data are controls/reference/risk data, not replacements.

1.4 B3 investor participation and foreign investor movement

These are economically required but automatic live source-map remains unresolved. The contract is:

  1. Add manual official-file importer support for both datasets.
  2. Add parser fixtures only after exact official B3 report labels and representative file headers are present in configs/manual/b3_official_report_catalog.yaml.
  3. If exact official labels/files are not available, leave the datasets as source_map_only_manual_official_workflow and document the blocker.

Minimum target schema when parser is enabled:

  • ref_date
  • market_segment
  • investor_type or residency_type
  • buy_value if present
  • sell_value if present
  • net_flow if present or derivable
  • trading_value_share if present or derivable from same-source denominators
  • unit
  • available_date
  • source_report_section
  • provenance columns

Do not substitute COTAHIST volume, CVM fund flows, or BCB balance-of-payments for this dataset. Those can be separate features, not substitutes.

B3 tests required

  • Manual importer refuses unknown dataset/report-section pairs.
  • Manual importer writes raw bytes and manifest provenance with manual download timestamp and operator note.
  • IndexReport parser/normalizer works through the manual importer path.
  • Traded-securities parser fixture produces non-null symbols and no duplicate primary keys.
  • At least one B3 reference-rate/risk parser fixture produces long-form rows.
  • Investor/foreign-flow parser fixture only exists if exact official report labels/headers are configured; otherwise tests must assert the datasets remain manual-workflow/source-map-only.
  • CLI tests prove no automatic B3 daily-bulletin downloader is exposed as live.

2. BCB Copom: populate and validate official manual source

Decision

BCB Copom calendar/decisions are important enough that an empty schema is not enough. Because the official BCB pages are JS-rendered for plain HTTP clients and no official structured endpoint is fixture-proven in the repo, the correct near-term implementation is a populated official-source manual YAML.

Required config/data

Replace the empty configs/manual/bcb_copom.yaml with populated data.

Coverage requirement:

  • Historical realized Copom decisions and meetings from the project’s model start date. Use 2002-01-01 as minimum unless repo config defines a later first modeling date.
  • Current-year and next published scheduled meetings from the official calendar page.
  • Future scheduled meetings may have calendar rows but must not have decision rows or target-rate values.

Manual YAML must include top-level provenance:

source: bcb_copom_manual_official_fallback
source_version: manual_official_bcb_copom_v2
calendar_source_url: https://www.bcb.gov.br/controleinflacao/calendariocopom
decision_source_url: https://www.bcb.gov.br/controleinflacao/historicotaxasjuros
source_access_date: YYYY-MM-DD
maintainer_note: "Rows copied from official BCB Copom calendar and target-rate history pages."

Calendar row minimum fields:

  • meeting_id
  • meeting_number if available
  • meeting_year
  • start_date if available
  • end_date if available
  • decision_date
  • status: realized or scheduled
  • available_date: for scheduled calendar rows, the date the schedule is known from official source; for realized rows, no later than decision date if schedule was known earlier, but must not create decision leakage.
  • source_url

Decision row minimum fields:

  • meeting_id
  • decision_date
  • target_selic_pct
  • previous_target_selic_pct
  • decision_bp
  • decision_label: hike, cut, hold
  • available_date: decision date unless official release timestamp is modeled more precisely.
  • source_url

Required BCB behavior

  • bcb_copom_calendar and bcb_copom_decisions may become manual_source_map / manual_official_file_ingest, not merely source_map_only_manual_pending, once populated.
  • The pipeline should write silver bcb_copom_calendar and bcb_copom_decisions from the manual file.
  • Add a gold/source-specific event panel with:
    • days_to_next_copom
    • days_since_last_copom
    • is_copom_decision_day
    • is_copom_window_t_minus_1
    • is_copom_window_t_minus_2
    • is_copom_window_t_minus_5
    • is_copom_window_t_plus_1
  • Pre-decision days may know scheduled dates but must not know decision values.

BCB tests required

  • Committed manual file is non-empty and covers model start through current schedule.
  • No duplicate meeting_id.
  • No duplicate decision_date among decision rows.
  • decision_bp == round((target - previous) * 100) for all rows with target/previous.
  • Decision labels match sign of decision_bp.
  • Scheduled future rows have no decision values.
  • Event-window panel does not leak future decision values into pre-decision rows.
  • Cross-check realized decision dates against SGS Selic target changes where feasible, with tolerance/documented exceptions.

3. CVM FII monthly reports: make live

Decision

CVM FII monthly structured reports can be made live. The official page states FII monthly reports are structured periodic documents and the dataset provides the last five years, updated weekly with reapresentations. The direct directory exposes annual ZIP files such as inf_mensal_fii_2016.zip, ..., inf_mensal_fii_2026.zip.

Required config

Promote cvm_fii_reports from source_map_only_official_page to live_download.

Suggested config:

- dataset_id: cvm_fii_reports
  source_map_status: live_download
  raw_format: zip_csv
  period_routing:
    annual_from: 2016
  source_urls:
    - name: fii_inf_mensal_annual
      url_template: "https://dados.cvm.gov.br/dados/FII/DOC/INF_MENSAL/DADOS/inf_mensal_fii_{year}.zip"
      filename_template: "inf_mensal_fii_{year}.zip"

Prefer direct-directory discovery if natural in existing CVM code, but hardcoded annual template is acceptable because the official directory exposes stable annual ZIPs.

Required parser/normalizer

Implement:

  • src/bralpha/parsing/cvm_fii.py
  • src/bralpha/normalization/cvm_fii.py
  • extend src/bralpha/pipelines/cvm_ingest.py

Parser must preserve table/file identity inside the ZIP.

Minimum silver schema:

  • ref_date
  • available_date
  • fund_id
  • report_type / table_name
  • metric_name
  • metric_value where numeric
  • metric_text where nonnumeric
  • source_file
  • inner_filename
  • source_version
  • provenance columns

If a richer normalized table is straightforward from the official columns, implement it, but do not block on perfect economics in this issue. A generic structured long metric panel is acceptable as first live foundation.

Required FII aggregates

Create a source-specific low-dimensional aggregate panel if numeric fields are present:

  • counts of reporting funds by ref_date
  • sums/means for numeric metrics grouped by metric_name
  • optional fund_type/segment group if available without PIT leakage

Do not join to B3 securities yet unless a PIT FII security master is already available.

FII tests required

  • Annual resource generation/discovery from 2016 onward.
  • Parser fixture with at least two tables or two numeric/text fields.
  • Window filtering after annual ZIP parse.
  • available_date uses HTTP/resource metadata when present; otherwise first-seen rows are marked model_usable=false if you add a model usability flag.
  • Quality checks for non-null ref_date, fund_id, and no duplicate primary keys.

4. CVM FIDC monthly reports: make live

Decision

CVM FIDC monthly reports can be made live. The official page says it provides the last twelve monthly files updated weekly with new presentations/reapresentations and also provides historical files since 2019 and since 2013. The official direct directories expose:

  • current monthly files: https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/inf_mensal_fidc_YYYYMM.zip
  • historical annual files: https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/HIST/inf_mensal_fidc_YYYY.zip

Required config

Promote cvm_fidc_reports from source_map_only_official_page to live_download.

Suggested config:

- dataset_id: cvm_fidc_reports
  source_map_status: live_download
  raw_format: zip_csv_multi_table
  period_routing:
    historical_annual_from: 2013
    current_monthly_directory: "https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/"
    historical_directory: "https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/HIST/"
  source_urls:
    - name: fidc_monthly
      url_template: "https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/inf_mensal_fidc_{period:%Y%m}.zip"
      filename_template: "inf_mensal_fidc_{period:%Y%m}.zip"
    - name: fidc_historical_annual
      url_template: "https://dados.cvm.gov.br/dados/FIDC/DOC/INF_MENSAL/DADOS/HIST/inf_mensal_fidc_{year}.zip"
      filename_template: "inf_mensal_fidc_{year}.zip"

Routing rule:

  • Use current monthly files when the requested period is present in the current direct directory.
  • Use historical annual files for older years/months.
  • Do not guess future monthly files. Missing expected resources should fail clearly or produce failed manifest rows.

Required parser/normalizer

Implement:

  • src/bralpha/parsing/cvm_fidc.py
  • src/bralpha/normalization/cvm_fidc.py
  • extend src/bralpha/pipelines/cvm_ingest.py

Parser must support multiple tables inside ZIPs and preserve table_name from inner filenames.

Important official layout note to encode in docs/tests:

  • The CVM page states fields changed from 11/2019 onward and that 10/2023 has a document-version transition with new columns and a new table X. Do not force a single rigid schema across all vintages. Use diagonal/long normalization with table identity and raw column preservation.

Minimum silver schema:

  • ref_date
  • available_date
  • fund_id
  • class_id / cnpj_classe if present
  • table_name
  • metric_name
  • metric_value where numeric
  • metric_text where nonnumeric
  • source_file
  • inner_filename
  • source_version
  • provenance columns

Required FIDC aggregates

Create a low-dimensional source-specific panel for model foundations:

  • reporting fund count by ref_date
  • numeric metric sums/means by table_name and metric_name
  • specific named metrics only if column names are stable in fixtures, e.g. PL/NAV, quota value, delinquency/overdue/default fields, subordinated/senior fields.

Avoid overfitting to one layout version. Preserve generic long metrics if stable named metrics are uncertain.

FIDC tests required

  • Current monthly resource discovery/generation for available months.
  • Historical annual resource generation/discovery from 2013 onward.
  • Parser fixture for pre-11/2019 and post-10/2023 layout variants, if representative fields are available.
  • Parser fixture with multiple tables inside one ZIP.
  • Window filtering after annual/historical ZIP parse.
  • Quality checks for non-null ref_date, fund_id when present, table name, and no duplicate primary keys.

Cross-source acceptance criteria

This issue is complete when:

  1. B3 daily-bulletin automatic download remains non-live unless a real date-specific official HTTP workflow is fixture-proven.
  2. A controlled B3 manual official-file import pipeline exists and is documented.
  3. b3_index_daily can be populated from a manually supplied official IndexReport file.
  4. b3_traded_securities can be populated from at least one manually supplied official Instruments/Tradable Security file, or remains explicitly blocked with a parser/catalog reason.
  5. b3_reference_rates can be populated from at least one manually supplied official reference/risk/rate report, or remains explicitly blocked with a parser/catalog reason.
  6. B3 investor/foreign-flow datasets either have confirmed official report labels and manual-import parsers, or remain source-map-only with a precise official-page/manual workflow blocker.
  7. configs/manual/bcb_copom.yaml is populated with official calendar/decision rows from model start through current schedule, with no future decision leakage.
  8. BCB Copom silver and event panels are generated from the populated manual source and documented as manual-official data.
  9. cvm_fii_reports is live with official annual ZIP ingestion, parser fixtures, and source-specific silver output.
  10. cvm_fidc_reports is live with current monthly + historical annual routing, parser fixtures, and source-specific silver output.
  11. All new live/manual-ingest datasets write manifests with raw provenance and pass source-specific quality checks.
  12. No live-network tests are added.
  13. No raw downloaded data is committed.

Suggested implementation order

  1. Implement B3 manual official-file import infrastructure and catalog validation.
  2. Wire b3_index_daily to the manual importer using the existing IndexReport parser fixture.
  3. Add B3 traded-security/reference-rate parser fixtures if representative official files are available; otherwise lock them as blocked with exact reason.
  4. Populate and validate configs/manual/bcb_copom.yaml; add Copom event panel.
  5. Make cvm_fii_reports live.
  6. Make cvm_fidc_reports live.
  7. Update docs and tests.

Follow-up issue expected

After this issue, open a separate B3 live-download investigation issue to capture the actual browser/network request fixtures for the official B3 date-specific daily-bulletin workflow. That future issue can promote B3 manual official-file ingestion to automatic live_download once the official request/response contract is deterministic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions