Implement CVM raw-to-research spine

# Codex Design Contract: CVM Raw-to-Research Spine

You are implementing the CVM raw-to-research layer for Brazil-RV.

This comes after CVM structured ingestion. Do not modify CVM ingestion except for tiny bugs found by tests. Do not add new CVM datasets. Do not add CDA/IPE/FII/FIDC/company filing normalization. Do not add modeling, backtesting, portfolio construction, event-risk overlays, NLP, document-body parsing, PDF parsing, scraping, browser automation, or new dependencies.

Read first:

- `AGENTS.md`
- `docs/PROJECT_PLAN.md`
- `docs/TIMING_AND_AVAILABILITY_POLICY.md`
- `docs/CVM_SOURCE_MAP.md`
- `configs/datasets/cvm.yaml`
- `src/bralpha/normalization/cvm_funds.py`
- `src/bralpha/pipelines/cvm_ingest.py`
- BCB/IBGE/Tesouro/FRED raw-to-research implementations for structure only.

## Current CVM silver scope

Only these current CVM silver datasets are model-usable in this PR:

```text
data/silver/cvm_fund_daily_reports/
data/silver/cvm_fund_registry_current/
```

These current CVM datasets are raw/bronze-only or source-mapped and must not be used for model-ready gold panels in this PR:

```text
data/bronze/cvm/cvm_fund_registry_history/
data/bronze/cvm/cvm_fund_class_registry/
cvm_fund_portfolio_cda
cvm_company_ipe_metadata
cvm_fii_reports
cvm_fidc_reports
cvm_company_itr
cvm_company_dfp
cvm_public_offerings
cvm_sanctions_regulatory_events
```

Do not try to classify historical funds using current registry fields in a way that creates point-in-time leakage. Current registry is reference/audit metadata unless explicitly used as current snapshot metadata.

---

# Goal

Convert current CVM source-specific silver tables into research-ready CVM gold panels:

```text
data/silver/cvm_fund_daily_reports/
data/silver/cvm_fund_registry_current/
    -> data/gold/cvm/{panel}/
```

The CVM research layer should make local fund-flow and fund-stock data usable for daily sequence models while avoiding high-cardinality per-fund model features and avoiding hand-engineered alpha features.

---

# Core principles

## 1. Source-specific silver remains immutable

Do not mutate or overwrite `data/silver/...`.

Gold outputs go only to:

```text
data/gold/cvm/{panel}/
```

## 2. Strict point-in-time policy

Every model-usable daily row must have:

```text
ref_date
available_date
```

For observation panels:

```text
ref_date = CVM report/economic date from silver
available_date = model-usable decision date from silver
```

For daily aligned flow panels:

```text
ref_date = model/as-of date = observation_available_date
available_date = ref_date
observation_ref_date = original CVM report date
observation_available_date = original CVM available_date
```

For as-of state panels:

```text
ref_date = model/as-of date
available_date = ref_date
observation_ref_date = original CVM report date
observation_available_date = original CVM available_date
```

Only use observations satisfying:

```text
observation_available_date <= ref_date
```

Never use `download_timestamp_utc` as historical availability.

## 3. Do not forward-fill flow variables

Subscriptions and redemptions are daily flow observations.

Do not forward-fill:

```text
subscriptions
redemptions
```

They should appear on the model date when they become available.

## 4. Forward-fill only state variables

These can be carried as latest available state:

```text
portfolio_value
nav
shareholder_count
fund_count/report_count coverage fields
```

Do not forward-fill subscriptions or redemptions.

## 5. Avoid high-cardinality fund-level model panels

Do not create model-ready daily-long rows for every fund. That would be too high-dimensional and noisy for the first neural model.

Keep per-fund data as observation/audit panels. Model-oriented daily-long should use low-cardinality aggregate groups only.

Allowed aggregate groups in this PR:

```text
all funds
fund_type from CVM daily report silver
```

Do not use current registry classes for historical model features in this PR.

## 6. Transformer-aware feature rule

Allowed structural/preprocessing fields:

```text
fund_id
fund_type
ref_date
available_date
observation_ref_date
observation_available_date
official portfolio_value/nav/quota_value/subscriptions/redemptions/shareholder_count
raw official value fields
fund/report counts
non-null metric counts
coverage/missingness flags
feature_id
group_type/group_value
staleness_days
source_version
```

Allowed structural aggregation:

```text
sum official values by low-cardinality group and date
count reported funds / non-null metric counts by group and date
```

Do not add:

```text
net flow
flow ratios
subscriptions divided by NAV
redemptions divided by NAV
momentum
rolling sums
rolling means
rolling z-scores
rolling volatility
stress labels
fund return estimates
quota returns
drawdowns
shares by group
classification labels from current registry
portfolio/backtest fields
```

We preserve subscriptions and redemptions separately. The model can learn their difference/interaction later if useful.

---

# Required files

Create only files with real code:

```text
docs/CVM_RAW_TO_RESEARCH_SPINE.md

configs/derived/cvm.yaml

src/bralpha/derived/cvm/io.py
src/bralpha/derived/cvm/schemas.py
src/bralpha/derived/cvm/calendar.py
src/bralpha/derived/cvm/fund_reports.py
src/bralpha/derived/cvm/registry.py
src/bralpha/derived/cvm/daily_long.py
src/bralpha/derived/cvm/quality.py

src/bralpha/pipelines/cvm_research_spine.py

tests/test_cvm_research_spine_io.py
tests/test_cvm_fund_report_research_panels.py
tests/test_cvm_registry_research_panels.py
tests/test_cvm_daily_long_panel.py
```

Do not create empty modules.

Update `src/bralpha/infra/config.py` only as needed to add `load_cvm_research_config(repo_root)`.

---

# Config

Add:

```text
configs/derived/cvm.yaml
```

Minimum content:

```yaml
cvm_research:
  calendar:
    default: business_days_mon_fri
    timezone: America/Sao_Paulo

  fund_reports:
    include_per_fund_observation: true
    include_group_observation: true
    include_flow_daily: true
    include_state_asof_daily: true
    group_by:
      - all
      - fund_type
    max_groups: 100

  registry:
    include_current_reference: true

  daily_long:
    include_fund_flows: true
    include_fund_state: true
```

Rules:

- Keep config small.
- Do not add flags for net flow, ratios, returns, rolling windows, or stress features.
- `max_groups` protects accidental high-cardinality grouping.

---

# Required gold panels

## 1. `fund_daily_observation`

Output:

```text
data/gold/cvm/fund_daily_observation/
```

Input:

```text
data/silver/cvm_fund_daily_reports/
```

Purpose: per-fund official CVM daily report observations for audit and future aggregation.

Required columns:

```text
ref_date
available_date
availability_policy
fund_id
fund_type
portfolio_value
nav
quota_value
subscriptions
redemptions
shareholder_count
raw_vl_total
raw_vl_patrim_liq
raw_vl_quota
raw_captc_dia
raw_resg_dia
raw_nr_cotst
has_portfolio_value
has_nav
has_quota_value
has_subscriptions
has_redemptions
has_shareholder_count
source_version
```

Rules:

- Preserve official numeric values from silver.
- Preserve raw official fields.
- Add only missingness flags.
- Do not compute net flow, ratios, returns, rolling values, stress labels, or classification labels.
- Date filter by `ref_date` for observation-panel builds.

Primary key:

```text
ref_date, fund_id
```

Quality checks:

```text
required columns present
no duplicate primary keys
available_date >= ref_date where available_date is not null
```

---

## 2. `fund_group_observation`

Output:

```text
data/gold/cvm/fund_group_observation/
```

Input:

```text
fund_daily_observation
```

Purpose: low-cardinality official aggregate observations by date and group.

Required columns:

```text
ref_date
available_date
group_type
group_value
feature_id
portfolio_value
nav
subscriptions
redemptions
shareholder_count
fund_count
portfolio_value_count
nav_count
subscriptions_count
redemptions_count
shareholder_count_count
source_version
```

Allowed groups:

```text
group_type = all, group_value = all
group_type = fund_type, group_value = normalized fund_type or unknown
```

Rules:

- Aggregate only low-cardinality groups configured in `group_by`.
- `available_date = max(available_date)` across contributing rows for each `ref_date/group`.
- Sum official numeric values using null-safe semantics:
  - sum non-null values;
  - if all values for a metric are null, output null for that metric.
- Count reported funds and non-null metric counts.
- `feature_id` deterministic from `group_type/group_value`.
- Do not compute net flow, ratios, shares, momentum, rolling values, or stress labels.
- Do not aggregate by fund_id.
- Raise a clear error if group count exceeds `max_groups`.

Primary key:

```text
ref_date, group_type, group_value
```

---

## 3. `fund_flows_daily`

Output:

```text
data/gold/cvm/fund_flows_daily/
```

Input:

```text
fund_group_observation
```

Purpose: model-date aligned official flow observations.

Required columns:

```text
ref_date
available_date
observation_ref_date
observation_available_date
group_type
group_value
feature_id
subscriptions
redemptions
subscriptions_count
redemptions_count
fund_count
source_version
```

Rules:

- `observation_ref_date = fund_group_observation.ref_date`.
- `observation_available_date = fund_group_observation.available_date`.
- `ref_date = observation_available_date`.
- `available_date = ref_date`.
- Keep rows only where `ref_date` falls in requested `[start, end]`.
- Do not forward-fill.
- Do not compute net flow, ratios, rolling sums, or momentum.

Primary key:

```text
ref_date, group_type, group_value, observation_ref_date
```

Note: include `observation_ref_date` in the key because multiple CVM report dates could theoretically become available on the same model date.

---

## 4. `fund_state_asof_daily`

Output:

```text
data/gold/cvm/fund_state_asof_daily/
```

Input:

```text
fund_group_observation
```

Purpose: daily latest-available fund stock/state panel.

Required columns:

```text
ref_date
available_date
group_type
group_value
feature_id
observation_ref_date
observation_available_date
portfolio_value
nav
shareholder_count
fund_count
portfolio_value_count
nav_count
shareholder_count_count
is_available
is_observed_on_ref_date
staleness_days
source_version
```

Rules:

- `ref_date` is the model/as-of date.
- `available_date = ref_date`.
- Use latest group observation satisfying:

```text
observation_available_date <= ref_date
```

- Build/read group observation history through `end`, not only `[start, end]`.
- Emit rows only after first availability.
- Do not include subscriptions/redemptions in this panel.
- Do not forward-fill flow variables.
- Preserve latest missing state observation if all values are missing; do not silently fall back to older numeric values.
- `staleness_days = ref_date - observation_available_date`.
- Use long/skinny format by group; do not pivot wide.

Primary key:

```text
ref_date, feature_id
```

Implementation hint:

- Use Polars `join_asof` by `feature_id`.
- Use Mon-Fri business-day calendar for first pass.

---

## 5. `fund_registry_current_reference`

Output:

```text
data/gold/cvm/fund_registry_current_reference/
```

Input:

```text
data/silver/cvm_fund_registry_current/
```

Purpose: current fund registry/reference metadata for audit and future mapping.

Required columns:

```text
fund_id
fund_type
fund_name
cvm_code
registration_date
constitution_date
cancellation_date
status
status_start_date
activity_start_date
class_name
class_start_date
benchmark_or_return_target
condominium_type
is_fund_of_funds
is_exclusive
is_long_term_tax
public_target
admin_id
admin_name
manager_id
manager_name
custodian_id
custodian_name
auditor_id
auditor_name
controller_id
controller_name
snapshot_date
source_version
```

Rules:

- Preserve current registry fields.
- This is reference metadata, not point-in-time historical classification.
- Do not join it into historical fund-flow model panels in this PR.
- Do not infer classes from fund names.
- Do not create daily model rows from current registry metadata.

Primary key:

```text
fund_id
```

Write unpartitioned or partitioned by `snapshot_date` year only if existing helper requires date partitioning. Prefer unpartitioned reference output if convenient.

---

## 6. `cvm_daily_long`

Output:

```text
data/gold/cvm/daily_long/
```

Inputs:

```text
fund_flows_daily
fund_state_asof_daily
```

Purpose: unified long-form CVM panel for later cross-source model assembly.

Required columns:

```text
ref_date
available_date
source_family
feature_id
value_name
value
unit
observation_ref_date
observation_available_date
is_available
staleness_days
source_version
```

Rules:

- Long-format only. Do not pivot wide.
- Include only aggregate group panels, not per-fund rows.
- Flow value names:
  - `subscriptions`
  - `redemptions`
  - `subscriptions_count`
  - `redemptions_count`
  - `fund_count`
- State value names:
  - `portfolio_value`
  - `nav`
  - `shareholder_count`
  - `portfolio_value_count`
  - `nav_count`
  - `shareholder_count_count`
  - `fund_count`
- `source_family` values:
  - `cvm_fund_flows`
  - `cvm_fund_state`
- Drop rows where `value` is null.
- Do not compute net flow, ratios, returns, shares, rolling values, z-scores, or stress labels.

Primary key:

```text
ref_date, source_family, feature_id, value_name
```

If `fund_flows_daily` can produce multiple `observation_ref_date` rows for the same `ref_date/feature_id/value_name`, then include `observation_ref_date` in the daily-long primary key. Prefer uniqueness without collisions; tests must cover this.

---

# IO and performance

Create:

```text
src/bralpha/derived/cvm/io.py
```

Rules:

- Read from `data/silver/{dataset_id}/`.
- Write to `data/gold/cvm/{panel}/`.
- Use partitioned Parquet by year for date-indexed panels.
- Use exact panel primary keys for gold writes, with `augment_source_dataset_key=False`.
- Do not mutate source silver.
- Missing optional inputs should skip panels with clear status in full runs.
- Explicit selected panel with missing required input should raise a clear error.
- Date-bounded reads must prune nested `year=YYYY` partitions because CVM silver is partitioned by `year/month`.
- For as-of panels, read observation/group history with `start=None, end=end`, then emit rows only for `[start, end]`.

Use Polars and existing Parquet helpers. Do not add Dask/Spark/Ray/database/orchestration.

Performance requirements:

- Do not build high-cardinality fund-id dense as-of panels.
- Aggregations should be Polars-native.
- Group panels should have low cardinality only.
- Avoid row-wise Python loops over fund daily data.

---

# Pipeline

Add:

```text
src/bralpha/pipelines/cvm_research_spine.py
```

CLI:

```bash
python -m bralpha.pipelines.cvm_research_spine \
  --repo-root . \
  --start 2010-01-01 \
  --end 2026-01-01
```

Optional repeated flag:

```text
--panel fund_daily_observation
--panel fund_group_observation
--panel fund_flows_daily
--panel fund_state_asof_daily
--panel fund_registry_current_reference
--panel daily_long
```

Behavior:

- Read existing CVM silver only.
- Do not run CVM ingestion.
- Do not hit live CVM.
- Do not read raw/bronze-only CVM datasets.
- Write outputs under `data/gold/cvm/`.
- Return/print concise status by panel.
- Skip missing optional inputs in full run.
- Raise on missing required inputs when a panel is explicitly selected.

---

# Documentation

Add:

```text
docs/CVM_RAW_TO_RESEARCH_SPINE.md
```

Document:

1. Input CVM silver datasets.
2. Output CVM gold panels.
3. Strict point-in-time policy.
4. Difference between flow panels and state/as-of panels.
5. Why per-fund observations are preserved but not emitted to daily-long.
6. Why only `all` and `fund_type` groups are included in this PR.
7. Transformer-aware feature rule.
8. What is intentionally excluded:
   - net flow,
   - ratios,
   - returns,
   - rolling sums,
   - z-scores,
   - stress labels,
   - historical classification from current registry,
   - registry-history/class-registry bronze-only datasets,
   - CDA/IPE/FII/FIDC/company filing datasets,
   - portfolio/backtest logic.
9. Known limitation: current fund registry is reference-only until historical class registry normalization is implemented.

---

# Tests

Use synthetic fixtures only. Do not download CVM data.

## IO

- Missing optional inputs skip in full pipeline.
- Explicit selected panel with missing required input raises.
- Outputs write under `data/gold/cvm/`.
- Date-bounded reads prune nested `year=YYYY/month=M` partitions.
- Gold writes use exact panel primary keys.

## Fund daily observation

- Preserves official values and raw official fields.
- Adds missingness flags.
- Does not compute net flow, ratios, returns, or rolling values.
- Primary keys are unique.

Do not implement this as a static banned-feature denylist test. Validate behavior and expected columns through schema/order and value assertions.

## Fund group observation

- Builds `all` group and `fund_type` group.
- Sums official values correctly with null-aware behavior.
- Produces non-null metric counts and fund counts.
- `available_date` is max contributing availability date.
- `feature_id` is deterministic.
- `max_groups` raises when exceeded.

## Fund flows daily

- Aligns flows to `observation_available_date`.
- `ref_date = available_date = observation_available_date`.
- Does not forward-fill.
- Preserves subscriptions and redemptions separately.
- Handles multiple observation dates becoming available on same model date without duplicate-key collisions.

## Fund state as-of daily

- Uses latest group observation with `observation_available_date <= ref_date`.
- Uses pre-window group history at output start.
- Does not emit rows before first availability.
- Does not include subscriptions/redemptions.
- `staleness_days` is correct.

## Registry reference

- Preserves current registry metadata.
- Does not create daily model rows.
- Does not join current registry classifications into historical flow panels.

## Daily long

- Includes aggregate flow and state rows when inputs exist.
- Drops null value rows.
- Uses long primary key without collisions.
- Does not include per-fund rows.
- Does not pivot wide.
- Does not include registry reference metadata.

---

# Acceptance criteria

The PR is complete only when:

1. `docs/CVM_RAW_TO_RESEARCH_SPINE.md` exists.
2. `configs/derived/cvm.yaml` exists.
3. All required CVM gold panels are implemented.
4. Outputs go only to `data/gold/cvm/{panel}/`.
5. Source-specific CVM silver inputs are not mutated.
6. Every daily model-usable row has `ref_date` and `available_date`.
7. Flow panels are not forward-filled.
8. State/as-of panels never use observations with `observation_available_date > ref_date`.
9. Per-fund observations are preserved but not emitted to `daily_long`.
10. Current registry is reference-only and not used as historical classification in model panels.
11. No net-flow, ratio, return, rolling, z-score, stress-label, stationarity, or portfolio/backtest features are added.
12. No live CVM downloads happen in this pipeline.
13. No new dependencies are added.
14. Tests cover IO, observations, group aggregation, flow alignment, state as-of, registry reference, and daily-long.
15. `python -m pytest` passes.
16. `python -m ruff check .` passes.
17. No generated data is committed.

Keep the implementation structural, point-in-time safe, low-cardinality, and efficient.


Implement CVM raw-to-research spine #33

Description

Codex Design Contract: CVM Raw-to-Research Spine

Current CVM silver scope

Goal

Core principles

1. Source-specific silver remains immutable

2. Strict point-in-time policy

3. Do not forward-fill flow variables

4. Forward-fill only state variables

5. Avoid high-cardinality fund-level model panels

6. Transformer-aware feature rule

Required files

Config

Required gold panels

1. fund_daily_observation

2. fund_group_observation

3. fund_flows_daily

4. fund_state_asof_daily

5. fund_registry_current_reference

6. cvm_daily_long

IO and performance

Pipeline

Documentation

Tests

IO

Fund daily observation

Fund group observation

Fund flows daily

Fund state as-of daily

Registry reference

Daily long

Acceptance criteria

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. `fund_daily_observation`

2. `fund_group_observation`

3. `fund_flows_daily`

4. `fund_state_asof_daily`

5. `fund_registry_current_reference`

6. `cvm_daily_long`