Implement ANP raw-to-research spine

# Codex Design Contract: ANP Raw-to-Research Spine

You are implementing the ANP raw-to-research layer for Brazil-RV.

This comes after ANP structured ingestion. Do not modify ANP ingestion except for tiny bugs found by tests. Do not add new ANP datasets. Do not implement deferred downstream movements, royalties, reference prices, registries, PMQC, stock, storage, import/export, or RenovaBio/CBIO datasets. Do not add modeling, backtesting, portfolio logic, event-risk overlays, NLP, PDF/XLSX parsing, scraping, browser automation, or new dependencies.

Read first:

- `AGENTS.md`
- `docs/PROJECT_PLAN.md`
- `docs/TIMING_AND_AVAILABILITY_POLICY.md`
- `docs/ANP_SOURCE_MAP.md`
- `configs/datasets/anp.yaml`
- `src/bralpha/normalization/anp_fuels.py`
- `src/bralpha/pipelines/anp_ingest.py`
- ONS/CVM/FRED/Tesouro/IBGE/BCB raw-to-research implementations for structure only.

---

## Current ANP silver scope

Only these current ANP silver datasets are model-usable in this PR:

```text
data/silver/anp_fuel_prices_weekly/
data/silver/anp_fuel_sales_monthly/
data/silver/anp_oil_gas_production_monthly/
```

These datasets are source-mapped/deferred and must not be used in this PR:

```text
anp_downstream_movements
anp_government_take_royalties
anp_government_take_special_participation
anp_reference_oil_price
anp_reference_gas_price
anp_fuel_station_registry
anp_glp_reseller_registry
anp_distributor_registry
anp_storage_capacity
anp_import_export_authorizations
anp_quality_monitoring_pmqc
anp_fuel_stock_data
anp_renovabio_cbio
```

---

# Goal

Convert current ANP source-specific silver tables into research-ready ANP gold panels:

```text
data/silver/anp_fuel_prices_weekly/
data/silver/anp_fuel_sales_monthly/
data/silver/anp_oil_gas_production_monthly/
    -> data/gold/anp/{panel}/
```

The ANP research layer should make official fuel-price, fuel-demand, and oil/gas production information usable for daily sequence models without adding hand-engineered alpha features.

---

# Core principles

## 1. Source-specific silver remains immutable

Do not mutate or overwrite:

```text
data/silver/anp_*/
```

Gold outputs go only to:

```text
data/gold/anp/{panel}/
```

## 2. Strict point-in-time policy

Every model-usable row must have:

```text
ref_date
available_date
```

For observation panels:

```text
ref_date = ANP observation/reference date
available_date = model-usable decision date from silver
```

For as-of panels:

```text
ref_date = model/as-of date
available_date = ref_date
observation_ref_date = original ANP ref_date
observation_available_date = original ANP available_date
```

Only use observations satisfying:

```text
observation_available_date <= ref_date
```

Never use `download_timestamp_utc` as historical availability.

## 3. No hourly aggregation is needed for ANP

Current ANP silver has no hourly data.

The hourly-to-daily concern from ONS does not apply here. Do **not** create hourly logic, hourly calendars, or intraday aggregation code for ANP.

The relevant ANP alignment problems are instead:

```text
station/product weekly survey microdata -> low-cardinality official aggregate observations -> latest-available daily state
monthly sales/production observations -> latest-available daily state
```

## 4. Aggregate station microdata before model-facing panels

`anp_fuel_prices_weekly` is station-level microdata. Do not emit station-level rows to model-facing `daily_long`; that would be high-cardinality noise.

Preserve station rows in an audit observation panel, but model-facing panels must use low-cardinality aggregates only.

Allowed fuel-price groups:

```text
all Brazil by product
region by product
state by product
```

Do not aggregate by municipality, station, brand, street, or CNPJ in this PR.

## 5. Transformer-aware feature rule

Allowed structural/preprocessing fields:

```text
observation_id
ref_date
available_date
observation_ref_date
observation_available_date
region
state
product
location
metric_type
resource_family
sale_price
purchase_price
sales_volume_m3
metric_value
count fields
missingness fields
feature_id
source_family
value_name
unit
staleness_days
source_version
```

Allowed structural aggregation:

```text
mean official sale/purchase prices by product/geography/ref_date
sum official monthly sales volumes by product/geography/ref_date
sum official production/gas metrics by metric_type/product/location/geography/ref_date
count contributing observations and non-null metrics
```

Do not add:

```text
price changes
fuel spreads
sale-purchase spreads
pass-through
inflation proxies
monthly changes
growth rates
shares
ratios
per-capita values
rolling sums
rolling means
rolling z-scores
rolling volatility
seasonal anomalies
stationarity transforms
BOE conversion
flaring share
Petrobras labels
stress labels
portfolio/backtest fields
```

If later we want fuel inflation proxies, diesel-gasoline spreads, oil-equivalent conversion, flaring shares, or rolling anomalies, that belongs in a later model-feature or cross-source assembly layer, not this ANP raw-to-research spine.

---

# Required files

Create only files with real code:

```text
docs/ANP_RAW_TO_RESEARCH_SPINE.md

configs/derived/anp.yaml

src/bralpha/derived/anp/io.py
src/bralpha/derived/anp/schemas.py
src/bralpha/derived/anp/calendar.py
src/bralpha/derived/anp/fuel_prices.py
src/bralpha/derived/anp/fuel_sales.py
src/bralpha/derived/anp/oil_gas.py
src/bralpha/derived/anp/daily_long.py
src/bralpha/derived/anp/quality.py

src/bralpha/pipelines/anp_research_spine.py

tests/test_anp_research_spine_io.py
tests/test_anp_fuel_price_research_panels.py
tests/test_anp_fuel_sales_research_panels.py
tests/test_anp_oil_gas_research_panels.py
tests/test_anp_daily_long_panel.py
```

Do not create empty modules.

Update `src/bralpha/infra/config.py` only as needed to add:

```python
load_anp_research_config(repo_root)
```

---

# Config

Add:

```text
configs/derived/anp.yaml
```

Minimum content:

```yaml
anp_research:
  calendar:
    default: business_days_mon_fri
    timezone: America/Sao_Paulo

  fuel_prices:
    include_station_observation: true
    include_group_observation: true
    group_by:
      - all
      - region
      - state
    max_groups: 500
    aggregation: mean

  fuel_sales:
    include_observation: true
    include_group_observation: true
    group_by:
      - all
      - region
      - state
    max_groups: 500

  oil_gas:
    include_observation: true
    include_group_observation: true
    group_by:
      - all
      - region
      - state
    max_groups: 1000

  asof:
    include_state_asof_daily: true
    max_features: 20000

  daily_long:
    include_fuel_prices: true
    include_fuel_sales: true
    include_oil_gas: true
```

Rules:

- Keep config small.
- Do not add flags for spreads, changes, returns, ratios, rolling features, or stress labels.
- `max_groups` and `max_features` protect against accidental high-cardinality expansion.

---

# Required gold panels

## 1. `fuel_price_station_observation`

Output:

```text
data/gold/anp/fuel_price_station_observation/
```

Input:

```text
data/silver/anp_fuel_prices_weekly/
```

Purpose: station-level official ANP price observations for audit and future aggregation.

Required columns:

```text
observation_id
ref_date
available_date
availability_policy
region
state
municipality
station_name
station_cnpj
product
sale_price
purchase_price
unit
brand
resource_family
has_sale_price
has_purchase_price
source_version
```

Rules:

- Preserve official sale and purchase prices separately.
- Preserve `purchase_price = null` after August 2020 where applicable.
- Add only missingness flags.
- Do not compute spreads, changes, ratios, pass-through, inflation proxies, or station-level returns.
- Date filter by `ref_date`.

Primary key:

```text
observation_id
```

Quality checks:

```text
required columns present
no duplicate primary keys
available_date >= ref_date where available_date is not null
```

---

## 2. `fuel_price_group_observation`

Output:

```text
data/gold/anp/fuel_price_group_observation/
```

Input:

```text
fuel_price_station_observation
```

Purpose: low-cardinality official fuel-price aggregates by product/date/geography.

Required columns:

```text
ref_date
available_date
availability_policy
group_type
group_value
product
feature_id
sale_price
purchase_price
station_count
sale_price_count
purchase_price_count
unit
source_version
```

Allowed groups:

```text
group_type = all, group_value = all
group_type = region, group_value = normalized region or unknown
group_type = state, group_value = normalized state or unknown
```

Rules:

- Aggregate by `ref_date, group_type, group_value, product`.
- Use mean over non-null official station prices.
- If all values for a metric are null, output null for that metric.
- `available_date = max(available_date)` across contributing station rows.
- `station_count = number of station/product observations`.
- Count non-null sale and purchase prices separately.
- `feature_id` deterministic from `group_type/group_value/product`.
- Raise if group count exceeds `max_groups`.
- Do not compute sale-purchase spread, fuel spreads, changes, ratios, or rolling values.

Primary key:

```text
ref_date, group_type, group_value, product
```

---

## 3. `fuel_sales_observation`

Output:

```text
data/gold/anp/fuel_sales_observation/
```

Input:

```text
data/silver/anp_fuel_sales_monthly/
```

Purpose: official state/product monthly fuel-sales observations.

Required columns:

```text
ref_date
available_date
availability_policy
year
month
region
state
product
sales_volume_m3
unit
has_sales_volume_m3
source_version
```

Rules:

- Preserve official monthly sales volume.
- Add only missingness flag.
- Do not compute monthly changes, shares, ratios, per-capita values, or rolling values.
- Date filter by `ref_date`.

Primary key:

```text
ref_date, state, product
```

---

## 4. `fuel_sales_group_observation`

Output:

```text
data/gold/anp/fuel_sales_group_observation/
```

Input:

```text
fuel_sales_observation
```

Purpose: low-cardinality monthly fuel-demand aggregates.

Required columns:

```text
ref_date
available_date
availability_policy
group_type
group_value
product
feature_id
sales_volume_m3
sales_volume_count
state_count
unit
source_version
```

Rules:

- Aggregate by `ref_date, group_type, group_value, product`.
- Allowed groups: `all`, `region`, `state`.
- Sum official sales volumes over non-null rows.
- If all values are null, output null.
- Count contributing rows and non-null volume rows.
- `available_date = max(available_date)` across contributing rows.
- `feature_id` deterministic from `group_type/group_value/product`.
- Raise if group count exceeds `max_groups`.
- Do not compute changes, shares, ratios, or rolling values.

Primary key:

```text
ref_date, group_type, group_value, product
```

---

## 5. `oil_gas_production_observation`

Output:

```text
data/gold/anp/oil_gas_production_observation/
```

Input:

```text
data/silver/anp_oil_gas_production_monthly/
```

Purpose: official monthly ANP petroleum/LGN/gas metric observations.

Required columns:

```text
ref_date
available_date
availability_policy
year
month
region
state
location
product
metric_type
metric_value
unit
resource_family
has_metric_value
source_version
```

Rules:

- Preserve official metric type/value/unit from silver.
- Add only missingness flag.
- Do not compute BOE, rates, ratios, changes, flaring shares, or Petrobras labels.
- Date filter by `ref_date`.

Primary key:

```text
ref_date, state, location, metric_type
```

---

## 6. `oil_gas_group_observation`

Output:

```text
data/gold/anp/oil_gas_group_observation/
```

Input:

```text
oil_gas_production_observation
```

Purpose: low-cardinality monthly petroleum/gas state aggregates.

Required columns:

```text
ref_date
available_date
availability_policy
group_type
group_value
location
product
metric_type
feature_id
metric_value
metric_value_count
state_count
unit
source_version
```

Rules:

- Aggregate by `ref_date, group_type, group_value, location, product, metric_type`.
- Allowed groups: `all`, `region`, `state`.
- Sum official metric values over non-null rows.
- If all values are null, output null.
- Count contributing rows and non-null metric rows.
- `available_date = max(available_date)` across contributing rows.
- `feature_id` deterministic from `group_type/group_value/location/product/metric_type`.
- Raise if group count exceeds `max_groups`.
- Do not compute BOE, flaring shares, gas availability ratios, changes, or rolling values.

Primary key:

```text
ref_date, group_type, group_value, location, product, metric_type
```

---

## 7. `anp_state_asof_daily`

Output:

```text
data/gold/anp/state_asof_daily/
```

Inputs:

```text
fuel_price_group_observation
fuel_sales_group_observation
oil_gas_group_observation
```

Purpose: daily latest-available ANP state panel for model dates.

Required columns:

```text
ref_date
available_date
source_family
feature_id
observation_ref_date
observation_available_date
value_name
value
unit
is_available
is_observed_on_ref_date
staleness_days
source_version
```

Rules:

- Long-format only.
- `ref_date` is model/as-of date.
- `available_date = ref_date`.
- Use latest observation satisfying:

```text
observation_available_date <= ref_date
```

- Build/read observation history through `end`, not only `[start, end]`.
- Emit rows only after first availability.
- Preserve latest missing official observation if it is latest; do not silently fall back to older numeric value.
- `staleness_days = ref_date - observation_available_date`.
- Weekly fuel-price aggregates should be carried forward as latest state with increasing `staleness_days`.
- Monthly sales/production aggregates should be carried forward as latest state with higher `staleness_days`.
- Do not compute changes, ratios, spreads, rolling features, z-scores, stress labels, or stationarity transforms.
- Raise if selected feature count exceeds `max_features`.

Value families:

```text
source_family = anp_fuel_price
value_name = sale_price, purchase_price, station_count, sale_price_count, purchase_price_count

source_family = anp_fuel_sales
value_name = sales_volume_m3, sales_volume_count, state_count

source_family = anp_oil_gas
value_name = metric_value, metric_value_count, state_count
```

Primary key:

```text
ref_date, source_family, feature_id, value_name
```

---

## 8. `anp_daily_long`

Output:

```text
data/gold/anp/daily_long/
```

Input:

```text
anp_state_asof_daily
```

Purpose: unified long-form ANP panel for later cross-source assembly.

Required columns:

```text
ref_date
available_date
source_family
feature_id
value_name
value
unit
observation_ref_date
observation_available_date
is_available
staleness_days
source_version
```

Rules:

- This can be a selected copy of `anp_state_asof_daily`.
- Drop rows where `value` is null.
- Keep long format; do not pivot wide.
- Do not include station-level rows.
- Do not include source-mapped/deferred datasets.
- Do not compute derived features.

Primary key:

```text
ref_date, source_family, feature_id, value_name
```

---

# IO and performance

Create:

```text
src/bralpha/derived/anp/io.py
```

Rules:

- Read from `data/silver/{dataset_id}/`.
- Write to `data/gold/anp/{panel}/`.
- Use partitioned Parquet by year for date-indexed panels.
- Use exact panel primary keys for gold writes, with `augment_source_dataset_key=False`.
- Do not mutate source silver.
- Missing optional inputs should skip panels with clear status in full runs.
- Explicit selected panel with missing required input should raise a clear error.
- Date-bounded reads must prune nested `year=YYYY` partitions.
- For as-of panels, read observation/group history with `start=None, end=end`, then emit rows only for `[start, end]`.

Performance requirements:

- Use Polars-native aggregations for station-level fuel-price microdata.
- Do not create dense station-level daily as-of panels.
- Do not pivot wide.
- Avoid row-wise Python loops over fuel-price microdata.
- Keep model-facing output low-cardinality and long-format.

---

# Pipeline

Add:

```text
src/bralpha/pipelines/anp_research_spine.py
```

CLI:

```bash
python -m bralpha.pipelines.anp_research_spine \
  --repo-root . \
  --start 2010-01-01 \
  --end 2026-01-01
```

Optional repeated flag:

```text
--panel fuel_price_station_observation
--panel fuel_price_group_observation
--panel fuel_sales_observation
--panel fuel_sales_group_observation
--panel oil_gas_production_observation
--panel oil_gas_group_observation
--panel state_asof_daily
--panel daily_long
```

Behavior:

- Read existing ANP silver only.
- Do not run ANP ingestion.
- Do not hit live ANP.
- Do not read source-mapped/deferred ANP datasets.
- Write outputs under `data/gold/anp/`.
- Return/print concise status by panel.
- Skip missing optional inputs in full run.
- Raise on missing required inputs when a panel is explicitly selected.

---

# Documentation

Add:

```text
docs/ANP_RAW_TO_RESEARCH_SPINE.md
```

Document:

1. Input ANP silver datasets.
2. Output ANP gold panels.
3. Strict point-in-time policy.
4. Why ANP has no hourly-to-daily aggregation in this PR.
5. Why station-level fuel-price observations are preserved but not emitted to `daily_long`.
6. Aggregation rules:
   - station prices -> mean by product/geography/date + counts;
   - sales volumes -> sum by product/geography/date + counts;
   - production/gas metrics -> sum by metric/product/location/geography/date + counts.
7. Transformer-aware feature rule.
8. What is intentionally excluded:
   - price changes,
   - spreads,
   - pass-through,
   - inflation proxies,
   - growth rates,
   - shares,
   - ratios,
   - rolling features,
   - z-scores,
   - seasonal anomalies,
   - BOE conversion,
   - flaring shares,
   - Petrobras labels,
   - source-mapped/deferred ANP datasets,
   - portfolio/backtest logic.

---

# Tests

Use synthetic fixtures only. Do not download ANP data.

## IO

- Missing optional inputs skip in full pipeline.
- Explicit selected panel with missing required input raises.
- Outputs write under `data/gold/anp/`.
- Date-bounded reads prune nested `year=YYYY` partitions.
- Gold writes use exact primary keys.
- Source silver is not mutated.

## Fuel-price station observation

- Preserves official sale and purchase prices.
- Preserves null purchase prices.
- Adds missingness flags.
- Does not aggregate station rows in this panel.
- Primary keys are unique.

## Fuel-price group observation

- Builds `all`, `region`, and `state` groups by product.
- Uses mean over non-null sale/purchase prices.
- Emits station and non-null price counts.
- `available_date = max(contributing available_date)`.
- `feature_id` is deterministic.
- `max_groups` raises when exceeded.

Do not add static banned-feature denylist tests.

## Fuel-sales observation and group observation

- Observation panel preserves official monthly state/product sales volume.
- Group panel sums official sales volume by configured group/product.
- Emits count fields.
- `available_date = max(contributing available_date)`.
- Does not compute changes, shares, ratios, or rolling values.

## Oil/gas observation and group observation

- Observation panel preserves metric type/value/unit/resource family.
- Group panel sums official metric values by configured group/location/product/metric_type.
- Emits count fields.
- `available_date = max(contributing available_date)`.
- Does not compute BOE, flaring shares, ratios, or changes.

## State as-of

- Uses latest observation with `observation_available_date <= ref_date`.
- Uses pre-window observation history at output start.
- Does not emit rows before first availability.
- Carries weekly fuel-price state forward with increasing `staleness_days`.
- Carries monthly sales/production state forward with higher `staleness_days`.
- Preserves latest missing official observation rather than falling back to older numeric value.
- `max_features` guard works.

## Daily long

- Includes all source families when inputs exist.
- Drops null value rows.
- Uses long primary key.
- Does not pivot wide.
- Does not include station-level rows.
- Does not include source-mapped/deferred datasets.

Do not add static banned-feature denylist tests.

---

# Acceptance criteria

The PR is complete only when:

1. `docs/ANP_RAW_TO_RESEARCH_SPINE.md` exists.
2. `configs/derived/anp.yaml` exists.
3. Required ANP gold panels are implemented.
4. Outputs go only to `data/gold/anp/{panel}/`.
5. Source-specific ANP silver inputs are not mutated.
6. Every model-usable row has `ref_date` and `available_date`.
7. Station-level fuel-price rows are preserved for audit but excluded from `daily_long`.
8. Aggregates use low-cardinality `all`, `region`, and `state` groups only.
9. As-of panels never use observations with `observation_available_date > ref_date`.
10. Weekly fuel prices and monthly sales/production are carried as latest available state with `staleness_days`.
11. No price changes, spreads, pass-through, inflation proxies, growth rates, shares, ratios, rolling features, z-scores, BOE conversions, flaring shares, stress labels, or portfolio/backtest features are added.
12. No live ANP downloads happen in this pipeline.
13. No new dependencies are added.
14. Tests cover IO, fuel-price station observations, fuel-price aggregates, sales aggregates, production aggregates, as-of state, and daily-long.
15. `python -m pytest` passes.
16. `python -m ruff check .` passes.
17. No generated data is committed.

Keep the implementation structural, daily-aligned, point-in-time safe, low-cardinality, and efficient.


Implement ANP raw-to-research spine #40

Description

Codex Design Contract: ANP Raw-to-Research Spine

Current ANP silver scope

Goal

Core principles

1. Source-specific silver remains immutable

2. Strict point-in-time policy

3. No hourly aggregation is needed for ANP

4. Aggregate station microdata before model-facing panels

5. Transformer-aware feature rule

Required files

Config

Required gold panels

1. fuel_price_station_observation

2. fuel_price_group_observation

3. fuel_sales_observation

4. fuel_sales_group_observation

5. oil_gas_production_observation

6. oil_gas_group_observation

7. anp_state_asof_daily

8. anp_daily_long

IO and performance

Pipeline

Documentation

Tests

IO

Fuel-price station observation

Fuel-price group observation

Fuel-sales observation and group observation

Oil/gas observation and group observation

State as-of

Daily long

Acceptance criteria

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. `fuel_price_station_observation`

2. `fuel_price_group_observation`

3. `fuel_sales_observation`

4. `fuel_sales_group_observation`

5. `oil_gas_production_observation`

6. `oil_gas_group_observation`

7. `anp_state_asof_daily`

8. `anp_daily_long`