Feature upgrades: deterministic source-specific and RV features with preprocessing metadata

[gabrool]

Goal

Add the deterministic feature upgrades needed before the preprocessing/tensor layer. These are features that are useful for Brazil low-frequency relative value and should not be left for a transformer to rediscover from raw levels alone.

This issue must also update preprocessing metadata for every new feature and demote superseded/useless raw features from first-model defaults.

Scope

In scope:

• Add deterministic feature panels for currently implemented source families.
• Add cross-market/RV features that are central to Brazil rates/FX/equity RV.
• Add preprocessing metadata rules for every new feature.
• Mark superseded raw features model_default: false in configs/modeling/feature_preprocessing.yaml.
• Add PIT-safe warmup/history handling for rolling features.
• Add formula and metadata tests.

Out of scope:

• No fitted scalers or actual preprocessing execution.
• No tensors or model training.
• No new external data sources.
• No new raw ingestion endpoints.
• No target/label changes except using already available feature dates for tests if needed.

Global implementation requirements

1. All new feature builders must emit daily-long-compatible rows:

ref_date
available_date
source_family
feature_id
value_name
value
unit
observation_ref_date
observation_available_date
is_available
staleness_days
source_version

If the source family carries PIT fields, preserve them too:

availability_policy
availability_basis
revision_policy
vintage_id
model_usable
model_usable_reason

2. Features must be computed as of ref_date only. No rolling/statistical feature may use observations with observation_available_date > ref_date.

3. Rolling features must receive enough warmup history before the requested output window. Use the model contract defaults:

lookback_business_days = 256
feature_warmup_business_days = 504

Feature builders may accept extended history and then filter output back to [start, end]. Add tests proving first output dates can use pre-window history.

4. Use these shared naming conventions:

source_family = <source>_<feature_group>_feature
feature_id = stable family/entity key
value_name = stable metric name

5. Do not delete raw/research panels. “Remove useless features” means: update preprocessing metadata so superseded/noisy/raw features are model_default: false, and ensure new upgraded feature panels are the first-model defaults.

6. Add configs/derived/model_feature_upgrades.yaml if useful for enabling/disabling groups, but Codex must not move feature selection decisions into vague config defaults. The feature list below is mandatory.

Stack plan

Implement as stacked PRs:

1. Feature-upgrade foundation and B3 market/rates features.
2. BCB, FRED, Tesouro, and Brazil/global RV features.
3. IBGE and Focus macro-release features.
4. ANP, ONS, CVM, Novo CAGED, Receita features.
5. Metadata cleanup/removals and full coverage tests.

Each PR must run:

python -m ruff check .
python -m pytest

Acceptance criteria

This issue is complete when:

1. Every feature listed in the follow-up comments exists and has tests.
2. Every new feature has exactly one preprocessing metadata rule.
3. Superseded/noisy raw features are marked model_default: false.
4. Rolling features are PIT-safe and use warmup history.
5. Daily-long upgraded feature panels contain no future leakage and preserve availability metadata.

[gabrool]

Required features: B3 rates, futures, indexes, composition

Shared formulas

Use these conventions:

log_return_nbd = log(value_t / value_t-n) if both values > 0
pct_change_nbd = 100 * (value_t / value_t-n - 1) if denominator != 0
bp_change_nbd = value_bp_t - value_bp_t-n
realized_vol_nbd_ann = std(log_return_1bd over last n observations) * sqrt(252)

Use observation sequence order within each feature/entity, not calendar-day differences. Null if insufficient history.

B3 DI curve features

Add src/bralpha/derived/b3/di_features.py and a panel, e.g. b3_di_feature_daily.

Inputs: di_curve_grid_daily with curve_id, tenor_business_days, implied_annual_rate_bp, log_discount_factor, is_interpolated, is_extrapolated.

Per-tenor feature_id:

b3_di_curve:{curve_id}:{tenor_business_days}bd

Emit these value_names for every tenor available:

rate_level_bp
rate_change_1bd_bp
rate_change_5bd_bp
rate_change_21bd_bp
log_discount_factor
log_df_change_1bd
log_df_change_5bd
log_df_change_21bd
rolldown_1bd_bp
rolldown_5bd_bp
rolldown_21bd_bp
is_interpolated
is_extrapolated

Formulas:

• rate_level_bp = implied_annual_rate_bp.
• rate_change_* = current rate_level_bp - lagged rate_level_bp.
• log_df_change_* = current log_discount_factor - lagged log_discount_factor.
• rolldown_hbd_bp = interpolated_rate_at_tenor(T - h) - rate_at_T using the same day’s curve and log-DF interpolation. Null when T - h <= 0 or no bracket exists.

Cross-tenor feature_id:

b3_di_curve:{curve_id}:shape

Emit:

slope_21_252_bp       = rate_252 - rate_21
slope_63_252_bp       = rate_252 - rate_63
slope_252_504_bp      = rate_504 - rate_252
slope_504_1260_bp     = rate_1260 - rate_504
butterfly_63_252_504_bp    = 2 * rate_252 - rate_63 - rate_504
butterfly_252_504_1260_bp  = 2 * rate_504 - rate_252 - rate_1260
forward_21_63_bp
forward_63_126_bp
forward_126_252_bp
forward_252_504_bp
forward_504_756_bp
forward_756_1260_bp

Forward formula from log discount factors:

forward_rate = exp((log_df_T1 - log_df_T2) * 252 / (T2 - T1)) - 1
forward_bp = forward_rate * 10_000

Use only brackets that exist. Null if either tenor is unavailable or T2 <= T1.

Preprocessing metadata for DI features:

• rate_level_bp, slopes, butterflies, forwards, rolldowns, rate changes: identity, stationaryish, train MAD or train quantile robust scaling. Use train MAD 10 for levels/slopes/forwards; train quantile 0.001/0.999 for changes.
• log_discount_factor: already_log, robust scale.
• log_df_change_*: already_return, robust scale.
• flags: binary rule.

Demote old DI raw/grid defaults:

• Set model_default: false for di_curve_grid_daily and di_curve_contract_daily canonical level rows now superseded by b3_di_curve_feature, except keep raw panels for audit.

B3 continuous futures features

Add src/bralpha/derived/b3/futures_features.py, source family b3_futures_feature.

Input: continuous_futures_daily.

Feature ID:

b3_futures:{continuous_id}

Emit:

log_settlement
log_return_1bd
log_return_5bd
log_return_21bd
same_contract_log_return_1bd
realized_vol_5bd_ann
realized_vol_21bd_ann
realized_vol_63bd_ann
volume_log1p
open_interest_log1p
volume_open_interest_ratio
roll_gap_pct
is_roll_date
is_tradeable
business_days_to_maturity

Formulas:

• log_settlement = log(settlement) when settlement > 0, else null.
• log_return_* from settlement.
• same_contract_log_return_1bd: use lag only if current and previous selected contract IDs are the same; otherwise null.
• realized_vol_* uses 1bd log returns.
• volume_log1p = log1p(volume); same for OI.
• volume_open_interest_ratio = volume / open_interest if OI > 0.
• roll_gap_pct = 100 * (quote_pct_change_1d - same_contract_quote_pct_change_1d) if both exist; else null.

Preprocessing:

• log levels: already_log.
• returns and roll gap: already_return or identity for percent roll gap; train quantile robust scale.
• realized vol and ratios: log1p_positive, train quantile robust scale.
• log1p liquidity features: already_log, robust scale.
• flags: binary.
• maturity distance: bounded 0..3000, robust scale.

Demote old continuous futures raw values in metadata once these features exist: settlement, raw quote diff/pct change, raw volume/OI should be model_default: false in favor of b3_futures_feature.

B3 index features

Add source family b3_index_feature from index_daily.

Feature ID:

b3_index:{index_id}

Emit:

log_close
log_return_1bd
log_return_5bd
log_return_21bd
realized_vol_21bd_ann
realized_vol_63bd_ann
intraday_range_log      = log(high / low) if high > 0 and low > 0
close_drawdown_252bd_pct = 100 * (close / trailing_252bd_max_close - 1)
volume_log1p
financial_volume_log1p
number_of_trades_log1p

Preprocessing:

• log close: already_log.
• returns/range/drawdown: return-style train quantile robust scale.
• volumes/counts: already_log or count/log rule as appropriate.

Demote raw index_daily price/liquidity model defaults after this panel exists.

B3 index composition features

Add source family b3_index_composition_feature from index_composition_daily.

Feature ID:

b3_index_composition:{index_id}

Emit:

constituent_count
weight_top1_pct
top5_weight_pct
top10_weight_pct
hhi_weight
effective_constituents

Formulas:

• Sort weights descending per index/ref_date.
• If weights are fractions <= 1, convert to percent internally for top weights. If max weight > 1, treat as percent.
• hhi_weight = sum(weight_fraction^2).
• effective_constituents = 1 / hhi_weight if HHI > 0.

Preprocessing:

• counts/effective constituents: count/log1p robust scale.
• top weights: bounded percent 0..100 robust scale.
• HHI: bounded 0..1 robust scale.

Demote raw theoretical_quantity; keep raw weight non-default if this aggregate composition feature exists.

[gabrool]

Required features: BCB, FRED, Tesouro, Brazil/global RV

BCB SGS upgraded features

Extend current bcb_sgs_feature or add bcb_policy_reserves_feature if cleaner.

Add:

real_policy_rate_12m_ipca_bp = selic_target_level_pa * 100 - ipca_12m_sum_pct * 100
real_policy_rate_3m_ann_ipca_bp = selic_target_level_pa * 100 - ipca_3m_ann_pct * 100
reserves_log_change_21bd
reserves_log_change_63bd
reserves_drawdown_from_504bd_high_pct

Preprocessing:

• real policy rates: bp level, identity, train MAD 10, robust scale.
• reserves log changes: already_return, train quantile 0.001/0.999, robust scale.
• reserves drawdown: percent drawdown, identity, train quantile 0.001/0.999, robust scale.

BCB PTAX features

Add source family bcb_ptax_feature from ptax_selected_daily.

Feature ID:

bcb_ptax:{currency_code}

Emit:

mid_rate = (bid_rate + ask_rate) / 2
log_mid_rate
log_return_1bd
log_return_5bd
log_return_21bd
realized_vol_21bd_ann
bid_ask_spread_bp = 10_000 * (ask_rate - bid_rate) / mid_rate
mid_parity = (bid_parity + ask_parity) / 2 if both exist
log_mid_parity
parity_bid_ask_spread_bp = 10_000 * (ask_parity - bid_parity) / mid_parity

Null if denominators are non-positive or missing.

Preprocessing:

• log levels: already_log.
• returns/spreads: return-style/identity with train quantile robust scale.
• realized vol: log1p_positive + robust scale.

Demote raw ptax bid/ask/parity rows from first-model defaults once these exist.

BCB Focus upgraded features

Add source family bcb_focus_feature from focus_expectation_asof_daily.

Feature ID:

bcb_focus:{expectation_key}

Emit:

median_level
mean_level
std_dev_log1p
respondents_log1p
median_revision_1bd
median_revision_5bd
median_revision_21bd
mean_revision_1bd
mean_revision_5bd
mean_revision_21bd
dispersion_to_abs_median = std_dev / max(abs(median), 1.0)

Top5 spread features:

For rows that differ only by is_top5, create a matched key with is_top5 removed and emit:

top5_minus_general_median
top5_minus_general_mean

Only emit when both top5 and general are available as-of the same ref_date for the same endpoint, indicator, detail, reference period, meeting, calculation type, and base calculation.

Preprocessing:

• levels: identity, train quantile robust scale.
• revisions and top5 spreads: identity, train quantile robust scale.
• std_dev_log1p and respondents_log1p: already_log, robust scale.
• dispersion ratio: log1p_positive, train quantile robust scale.

Demote raw Focus mean, median, std_dev, respondents to model_default: false once this feature panel exists. min_value/max_value remain false.

FRED upgraded features

Add src/bralpha/derived/fred/features.py, source families:

fred_rate_feature
fred_market_feature

Rate feature IDs:

fred_rate:{series_id_lower}
fred_rate:curve

For each rate/spread series in DGS2,DGS5,DGS10,DGS30,DFF,SOFR,DFEDTARU,DFEDTARL,DFII5,DFII10,DFII30,T5YIE,T10YIE,BAA10Y,AAA10Y,DBAA,DAAA emit:

level_bp = value_percent * 100
change_1bd_bp
change_5bd_bp
change_21bd_bp

Curve/spread features:

ust_10y_2y_slope_bp = DGS10*100 - DGS2*100
ust_30y_5y_slope_bp = DGS30*100 - DGS5*100
ust_10y_real_rate_bp = DFII10*100
ust_10y_breakeven_bp = T10YIE*100
baa_credit_spread_bp = BAA10Y*100
aaa_credit_spread_bp = AAA10Y*100
fed_target_mid_bp = ((DFEDTARU + DFEDTARL) / 2) * 100

Also emit 1/5/21bd changes for the curve/spread features above.

Market feature IDs:

fred_market:{series_id_lower}

For DTWEXBGS,DTWEXAFEGS,DTWEXEMEGS,DEXCHUS,SP500,NASDAQCOM,DCOILWTICO,DCOILBRENTEU,PCOPPUSDM emit:

log_level
log_return_1bd
log_return_5bd
log_return_21bd
realized_vol_21bd_ann

Use signed-safe log for oil series if non-positive; otherwise positive log.

For VIXCLS emit:

log1p_level
change_1bd
change_5bd
change_21bd

Preprocessing:

• bp levels/spreads: identity + train MAD 10 + robust scale.
• bp changes: identity + train quantile + robust scale.
• log levels: already_log + robust scale.
• log returns: already_return + train quantile + robust scale.
• realized vol/VIX log level: log/positive-level robust scale.

Demote raw FRED source_family=fred, value_name=value from first-model defaults once upgraded features exist.

Tesouro upgraded features

Add source family tesouro_feature.

From tesouro_direto_prices_rates by security feature key:

mid_rate_pct = (buy_rate + sell_rate) / 2
mid_rate_bp = mid_rate_pct * 100
rate_bid_ask_spread_bp = abs(sell_rate - buy_rate) * 100
mid_price = (buy_price + sell_price) / 2
price_bid_ask_spread_pct = 100 * abs(sell_price - buy_price) / mid_price
mid_rate_change_1bd_bp
mid_rate_change_5bd_bp
mid_rate_change_21bd_bp
mid_price_log_return_1bd
mid_price_log_return_5bd
mid_price_log_return_21bd

From tesouro_direto_flows:

Aggregate by date and security/type where possible. Use operation_type if present; otherwise parse sale/redemption from feature ID.

Emit:

sales_value_brl
redemptions_value_brl
net_flow_value_brl = sales - redemptions
gross_flow_value_brl = sales + redemptions
redemption_share_pct = 100 * redemptions / gross_flow_value_brl
sales_quantity
redemptions_quantity
net_quantity
investor_count

Join to tesouro_direto_stock by security type/bucket when possible and emit:

net_flow_to_stock_value_pct = 100 * net_flow_value_brl / stock_value

From tesouro_direto_stock and tesouro_dpf_stock:

stock_value_log
stock_value_change_21bd_pct
investor_count_log1p
quantity_log1p

Preprocessing:

• rates/spreads: bp identity + robust scale.
• price log returns: already_return.
• BRL flows: signed_log1p.
• gross/sales/redemptions/stock: log1p_positive or log_positive as applicable.
• shares/ratios: bounded percent when 0..100, otherwise train quantile robust scale.

Demote raw Tesouro daily-long rows from first-model defaults once tesouro_feature exists.

[gabrool]

Required features: cross-market Brazil RV features

Add source family br_rv_cross_feature. Build these only from same-date upgraded/as-of feature panels. If any required input is unavailable for a date, emit null.

Feature IDs and value names:

• br_rv_cross:policy, brl_policy_carry_vs_fed_mid_bp = selic_target_level_bp - fed_target_mid_bp
• br_rv_cross:policy, brl_policy_carry_vs_fed_funds_bp = selic_target_level_bp - fred_rate:dff level_bp
• br_rv_cross:rates, brl_di_2y_minus_ust_2y_bp = b3 DI 504bd rate_level_bp - DGS2 level_bp
• br_rv_cross:rates, brl_di_5y_minus_ust_5y_bp = b3 DI 1260bd rate_level_bp - DGS5 level_bp
• br_rv_cross:rates, brl_di_2y_real_minus_us_2y_proxy_bp = b3 DI 504bd rate_level_bp - ipca_12m_sum_pct * 100 - DGS2 level_bp
• br_rv_cross:fx, brl_fx_idiosyncratic_return_5bd = USD/BRL log_return_5bd - DTWEXEMEGS log_return_5bd
• br_rv_cross:fx, brl_fx_idiosyncratic_return_21bd = USD/BRL log_return_21bd - DTWEXEMEGS log_return_21bd
• br_rv_cross:equity, ibov_sp500_relative_return_21bd = B3 Ibovespa log_return_21bd - SP500 log_return_21bd

Metadata:

• Carry and DI/UST spreads: identity, bp units, train MAD 10, robust scale.
• Relative FX/equity returns: already_return, train quantile 0.001/0.999, robust scale.

Availability:

• available_date = ref_date.
• observation_available_date = max(input observation_available_date) where available.
• staleness_days = max(input staleness_days).

Tests:

• Synthetic same-date inputs produce exact outputs.
• Missing inputs produce nulls.
• Features never appear before all contributing inputs are available.

[gabrool]

Required features: IBGE macro-release features

Add source family ibge_sidra_feature from sidra_asof_daily. Compute by feature_id time series using observation dates, not repeated daily rows. Use only observations available as of each ref_date.

Feature ID:

ibge_sidra_feature:{original_feature_id}

Inflation datasets

For feature IDs containing ibge_sidra:(ipca|ipca15|inpc): emit:

monthly_pct
change_1obs_pp
trailing_3obs_sum_pct
trailing_12obs_sum_pct
trailing_3obs_ann_pct = ((1 + trailing_3obs_sum_pct / 100) ** 4 - 1) * 100

Null rolling features until enough observations exist.

Metadata:

• monthly/trailing percent: identity, train quantile 0.001/0.999, robust scale.
• changes: identity, train quantile 0.001/0.999, robust scale.

Activity and cost index datasets

For feature IDs containing any of:

gdp_volume_index
pim_industrial_production
pmc_retail_index
pmc_broad_retail_index
pmc_retail_by_activity
pmc_broad_retail_by_activity
pms_services_index
pms_services_segments
pms_services_activities
sinapi_cost_m2

Emit:

log_level
log_change_1obs
log_change_3obs
yoy_log_change

Use 12 observations for monthly/moving-quarter series and 4 observations for quarterly GDP series. Determine frequency from SIDRA config metadata.

Metadata:

• log_level: already_log, robust scale.
• log changes: already_return, train quantile 0.001/0.999, robust scale.

GDP volume change dataset

For gdp_volume_change emit:

level_pct
change_1obs_pp
yoy_change_pp

Use 4 observations for yoy.

Metadata: identity, train quantile robust scale.

GDP current values

For gdp_current_values emit:

signed_log_level
signed_log_change_1obs
yoy_signed_log_change

Metadata:

• signed log level: already_log if emitted as signed-log value, robust scale.
• changes: already_return, robust scale.

PNAD labor rate datasets

For:

pnad_unemployment_rate
pnad_participation_rate
pnad_underutilization_rate

Emit:

level_pct
change_1obs_pp
yoy_change_pp

Metadata: bounded percent 0..100, robust scale for level; changes use train quantile robust scale.

PNAD labor level/income datasets

For:

pnad_occupation_level
pnad_underutilization_level
pnad_real_income
pnad_real_income_usual
pnad_real_income_mass
pnad_real_income_mass_usual

Emit:

log_level
log_change_1obs
yoy_log_change

Metadata: log level already_log; log changes already_return.

Required tests

• Monthly IPCA synthetic data gives exact 3obs, 12obs, and annualized values.
• Quarterly GDP uses 4 observations for yoy.
• Monthly activity index uses 12 observations for yoy.
• First output date can use pre-window history.
• No feature uses an unavailable observation.

[gabrool]

Required features: ANP, ONS, CVM, Novo CAGED, Receita

ANP features

Add source family anp_fuel_feature from ANP daily-long/as-of rows.

Fuel prices by product/region/state feature key:

sale_price_log
purchase_price_log
sale_price_log_change_1obs
sale_price_log_change_4obs
sale_purchase_spread_pct = 100 * (sale_price - purchase_price) / sale_price
station_count_log1p

Cross-product parity/spreads at same geography/date:

ethanol_gasoline_parity = ethanol_sale_price / gasoline_sale_price
diesel_gasoline_spread_pct = 100 * (diesel_sale_price / gasoline_sale_price - 1)

Fuel sales:

sales_volume_log
sales_volume_yoy_log_change
sales_volume_3obs_sum_log
state_count_log1p

Oil/gas production:

production_log
production_yoy_log_change
production_3obs_sum_log
state_count_log1p

Metadata:

• log levels: already_log.
• log changes: already_return.
• parity/spread: identity, train quantile robust scale.
• counts: already_log or count_log1p rule.

ONS features

Add source family ons_power_feature.

Hydro/EAR and ENA:

stored_energy_percent_level
stored_energy_percent_change_5bd
stored_energy_percent_change_21bd
stored_energy_percent_seasonal_z
stored_energy_mwmes_log
ena_percent_mlt_level
ena_percent_mlt_change_5bd
ena_percent_mlt_seasonal_z

Seasonal z-score rule:

• Compute using only prior observations available before ref_date.
• Group by subsystem and calendar month.
• Use expanding historical mean/std for that month with at least 24 prior monthly/daily observations; otherwise null.

CMO/load/energy balance:

cmo_log1p
cmo_log_change_1obs
load_log
load_deviation_21bd_pct = 100 * (load / trailing_21bd_mean_load - 1)
hydro_generation_share_pct
thermal_generation_share_pct
wind_generation_share_pct
solar_generation_share_pct
interchange_to_load_pct
hour_count_log1p

Generation share denominator:

total_generation = hydro + thermal + wind + solar + other

Null shares if denominator <= 0.

Metadata:

• shares and percent levels: bounded percent rules, hard clip sensible ranges; generation shares 0..100, load deviations -100..300.
• log levels: already_log or log1p_positive.
• changes: already_return/identity, train quantile robust scale.
• seasonal z: identity, train quantile robust scale.

CVM features

Add source family cvm_fund_feature from CVM fund group observations/state.

Emit by group feature key:

subscriptions_log
redemptions_log
net_flow_brl = subscriptions - redemptions
gross_flow_brl = subscriptions + redemptions
net_flow_to_nav_pct = 100 * net_flow_brl / nav
gross_flow_to_nav_pct = 100 * gross_flow_brl / nav
redemption_share_pct = 100 * redemptions / gross_flow_brl
nav_log
portfolio_value_log
shareholder_count_log1p
fund_count_log1p
shareholder_count_change_21bd_pct

Null ratios when denominator <= 0.

Metadata:

• subscriptions/redemptions/gross/nav/portfolio: already_log if emitted as log, otherwise log1p_positive.
• net flow: signed_log1p.
• ratios/shares: identity, train quantile robust scale; bounded 0..100 only for redemption share.
• counts: count/log1p.

Novo CAGED features

Add source family novo_caged_feature from movement group observations.

The current grouped panel has movement_sign; build combined features for each group by matching admission and dismissal signs.

Emit:

admissions_count
dismissals_count
net_jobs = admissions_count - dismissals_count
gross_jobs = admissions_count + dismissals_count
admission_dismissal_ratio = admissions_count / dismissals_count
net_jobs_yoy_change
net_jobs_to_gross_pct = 100 * net_jobs / gross_jobs
wage_mean_log
wage_mean_yoy_log_change
contract_hours_mean
state_positive_diffusion_share_pct
sector_positive_diffusion_share_pct

Diffusion shares:

• For state diffusion, across state groups in the same ref_date, share with net_jobs > 0.
• For sector diffusion, across cnae_section groups in the same ref_date, share with net_jobs > 0.

Metadata:

• counts/gross: count_log1p.
• net jobs: signed_log1p.
• ratios/shares: identity + train quantile robust scale; diffusion shares bounded 0..100.
• wage log: already_log.
• contract hours: bounded 0..80.

Receita features

Add source family receita_feature from Receita feature observations/as-of rows.

Emit by tax collection feature key:

collection_signed_log
collection_yoy_pct = 100 * (value / value_lag_12obs - 1)
collection_3obs_sum_signed_log
collection_12obs_sum_signed_log
category_share_pct = 100 * value / total_collection_same_ref_date

If BCB/IBGE IPCA 12m is available on the same date, also emit:

real_collection_yoy_pct = collection_yoy_pct - ipca_12m_sum_pct

Null if denominator is zero/missing or if no lag exists.

Metadata:

• signed log levels/sums: already_log if emitted signed-log, otherwise signed_log1p.
• yoy and real yoy: identity + train quantile robust scale.
• category share: bounded percent 0..100 robust scale.

[gabrool]

Metadata updates, model-default cleanup, and tests

Preprocessing metadata for new features

Every new source family must have explicit rules in configs/modeling/feature_preprocessing.yaml. Do not add a catch-all.

Required new source-family rule groups:

b3_di_curve_feature
b3_futures_feature
b3_index_feature
b3_index_composition_feature
bcb_ptax_feature
bcb_focus_feature
fred_rate_feature
fred_market_feature
tesouro_feature
br_rv_cross_feature
ibge_sidra_feature
anp_fuel_feature
ons_power_feature
cvm_fund_feature
novo_caged_feature
receita_feature

If a feature is emitted as a log, use already_log. If it is emitted as a log return, use already_return. Do not apply log transforms twice.

Superseded model-default cleanup

Update existing metadata so these existing raw rows are model_default: false after upgraded replacements exist:

continuous_futures_daily raw settlement / raw one-day quote changes / raw volume / raw open_interest
index_daily raw price and liquidity rows
di_curve_grid_daily raw canonical rows superseded by b3_di_curve_feature
di_curve_contract_daily raw canonical rows superseded by b3_di_curve_feature
ptax raw bid/ask/parity rows
focus raw mean/median/std_dev/respondents rows once bcb_focus_feature exists
fred raw value rows
ibge_sidra raw value rows
tesouro_direto_prices_rates raw rows
tesouro_direto_flows raw rows
tesouro_direto_stock raw rows
tesouro_dpf_stock raw rows
anp raw daily-long rows
ons raw daily-long rows
cvm raw daily-long rows
novo_caged_movements raw daily-long rows
receita_tax_collection raw daily-long rows

Keep these already-false decisions:

listed_market_daily individual-security price/liquidity rows
Focus min_value and max_value
raw BCB SGS rows with engineered replacements
redundant DI compatibility columns

Do not remove research outputs. The later first-model assembly should prefer upgraded feature families via model_default metadata.

Required tests

For each new feature builder:

1. Formula tests with small synthetic data.
2. PIT tests proving no future observation is used.
3. Warmup tests proving pre-window history is used and output is filtered back to [start, end].
4. Missing-input tests proving null output rather than accidental stale output.
5. Metadata tests proving every new feature row matches exactly one preprocessing rule.
6. Superseded raw-feature tests proving the old raw rows are model_default: false.

Specific required tests:

• B3 DI forward formula from two log discount factors.
• B3 DI rolldown uses same-day curve only.
• B3 futures realized volatility uses only lagged returns.
• PTAX bid/ask spread and log return formulas.
• Focus revisions and Top5-minus-general matching.
• FRED UST slopes and rate changes.
• Tesouro net flow and flow-to-stock ratio.
• IBGE monthly 12-observation yoy and quarterly 4-observation yoy.
• ANP ethanol/gasoline parity and monthly yoy.
• ONS generation shares and seasonal z-score using only prior years.
• CVM net-flow-to-NAV ratios.
• Novo CAGED net jobs and diffusion shares.
• Receita category share and real yoy when IPCA is available.
• Cross-market DI/UST spread and BRL versus dollar-index return difference.

Documentation

Add or update:

docs/FEATURE_UPGRADES_CONTRACT.md
docs/FEATURE_PREPROCESSING_METADATA_CONTRACT.md

The feature-upgrades doc must explain formulas, PIT/warmup policy, canonical upgraded feature families, demoted raw feature groups, and that values remain unpreprocessed until the later preprocessing layer.