Blocker: implement actual feature preprocessing engine from metadata before modeling

[gabrool]

Design contract for Codex to convert configs/modeling/feature_preprocessing.yaml from metadata-only intent into a real train-only preprocessing layer that produces model-ready, tensor-safe features. Scope includes clean preprocessing package structure, rule matching, transforms, train-only winsor/scaler fitting, missing masks, state persistence, validation, docs, and tests. Detailed contract follows in comments.

[gabrool]

Detailed design contract, part 1 — objective and package boundaries

This issue replaces the current metadata-only feature preprocessing contract with a real preprocessing layer. The current contract explicitly says it only records intent and does not transform values, fit scalers, build tensors, or create targets. This issue must change that: after completion, model-default features emitted by the research spines must be transformable into train-only fitted, tensor-safe model inputs.

Do not implement the transformer model here. Do not change target construction except for compatibility checks. Do not add broad new alpha features. This issue is only the model-feature preprocessing layer.

Non-negotiable objective

The preprocessing implementation must be point-in-time safe, train-only fitted, deterministic, auditable, and feature-specific. Every emitted model-facing feature row must match exactly one preprocessing rule. Every model-default row must be transformed according to that rule. No validation/test/live row may influence fitted clipping/scaling statistics.

Required package layout

Create a clean src/bralpha/preprocessing/ package. Keep responsibilities narrow and natural:

• schema.py: dataclasses or Pydantic models for preprocessing rules, selectors, fitted state, feature-index records, and output schemas.
• config.py: load and validate configs/modeling/feature_preprocessing.yaml; compute config hash; validate unique rule IDs and priorities.
• matching.py: selector/rule matching only. It must enforce exactly-one matching and must not use priority to resolve overlaps.
• sources.py: adapters that convert supported research panels into one canonical long feature-observation frame.
• transforms.py: pure stateless value transforms and domain validation.
• winsorization.py: train-only fitting/apply logic for quantile, MAD, hard-clip, and none.
• scalers.py: train-only fitting/apply logic for robust z-score and none.
• state.py: fitted preprocessing state construction, JSON serialization/deserialization, version/hash validation, and feature-index persistence.
• engine.py: orchestration for fit_transform_train() and transform_with_state().
• io.py: read/write preprocessed parquet outputs and state artifacts.
• audit.py: coverage, leakage, missingness, train-stat, and no-NaN checks.

Add one pipeline entry point, for example src/bralpha/pipelines/preprocess_features.py, that wires these modules together. Keep CLI/pipeline code thin; core logic belongs in bralpha.preprocessing.

Documentation

Replace or update docs/FEATURE_PREPROCESSING_METADATA_CONTRACT.md with docs/FEATURE_PREPROCESSING_CONTRACT.md. The new doc must explain both the metadata and the actual fit/apply behavior, including exact transform order, fitted state format, missing-mask semantics, and train-only leakage guarantees.

[gabrool]

Detailed design contract, part 2 — canonical feature-observation frame

Preprocessing must operate on a single canonical long frame, not on ad hoc source-specific frames. Implement adapters in bralpha.preprocessing.sources that convert current research outputs into this shape.

Required canonical columns:

• ref_date: model/as-of date of the feature row.
• available_date: model-usable date; must be <= ref_date for model-ready inputs.
• panel: source panel name, for example b3_feature_daily, continuous_futures_daily, di_curve_grid_daily, or daily_long.
• source_family: row source family when available, for example bcb_sgs_feature, ptax, fred, ons_ear_subsystem, b3_di_curve_feature.
• feature_id: stable source feature/entity identifier.
• value_name: numeric field being preprocessed.
• unit: source unit when present.
• raw_value: original numeric/boolean value before preprocessing.
• source_version and existing PIT columns where present.
• rule_id, semantic_type, raw_unit, stationarity, model_default, and model_default_reason after rule matching.
• feature_key: stable model feature/token key.
• stat_key: key used for fitting winsor/scaler statistics.

Source adapters

Daily-long research families already have source_family, feature_id, value_name, value, and unit; adapt them directly.

B3 B3_FEATURE_DAILY_COLUMNS style panels also adapt directly.

Wide B3 panels must be melted only for columns that are covered by preprocessing rules. At minimum support these panels because rules already exist for them:

• continuous_futures_daily
• di_curve_grid_daily
• di_curve_contract_daily
• index_daily
• listed_market_daily
• index_composition_daily

Use an explicit source-panel registry, preferably configs/modeling/feature_sources.yaml, to declare each wide panel’s primary keys, entity columns, metadata columns, and meltable value columns. Do not infer wide-panel identity from all non-numeric columns silently.

Feature keys and stat keys

feature_key must be stable across runs and unique enough to avoid collisions. Recommended format:

{panel}|{source_family}|{feature_id}|{value_name}|{unit}

For wide B3 panels, feature_id should be built from the panel’s declared entity columns, for example curve ID + tenor for grid rows or continuous ID for futures rows.

Fitted statistics must usually be per feature_key. However, coordinate/control features such as curve tenor and maturity-distance controls must not be fitted per constant entity if that would collapse the feature to all zeros. Add an explicit stat_scope field to preprocessing metadata, defaulting to feature_key, with allowed values:

• feature_key: fit separate stats per model feature series. Default for dynamic prices, returns, rates, liquidity, macro levels, flows, and spreads.
• rule_id: pool all rows matching the rule. Use for coordinate/control features where per-feature scaling would destroy cross-sectional information, such as curve tenors and maturity distances.
• selector_value_name: pool by panel/source family plus value name when several dynamic entities share the same natural unit and the rule explicitly says to do so.

Codex must add explicit stat_scope entries to the metadata where needed rather than relying on hidden heuristics. The first required pooled scopes are curve-tenor and maturity-distance controls.

[gabrool]

Detailed design contract, part 3 — exact transform order and algorithms

The preprocessing order is fixed and must be documented and tested:

1. Rule-match every feature observation.
2. Keep all rows for audit coverage, then filter model_default: true for the first model output.
3. Record raw missingness before any transform.
4. Validate domain constraints from metadata.
5. Apply the stateless transform.
6. Apply hard clipping where configured.
7. Fit train-only winsorization bounds and apply them.
8. Fit train-only scaler state and apply it.
9. Apply missing policy and emit tensor-safe values plus masks.
10. Persist fitted state and feature index.

Domain validation

• positive_required: true: values must be strictly positive before log_positive; invalid values become null and set invalid_mask = 1.
• allow_negative: false: negative values become invalid/null unless the transform explicitly allows them.
• Hard hard_min / hard_max are not a substitute for domain validation; they are clipping rules after validation.
• Do not silently coerce invalid log inputs to zero or epsilon.

Transforms

Implement exactly these transform names from metadata:

• identity: x
• binary: boolean/int/string-safe conversion to 0.0/1.0; null handled by missing policy
• count_log1p: require x >= 0; output log1p(x)
• log_positive: require x > 0; output natural log
• log1p_positive: require x >= 0; output log1p(x)
• signed_log1p: sign(x) * log1p(abs(x))
• percent_to_bp: x * 100.0
• already_log: x
• already_return: x
• clip_only: apply hard clip and no other transform

All transforms must preserve nulls until missing-policy application.

Winsorization

• none: no train-fitted clipping.
• train_quantile: fit lower/upper quantiles on transformed train values, after any hard clip.
• hard_clip_then_train_quantile: hard clip first, then fit train quantiles.
• train_mad: fit median and MAD-based bounds. Interpret winsor_lower and winsor_upper as signed MAD-sigma multiples around the median. Use scaled MAD 1.4826 * median(abs(x - median)); if zero or null, fall back to IQR/1.349, then std, then 1.0.

Winsor bounds are fitted on train rows only for each stat_key.

Scaling

• none: output winsorized/transformed value unchanged.
• train_robust_zscore: fit median and robust scale on train rows only for each stat_key. Use IQR/1.349 first, then scaled MAD, then std, then 1.0. Output (x - median) / scale.

Constant features are allowed but must be audited. Their scale fallback should be 1.0, producing zero-centered values where appropriate. This is another reason coordinate features need pooled stat_scope rather than per-feature stats.

Missing policies and tensor values

Emit both nullable audit values and tensor-safe values:

• processed_value: nullable final transformed/clipped/scaled value.
• tensor_value: finite float used by model tensors; fill missing/invalid processed_value with 0.0.
• observed_mask: 1.0 if processed_value is present and valid, else 0.0.
• raw_missing_mask: 1.0 if raw value was missing, else 0.0.
• invalid_mask: 1.0 if a non-null raw value failed domain/parse validation, else 0.0.

For missing_policy: preserve_null_add_mask, missing values stay null in processed_value and have observed_mask = 0.0 in tensor output.

For missing_policy: false_fill, nulls become 0.0, processed_value = 0.0, and observed_mask = 1.0 because the feature semantics say missing means false. Still retain raw_missing_mask for audit.

[gabrool]

Detailed design contract, part 4 — train-only fitted state and outputs

Fitting API

Implement a pure Python API similar to:

• fit_preprocessing(feature_frame, rules, train_start, train_end, *, model_default_only=True) -> FeaturePreprocessingState
• transform_features(feature_frame, state) -> pl.DataFrame
• fit_transform_train(...) -> tuple[pl.DataFrame, FeaturePreprocessingState]

The fit window must be explicit. The engine must refuse to fit without train_start and train_end. Fitting statistics on the full available history is not allowed.

Rows used for fitting must satisfy:

• train_start <= ref_date <= train_end
• available_date <= ref_date
• exactly one matched rule
• model_default = true, unless a caller is running audit-only mode
• non-null and valid value after domain validation/transform for stats that need numeric data

State artifact

Persist a JSON state artifact, not Python pickle. It must include:

• state_version
• created timestamp
• git/source version if available
• preprocessing config hash
• feature-source config hash if added
• train start/end dates
• decision calendar ID if available
• full matched rule summary by rule_id
• feature_index: stable mapping from feature_key to integer column/token index
• stat_groups: fitted winsor/scaler/domain summary by stat_key
• audit counts: train row count, non-null count, invalid count, missing count, constant-feature flag

transform_with_state must verify the config hash by default. If the config changed, it should fail unless an explicit audit override is passed.

Feature universe and unknown features

The first model’s feature universe must be deterministic. Default behavior:

• Include model_default: true features that appear in the training window and have at least one valid train observation or are binary false-fill features.
• Fail if a model-default feature appears in validation/test/live that is not in the fitted feature_index, unless the caller explicitly runs allow_unseen_features='drop_with_audit'.
• Fail if a model-default feature has zero valid train observations and is not false-fill/binary.
• Do not include model_default: false rows in tensor outputs, but keep them in rule-coverage audits.

Output parquet schemas

Write a preprocessed long frame with at least:

• ref_date
• available_date
• feature_index
• feature_key
• stat_key
• panel
• source_family
• feature_id
• value_name
• unit
• rule_id
• semantic_type
• raw_unit
• stationarity
• raw_value
• transformed_value
• winsorized_value
• processed_value
• tensor_value
• observed_mask
• raw_missing_mask
• invalid_mask
• source_version

A wide/tensor-ready output is also required, but it can remain simple for now:

• data/gold/model_features/preprocessed_long/ for long output.
• data/gold/model_features/tensor_daily/ with one row per ref_date and array/list columns or dense wide columns for tensor_value and observed_mask.
• data/artifacts/preprocessing/{run_id}/state.json and feature_index.parquet.

The tensor output must contain no NaN or infinite values. Null audit values are allowed only in the long audit columns, never in tensor_value.

[gabrool]

Detailed design contract, part 5 — leakage and audit gates

The preprocessing layer must be stricter than the research spines. It is the final gate before modeling.

Required audits

Implement audit functions that fail by default on:

• Any feature observation with no matching preprocessing rule.
• Any feature observation matching more than one rule.
• Any model-default feature excluded because it has no fitted stats.
• Any validation/test/live feature not present in the fitted state, unless explicitly dropped with audit.
• Any row with available_date > ref_date in model-ready input.
• Any fitted statistic using rows outside the train date range.
• Any NaN or infinite tensor_value.
• Any scaler with null/zero scale after all fallbacks.
• Any transform-domain violation rate above a configurable threshold; default threshold is zero for positive-required features.

Coverage reports

Write machine-readable audit reports beside the state artifact:

• rule_coverage.parquet: counts by rule and model-default flag.
• feature_coverage.parquet: counts by feature key, missingness, invalids, train coverage, first/last date.
• stat_groups.parquet: fitted bounds/scales by stat key.
• dropped_features.parquet: any excluded or unseen feature and reason.

No leakage examples that must be tested

• A validation outlier must not change train quantile winsor bounds.
• A test-period level shift must not change robust median/scale.
• Applying the saved state twice to the same data must produce byte-stable outputs except for file metadata.
• Refitting on a longer train window may change state, but applying an old state to old data must remain stable.

Current metadata compatibility

The current YAML must remain the single source of preprocessing intent. Codex may add required fields such as stat_scope, but must not hard-code special feature behavior in Python when it belongs in metadata. Source adapters may know how to reshape panels; feature-specific transform decisions belong in feature_preprocessing.yaml.

Unknown future rules should fail coverage tests until reviewed.

[gabrool]

Detailed design contract, part 6 — required tests

Add focused unit tests and integration tests. Minimum coverage:

1. Config loading rejects duplicate rule IDs, duplicate priorities, unknown transform names, unknown winsor/scaler names, and invalid selector regexes.
2. Rule matching succeeds for exact, regex, source-family, panel, value-name, unit, and feature-id selectors.
3. Rule matching fails on zero matches and multiple matches; priority must not resolve ambiguity.
4. Every current metadata rule is exercised by at least one fixture row or documented as audit-only.
5. model_default: false rows are matched and audited but excluded from first-model tensor output.
6. identity, binary, count_log1p, log_positive, log1p_positive, signed_log1p, percent_to_bp, already_log, already_return, and clip_only are tested.
7. Invalid values for positive/log/count features become null plus invalid_mask = 1; they are not coerced to epsilon or zero.
8. Binary false-fill converts missing to zero and keeps tensor-safe output.
9. Preserve-null missing policy emits tensor_value = 0.0 with observed_mask = 0.0.
10. Hard clipping happens before train quantile winsorization for hard_clip_then_train_quantile.
11. Train quantile winsorization is fitted on train dates only.
12. Train MAD winsorization uses median and scaled MAD, with fallback behavior tested.
13. Robust z-score scaling uses train median and robust scale only.
14. Constant train features get safe scale fallback and are audited.
15. stat_scope = rule_id preserves cross-sectional coordinate information that would be lost under per-feature scaling.
16. Saved state round-trips through JSON and produces identical transformed values after reload.
17. Config hash mismatch fails by default.
18. Unknown validation/live feature fails by default.
19. End-to-end small fixture from daily-long plus B3 wide panels produces a preprocessed long frame and tensor-safe output with no NaN/inf.
20. PIT gate rejects any input row with available_date > ref_date.

Update existing tests that currently assert metadata-only behavior. Keep all existing source/timing tests passing.

[gabrool]

Detailed design contract, part 7 — implementation sequence and definition of done

Suggested stacked PR sequence:

1. Add preprocessing schemas/config loader/matcher and coverage tests.
2. Add source adapters and canonical feature-observation frame tests.
3. Add stateless transforms plus domain/missing-mask tests.
4. Add train-only winsor/scaler fitting, state serialization, and leakage tests.
5. Add pipeline/IO/tensor-safe outputs and end-to-end fixture tests.
6. Update docs and remove wording that says preprocessing is metadata-only.

Definition of done

This issue is complete only when:

• configs/modeling/feature_preprocessing.yaml drives real preprocessing, not just documentation.
• Every emitted feature row from supported panels matches exactly one rule in audit mode.
• The first-model output includes only model_default: true rows.
• All fitted statistics are train-only and persisted in a JSON state artifact.
• Applying a saved state to validation/test/live data does not refit anything.
• Missing and invalid values are represented by tensor-safe values plus masks.
• Tensor output contains no NaN or infinite values.
• Source adapters cover current B3 wide panels, B3 feature daily panels, and all daily-long source families.
• stat_scope is explicit where per-feature scaling would destroy coordinate/control features.
• Docs explain the actual preprocessing contract, state artifact, and output schemas.
• All required tests pass.

Validation expected in PR:

• python -m ruff check .
• python -m pytest

Do not close this issue with only metadata, docs, or partial transforms. The output must be usable as the first real model input layer.