Scope 14: Per-Column Strategy Customisability — per_column_strategy and per_column_constant_fill

[DEVunderdog]

Problem Statement

When a data scientist uses DataForgeML to impute a dataset, the automated routing engine selects imputation strategies based on statistical signals — missingness severity, mechanism, distribution shape, and dataset size. This works well for the majority of columns, but fails when the user has domain knowledge the engine cannot derive.

The only current override path is mnar_columns, which forces MNAR semantics: a data-derived mean/median fill plus a binary missingness indicator. This is semantically wrong for three documented cases:

1. A column classified as MCAR Minor might be known to follow a strong pattern predictable from other columns — the user wants Regression rather than Mean, but mnar_columns gives them a scalar fill with an indicator they do not need.
2. A column with High severity might be known to be truly random (e.g. a sensor dropout with no correlation to data values) — the user wants Median rather than MICE/KNN, but there is no clean override path.
3. A column where null semantically means zero (e.g. a transaction count column where missing means no transactions occurred) — the user wants Constant(0), but mnar_columns would fill with the observed mean of non-missing rows, which is actively wrong.

There is also no defined insertion point in the routing priority chain for a user override, making future extension of the chain fragile.

Solution

Add two new fields to NumericImputationConfig:

• per_column_strategy: dict[str, ImputationStrategy] — an explicit per-column strategy assignment that fires at Priority 1.5 in the routing chain, after DropCandidate (which remains a hard gate) but before MNAR routing, bypassing all routing priorities 2–7.
• per_column_constant_fill: dict[str, float] — the companion fill value required when a column is declared Constant in per_column_strategy.

The user declares overrides directly on the config:

config.numeric.per_column_strategy = {
    "transaction_count": ImputationStrategy.Constant,
    "sensor_reading":    ImputationStrategy.Median,
    "income":            ImputationStrategy.Regression,
}
config.numeric.per_column_constant_fill = {
    "transaction_count": 0.0,
}

ColumnImputationRecord.signals records "per_column_strategy_override: user forced strategy=X" for every overridden column, preserving full audit visibility.

Construction-time validation catches contradictions before any data is touched. Fit-time validation catches size-guard violations before any model is trained.

User Stories

1. As a data scientist, I want to force Regression imputation on a specific column that the engine routes to Mean, so that I can apply my domain knowledge that the column is strongly predictable from other features.
2. As a data scientist, I want to force Median imputation on a High-severity column that the engine routes to MICE, so that I can override the model-based strategy when I know the missingness is truly random.
3. As a data scientist, I want to fill a transaction-count column with Constant(0) when null means no transactions occurred, so that imputed values reflect the correct domain interpretation rather than the column mean.
4. As a data scientist, I want to force KNN on a column the engine routes to Mean, so that I can leverage neighbour-based imputation when I know similar rows exist in my dataset.
5. As a data scientist, I want to force MICE on a column the engine routes to Median, so that I can use joint multi-column imputation when I know the missingness is jointly structured.
6. As a data scientist, I want to force Mode on a Continuous numeric column, so that I can handle a domain-specific column where the mode is the most appropriate fill.
7. As a data scientist, I want the override to fire after the DropCandidate gate, so that the 50%-missing floor remains a reliable library guarantee and is not accidentally bypassed.
8. As a data scientist, I want a ValueError at config construction time if I declare a column in both mnar_columns and per_column_strategy, so that contradictory declarations are caught before any data is touched.
9. As a data scientist, I want a ValueError at config construction time if I declare Constant strategy for a column without providing a fill value in per_column_constant_fill, so that the required domain knowledge is never silently defaulted.
10. As a data scientist, I want a ValueError at config construction time if I use Passthrough, Indicator, MNAR, or Dropped in per_column_strategy, so that internal-only and semantically-reserved strategies are clearly separated from user-facing ones.
11. As a data scientist, I want a ValueError at fit time if I force a model-based strategy (KNN, Regression, MICE) but my dataset does not meet the size guard for that strategy, so that I am told exactly what to change rather than silently receiving a Median fill.
12. As a data scientist, I want ColumnImputationRecord.signals to record "per_column_strategy_override: user forced strategy=X" when an override fires, so that I can audit which columns were routed by user intent rather than the automatic engine.
13. As a data scientist, I want per_column_strategy to appear in the to_dict / from_dict round-trip of NumericImputationConfig, so that overrides survive serialisation and can be stored alongside a fitted imputer.
14. As a data scientist, I want per_column_constant_fill to appear in the to_dict / from_dict round-trip of NumericImputationConfig, so that the fill value survives serialisation alongside the strategy declaration.
15. As a data scientist, I want add_indicator_columns to still apply independently when per_column_strategy also overrides a column, so that indicator and strategy decisions remain orthogonal and I do not need to re-declare the column to get both.
16. As a data scientist, I want suggest_refit_config (Scope 3) to skip MNAR-declared columns when it populates per_column_strategy, so that the auto-generated refit config never produces a construction-time conflict error.
17. As a data scientist, I want the ValueError for Dropped in per_column_strategy to name the correct alternative (PipelineConfig.exclude_columns), so that I know where to go when I want to force-remove a column.
18. As a data scientist, I want the ValueError for MNAR in per_column_strategy to name the correct alternative (mnar_columns), so that I know where to go when I want MNAR semantics.
19. As a data scientist, I want the ValueError for size-guard violations to name the specific guard, the current dataset values, and the configured threshold, so that I know exactly what to change.
20. As a data scientist, I want the strategy recorded in ColumnImputationRecord to always match what per_column_strategy declared, so that the audit trail never contradicts my config.

Implementation Decisions

1. New fields on NumericImputationConfig

Two new fields are added:

• per_column_strategy: dict[str, ImputationStrategy] — defaults to empty dict. Maps column name → forced strategy.
• per_column_constant_fill: dict[str, float] — defaults to empty dict. Maps column name → constant fill value. Required when any column is declared Constant in per_column_strategy.

Both fields participate in to_dict / from_dict serialisation.

2. Construction-time validation (__post_init__)

NumericImputationConfig.__post_init__ enforces four rules, raising ValueError immediately:

• Invalid strategy: any column in per_column_strategy with value Passthrough, Indicator, Dropped, or MNAR → ValueError naming the blocked strategy and the correct alternative.
• Missing Constant fill: any column with per_column_strategy[col] == Constant that has no entry in per_column_constant_fill → ValueError.
• MNAR conflict: ImputationConfig.__post_init__ (the parent config) raises ValueError if any column appears in both mnar_columns and per_column_strategy. This validation lives one level up because it requires visibility of both fields simultaneously.

3. Override priority: Priority 1.5 in _fit_one

per_column_strategy fires at Priority 1.5 — after the DropCandidate check (Priority 1) and before MNAR routing (Priority 2). If the column is in per_column_strategy, the declared strategy is used directly and all routing priorities 2–7 are bypassed. The signal "per_column_strategy_override: user forced strategy=X" is appended to ColumnImputationRecord.signals.

For Constant strategy, the fill value is read from per_column_constant_fill[col].

4. Fit-time size-guard validation

When per_column_strategy forces a model-based strategy (KNN, Regression, MICE), the relevant size guard is checked at the start of NumericImputer.fit() before any model is trained:

• Regression: n_rows >= regression_min_rows, else ValueError.
• KNN: n_rows <= knn_max_rows AND n_features <= knn_max_features, else ValueError.
• MICE: same size guards as the routing engine's MICE path.

The ValueError message names the column, the declared strategy, the specific guard that was not met, the current dataset value, and the configured threshold.

5. ImputationConfig construction-time MNAR conflict check

ImputationConfig.__post_init__ computes the intersection of mnar_columns and per_column_strategy.keys(). If non-empty, raises ValueError naming the conflicting columns.

6. suggest_refit_config MNAR-skip constraint (Scope 3 dependency)

When suggest_refit_config (Issue #92) populates per_column_strategy[col] = Median for low-R² columns, it must skip any column already in mnar_columns. MNAR columns do not go through the routing engine; their R² is not a meaningful signal and adding them to per_column_strategy would fail the construction-time conflict check.

7. No routing-engine bypass for DropCandidate

DropCandidate (Priority 1) fires unconditionally before the per_column_strategy check. A column that is >50% missing is dropped regardless of any override declaration. There is no mechanism in this scope to rescue a DropCandidate column; that requires a dedicated escape hatch in a future scope.

8. add_indicator_columns is orthogonal

If a column appears in both per_column_strategy and add_indicator_columns, the indicator is still added. Strategy and indicator decisions are independent.

Testing Decisions

A good test for this feature asserts external behaviour — the strategy recorded in ColumnImputationRecord, the error raised at construction or fit time, the signal text — not internal routing logic. Tests should not inspect which branch of _fit_one was taken; they should observe what came out.

Seam 1: NumericImputationConfig construction — test_imputation_config.py

Pure unit tests with no DataFrames. Tests for:

• ValueError when Passthrough, Indicator, Dropped, or MNAR appear in per_column_strategy (one test per blocked strategy)
• ValueError when Constant is declared with no companion per_column_constant_fill entry
• ValueError from ImputationConfig.__post_init__ when a column appears in both mnar_columns and per_column_strategy
• Valid construction succeeds with all allowed strategies
• to_dict / from_dict round-trip preserves both new fields
• Empty dicts are the correct default

Prior art: the entire existing test_imputation_config.py — all tests are pure config construction and round-trip assertions.

Seam 2: NumericImputer.fit() routing — test_numeric_imputer.py

Tests using minimal StructuralProfileResult stubs (no real profiler). Tests for:

• Override fires at Priority 1.5: a column with MCAR Minor profile + per_column_strategy = Regression produces strategy=Regression in the record
• Override fires before MNAR priority: a column in per_column_strategy that is not in mnar_columns does not receive MNAR treatment
• DropCandidate still fires: a column with DropCandidate flag still produces strategy=Dropped even when it appears in per_column_strategy
• Signal is recorded: "per_column_strategy_override: user forced strategy=X" appears in record.signals
• Constant fill uses per_column_constant_fill: the record's fill_value matches the declared constant

Prior art: the existing profile-stub pattern in test_numeric_imputer.py — _make_profile and _numeric_cp helpers.

Seam 3: ImputationOrchestrator.fit() size-guard validation — test_orchestrator.py

Tests using real small DataFrames (no profiler stub needed for the error path). Tests for:

• ValueError at fit time when per_column_strategy = Regression and n_rows < regression_min_rows
• ValueError at fit time when per_column_strategy = KNN and n_rows > knn_max_rows
• Message names the column, strategy, guard, actual value, and threshold

Prior art: existing orchestrator tests.

Out of Scope

• per_column_strategy for non-Numeric semantic types. Categorical and Boolean columns have trivial routing (always Mode); there is no routing tree to override. If a future scope adds complex routing for those types, per-column overrides for them will be designed separately.
• Rescuing DropCandidate columns. A column that is >50% missing cannot be kept via per_column_strategy. A dedicated escape hatch (e.g. force_keep_columns) is a future scope.
• per_column_strategy for Dropped strategy. Force-dropping a column below the 50% threshold is out of scope. Use PipelineConfig.exclude_columns (Hard Exclusion) for intentional column removal.
• User-configurable MNAR fill override. Issue Scope 8: MNAR Treatment — data-derived fill replaces sentinel constant #97 (Scope 8) decided that the MNAR fill is always data-derived (observed mean or median). No user-specified MNAR fill constant is exposed in this or any scope.
• per_column_n_neighbors and per_column_max_iter. These parameter overrides are already defined in the CONTEXT.md glossary and are populated by suggest_refit_config (Issue Scope 3: Multi-round Iteration and Feedback Loop — ImputationFitDiagnostic and suggest_refit_config #92, Scope 3). Their implementation is owned by Scope 3, not this scope.

Further Notes

Dependencies on open GitHub Issues

• Issue Scope 0: Regression Imputer Overhaul — IterativeImputer, NonlinearityTag, dynamic convergence #89 (Scope 0) — Regression Internals: when a user forces Regression via per_column_strategy, the quality of the imputed values depends on Scope 0's IterativeImputer and NonlinearityTag-driven estimator selection. Scope 14 can ship before Scope 0 (the override mechanism is correct regardless), but imputation quality for forced Regression will improve once Scope 0 lands.
• Issue Scope 1: KNNImputer Hyperparameter Selection — adaptive k, reliability-based weights, NaN-safe scaling #90 (Scope 1) — KNN Overhaul: same relationship for forced KNN overrides. Scope 14 can ship before Scope 1; adaptive k and NaN-safe scaling land with Scope 1.
• Issue Scope 2: MICE Overhaul — estimator selection, dynamic convergence, and data-adaptive hyperparameters #91 (Scope 2) — MICE Overhaul: same relationship for forced MICE overrides. Estimator selection and dynamic convergence land with Scope 2.
• Issue Scope 3: Multi-round Iteration and Feedback Loop — ImputationFitDiagnostic and suggest_refit_config #92 (Scope 3) — suggest_refit_config: Scope 3 writes to per_column_strategy and per_column_max_iter. It depends on both fields existing in NumericImputationConfig — introduced by this scope. Scope 3 must also implement the MNAR-skip constraint (Implementation Decision 6 above). Scope 14 should ship before or together with Scope 3.
• Issue Scope 8: MNAR Treatment — data-derived fill replaces sentinel constant #97 (Scope 8) — MNAR Data-Derived Fill: the motivating use case for Constant strategy (transaction count = 0) is explicitly differentiated from MNAR semantics. The clean separation between per_column_strategy = Constant (user-specified domain fill) and mnar_columns (data-derived fill + indicator) relies on Scope 8 having completed the MNAR path rewrite. Scope 14 works correctly before Scope 8 ships but the conceptual boundary is sharpest once both are in place.

ADR 0029

Two architectural decisions from this scope are recorded in ADR 0029: (1) per_column_strategy fires after DropCandidate and before MNAR, and (2) size-guard violations raise ValueError at fit time rather than falling back silently to Median. The rationale and rejected alternatives are documented there.