Skip to content

feat: ImputationOrchestrator.fit() string sentinel threading + end-to-end integration test #183

Description

@DEVunderdog

Parent

#176

What to build

Wire string sentinel declarations from the profile into Phase 2 and verify the full pipeline with an integration test.

ImputationOrchestrator.fit() (imputation/orchestrator.py):

  • Change the existing _resolve_effective_nulls(train_df) call to _resolve_effective_nulls(train_df, string_sentinels=profile.string_sentinels) so replace-semantics normalization applies to the training split before any fill values are computed.
  • Pass string_sentinels=profile.string_sentinels to the FittedImputer constructor so transform-time normalization uses the same declarations.

Integration test (tests/integration/test_imputation_end_to_end.py):

  • DataFrame with a String/Utf8 column (status) containing a mix of real values, declared sentinels ("N/A", "missing"), and the hardcoded default "?".
  • ProfileConfig declares string_sentinels={"status": ["N/A", "missing"]}.
  • StructuralProfiler.profile() runs; the returned profile carries the declarations.
  • ImputationOrchestrator.fit() produces a FittedImputer.
  • FittedImputer.transform() on the test split produces a result where:
    • "N/A" and "missing" values are filled (treated as effective null).
    • "?" values are not filled (replace semantics: "?" is not in the declaration and the hardcoded set does not apply for this column).
    • The fill value is derived exclusively from non-sentinel observations.

Acceptance criteria

  • ImputationOrchestrator.fit() passes string_sentinels=profile.string_sentinels to _resolve_effective_nulls
  • ImputationOrchestrator.fit() passes string_sentinels=profile.string_sentinels to the FittedImputer constructor
  • Unit test verifies that the FittedImputer returned by fit() has string_sentinels equal to profile.string_sentinels
  • Integration test: "N/A" and "missing" are filled in the transform output
  • Integration test: "?" is not filled (replace semantics confirmed)
  • Integration test: fill value is computed from non-sentinel observations only
  • Columns with no declared sentinels are completely unaffected — existing integration tests continue to pass unchanged
  • Docstrings updated per ADR-0034

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions