Skip to content

feat: MissingnessProfiler string sentinel replace semantics #179

Description

@DEVunderdog

Parent

#176

What to build

Update MissingnessProfiler._profile_column in profiling/_missingness_profiler.py to accept an optional string_sentinels: list[str] | None parameter (default None).

When string_sentinels is provided and the column is String/Utf8, the eff_null boolean mask uses only the declared values (case-insensitive) for sentinel matching — the hardcoded _SENTINEL_STRINGS set is not applied for that column. Empty/whitespace-only strings are always included in the mask regardless of any declaration.

_run() passes profile_config.string_sentinels.get(col_name) for each column so the profiler reads declared sentinels from config without structural changes to its call signature from the orchestrator.

When string_sentinels=None, output is identical to the current behaviour — hardcoded _SENTINEL_STRINGS apply.

Acceptance criteria

  • _profile_column gains string_sentinels: list[str] | None = None
  • Declared sentinel values are counted in effective_null_count alongside standard Polars nulls
  • Hardcoded defaults are not counted for a column that has a declaration (replace semantics)
  • Empty/whitespace strings are counted regardless of whether a declaration exists
  • effective_null_ratio reflects the declaration-inclusive count
  • Severity classification (Minor/Moderate/High/Severe) uses the declaration-inclusive ratio
  • A column with >50% declared-sentinel rows receives a DropCandidate flag
  • Declared sentinel rows participate in the missingness indicator used for MAR correlation detection
  • When string_sentinels=None, output is identical to current behaviour (no regression)
  • Columns with no declaration are completely unaffected
  • Unit tests in tests/unit/profiling/test_missingness_profiler.py covering all criteria above
  • Docstrings updated per ADR-0034

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions