Skip to content

feat: profiling/orchestrator.py MAR detection — string sentinel replace semantics #180

Description

@DEVunderdog

Parent

#176

What to build

Update the MAR correlation detection loop in profiling/orchestrator.py to use user-declared string sentinels with replace semantics when building per-column missingness masks.

Where the orchestrator currently references _SENTINEL_STRINGS unconditionally for String/Utf8 columns (around lines 357 and 375), it should instead check profile_config.string_sentinels.get(col_name):

  • If a declaration exists for the column → use only the declared values (case-insensitive) for sentinel matching in the mask
  • If no declaration exists → fall back to _SENTINEL_STRINGS (unchanged current behaviour)

Empty/whitespace detection is always applied unconditionally.

Acceptance criteria

  • MAR correlation missingness mask for a String/Utf8 column with a declared sentinel uses only the declared values, not _SENTINEL_STRINGS
  • MAR correlation missingness mask for a String/Utf8 column with no declaration continues to use _SENTINEL_STRINGS (no regression)
  • Empty/whitespace strings are always included in the mask regardless of declaration
  • Columns of other dtypes are completely unaffected
  • Unit tests in tests/unit/profiling/test_pipeline_config.py or test_missingness_profiler.py (whichever already covers MAR detection paths) verify the replace-semantics mask
  • Docstrings updated per ADR-0034 for any modified in-scope methods

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions