Skip to content

feat: extend _resolve_effective_nulls with user-declared string sentinel replace semantics #178

Description

@DEVunderdog

Parent

#176

What to build

Extend _resolve_effective_nulls in utils/_null_normalization.py to accept an optional string_sentinels: dict[str, list[str]] | None parameter (default None).

For each String/Utf8 column in the DataFrame:

  • If the column name is present in string_sentinels, match only the declared values (case-insensitive via .str.to_uppercase()). The hardcoded _SENTINEL_STRINGS set is not applied for that column.
  • If the column name is absent, fall back to the existing _SENTINEL_STRINGS behaviour (no change).
  • Empty/whitespace-only string detection is always applied unconditionally, regardless of any declaration.

The function remains pure and side-effect-free. When string_sentinels is None or {}, behaviour is strictly identical to the current implementation — no existing caller is broken.

Acceptance criteria

  • _resolve_effective_nulls signature gains string_sentinels: dict[str, list[str]] | None = None
  • String/Utf8 column with no declaration → hardcoded _SENTINEL_STRINGS applies (regression: current behaviour preserved)
  • String/Utf8 column with user-declared sentinels → only declared values converted to null; a hardcoded default not in the declaration (e.g. "?") is not converted
  • Declared sentinel matching is case-insensitive
  • Multiple declared sentinels for one column → all converted; any other value unchanged
  • Empty/whitespace strings always converted to null regardless of whether a declaration exists for that column
  • Column present in string_sentinels but absent from the DataFrame → no error, rest of DataFrame unchanged
  • string_sentinels=None → output byte-for-byte identical to calling with no argument
  • All cases above covered by unit tests in tests/unit/imputation/test_null_normalization.py
  • Docstring updated per ADR-0034

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions