Context
contains_expected returns True when ctx.expected_output is None (evaluators.py:71-72), and word_overlap returns overlap=1.0 (evaluators.py:152). Meanwhile the wrapper resolves expected to None whenever no EvalCase reaches the function (wrapper.py:172-176) — a legitimate path for single-shot evals, but also exactly what a wiring mistake produces.
Impact
Chained failure mode: case wiring forgotten → expected=None → contains_expected passes everything, word_overlap reports a perfect 1.0. Each behavior is defensible in isolation; stacked, they form a silent path where a broken eval looks healthy.
Suggestion
Either a strict: bool = False param (None expected → fail or error), or emit the score as skipped=True instead of a vacuous pass so reporting distinguishes "not applicable" from "verified". Related to the zero-evaluators trap in #117 — same family of silent passes.
Context
contains_expectedreturnsTruewhenctx.expected_output is None(evaluators.py:71-72), andword_overlapreturnsoverlap=1.0(evaluators.py:152). Meanwhile the wrapper resolvesexpectedtoNonewhenever noEvalCasereaches the function (wrapper.py:172-176) — a legitimate path for single-shot evals, but also exactly what a wiring mistake produces.Impact
Chained failure mode: case wiring forgotten →
expected=None→contains_expectedpasses everything,word_overlapreports a perfect 1.0. Each behavior is defensible in isolation; stacked, they form a silent path where a broken eval looks healthy.Suggestion
Either a
strict: bool = Falseparam (None expected → fail or error), or emit the score asskipped=Trueinstead of a vacuous pass so reporting distinguishes "not applicable" from "verified". Related to the zero-evaluators trap in #117 — same family of silent passes.