Context
@suite.eval() with no evaluators= and no per-case evaluators produces scores == [], and the wrapper computes passed=all(s.passed for s in scores) (wrapper.py:100) — all([]) is True, so the case is green no matter what the task returned. validate_evaluators([]) accepts the empty list without complaint.
Impact
The classic always-green eval suite: a wiring mistake (forgot evaluators=, or per-case evaluators not attached) is indistinguishable from a passing eval. This contradicts the framework's own stated philosophy — validate_evaluators's docstring says failures belong loud at the boundary.
Suggestion
In the wrapper, after merging suite-level and per-case evaluators: if the merged list is empty, raise (preferred) or at least emit a prominent warning. Registration-time rejection isn't enough on its own because per-case evaluators are only known at runtime — the runtime guard covers both.
Context
@suite.eval()with noevaluators=and no per-case evaluators producesscores == [], and the wrapper computespassed=all(s.passed for s in scores)(wrapper.py:100) —all([])isTrue, so the case is green no matter what the task returned.validate_evaluators([])accepts the empty list without complaint.Impact
The classic always-green eval suite: a wiring mistake (forgot
evaluators=, or per-case evaluators not attached) is indistinguishable from a passing eval. This contradicts the framework's own stated philosophy —validate_evaluators's docstring says failures belong loud at the boundary.Suggestion
In the wrapper, after merging suite-level and per-case evaluators: if the merged list is empty, raise (preferred) or at least emit a prominent warning. Registration-time rejection isn't enough on its own because per-case evaluators are only known at runtime — the runtime guard covers both.