Tracked for when the `protest history` browse CLI is reintroduced (code archived on `archive/history-cli`).
Problem
`list`'s score-trend arrows (`_track_cases` + `_rich_score_arrows`) pile every case's score from every run into one flat list, then compare `values[0]` vs `values[-1]`. That compares two arbitrary individual case scores — not a per-run mean — and worse, it flattens across eval-definition changes. The one view that claims to show "improvement over time" draws an arrow over a changed thermometer.
The right unit: the epoch
"Improvement over time" only has meaning within a stable definition. Two axes:
- Improve the SUT (prompt/model): eval fixed, no hash changes → scores comparable → this is the real trend.
- Improve the eval (rubric/judge/threshold): `eval_hash` changes → comparing 0.60→0.50 across it is meaningless.
Epoch of score `s` for a suite = maximal run of consecutive runs where the set of `(case_name, case_hash, eval_hash)` carrying `s` is identical.
Within an epoch, mean-per-run is comparable; at a boundary, start a new series (do not smooth over it). Open design question: stable-cohort (track the subset of cases unchanged over the window, report churn separately — smoother, answers "for what I didn't touch, am I improving?") vs strict epoch (reset on any change — honest but choppy in active dev).
No storage change: `case_hash`/`eval_hash` already persisted per case.
Note: `compare` already self-heals (it only diffs adjacent runs, so same-hash-or-not suffices). This issue is specifically about the trend view lying across boundaries.
Tracked for when the `protest history` browse CLI is reintroduced (code archived on `archive/history-cli`).
Problem
`list`'s score-trend arrows (`_track_cases` + `_rich_score_arrows`) pile every case's score from every run into one flat list, then compare `values[0]` vs `values[-1]`. That compares two arbitrary individual case scores — not a per-run mean — and worse, it flattens across eval-definition changes. The one view that claims to show "improvement over time" draws an arrow over a changed thermometer.
The right unit: the epoch
"Improvement over time" only has meaning within a stable definition. Two axes:
Within an epoch, mean-per-run is comparable; at a boundary, start a new series (do not smooth over it). Open design question: stable-cohort (track the subset of cases unchanged over the window, report churn separately — smoother, answers "for what I didn't touch, am I improving?") vs strict epoch (reset on any change — honest but choppy in active dev).
No storage change: `case_hash`/`eval_hash` already persisted per case.
Note: `compare` already self-heals (it only diffs adjacent runs, so same-hash-or-not suffices). This issue is specifically about the trend view lying across boundaries.