Rebalance filter priority_score: raise keyword-overlap weight / lower 0.25·credibility blend

[smodee]

Context

Follow-up split out of #13 (closed) and PR #30. PR #30 promoted ~22 national/international outlets from unknown (domain_score 0.2) to Tier 3 trusted_media (0.6) so legitimate outbreak reporting was no longer floored below the filter's credibility threshold. It also added a topical relevance term to the search-stage score (#4).

Problem

The relevance term added in #30 lives in search-stage ranking only — it does not feed the filter's keep decision. The filter's compute_priority_score still blends in 0.25·credibility heavily. Now that reputable outlets sit at Tier 3 (0.6), that credibility weight can push off-topic pieces from those same outlets over the keep threshold on authority alone.

Observed during the #3/#4/#13 review: an H5N1 run kept a CBS transcript that was off-topic — admitted on credibility, not relevance. This is the direct, PR-acknowledged trade-off of the Tier 3 promotions ("Known interaction to flag for reviewers" in #30).

Proposed change

In bioscancast/stages/filtering/heuristics.py (compute_priority_score / heuristic weights in stages/filtering/config.py):

• Raise the keyword-overlap / relevance weight in the filter's keep decision, and/or
• Lower the 0.25·credibility blend weight,

so a Tier 3 domain no longer clears the borderline on authority alone when topical overlap is low.

Notes / guardrails

• Keep official-source recall at 1.0 — the sweep in Heuristic filtering scores too low for real-world search results #13 showed official recall holds at every threshold; don't regress it.
• The threshold itself is a non-lever (flat plateau 0.60–0.775, per Heuristic filtering scores too low for real-world search results #13's sweep) — this is about the weight blend, not the cutoff.
• Re-run the offline sweep in scripts/sweep_filter_params.py against the hand-labeled pools (data/investigations/live_pools/labels.json) to confirm precision improves without dropping official recall.

Full method + numbers: data/investigations/findings-issues-3-4-13.md.

[smodee]

Status: mechanism landed in #48 + precision/ranking improvement now empirically confirmed → closing.

#48 (merged c8962a6) implemented the rebalance: keyword_overlap 0.40→0.50, domain 0.20→0.10, credibility blend 0.25→0.15, plus a structural official_recall_guarantee (officials kept unconditionally, so recall=1.0 no longer depends on the weights).

The acceptance criterion here — confirm precision improves without dropping official recall — has now been checked. The original sweep harness (sweep_filter_params.py + the hand-labeled live_pools) no longer exists on any branch, so it was rebuilt with an LLM judge in place of the human annotator: 8 fresh forecasting questions, 212 organic results captured via the live search stage, labeled by gpt-4o for forecast-usefulness (rubric anchored to topicality, not authority), swept against the filter's priority_score weights.

Results:

• Ranking quality MAP 0.628 → 0.728 (+16%), R-precision 0.507 → 0.602 (+19%); improves on every question with positives (none regress; measles most, AP 0.303→0.566).
• Decomposition confirms the gain comes from the weight/blend change; the official_recall_guarantee has zero effect on ranking (pure recall preservation, as designed).
• Off-topic high-authority false keeps: 1 → 0.
• Direction is validated and Filter: relevance beats authority in the keep decision; guarantee official recall (#44) #48's weights are conservative — a more aggressive kw .55 / dom .05 / blend .10 variant scores higher still (MAP 0.750), consistent with Filter: relevance beats authority in the keep decision; guarantee official recall (#44) #48's "weights are directional" note.

Caveat: LLM-judge labels on a single fresh news snapshot (one backend, 28 positives) — directional confirmation, not a definitive precision figure. The keep-stage effect is small in absolute terms because the 0.65 threshold already filters nearly all organic content (on-topic survival depends on the downstream LLM-rescue path) — i.e. #48 improves the ranking, corroborating the findings-issues-3-4-13.md conclusion that the filter weights aren't the dominant lever (upstream search relevance / the LLM path are).

Closing as done. Optional future micro-tune (push domain weight lower / blend toward 0.10) is not required and shouldn't be over-fit to one snapshot — noting the headroom for whenever a broader sweep runs.