Evaluation Corpus

This directory contains public-safe synthetic material for M7 evaluation. It must never contain real repositories, real scan output, real GHAS exports, raw private findings, internal hostnames, or credentials.

The committed corpus is under eval/synthetic-corpus/:

• checkout/ is a tiny fake local checkout.
• gitleaks.synthetic.toml is a synthetic-only Gitleaks rulepack.
• targets.local.example.yaml points at the fake checkout when commands run from the repository root.
• expected-findings.example.json is the ground truth consumed by security-scanner evaluate.

Example workflow:

uv run security-scanner scan \
  --manifest eval/synthetic-corpus/targets.local.example.yaml \
  --output private/eval-findings.jsonl

uv run security-scanner evaluate \
  --expected eval/synthetic-corpus/expected-findings.example.json \
  --findings private/eval-findings.jsonl

evaluate reports true positives, false positives, false negatives, precision, and recall. Its default gate fails when any expected synthetic finding is missed, precision is below 0.90, or recall is below 0.99.