[FEATURE] Rule validation + test framework (`codelens rule-validate`, `codelens rule-test`)

[Wolfvin]

Summary

Add codelens rule-validate (catch typos, unknown keys, schema violations) and codelens rule-test (snapshot testing for rule YAML with positive/negative samples). Currently CodeLens silently skips malformed rules, creating false sense of security.

Worker consensus (4 reports)

Worker	Source	Contribution
Opengrep	update!/CodeLens_Opengrep_Upgrade_Analysis.md #40	codelens test <rule.yaml> — snapshot testing. Inline # ruleid: <id> / # ok: <no-finding-expected> markers OR separate <rule>.test.yaml + targets/<name>.<ext> + <name>.expected.json. Fixtest: <target>.<ext> + <target>.fixed.<ext>.
Opengrep	same file #41	codelens rule-validate <rule.yaml> (rename existing validate → registry-validate). YAML syntax + schema + pattern parseability + cross-field validation. Exit codes 0/1/2.
Opengrep	same file #55	codelens show <file> — AST dump (S-expression) + --explain for step-by-step metavar binding. DX tool for rule author.
Semgrep	update!/CodeLens_Upgrade_Issues_from_Semgrep.md CL-005	codelens rule-validate [--strict] — detect typos (pattern-eiter vs pattern-either), unknown keys, missing required fields, wrong types, unsupported languages. Exit codes 0 (valid) / 1 (error) / 2 (warning + --strict). Pre-commit hook integration.
Semgrep	same file CL-006	codelens rule-test <rule-path> + .test.yaml fixture format. Output: <rule-id>: PASS (3/3 samples). Seed by migrating ≥10 rules from existing python_security.yaml. CI integration.

Proposed scope (P1, 2-3 weeks total)

Part 1 — rule-validate (P1, 1 week)

• Rename existing validate → registry-validate (deprecation warning 1 version before rename)
• New codelens rule-validate <rule.yaml>:
  • YAML syntax check (unclosed quote, indentation)
  • Schema validation (required fields id, message, severity, language; enum severity: critical|high|medium|low|info)
  • Pattern parseability (compile pattern: to AST for target language, detect syntax error)
  • Cross-field validation (patterns: and pattern: mutually exclusive; fix: requires pattern: or patterns:)
• Exit codes: 0 valid / 1 error / 2 warning (with --strict)
• Pre-commit hook integration
• New files: scripts/commands/rule_validate.py, scripts/rule_validator.py, references/rule-schema.json

Part 2 — rule-test (P1, 1-2 weeks)

• New codelens rule-test <rule-path> command
• Fixture format: <rule>.yaml + <rule>.test.yaml (inline with # ruleid: <id> / # ok: <no-finding-expected> markers)
• OR separate files: <rule>.yaml + targets/<name>.<ext> + <name>.expected.json
• Fixtest: <target>.<ext> + <target>.fixed.<ext> (verifies autofix output)
• Diagnosis: diff actual vs expected, pass/fail per rule, non-zero exit on fail
• --test-ignore-todo flag skips # todoruleid markers
• Output: human-readable summary + optional --json for CI
• CI workflow: .github/workflows/codelens-rule-tests.yml
• Seed: migrate ≥10 rules from python_security.yaml and javascript_security.yaml
• New files: scripts/commands/rule_test.py, scripts/rule_test_runner.py, tests/rule_fixtures/

Part 3 — show command (P3, 1 week, optional)

• codelens show <file> — tree-sitter AST dump (S-expression)
• codelens show --pattern <pat> --lang <lang> — compiled pattern AST
• --explain — step-by-step metavariable binding when match occurs
• --html — interactive AST tree
• New files: scripts/commands/show.py

Acceptance criteria

• rule-validate catches: typos, unknown keys, missing required fields, wrong types, unparseable patterns
• rule-validate exit codes correct (0/1/2)
• rule-test runs against fixture directory, reports pass/fail per rule
• 10+ existing rules migrated to test format
• CI workflow runs rule-test on every PR touching scripts/rules/
• Pre-commit hook for rule-validate documented

Related

• Depends on Rule pattern engine issue (the Semgrep-compatible YAML one) — rule-validate's pattern parseability check needs the pattern parser to exist first.