Agent/Skill eval framework

[sendtoshailesh]

Description

Investigate and prototype a behavioral eval framework for Git-Ape agents and skills. This work was originally scoped as Section 3 of #17 but is being split out into its own issue so that #17 can land focused on CONTRIBUTING.md + structural PR validation CI, while eval design proceeds as a separate exploratory track.

Agents and skills in this repo are Markdown-based (frontmatter + procedural prose). We need a way to verify that:

• Given a user intent (e.g., "deploy a function app"), the right agent is selected and it invokes the expected skills in the expected order.
• Given mock Azure CLI / MCP responses, a skill produces the expected report shape and content.
• Critical-path regressions (requirements → template → security gate → deploy) are caught before merge.

Scope

1. Investigate frameworks

• Evaluate openai/evals — Completion Function Protocol, model coupling, Python-only ecosystem fit.
• Compare against a lighter custom Node.js harness aligned with this repo's existing tooling (e.g., scripts/validate-structure.js from CONTRIBUTING.md and contribution verification CI #17).
• Document the trade-offs and pick one. Decision must be recorded in evals/README.md.

2. Define eval scenarios (JSON schema)

• Schema fields: id, user_intent, expected_agent, expected_skill_sequence, assertions[] (e.g., skill_order, skill_present, skill_absent).
• At least one scenario for the core deployment path: deploy-function-app covering prereq-check → azure-naming-research → azure-rest-api-reference → azure-resource-availability → azure-security-analyzer → azure-deployment-preflight → azure-integration-tester.
• Scenarios live under evals/scenarios/*.json.

3. Build eval runner

• evals/run-eval.js (or chosen tool) that:
  • Loads all scenarios from evals/scenarios/.
  • Cross-references expected_agent against real agent definitions in agents/ (or wherever they live).
  • Cross-references expected_skill_sequence and assertion targets against real skills under skills/.
  • Fails on dangling references and assertion violations.
• Wire into PR validation CI as a non-blocking job initially; promote to blocking once stable.

4. Skill output testing (stretch)

• Mock Azure CLI / MCP responses fixtures under evals/fixtures/.
• Snapshot-style test that a skill's procedure produces the expected report sections/headings.

Prior art

PR #40 (now superseded for the #17 scope) contained an initial spike of this work in evals/:

• evals/README.md — investigation notes and a tentative decision favoring a custom Node harness over openai/evals.
• evals/scenarios/deploy-function-app.json — first draft of the core deployment scenario.
• evals/run-eval.js — first cut of a validator that resolves agent/skill references.

That spike was not merged. This issue should re-evaluate it from scratch (the codebase has since evolved) but may reuse the artifacts as a starting point.

Acceptance criteria

• Decision recorded in evals/README.md: openai/evals vs. custom harness, with rationale.
• Scenario JSON schema documented.
• At least one scenario defined for the core deployment path (deploy-function-app).
• Runner script validates scenarios against real agent/skill definitions (no dangling references).
• Runner wired into CI (non-blocking is acceptable for the first pass).
• CONTRIBUTING.md updated with a short "How to add an eval scenario" section once the harness is chosen.

Out of scope

• LLM-in-the-loop / model-graded evals (can be a follow-up).
• Live Azure calls in CI (use mocks/fixtures only).

Related

• Split out from CONTRIBUTING.md and contribution verification CI #17.
• Superseded spike: Add CONTRIBUTING.md, PR validation CI, and eval framework #40.

[arnaudlh]

@sendtoshailesh included waza into agent engineering devcontainer for 0.0.2.