Skip to content

Commit eaf8724

Browse files
committed
feat(showcase): add eval tier configuration and gitignore entries
Add eval-tiers.json defining three evaluation tiers (Gold Standard, Key Partners, Full Matrix) with fail-fast on tier 1. Add gitignore entries for .eval-results/ and .eval-baseline.json local artifacts.
1 parent d0cac2e commit eaf8724

2 files changed

Lines changed: 24 additions & 0 deletions

File tree

showcase/.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,7 @@ shell/src/data/*.json
1919
shell-dojo/src/data/*.json
2020
shell-docs/src/data/*.json
2121
shell-dashboard/src/data/*.json
22+
23+
# Eval results (local, not committed)
24+
.eval-results/
25+
.eval-baseline.json

showcase/eval-tiers.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"_comment": "Tiered evaluation order. Tier 1 fail_fast stops eval early on gold-standard regression.",
3+
"tiers": [
4+
{
5+
"name": "Gold Standard",
6+
"slugs": ["langgraph-python"],
7+
"fail_fast": true
8+
},
9+
{
10+
"name": "Key Partners",
11+
"slugs": ["mastra", "crewai-crews", "google-adk", "langgraph-typescript"],
12+
"fail_fast": false
13+
},
14+
{
15+
"name": "Full Matrix",
16+
"slugs": "*",
17+
"fail_fast": false
18+
}
19+
]
20+
}

0 commit comments

Comments
 (0)