Commit 8424575
authored
fix(aimock): validate fixtures at load time to prevent runtime 500s (CopilotKit#3973)
## Summary
Follow-up hardening to CopilotKit#3971. aimock supports fixture schema validation
at startup via `--validate-on-load`, but the flag is **opt-in** and the
showcase Dockerfile was not passing it. That meant fixtures with
unrecognized response keys (e.g. `"text"` instead of `"content"`) loaded
silently and only surfaced as HTTP 500s at request time — which is
exactly what crashed crewai-crews and triggered CopilotKit#3971.
This PR wires up two independent safety nets so a broken fixture can't
ship again:
1. **Dockerfile (fail-fast at container boot)** —
`showcase/aimock/Dockerfile` now passes `--validate-on-load`. If any
fixture fails the aimock schema, the container exits non-zero instead of
starting and serving 500s. Railway will not promote a bad build.
2. **CI test (fail-fast in PR review)** — new vitest spec at
`showcase/scripts/__tests__/aimock-fixtures.test.ts` imports
`loadFixtureFile` + `validateFixtures` from `@copilotkit/aimock` and
asserts zero errors against both `feature-parity.json` and `smoke.json`.
Runs inside the existing ` Showcase: Validate` workflow
(`showcase/scripts` vitest suite) on every PR that touches
`showcase/**`.
## Verification
**Red-green on the vitest spec:**
- Rebased onto the tip of main *before* CopilotKit#3971 merged: the spec fails
with 5 errors — exactly the 5 broken `"text"` fixtures (`plan`, `steps`,
`mars`, `dashboard`, `report`) that CopilotKit#3971 repaired.
- Rebased forward onto main *after* CopilotKit#3971: spec passes with 0 errors,
all 549 showcase/scripts tests green.
**Red-green on the Dockerfile:**
- Current fixtures + `--validate-on-load`: container boots cleanly, logs
`Loaded 39 fixture(s) from /fixtures`.
- Injecting an intentionally broken fixture (`response: { "text": "..."
}`): container fails to start with `[aimock] Fixture 0: response is not
a recognized type (must have content, toolCalls, error, or embedding)` /
`Validation failed: 1 error(s), 0 warning(s)` and non-zero exit.
## Test plan
- [x] Local: full `showcase/scripts` vitest suite passes (549/549)
- [x] Local: `pnpm run test` (monorepo) passes
- [x] Docker: image builds and starts with `--validate-on-load` against
current fixtures
- [x] Docker red-green: broken fixture fails container start with
non-zero exit
- [ ] CI: ` Showcase: Validate` job runs the new test file on PR26 files changed
Lines changed: 378 additions & 265 deletions
File tree
- .github/workflows
- examples/integrations
- adk
- agno
- crewai-crews
- langgraph-fastapi
- langgraph-js
- langgraph-python-threads
- langgraph-python
- llamaindex
- mastra
- ms-agent-framework-dotnet
- ms-agent-framework-python
- pydantic-ai
- strands-python
- scripts/doc-tests/fixtures
- showcase
- aimock
- scripts
- __tests__
- e2e
- create-integration
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
10 | 12 | | |
| 13 | + | |
| 14 | + | |
11 | 15 | | |
12 | 16 | | |
13 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
0 commit comments