Skip to content

Commit 8424575

Browse files
authored
fix(aimock): validate fixtures at load time to prevent runtime 500s (CopilotKit#3973)
## Summary Follow-up hardening to CopilotKit#3971. aimock supports fixture schema validation at startup via `--validate-on-load`, but the flag is **opt-in** and the showcase Dockerfile was not passing it. That meant fixtures with unrecognized response keys (e.g. `"text"` instead of `"content"`) loaded silently and only surfaced as HTTP 500s at request time — which is exactly what crashed crewai-crews and triggered CopilotKit#3971. This PR wires up two independent safety nets so a broken fixture can't ship again: 1. **Dockerfile (fail-fast at container boot)** — `showcase/aimock/Dockerfile` now passes `--validate-on-load`. If any fixture fails the aimock schema, the container exits non-zero instead of starting and serving 500s. Railway will not promote a bad build. 2. **CI test (fail-fast in PR review)** — new vitest spec at `showcase/scripts/__tests__/aimock-fixtures.test.ts` imports `loadFixtureFile` + `validateFixtures` from `@copilotkit/aimock` and asserts zero errors against both `feature-parity.json` and `smoke.json`. Runs inside the existing ` Showcase: Validate` workflow (`showcase/scripts` vitest suite) on every PR that touches `showcase/**`. ## Verification **Red-green on the vitest spec:** - Rebased onto the tip of main *before* CopilotKit#3971 merged: the spec fails with 5 errors — exactly the 5 broken `"text"` fixtures (`plan`, `steps`, `mars`, `dashboard`, `report`) that CopilotKit#3971 repaired. - Rebased forward onto main *after* CopilotKit#3971: spec passes with 0 errors, all 549 showcase/scripts tests green. **Red-green on the Dockerfile:** - Current fixtures + `--validate-on-load`: container boots cleanly, logs `Loaded 39 fixture(s) from /fixtures`. - Injecting an intentionally broken fixture (`response: { "text": "..." }`): container fails to start with `[aimock] Fixture 0: response is not a recognized type (must have content, toolCalls, error, or embedding)` / `Validation failed: 1 error(s), 0 warning(s)` and non-zero exit. ## Test plan - [x] Local: full `showcase/scripts` vitest suite passes (549/549) - [x] Local: `pnpm run test` (monorepo) passes - [x] Docker: image builds and starts with `--validate-on-load` against current fixtures - [x] Docker red-green: broken fixture fails container start with non-zero exit - [ ] CI: ` Showcase: Validate` job runs the new test file on PR
2 parents 550150a + 513d72e commit 8424575

26 files changed

Lines changed: 378 additions & 265 deletions

File tree

.github/workflows/showcase_aimock-e2e.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ jobs:
7878
- name: Start aimock
7979
run: |
8080
npm install -g @copilotkit/aimock@latest
81-
npx aimock --port 4010 --host 127.0.0.1 --fixtures showcase/aimock/feature-parity.json &
81+
npx aimock --port 4010 --host 127.0.0.1 --fixtures showcase/aimock/feature-parity.json --validate-on-load &
8282
# Wait for aimock to be ready
8383
for i in $(seq 1 20); do
8484
curl -sf http://localhost:4010/ > /dev/null 2>&1 && break

.github/workflows/showcase_validate.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,14 @@ on:
44
pull_request:
55
paths:
66
- "showcase/**"
7+
- "examples/integrations/**/fixtures/**"
8+
- "scripts/doc-tests/fixtures/**"
79
push:
810
branches: [main]
911
paths:
1012
- "showcase/**"
13+
- "examples/integrations/**/fixtures/**"
14+
- "scripts/doc-tests/fixtures/**"
1115

1216
concurrency:
1317
group: showcase-validate-${{ github.ref }}

.github/workflows/test_doc-examples.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ jobs:
4242
echo "aimock binary: $AIMOCK_BIN"
4343
ls -la "$AIMOCK_BIN" || echo "binary not found at expected path"
4444
which aimock || echo "aimock not on PATH"
45-
nohup node $(npm root -g)/@copilotkit/aimock/dist/cli.js --fixtures scripts/doc-tests/fixtures > /tmp/aimock.log 2>&1 &
45+
nohup node $(npm root -g)/@copilotkit/aimock/dist/cli.js --fixtures scripts/doc-tests/fixtures --validate-on-load > /tmp/aimock.log 2>&1 &
4646
for i in $(seq 1 60); do
4747
if curl -sf http://localhost:4010/health; then
4848
echo "aimock ready"

examples/integrations/adk/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ services:
33
image: ghcr.io/copilotkit/aimock:latest
44
volumes:
55
- ./fixtures:/fixtures:ro
6-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
6+
command:
7+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
78

89
agent:
910
build:

examples/integrations/agno/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ services:
33
image: ghcr.io/copilotkit/aimock:latest
44
volumes:
55
- ./fixtures:/fixtures:ro
6-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
6+
command:
7+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
78

89
agent:
910
build:

examples/integrations/crewai-crews/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ services:
33
image: ghcr.io/copilotkit/aimock:latest
44
volumes:
55
- ./fixtures:/fixtures:ro
6-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
6+
command:
7+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
78

89
agent:
910
build:

examples/integrations/langgraph-fastapi/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ services:
66
image: ghcr.io/copilotkit/aimock:latest
77
volumes:
88
- ./fixtures:/fixtures:ro
9-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
9+
command:
10+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
1011

1112
agent:
1213
build:

examples/integrations/langgraph-js/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ services:
66
image: ghcr.io/copilotkit/aimock:latest
77
volumes:
88
- ./fixtures:/fixtures:ro
9-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
9+
command:
10+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
1011

1112
agent:
1213
build:

examples/integrations/langgraph-python-threads/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ services:
66
image: ghcr.io/copilotkit/aimock:latest
77
volumes:
88
- ./fixtures:/fixtures:ro
9-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
9+
command:
10+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
1011

1112
agent:
1213
build:

examples/integrations/langgraph-python/docker-compose.test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ services:
66
image: ghcr.io/copilotkit/aimock:latest
77
volumes:
88
- ./fixtures:/fixtures:ro
9-
command: ["--fixtures", "/fixtures", "--host", "0.0.0.0"]
9+
command:
10+
["--fixtures", "/fixtures", "--host", "0.0.0.0", "--validate-on-load"]
1011

1112
agent:
1213
build:

0 commit comments

Comments
 (0)