Skip to content

Commit 3f62cd4

Browse files
committed
fix: validate aimock fixtures at load time to prevent runtime 500s
Follow-up to CopilotKit#3971. aimock supports fixture schema validation at startup via --validate-on-load, but it's opt-in. The showcase Dockerfile did not pass the flag, so fixtures with unrecognized response keys (e.g. "text" instead of "content") loaded silently and only failed at request time with HTTP 500. That's what crashed crewai-crews on startup. Changes: - showcase/aimock/Dockerfile: pass --validate-on-load so broken fixtures fail the container boot, not individual requests. - showcase/scripts/__tests__/aimock-fixtures.test.ts: new vitest spec that loads feature-parity.json and smoke.json via @copilotkit/aimock's loadFixtureFile + validateFixtures and asserts zero errors. Runs as part of the existing showcase-validate CI workflow. - showcase/scripts/package.json: add @copilotkit/aimock dependency for the validator import. Verified red-green: with the pre-CopilotKit#3971 broken "text" fixtures, validateFixtures flags 5 errors; post-CopilotKit#3971 it returns zero. Docker red-green: container with an intentionally broken fixture fails to start with "Validation failed: 1 error(s)" and non-zero exit.
1 parent 728363f commit 3f62cd4

4 files changed

Lines changed: 42 additions & 0 deletions

File tree

pnpm-lock.yaml

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

showcase/aimock/Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@ CMD ["--proxy-only", \
66
"--provider-anthropic", "https://api.anthropic.com", \
77
"--provider-gemini", "https://generativelanguage.googleapis.com", \
88
"--fixtures", "/fixtures", \
9+
"--validate-on-load", \
910
"--host", "0.0.0.0", \
1011
"--port", "4010"]
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
import { describe, it, expect } from "vitest";
2+
import path from "path";
3+
import { loadFixtureFile, validateFixtures } from "@copilotkit/aimock";
4+
5+
const AIMOCK_DIR = path.resolve(__dirname, "..", "..", "aimock");
6+
7+
const FIXTURE_FILES = ["feature-parity.json", "smoke.json"];
8+
9+
describe("showcase aimock fixtures", () => {
10+
for (const file of FIXTURE_FILES) {
11+
it(`${file} loads and validates with zero errors`, () => {
12+
const filePath = path.join(AIMOCK_DIR, file);
13+
const fixtures = loadFixtureFile(filePath);
14+
15+
// If loadFixtureFile returns [], the file itself is broken (unreadable,
16+
// invalid JSON, or missing "fixtures" array). Treat as fatal.
17+
expect(
18+
fixtures.length,
19+
`${file} produced 0 fixtures — file is unreadable or malformed`,
20+
).toBeGreaterThan(0);
21+
22+
const results = validateFixtures(fixtures);
23+
const errors = results.filter((r) => r.severity === "error");
24+
25+
if (errors.length > 0) {
26+
const detail = errors
27+
.map((e) => ` [${e.fixtureIndex}] ${e.message}`)
28+
.join("\n");
29+
throw new Error(
30+
`${file} has ${errors.length} fixture validation error(s):\n${detail}`,
31+
);
32+
}
33+
34+
expect(errors).toEqual([]);
35+
});
36+
}
37+
});

showcase/scripts/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
"test:watch": "vitest"
1515
},
1616
"dependencies": {
17+
"@copilotkit/aimock": "^1.10.0",
1718
"@playwright/test": "^1.59.1",
1819
"ajv": "^8.17.1",
1920
"ajv-formats": "^3.0.1",

0 commit comments

Comments
 (0)