diff --git a/docs/drift-detection.html b/docs/drift-detection.html index 46903a2..0a9cf79 100644 --- a/docs/drift-detection.html +++ b/docs/drift-detection.html @@ -281,20 +281,92 @@

All three agree

-

Running Drift Tests

+

What Drift Looks Like

-
-
Run drift tests shell
-
# Set API keys for providers you want to test
-export OPENAI_API_KEY=sk-...
-export ANTHROPIC_API_KEY=sk-ant-...
-export GOOGLE_API_KEY=AI...
-
-# Run all drift tests
-pnpm test:drift
+        
 
-# Run for a specific provider
-pnpm test:drift -- --grep "OpenAI Chat"
+ +
+
$ pnpm test:drift
+ [critical] + AIMOCK DRIFT — field in SDK + real API but missing + from mock
+ Path:    choices[].message.refusal
+ SDK:     null    + Real: null    + Mock: <absent>
+
+ [critical] + TYPE MISMATCH — real API and mock disagree on + type
+ Path:    content[].input
+ SDK:     object    + Real: object    + Mock: string
+
+ [warning] + PROVIDER ADDED FIELD — in real API but not in SDK + or mock
+ Path:    choices[].message.annotations
+ SDK:     <absent> +    Real: array    + Mock: <absent>
+
+ + 2 critical (test fails) · 1 warning (logged) · detected before any user + reported it

Test Files

diff --git a/docs/index.html b/docs/index.html index 6c0aceb..66ef2ff 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1363,10 +1363,7 @@

Chaos Testing

📊

Drift Detection

-

- Daily CI validation against real APIs. Know immediately when provider behavior changes - break your fixtures. -

+

Fixtures stay accurate as providers evolve. Fixes ship before your tests break.

@@ -1375,10 +1372,10 @@

Drift Detection

-

Verified against real APIs, every day

+

Your mocks never go stale

- aimock's drift detection runs daily against live provider APIs. When response formats - change, you know immediately — not when your tests break in production. + LLM providers change their response formats without warning. aimock catches it within 24 + hours and ships the fix before your tests break.

@@ -1386,24 +1383,24 @@

Verified against real APIs, every day

1

Real API calls

- Daily CI hits actual OpenAI, Anthropic, Gemini endpoints to capture current response - formats. + Daily CI hits actual OpenAI, Anthropic, Gemini endpoints to detect response format + changes.

2
-

Response validation

+

Compare & detect

- Compares real responses against aimock's fixture format. Schema changes are caught - instantly. + Response schemas are validated against aimock's builders. Any drift is flagged + automatically.

3
-

Auto-remediation

+

Fix & ship

- Drift detected → PR opened → fixtures, skills, and docs updated - automatically. Zero manual effort. + Drift detected → fixtures, builders, skills, and docs updated → PR shipped. + Always current, within a day.