Skip to content

Commit af9fbd5

Browse files
tylerslatontylerslaton
authored
fix(showcase/aimock): correct D5 fixture matching for agent-config + shared-state-streaming (CopilotKit#4737)
## Summary Two D5 fixture-correctness fixes for the langgraph-python column. Together they flip `agent-config` to D5 and unblock the recursion-loop failure mode for `shared-state-streaming` (the cell still has a separate framework-level state-binding regression that prevents it from rendering — tracked separately). - **agent-config**: drop the "introduce yourself per your config" baseline fixture. It substring-matched every prefixed probe prompt (`tone:professional`, `expertise:beginner`, `responseLength:concise`, …), forcing byte-identical responses across all six knob probes and masking the per-knob behavior the cell is meant to demonstrate. - **shared-state-streaming**: add toolCallId-keyed follow-up fixtures (poem / email / quantum) above each first-turn entry so aimock returns content-only after the tool result lands. Without these, the unchanged last user message kept re-matching the first-turn fixture, the agent re-fired `write_document`, and langgraph hit its 100-step recursion limit. Synced to `showcase/aimock/d5-all.json` so replay-mode picks up both fixes. ## Test plan - [x] Local replay: `agent-config` flips GREEN (each knob probe receives its dedicated fixture) - [x] Local replay: `shared-state-streaming` no longer hits `GraphRecursionError`; remaining failure is the framework state-binding bug (separate work) - [x] aimock fixture count went up by the expected amount on container restart (`Loaded N fixture(s)`) - [x] lefthook hooks pass (lint-fix, sync-lockfile, test-and-check-packages all clean)
2 parents 3aa64b0 + ca22e38 commit af9fbd5

3 files changed

Lines changed: 108 additions & 56 deletions

File tree

‎showcase/aimock/d5-all.json‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -377,14 +377,6 @@
377377
]
378378
}
379379
},
380-
{
381-
"match": {
382-
"userMessage": "introduce yourself per your config"
383-
},
384-
"response": {
385-
"content": "Hello — I'm operating with the configured tone, expertise, and responseLength forwarded by the CopilotKit provider. Adjust the controls above the chat and my system prompt will rebuild on the next turn."
386-
}
387-
},
388380
{
389381
"match": {
390382
"userMessage": "good name for a goldfish"
@@ -2070,6 +2062,15 @@
20702062
]
20712063
}
20722064
},
2065+
{
2066+
"match": {
2067+
"userMessage": "poem about autumn leaves",
2068+
"toolCallId": "call_d5_write_document_poem_001"
2069+
},
2070+
"response": {
2071+
"content": "Done — the poem has been written into the shared document state."
2072+
}
2073+
},
20732074
{
20742075
"match": {
20752076
"userMessage": "poem about autumn leaves",
@@ -2086,6 +2087,15 @@
20862087
]
20872088
}
20882089
},
2090+
{
2091+
"match": {
2092+
"userMessage": "polite email declining",
2093+
"toolCallId": "call_d5_write_document_email_001"
2094+
},
2095+
"response": {
2096+
"content": "Done — the decline-email draft has been written into the shared document state."
2097+
}
2098+
},
20892099
{
20902100
"match": {
20912101
"userMessage": "polite email declining",
@@ -2102,6 +2112,15 @@
21022112
]
21032113
}
21042114
},
2115+
{
2116+
"match": {
2117+
"userMessage": "quantum computing for a curious teenager",
2118+
"toolCallId": "call_d5_write_document_quantum_001"
2119+
},
2120+
"response": {
2121+
"content": "Done — the quantum-computing explainer has been written into the shared document state."
2122+
}
2123+
},
21052124
{
21062125
"match": {
21072126
"userMessage": "quantum computing for a curious teenager",

‎showcase/harness/fixtures/d5/agent-config.json‎

Lines changed: 53 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,53 @@
1-
{
2-
"_comment": "D5 fixture for /demos/agent-config. Six prompts, three knob pairs (tone / expertise / responseLength), with each value producing a distinct response. Substring keys are uncommon enough not to collide with adjacent fixtures in d5-all.json.",
3-
"fixtures": [
4-
{
5-
"match": { "userMessage": "tone:professional" },
6-
"response": {
7-
"content": "Greetings. I am operating in professional tone. I will provide measured, formal responses calibrated to your stated preferences and refrain from colloquialisms in my replies."
8-
}
9-
},
10-
{
11-
"match": { "userMessage": "tone:casual" },
12-
"response": {
13-
"content": "Hey! Casual mode here — I'll keep things friendly and easygoing. Just shoot me whatever you want to know and I'll riff on it without sounding like a press release."
14-
}
15-
},
16-
{
17-
"match": { "userMessage": "expertise:beginner" },
18-
"response": {
19-
"content": "Sure! Think of CopilotKit as a friendly toolkit. It helps you add an AI helper to your app. The helper can answer questions, run small tasks, and show buttons or charts when needed."
20-
}
21-
},
22-
{
23-
"match": { "userMessage": "expertise:expert" },
24-
"response": {
25-
"content": "CopilotKit composes a runtime adapter (Express/Hono) over the AG-UI SSE protocol; the React client wires hooks (useFrontendTool, useAgentContext) into a typed agent runner. The architecture front-runs round-trip latency by streaming TEXT_MESSAGE_CHUNK and TOOL_CALL events on the same channel."
26-
}
27-
},
28-
{
29-
"match": { "userMessage": "responseLength:concise" },
30-
"response": {
31-
"content": "Agent context is a typed payload sent each turn."
32-
}
33-
},
34-
{
35-
"match": { "userMessage": "responseLength:detailed" },
36-
"response": {
37-
"content": "Agent context is a typed payload published from the frontend on every turn through the useAgentContext hook. The payload is forwarded into the agent's runtime context (LangGraph 0.6 introduced the `context` channel as the supported relay for per-run frontend-supplied data; legacy `properties` flowed via `forwardedProps` and did not land in `RunnableConfig`). On the Python side, CopilotKitMiddleware reads the value off the runtime context, then routes it into the system-prompt builder so the model sees the user's tone, expertise, and length preferences before each call. The result is per-turn behavior change without a model swap."
38-
}
39-
},
40-
{
41-
"match": { "userMessage": "introduce yourself per your config" },
42-
"response": {
43-
"content": "Hello — I'm operating with the configured tone, expertise, and responseLength forwarded by the CopilotKit provider. Adjust the controls above the chat and my system prompt will rebuild on the next turn."
44-
}
45-
}
46-
]
47-
}
1+
{
2+
"_comment": "D5 fixture for /demos/agent-config. Six prompts, three knob pairs (tone / expertise / responseLength), with each value producing a distinct response. Substring keys are uncommon enough not to collide with adjacent fixtures in d5-all.json.",
3+
"fixtures": [
4+
{
5+
"match": {
6+
"userMessage": "tone:professional"
7+
},
8+
"response": {
9+
"content": "Greetings. I am operating in professional tone. I will provide measured, formal responses calibrated to your stated preferences and refrain from colloquialisms in my replies."
10+
}
11+
},
12+
{
13+
"match": {
14+
"userMessage": "tone:casual"
15+
},
16+
"response": {
17+
"content": "Hey! Casual mode here — I'll keep things friendly and easygoing. Just shoot me whatever you want to know and I'll riff on it without sounding like a press release."
18+
}
19+
},
20+
{
21+
"match": {
22+
"userMessage": "expertise:beginner"
23+
},
24+
"response": {
25+
"content": "Sure! Think of CopilotKit as a friendly toolkit. It helps you add an AI helper to your app. The helper can answer questions, run small tasks, and show buttons or charts when needed."
26+
}
27+
},
28+
{
29+
"match": {
30+
"userMessage": "expertise:expert"
31+
},
32+
"response": {
33+
"content": "CopilotKit composes a runtime adapter (Express/Hono) over the AG-UI SSE protocol; the React client wires hooks (useFrontendTool, useAgentContext) into a typed agent runner. The architecture front-runs round-trip latency by streaming TEXT_MESSAGE_CHUNK and TOOL_CALL events on the same channel."
34+
}
35+
},
36+
{
37+
"match": {
38+
"userMessage": "responseLength:concise"
39+
},
40+
"response": {
41+
"content": "Agent context is a typed payload sent each turn."
42+
}
43+
},
44+
{
45+
"match": {
46+
"userMessage": "responseLength:detailed"
47+
},
48+
"response": {
49+
"content": "Agent context is a typed payload published from the frontend on every turn through the useAgentContext hook. The payload is forwarded into the agent's runtime context (LangGraph 0.6 introduced the `context` channel as the supported relay for per-run frontend-supplied data; legacy `properties` flowed via `forwardedProps` and did not land in `RunnableConfig`). On the Python side, CopilotKitMiddleware reads the value off the runtime context, then routes it into the system-prompt builder so the model sees the user's tone, expertise, and length preferences before each call. The result is per-turn behavior change without a model swap."
50+
}
51+
}
52+
]
53+
}

‎showcase/harness/fixtures/d5/shared-state-streaming.json‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,15 @@
11
{
2-
"_comment": "D5 fixture for /demos/shared-state-streaming. Three pill prompts produce `write_document` tool calls with non-trivial content payloads (≥ 100 chars) so the document streams into shared state and the DocumentView's char-count assertion can verify a substantive document was written. `arguments` is shown JSON-stringified here for parity with the OpenAI wire shape; aimock's `normalizeResponse` (see node_modules/@copilotkit/aimock/dist/fixture-loader) auto-stringifies object-valued arguments at load time, so raw-object form would also work.",
2+
"_comment": "D5 fixture for /demos/shared-state-streaming. Three pill prompts produce `write_document` tool calls with non-trivial content payloads (≥ 100 chars) so the document streams into shared state and the DocumentView's char-count assertion can verify a substantive document was written. Each pill has TWO entries: a first-turn (matches by userMessage substring) returning toolCall + content with a STATIC tool_call_id, and a follow-up keyed on userMessage + that toolCallId returning content-only. The follow-up MUST appear ABOVE the first-turn entry so the matcher returns it after the tool result is appended (otherwise the unchanged last user message keeps re-matching the first-turn fixture, the agent re-fires write_document, and langgraph runs into its 100-step recursion limit). See showcase/harness/fixtures/d5/README.md (\"Mid-loop re-invocations after a tool call\"). `arguments` is shown JSON-stringified here for parity with the OpenAI wire shape; aimock auto-stringifies object-valued arguments at load time, so raw-object form would also work.",
33
"fixtures": [
4+
{
5+
"match": {
6+
"userMessage": "poem about autumn leaves",
7+
"toolCallId": "call_d5_write_document_poem_001"
8+
},
9+
"response": {
10+
"content": "Done — the poem has been written into the shared document state."
11+
}
12+
},
413
{
514
"match": { "userMessage": "poem about autumn leaves" },
615
"response": {
@@ -14,6 +23,15 @@
1423
]
1524
}
1625
},
26+
{
27+
"match": {
28+
"userMessage": "polite email declining",
29+
"toolCallId": "call_d5_write_document_email_001"
30+
},
31+
"response": {
32+
"content": "Done — the decline-email draft has been written into the shared document state."
33+
}
34+
},
1735
{
1836
"match": { "userMessage": "polite email declining" },
1937
"response": {
@@ -27,6 +45,15 @@
2745
]
2846
}
2947
},
48+
{
49+
"match": {
50+
"userMessage": "quantum computing for a curious teenager",
51+
"toolCallId": "call_d5_write_document_quantum_001"
52+
},
53+
"response": {
54+
"content": "Done — the quantum-computing explainer has been written into the shared document state."
55+
}
56+
},
3057
{
3158
"match": { "userMessage": "quantum computing for a curious teenager" },
3259
"response": {

0 commit comments

Comments
 (0)