Skip to content

Commit 22a8446

Browse files
AlemTuzlakAlemTuzlak
committed
fix(showcase/aimock): correct D5 fixture matching for agent-config + shared-state-streaming
agent-config: drop the "introduce yourself per your config" baseline — it substring-matched every prefixed probe prompt (tone:, expertise:, responseLength:), forcing byte-identical responses and masking the per-knob behavior the cell is meant to demonstrate. shared-state-streaming: add toolCallId-keyed follow-up fixtures (poem/email/quantum) above each first-turn entry so the matcher returns content-only after the tool result lands. Without these the unchanged last user message kept re-matching the first-turn fixture, the agent re-fired write_document, and langgraph hit its 100-step recursion limit. The frontend state-binding regression that prevents the streamed document from rendering is a separate framework-level issue. Synced to showcase/aimock/d5-all.json so replay-mode picks up both fixes.
1 parent 462e200 commit 22a8446

3 files changed

Lines changed: 154 additions & 105 deletions

File tree

‎showcase/aimock/d5-all.json‎

Lines changed: 73 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -377,14 +377,6 @@
377377
]
378378
}
379379
},
380-
{
381-
"match": {
382-
"userMessage": "introduce yourself per your config"
383-
},
384-
"response": {
385-
"content": "Hello — I'm operating with the configured tone, expertise, and responseLength forwarded by the CopilotKit provider. Adjust the controls above the chat and my system prompt will rebuild on the next turn."
386-
}
387-
},
388380
{
389381
"match": {
390382
"userMessage": "good name for a goldfish"
@@ -1050,12 +1042,84 @@
10501042
"content": "Reasoning: first, I identified the question requires step-by-step thinking. Then, I broke it into sub-steps and worked through each one. Finally, I aggregated the partial answers into a single response. The reasoning block above the answer demonstrates intermediate-thought rendering."
10511043
}
10521044
},
1045+
{
1046+
"match": {
1047+
"userMessage": "poem about autumn leaves",
1048+
"toolCallId": "call_d5_write_document_poem_001"
1049+
},
1050+
"response": {
1051+
"content": "Done — the poem has been written into the shared document state."
1052+
}
1053+
},
1054+
{
1055+
"match": {
1056+
"userMessage": "poem about autumn leaves"
1057+
},
1058+
"response": {
1059+
"content": "Streaming the poem now.",
1060+
"toolCalls": [
1061+
{
1062+
"id": "call_d5_write_document_poem_001",
1063+
"name": "write_document",
1064+
"arguments": "{\"document\":\"Crimson and amber in slow descent, / each leaf a quiet ledger of summer spent. / The wind, a courier with nothing to say, / files them gently into the morning's gray. / Somewhere a kettle hums, and afternoons grow brief — / autumn keeps its books in vermilion and gold leaf.\"}"
1065+
}
1066+
]
1067+
}
1068+
},
1069+
{
1070+
"match": {
1071+
"userMessage": "polite email declining",
1072+
"toolCallId": "call_d5_write_document_email_001"
1073+
},
1074+
"response": {
1075+
"content": "Done — the decline-email draft has been written into the shared document state."
1076+
}
1077+
},
1078+
{
1079+
"match": {
1080+
"userMessage": "polite email declining"
1081+
},
1082+
"response": {
1083+
"content": "Drafting the email now.",
1084+
"toolCalls": [
1085+
{
1086+
"id": "call_d5_write_document_email_001",
1087+
"name": "write_document",
1088+
"arguments": "{\"document\":\"Hi — thanks for sending the invite for Tuesday afternoon. Unfortunately I won't be able to make it this week. I'd love to find time later in the month if your schedule allows. In the meantime, feel free to send any pre-reads my way and I'll review them async so we don't lose momentum. Best, [name]\"}"
1089+
}
1090+
]
1091+
}
1092+
},
1093+
{
1094+
"match": {
1095+
"userMessage": "quantum computing for a curious teenager",
1096+
"toolCallId": "call_d5_write_document_quantum_001"
1097+
},
1098+
"response": {
1099+
"content": "Done — the quantum-computing explainer has been written into the shared document state."
1100+
}
1101+
},
1102+
{
1103+
"match": {
1104+
"userMessage": "quantum computing for a curious teenager"
1105+
},
1106+
"response": {
1107+
"content": "Streaming the explainer now.",
1108+
"toolCalls": [
1109+
{
1110+
"id": "call_d5_write_document_quantum_001",
1111+
"name": "write_document",
1112+
"arguments": "{\"document\":\"A regular computer stores information in bits — tiny switches that are either on (1) or off (0). A quantum computer uses qubits, which can sit in a fuzzy superposition of both states at once until you check them. Stack many qubits together and they can explore lots of possibilities in parallel, which is why people are excited.\\n\\nThis doesn't make quantum computers faster at everything. They're great at problems with hidden structure — like factoring big numbers, simulating molecules, or searching certain databases — but useless for, say, opening Excel. Today's machines are noisy and small, so we mostly use them to test ideas rather than replace your laptop.\"}"
1113+
}
1114+
]
1115+
}
1116+
},
10531117
{
10541118
"match": {
10551119
"userMessage": "stream the counter to 5"
10561120
},
10571121
"response": {
1058-
"content": "Streaming state updates: counter advanced from 0 through 1, 2, 3, 4 to 5. Each intermediate value was reflected in the shared state and visible to the UI mid-stream."
1122+
"content": "Streaming counter from 0 to 5."
10591123
}
10601124
},
10611125
{
@@ -2070,54 +2134,6 @@
20702134
]
20712135
}
20722136
},
2073-
{
2074-
"match": {
2075-
"userMessage": "poem about autumn leaves",
2076-
"toolName": "write_document"
2077-
},
2078-
"response": {
2079-
"content": "Streaming the poem now.",
2080-
"toolCalls": [
2081-
{
2082-
"id": "call_d5_write_document_poem_001",
2083-
"name": "write_document",
2084-
"arguments": "{\"document\":\"Crimson and amber in slow descent, / each leaf a quiet ledger of summer spent. / The wind, a courier with nothing to say, / files them gently into the morning's gray. / Somewhere a kettle hums, and afternoons grow brief — / autumn keeps its books in vermilion and gold leaf.\"}"
2085-
}
2086-
]
2087-
}
2088-
},
2089-
{
2090-
"match": {
2091-
"userMessage": "polite email declining",
2092-
"toolName": "write_document"
2093-
},
2094-
"response": {
2095-
"content": "Drafting the email now.",
2096-
"toolCalls": [
2097-
{
2098-
"id": "call_d5_write_document_email_001",
2099-
"name": "write_document",
2100-
"arguments": "{\"document\":\"Hi — thanks for sending the invite for Tuesday afternoon. Unfortunately I won't be able to make it this week. I'd love to find time later in the month if your schedule allows. In the meantime, feel free to send any pre-reads my way and I'll review them async so we don't lose momentum. Best, [name]\"}"
2101-
}
2102-
]
2103-
}
2104-
},
2105-
{
2106-
"match": {
2107-
"userMessage": "quantum computing for a curious teenager",
2108-
"toolName": "write_document"
2109-
},
2110-
"response": {
2111-
"content": "Streaming the explainer now.",
2112-
"toolCalls": [
2113-
{
2114-
"id": "call_d5_write_document_quantum_001",
2115-
"name": "write_document",
2116-
"arguments": "{\"document\":\"A regular computer stores information in bits — tiny switches that are either on (1) or off (0). A quantum computer uses qubits, which can sit in a fuzzy superposition of both states at once until you check them. Stack many qubits together and they can explore lots of possibilities in parallel, which is why people are excited.\\n\\nThis doesn't make quantum computers faster at everything. They're great at problems with hidden structure — like factoring big numbers, simulating molecules, or searching certain databases — but useless for, say, opening Excel. Today's machines are noisy and small, so we mostly use them to test ideas rather than replace your laptop.\"}"
2117-
}
2118-
]
2119-
}
2120-
},
21212137
{
21222138
"match": {
21232139
"userMessage": "What do you know about me from my context"

‎showcase/harness/fixtures/d5/agent-config.json‎

Lines changed: 53 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,53 @@
1-
{
2-
"_comment": "D5 fixture for /demos/agent-config. Six prompts, three knob pairs (tone / expertise / responseLength), with each value producing a distinct response. Substring keys are uncommon enough not to collide with adjacent fixtures in d5-all.json.",
3-
"fixtures": [
4-
{
5-
"match": { "userMessage": "tone:professional" },
6-
"response": {
7-
"content": "Greetings. I am operating in professional tone. I will provide measured, formal responses calibrated to your stated preferences and refrain from colloquialisms in my replies."
8-
}
9-
},
10-
{
11-
"match": { "userMessage": "tone:casual" },
12-
"response": {
13-
"content": "Hey! Casual mode here — I'll keep things friendly and easygoing. Just shoot me whatever you want to know and I'll riff on it without sounding like a press release."
14-
}
15-
},
16-
{
17-
"match": { "userMessage": "expertise:beginner" },
18-
"response": {
19-
"content": "Sure! Think of CopilotKit as a friendly toolkit. It helps you add an AI helper to your app. The helper can answer questions, run small tasks, and show buttons or charts when needed."
20-
}
21-
},
22-
{
23-
"match": { "userMessage": "expertise:expert" },
24-
"response": {
25-
"content": "CopilotKit composes a runtime adapter (Express/Hono) over the AG-UI SSE protocol; the React client wires hooks (useFrontendTool, useAgentContext) into a typed agent runner. The architecture front-runs round-trip latency by streaming TEXT_MESSAGE_CHUNK and TOOL_CALL events on the same channel."
26-
}
27-
},
28-
{
29-
"match": { "userMessage": "responseLength:concise" },
30-
"response": {
31-
"content": "Agent context is a typed payload sent each turn."
32-
}
33-
},
34-
{
35-
"match": { "userMessage": "responseLength:detailed" },
36-
"response": {
37-
"content": "Agent context is a typed payload published from the frontend on every turn through the useAgentContext hook. The payload is forwarded into the agent's runtime context (LangGraph 0.6 introduced the `context` channel as the supported relay for per-run frontend-supplied data; legacy `properties` flowed via `forwardedProps` and did not land in `RunnableConfig`). On the Python side, CopilotKitMiddleware reads the value off the runtime context, then routes it into the system-prompt builder so the model sees the user's tone, expertise, and length preferences before each call. The result is per-turn behavior change without a model swap."
38-
}
39-
},
40-
{
41-
"match": { "userMessage": "introduce yourself per your config" },
42-
"response": {
43-
"content": "Hello — I'm operating with the configured tone, expertise, and responseLength forwarded by the CopilotKit provider. Adjust the controls above the chat and my system prompt will rebuild on the next turn."
44-
}
45-
}
46-
]
47-
}
1+
{
2+
"_comment": "D5 fixture for /demos/agent-config. Six prompts, three knob pairs (tone / expertise / responseLength), with each value producing a distinct response. Substring keys are uncommon enough not to collide with adjacent fixtures in d5-all.json.",
3+
"fixtures": [
4+
{
5+
"match": {
6+
"userMessage": "tone:professional"
7+
},
8+
"response": {
9+
"content": "Greetings. I am operating in professional tone. I will provide measured, formal responses calibrated to your stated preferences and refrain from colloquialisms in my replies."
10+
}
11+
},
12+
{
13+
"match": {
14+
"userMessage": "tone:casual"
15+
},
16+
"response": {
17+
"content": "Hey! Casual mode here — I'll keep things friendly and easygoing. Just shoot me whatever you want to know and I'll riff on it without sounding like a press release."
18+
}
19+
},
20+
{
21+
"match": {
22+
"userMessage": "expertise:beginner"
23+
},
24+
"response": {
25+
"content": "Sure! Think of CopilotKit as a friendly toolkit. It helps you add an AI helper to your app. The helper can answer questions, run small tasks, and show buttons or charts when needed."
26+
}
27+
},
28+
{
29+
"match": {
30+
"userMessage": "expertise:expert"
31+
},
32+
"response": {
33+
"content": "CopilotKit composes a runtime adapter (Express/Hono) over the AG-UI SSE protocol; the React client wires hooks (useFrontendTool, useAgentContext) into a typed agent runner. The architecture front-runs round-trip latency by streaming TEXT_MESSAGE_CHUNK and TOOL_CALL events on the same channel."
34+
}
35+
},
36+
{
37+
"match": {
38+
"userMessage": "responseLength:concise"
39+
},
40+
"response": {
41+
"content": "Agent context is a typed payload sent each turn."
42+
}
43+
},
44+
{
45+
"match": {
46+
"userMessage": "responseLength:detailed"
47+
},
48+
"response": {
49+
"content": "Agent context is a typed payload published from the frontend on every turn through the useAgentContext hook. The payload is forwarded into the agent's runtime context (LangGraph 0.6 introduced the `context` channel as the supported relay for per-run frontend-supplied data; legacy `properties` flowed via `forwardedProps` and did not land in `RunnableConfig`). On the Python side, CopilotKitMiddleware reads the value off the runtime context, then routes it into the system-prompt builder so the model sees the user's tone, expertise, and length preferences before each call. The result is per-turn behavior change without a model swap."
50+
}
51+
}
52+
]
53+
}

‎showcase/harness/fixtures/d5/shared-state-streaming.json‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,15 @@
11
{
2-
"_comment": "D5 fixture for /demos/shared-state-streaming. Three pill prompts produce `write_document` tool calls with non-trivial content payloads (≥ 100 chars) so the document streams into shared state and the DocumentView's char-count assertion can verify a substantive document was written. `arguments` is shown JSON-stringified here for parity with the OpenAI wire shape; aimock's `normalizeResponse` (see node_modules/@copilotkit/aimock/dist/fixture-loader) auto-stringifies object-valued arguments at load time, so raw-object form would also work.",
2+
"_comment": "D5 fixture for /demos/shared-state-streaming. Three pill prompts produce `write_document` tool calls with non-trivial content payloads (≥ 100 chars) so the document streams into shared state and the DocumentView's char-count assertion can verify a substantive document was written. Each pill has TWO entries: a first-turn (matches by userMessage substring) returning toolCall + content with a STATIC tool_call_id, and a follow-up keyed on userMessage + that toolCallId returning content-only. The follow-up MUST appear ABOVE the first-turn entry so the matcher returns it after the tool result is appended (otherwise the unchanged last user message keeps re-matching the first-turn fixture, the agent re-fires write_document, and langgraph runs into its 100-step recursion limit). See showcase/harness/fixtures/d5/README.md (\"Mid-loop re-invocations after a tool call\"). `arguments` is shown JSON-stringified here for parity with the OpenAI wire shape; aimock auto-stringifies object-valued arguments at load time, so raw-object form would also work.",
33
"fixtures": [
4+
{
5+
"match": {
6+
"userMessage": "poem about autumn leaves",
7+
"toolCallId": "call_d5_write_document_poem_001"
8+
},
9+
"response": {
10+
"content": "Done — the poem has been written into the shared document state."
11+
}
12+
},
413
{
514
"match": { "userMessage": "poem about autumn leaves" },
615
"response": {
@@ -14,6 +23,15 @@
1423
]
1524
}
1625
},
26+
{
27+
"match": {
28+
"userMessage": "polite email declining",
29+
"toolCallId": "call_d5_write_document_email_001"
30+
},
31+
"response": {
32+
"content": "Done — the decline-email draft has been written into the shared document state."
33+
}
34+
},
1735
{
1836
"match": { "userMessage": "polite email declining" },
1937
"response": {
@@ -27,6 +45,15 @@
2745
]
2846
}
2947
},
48+
{
49+
"match": {
50+
"userMessage": "quantum computing for a curious teenager",
51+
"toolCallId": "call_d5_write_document_quantum_001"
52+
},
53+
"response": {
54+
"content": "Done — the quantum-computing explainer has been written into the shared document state."
55+
}
56+
},
3057
{
3158
"match": { "userMessage": "quantum computing for a curious teenager" },
3259
"response": {

0 commit comments

Comments
 (0)