Skip to content

Commit d2d7224

Browse files
committed
docs(showcase): add GOTCHAS.md — framework edge cases from D5 grind
What we learned getting 18 integrations to D5 green — and what was still wrong even when green. Cross-framework patterns (V1/V2, testids, runtime hoisting, agent naming, multimodal shims, Pydantic models), per-framework edge cases for all 15 frameworks, aimock fixture gotchas, and the 6 things that were "green but wrong."
1 parent 7f997a9 commit d2d7224

1 file changed

Lines changed: 130 additions & 0 deletions

File tree

showcase/GOTCHAS.md

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Showcase GOTCHAS — Framework & Integration Edge Cases
2+
3+
What we learned from getting all 18 integrations to D5 green. Many of these are things that were "green" but still wrong — passing probes while the underlying wiring was fragile, framework-specific, or relying on coincidence. This document exists so we don't re-learn these when rebuilding.
4+
5+
---
6+
7+
## Cross-Framework Patterns
8+
9+
**V1 vs V2 CopilotKit imports cause silent failures.** V1 is `@copilotkit/react-core`, V2 is `@copilotkit/react-core/v2`. Mixing V1 provider with V2 hooks (e.g., `useRenderTool`) silently fails — the tool rendering pipeline never wires up. Agent discovery also breaks: V2 runtime needs V2 provider. Found on: ms-agent-dotnet (auth), built-in-agent (interrupts), spring-ai (tool-rendering).
10+
11+
**Custom `assistantMessage` slot renderers must carry `data-testid="copilot-assistant-message"`.** The byoc-hashbrown demo overrides the slot with a HashBrown renderer. Without the testid, the probe (and any external consumer) sees 0 assistant messages. Every integration's hashbrown-renderer.tsx needed this fix independently. This is the #1 argument for a shared frontend.
12+
13+
**`copilotRuntimeNextJSAppRouterEndpoint()` must be hoisted to module scope.** Calling it inside the POST handler (per-request) causes `handleServiceAdapter` to repeatedly re-wrap `runtime.agents` in Promise layers. Under concurrent requests (/info + agent/run), this creates a race condition where the agents list is stale. Found on: google-adk (all 6 dedicated routes).
14+
15+
**Agent names must match exactly between frontend and backend.** `useAgent("agentic-chat-reasoning")` must match the backend registration. Dashes vs underscores, trailing hyphens, typos — all cause silent "Agent not found" errors that crash the page via React error boundary.
16+
17+
**`onRunInitialized` multimodal shim is framework-dependent.** langgraph-python NEEDS it (the `@ag-ui/langgraph` converter only understands legacy `binary` parts). langroid does NOT need it (speaks AG-UI directly — adding the shim causes double-encoding). Per-framework boolean, not a universal.
18+
19+
**Content parts from AG-UI arrive as Pydantic model instances, not dicts.** `isinstance(part, dict)` silently drops them. Must check `hasattr(part, "model_dump")` and call `model_dump(by_alias=True)`. Affected: langroid, ms-agent-python, pydantic-ai.
20+
21+
**`from __future__ import annotations` breaks Pydantic tool schemas.** PEP 563 makes all annotations strings. When LlamaIndex `AGUIChatWorkflow` passes `backend_tools` to Pydantic for schema generation, `Annotated[str, "..."]` is a raw string instead of a resolved type. Affected: llamaindex, crewai-crews, pydantic-ai, ag2. Fix: remove the import from files defining tools.
22+
23+
---
24+
25+
## Per-Framework Edge Cases
26+
27+
### langgraph-python (Reference — always compare against this)
28+
29+
- `a2ui_dynamic` graph owns `generate_a2ui` tool — runtime MUST NOT auto-inject (`injectA2UITool: false`). Double-injection confuses the LLM.
30+
- `server.mjs` must register ALL graphs from `langgraph.json`. We found it registering 5 of 25 — every unregistered graph returned 404.
31+
- Health probe uses `/ok` (langgraph-cli convention), not `/health`.
32+
- Version pinning: `langchain>=1.2.0` imports from `langgraph.runtime.ExecutionInfo` which doesn't exist in `langgraph==1.0.5`.
33+
34+
### langgraph-typescript
35+
36+
- Same server.mjs graph registration issue as langgraph-python.
37+
- Trailing slash on `deploymentUrl` matters for dedicated API routes. Missing it causes 404.
38+
- esbuild architecture mismatch on ARM Mac Docker builds. Passes in CI (Depot x86), fails locally on Apple Silicon.
39+
40+
### agno
41+
42+
- `reasoning=True` does multi-call chain-of-thought which breaks aimock (only first call matches). Disable for aimock-backed tests.
43+
- Agno's stock AGUI handler emits `STEP_STARTED`/`STEP_FINISHED` for reasoning — CopilotKit ignores these. The `reasoningMessage` slot requires `REASONING_MESSAGE_*` events. We built a custom handler, then reverted to stock AGUI.
44+
- Internal tool execution creates infinite fixture loops (same pattern as AG2).
45+
46+
### spring-ai
47+
48+
- **Java backend** — Maven build, fundamentally different toolchain.
49+
- `StreamingToolAgent.streamFirstTurn()` must include `toolCallbacks` with `internalToolExecutionEnabled=false`. Without this, aimock can't match `toolName: "get_weather"` — falls through to text-only fixture, weather card never renders.
50+
- AG-UI Java SDK not on Maven Central. Must clone and `mvn install` in Dockerfile.
51+
52+
### mastra
53+
54+
- **JS object shorthand key trap:** `{ weatherTool }` expands to function name `"weatherTool"`, not `"get_weather"`. Must use explicit keys: `{ get_weather: weatherTool }`.
55+
- `byocHashbrownAgent` needs its own dedicated agent with the hashbrown system prompt. The `weatherAgent` produces plain text → `useJsonParser` returns null → empty dashboard → timeout.
56+
- ~280s cold start (V8 JIT + Mastra boot). Watchdog can kill it before ready.
57+
58+
### ms-agent-python
59+
60+
- `AgentFrameworkAgent.run()` expects `input_data: dict`. The `_MultimodalAgent` override used `*args/**kwargs``TypeError` at runtime.
61+
- The override must `yield` events (async generator), not `return` (coroutine).
62+
- OpenAI `store=True` breaks aimock fixture matching. Set `store=False`.
63+
64+
### ms-agent-dotnet
65+
66+
- C# / .NET backend.
67+
- Auth page had V1 `CopilotKit` import → agent discovery failed → "Agent not found".
68+
69+
### built-in-agent
70+
71+
- **No Python backend.** TanStack AI `BuiltInAgent` runs in-process in Next.js.
72+
- `type: "tanstack"` with `convertTanStackStream` has a `runFinished` flag that blocks ALL events after first `RUN_FINISHED`. For byoc, must use `type: "custom"`.
73+
- OpenAI Responses API does NOT support `response_format: { type: "json_object" }` through TanStack adapter. The call silently fails — aimock never receives a request.
74+
75+
### crewai-crews
76+
77+
- `from __future__ import annotations` breaks `InterruptScheduling` import stubs in tests.
78+
- Backend tool execution doesn't cycle back to aimock for text follow-up. Known adapter limitation.
79+
80+
### pydantic-ai
81+
82+
- `_classify_binary_part()` has the `isinstance(part, dict)` bug. Pydantic models from AG-UI need `model_dump()`.
83+
- `starlette>=1.0.0` removes `on_startup`. Pin `starlette<1.0.0`.
84+
85+
### llamaindex
86+
87+
- `from __future__ import annotations` breaks Pydantic tool schema validation specifically when `backend_tools` are present but the response is text-only.
88+
89+
### langroid
90+
91+
- Custom AGUI handler (hand-written SSE, not a framework adapter).
92+
- `_normalize_part()` must handle Pydantic model instances via `model_dump(by_alias=True)`.
93+
- Does NOT need `onRunInitialized` multimodal shim.
94+
95+
### AG2
96+
97+
- `AGUIStream` requires plain string content. Multipart arrays cause 400 errors. `ContentFlattenerShim` handles conversion.
98+
- Internal tool execution + aimock = infinite loop. Fix: `max_consecutive_auto_reply` or `hasToolResult` in fixtures.
99+
100+
### google-adk
101+
102+
- **Underscores required for ALL agent names.** Every other framework uses dashes.
103+
- All 6 dedicated route files called `copilotRuntimeNextJSAppRouterEndpoint()` per-request → race condition. Fixed by hoisting to module scope.
104+
105+
### claude-sdk-python
106+
107+
- Transient "empty assistant text" flaps (fc=1, self-healing, not reproducible locally). Suspected SSE stream interruption on Railway.
108+
109+
---
110+
111+
## Aimock & Fixture Edge Cases
112+
113+
**Check fixtures FIRST.** When an agent misbehaves through aimock, the fixture determines behavior — the real LLM is never consulted.
114+
115+
**`sequenceIndex` counters are global.** They persist across all test runs within the same aimock process. Use `hasToolResult` (stateless) instead for fixtures shared across integrations.
116+
117+
**Tool-rendering fixtures need `toolName` in match criteria.** If the request doesn't include tool definitions, the fixture falls through to text-only. Spring-ai omitted tools; mastra's shorthand keys produced wrong function names.
118+
119+
**PDF turn is fragile.** Two-turn multimodal probe: if turn 2's message doesn't match the PDF fixture, the image fixture matches instead. The PDF fixture must be the most specific match.
120+
121+
---
122+
123+
## What Was "Green" But Still Wrong
124+
125+
1. **18 copies of identical frontend code** — every fix was a blitz. One missed integration = one regression.
126+
2. **V1/V2 imports inconsistent** — some pages used V1 provider with V2 hooks and happened to work because the feature didn't exercise the broken path.
127+
3. **Most integrations still on V1 runtime API**`copilotRuntimeNextJSAppRouterEndpoint` + `ExperimentalEmptyAdapter` instead of V2's `createCopilotRuntimeHandler` + `InMemoryAgentRunner`. Only `built-in-agent` fully uses V2. The V1 API has the per-request race condition (hoisting to module scope was a band-aid, not a migration).
128+
4. **Agent name mismatches masked by default fallback** — features passed because the runtime fell back to `"default"`, not because the correct agent was wired.
129+
5. **Missing testids on custom renderers** — probe assertions were weak enough to pass via fallback selectors.
130+
6. **`onRunInitialized` shim applied where unnecessary** — worked by coincidence because legacy format round-tripped correctly.

0 commit comments

Comments
 (0)