|
| 1 | +# Showcase GOTCHAS — Framework & Integration Edge Cases |
| 2 | + |
| 3 | +What we learned from getting all 18 integrations to D5 green. Many of these are things that were "green" but still wrong — passing probes while the underlying wiring was fragile, framework-specific, or relying on coincidence. This document exists so we don't re-learn these when rebuilding. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Cross-Framework Patterns |
| 8 | + |
| 9 | +**V1 vs V2 CopilotKit imports cause silent failures.** V1 is `@copilotkit/react-core`, V2 is `@copilotkit/react-core/v2`. Mixing V1 provider with V2 hooks (e.g., `useRenderTool`) silently fails — the tool rendering pipeline never wires up. Agent discovery also breaks: V2 runtime needs V2 provider. Found on: ms-agent-dotnet (auth), built-in-agent (interrupts), spring-ai (tool-rendering). |
| 10 | + |
| 11 | +**Custom `assistantMessage` slot renderers must carry `data-testid="copilot-assistant-message"`.** The byoc-hashbrown demo overrides the slot with a HashBrown renderer. Without the testid, the probe (and any external consumer) sees 0 assistant messages. Every integration's hashbrown-renderer.tsx needed this fix independently. This is the #1 argument for a shared frontend. |
| 12 | + |
| 13 | +**`copilotRuntimeNextJSAppRouterEndpoint()` must be hoisted to module scope.** Calling it inside the POST handler (per-request) causes `handleServiceAdapter` to repeatedly re-wrap `runtime.agents` in Promise layers. Under concurrent requests (/info + agent/run), this creates a race condition where the agents list is stale. Found on: google-adk (all 6 dedicated routes). |
| 14 | + |
| 15 | +**Agent names must match exactly between frontend and backend.** `useAgent("agentic-chat-reasoning")` must match the backend registration. Dashes vs underscores, trailing hyphens, typos — all cause silent "Agent not found" errors that crash the page via React error boundary. |
| 16 | + |
| 17 | +**`onRunInitialized` multimodal shim is framework-dependent.** langgraph-python NEEDS it (the `@ag-ui/langgraph` converter only understands legacy `binary` parts). langroid does NOT need it (speaks AG-UI directly — adding the shim causes double-encoding). Per-framework boolean, not a universal. |
| 18 | + |
| 19 | +**Content parts from AG-UI arrive as Pydantic model instances, not dicts.** `isinstance(part, dict)` silently drops them. Must check `hasattr(part, "model_dump")` and call `model_dump(by_alias=True)`. Affected: langroid, ms-agent-python, pydantic-ai. |
| 20 | + |
| 21 | +**`from __future__ import annotations` breaks Pydantic tool schemas.** PEP 563 makes all annotations strings. When LlamaIndex `AGUIChatWorkflow` passes `backend_tools` to Pydantic for schema generation, `Annotated[str, "..."]` is a raw string instead of a resolved type. Affected: llamaindex, crewai-crews, pydantic-ai, ag2. Fix: remove the import from files defining tools. |
| 22 | + |
| 23 | +--- |
| 24 | + |
| 25 | +## Per-Framework Edge Cases |
| 26 | + |
| 27 | +### langgraph-python (Reference — always compare against this) |
| 28 | + |
| 29 | +- `a2ui_dynamic` graph owns `generate_a2ui` tool — runtime MUST NOT auto-inject (`injectA2UITool: false`). Double-injection confuses the LLM. |
| 30 | +- `server.mjs` must register ALL graphs from `langgraph.json`. We found it registering 5 of 25 — every unregistered graph returned 404. |
| 31 | +- Health probe uses `/ok` (langgraph-cli convention), not `/health`. |
| 32 | +- Version pinning: `langchain>=1.2.0` imports from `langgraph.runtime.ExecutionInfo` which doesn't exist in `langgraph==1.0.5`. |
| 33 | + |
| 34 | +### langgraph-typescript |
| 35 | + |
| 36 | +- Same server.mjs graph registration issue as langgraph-python. |
| 37 | +- Trailing slash on `deploymentUrl` matters for dedicated API routes. Missing it causes 404. |
| 38 | +- esbuild architecture mismatch on ARM Mac Docker builds. Passes in CI (Depot x86), fails locally on Apple Silicon. |
| 39 | + |
| 40 | +### agno |
| 41 | + |
| 42 | +- `reasoning=True` does multi-call chain-of-thought which breaks aimock (only first call matches). Disable for aimock-backed tests. |
| 43 | +- Agno's stock AGUI handler emits `STEP_STARTED`/`STEP_FINISHED` for reasoning — CopilotKit ignores these. The `reasoningMessage` slot requires `REASONING_MESSAGE_*` events. We built a custom handler, then reverted to stock AGUI. |
| 44 | +- Internal tool execution creates infinite fixture loops (same pattern as AG2). |
| 45 | + |
| 46 | +### spring-ai |
| 47 | + |
| 48 | +- **Java backend** — Maven build, fundamentally different toolchain. |
| 49 | +- `StreamingToolAgent.streamFirstTurn()` must include `toolCallbacks` with `internalToolExecutionEnabled=false`. Without this, aimock can't match `toolName: "get_weather"` — falls through to text-only fixture, weather card never renders. |
| 50 | +- AG-UI Java SDK not on Maven Central. Must clone and `mvn install` in Dockerfile. |
| 51 | + |
| 52 | +### mastra |
| 53 | + |
| 54 | +- **JS object shorthand key trap:** `{ weatherTool }` expands to function name `"weatherTool"`, not `"get_weather"`. Must use explicit keys: `{ get_weather: weatherTool }`. |
| 55 | +- `byocHashbrownAgent` needs its own dedicated agent with the hashbrown system prompt. The `weatherAgent` produces plain text → `useJsonParser` returns null → empty dashboard → timeout. |
| 56 | +- ~280s cold start (V8 JIT + Mastra boot). Watchdog can kill it before ready. |
| 57 | + |
| 58 | +### ms-agent-python |
| 59 | + |
| 60 | +- `AgentFrameworkAgent.run()` expects `input_data: dict`. The `_MultimodalAgent` override used `*args/**kwargs` → `TypeError` at runtime. |
| 61 | +- The override must `yield` events (async generator), not `return` (coroutine). |
| 62 | +- OpenAI `store=True` breaks aimock fixture matching. Set `store=False`. |
| 63 | + |
| 64 | +### ms-agent-dotnet |
| 65 | + |
| 66 | +- C# / .NET backend. |
| 67 | +- Auth page had V1 `CopilotKit` import → agent discovery failed → "Agent not found". |
| 68 | + |
| 69 | +### built-in-agent |
| 70 | + |
| 71 | +- **No Python backend.** TanStack AI `BuiltInAgent` runs in-process in Next.js. |
| 72 | +- `type: "tanstack"` with `convertTanStackStream` has a `runFinished` flag that blocks ALL events after first `RUN_FINISHED`. For byoc, must use `type: "custom"`. |
| 73 | +- OpenAI Responses API does NOT support `response_format: { type: "json_object" }` through TanStack adapter. The call silently fails — aimock never receives a request. |
| 74 | + |
| 75 | +### crewai-crews |
| 76 | + |
| 77 | +- `from __future__ import annotations` breaks `InterruptScheduling` import stubs in tests. |
| 78 | +- Backend tool execution doesn't cycle back to aimock for text follow-up. Known adapter limitation. |
| 79 | + |
| 80 | +### pydantic-ai |
| 81 | + |
| 82 | +- `_classify_binary_part()` has the `isinstance(part, dict)` bug. Pydantic models from AG-UI need `model_dump()`. |
| 83 | +- `starlette>=1.0.0` removes `on_startup`. Pin `starlette<1.0.0`. |
| 84 | + |
| 85 | +### llamaindex |
| 86 | + |
| 87 | +- `from __future__ import annotations` breaks Pydantic tool schema validation specifically when `backend_tools` are present but the response is text-only. |
| 88 | + |
| 89 | +### langroid |
| 90 | + |
| 91 | +- Custom AGUI handler (hand-written SSE, not a framework adapter). |
| 92 | +- `_normalize_part()` must handle Pydantic model instances via `model_dump(by_alias=True)`. |
| 93 | +- Does NOT need `onRunInitialized` multimodal shim. |
| 94 | + |
| 95 | +### AG2 |
| 96 | + |
| 97 | +- `AGUIStream` requires plain string content. Multipart arrays cause 400 errors. `ContentFlattenerShim` handles conversion. |
| 98 | +- Internal tool execution + aimock = infinite loop. Fix: `max_consecutive_auto_reply` or `hasToolResult` in fixtures. |
| 99 | + |
| 100 | +### google-adk |
| 101 | + |
| 102 | +- **Underscores required for ALL agent names.** Every other framework uses dashes. |
| 103 | +- All 6 dedicated route files called `copilotRuntimeNextJSAppRouterEndpoint()` per-request → race condition. Fixed by hoisting to module scope. |
| 104 | + |
| 105 | +### claude-sdk-python |
| 106 | + |
| 107 | +- Transient "empty assistant text" flaps (fc=1, self-healing, not reproducible locally). Suspected SSE stream interruption on Railway. |
| 108 | + |
| 109 | +--- |
| 110 | + |
| 111 | +## Aimock & Fixture Edge Cases |
| 112 | + |
| 113 | +**Check fixtures FIRST.** When an agent misbehaves through aimock, the fixture determines behavior — the real LLM is never consulted. |
| 114 | + |
| 115 | +**`sequenceIndex` counters are global.** They persist across all test runs within the same aimock process. Use `hasToolResult` (stateless) instead for fixtures shared across integrations. |
| 116 | + |
| 117 | +**Tool-rendering fixtures need `toolName` in match criteria.** If the request doesn't include tool definitions, the fixture falls through to text-only. Spring-ai omitted tools; mastra's shorthand keys produced wrong function names. |
| 118 | + |
| 119 | +**PDF turn is fragile.** Two-turn multimodal probe: if turn 2's message doesn't match the PDF fixture, the image fixture matches instead. The PDF fixture must be the most specific match. |
| 120 | + |
| 121 | +--- |
| 122 | + |
| 123 | +## What Was "Green" But Still Wrong |
| 124 | + |
| 125 | +1. **18 copies of identical frontend code** — every fix was a blitz. One missed integration = one regression. |
| 126 | +2. **V1/V2 imports inconsistent** — some pages used V1 provider with V2 hooks and happened to work because the feature didn't exercise the broken path. |
| 127 | +3. **Most integrations still on V1 runtime API** — `copilotRuntimeNextJSAppRouterEndpoint` + `ExperimentalEmptyAdapter` instead of V2's `createCopilotRuntimeHandler` + `InMemoryAgentRunner`. Only `built-in-agent` fully uses V2. The V1 API has the per-request race condition (hoisting to module scope was a band-aid, not a migration). |
| 128 | +4. **Agent name mismatches masked by default fallback** — features passed because the runtime fell back to `"default"`, not because the correct agent was wired. |
| 129 | +5. **Missing testids on custom renderers** — probe assertions were weak enough to pass via fallback selectors. |
| 130 | +6. **`onRunInitialized` shim applied where unnecessary** — worked by coincidence because legacy format round-tripped correctly. |
0 commit comments