Skip to content

Commit 7c3edca

Browse files
committed
fix(showcase): unbreak multimodal demo end-to-end (sample buttons auto-send, dedupe, proxy)
The langgraph-python multimodal-attachments demo had a stack of bugs that compounded each other. Fixing them required touching the local docker-compose, the aimock fixtures, the LangChain middleware, the client-side AG-UI shim, and the sample-attachment buttons. This commit lands the full set together because they only make sense as a unit — verified end-to-end against `showcase up langgraph-python` in a headed browser. New e2e suite pins each regression. Supersedes CopilotKit#4584 (the original fix from May 1 that never landed — this is a fresh port onto the post-refactor file layout where page.tsx is split into legacy-converter-shim.tsx, multimodal-chat.tsx, file-to-data-attachment.ts). What was broken and what changed: 1. Random uploads crashed with `Failed to fetch`. aimock returned HTTP 404 on no-match, the LangGraph SDK surfaced `NotFoundError`, the AG-UI stream surfaced a `RUN_ERROR`, the demo crashed. Added `--proxy-only` + `--provider-openai https://api.openai.com` to the local aimock command so unmatched user prompts fall through to real OpenAI (mirrors the Railway aimock setup). 2. Bundled-sample fixtures keyed on user-visible canned prompts. The auto-prompts are deliberately long, specific, and natural- reading ("can you tell me what is in this demo image/pdf I just attached") so they (a) render cleanly as the user message bubble, and (b) can't collide with arbitrary user prompts — random uploads phrase questions differently and fall through to the proxy. 3. Sample buttons now auto-send via `useAgent`. The previous DataTransfer-based path queued the attachment via the chat's hidden file input, then required clicking send while the attachment was still uploading — `CopilotChat.onSubmitInput` rejects submits during upload AND clears the input regardless, so the canned prompt was eaten. Rewrite to call `agent.addMessage(...)` + `copilotkit.runAgent({ agent })` directly with the base64'd content part, sidestepping the upload race entirely. 4. PDF flattened text bled into the rendered user message. `_PdfFlattenMiddleware` ran in `before_model` and returned `{"messages": rewritten}`, which persisted to agent state. The chat UI then rendered the `[Attached document]\n<pdf body>` text part inline with the user prompt. Switched to `wrap_model_call` so the PDF→text rewrite is scoped to the outgoing model request only and never pollutes state. 5. Attachments doubled (and PDFs rendered as broken `<img>`). The `@ag-ui/langgraph` round-trip translates outgoing `binary` parts to LangChain `image_url` and incoming `image_url` back to `image` AG-UI parts — regardless of mimeType, so PDFs came back as `type: "image"` with `mimeType: "application/pdf"` and were forced into `ImageAttachment`, where the load failed and the chat showed two "Failed to load image" boxes. Plus the user's original modern part survived alongside the round-tripped one, doubling visible chips. Added a `dedupeUserMessageMedia` subscriber on both `onMessagesSnapshotEvent` and `onRunFinalized` to: - dedupe media parts by `source.value` so the local + round- tripped copy collapse to one chip - re-key part `type` from `mimeType` so PDFs route to `DocumentAttachment` (icon + filename) and images to `ImageAttachment`. Also flipped the `onRunInitialized` shim from REPLACE to APPEND — keep the modern part for the UI AND emit a legacy `binary` sibling for the converter. 6. Regression suite (`tests/e2e/multimodal.spec.ts`). Replaces the pre-rewrite suite with five focused tests: - page loads with all expected affordances - sample image: auto-sends, EXACTLY ONE `<img>`, assistant references the logo - sample PDF: auto-sends, EXACTLY ONE `DocumentAttachment` chip ("PDF" label), NO `<img>`, no `[Attached document]` text bleed - image then PDF in the same session: each message keeps its own single chip, no cross-contamination - PDF then image in the same session: symmetric All 5 pass against the live local stack (15.4s).
1 parent 15db0bb commit 7c3edca

8 files changed

Lines changed: 816 additions & 222 deletions

File tree

showcase/aimock/feature-parity.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -563,15 +563,15 @@
563563
},
564564
{
565565
"match": {
566-
"userMessage": "this PDF"
566+
"userMessage": "can you tell me what is in this demo pdf I just attached"
567567
},
568568
"response": {
569569
"content": "The attached PDF is the CopilotKit Quickstart guide. It walks through three steps: install `@copilotkit/react-core` and `@copilotkit/react-ui`, wrap your app in a `<CopilotKit runtimeUrl=\"/api/copilotkit\">` provider, and drop a `<CopilotChat />` wherever you want the assistant to appear."
570570
}
571571
},
572572
{
573573
"match": {
574-
"userMessage": "this image"
574+
"userMessage": "can you tell me what is in this demo image I just attached"
575575
},
576576
"response": {
577577
"content": "The attached image is the CopilotKit logo — a clean, geometric mark used across CopilotKit branding."

showcase/docker-compose.local.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,21 @@ services:
4646
- ./aimock/d5-all.json:/showcase-fixtures/d5-all.json:ro
4747
- ./aimock/smoke.json:/showcase-fixtures/smoke.json:ro
4848
- ./aimock/feature-parity.json:/showcase-fixtures/feature-parity.json:ro
49+
# `--proxy-only` + `--provider-openai` mirrors the Railway aimock setup:
50+
# matched fixtures short-circuit deterministically; unmatched requests
51+
# fall through to real OpenAI (using OPENAI_API_KEY from .env). Without
52+
# these flags aimock returns 404 on no-match, the LangGraph SDK
53+
# interprets that as `NotFoundError`, and the demo crashes with
54+
# "Failed to fetch" instead of seeing a real model response.
4955
command:
5056
[
5157
"--port",
5258
"4010",
5359
"--host",
5460
"0.0.0.0",
61+
"--proxy-only",
62+
"--provider-openai",
63+
"https://api.openai.com",
5564
"--fixtures",
5665
"/showcase-fixtures/d5-all.json",
5766
"--fixtures",

0 commit comments

Comments
 (0)