Commit 7c3edca
committed
fix(showcase): unbreak multimodal demo end-to-end (sample buttons auto-send, dedupe, proxy)
The langgraph-python multimodal-attachments demo had a stack of bugs
that compounded each other. Fixing them required touching the local
docker-compose, the aimock fixtures, the LangChain middleware, the
client-side AG-UI shim, and the sample-attachment buttons. This
commit lands the full set together because they only make sense as
a unit — verified end-to-end against `showcase up langgraph-python`
in a headed browser. New e2e suite pins each regression.
Supersedes CopilotKit#4584 (the original fix from May 1 that never landed —
this is a fresh port onto the post-refactor file layout where
page.tsx is split into legacy-converter-shim.tsx, multimodal-chat.tsx,
file-to-data-attachment.ts).
What was broken and what changed:
1. Random uploads crashed with `Failed to fetch`. aimock returned
HTTP 404 on no-match, the LangGraph SDK surfaced `NotFoundError`,
the AG-UI stream surfaced a `RUN_ERROR`, the demo crashed.
Added `--proxy-only` + `--provider-openai https://api.openai.com`
to the local aimock command so unmatched user prompts fall through
to real OpenAI (mirrors the Railway aimock setup).
2. Bundled-sample fixtures keyed on user-visible canned prompts.
The auto-prompts are deliberately long, specific, and natural-
reading ("can you tell me what is in this demo image/pdf I just
attached") so they (a) render cleanly as the user message bubble,
and (b) can't collide with arbitrary user prompts — random
uploads phrase questions differently and fall through to the
proxy.
3. Sample buttons now auto-send via `useAgent`. The previous
DataTransfer-based path queued the attachment via the chat's
hidden file input, then required clicking send while the
attachment was still uploading — `CopilotChat.onSubmitInput`
rejects submits during upload AND clears the input regardless,
so the canned prompt was eaten. Rewrite to call
`agent.addMessage(...)` + `copilotkit.runAgent({ agent })`
directly with the base64'd content part, sidestepping the
upload race entirely.
4. PDF flattened text bled into the rendered user message.
`_PdfFlattenMiddleware` ran in `before_model` and returned
`{"messages": rewritten}`, which persisted to agent state. The
chat UI then rendered the `[Attached document]\n<pdf body>` text
part inline with the user prompt. Switched to `wrap_model_call`
so the PDF→text rewrite is scoped to the outgoing model request
only and never pollutes state.
5. Attachments doubled (and PDFs rendered as broken `<img>`). The
`@ag-ui/langgraph` round-trip translates outgoing `binary` parts
to LangChain `image_url` and incoming `image_url` back to `image`
AG-UI parts — regardless of mimeType, so PDFs came back as
`type: "image"` with `mimeType: "application/pdf"` and were
forced into `ImageAttachment`, where the load failed and the
chat showed two "Failed to load image" boxes. Plus the user's
original modern part survived alongside the round-tripped one,
doubling visible chips.
Added a `dedupeUserMessageMedia` subscriber on both
`onMessagesSnapshotEvent` and `onRunFinalized` to:
- dedupe media parts by `source.value` so the local + round-
tripped copy collapse to one chip
- re-key part `type` from `mimeType` so PDFs route to
`DocumentAttachment` (icon + filename) and images to
`ImageAttachment`.
Also flipped the `onRunInitialized` shim from REPLACE to APPEND
— keep the modern part for the UI AND emit a legacy `binary`
sibling for the converter.
6. Regression suite (`tests/e2e/multimodal.spec.ts`). Replaces the
pre-rewrite suite with five focused tests:
- page loads with all expected affordances
- sample image: auto-sends, EXACTLY ONE `<img>`, assistant
references the logo
- sample PDF: auto-sends, EXACTLY ONE `DocumentAttachment` chip
("PDF" label), NO `<img>`, no `[Attached document]` text bleed
- image then PDF in the same session: each message keeps its own
single chip, no cross-contamination
- PDF then image in the same session: symmetric
All 5 pass against the live local stack (15.4s).1 parent 15db0bb commit 7c3edca
8 files changed
Lines changed: 816 additions & 222 deletions
File tree
- showcase
- aimock
- integrations/langgraph-python
- src
- agents
- app/demos/multimodal
- tests/e2e
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
563 | 563 | | |
564 | 564 | | |
565 | 565 | | |
566 | | - | |
| 566 | + | |
567 | 567 | | |
568 | 568 | | |
569 | 569 | | |
570 | 570 | | |
571 | 571 | | |
572 | 572 | | |
573 | 573 | | |
574 | | - | |
| 574 | + | |
575 | 575 | | |
576 | 576 | | |
577 | 577 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
49 | 55 | | |
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
53 | 59 | | |
54 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
55 | 64 | | |
56 | 65 | | |
57 | 66 | | |
| |||
0 commit comments