fix(showcase): unbreak multimodal demo end-to-end (auto-send, dedupe, proxy)#4761
Merged
Conversation
…o-send, dedupe, proxy) The langgraph-python multimodal-attachments demo had a stack of bugs that compounded each other. Fixing them required touching the local docker-compose, the aimock fixtures, the LangChain middleware, the client-side AG-UI shim, and the sample-attachment buttons. This commit lands the full set together because they only make sense as a unit — verified end-to-end against `showcase up langgraph-python` in a headed browser. New e2e suite pins each regression. Supersedes #4584 (the original fix from May 1 that never landed — this is a fresh port onto the post-refactor file layout where page.tsx is split into legacy-converter-shim.tsx, multimodal-chat.tsx, file-to-data-attachment.ts). What was broken and what changed: 1. Random uploads crashed with `Failed to fetch`. aimock returned HTTP 404 on no-match, the LangGraph SDK surfaced `NotFoundError`, the AG-UI stream surfaced a `RUN_ERROR`, the demo crashed. Added `--proxy-only` + `--provider-openai https://api.openai.com` to the local aimock command so unmatched user prompts fall through to real OpenAI (mirrors the Railway aimock setup). 2. Bundled-sample fixtures keyed on user-visible canned prompts. The auto-prompts are deliberately long, specific, and natural- reading ("can you tell me what is in this demo image/pdf I just attached") so they (a) render cleanly as the user message bubble, and (b) can't collide with arbitrary user prompts — random uploads phrase questions differently and fall through to the proxy. 3. Sample buttons now auto-send via `useAgent`. The previous DataTransfer-based path queued the attachment via the chat's hidden file input, then required clicking send while the attachment was still uploading — `CopilotChat.onSubmitInput` rejects submits during upload AND clears the input regardless, so the canned prompt was eaten. Rewrite to call `agent.addMessage(...)` + `copilotkit.runAgent({ agent })` directly with the base64'd content part, sidestepping the upload race entirely. 4. PDF flattened text bled into the rendered user message. `_PdfFlattenMiddleware` ran in `before_model` and returned `{"messages": rewritten}`, which persisted to agent state. The chat UI then rendered the `[Attached document]\n<pdf body>` text part inline with the user prompt. Switched to `wrap_model_call` so the PDF→text rewrite is scoped to the outgoing model request only and never pollutes state. 5. Attachments doubled (and PDFs rendered as broken `<img>`). The `@ag-ui/langgraph` round-trip translates outgoing `binary` parts to LangChain `image_url` and incoming `image_url` back to `image` AG-UI parts — regardless of mimeType, so PDFs came back as `type: "image"` with `mimeType: "application/pdf"` and were forced into `ImageAttachment`, where the load failed and the chat showed two "Failed to load image" boxes. Plus the user's original modern part survived alongside the round-tripped one, doubling visible chips. Added a `dedupeUserMessageMedia` subscriber on both `onMessagesSnapshotEvent` and `onRunFinalized` to: - dedupe media parts by `source.value` so the local + round- tripped copy collapse to one chip - re-key part `type` from `mimeType` so PDFs route to `DocumentAttachment` (icon + filename) and images to `ImageAttachment`. Also flipped the `onRunInitialized` shim from REPLACE to APPEND — keep the modern part for the UI AND emit a legacy `binary` sibling for the converter. 6. Regression suite (`tests/e2e/multimodal.spec.ts`). Replaces the pre-rewrite suite with five focused tests: - page loads with all expected affordances - sample image: auto-sends, EXACTLY ONE `<img>`, assistant references the logo - sample PDF: auto-sends, EXACTLY ONE `DocumentAttachment` chip ("PDF" label), NO `<img>`, no `[Attached document]` text bleed - image then PDF in the same session: each message keeps its own single chip, no cross-contamination - PDF then image in the same session: symmetric All 5 pass against the live local stack (15.4s).
AlemTuzlak
requested review from
jpr5,
marthakelly,
mme,
ranst91 and
tylerslaton
as code owners
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
5 tasks
Contributor
📣 Social Copy GeneratorGenerate social media copies (Twitter/X, LinkedIn, Blog Post) for this PR using Claude.
|
6 tasks
AlemTuzlak
added a commit
that referenced
this pull request
May 12, 2026
…-out, gen-ui-agent progression, multimodal D5 (#4776) ## Summary Four independent showcase production bugs Alem reported, plus the D5 multimodal harness regression they unblocked. D5 result: **37 → 39 of 40 features passing** on `langgraph-python`. The single remaining failure (`tool-rendering-reasoning-chain`) is a separate agent/runtime bug — root cause identified, scoped out below. ## What changed and why ### 1. `shared-state-read-write` — wrong-fixture fallthrough **Symptom:** "Greet me" returned the generic `showcase assistant` blurb; "Plan a weekend" returned the generic 5-step content-marketing plan. Only "Remember something" worked. **Cause:** The pill prompts `"Say hi and introduce yourself."` and `"Suggest a weekend plan based on my interests."` were matching the bare `userMessage: "hi"` / `"plan"` catch-alls in `feature-parity.json` (which loads after `d5-all.json`). The shared-state demo had no pill-specific fixtures to win first. **Fix:** Added two fixtures to `harness/fixtures/d5/shared-state.json` (mirrored to `aimock/d5-all.json`) keyed on longer unique substrings. Responses now reference the Preferences panel / interests panel so the demo's intent is clear. ### 2. `auth` — sign-out never demonstrates rejection **Symptom:** Clicking "Sign out" unmounted `<CopilotKit>` and bounced the user back to the SignInCard. The demo's whole point — runtime returning 401 on unauthenticated requests — was never visible. **Cause:** `page.tsx` rendered SignInCard whenever `!isAuthenticated`, which included the post-sign-out state. **Fix (matches `qa/auth.md` spec):** - `useDemoAuth` tracks a `hasEverSignedIn` flag (in-memory; resets on full page reload). - `page.tsx` only shows SignInCard for first-paint visitors; signed-in and signed-out states both keep `<CopilotKit>` mounted. - `AuthBanner` is bi-state — green when authed, amber when signed out — and exposes a new `data-testid="auth-authenticate-button"` for re-sign-in. - `<CopilotKit onError={...}>` populates a `data-testid="auth-demo-error"` surface so the 401 from the runtime is visible to the user the moment they send an unauthenticated message. - Updated `tests/e2e/auth.spec.ts` accordingly (the old "SignInCard re-mounts after sign-out" test was pinning the regression). ### 3. `gen-ui-agent` — step progression short-circuited **Symptom:** Clicking a pill rendered `All 3 steps complete` instantly. No pending → in_progress → completed animation. The card just appeared in its final state. **Cause:** The aimock fixture emitted a single `set_steps` tool call with all three steps already `status: completed`, even though `gen_ui_agent.py`'s SYSTEM_PROMPT instructs the LLM to make seven sequential calls (1 all-pending seed + 6 transitions). **Fix:** Regenerated `gen-ui-agent.json` as a 7-leg `toolCallId` chain per pill (24 fixtures total = 3 pills × 8 legs each): - Seed leg keyed only on `userMessage` (no `hasToolResult` gate — matches the pattern PR #4770 established, since `hasToolResult: false` blocks the seed firing on subsequent pills in a multi-pill session). - Six `toolCallId`-keyed transitions advancing each step pending → in_progress → completed in order. - Final narration after the 7th set_steps' tool result. - Order: `toolCallId` legs FIRST, `userMessage` seed LAST, so the most specific matcher wins first-match-wins. Mirrored verbatim into `d5-all.json`. Added a regression test in `tests/e2e/gen-ui-agent.spec.ts` that asserts at least one step is observed in `pending` state during the run. ### 4. Multimodal D5 — runner sends a competing message **Symptom:** D5 multimodal failed with `timeout: assistant did not respond within 30000ms (baseline=0, current=0)` even though the LangGraph agent's runs all succeeded. **Cause:** The sample-attachment buttons auto-send the user message via `agent.addMessage` + `copilotkit.runAgent` (the behavior PR #4761 restored). But the D5 conversation runner ALSO typed `input` and pressed Enter immediately after `preFill`, sending a SECOND user message. Both messages were matched and processed by aimock (debug logs confirm), but the resulting v1 LangGraph runtime SSE stream got tangled — Chrome DevTools showed `POST /api/copilotkit-multimodal` stuck in `statusCode: pending`, and the assistant message never rendered. **Fix (3 small pieces):** - `conversation-runner.ts`: added `skipSend?: boolean` to `ConversationTurn`. Distinct from the existing `skipFill` (which still presses Enter once the textarea has content) — `skipSend` skips both fill and Enter entirely. Used when `preFill` handles the whole submission via the agent surface. - `d5-multimodal.ts`: set `skipSend: true` on both image and PDF turns, bumped `responseTimeoutMs: 60_000` (the Playwright e2e spec uses 60-90s for the same path). - `feature-parity.json`: PDF auto-prompt fixture now says "The attached PDF document is..." instead of "The attached PDF is..." so the existing `buildModalityAssertion("document")` check still lands. ## Out of scope (separate task) - **`tool-rendering-reasoning-chain`** — the one remaining D5 failure. Root cause: the Tokyo pill uses Responses-API mode, producing a `reasoning` role message in the conversation. On Turn 2, that message survives in history and the runtime returns `RUN_ERROR: "message role is not supported"`. This is an agent/runtime issue, not a fixture-level fix. - **`book-a-call` Cancel path** — clicking "None of these work" produces a contradictory "Booked!" narration. Aimock's JSON-fixture matcher can't inspect tool-result content (only `userMessage` + `toolCallId` + `hasToolResult` + `toolName`), so this needs either a React-level UI redesign or an aimock package extension. ## Test plan - [x] `bin/showcase fixtures validate` — 0 errors - [x] `bin/showcase test langgraph-python --d5` — 39 of 40 features passing (was 37 before) - [x] Manual verification of `/demos/auth` end-to-end on local: sign in → sign out → send unauthenticated message → 401 error surface displays with banner still amber, no white-screen - [x] Manual verification of `/demos/shared-state-read-write` on local: Greet me and Plan a weekend return shared-state-aware responses instead of the generic catch-alls - [x] Direct aimock probe (`curl /v1/chat/completions`) verifies the `set_steps_launch_001` → `_002` → ... → `_007` chain emits the expected progression states - [ ] New `tests/e2e/shared-state-read-write.spec.ts` Playwright suite (4 tests including the wrong-fixture regression assertions) — runs in CI 🤖 Generated with [Claude Code](https://claude.com/claude-code)
pull Bot
pushed a commit
to justinlietz93/CopilotKit
that referenced
this pull request
May 12, 2026
…-out, gen-ui-agent progression, multimodal D5
Four independent showcase production bugs Alem reported, plus the
D5 multimodal harness regression they unblocked.
Shared-state-read-write: "Greet me" ("Say hi and introduce yourself.")
and "Plan a weekend" ("Suggest a weekend plan based on my interests.")
were matching the bare `hi` and `plan` catch-alls in feature-parity.json
and returning the generic showcase-assistant blurb / 5-step content plan
instead of shared-state-aware responses. Added pill-specific fixtures in
shared-state.json (mirrored into d5-all.json) so the longer userMessage
substrings win first-match-wins ahead of feature-parity.
Auth sign-out: signing out unmounted CopilotKit entirely and bounced
the user back to the SignInCard, so the demo never showcased the
runtime returning 401 — its whole point. The QA contract in
qa/auth.md spelled out the intended UX. Restored it: CopilotKit stays
mounted after the first sign-in, the AuthBanner flips to an amber
"Signed out — the agent will reject your messages" state with a
re-Sign-in button, and CopilotKit's `onError` callback drives a
`data-testid="auth-demo-error"` surface that displays the runtime's
401 the moment the user sends an unauthenticated message. Updated the
e2e spec to match (the old "SignInCard re-mounts after sign-out" test
pinned the regression).
Gen-ui-agent: the aimock fixture short-circuited the 7-step
progression spelled out in `gen_ui_agent.py`'s SYSTEM_PROMPT to a
single set_steps call with all three steps already `completed`, so
the InlineAgentStateCard rendered the final 3/3 state instantly with
no sequential pending → in_progress → completed animation.
Regenerated as a 7-leg toolCallId chain per pill (8 fixtures × 3
pills): seed leg keyed on userMessage with NO `hasToolResult` gate
(matching PR CopilotKit#4770's pattern — `hasToolResult: false` would block the
seed from firing on the second pill in a multi-pill session), then
six toolCallId-keyed transitions, then a final narration. Fixture
order: toolCallId legs FIRST so the most specific match wins.
Multimodal D5: the sample-attachment buttons auto-send via
`agent.addMessage + copilotkit.runAgent` (restored in PR CopilotKit#4761), but
the D5 harness still typed `input` + pressed Enter via the runner
after `preFill`, sending a second user message that competed with the
in-flight image upload — the v1 LangGraph runtime SSE stream got
tangled (browser DevTools showed `statusCode: pending` indefinitely)
and the assistant message never rendered. Added `skipSend?: boolean`
to ConversationTurn (distinct from `skipFill`, which still presses
Enter once the textarea has content) and switched d5-multimodal.ts to
`skipSend: true` with `responseTimeoutMs: 60_000` so the runner waits
on the assistant response without poking the chat further. Bumped the
PDF auto-prompt fixture in feature-parity.json to include the word
"document" so the existing `buildModalityAssertion("document")` check
still lands.
D5 result: 37 → 39 of 40 features passing. Only
`tool-rendering-reasoning-chain` remains and is a separate
agent/runtime bug (Tokyo Responses-API `reasoning` message survives
into the next turn's conversation history, runtime returns
`RUN_ERROR: "message role is not supported"`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-lands the multimodal-attachments fix from #4584 (May 1, never merged) onto current
main, ported to the post-refactor file layout wherepage.tsxwas split intolegacy-converter-shim.tsx,multimodal-chat.tsx, andfile-to-data-attachment.ts.Auto-send was the visible regression: clicking Try with sample image / Try with sample PDF only queued the attachment chip instead of sending the canned prompt. This PR restores the full end-to-end behavior plus five regression tests so it can't silently break again.
What was broken and what changed
Random uploads crashed with
Failed to fetch. aimock returned HTTP 404 on no-match, the LangGraph SDK surfacedNotFoundError, the AG-UI stream surfaced aRUN_ERROR, the demo crashed. → Added--proxy-only+--provider-openai https://api.openai.comto the local aimock command so unmatched user prompts fall through to real OpenAI (mirrors Railway).Bundled-sample fixtures keyed on user-visible canned prompts. Auto-prompts are now natural and specific ("can you tell me what is in this demo image/pdf I just attached") so they render cleanly as the user message bubble AND can't collide with arbitrary user prompts — random uploads phrase questions differently and fall through to the proxy.
Sample buttons now auto-send via
useAgent. The previous DataTransfer path queued the attachment via the chat's hidden file input but required clicking send while the attachment was still uploading —CopilotChat.onSubmitInputrejects submits during upload AND clears the input regardless, so the canned prompt was eaten. Rewrite callsagent.addMessage(...)+copilotkit.runAgent({ agent })directly with the base64'd content part.PDF flattened text bled into the rendered user message.
_PdfFlattenMiddlewareran inbefore_modeland persisted the rewrite to agent state. Switched towrap_model_callso the PDF→text rewrite is scoped to the model request only.Attachments doubled (and PDFs rendered as broken
<img>). The@ag-ui/langgraphround-trip mis-tags PDFs asimageand re-injects the user's original modern part, doubling chips. AddeddedupeUserMessageMediasubscriber ononMessagesSnapshotEvent+onRunFinalizedto dedupe bysource.valueand re-key type from mimeType. Also flippedonRunInitializedfrom REPLACE to APPEND so the modern part stays for the UI alongside a legacybinarysibling for the converter.Regression suite (
tests/e2e/multimodal.spec.ts). Five focused tests, all pass against live local stack (15.4s):<img>, assistant references the logoDocumentAttachmentchip ("PDF" label), NO<img>, no[Attached document]text bleedTest plan
showcase up langgraph-python— both sample buttons auto-send; image renders as<img>, PDF renders as PDF chip; random paperclip uploads go through proxyBASE_URL=http://localhost:3100 CI=1 npx playwright test multimodal.spec.ts— 5 / 5 passingCloses
Closes #4584.