Skip to content

Commit cab83d3

Browse files
committed
feat(shell-docs): port platform pages from upstream
Adds multimodal-attachments, runtime-server-adapter, and vs-code-extension pages under a new ---Platform--- section in meta.json. threads.mdx was already in place and left untouched. fumadocs-ui import lines are stripped since shell-docs pulls components from its MDX registry instead.
1 parent 386accd commit cab83d3

4 files changed

Lines changed: 1318 additions & 0 deletions

File tree

showcase/shell-docs/src/content/docs/meta.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@
1818
"shared-state",
1919
"---Threads---",
2020
"threads",
21+
"---Platform---",
22+
"multimodal-attachments",
23+
"runtime-server-adapter",
24+
"vs-code-extension",
2125
"---Backend---",
2226
"...backend",
2327
"---Premium Features---",
Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
---
2+
title: Multimodal Attachments
3+
icon: "lucide/Paperclip"
4+
description: Let users send images, audio, video, and documents to the AI alongside their messages.
5+
---
6+
7+
You have a working CopilotChat and want users to attach files — images, PDFs, audio, video — that the AI can see and respond to. By the end of this guide, your chat will support drag-and-drop file attachments with previews, lightbox viewing, and multimodal AI responses.
8+
9+
## Quick start
10+
11+
Add `attachments` to your `CopilotChat` component:
12+
13+
```tsx title="page.tsx"
14+
import { CopilotChat } from "@copilotkit/react-core/v2";
15+
16+
<CopilotChat
17+
agentId="my-agent"
18+
attachments={{ enabled: true }} // [!code highlight]
19+
/>
20+
```
21+
22+
That's it. Users can now click the attachment button or drag-and-drop files into the chat. The files are sent as part of the message content to your agent.
23+
24+
## Configuration
25+
26+
The `attachments` prop accepts an `AttachmentsConfig` object:
27+
28+
```tsx
29+
<CopilotChat
30+
attachments={{
31+
enabled: true,
32+
accept: "image/*", // MIME filter (default: "*/*")
33+
maxSize: 10 * 1024 * 1024, // 10MB limit (default: 20MB)
34+
}}
35+
/>
36+
```
37+
38+
| Option | Type | Default | Description |
39+
| --- | --- | --- | --- |
40+
| `enabled` | `boolean` || Enable file attachments in the chat input. |
41+
| `accept` | `string` | `"*/*"` | MIME type filter. Supports patterns like `"image/*"`, `".pdf,.docx"`, or comma-separated lists. |
42+
| `maxSize` | `number` | `20 * 1024 * 1024` | Maximum file size in bytes. |
43+
| `onUpload` | `(file: File) => AttachmentUploadResult \| Promise<...>` || Custom upload handler. See [Custom upload handler](#custom-upload-handler). |
44+
| `onUploadFailed` | `(error: AttachmentUploadError) => void` || Called when a file fails validation or upload. See [Handling upload errors](#handling-upload-errors). |
45+
46+
## Supported file types
47+
48+
Attachments are categorized by modality based on their MIME type:
49+
50+
| Modality | MIME types | Preview | AI support |
51+
| --- | --- | --- | --- |
52+
| **Image** | `image/*` | Thumbnail with lightbox | Supported by most vision-capable models (GPT-4o, Claude, etc.) |
53+
| **Audio** | `audio/*` | Audio player | Model-dependent |
54+
| **Video** | `video/*` | Thumbnail with play button + lightbox | Model-dependent |
55+
| **Document** | Everything else | File icon + name; PDF and text get lightbox preview | Sent as file content — model support varies |
56+
57+
<Callout type="info">
58+
Not all models support all modalities. For example, OpenAI's GPT-4o supports images but not audio file parts. If the model doesn't support a file type, you'll get a `RUN_ERROR` event. Use the `onError` callback to handle this gracefully.
59+
</Callout>
60+
61+
## Custom upload handler
62+
63+
By default, files are read as base64 and sent inline. For large files or production apps, you'll want to upload to your own storage and pass a URL instead.
64+
65+
`onUpload` returns an `AttachmentUploadResult` — a discriminated union with two variants:
66+
67+
<Tabs items={["Base64 (inline)", "URL (hosted)"]}>
68+
<Tab value="Base64 (inline)">
69+
```tsx
70+
<CopilotChat
71+
attachments={{
72+
enabled: true,
73+
onUpload: async (file) => {
74+
const buffer = await file.arrayBuffer();
75+
const base64 = btoa(String.fromCharCode(...new Uint8Array(buffer)));
76+
return {
77+
type: "data",
78+
value: base64,
79+
mimeType: file.type,
80+
};
81+
},
82+
}}
83+
/>
84+
```
85+
</Tab>
86+
<Tab value="URL (hosted)">
87+
```tsx
88+
<CopilotChat
89+
attachments={{
90+
enabled: true,
91+
onUpload: async (file) => {
92+
const formData = new FormData();
93+
formData.append("file", file);
94+
const res = await fetch("/api/upload", { method: "POST", body: formData });
95+
const { url } = await res.json();
96+
return {
97+
type: "url",
98+
value: url,
99+
mimeType: file.type,
100+
};
101+
},
102+
}}
103+
/>
104+
```
105+
</Tab>
106+
</Tabs>
107+
108+
### Adding metadata
109+
110+
You can attach custom metadata to any upload. It's included in the `InputContent` part sent to the agent and accessible via `metadata` on the content part.
111+
112+
```tsx
113+
onUpload: async (file) => {
114+
const url = await uploadToStorage(file);
115+
return {
116+
type: "url",
117+
value: url,
118+
mimeType: file.type,
119+
metadata: {
120+
uploadedBy: currentUser.id,
121+
category: "support-ticket",
122+
},
123+
};
124+
},
125+
```
126+
127+
The filename is always included in metadata automatically — you don't need to add it yourself.
128+
129+
## Handling upload errors
130+
131+
Use `onUploadFailed` to react when a file is rejected or an upload fails — for example, to show a toast:
132+
133+
```tsx
134+
<CopilotChat
135+
attachments={{
136+
enabled: true,
137+
accept: "image/*",
138+
maxSize: 5 * 1024 * 1024, // 5MB
139+
onUploadFailed: (error) => {
140+
// error.reason: "file-too-large" | "invalid-type" | "upload-failed"
141+
// error.file: the original File object
142+
// error.message: human-readable description
143+
toast.error(error.message);
144+
},
145+
}}
146+
/>
147+
```
148+
149+
| Reason | When it fires |
150+
| --- | --- |
151+
| `invalid-type` | File doesn't match the `accept` filter. |
152+
| `file-too-large` | File exceeds `maxSize`. |
153+
| `upload-failed` | The `onUpload` handler threw, or the default base64 reader failed. |
154+
155+
For errors that happen _after_ the message is sent (e.g., the model doesn't support the file type), use the `onError` callback on `CopilotChat`:
156+
157+
```tsx
158+
<CopilotChat
159+
attachments={{ enabled: true }}
160+
onError={(event) => {
161+
console.error(`[${event.code}]`, event.error.message);
162+
}}
163+
/>
164+
```
165+
166+
## How it works
167+
168+
When a user attaches files and sends a message, CopilotKit:
169+
170+
1. Reads each file (via the default base64 reader or your `onUpload` handler)
171+
2. Builds an array of `InputContent` parts — text + one part per attachment
172+
3. Adds the message to the agent with `content: [{ type: "text", ... }, { type: "image", source: ... }, ...]`
173+
4. The agent receives the multimodal content via the AG-UI protocol and forwards it to the model
174+
175+
The attachments are part of the standard AG-UI `InputContent` schema, so any AG-UI-compatible agent (BuiltInAgent, LangGraph, custom) can receive them.
176+
177+
<Callout type="info">
178+
Migrating from the old `imageUploadsEnabled` prop? See the [migration guide](/migration-guides/migrate-attachments) for a step-by-step walkthrough.
179+
</Callout>

0 commit comments

Comments
 (0)