You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: add blob attachment type for inline base64 data
Add support for a new 'blob' attachment type that allows sending
base64-encoded content (e.g. images) directly without disk I/O.
Generated types will be updated automatically when the runtime publishes
the new schema to @github/copilot. This commit includes:
- Add blob variant to Node.js and Python hand-written types
- Export attachment types from Python SDK public API
- Update docs: image-input.md, all language READMEs, streaming-events.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy file name to clipboardExpand all lines: docs/features/image-input.md
+193-8Lines changed: 193 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,9 @@
1
1
# Image Input
2
2
3
-
Send images to Copilot sessions by attaching them as file attachments. The runtime reads the file from disk, converts it to base64 internally, and sends it to the LLM as an image content block — no manual encoding required.
3
+
Send images to Copilot sessions as attachments. There are two ways to attach images:
4
+
5
+
-**File attachment** (`type: "file"`) — provide an absolute path; the runtime reads the file from disk, converts it to base64, and sends it to the LLM.
6
+
-**Blob attachment** (`type: "blob"`) — provide base64-encoded data directly; useful when the image is already in memory (e.g., screenshots, generated images, or data from an API).
4
7
5
8
## Overview
6
9
@@ -25,11 +28,12 @@ sequenceDiagram
25
28
| Concept | Description |
26
29
|---------|-------------|
27
30
|**File attachment**| An attachment with `type: "file"` and an absolute `path` to an image on disk |
28
-
|**Automatic encoding**| The runtime reads the image, converts it to base64, and sends it as an `image_url` block |
31
+
|**Blob attachment**| An attachment with `type: "blob"`, base64-encoded `data`, and a `mimeType` — no disk I/O needed |
32
+
|**Automatic encoding**| For file attachments, the runtime reads the image and converts it to base64 automatically |
29
33
|**Auto-resize**| The runtime automatically resizes or quality-reduces images that exceed model-specific limits |
30
34
|**Vision capability**| The model must have `capabilities.supports.vision = true` to process images |
31
35
32
-
## Quick Start
36
+
## Quick Start — File Attachment
33
37
34
38
Attach an image file to any message using the file attachment type. The path must be an absolute path to an image on disk.
When you already have image data in memory (e.g., a screenshot captured by your app, or an image fetched from an API), use a blob attachment to send it directly without writing to disk.
Supported image formats include JPG, PNG, GIF, and other common image types. The runtime reads the image from disk and converts it as needed before sending to the LLM. Use PNG or JPEG for best results, as these are the most widely supported formats.
405
+
Supported image formats include JPG, PNG, GIF, and other common image types. For file attachments, the runtime reads the image from disk and converts it as needed. For blob attachments, you provide the base64 data and MIME type directly. Use PNG or JPEG for best results, as these are the most widely supported formats.
221
406
222
407
The model's `capabilities.limits.vision.supported_media_types` field lists the exact MIME types it accepts.
223
408
@@ -283,10 +468,10 @@ These image blocks appear in `tool.execution_complete` event results. See the [S
283
468
|-----|---------|
284
469
|**Use PNG or JPEG directly**| Avoids conversion overhead — these are sent to the LLM as-is |
285
470
|**Keep images reasonably sized**| Large images may be quality-reduced, which can lose important details |
286
-
|**Use absolute paths**| The runtime reads files from disk; relative paths may not resolve correctly |
287
-
|**Check vision support first**|Sending images to a non-vision model wastes tokens on the file path without visual understanding|
288
-
|**Multiple images are supported**|Attach several file attachments in one message, up to the model's `max_prompt_images` limit|
289
-
|**Images are not base64 in your code**|You provide a file path — the runtime handles encoding, resizing, and format conversion|
471
+
|**Use absolute paths for file attachments**| The runtime reads files from disk; relative paths may not resolve correctly |
472
+
|**Use blob attachments for in-memory data**|When you already have base64 data (e.g., screenshots, API responses), blob avoids unnecessary disk I/O|
473
+
|**Check vision support first**|Sending images to a non-vision model wastes tokens without visual understanding|
474
+
|**Multiple images are supported**|Attach several attachments in one message, up to the model's `max_prompt_images` limit|
290
475
|**SVG is not supported**| SVG files are text-based and excluded from image processing |
Copy file name to clipboardExpand all lines: dotnet/README.md
+17-3Lines changed: 17 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -265,21 +265,35 @@ session.On(evt =>
265
265
266
266
## Image Support
267
267
268
-
The SDK supports image attachments via the `Attachments` parameter. You can attach images by providing their file path:
268
+
The SDK supports image attachments via the `Attachments` parameter. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
Supported image formats include JPG, PNG, GIF, and other common image types. The agent's `view` tool can also read images directly from the filesystem, so you can also ask questions like:
The SDK supports image attachments via the `Attachments` field in `MessageOptions`. You can attach images by providing their file path:
181
+
The SDK supports image attachments via the `Attachments` field in `MessageOptions`. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
Supported image formats include JPG, PNG, GIF, and other common image types. The agent's `view` tool can also read images directly from the filesystem, so you can also ask questions like:
Copy file name to clipboardExpand all lines: nodejs/README.md
+14-1Lines changed: 14 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -297,9 +297,10 @@ See `SessionEvent` type in the source for full details.
297
297
298
298
## Image Support
299
299
300
-
The SDK supports image attachments via the `attachments` parameter. You can attach images by providing their file path:
300
+
The SDK supports image attachments via the `attachments` parameter. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
301
301
302
302
```typescript
303
+
// File attachment — runtime reads from disk
303
304
awaitsession.send({
304
305
prompt: "What's in this image?",
305
306
attachments: [
@@ -309,6 +310,18 @@ await session.send({
309
310
},
310
311
],
311
312
});
313
+
314
+
// Blob attachment — provide base64 data directly
315
+
awaitsession.send({
316
+
prompt: "What's in this image?",
317
+
attachments: [
318
+
{
319
+
type: "blob",
320
+
data: base64ImageData,
321
+
mimeType: "image/png",
322
+
},
323
+
],
324
+
});
312
325
```
313
326
314
327
Supported image formats include JPG, PNG, GIF, and other common image types. The agent's `view` tool can also read images directly from the filesystem, so you can also ask questions like:
The SDK supports image attachments via the `attachments` parameter. You can attach images by providing their file path:
237
+
The SDK supports image attachments via the `attachments` parameter. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
238
238
239
239
```python
240
+
# File attachment — runtime reads from disk
240
241
await session.send({
241
242
"prompt": "What's in this image?",
242
243
"attachments": [
@@ -246,6 +247,18 @@ await session.send({
246
247
}
247
248
]
248
249
})
250
+
251
+
# Blob attachment — provide base64 data directly
252
+
await session.send({
253
+
"prompt": "What's in this image?",
254
+
"attachments": [
255
+
{
256
+
"type": "blob",
257
+
"data": base64_image_data,
258
+
"mimeType": "image/png",
259
+
}
260
+
]
261
+
})
249
262
```
250
263
251
264
Supported image formats include JPG, PNG, GIF, and other common image types. The agent's `view` tool can also read images directly from the filesystem, so you can also ask questions like:
0 commit comments