Skip to content

Commit 698b259

Browse files
MackinnonBuckCopilotCopilot
authored
feat: add blob attachment type for inline base64 data (#731)
* feat: add blob attachment type for inline base64 data Add support for a new 'blob' attachment type that allows sending base64-encoded content (e.g. images) directly without disk I/O. Generated types will be updated automatically when the runtime publishes the new schema to @github/copilot. This commit includes: - Add blob variant to Node.js and Python hand-written types - Export attachment types from Python SDK public API - Update docs: image-input.md, all language READMEs, streaming-events.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update dotnet/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add E2E tests for blob attachments across all 4 SDKs Add blob attachment E2E tests for Node.js, Python, Go, and .NET SDKs. Each test sends a message with an inline base64-encoded PNG blob attachment and verifies the request is accepted by the replay proxy. - nodejs/test/e2e/session_config.test.ts: should accept blob attachments - python/e2e/test_session.py: test_should_accept_blob_attachments - go/internal/e2e/session_test.go: TestSessionBlobAttachment - dotnet/test/SessionTests.cs: Should_Accept_Blob_Attachments - test/snapshots/: request-only YAML snapshots for replay proxy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(python): break long base64 string to satisfy ruff E501 line length Split the inline base64-encoded PNG data in the blob attachment E2E test into a local variable with implicit string concatenation so every line stays within the 100-character limit enforced by ruff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent dde88fb commit 698b259

File tree

16 files changed

+435
-24
lines changed

16 files changed

+435
-24
lines changed

‎docs/features/image-input.md‎

Lines changed: 197 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# Image Input
22

3-
Send images to Copilot sessions by attaching them as file attachments. The runtime reads the file from disk, converts it to base64 internally, and sends it to the LLM as an image content block — no manual encoding required.
3+
Send images to Copilot sessions as attachments. There are two ways to attach images:
4+
5+
- **File attachment** (`type: "file"`) — provide an absolute path; the runtime reads the file from disk, converts it to base64, and sends it to the LLM.
6+
- **Blob attachment** (`type: "blob"`) — provide base64-encoded data directly; useful when the image is already in memory (e.g., screenshots, generated images, or data from an API).
47

58
## Overview
69

@@ -25,11 +28,12 @@ sequenceDiagram
2528
| Concept | Description |
2629
|---------|-------------|
2730
| **File attachment** | An attachment with `type: "file"` and an absolute `path` to an image on disk |
28-
| **Automatic encoding** | The runtime reads the image, converts it to base64, and sends it as an `image_url` block |
31+
| **Blob attachment** | An attachment with `type: "blob"`, base64-encoded `data`, and a `mimeType` — no disk I/O needed |
32+
| **Automatic encoding** | For file attachments, the runtime reads the image and converts it to base64 automatically |
2933
| **Auto-resize** | The runtime automatically resizes or quality-reduces images that exceed model-specific limits |
3034
| **Vision capability** | The model must have `capabilities.supports.vision = true` to process images |
3135

32-
## Quick Start
36+
## Quick Start — File Attachment
3337

3438
Attach an image file to any message using the file attachment type. The path must be an absolute path to an image on disk.
3539

@@ -75,15 +79,15 @@ session = await client.create_session({
7579
"on_permission_request": lambda req, inv: PermissionRequestResult(kind="approved"),
7680
})
7781

78-
await session.send({
79-
"prompt": "Describe what you see in this image",
80-
"attachments": [
82+
await session.send(
83+
"Describe what you see in this image",
84+
attachments=[
8185
{
8286
"type": "file",
8387
"path": "/absolute/path/to/screenshot.png",
8488
},
8589
],
86-
})
90+
)
8791
```
8892

8993
</details>
@@ -215,9 +219,190 @@ await session.SendAsync(new MessageOptions
215219

216220
</details>
217221

222+
## Quick Start — Blob Attachment
223+
224+
When you already have image data in memory (e.g., a screenshot captured by your app, or an image fetched from an API), use a blob attachment to send it directly without writing to disk.
225+
226+
<details open>
227+
<summary><strong>Node.js / TypeScript</strong></summary>
228+
229+
```typescript
230+
import { CopilotClient } from "@github/copilot-sdk";
231+
232+
const client = new CopilotClient();
233+
await client.start();
234+
235+
const session = await client.createSession({
236+
model: "gpt-4.1",
237+
onPermissionRequest: async () => ({ kind: "approved" }),
238+
});
239+
240+
const base64ImageData = "..."; // your base64-encoded image
241+
await session.send({
242+
prompt: "Describe what you see in this image",
243+
attachments: [
244+
{
245+
type: "blob",
246+
data: base64ImageData,
247+
mimeType: "image/png",
248+
displayName: "screenshot.png",
249+
},
250+
],
251+
});
252+
```
253+
254+
</details>
255+
256+
<details>
257+
<summary><strong>Python</strong></summary>
258+
259+
```python
260+
from copilot import CopilotClient
261+
from copilot.types import PermissionRequestResult
262+
263+
client = CopilotClient()
264+
await client.start()
265+
266+
session = await client.create_session({
267+
"model": "gpt-4.1",
268+
"on_permission_request": lambda req, inv: PermissionRequestResult(kind="approved"),
269+
})
270+
271+
base64_image_data = "..." # your base64-encoded image
272+
await session.send(
273+
"Describe what you see in this image",
274+
attachments=[
275+
{
276+
"type": "blob",
277+
"data": base64_image_data,
278+
"mimeType": "image/png",
279+
"displayName": "screenshot.png",
280+
},
281+
],
282+
)
283+
```
284+
285+
</details>
286+
287+
<details>
288+
<summary><strong>Go</strong></summary>
289+
290+
<!-- docs-validate: hidden -->
291+
```go
292+
package main
293+
294+
import (
295+
"context"
296+
copilot "github.com/github/copilot-sdk/go"
297+
)
298+
299+
func main() {
300+
ctx := context.Background()
301+
client := copilot.NewClient(nil)
302+
client.Start(ctx)
303+
304+
session, _ := client.CreateSession(ctx, &copilot.SessionConfig{
305+
Model: "gpt-4.1",
306+
OnPermissionRequest: func(req copilot.PermissionRequest, inv copilot.PermissionInvocation) (copilot.PermissionRequestResult, error) {
307+
return copilot.PermissionRequestResult{Kind: copilot.PermissionRequestResultKindApproved}, nil
308+
},
309+
})
310+
311+
base64ImageData := "..."
312+
mimeType := "image/png"
313+
displayName := "screenshot.png"
314+
session.Send(ctx, copilot.MessageOptions{
315+
Prompt: "Describe what you see in this image",
316+
Attachments: []copilot.Attachment{
317+
{
318+
Type: copilot.Blob,
319+
Data: &base64ImageData,
320+
MIMEType: &mimeType,
321+
DisplayName: &displayName,
322+
},
323+
},
324+
})
325+
}
326+
```
327+
<!-- /docs-validate: hidden -->
328+
329+
```go
330+
mimeType := "image/png"
331+
displayName := "screenshot.png"
332+
session.Send(ctx, copilot.MessageOptions{
333+
Prompt: "Describe what you see in this image",
334+
Attachments: []copilot.Attachment{
335+
{
336+
Type: copilot.Blob,
337+
Data: &base64ImageData, // base64-encoded string
338+
MIMEType: &mimeType,
339+
DisplayName: &displayName,
340+
},
341+
},
342+
})
343+
```
344+
345+
</details>
346+
347+
<details>
348+
<summary><strong>.NET</strong></summary>
349+
350+
<!-- docs-validate: hidden -->
351+
```csharp
352+
using GitHub.Copilot.SDK;
353+
354+
public static class BlobAttachmentExample
355+
{
356+
public static async Task Main()
357+
{
358+
await using var client = new CopilotClient();
359+
await using var session = await client.CreateSessionAsync(new SessionConfig
360+
{
361+
Model = "gpt-4.1",
362+
OnPermissionRequest = (req, inv) =>
363+
Task.FromResult(new PermissionRequestResult { Kind = PermissionRequestResultKind.Approved }),
364+
});
365+
366+
var base64ImageData = "...";
367+
await session.SendAsync(new MessageOptions
368+
{
369+
Prompt = "Describe what you see in this image",
370+
Attachments = new List<UserMessageDataAttachmentsItem>
371+
{
372+
new UserMessageDataAttachmentsItemBlob
373+
{
374+
Data = base64ImageData,
375+
MimeType = "image/png",
376+
DisplayName = "screenshot.png",
377+
},
378+
},
379+
});
380+
}
381+
}
382+
```
383+
<!-- /docs-validate: hidden -->
384+
385+
```csharp
386+
await session.SendAsync(new MessageOptions
387+
{
388+
Prompt = "Describe what you see in this image",
389+
Attachments = new List<UserMessageDataAttachmentsItem>
390+
{
391+
new UserMessageDataAttachmentsItemBlob
392+
{
393+
Data = base64ImageData,
394+
MimeType = "image/png",
395+
DisplayName = "screenshot.png",
396+
},
397+
},
398+
});
399+
```
400+
401+
</details>
402+
218403
## Supported Formats
219404

220-
Supported image formats include JPG, PNG, GIF, and other common image types. The runtime reads the image from disk and converts it as needed before sending to the LLM. Use PNG or JPEG for best results, as these are the most widely supported formats.
405+
Supported image formats include JPG, PNG, GIF, and other common image types. For file attachments, the runtime reads the image from disk and converts it as needed. For blob attachments, you provide the base64 data and MIME type directly. Use PNG or JPEG for best results, as these are the most widely supported formats.
221406

222407
The model's `capabilities.limits.vision.supported_media_types` field lists the exact MIME types it accepts.
223408

@@ -283,10 +468,10 @@ These image blocks appear in `tool.execution_complete` event results. See the [S
283468
|-----|---------|
284469
| **Use PNG or JPEG directly** | Avoids conversion overhead — these are sent to the LLM as-is |
285470
| **Keep images reasonably sized** | Large images may be quality-reduced, which can lose important details |
286-
| **Use absolute paths** | The runtime reads files from disk; relative paths may not resolve correctly |
287-
| **Check vision support first** | Sending images to a non-vision model wastes tokens on the file path without visual understanding |
288-
| **Multiple images are supported** | Attach several file attachments in one message, up to the model's `max_prompt_images` limit |
289-
| **Images are not base64 in your code** | You provide a file path — the runtime handles encoding, resizing, and format conversion |
471+
| **Use absolute paths for file attachments** | The runtime reads files from disk; relative paths may not resolve correctly |
472+
| **Use blob attachments for in-memory data** | When you already have base64 data (e.g., screenshots, API responses), blob avoids unnecessary disk I/O |
473+
| **Check vision support first** | Sending images to a non-vision model wastes tokens without visual understanding |
474+
| **Multiple images are supported** | Attach several attachments in one message, up to the model's `max_prompt_images` limit |
290475
| **SVG is not supported** | SVG files are text-based and excluded from image processing |
291476

292477
## See Also

‎docs/features/streaming-events.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -639,7 +639,7 @@ The user sent a message. Recorded for the session timeline.
639639
|------------|------|----------|-------------|
640640
| `content` | `string` | ✅ | The user's message text |
641641
| `transformedContent` | `string` | | Transformed version after preprocessing |
642-
| `attachments` | `Attachment[]` | | File, directory, selection, or GitHub reference attachments |
642+
| `attachments` | `Attachment[]` | | File, directory, selection, blob, or GitHub reference attachments |
643643
| `source` | `string` | | Message source identifier |
644644
| `agentMode` | `string` | | Agent mode: `"interactive"`, `"plan"`, `"autopilot"`, or `"shell"` |
645645
| `interactionId` | `string` | | CAPI interaction ID |

‎dotnet/README.md‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -271,18 +271,33 @@ session.On(evt =>
271271

272272
## Image Support
273273

274-
The SDK supports image attachments via the `Attachments` parameter. You can attach images by providing their file path:
274+
The SDK supports image attachments via the `Attachments` parameter. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
275275

276276
```csharp
277+
// File attachment — runtime reads from disk
277278
await session.SendAsync(new MessageOptions
278279
{
279280
Prompt = "What's in this image?",
280281
Attachments = new List<UserMessageDataAttachmentsItem>
281282
{
282-
new UserMessageDataAttachmentsItem
283+
new UserMessageDataAttachmentsItemFile
283284
{
284-
Type = UserMessageDataAttachmentsItemType.File,
285-
Path = "/path/to/image.jpg"
285+
Path = "/path/to/image.jpg",
286+
DisplayName = "image.jpg",
287+
}
288+
}
289+
});
290+
291+
// Blob attachment — provide base64 data directly
292+
await session.SendAsync(new MessageOptions
293+
{
294+
Prompt = "What's in this image?",
295+
Attachments = new List<UserMessageDataAttachmentsItem>
296+
{
297+
new UserMessageDataAttachmentsItemBlob
298+
{
299+
Data = base64ImageData,
300+
MimeType = "image/png",
286301
}
287302
}
288303
});

‎dotnet/test/SessionTests.cs‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,29 @@ public async Task DisposeAsync_From_Handler_Does_Not_Deadlock()
538538
await disposed.Task.WaitAsync(TimeSpan.FromSeconds(10));
539539
}
540540

541+
[Fact]
542+
public async Task Should_Accept_Blob_Attachments()
543+
{
544+
var session = await CreateSessionAsync();
545+
546+
await session.SendAsync(new MessageOptions
547+
{
548+
Prompt = "Describe this image",
549+
Attachments =
550+
[
551+
new UserMessageDataAttachmentsItemBlob
552+
{
553+
Data = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==",
554+
MimeType = "image/png",
555+
DisplayName = "test-pixel.png",
556+
},
557+
],
558+
});
559+
560+
// Just verify send doesn't throw — blob attachment support varies by runtime
561+
await session.DisposeAsync();
562+
}
563+
541564
private static async Task WaitForAsync(Func<bool> condition, TimeSpan timeout)
542565
{
543566
var deadline = DateTime.UtcNow + timeout;

‎go/README.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,9 +181,10 @@ Event types: `SessionLifecycleCreated`, `SessionLifecycleDeleted`, `SessionLifec
181181

182182
## Image Support
183183

184-
The SDK supports image attachments via the `Attachments` field in `MessageOptions`. You can attach images by providing their file path:
184+
The SDK supports image attachments via the `Attachments` field in `MessageOptions`. You can attach images by providing their file path, or by passing base64-encoded data directly using a blob attachment:
185185

186186
```go
187+
// File attachment — runtime reads from disk
187188
_, err = session.Send(context.Background(), copilot.MessageOptions{
188189
Prompt: "What's in this image?",
189190
Attachments: []copilot.Attachment{
@@ -193,6 +194,19 @@ _, err = session.Send(context.Background(), copilot.MessageOptions{
193194
},
194195
},
195196
})
197+
198+
// Blob attachment — provide base64 data directly
199+
mimeType := "image/png"
200+
_, err = session.Send(context.Background(), copilot.MessageOptions{
201+
Prompt: "What's in this image?",
202+
Attachments: []copilot.Attachment{
203+
{
204+
Type: copilot.Blob,
205+
Data: &base64ImageData,
206+
MIMEType: &mimeType,
207+
},
208+
},
209+
})
196210
```
197211

198212
Supported image formats include JPG, PNG, GIF, and other common image types. The agent's `view` tool can also read images directly from the filesystem, so you can also ask questions like:

0 commit comments

Comments
 (0)