Skip to content

copilot-runtime.mdx

Latest commit

 

History

History
341 lines (260 loc) · 13.4 KB

copilot-runtime.mdx

File metadata and controls

341 lines (260 loc) · 13.4 KB
title CopilotRuntime
description Server-side runtime for hosting CopilotKit agents

The CopilotRuntime package gives you everything you need to host CopilotKit agents on your own infrastructure. It owns the agent registry, wires an AgentRunner, exposes HTTP endpoints (Express or Hono), and optionally coordinates middleware and transcription services.

Basic Setup

import { CopilotRuntime, InMemoryAgentRunner } from "@copilotkit/runtime";
import { BasicAgent } from "@copilotkit/runtime/v2";

const runtime = new CopilotRuntime({
  agents: {
    default: new BasicAgent({
      model: "openai/gpt-4o",
      prompt: "You are a helpful assistant.",
    }),
  },
  runner: new InMemoryAgentRunner(),
});

Once you create a runtime instance you can mount one of the provided HTTP endpoints (Express or Hono) described below.

CopilotRuntime Constructor

new CopilotRuntime(options: CopilotRuntimeOptions)

CopilotRuntimeOptions

Option Type Description
agents Promise<Record<string, AbstractAgent>> | Record<string, AbstractAgent> Required. A map from agent identifier to an AbstractAgent. You can return the map synchronously or asynchronously.
runner AgentRunner Optional. Defaults to new InMemoryAgentRunner(). Controls how agent runs are scheduled and streamed.
transcriptionService TranscriptionService Optional. Enables /transcribe and reports audioFileTranscriptionEnabled: true from /info.
beforeRequestMiddleware BeforeRequestMiddleware Optional. Runs before every request. Can mutate the incoming Request or return void.
afterRequestMiddleware AfterRequestMiddleware Optional. Runs after every handler resolves. Receives the response along with parsed messages, thread ID, and run ID extracted from the SSE stream. Use this for logging, metrics, telemetry, etc.

Passing Agents

  • Provide a plain object mapping agent ids to instances. The helper BasicAgent covers common OpenAI/Anthropic/Gemini setups, but you can supply any AbstractAgent.
  • Because agents may be an async function, you can lazily load credentials or fetch configuration before returning the record.
  • Each request clones the selected agent instance so per-request state (messages, headers, tool registration) does not leak between threads.
  • Authorization and x-* headers on the incoming request are forwarded to the cloned agent automatically. Override or strip them inside beforeRequestMiddleware if needed.
const runtime = new CopilotRuntime({
  agents: async () => ({
    default: buildDefaultAgent(),
    support: buildSupportAgent(),
  }),
});

Choosing an AgentRunner

InMemoryAgentRunner is the default and streams events directly from your Node.js process. Swap it with a custom AgentRunner if you need to enqueue work or forward to another service:

class QueueBackedRunner implements AgentRunner {
  /* ... */
}

const runtime = new CopilotRuntime({
  agents,
  runner: new QueueBackedRunner(),
});

Middleware Hooks

const runtime = new CopilotRuntime({
  agents,
  beforeRequestMiddleware: async ({ request, path }) => {
    const auth = request.headers.get("authorization");
    if (!auth) {
      throw new Response("Unauthorized", { status: 401 });
    }
  },
  afterRequestMiddleware: async ({
    response,
    path,
    messages,
    threadId,
    runId,
  }) => {
    console.info("Handled", path, response.status);
    console.info("Thread:", threadId, "Run:", runId);
    console.info("Messages:", messages);
  },
});

Throwing a Response from beforeRequestMiddleware short-circuits the request. afterRequestMiddleware runs in the background and should swallow its own errors.

AfterRequestMiddlewareParameters

Parameter Type Description
runtime CopilotRuntime The runtime instance.
response Response A clone of the response sent to the client. The body is still readable.
path string The matched route path (e.g. /agent/default/run).
messages Message[] Reconstructed messages from the SSE stream. Empty for non-SSE responses. Each message has id, role, and optionally content, toolCalls, or toolCallId.
threadId string | undefined Thread ID from the RUN_STARTED event.
runId string | undefined Run ID from the RUN_STARTED event.

This makes afterRequestMiddleware suitable for telemetry, audit logging, and post-run analytics without needing to manually parse the streamed response.

Audio Transcription

Enable audio transcription by providing a transcriptionService. The @copilotkit/voice package includes providers:

import { CopilotRuntime } from "@copilotkit/runtime";
import { TranscriptionServiceOpenAI } from "@copilotkit/voice";
import OpenAI from "openai";

const runtime = new CopilotRuntime({
  agents: { default: yourAgent },
  transcriptionService: new TranscriptionServiceOpenAI({
    openai: new OpenAI({ apiKey: process.env.OPENAI_API_KEY }),
  }),
});

This enables the /transcribe endpoint and shows a microphone button in the chat UI. The /info endpoint will report audioFileTranscriptionEnabled: true.

For custom providers, extend TranscriptionService from runtime:

import {
  TranscriptionService,
  TranscribeFileOptions,
} from "@copilotkit/runtime";

class MyTranscriptionService extends TranscriptionService {
  async transcribeFile(options: TranscribeFileOptions): Promise<string> {
    // options.audioFile, options.mimeType, options.size
    return "transcribed text";
  }
}

HTTP Endpoints

Import the helper that matches your server framework and the transport style you want to expose.

Hono (REST-style)

import { createCopilotEndpoint } from "@copilotkit/runtime";

const copilot = createCopilotEndpoint({ runtime, basePath: "/api/copilotkit" });

const app = new Hono();
app.route("/", copilot);

This mounts five routes under the base path:

Method Path Purpose
POST /agent/:agentId/run Streams agent events (SSE).
POST /agent/:agentId/connect Creates a live WebSocket/stream connection.
POST /agent/:agentId/stop/:threadId Cancels an in-flight thread.
GET /info Returns runtime metadata and agent list.
POST /transcribe Proxies audio transcription (requires transcriptionService).

Hono (Single-route)

import { createCopilotEndpointSingleRoute } from "@copilotkit/runtime";

const copilot = createCopilotEndpointSingleRoute({
  runtime,
  basePath: "/api/copilotkit",
});

const app = new Hono();
app.route("/", copilot);

All interactions happen through a single POST /api/copilotkit endpoint. The client sends a JSON envelope:

{
  "method": "agent/run",
  "params": { "agentId": "default" },
  "body": { "messages": [...], "threadId": "thread-123" }
}

Allowed method values are:

  • agent/run
  • agent/connect
  • agent/stop
  • info
  • transcribe

Pair this server endpoint with the React provider flag <CopilotKitProvider useSingleEndpoint />.

Next.js (App Router) with Hono

When you're inside a Next.js app/ route, export the Hono handlers directly:

// app/api/copilotkit/[[...slug]]/route.ts
import { CopilotRuntime, createCopilotEndpoint } from "@copilotkit/runtime";
import { handle } from "hono/vercel";

const runtime = new CopilotRuntime({ agents, runner });
const app = createCopilotEndpoint({ runtime, basePath: "/api/copilotkit" });

export const GET = handle(app);
export const POST = handle(app);

Swap createCopilotEndpoint for createCopilotEndpointSingleRoute if you configure the client with useSingleEndpoint.

Express (REST-style)

import express from "express";
import { createCopilotEndpointExpress } from "@copilotkit/runtime/express";

const app = express();
app.use(
  "/api/copilotkit",
  createCopilotEndpointExpress({ runtime, basePath: "/" }),
);

The router mirrors the Hono REST routes and automatically enables permissive CORS (allowing all origins, headers, and methods). Adjust headers upstream if you need tighter controls.

Express (Single-route)

import express from "express";
import { createCopilotEndpointSingleRouteExpress } from "@copilotkit/runtime/express";

const app = express();
app.use(
  "/api/copilotkit",
  createCopilotEndpointSingleRouteExpress({ runtime, basePath: "/" }),
);

Single-route Express works identically to the Hono version: send the JSON envelope described earlier, and set useSingleEndpoint on the client.

NestJS (Express backend)

NestJS uses Express by default. Mount the Express router in main.ts:

// main.ts
import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module";
import { NestExpressApplication } from "@nestjs/platform-express";
import { CopilotRuntime } from "@copilotkit/runtime";
import { createCopilotEndpointExpress } from "@copilotkit/runtime/express";
import { BasicAgent } from "@copilotkit/runtime/v2";

async function bootstrap() {
  const app = await NestFactory.create<NestExpressApplication>(AppModule);

  const runtime = new CopilotRuntime({
    agents: {
      default: new BasicAgent({
        model: "openai/gpt-4o",
        prompt: "You are helpful.",
      }),
    },
  });

  // Mount under /api/copilotkit (REST-style routes)
  app.use(
    "/api/copilotkit",
    createCopilotEndpointExpress({ runtime, basePath: "/" }),
  );

  await app.listen(3000);
}
bootstrap();

Available routes (no global prefix):

  • POST /api/copilotkit/agent/:agentId/run
  • POST /api/copilotkit/agent/:agentId/connect
  • POST /api/copilotkit/agent/:agentId/stop/:threadId
  • GET /api/copilotkit/info
  • POST /api/copilotkit/transcribe

If you prefer the single-route transport, mount the single-route helper instead and set useSingleEndpoint on the client:

import { createCopilotEndpointSingleRouteExpress } from "@copilotkit/runtime/express";

app.use(
  "/api/copilotkit",
  createCopilotEndpointSingleRouteExpress({ runtime, basePath: "/" }),
);

Notes:

  • If you call app.setGlobalPrefix('api'), the effective paths become /api/copilotkit/... (or /api/copilotkit for single-route).
  • Endpoints stream Server‑Sent Events; ensure your proxy honors text/event-stream and keep‑alive.
  • The router enables permissive CORS; tighten via Nest’s enableCors if needed.

Runtime Metadata

Calling GET /info (or the info single-route method) returns:

{
  "version": "0.0.20",
  "agents": {
    "default": {
      "name": "default",
      "className": "BasicAgent",
      "description": "You are a helpful assistant."
    }
  },
  "audioFileTranscriptionEnabled": false
}

Use this to verify deployments and surface diagnostics in your UI.

Next Steps

  • Configure the React client with runtimeUrl (and useSingleEndpoint if you chose the single endpoint helper).
  • Register frontend tools with CopilotKitProvider so agents can trigger UI actions.
  • Extend the runtime runner or middleware hooks to integrate logging, rate limiting, or background queues.