Merge branch 'feat/proxy-exchange-capture' into dev

Author: jcaose

README.md

@@ -36,6 +36,7 @@ A reverse-engineered proxy for the GitHub Copilot API that exposes it as an Open
 - **Usage Dashboard**: A web-based dashboard to monitor your Copilot API usage, view quotas, and see detailed statistics.
 - **Rate Limit Control**: Manage API usage with rate-limiting options (`--rate-limit`) and a waiting mechanism (`--wait`) to prevent errors from rapid requests.
 - **Manual Request Approval**: Manually approve or deny each API request for fine-grained control over usage (`--manual`).
+- **Opt-In Exchange Capture**: Persist proxied request and response exchanges to JSONL for debugging or audit purposes (`--capture`, `--capture-path`).
 - **Token Visibility**: Option to display GitHub and Copilot tokens during authentication and refresh for debugging (`--show-token`).
 - **Flexible Authentication**: Authenticate interactively or provide a GitHub token directly, suitable for CI/CD environments.
 - **Support for Different Account Types**: Works with individual, business, and enterprise GitHub Copilot plans.
@@ -161,6 +162,8 @@ The following command line options are available for the `start` command:
 | --wait         | Wait instead of error when rate limit is hit                                  | false      | -w    |
 | --github-token | Provide GitHub token directly (must be generated using the `auth` subcommand) | none       | -g    |
 | --claude-code  | Generate a command to launch Claude Code with Copilot API config              | false      | -c    |
+| --capture      | Persist proxied request and response exchanges to disk                        | false      | none  |
+| --capture-path | Path to the JSONL file used for exchange capture                              | auto       | none  |
 | --show-token   | Show GitHub and Copilot tokens on fetch and refresh                           | false      | none  |
 | --proxy-env    | Initialize proxy from environment variables                                   | false      | none  |
 
@@ -238,6 +241,12 @@ npx copilot-api@latest start --rate-limit 30 --wait
 # Provide GitHub token directly
 npx copilot-api@latest start --github-token ghp_YOUR_TOKEN_HERE
 
+# Persist proxied exchanges to the default capture file
+npx copilot-api@latest start --capture
+
+# Persist proxied exchanges to a specific JSONL file
+npx copilot-api@latest start --capture --capture-path ./captures/session.jsonl
+
 # Run only the auth flow
 npx copilot-api@latest auth
 
@@ -269,6 +278,20 @@ After starting the server, a URL to the Copilot Usage Dashboard will be displaye
     `https://ericc-ch.github.io/copilot-api?endpoint=http://localhost:4141/usage`
     - If you use the `start.bat` script on Windows, this page will open automatically.
 
+## Exchange Capture
+
+When `--capture` is enabled, the proxy appends one JSON object per proxied exchange to a JSONL file. Captures include the request body, the final response body, request metadata, token usage, and correlation fields such as request IDs when available.
+
+By default, capture files are written under:
+
+```sh
+~/.local/share/copilot-api/captures/YYYY-MM-DD.jsonl
+```
+
+Use `--capture-path` to write to a specific JSONL file instead. Sensitive values such as authorization headers and token-like fields are redacted before they are written to disk.
+
+When using the default daily capture path, the current day remains an appendable `.jsonl` file and older daily capture files are automatically compressed to `.jsonl.gz`.
+
 The dashboard provides a user-friendly interface to view your Copilot usage data:
 
 - **API Endpoint URL**: The dashboard is pre-configured to fetch data from your local server endpoint via the URL query parameter. You can change this URL to point to any other compatible API endpoint.

docs/rfcs/opt-in-proxy-exchange-capture.md

@@ -0,0 +1,152 @@
+# RFC: Opt-In Proxy Exchange Capture
+
+## Summary
+
+Add an opt-in runtime feature that persists proxied request and response exchanges to disk as JSONL. The feature is disabled by default and does not change existing proxy behavior or terminal usage logging.
+
+## Motivation
+
+The proxy currently provides terminal summaries for request usage, but not a durable record of payloads, response IDs, tool-call traffic, or translated streamed output. That makes debugging translation issues and upstream behavior harder than it needs to be.
+
+An opt-in capture mode gives us persistent, structured traces without making default runs noisy or risky.
+
+## Goals
+
+- Keep capture disabled by default.
+- Persist request and final response bodies for proxied model traffic.
+- Preserve current client-visible behavior, especially for streaming endpoints.
+- Redact sensitive values before writing to disk.
+- Keep storage append-friendly and easy to inspect manually.
+- Reduce disk growth for older captures automatically.
+
+## Non-Goals
+
+- Capturing GitHub auth and token refresh flows.
+- Persisting raw SSE streams byte-for-byte.
+- Replacing existing terminal logging.
+- Building a UI or search layer for captured data.
+
+## User Interface
+
+New `start` options:
+
+- `--capture`
+- `--capture-path <path>`
+
+Behavior:
+
+- `--capture` enables persistence.
+- `--capture-path` overrides the default JSONL location.
+- Without `--capture`, no exchanges are written to disk.
+- In default-path mode, the current day stays as plain `.jsonl` and prior-day files are automatically gzipped.
+
+Default path:
+
+```text
+~/.local/share/copilot-api/captures/YYYY-MM-DD.jsonl
+```
+
+## Design
+
+A dedicated capture module is responsible for:
+
+- generating and carrying request correlation IDs
+- appending JSONL records
+- redacting token-like values
+- measuring request and response byte sizes
+- reconstructing final streamed responses for persistence
+- compressing prior-day default capture files to `.jsonl.gz`
+
+Capture records are intentionally aligned with the proxy's effective upstream interaction, not a byte-for-byte copy of the original inbound request. In practice that means:
+
+- normalized request fields added by the proxy, such as model normalization or inferred limits, are reflected in the saved record
+- compatibility routes may persist translated request and response shapes rather than the original client-facing wire format
+- the goal of capture is operational debugging of what the proxy effectively sent and received, not raw inbound replay fidelity
+
+Each persisted record includes:
+
+- timestamp
+- route and HTTP method
+- upstream target path
+- model and reasoning level when available
+- local request ID
+- upstream status
+- upstream response ID when available
+- `previous_response_id` when available
+- request bytes
+- response bytes
+- request body
+- final response body
+- usage totals when available
+
+For compatibility routes, the saved bodies should be interpreted as the proxy's translated interaction model. They are useful for debugging behavior through the proxy, but they are not guaranteed to exactly match the original client payload shape.
+
+## Streaming
+
+For streaming endpoints, the proxy continues forwarding events unchanged to the client. In parallel, it reconstructs a final logical response for persistence.
+
+This applies to:
+
+- OpenAI chat completions streams
+- Responses API streams routed back to chat completions
+- Anthropic message streams translated from chat completions
+
+The design intentionally stores the final reconstructed response instead of raw SSE chunks. That keeps logs compact and directly usable, at the cost of losing exact chunk timing and wire-level fidelity.
+
+## Storage Lifecycle
+
+The active capture file remains uncompressed so the proxy can append to it efficiently.
+
+When capture uses the default daily path:
+
+- today's file is written as `YYYY-MM-DD.jsonl`
+- older `.jsonl` files are automatically gzip-compressed to `.jsonl.gz`
+- compression runs when capture writes begin for a new process/day
+- compressed files are not reopened for append
+
+This keeps the hot file simple while reducing long-term storage usage for older days.
+
+## Redaction
+
+Before any exchange is written to disk, the capture layer redacts:
+
+- authorization headers
+- token-like keys
+- token-like string values
+
+This keeps the default capture mode suitable for debugging without writing obvious credentials to disk.
+
+## Scope
+
+Capture is implemented for proxied model traffic:
+
+- `/v1/chat/completions`
+- `/v1/responses`
+- `/v1/messages`
+- `/v1/embeddings`
+
+Auth and token management flows are intentionally out of scope.
+
+## Change Summary
+
+Implemented changes on branch `feat/proxy-exchange-capture`:
+
+- added CLI flags in `src/start.ts`
+- added capture config to runtime state
+- added default capture path handling
+- added `src/lib/exchange-capture.ts` for persistence, redaction, and stream reconstruction
+- added automatic gzip compression for prior daily capture files in default-path mode
+- threaded request IDs and upstream status through Copilot service calls
+- integrated persistence into chat, responses, anthropic, and embeddings routes
+- documented the feature in `README.md`
+- added unit coverage for redaction and stream reconstruction
+- added unit coverage for previous-day gzip compression
+
+## Validation
+
+Validated in the feature worktree with:
+
+- `bun test`
+- `bunx --bun tsc --noEmit`
+- `bunx --bun eslint --cache src tests`
+- `bunx --bun tsdown`

src/lib/api-config.ts

@@ -17,7 +17,11 @@ export const copilotBaseUrl = (state: State) =>
   state.accountType === "individual" ?
     "https://api.githubcopilot.com"
   : `https://api.${state.accountType}.githubcopilot.com`
-export const copilotHeaders = (state: State, vision: boolean = false) => {
+export const copilotHeaders = (
+  state: State,
+  vision: boolean = false,
+  requestId: string = randomUUID(),
+) => {
   const headers: Record<string, string> = {
     Authorization: `Bearer ${state.copilotToken}`,
     "content-type": standardHeaders()["content-type"],
@@ -27,7 +31,7 @@ export const copilotHeaders = (state: State, vision: boolean = false) => {
     "user-agent": USER_AGENT,
     "openai-intent": "conversation-panel",
     "x-github-api-version": API_VERSION,
-    "x-request-id": randomUUID(),
+    "x-request-id": requestId,
     "x-vscode-user-agent-library-version": "electron-fetch",
   }