Feature Request: Support embeddings and OpenAI Realtime API endpoints

[iXandru]

Summary

Feature request: please add/verify OpenAI-compatible support for embeddings and the current OpenAI Realtime API surface in LLM-API-Key-Proxy.

I’m using this proxy as the OpenAI-compatible gateway in front of tools that expect more than chat completions. The main immediate use case is Gbrain as an OpenClaw memory/context engine:

• Gbrain repo: https://github.com/garrytan/gbrain
• Proxy repo: https://github.com/b3nw/LLM-API-Key-Proxy

I checked a local Gbrain clone before rewriting this. Gbrain’s AI gateway supports OpenAI/openai-compatible embedding providers and calls the standard embeddings route for openai-compatible providers. In the relevant path, it posts to:

• POST {baseUrl}/embeddings

So when the configured base URL includes /v1, the proxy needs to expose:

• POST /v1/embeddings

Without that, Gbrain/OpenClaw memory indexing and semantic search can’t route cleanly through the proxy.

Why this matters

For OpenClaw, Gbrain is intended to act as the long-term memory / context engine: importing content, embedding it, and serving hybrid semantic search over that context. That makes the embeddings endpoint a core compatibility requirement, not a nice-to-have.

Realtime support is the second compatibility gap. Live voice / realtime clients use different transports and body types from ordinary /v1/chat/completions or /v1/responses, so supporting only normal JSON chat routes is not enough for OpenAI-compatible realtime model usage.

Requested endpoint support

1. Embeddings

• POST /v1/embeddings

Expected behavior:

• Forward OpenAI-compatible embedding requests and responses without assuming chat-completion semantics.
• Preserve request fields such as model, input, and dimensions where provided.
• Support normal OpenAI-style responses shaped like data[].embedding plus usage metadata.

Minimal acceptance:

• Gbrain can configure an openai-compatible embedding provider pointed at the proxy and successfully embed text through POST /v1/embeddings.

2. Realtime HTTP setup/control endpoints

Please add passthrough support for the current OpenAI Realtime HTTP endpoints, especially:

• POST /v1/realtime/client_secrets
  • Used to mint ephemeral client secrets for browser/mobile/WebRTC/WebSocket clients.
  • Should support session types such as realtime and transcription.
• POST /v1/realtime/calls
  • Used for WebRTC SDP setup.
  • Important: this may use Content-Type: application/sdp, so the proxy should not force JSON-only body handling.
• POST /v1/realtime/transcription_sessions
  • Used for transcription-only realtime sessions / ephemeral credentials.
• POST /v1/realtime/sessions
  • Older/beta-style realtime session creation; useful for compatibility with clients that have not fully migrated.
• POST /v1/realtime/translations
  • Useful if the proxy aims to cover realtime translation flows too.

3. Realtime WebSocket endpoint

• GET /v1/realtime as a WebSocket upgrade endpoint

Expected behavior:

• Preserve query parameters such as ?model=... or ?intent=transcription.
• Support bidirectional WebSocket forwarding.
• Preserve relevant auth, subprotocol, and beta/GA headers where applicable.
• Do not buffer or reinterpret realtime event frames as normal HTTP JSON.

4. Streaming passthrough

Please ensure streaming/chunked/SSE responses are forwarded incrementally rather than buffered until completion, including:

• POST /v1/chat/completions with stream: true
• streaming POST /v1/responses
• any realtime-related streaming response paths where applicable

Implementation notes / gotchas

• Realtime is transport-specific. WebRTC setup uses SDP bodies; WebSocket sessions need HTTP upgrade; normal route forwarding alone will not cover it.
• Ephemeral client secrets should not be logged as plaintext.
• Embeddings should not be routed through chat-completion request/response handling.
• A robust approach may be path/method based passthrough that preserves body type, content type, streaming behavior, and upgrade semantics.

Minimal acceptance criteria

• POST /v1/embeddings through the proxy returns an OpenAI-compatible embeddings response.
• Gbrain can use the proxy as an openai-compatible embedding base URL for memory/context indexing.
• POST /v1/realtime/client_secrets can mint ephemeral realtime credentials through the proxy.
• POST /v1/realtime/calls can pass WebRTC SDP setup through the proxy.
• wss://<proxy>/v1/realtime?model=... can upgrade and forward bidirectional realtime events.
• Streaming responses are forwarded incrementally rather than buffered.

Thanks for maintaining this project — this would make it much more useful as a real OpenAI-compatible gateway for agent stacks, not just chat routing.

[claw-io]

Splitting this into three follow-up issues for the Realtime work (the embeddings + streaming parts are already supported and I'm asking for failing-request evidence on those in the main reply below):

• Support OpenAI Realtime session control endpoints (JSON-only) #54 — Realtime session control endpoints (JSON-only): /v1/realtime/client_secrets, /sessions, /transcription_sessions, /translations
• Support OpenAI Realtime SIP/SDP calls endpoint (POST /v1/realtime/calls) #55 — Realtime SIP/SDP calls endpoint: POST /v1/realtime/calls with application/sdp
• Support OpenAI Realtime WebSocket transport (GET /v1/realtime) #56 — Realtime WebSocket audio transport: GET /v1/realtime?model=…

[claw-io]

Thanks for the detailed write-up — I checked the proxy source and want to give a precise split, because some of this is already supported and some of it is a real gap.

✅ Embeddings — already supported

POST /v1/embeddings is wired up. The route is registered in src/proxy_app/main.py:1423 and the request schema at src/proxy_app/main.py:178 already accepts model, input (string or list), input_type, dimensions, and user — which is everything the OpenAI spec needs for the Gbrain/embedding-provider use case you describe:

# src/proxy_app/main.py:178-183
class EmbeddingRequest(BaseModel):
    model: str
    input: Union[str, List[str]]
    input_type: Optional[str] = None
    dimensions: Optional[int] = None
    user: Optional[str] = None

# src/proxy_app/main.py:1423-1488
@app.post("/v1/embeddings")
async def embeddings(
    request: Request,
    body: EmbeddingRequest,
    client: RotatingClient = Depends(get_rotating_client),
    batcher: Optional[EmbeddingBatcher] = Depends(get_embedding_batcher),
    _=Depends(verify_api_key),
):
    """
    OpenAI-compatible endpoint for creating embeddings.
    Supports two modes based on the USE_EMBEDDING_BATCHER flag:
    - True: Uses a server-side batcher for high throughput.
    - False: Passes requests directly to the provider.
    """
    ...
    response = await client.aembedding(request=request, **request_data)
    return response

When USE_EMBEDDING_BATCHER=true, the response is also assembled with a proper data[] list, model, and usage block (main.py:1469-1478) so it matches the OpenAI response shape Gbrain expects. For the OpenClaw/Gbrain memory-indexing case, POST {baseUrl}/embeddings with baseUrl containing /v1 should "just work" today — POST /v1/embeddings is reachable.

Before we change anything: can you share the failing request? Specifically:

• The exact URL Gbrain is hitting (with /v1 prefix or without),
• The request body Gbrain sends,
• The status code and body of the proxy's response,
• The proxy's log line for that request (from proxy.log or the WebUI Log Explorer).

If the proxy is returning 404, I'd guess a path-mismatch; if it's 400, likely a model-not-routed issue or a JSON-shape mismatch; if it's 200 with an unexpected body, that's a real shape gap and I'd want to see the diff.

✅ Streaming is genuinely incremental — not buffered

Streaming responses on POST /v1/chat/completions (and POST /v1/responses) are forwarded chunk-by-chunk. The handler returns a StreamingResponse(..., media_type="text/event-stream") over a generator that yields inside the async for loop:

# src/proxy_app/main.py:1031-1047
is_streaming = request_data.get("stream", False)

if is_streaming:
    response_generator = await client.acompletion(
        request=request, **request_data
    )
    return StreamingResponse(
        streaming_response_wrapper(
            request, request_data, response_generator, raw_logger
        ),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",
        },
    )

# src/proxy_app/main.py:776-794
async def streaming_response_wrapper(
    request: Request,
    request_data: dict,
    response_stream: AsyncGenerator[str, None],
    logger: Optional[RawIOLogger] = None,
) -> AsyncGenerator[str, None]:
    ...
    try:
        async for chunk_str in response_stream:
            if await request.is_disconnected():
                logging.warning("Client disconnected, stopping stream.")
                break
            yield chunk_str
            ...

X-Accel-Buffering: no also disables nginx-style buffering if you're behind a reverse proxy. Same pattern on /v1/responses at main.py:1152-1192.

❌ OpenAI Realtime — confirmed gap, split into three new issues

The Realtime half of your report is correct. Searched src/ for realtime, client_secrets, realtime/calls, realtime/sessions, realtime/transcription_sessions, and application/sdp — zero hits. The only WebSocket route is a quota-stats push channel, not an audio transport:

# src/proxy_app/main.py:2213
@app.websocket("/v1/ws")
async def websocket_endpoint(websocket: WebSocket):
    ...
    while True:
        ...
        stats = await client.get_quota_stats()
        await websocket.send_json({"type": "quota_stats", "data": stats})
        ...
        await asyncio.sleep(10)

A GET /v1/realtime?model=... audio client connecting here will just receive quota-stats envelopes and nothing else.

I've split the Realtime work into three separate issues because each has a different transport shape, different test matrix, and different shipping risk:

• Realtime session control endpoints (JSON-only: POST /v1/realtime/client_secrets, /sessions, /transcription_sessions, /translations) — stateless JSON passthrough. Bundled because all four share the same handler shape (closest analog is /v1/embeddings at main.py:1423) and ship cleanly in one PR. Initial implementation proposed.
• Realtime SIP/SDP calls (POST /v1/realtime/calls with application/sdp body) — separate issue. Different transport: raw SDP bytes, not JSON. Needs await request.body() instead of await request.json(), body-bytes logging, and Location-header preservation. Initial implementation proposed with tests.
• Realtime WebSocket audio transport (GET /v1/realtime?model=...&intent=transcription) — stateful WebSocket bridge. The biggest design surface of the three. Initial implementation proposed as a passthrough bridge with no provider rotation mid-session, building on CodexWebSocketPool patterns in src/rotator_library/providers/utilities/codex_ws_transport.py. Several open questions (quota accounting, compression, transcription intent routing) called out before coding.

I'll post links to all three new issues in the comments on this one for traceability.

One relevant note for the Realtime implementation

The project already speaks WebSocket outbound to OpenAI on the Codex provider path — src/rotator_library/providers/utilities/codex_ws_transport.py and the WebSocket branch in src/rotator_library/providers/codex_provider.py:1581-1683 (CODEX_USE_WEBSOCKET=true). So the websockets dependency and prior art for session/affinity management already exist; the gap is purely an inbound /v1/realtime* route. Bridging Realtime audio frames through RotatingClient is the design work, not the WebSocket plumbing.

References (HEAD on dev: d77a246 fix(fallback): enable MODEL_FALLBACK for streaming requests)

• src/proxy_app/main.py:1423 — POST /v1/embeddings route
• src/proxy_app/main.py:178-183 — EmbeddingRequest Pydantic model
• src/proxy_app/main.py:948-1104 — chat_completions handler
• src/proxy_app/main.py:1108-1239 — responses_api handler
• src/proxy_app/main.py:776 — streaming_response_wrapper (genuine yield-per-chunk)
• src/proxy_app/main.py:2213 — only WebSocket route in app: /v1/ws (quota push, not Realtime)
• src/rotator_library/providers/utilities/codex_ws_transport.py — existing outbound WS plumbing to OpenAI

[b3nw]

Closing — the two verifiable claims in this report have been confirmed supported by the current code, and the third (Realtime support) is now tracked in three sub-issues.

Verified
POST /v1/embeddings is wired up. Route handler at src/proxy_app/main.py:1431 (@app.post("/v1/embeddings")), Pydantic model EmbeddingRequest at src/proxy_app/main.py:178-183 accepts model, input (string or list), input_type, dimensions, user. The handler supports both batcher mode (USE_EMBEDDING_BATCHER=true) and direct passthrough via client.aembedding(). Response shape includes data[], model, usage — matches the OpenAI response shape for Gbrain and similar consumers.
Streaming is genuinely per-chunk. /v1/chat/completions returns a StreamingResponse over streaming_response_wrapper (main.py:776-798) which async for chunk_str in response_stream: ... yield chunk_str — no buffering, and X-Accel-Buffering: no disables nginx-style buffering behind reverse proxies. Same pattern on /v1/responses at main.py:1152-1192.
Tracked separately
The Realtime gap from the original report has been formally decomposed and filed as three sub-issues with their own transport shapes and review surfaces:

#54 — Realtime session control (JSON-only): stateless passthrough for client_secrets, sessions, transcription_sessions, translations
#55 — Realtime SIP/SDP calls: POST /v1/realtime/calls with application/sdp body
#56 — Realtime WebSocket audio transport: GET /v1/realtime?model=...
No further action under #53 — its content is fully captured by the sub-issues and the original 2026-06-18 reply.