Structured output (`output_type`) is enforced on every turn via `response_format`, suppressing tool calls on strict OpenAI-compatible servers

[harche]

Please read this first

• Have you read the docs? Yes.
• Have you searched for related issues? Yes.

Describe the bug

When an Agent is configured with both tools and an output_type, the SDK sends the output schema as response_format / text.format on every model call:

• openai_responses.py#L781 — response_format = Converter.get_response_format(output_schema)
• openai_responses.py#L857 — "text": response_format

Against OpenAI's hosted endpoint this is tolerated, because the server lets tool calls bypass a set response_format. Against strict OpenAI-compatible servers (e.g. vLLM) the guided-decoding grammar is enforced from the first generated token, which overrides tool_choice and suppresses tool calls entirely — the model is forced into the JSON schema on turn 1 and never calls its tools, hallucinating a schema-shaped answer instead.

Net effect: tool-using agents that also require structured output cannot call their tools on non-OpenAI backends (custom base_url, LiteLLM, vLLM, …).

Debug information

• Agents SDK version: 0.17.7 (also reproduces on main)
• Python version: 3.12
• Backend: vLLM serving google/gemma-4-31B-it via the OpenAI-compatible /v1 API

Repro steps

A minimal /v1/chat/completions request against a vLLM-served model, with a tool and a response_format, reproduces the underlying server behavior without the SDK:

{
  "model": "<any tool-capable model>",
  "messages": [{"role": "user", "content": "Find the token by calling get_token, then answer."}],
  "tools": [{"type": "function", "function": {"name": "get_token", "parameters": {"type": "object", "properties": {}}}}],
  "tool_choice": "auto",
  "response_format": {"type": "json_schema", "json_schema": {"name": "a", "strict": true,
    "schema": {"type": "object", "additionalProperties": false, "required": ["token"],
               "properties": {"token": {"type": "string"}}}}}
}

Result: finish_reason: "stop", tool_calls: null, content = a hallucinated {"token": "…"}. The tool is never called — even when tool_choice explicitly names the function. Drop the response_format and the same request calls the tool correctly.

(vLLM tracks the composition gap in vllm-project/vllm#17481 and RFC vllm-project/vllm#19097.)

Expected behavior

A tool-using agent with an output_type should still call its tools on OpenAI-compatible servers. The structured schema is a property of the final answer; tool-calling turns shouldn't be constrained by it.

Why this is worth addressing in the SDK, not just waiting on servers

There's no inherent reason to enforce the typed format on every turn. The SDK currently relies on OpenAI's server special-casing tool calls vs. response_format, which makes structured-output + tools non-portable across the OpenAI-compatible ecosystem the SDK otherwise supports.

Prior art: the Anthropic Agent SDK does this the portable way

The Anthropic Agent SDK treats structured output as a run-level option returned on the final result, decoupled from per-turn generation:

• ClaudeAgentOptions.output_format — "When specified, the agent returns structured data matching the schema."
• ResultMessage.structured_output — delivered once, on the terminal result message.

Tools run unconstrained through the loop; the schema is applied to the final output only.

Proposed change

Add an opt-in path (default off, so OpenAI-hosted behavior is unchanged) that defers structured output:

1. Run the agent loop without sending output_schema as text.format (so tool calls are never suppressed).
2. Once the model produces a final response with no tool calls, perform a final tool-free "structuring" model call that does apply the schema, and validate that into final_output.

This mirrors the Anthropic SDK's model and makes structured-output + tools portable to any OpenAI-compatible server while leaving the OpenAI-hosted fast path intact.

A reference application-level workaround (deferred structuring implemented in a provider) is here:
harche/lightspeed-agentic-sandbox@e6c0560

A PR implementing the opt-in SDK change is linked below.

[seratch]

Thanks for the detailed report and reproduction. I confirmed that the SDK passes output_type to each model invocation. However, the failure described here is not a general problem with custom base_url or OpenAI-compatible providers. OpenAI's API supports tool definitions and a structured response_format in the same request; the reported failure occurs because vLLM currently cannot reliably compose those two features and lets its guided-decoding constraint suppress tool calls.

The SDK already supports this provider-specific limitation through explicit two-stage orchestration or a custom Model/ModelProvider: run the vLLM tool-calling step without an output schema, then make a separate tool-free structured-output call. This keeps the extra request, cost, retries, hooks, and turn accounting explicit.

Adding a vLLM compatibility mode to the shared runner would introduce a hidden model call across streaming, server-managed state, approvals, and resume paths for behavior that other providers do not require. I am therefore going to close this issue and the linked PR. We can reconsider a provider-scoped adapter or documentation improvement if this limitation affects additional backends or cannot be handled through the existing extension points.

[harche]

Thanks @seratch , I will handle this in my application with something like harche/lightspeed-agentic-sandbox@23950a6