forge init 'Custom' provider produces forge.yaml the runtime cannot consume (custom URL never reaches LLM client)

[initializ-mk]

Summary

Picking the Custom provider during forge init (or via the Web UI wizard) writes a forge.yaml + .env pair that the runtime cannot consume. At forge run time the agent falls back to StubExecutor and every request fails with:

agent execution not configured for framework "forge"

The wizard collects the custom URL, model, and API key correctly — it just writes them to keys that the runtime never reads, while simultaneously writing a provider: value that the LLM client factory rejects outright.

Reproduction

1. forge init → choose Custom provider → enter base URL, model name (e.g. moonshotai/Kimi-K2.6), API key.
2. forge run (or forge serve).
3. Send any task to the agent.

Observed forge.yaml

model:
  provider: custom
  name: moonshotai/Kimi-K2.6

Observed .env

MODEL_BASE_URL=https://your-endpoint.example.com/v1
MODEL_API_KEY=sk-...

Observed log

{"error":"agent execution not configured for framework \"forge\"","level":"error","msg":"execute failed", ...}

If you look just upstream you'll also find:

{"level":"warn","msg":"failed to create LLM client, using stub","error":"unknown LLM provider: \"custom\""}

Root cause

Two disconnected sides of the same feature.

1. The factory rejects provider: custom — forge-core/llm/providers/factory.go:11-27

func NewClient(provider string, cfg llm.ClientConfig) (llm.Client, error) {
    switch provider {
    case "openai":    return NewOpenAIClient(cfg), nil
    case "anthropic": return NewAnthropicClient(cfg), nil
    case "gemini":    ...
    case "ollama":    return NewOllamaClient(cfg), nil
    default:
        return nil, fmt.Errorf("unknown LLM provider: %q", provider)
    }
}

There is no "custom" case. Runner.buildLLMClient propagates the error and the runner falls back to StubExecutor at forge-cli/runtime/runner.go:541-544.

2. The env vars the wizard writes are never read at runtime — forge-cli/cmd/init.go:389-395

if ctx.CustomBaseURL != "" {
    opts.EnvVars["MODEL_BASE_URL"] = ctx.CustomBaseURL
}
if ctx.CustomAPIKey != "" {
    opts.EnvVars["MODEL_API_KEY"] = ctx.CustomAPIKey
}

But ResolveModelConfig at forge-core/runtime/config.go:80-88 only honors per-provider variables:

if u := envVars["OPENAI_BASE_URL"]; u != "" && mc.Provider == "openai" { ... }
if u := envVars["ANTHROPIC_BASE_URL"]; u != "" && mc.Provider == "anthropic" { ... }
if u := envVars["OLLAMA_BASE_URL"]; u != "" && mc.Provider == "ollama" { ... }

MODEL_BASE_URL is grepped only by forge-ui (the wizard frontend) and the init renderer — no consumer in the runtime. Same story for MODEL_API_KEY (it's in runner.go:2672's builtinSecretKeys list for the secret overlay, but ResolveModelConfig never wires it into mc.Client.APIKey).

The Web UI wizard (forge-ui/handlers_create.go:81) emits the same MODEL_BASE_URL shape, so this affects both the TUI and the Web UI paths.

Expected behavior

forge init's Custom option is documented and presented as the way to point Forge at any OpenAI-compatible endpoint (OpenRouter, litellm, vLLM, Together AI, local proxies, etc.). The wizard collects all three required inputs. The output should produce a forge run that successfully constructs an LLM client and routes to the configured endpoint without manual forge.yaml / .env edits.

Suggested fix

Two plausible shapes — either works; my read is option A is the smallest correct change.

Option A — Wire Custom through the existing OpenAI-compatible path

In forge-cli/cmd/init.go, when the wizard returns ctx.CustomBaseURL != "":

• Write provider: openai (not custom) into forge.yaml.
• Write OPENAI_BASE_URL=<custom URL> and OPENAI_API_KEY=<custom key> into .env (instead of MODEL_BASE_URL / MODEL_API_KEY).

This needs zero changes in forge-core/. The existing OpenAI client + the existing ResolveModelConfig env-var branch handle it. Banner will read Model: openai/<custom-model-name> which is honest about the wire protocol.

Trade-off: loses the "Custom" label in the banner. Could be preserved with a comment in the generated forge.yaml if that matters.

Option B — Make custom a first-class provider type

• Add case "custom": to forge-core/llm/providers/factory.go that constructs an OpenAIClient with cfg.BaseURL and cfg.APIKey pre-populated.
• Add BaseURL string + BaseURLEnv string to types.ModelSpec (currently only Provider, Name, Version, OrganizationID, Fallbacks).
• Extend ResolveModelConfig to honor MODEL_BASE_URL / MODEL_API_KEY when mc.Provider == "custom".

More invasive (touches the schema + the factory + the resolver + their tests) but keeps provider: custom as a documented YAML value with explicit semantics.

Acceptance criteria

• forge init → Custom → forge run (no manual edits) → primary LLM client constructs and serves a request end-to-end.
• Banner reflects the configured custom endpoint accurately.
• Test in forge-cli/cmd/init_test.go pins the generated forge.yaml + .env shape for the Custom path.
• Integration test in forge-cli/runtime/ proves Runner.buildLLMClient succeeds for the Custom config shape (using httptest.NewServer as the OpenAI-compatible target).
• forge-ui/handlers_create.go:81 produces the same shape as the TUI.
• Migration note in CHANGELOG for any users who already have provider: custom in their checked-in forge.yaml.

Workaround (until fix lands)

Edit the generated files manually:

# forge.yaml
model:
  provider: openai
  name: moonshotai/Kimi-K2.6

# .env
OPENAI_BASE_URL=https://your-endpoint.example.com/v1
OPENAI_API_KEY=sk-...

Environment

• forge built from main (commit a572b21).
• Reproduced with provider: custom + name: moonshotai/Kimi-K2.6 against an OpenRouter-shaped endpoint.
• framework: forge, Telegram channel, default builtin tools.

[initializ-mk]

Update — second failure mode found after applying the workaround

After applying the documented workaround (changing provider: custom → provider: openai in forge.yaml and adding OPENAI_BASE_URL to .env), the agent still fails — this time silently routing to ChatGPT's backend instead of the configured endpoint:

{"base_url":"https://chatgpt.com/backend-api/codex","msg":"using OAuth credentials for provider","provider":"openai"}
...
{"error":"responses api stream error (status 400): {\"detail\":\"The 'moonshotai/Kimi-K2.6' model is not supported when using Codex with a ChatGPT account.\"}"}

This expands the root cause of #83 beyond MODEL_BASE_URL. There are actually two broken wires from the wizard, plus a latent precedence bug.

Wire 2 — MODEL_API_KEY is also written by the wizard but never read at runtime

forge-cli/cmd/init.go:393-395 writes:

if ctx.CustomAPIKey != "" {
    opts.EnvVars["MODEL_API_KEY"] = ctx.CustomAPIKey
}

But resolveAPIKey at forge-core/runtime/config.go:230-253 only reads provider-specific keys:

case "openai":
    if k := envVars["OPENAI_API_KEY"]; k != "" { mc.Client.APIKey = k }
    else if k := envVars["LLM_API_KEY"]; k != "" { mc.Client.APIKey = k }

MODEL_API_KEY is in runner.go:2672's builtinSecretKeys (so it loads from the encrypted secret store into the env), but no resolver branch reads it. So even after the user manually flips provider: custom → provider: openai, cfg.APIKey stays empty unless they ALSO rename MODEL_API_KEY → OPENAI_API_KEY in .env.

Latent bug — stored OAuth credentials silently override an explicit OPENAI_BASE_URL

forge-cli/runtime/runner.go:1682:

needsOAuth := provider == "openai" && (cfg.APIKey == "" || cfg.APIKey == "__oauth__")
if needsOAuth {
    token, err := oauth.LoadCredentials(provider)
    if err == nil && token != nil && token.RefreshToken != "" {
        // ...
        cfg.APIKey = token.AccessToken
        cfg.BaseURL = baseURL   // <-- overrides whatever OPENAI_BASE_URL set
        return providers.NewOAuthClient(cfg, provider, oauthCfg), nil
    }

If the user (a) has any stored OpenAI OAuth credential from a previous forge init run AND (b) hasn't set OPENAI_API_KEY (because the wizard wrote MODEL_API_KEY per Wire 2 above), the OAuth path wins and silently replaces the user's explicit OPENAI_BASE_URL with the ChatGPT Codex backend URL. The user has no signal — the banner still reads Model: openai/<their-model> — and ChatGPT then rejects the unknown model name with an error that points away from the real cause.

Suggested guardrails (in priority order):

1. An explicit OPENAI_BASE_URL should disqualify the OAuth-credentials path, because OAuth's BaseURL is mutually exclusive with a user-supplied endpoint. Either skip OAuth when cfg.BaseURL != "", or fail loud with a clear error explaining the conflict.
2. At minimum, log a WARN — not INFO — when OAuth credentials override an explicit OPENAI_BASE_URL. The current INFO log buries the fact that an explicit setting was ignored.

Revised fix recommendation (still favoring Option A from the original report)

The minimal, correct fix in forge-cli/cmd/init.go:389-395 is to write the provider-specific env vars that the resolver already reads, not the MODEL_* aliases:

// Was
opts.EnvVars["MODEL_BASE_URL"] = ctx.CustomBaseURL
opts.EnvVars["MODEL_API_KEY"]  = ctx.CustomAPIKey

// Should be (for the Custom-as-OpenAI-compatible path)
opts.EnvVars["OPENAI_BASE_URL"] = ctx.CustomBaseURL
opts.EnvVars["OPENAI_API_KEY"]  = ctx.CustomAPIKey

Plus forge-ui/handlers_create.go:81 (BaseURLEnv: "MODEL_BASE_URL") needs the matching change so the Web UI wizard emits the same shape.

Plus the OAuth-precedence guardrail above, so a user with a leftover OAuth credential doesn't get silently redirected to ChatGPT.

Revised acceptance criteria

Adding to the original list:

• When provider: openai + OPENAI_BASE_URL is set, the OAuth credentials path is either skipped or surfaces a clear error explaining the conflict — never silently overrides the user's OPENAI_BASE_URL.
• forge-cli/cmd/init.go emits OPENAI_BASE_URL and OPENAI_API_KEY (not MODEL_*) for the Custom-provider wizard path.
• forge-ui/handlers_create.go:81 emits the same shape as the TUI.
• Test that pins: stored OAuth credentials + explicit OPENAI_BASE_URL → either OAuth is skipped or the runner fails loud at startup, not via a 400 from the wrong endpoint at request time.

Updated full workaround

# forge.yaml
model:
  provider: openai
  name: moonshotai/Kimi-K2.6

# .env — note the keys must be OPENAI_*, not MODEL_*
OPENAI_BASE_URL=https://your-endpoint.example.com/v1
OPENAI_API_KEY=sk-...

If you previously ran the OAuth path during forge init, also clear the leftover credential so it can't silently override your explicit endpoint:

rm ~/.forge/credentials/openai*.json   # path varies per the credential-store layout

(Setting OPENAI_API_KEY alone is enough to bypass the OAuth path, but removing the stale credential is a clean follow-up.)