[Epic] Gradient plumbing: objective gradients & residual Jacobians from the BNGsim output-sensitivity tensor

[wshlavacek]

Epic / tracker. Surface objective gradients and least-squares residual Jacobians
w.r.t. free parameters by consuming BNGsim's output_sensitivities tensor, so PyBNF can host
gradient-based optimizers (#386). This issue tracks the work; the implementation ships as the
dependency-ordered child issues below, each independently verified by a finite-difference check
against PyBNF's own loss(θ).

Backend status — VERIFIED (bngsim 0.10.14, 2026-06-27)

• bngsim.capabilities()["features"]["output_sensitivities"] → True.
• Result.output_sensitivities(selectors, *, axis="parameter") returns ∂g/∂θ in native
  parameter space (no log/scale transform — PyBNF owns that, ADR-0029).
• Simulator(..., sensitivity_params=, sensitivity_ic=, sensitivity_method=) accepts the request.
• No further BNGsim work is needed for the deterministic-ODE path.

Scope correction (edition-2)

This epic is edition-2 only (edition >= 2), and the parameter-mapping is not the classic
__FREE/__REF story the original draft used. Edition-2 binds free parameters to the model
by id (ADR-0034) and expresses per-condition values as Mutation(var, operation, value)
perturbations (ADR-0028). So "per-condition pinning" is just the perturbation operator's local
derivative: = → zero that column; *c//c → chain-rule factor; +/- → factor 1. No
surrogate recovery. Forward sensitivities are deterministic-ODE only — network-free/stochastic
models are never candidates.

Convention pin (the reconciliation the optimizer must trust)

PyBNF's Gaussian loss is Σ r²/(2σ²), r = pred − obs. Define the least-squares residual
ρ_i = (pred_i − obs_i)/σ_i. Then loss = ½‖ρ‖², residual-Jacobian J_ij = (1/σ_i)·∂pred_i/∂θ_j
(θ native), and ∂loss/∂θ = Jᵀρ (not 2Jᵀρ). scipy.least_squares minimizes ½‖ρ‖² with the
same ρ/J, so the residual form and the scalar form agree by construction. Sampling-space applies
J → J·diag(∂θ/∂u) once, via priors/scale.py (ADR-0029).

Cut 1 — usable-gradient MVP (strict order)

• A — Capability gate + sensitivity-tensor preservation (net backend) — A — Gradient plumbing: capability gate + preserve the output-sensitivity tensor through net execution #447
• B — Free-param → sensitivity_params/sensitivity_ic routing (bind-by-id; condition operator chain-rule) — B — Gradient plumbing: route free params to sensitivity_params/sensitivity_ic (bind-by-id + condition chain-rule) #448
• C — Gaussian gradient + residual-Jacobian assembly, convention pin, native→sampling transform, FD-check — C — Gradient plumbing: Gaussian gradient + residual-Jacobian assembly, convention pin, native→sampling transform, FD-check #449 — closes the common case; unblocks Gradient-based local optimizers (D2D-style: TRF/Levenberg-Marquardt + L-BFGS-B) in the async Algorithm loop #386

Coverage layers (independent; each adds one FD-oracle case; not yet opened)

• D — Estimated-σ gradient term (−r²/σ³ + 1/σ, σ not in tensor)
• E — Scale-transformed objective (LOG10/lognormal)
• F — Trajectory-transform Jacobians (normalization ADR-0053, cumulative→incident ADR-0051)
• G — Asymmetric noise families (laplace / negative-binomial / student-t)
• H — SBML + Antimony backend variants (bngsim_sbml_model.py:618)
• I — Constraint & comparison-difference gradients (.con/.prop)
• J — Pre-equilibration / steady-state sensitivity continuity — [#385 layer J] Gradient through pre-equilibration / steady-state (blocked: bngsim sensitivities) #457 (bngsim Pass parameters directly to postprocessing #210 resolved in 0.11.7)

Backward compatibility

Purely additive and opt-in. The scalar (non-gradient) execution path is byte-unchanged when the
gradient path is inactive. Metaheuristic fits are unaffected.

---

Original issue text (pre-epic, superseded by the children above)

The original draft framed item 4 around the classic __REF surrogate-base split (ADR-0027);
that framing is superseded by the edition-2 bind-by-id + perturbation-operator mapping described
above and implemented in #448. The backend re-scoping ("What BNGsim ships today") is confirmed
against bngsim 0.10.14.