Gradient optimizers: finish #386 deliverable 6 — convergence-vs-baseline test + becker smoke test

[wshlavacek]

Summary

#386 deliverable 6 is "Tests: each optimizer recovers known parameters on a small FD-validated model; convergence vs a metaheuristic baseline on the same model; a smoke test mirroring examples/becker_d2d_gradient/." The recovery portion is done; the other two parts remain. This issue tracks them.

Done (for reference):

• Parameter-recovery tests through the real bngsim backend — tests/test_gradient_optimizer.py (rate/IC, estimated-σ, bound-active, multimodal multi-start).
• Offline scipy-oracle step-math parity for the runners — tests/test_gradient_runner.py.

Part A — convergence vs a metaheuristic baseline

A test that, on the same model, compares a gradient fit (trf / lbfgs) against a metaheuristic baseline (de) and asserts the gradient method does at least as well — the head-to-head #386 asked for, which the truth-recovery tests don't cover.

• The recovery harness already exposes all four algorithms, so this can be built on it directly: tests/recovery_harness.py _ALGORITHMS = {'de', 'am', 'trf', 'lbfgs'}, with build() / drive() running any of them on one config.
• Design choice to settle (the fuzzy part): what "convergence vs baseline" asserts. Candidates: (a) the gradient fit reaches an objective ≤ the metaheuristic's on the same model/budget; (b) it reaches the known optimum in far fewer objective evaluations (the real selling point of a gradient method — count success_count / trajectory length); (c) both. Recommend (b)+(a): same small model, both recover the truth, and the gradient fit uses materially fewer evaluations. Keep it deterministic (fixed random_seed) so it isn't flaky.

Part B — becker smoke test

A smoke test mirroring examples/becker_d2d_gradient/ (the D2D EpoR methodological reference: multi-start → TRF) driven through PyBNF's real scheduler (fit_type = trf, now with multi-start), not the standalone BNGsim-direct path the reference notebooks use.

• The reference currently lives only on the unmerged branch origin/examples/becker-d2d-gradient (an Antimony model BIOMD0000000271.ant, becker_d2d_data.npz, a README, and two notebooks — all standalone, no scheduler).
• A scheduler-driven smoke test needs at least the model + data fixtures on main (under tests/), exercising the SBML/Antimony gradient path end-to-end (the measurement-model layer + SBML forward sensitivities the assembly already supports). It does not require merging the notebooks.
• Scope is a smoke test: it runs the fit to completion without error and lands at a sane objective — not a tight parameter-recovery assertion on the full EpoR model (which would be slow/flaky). A trimmed model or a short iteration budget is fine.

Optional, separate from the test deliverable

Decide whether to merge the examples/becker-d2d-gradient branch as the on-main standalone reference example (the notebooks + data + README #386 points at). This has independent docs/example value but is not required to close deliverable 6 — the smoke test can ship with just its own fixtures. Track/merge it on its own merits.

Done criteria

• A convergence-vs-baseline test (Part A) on a small model, deterministic, asserting the chosen metric (recommend: gradient fit recovers the truth in materially fewer evaluations than de, and to an at-least-as-good objective).
• A becker (or becker-derived) smoke test (Part B) driving fit_type = trf multi-start through the scheduler to completion, with the needed model/data fixtures landed under tests/.
• Gradient-based local optimizers (D2D-style: TRF/Levenberg-Marquardt + L-BFGS-B) in the async Algorithm loop #386 deliverable 6 checkbox flipped once both land.
• (Optional, separate) becker reference example merged to main, or explicitly deferred.

References

• Parent: Gradient-based local optimizers (D2D-style: TRF/Levenberg-Marquardt + L-BFGS-B) in the async Algorithm loop #386, deliverable 6 (the recovery portion is done; these are the remaining two parts).
• Harness to build on: tests/recovery_harness.py (_ALGORITHMS, build/drive).
• Reference impl: examples/becker_d2d_gradient/ on branch origin/examples/becker-d2d-gradient.

[wshlavacek]

Both parts landed in b3dc654 (on main).

Part A — convergence vs a metaheuristic baseline. test_gradient_recovers_in_far_fewer_evaluations_than_de (parametrized over trf / lbfgs) in tests/test_gradient_optimizer.py, built on the recovery harness. On the same decay model + data, a gradient fit and a de baseline both recover the truth, but the gradient fit reaches an at-least-as-good objective in an order of magnitude fewer objective evaluations — recommendation (b)+(a). Deterministic (fixed random_seed + the synchronous fake client), so it is a fixed inequality, not a flaky race. Measured: trf 7 evals, lbfgs 18, vs de 480 (assert grad_score <= de_score and 10 * grad_evals < de_evals).

Part B — becker smoke test. test_trf_multistart_smoke_through_the_scheduler_on_becker_epor. fit_type = trf multi-start drives the Becker EpoR model to completion through PyBNF's real scheduler, fitting kon/koff to a zero-noise synthetic Epo_EpoR time course generated from the model's own published nominal parameters. This exercises the Antimony → SBML forward-sensitivity gradient path the small .bngl recovery fixtures don't: the model is carried verbatim, bngsim carries output sensitivities through the ODE solve, and the assembled residual Jacobian drives the async trust-region loop. Smoke scope — every start runs to termination at a sane objective, not a tight EpoR recovery. Needs no BNG2.pl (no network to expand); gated on bngsim_antimony + output_sensitivities.

Fixtures landed under tests/: tests/sbml_files/becker_epor.ant (BioModels BIOMD0000000271, vendored verbatim from the examples/becker_d2d_gradient reference) + tests/sbml_files/SOURCE.md (provenance). Data is generated at runtime from the model (the recovery-harness oracle pattern), so the fixture is self-contained and drift-free.

#386 deliverable 6 checkbox flipped (and deliverable 5, the discrete-events gate, which had also landed in 9021e38/#461 but was still showing partial).

Optional follow-up — deferred. Merging the examples/becker-d2d-gradient reference branch (the notebooks + D2D data + README) to main is left as a separate, optional docs/example task as the issue notes; the smoke test ships with its own fixtures and does not depend on it.

Closing.

[wshlavacek]

Follow-up (7154df0): the Part B smoke test now also exercises the measurement-model layer the issue calls for, not just a bare species. The first version scored a bare model species (Epo_EpoR), which sidesteps the observation layer — the arithmetic observableFormula path needs the petab math extra, which was simply not installed in my local venv (it is a declared [tests] extra, so CI has it). Installing it and using it:

• Added tests/sbml_files/becker_epor.xml (the SBML form — a formula-observable's namespace is built from parse_sbml; a .ant model routes through parse_bngl at config.py:1874 and exposes no species namespace, so the measurement-layer case needs the .xml).
• test_trf_multistart_smoke_on_becker_epor_sbml_measurement_layer: trf multi-start scoring Epo_cells = Epo_EpoRi + dEpoi (the D2D internalised-Epo observable, ADR-0036) through the scheduler — the kon/koff forward sensitivities flow through the formula's two-species chain rule into the residual Jacobian. Gated pytest.importorskip('petab') + bngsim_sbml + output_sensitivities; recovers the truth exactly.
• The bare-species .ant test is kept (renamed ..._antimony_species) as the complementary dependency-free path.
• recovery_harness.make_newera_config gained an observables={id: formula} kwarg.