[Plugin] Self-improving harness engine (autoresearch ratchet) — agent-side

[Interstellar-code]

Summary

Build a self-improving harness engine for Hermes as a plugin in this (forked) repo. It applies the Karpathy autoresearch ratchet — propose one atomic change → run a fixed-budget eval → keep if the metric improves, git revert if not — to the profile/persona layer first (SOUL.md, system_prompt, USER.md, MEMORY.md), then later to skills/plugins. The plugin owns all the heavy machinery and exposes it at /api/improve/*; the Switch UI /improve page (tracked in a separate hermes-switchui issue) is a thin consumer of that API.

This is the agent half of the docs/self-improving-agent-proposal.md design. Switch-UI work is out of scope here.

Why a plugin in hermes-agent (not Switch UI)

The engine needs in-process access to the gateway, sessions, profile files under ~/.hermes/profiles/<id>/, git, and the scheduler — all agent-side. Per the proposal §6c the decision is: engine = forked-agent plugin (structured metrics, eval runner, meta-agent scheduler, git ratchet, API), UI = capability-probed page. Keeps it fork-only (never-upstream feature category).

Scope (agent-side deliverables)

1. Metrics store — SQLite at ~/.hermes/improve.db (per-profile history so trends survive restarts): sessions/day, error+warn rate, task completion, retries, tokens/cost, eval scores, experiment results (results.tsv-equivalent).
2. Eval runner — replays scenario suites through the gateway, records pass rate + tokens + latency. Consider wrapping promptfoo or porting skill-creator's eval format rather than writing scoring from scratch.
3. Scenario suites — 10–20 multi-turn tasks per profile, seeded from real logged sessions (sessions API). Scored by binary checks ("used tool X before answering?", "asked before destructive action?", "under N tokens?", "respected SOUL.md constraint Y?") plus a small LLM-as-judge rubric for tone/persona fidelity, judged by a different model than the proposer. Hold out ~30% of scenarios from the proposing agent (anti-gaming).
4. Meta-agent proposer + scheduler — reads metrics + failing scenarios, proposes one sentence-level diff to one file, runs the suite against the candidate, emits proposal (diff + before/after per-scenario scores + cost). Only queues if offline score improved (or tied with a shorter prompt — bloat guard).
5. Git ratchet — commit-per-experiment on ~/.hermes/profiles/<id>/, instant rollback, full experiment log.
6. Experiment state machine — proposed → approved → live (observing) → verified | reverted (+ rejected → logged). Two-stage verification: offline eval, then a live observation window (N real sessions or T days) comparing live metrics vs baseline. Verified → baseline vN+1 + next proposal auto-generated; regressed → auto git revert + notify + log.
7. API /api/improve/* — proposals list/get, approve/reject/edit-then-approve, history, per-profile baseline curve, pause/resume per profile, capability probe endpoint (so UI can hide when plugin absent, like jobs/kanban).

Treat the three profile files differently (core design rule)

File	Nature	Approach
SOUL.md / system_prompt	Instructions (behavior)	✅ Ratchet loop applies — the editable artifact
USER.md	Facts about the user	❌ Never ratchet against task metrics; only fact/staleness check, human-approved corrections
MEMORY.md / memory/*.md	Accumulated data	⚠️ Hygiene job (dedupe/consolidate/prune), metric = retrieval relevance + size, NOT task score

Safety rails (non-negotiable for persona layer)

1. One sentence-level change per experiment — never wholesale rewrites.
2. Human approval on every write, indefinitely for this layer (UI shows diff + per-scenario before/after + cost).
3. Git ratchet per experiment, instant rollback.
4. Run the full suite, not just failing scenarios (catch regressions elsewhere).
5. Drift anchor: identity-invariant checks that can never be traded for task score.
6. One experiment in flight per profile (attribution). Token budget cap per cycle.

Phases (agent-side)

• P0 — Per-profile metrics + store + /api/improve skeleton. Surface sessions/day, error/warn, cost, token efficiency per profile. (Blocked by Open Q1 — see below.)
• P1 — Scenario suites for top 1–2 profiles from real sessions + eval runner; manual runs via API.
• P2 — Gated ratchet loop on SOUL.md/system_prompt: proposer → suite → proposal in queue → approve→git commit / reject→log.
• P3 — Memory hygiene job + USER.md staleness checker (separate metrics).
• P4 — Extend runner to skills/plugins (mostly configuration once P0–P2 infra exists).

Open questions (blockers — likely gateway-side fixes)

1. Do gateway logs/sessions carry a profile id tag so metrics can be per-profile? If not, that's the first gateway-side change.
2. Are skill invocations logged with enough structure (skill ↔ session ↔ outcome) to correlate errors per skill? (Needed for P4.)
3. Which model proposes vs judges? (Must differ.)
4. Token budget per experiment cycle.
5. Metrics store location — plugin-local ~/.hermes/improve.db (preferred, fork-isolated).

Out of scope (separate issues)

• Switch UI /improve page → hermes-switchui repo issue (companion to this one).
• Any upstream-able gateway change discovered via Open Q1/Q2 → its own issue if it must touch core (not plugin).

Reference

Full design: docs/self-improving-agent-proposal.md (§6b profile-first variant, §6c experiment lifecycle spec). Prior art to wrap not rebuild: Anthropic skill-creator evals, promptfoo, DSPy/GEPA. Risk literature: PostTrainBench (reward hacking), agent-drift, Darwin Gödel Machine.

[Interstellar-code]

Companion UI issue (Switch UI /improve page, thin consumer of this plugin's /api/improve/*): Interstellar-code/hermes-switchui#206