MCPJVM-382: Execution Orchestrator resiliency for long-running regression workflows

[nimbly-dev]

Description

Enterprise validation from #381 shows that watchers and external verification are directionally correct, but long-running regression workflows are still operationally blocked by Execution Orchestrator resiliency gaps.

The main issue is no longer whether the product can express downstream convergence checks. The main issue is whether execution_orchestration can keep long-running async plans moving, observable, and classifiable across tool-call timeout boundaries.

Examples observed during enterprise testing:

• long-running search-consumer workflow remained in_progress while live tool calls timed out at the 300-second boundary
• persisted suite progress could lag behind active execution
• watcher/external verification could be functionally correct while the outer orchestration lifecycle still felt stalled or opaque
• operators lacked a first-class bounded outer resume/poll model for long workflows

This Epic establishes Execution Orchestrator resiliency as a first-class capability for long-running regression workflows.

Problem Statement

watchers and external verification can prove downstream completion, but the current orchestration model is too fragile for enterprise async workflows when one live MCP/tool call is expected to stay open until business completion. The control-plane needs a bounded outer resiliency model that can:

• execute a bounded orchestration slice
• persist progress frequently enough to preserve visibility
• sleep and resume across tool-call boundaries
• continue the same in-progress suite run from the current plan rather than rerunning already-completed plans
• stop deterministically on success, fail-closed, timeout, or bounded outer poll exhaustion

Intended Capability Direction

The expected direction is an outer orchestration resiliency loop owned by project-level defaults rather than plan-level configuration.

Conceptually:

1. execute
2. persist suite/run progress
3. inspect current persisted state
4. if still in_progress, sleep for a bounded interval
5. resume the same suiteRunId from the current nextPlanOrder
6. stop on terminal state or bounded outer poll exhaustion

This is distinct from watcher-level polling:

• watcher polling verifies one business condition inside a plan
• orchestrator resiliency polling keeps the whole suite progressing across tool/time boundaries

This Epic is about continuing the same long-running suite run. It is not about an operator-driven “rerun this plan N times” capability, and it should not re-execute already-completed plans when resuming a waiting plan.

Project-Level Defaults Direction

This capability should be configured from project defaults, not from individual plans.

Illustrative direction only:

{
  "workspaces": [
    {
      "projectRoot": "C:/repo/app",
      "defaults": {
        "requestTimeoutMs": 30000,
        "retryMax": 1,
        "watchers": {
          "pollIntervalMs": 5000,
          "maxAttempts": 12,
          "maxTotalWaitMs": 60000
        },
        "orchestrator": {
          "resumePollMax": 30,
          "resumePollIntervalMs": 10000,
          "resumePollTimeoutMs": 300000
        }
      }
    }
  ]
}

The exact schema can still be refined, but the policy should remain project-owned rather than plan-authored.

Scope

In scope:

• establish Execution Orchestrator resiliency as a first-class capability for long-running regression workflows
• support bounded outer resume/poll behavior across tool-call timeout boundaries
• persist suite progress often enough to distinguish active work from stalled execution
• define project-level defaults for orchestrator resiliency behavior
• preserve deterministic terminal states and fail-closed semantics
• improve operator visibility for in_progress, resumed, timed out, and terminal runs
• allow child stories for implementation of required defaults, resume/poll behavior, progress persistence, terminal classification, and SKILL/docs updates

Out of scope:

• replacing watcher-level bounded waiting semantics
• replacing external verification contract/provider work
• mandatory SQLite state-store adoption as a prerequisite
• operator-driven repeated rerun of already-completed plans
• Performance Suite integration
• moving orchestration into a mandatory always-on backend service

Acceptance Criteria

• Execution Orchestrator resiliency is recognized as a first-class capability separate from watcher/provider semantics
• The capability supports bounded outer resume/poll behavior for long-running workflows
• Project-level defaults are the source of orchestrator resiliency configuration
• Resume behavior continues the same suiteRunId from the current in-progress plan rather than rerunning completed plans
• Persisted suite progress becomes sufficient to distinguish active work from stalled execution during long runs
• Terminal outcomes remain deterministic across resumed or multi-pass execution
• The Epic is decomposed into child stories for implementation of configuration, progress persistence, resume/poll behavior, terminal classification, and operational visibility

Notes from #381

This Epic is directly motivated by the enterprise blockers documented in #381, especially:

• Regression Orchestrator tool call times out before long plan completion
• Persisted progress can lag behind active long-running execution
• Orchestrator lacks an outer bounded retry/resume model for long plan workflows

watchers and external verification remain necessary, but they are not sufficient on their own without a resilient outer orchestration model.