Phase 5: Copilot live test suite

[tinkermonkey]

Create the live Copilot test file with five test cases that exercise CliProcess with backend: 'copilot' against the real Copilot CLI binary. Tests gate on binary availability and a dry-run authentication check. Copilot live tests are primarily for local validation — CI skips them gracefully when the binary is absent.

Requirements

• Create src/__tests__/live/copilot.live.test.ts
• beforeAll must: (1) check CliProcess('copilot').isAvailable(), (2) perform a minimal dry-run CliProcess('copilot').run() with a 30s timeout to verify authentication; if either check fails, skip the entire suite via describe.skipIf
• Use collectLive() and assertEventStreamStructure() from Phase 1 helpers
• Implement five test cases:
  1. Golden path: call assertEventStreamStructure() on collected events
  2. ReadyEvent session UUID: assert ReadyEvent.sessionId matches UUID format
  3. Text content emitted: assert at least one TextEvent with non-empty text
  4. Session resume: second run with sessionId from first run; assert ReadyEvent.sessionId matches
  5. AbortSignal mid-run: abort after ReadyEvent; assert final event is ErrorEvent { code: 'aborted' }
• No test must fail solely because the copilot binary is absent from the runner

Design Guidance

Implements the 'Copilot Live Test Suite' section of the architecture design. The authentication gating via dry-run invocation, the five test cases, and the 'No per-token cost' rationale are documented in the architecture's component design for copilot.live.test.ts.

Acceptance Criteria

• npm run test:live with no copilot binary in PATH shows Copilot live tests as skipped (not failed)
• Suite skips when copilot is present but not authenticated
• With a valid authenticated Copilot environment, all five tests pass
• Golden path test satisfies assertEventStreamStructure() (ReadyEvent, TextEvent, DoneEvent; no errors; seq monotonic; sessionIds match)
• AbortSignal test confirms final event is ErrorEvent { code: 'aborted' }
• Code is reviewed and approved

Dependencies

Phase 1, Phase 4

[tinkermonkey]

Implementation

---

The copilot live tests show as skipped (not failed) when the binary is absent, exactly as required.

---

Summary

Created src/__tests__/live/copilot.live.test.ts with:

Gating logic (module-level top-level await):

• Checks CliProcess('copilot').isAvailable() for binary presence
• If available, runs a dry-run collectLive with maxTimeout: 30 and marks authenticated only if no error events are emitted
• describe.skipIf(!isAvailable || !isAuthenticated) gates the entire suite

Five test cases:

1. Golden path — calls assertEventStreamStructure() which verifies ReadyEvent, TextEvent, DoneEvent presence, no errors, monotonic seq, and matching sessionIds
2. ReadyEvent session UUID — asserts sessionId matches UUID regex (/^[0-9a-f]{8}-...-[0-9a-f]{12}$/i)
3. Text content emitted — asserts at least one TextEvent with non-empty text
4. Session resume — uses sessionId from first run's ReadyEvent with isFirstMessage: false; asserts second run's ReadyEvent.sessionId matches
5. AbortSignal mid-run — aborts on ReadyEvent; asserts final event is ErrorEvent { code: 'aborted' }

Verified: All 92 unit tests pass; live suite shows 5 tests skipped (not failed) when copilot binary is absent.

---

Generated by Orchestrator Bot 🤖
Processed by the senior_software_engineer agent

[tinkermonkey]

🔄 Starting Review Cycle

Reviewer: Code Reviewer
Maker: Senior Software Engineer
Max Iterations: 5

The automated maker-checker review cycle is now starting. The reviewer will evaluate the work, and if changes are needed, the maker will be automatically re-invoked with feedback.

---

Review cycle initiated by Switchyard

[tinkermonkey]

Code Review

---

I've completed a thorough review of the copilot live test implementation. Here's my assessment:

---

Status

APPROVED

Issues Found

Critical (Must Fix)

None

High Priority (Should Fix)

None

Notes

The implementation is clean and well-structured. Here are the key observations from the review:

Correctness: All five required test cases are present and correctly implemented. TypeScript compiles cleanly with tsc --noEmit. The gating logic properly checks binary availability and performs a dry-run authentication check before enabling the suite.

Consistency with existing patterns: The copilot test closely mirrors claude.live.test.ts in structure — module-level top-level await for gating, describe.skipIf for conditional execution, shared helpers (collectLive, assertEventStreamStructure), consistent maxTimeout values, and identical AbortSignal test structure. The only meaningful difference is the auth gating approach, which is justified (see below).

Auth gating divergence from claude test is correct: The claude test uses getLiveCredentials() which checks env vars (ANTHROPIC_API_KEY / CLAUDE_CODE_OAUTH_TOKEN). The copilot test instead performs a dry-run collectLive call, which is the right approach because getLiveCredentials for non-claude backends just returns isAvailable as hasCredentials — it doesn't actually verify authentication. The dry-run approach matches the requirements and provides a real auth check.

Session resume uses ReadyEvent.sessionId: This differs from the claude test (which uses DoneEvent.sessionId), but is correct per the buildCopilotArgs documentation: "the UUID comes from the ReadyEvent.sessionId produced by the session/new response on the first message."

All imports are used, no dead code, no commented-out code, no naming violations.

Source Artefact Updates

• Issue body: The requirement states "beforeAll must: (1) check..." but the implementation uses module-level top-level await instead of beforeAll. This is the correct approach — describe.skipIf evaluates at module definition time, so the gating values must be available before describe runs. This matches the established pattern in claude.live.test.ts. Consider updating the issue body to say "module-level gating must" instead of "beforeAll must" to reflect the actual convention.

---

Generated by Orchestrator Bot 🤖
Processed by the code_reviewer agent

[tinkermonkey]

Review Cycle Complete

Status: APPROVED

Iterations: 1/5

Maker: Senior Software Engineer

Reviewer: Code Reviewer

Review approved after 1 iteration(s)

---

Automated review cycle by Switchyard

[tinkermonkey]

Starting Repair Cycle (Testing)

Agent: Senior Software Engineer

Test Types: compilation, unit, integration, ci

Max Iterations: 100

Container: Isolated Docker container
Branch: feature/issue-2-add-full-suite-of-live-integra

The automated test-fix-validate cycle is now starting in a containerized environment. Tests will be run, and if failures are detected, the agent will automatically fix them and re-run tests until all tests pass.

---

Repair cycle initiated by Switchyard

[tinkermonkey]

✅ Repair Cycle Complete

Container: repair-cycle-code-wrapper-8-f09bfe14

Exit Code: 0

Total Agent Calls: 4

Duration: 116.8s

• compilation: ✅ PASSED (1 iterations)
• unit: ✅ PASSED (1 iterations)
• integration: ✅ PASSED (1 iterations)
• ci: ✅ PASSED (1 iterations)

---

Repair cycle executed by Switchyard (containerized)