Phase 3: Claude live test suite

[tinkermonkey]

Create the live Claude Code test file with six test cases that exercise CliProcess against the real claude binary. Tests gate on binary availability and credential presence, assert on event stream structure only (never on LLM-generated text), and run serially.

Requirements

• Create src/__tests__/live/claude.live.test.ts
• Entire suite must be gated in beforeAll using describe.skipIf(!claudeAvailable || !hasCredentials) — no test should fail if prerequisites are absent
• Use collectLive() and assertEventStreamStructure() from Phase 1 helpers
• Implement six test cases:
  1. Golden path: trivial prompt ("respond with exactly the word hello"), call assertEventStreamStructure(), assert DoneEvent.usage.inputTokens > 0 and usage.outputTokens > 0
  2. ReadyEvent fields: assert ReadyEvent.sessionId is a non-empty string and ReadyEvent.model is truthy
  3. Session resume: two sequential runs; second run uses isFirstMessage: false with sessionId from first run's DoneEvent; assert second ReadyEvent.sessionId matches captured ID; assert no ErrorEvent in either run
  4. AbortSignal mid-run: prompt for a long response, abort after ReadyEvent is received; assert final event is ErrorEvent { code: 'aborted' }
  5. OAuth auth path: skip individually if CLAUDE_CODE_OAUTH_TOKEN not present; run with only OAuth token in subprocess env; assert no ErrorEvent
  6. API key auth path: skip individually if ANTHROPIC_API_KEY not present; run with only API key in subprocess env; assert no ErrorEvent
• Each test must set maxTimeout: 60 on ProcessOptions to prevent zombie processes
• Tests must never assert on specific generated text content

Design Guidance

Implements the 'Claude Live Test Suite' section of the architecture design. The six test cases, their prompt strings, assertion strategies, and gating patterns are fully specified in the architecture's component design table for claude.live.test.ts. The credential isolation approach (removing unused credential from subprocess env) is documented in the architecture's auth path coverage section.

Acceptance Criteria

• npm run test:live with no credentials set shows all Claude live tests as skipped (not failed)
• npm run test:live with valid ANTHROPIC_API_KEY runs all six tests and they pass
• Golden path test collects exactly one ReadyEvent, at least one TextEvent, and exactly one DoneEvent
• Session resume test confirms second run's ReadyEvent.sessionId matches first run's DoneEvent.sessionId
• AbortSignal test confirms final event has type === 'error' and code === 'aborted'
• OAuth test is individually skipped when CLAUDE_CODE_OAUTH_TOKEN is absent
• No test asserts on LLM-generated string content
• Code is reviewed and approved

Dependencies

Phase 1

[tinkermonkey]

Implementation

---

All 91 unit tests pass. Here's a summary of the work:

Summary

Created src/__tests__/live/claude.live.test.ts with six live integration tests for the claude backend.

Key design decisions

Suite gating: Used top-level await (valid in ESM) to call getLiveCredentials('claude') at module evaluation time, so describe.skipIf(!claudeAvailable || !hasCredentials) has the resolved values before the suite registers. This is the only correct approach given Vitest evaluates describe.skipIf synchronously.

Six test cases:

1. Golden path — calls assertEventStreamStructure(), asserts exactly one ReadyEvent/DoneEvent, and usage.inputTokens > 0 / usage.outputTokens > 0
2. ReadyEvent fields — asserts sessionId is a non-empty string and model is truthy
3. Session resume — two sequential runs; second uses isFirstMessage: false with sessionId from first DoneEvent; asserts second ReadyEvent.sessionId matches
4. AbortSignal mid-run — uses CliProcess directly (needed for mid-stream abort), aborts on ReadyEvent, asserts final event is ErrorEvent { code: 'aborted' }
5. OAuth auth path — it.skipIf(!process.env.CLAUDE_CODE_OAUTH_TOKEN), temporarily removes ANTHROPIC_API_KEY to isolate the OAuth credential path
6. API key auth path — it.skipIf(!process.env.ANTHROPIC_API_KEY), temporarily removes CLAUDE_CODE_OAUTH_TOKEN to isolate the API key path

All tests set maxTimeout: 60 and never assert on LLM-generated text content.

---

Generated by Orchestrator Bot 🤖
Processed by the senior_software_engineer agent

[tinkermonkey]

🔄 Starting Review Cycle

Reviewer: Code Reviewer
Maker: Senior Software Engineer
Max Iterations: 5

The automated maker-checker review cycle is now starting. The reviewer will evaluate the work, and if changes are needed, the maker will be automatically re-invoked with feedback.

---

Review cycle initiated by Switchyard

[tinkermonkey]

Code Review

---

Good. Now I have a complete picture. Here's my review:

---

Status

CHANGES NEEDED

Issues Found

Critical (Must Fix)

None

High Priority (Should Fix)

• Inline import() type instead of top-level import: On line 65 of claude.live.test.ts, the events array is typed with import('../../events/types.js').ClaudeEvent[] — an inline dynamic import type expression. Every other test file in the project imports ClaudeEvent via a top-level import type statement (see CliProcess.test.ts:7, recovery.test.ts:3, live-helpers.ts:3). This is inconsistent with the established codebase convention and hurts readability. ClaudeEvent should be added to the existing type import on line 4:

  import type { ReadyEvent, DoneEvent, ErrorEvent, ClaudeEvent } from '../../events/types.js';

  Then line 65 becomes:

  const events: ClaudeEvent[] = [];

• ReadyEvent fields test lacks structural guard: The ReadyEvent fields test (line 29) immediately does events.find(e => e.type === 'ready') as ReadyEvent and accesses .sessionId and .model without any prior check that a ReadyEvent exists in the stream. If the CLI fails to emit a ReadyEvent (e.g., network error, binary crash), find returns undefined, and the as ReadyEvent cast silently produces undefined — the test then throws TypeError: Cannot read properties of undefined (reading 'sessionId') instead of a clear assertion failure. The other tests either call assertEventStreamStructure(events) first (golden path, which validates presence) or have natural guards. Add either a call to assertEventStreamStructure(events) or an explicit expect(ready).toBeDefined() before accessing fields, consistent with how the golden path test works.

Source Artefact Updates

• Issue body: The requirement states "Entire suite must be gated in beforeAll using describe.skipIf(!claudeAvailable || !hasCredentials)". The phrase "in beforeAll" is contradictory — describe.skipIf evaluates synchronously at describe-registration time, before any beforeAll hooks run. The implementation correctly uses top-level await at module scope to resolve credential availability, then passes the resolved booleans to describe.skipIf. The requirement should be updated to: "Credential availability must be resolved before the suite registers (e.g., via top-level await), then used with describe.skipIf(!claudeAvailable || !hasCredentials) — no test should fail if prerequisites are absent."

---

Generated by Orchestrator Bot 🤖
Processed by the code_reviewer agent

[tinkermonkey]

Implementation

---

Revision Notes

• ✅ Inline import() type: Added ClaudeEvent to the existing top-level import type on line 4 and replaced the inline import('../../events/types.js').ClaudeEvent[] expression on line 65 with the clean ClaudeEvent[] form, consistent with every other test file in the project.
• ✅ ReadyEvent fields structural guard: Typed the ready variable as ReadyEvent | undefined, added expect(ready).toBeDefined() before accessing .sessionId and .model, and used non-null assertion (!) for the subsequent accesses — so a missing ReadyEvent now produces a clear Vitest assertion failure rather than a cryptic TypeError.

---

Generated by Orchestrator Bot 🤖
Processed by the senior_software_engineer agent

[tinkermonkey]

Code Review

---

Both fixes are clean and correct. Here's my review:

---

Status

APPROVED

Previous Issues Status

• ✅ Inline import() type instead of top-level import - RESOLVED: ClaudeEvent added to the top-level import type on line 4, and the inline import('../../events/types.js').ClaudeEvent[] on line 66 replaced with ClaudeEvent[].
• ✅ ReadyEvent fields test lacks structural guard - RESOLVED: ready is now typed as ReadyEvent | undefined, an expect(ready).toBeDefined() assertion precedes field access, and non-null assertions (!) are used for subsequent property accesses. A missing ReadyEvent now produces a clear Vitest failure instead of a cryptic TypeError.

New Issues Found

None

Source Artefact Updates

None

---

Generated by Orchestrator Bot 🤖
Processed by the code_reviewer agent

[tinkermonkey]

Review Cycle Complete

Status: APPROVED

Iterations: 2/5

Maker: Senior Software Engineer

Reviewer: Code Reviewer

Review approved after 2 iteration(s)

---

Automated review cycle by Switchyard

[tinkermonkey]

Starting Repair Cycle (Testing)

Agent: Senior Software Engineer

Test Types: compilation, unit, integration, ci

Max Iterations: 100

Container: Isolated Docker container
Branch: feature/issue-2-add-full-suite-of-live-integra

The automated test-fix-validate cycle is now starting in a containerized environment. Tests will be run, and if failures are detected, the agent will automatically fix them and re-run tests until all tests pass.

---

Repair cycle initiated by Switchyard

[tinkermonkey]

✅ Repair Cycle Complete

Container: repair-cycle-code-wrapper-6-cf065526

Exit Code: 0

Total Agent Calls: 4

Duration: 97.8s

• compilation: ✅ PASSED (1 iterations)
• unit: ✅ PASSED (1 iterations)
• integration: ✅ PASSED (1 iterations)
• ci: ✅ PASSED (1 iterations)

---

Repair cycle executed by Switchyard (containerized)