Skip to content

RUNBOOK.md

Latest commit

 

History

History
193 lines (125 loc) · 9.48 KB

RUNBOOK.md

File metadata and controls

193 lines (125 loc) · 9.48 KB
Raw

Showcase D5 (d6-all-pills) Runbook

Operational documentation for debugging and fixing showcase D5 probe failures locally. Intended audience: engineers and AI agents working on showcase integrations.

CLI Rules

ALWAYS use bin/showcase for all operations. Never raw docker compose or docker build.

bin/showcase up <slug>          # start container
bin/showcase rebuild <slug>     # code changes (new image)
bin/showcase test <slug> --d5   # run D5 probe
  • rebuild handles symlink dereferencing that raw docker build cannot (tools/ and shared-tools/ are symlinks to ../../shared/).
  • recreate for env/config changes (same image, new container).
  • rebuild for code changes (new image).

Fixture Matching

D5 fixtures use hasToolResult (not turnIndex) for multi-turn disambiguation.

Field Meaning
hasToolResult: false First LLM call -- no tool result in messages yet
hasToolResult: true Follow-up call -- tool result present

turnIndex counts assistant messages, which varies across frameworks. Do not use turnIndex in new fixtures.

The mcp-subagents supervisor chain should also avoid turnIndex; chain each follow-up on the prior tool call's toolCallId so repeated and interleaved pill clicks replay the same way.

Fixture locations

  • Source fixtures: showcase/harness/fixtures/d5/*.json -- edit these, then rebuild bundle.
  • Bundle: showcase/aimock/d5-all.json -- aggregate of all source fixtures. Rebuild after any source edit.

Aimock debug logging

Add --log-level debug to the aimock command in docker-compose.local.yml to see fixture match/miss per request.

Integration Patterns

These are canonical. Do not deviate.

HITL (hitl-steps, hitl-approve-deny, hitl-text-input)

Backend agent has tools=[] (no backend tools). Frontend registers tools via useHumanInTheLoop or useFrontendTool. CopilotKit injects frontend tool definitions into the LLM call. Every HITL integration follows this pattern without exception.

gen-ui-custom

langgraph-python uses the chart pattern (useComponent with render_pie_chart). All other integrations should also demonstrate meaningful custom generative UI. Do not replace charts/data-viz with trivial text-only components just to pass tests.

tool-rendering

Frontend registers useRenderTool for get_weather. The v2 API uses parameters (not args) in the render callback. Backend has the actual tool.

shared-state

Backend calls set_notes tool, must forward tool result back to LLM for the follow-up text response. Frameworks that don't auto-cycle (crewai, langroid) need explicit tool-execution loops.

Docker Compose Environment

All providers must be routed through aimock. Required env vars in x-integration-defaults:

OPENAI_API_KEY / OPENAI_BASE_URL          -> http://aimock:4010/v1
ANTHROPIC_API_KEY / ANTHROPIC_BASE_URL    -> http://aimock:4010
GOOGLE_API_KEY / GOOGLE_GEMINI_BASE_URL   -> http://aimock:4010
SPRING_AI_OPENAI_BASE_URL                 -> http://aimock:4010

Missing any of these means that provider's integrations bypass aimock and hit real APIs (or fail with an empty key).

Debugging Sequence

  1. Check container health: docker compose -f showcase/docker-compose.local.yml ps
  2. Check container logs: docker logs showcase-<slug> --tail 30
  3. Enable aimock debug: add --log-level debug to compose command
  4. Run test: bin/showcase test <slug> --d5 --verbose
  5. Check aimock logs for fixture matching: docker logs showcase-aimock 2>&1 | grep "Fixture matched\|No fixture"
  6. If fixture matches but test fails: frontend/runtime issue (check component rendering, testid attributes)
  7. If fixture does not match: check hasToolResult, userMessage substring matching
  8. If zero aimock requests: check base URL env var for that provider

Production Debugging

Harness probe logging

The harness emits structured logs at INFO level for probe lifecycle events:

  • probe.tick-start / probe.tick-complete — probe run lifecycle
  • probe.target-start / probe.target-complete — per-service results
  • probe.d6-all-pills.service-start / probe.d6-all-pills.service-complete — D5 per-service
  • probe.d6-all-pills.feature-complete — per-feature pass/fail with error details
  • probe.run-summary — single line with all service results
  • probe.d6-all-pills.pool-abort-release — browser pool starvation events

View with: RAILWAY_PROJECT_ID=6f8c6bff-a80d-4f8f-b78d-50b32bcf4479 railway logs --service showcase-harness --tail 200

Debug-level probe logging

For detailed conversation-runner traces (selector resolution, DOM text extraction, settle polling, per-turn lifecycle), set LOG_LEVEL=debug on the showcase-harness Railway service. This enables console.debug(...) output from the conversation runner and D5 scripts.

To enable temporarily: set the env var in Railway dashboard → showcase-harness → Variables → LOG_LEVEL=debug. The service auto-restarts. Remember to unset after debugging — debug output is verbose.

Triggering probes manually

curl -sf -X POST "https://showcase-harness-production.up.railway.app/api/probes/probe:d6-all-pills-e2e/trigger" \
  -H "Authorization: Bearer $OPS_TRIGGER_TOKEN" \
  -H "Content-Type: application/json"

Retrieve OPS_TRIGGER_TOKEN: RAILWAY_PROJECT_ID=6f8c6bff-a80d-4f8f-b78d-50b32bcf4479 railway variables --service showcase-harness --json | python3 -c "import json,sys; print(json.load(sys.stdin)['OPS_TRIGGER_TOKEN'])"

Rate limit: 5 minutes per probe ID.

Testing package.json changes

When package.json changes (new deps, version bumps), volume mounts don't cover node_modules. You MUST rebuild the Docker image: bin/showcase rebuild <slug>, then re-test. A passing bin/showcase test against a volume-mounted container does NOT validate the build.

Isolated Test Runs (--isolate)

Use --isolate when another agent or terminal session is already running showcase locally. It prevents port collisions and container name conflicts by scoping everything into a temporary overlay.

bin/showcase test <slug> --d5 --isolate

Operational notes

  • Required when sharing a machine: If any other session has showcase up running, use --isolate to avoid stomping on its containers and ports.

  • Parallel runs supported: Up to 46 concurrent --isolate runs. Each gets a unique port range (slot 0 = +200, slot 1 = +400, ...) and a scoped docker compose project name.

  • Originals are never modified: The flag writes temp copies of docker-compose.local.yml and shared/local-ports.json to $TMPDIR/showcase-isolate-$$/. If the process crashes, originals are untouched.

  • Cleanup is automatic: The temp directory and slot are released on exit (via trap EXIT). If a run was killed with SIGKILL, clean up manually:

    # Remove orphaned containers from a specific isolated run:
docker compose --project-name <name> down

# Clear all stale slot reservations:
rm -rf /tmp/showcase-isolate-slots/*

  • Stale .iso-bak files: If you see docker-compose.local.yml.iso-bak or local-ports.json.iso-bak, those are leftovers from the old (pre-PR#4570) isolate behavior. The new code auto-restores them on startup, but you can also clean up manually with git checkout on the affected files.

Anti-Patterns

Earned by bugs. Do not repeat.

  • NEVER change a demo's fundamental functionality to pass a test. The demo IS the point.
  • NEVER replace chart/data-viz gen-ui with trivial text components.
  • NEVER use turnIndex in new fixtures. Use hasToolResult.
  • NEVER use raw docker build. Symlinks break. Use bin/showcase rebuild.
  • NEVER assume "agent says done" means "D5 is green." Always run the actual test.
  • NEVER add a backend tool for something that should be a frontend HITL tool.

Aimock Fixture Deployment

When adding or modifying fixture files in showcase/aimock/, the showcase-aimock image must be rebuilt so production picks up the changes. CI handles this automatically -- any push to main that touches showcase/aimock/** triggers the Build & Deploy workflow to rebuild and redeploy the image.

For manual iteration (e.g. testing a fixture change before merging), build and push directly:

docker build --platform linux/amd64 -f showcase/aimock/Dockerfile -t ghcr.io/copilotkit/showcase-aimock:latest showcase/aimock/ --push

After pushing, redeploy the Railway service so it pulls the new image (the CI workflow does this automatically via serviceInstanceRedeploy).

When adding a new fixture file, update both:

  1. showcase/docker-compose.local.yml -- add a volume mount for the new file
  2. showcase/aimock/Dockerfile -- add a COPY line for the new file

Dev Iteration Speed

Each integration service bind-mounts its host src/ directory into the container via the volumes entry in docker-compose.local.yml:

volumes:
  - ./integrations/<slug>/src:/app/src

This means source edits take effect on container restart without rebuilding the Docker image. The workflow becomes:

  1. Edit code under showcase/integrations/<slug>/src/
  2. Restart the container: bin/showcase restart <slug>
  3. Re-run the test: bin/showcase test <slug> --d5

Use bin/showcase rebuild <slug> only when you change dependencies (requirements.txt, package.json) or non-src files (Dockerfile, entrypoint). For pure src/ changes, restart is sufficient and much faster.