Skip to content

TESTING.md

Latest commit

 

History

History
45 lines (33 loc) · 7.46 KB

TESTING.md

File metadata and controls

45 lines (33 loc) · 7.46 KB
Raw

Test-Gating Matrix

This matrix documents which CI workflows fire on which triggers, what they test, and whether they gate merges. Companion to QA-COVERAGE.md -- that document tracks per-demo coverage; this one tracks per-workflow gating.

Scope: all testing-related workflows (unit, integration, e2e, smoke) across the monorepo. Data read directly from .github/workflows/*.yml on the current branch.

Matrix

Workflow file Name (CI UI) Trigger Path filter Required? What it tests
.github/workflows/test_unit.yml test / unit push (main), pull_request (main), workflow_dispatch paths-ignore: docs/**, README.md, examples/** No Vitest unit suite across Node 20/22/24 for all TS packages
.github/workflows/test_unit-python-sdk.yml test / unit / python-sdk push (main), pull_request (main) sdk-python/**, this workflow No pytest against sdk-python/ under Python 3.12 + Poetry
.github/workflows/test_integration-runtime.yml test / integration / runtime push (main), pull_request (main), workflow_dispatch packages/runtime/**, this workflow No Runtime server integration tests (Node, possibly others)
.github/workflows/test_integration-docs.yml test / integration / docs push (main), pull_request docs/** No Extracts code blocks from docs, runs them against aimock (model-name + doc-test)
.github/workflows/test_e2e-dojo.yml test / e2e / dojo push (main), pull_request (main), workflow_dispatch packages/**, sdk-python/**, this workflow, .changeset No ag-ui dojo end-to-end matrix on Depot runners
.github/workflows/test_e2e-legacy-v1.yml test / e2e / legacy-v1 push (main), pull_request (main), workflow_dispatch examples/**, this workflow, .changeset No Legacy v1.x examples (form-filling, travel, research-canvas, chat-with-your-data, state-machine)
.github/workflows/showcase_validate.yml Showcase: Validate push (main), pull_request showcase/**, examples/integrations/**/fixtures/**, scripts/doc-tests/fixtures/** No Build-pipeline Vitest + manifest/registry validation + shell build
.github/workflows/test_e2e-showcase-on-demand.yml test / e2e / showcase / on-demand issue_comment (/test-aimock), workflow_dispatch n/a (comment-gated) No aimock-backed Playwright E2E on demand per-package
.github/workflows/test_smoke-starter.yml test / smoke / starter schedule (0 */6 * * *), workflow_run (publish / release), pull_request, workflow_dispatch examples/integrations/**, this workflow No Docker-compose smoke for 12 starter integrations (build + curl)
.github/workflows/test_smoke-starter-deployed.yml test / smoke / starter-deployed schedule (0 */6 * * *), workflow_run (Showcase: Build & Deploy), workflow_dispatch n/a (scheduled / post-deploy) No Playwright E2E against live deployed starter URLs (@starter-health/-agent/-chat)

Which tests run on a typical PR?

  • packages/** (runtime/SDK): test / unit, test / integration, test / e2e / dojo, static / quality, static / check binaries, plus static / danger if packages/sdk-js/src/langgraph.ts is touched.
  • sdk-python/**: test / unit / python-sdk, test / e2e / dojo, static / check binaries, plus static / danger if copilotkit/langgraph_agent.py is touched. test / unit also fires (paths-ignore does not exclude sdk-python).
  • showcase/**: Showcase: Validate, test / unit (paths-ignore does not exclude showcase), static / quality, static / check binaries. No Playwright E2E runs automatically -- comment /test-aimock on the PR to trigger test / e2e / showcase / on-demand.
  • examples/** (legacy v1.x): test / e2e / legacy-v1, static / check binaries. test / unit is excluded via paths-ignore.
  • examples/integrations/** (starters): test / smoke / starter (Docker), Showcase: Validate (for fixtures only), static / check binaries.
  • docs/**: test / integration / docs only. test / unit and static / quality are excluded via paths-ignore.
  • .github/workflows/**: each workflow that lists its own path in its trigger runs (most do). No single "workflows changed" catch-all.

Required status checks

None of the workflows above are enforced as required status checks. The active PROTECT_OUR_MAIN ruleset on main requires zero status contexts -- merges are gated only by review approval, not by CI outcome.

The legacy classic branch protection required_status_checks.contexts array contains stale entries (e.g. test / unit, Showcase: Validate) that appear in GitHub's API responses but are not evaluated by the active ruleset. If you see a CI workflow marked as "required" in an older doc or script, treat that as ghost data: nothing in GitHub's current enforcement path consumes it.

Practical consequence: a red CI run does not block merge. Reviewers must eyeball gh pr checks before approving.

Footnotes

  • test / unit matrix is Node 20/22/24; the other workflows pin a single Node version each (22 most common).
  • test / e2e / dojo uses Depot runners (depot-ubuntu-24.04); all others use standard GitHub runners.
  • test / smoke / starter and test / smoke / starter-deployed both run every 6h; the former validates Docker-build integrity of examples/integrations/, the latter validates the deployed Railway services.
  • workflow_run triggers fire after another workflow completes -- they do not gate the triggering PR, they run post-merge.