Skip to content

AGENTS.md

Latest commit

 

History

History
156 lines (95 loc) · 4.5 KB

AGENTS.md

File metadata and controls

156 lines (95 loc) · 4.5 KB
Raw

Examples E2E (Playwright) — Notes for Agents & Future Devs

This folder contains an end-to-end (e2e) smoke test harness for the repository’s examples/.

The goals are:

  • Provide a consistent way to smoke-test examples locally and in CI.
  • Support multiple example “shapes” (Next-only vs hybrid examples that also have an agent).
  • Keep tests lightweight and stable (avoid flakiness, avoid requiring real API keys).

How the harness works

Selecting which example to run

This suite intentionally runs one example at a time.

  • The active example is selected via the EXAMPLE environment variable.
  • If EXAMPLE is not set, it defaults to form-filling.

The Playwright config (playwright.config.ts) uses EXAMPLE to:

  • Set the webServer.cwd to the chosen example directory (examples/v1/${EXAMPLE}).
  • Choose the webServer.command used to start the app.

Why each spec has const EXAMPLE = process.env.EXAMPLE ?? "form-filling";

Each spec file is gated so that when you run the suite for one example:

  • The matching spec runs.
  • All other specs skip.

This makes it easy to run a CI matrix (one job per example) while keeping all tests in one folder.

Example types

Next.js-only examples

These can be started with:

  • pnpm dev

Hybrid examples (UI + agent)

Some examples have a Python agent that can be run alongside the UI.

For e2e smoke tests we typically only need the UI to boot, so the Playwright config treats these as “hybrid”:

  • travel
  • research-canvas

For hybrid examples the webServer.command is:

  • pnpm dev:ui

This avoids starting the Python agent during UI-only smoke tests.

Local setup

Install Playwright harness deps

From examples/e2e:

  • pnpm install
  • pnpm exec playwright install --with-deps chromium

Install the example’s deps

Each example has its own package.json.

Install deps in the example directory you want to test, e.g.:

  • cd examples/v1/travel && pnpm install

Notes:

  • If an example has a postinstall that requires non-Node tooling (e.g. Python uv), you may want to run pnpm install --ignore-scripts for CI-like behavior.

Run a single example

From examples/e2e:

  • EXAMPLE=form-filling pnpm test
  • EXAMPLE=travel pnpm test
  • EXAMPLE=research-canvas pnpm test
  • EXAMPLE=chat-with-your-data pnpm test
  • EXAMPLE=state-machine pnpm test

When EXAMPLE is set, you should see 1 passed and the other example specs skipped.

Test layout

  • Tests live under tests/v1.x/.
  • Each example gets a single smoke spec (minimal assertions).

Examples:

  • tests/v1.x/form-filling.spec.ts
  • tests/v1.x/travel.spec.ts

Writing smoke tests (guidelines)

Keep smoke tests:

  • Stable: prefer getByRole selectors and obvious headings/buttons.
  • Cheap: do not rely on LLM outputs.
  • Non-invasive: avoid sending chat messages or triggering expensive background work.

Patterns used:

  • Gate the spec:
    • test.skip(EXAMPLE !== "<example>", ...)
  • Prefer:
    • await expect(page).toHaveTitle(/.../)
    • await expect(page.getByRole("heading", { name: "..." })).toBeVisible()

If an example auto-opens Copilot UI / triggers calls, prefer adding a query param to disable it (e.g. travel uses /?copilotOpen=false).

CI (GitHub Actions)

Workflow:

  • .github/workflows/test_e2e-legacy-v1.yml

It runs a matrix of:

  • form-filling
  • travel
  • research-canvas
  • chat-with-your-data
  • state-machine

Key CI behaviors:

  • Installs examples/e2e deps and Playwright Chromium.
  • Installs the selected example’s deps.
  • Uses pnpm install --frozen-lockfile for deterministic installs.
  • For research-canvas, installs with --ignore-scripts to avoid requiring Python tooling just to run UI smoke tests.

Artifacts:

  • Always uploads Playwright output (test-results and playwright-report) for debugging.

Common issues / debugging

  • "next: command not found": the selected example’s node_modules are missing; run pnpm install in that example directory.
  • "module not found" for a transitive dep (e.g. shiki): add it explicitly to the example’s dependencies and reinstall.
  • Next.js dev warnings about cross-origin (allowedDevOrigins): currently treated as warnings; tests can still pass.

Adding a new example

  1. Ensure the example can be started via pnpm dev (Next-only) or pnpm dev:ui (hybrid).
  2. Add a new spec under tests/v1.x/<example>.spec.ts.
  3. Run locally:
    • EXAMPLE=<example> pnpm test
  4. Add the example name to the CI matrix in:
    • .github/workflows/test_e2e-legacy-v1.yml