Skip to content

Add Playwright E2E tests to CI pipeline #296

@brianlan

Description

@brianlan

Summary

Wire npm run test:ui (Playwright E2E tests) into the existing pr-checks.yml CI workflow so security-critical browser tests run on every PR.

Background / Context

PR #292 hardens the GraphSandbox iframe with a restrictive CSP and sandbox="allow-scripts", and adds Playwright browser tests (frontend/tests/graph-sandbox-security.spec.ts) that verify:

  • Malicious DSL cannot make outbound network requests from within the iframe
  • Malicious DSL cannot access window.parent.document

These tests currently run only locally. If they are not executed in CI, regressions to the CSP or sandbox configuration will not be caught automatically. This is especially important before #293 removes the frontend semantic DSL validator, at which point the iframe CSP/sandbox becomes the sole client-side security boundary.

Problem

The pr-checks.yml workflow runs only vitest (npm test -- --run) for frontend tests and Docker health curls for smoke tests. No CI job executes Playwright specs. A future PR could accidentally weaken the CSP or sandbox attribute, and all existing CI checks would still pass.

Goal / Expected Behavior

Every PR triggers the Playwright E2E test suite in CI, and the workflow fails if any Playwright test fails.

Scope

  • Add a CI job that runs cd frontend && npm run test:ui in pr-checks.yml
  • Ensure the job has the same Node.js/npm setup as the existing frontend test job
  • The job should fail the workflow on Playwright test failure
  • Cache Playwright browser binaries between runs for reasonable performance

Out of Scope

  • Adding new Playwright tests (existing tests in frontend/tests/ are sufficient)
  • Modifying Playwright configuration files
  • Changing the existing vitest or Docker smoke jobs
  • Adding test:smoke:compose to CI (it requires Docker Compose, which is already covered separately)

Chosen Implementation Approach

Add a new job to .github/workflows/pr-checks.yml that:

  1. Checks out the repo
  2. Sets up Node.js (same version as existing frontend job)
  3. Installs frontend dependencies
  4. Installs Playwright browsers (npx playwright install --with-deps)
  5. Runs cd frontend && npm run test:ui

Implementation Plan

  1. Inspect .github/workflows/pr-checks.yml for existing job patterns
  2. Add a playwright-tests job modeled after the existing frontend-tests job
  3. Ensure the job uses working-directory: frontend
  4. Verify the job installs browsers before running tests

Relevant Files / Areas

  • .github/workflows/pr-checks.yml

Tests Required

  • The new CI job itself serves as the test; verify it passes on a PR after implementation

Manual Verification / Self-Check

  • Open a test PR and confirm the new Playwright job appears and passes
  • Confirm that breaking a CSP directive causes the Playwright security tests to fail and the CI job to fail

Reviewer Acceptance Checklist

  • New CI job runs npm run test:ui on every PR
  • Job fails when a Playwright test fails
  • Job caches browser installs for reasonable performance
  • No disruption to existing CI jobs

Dependencies

Follow-Up Work

None.

Definition of Done

  • pr-checks.yml includes a job that runs Playwright tests
  • The job passes on a PR that includes the current Playwright tests
  • The job fails when a Playwright test is intentionally broken

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions