Skip to content

Setup-phase failures (adapter.setup / tutorial.setup) skip teardown → seeded data leaks #15

@jbrecht

Description

@jbrecht

Problem

runTeardown() only runs on a step failure or after all steps succeed. A throw during adapter.setup() or tutorial.setup() skips it entirely — the only thing the outer finally does is browser.close().

In dist/pipeline/record.js (0.9.0):

await adapter.setup(page, ctx);          // line ~90 — throw here → no teardown
if (tutorial.setup) {
  await tutorial.setup(page, ctx);       // line ~93 — throw here → no teardown
}
// ...
for (...) {
  try { ...step... }
  catch (cause) { await runTeardown(); ... }   // teardown ONLY on step failure
}
await runTeardown();                     // ...or after all steps succeed

This means ctx.onTeardown(...) registered inside a setup() is a no-op on the setup path — exactly the path most likely to seed data and then throw (a flaky signIn, a warm-up page.goto that times out).

Why it matters

Per-tutorial setup (#8) is designed to seed state, and tutorials run against a shared, persistent test DB. A setup-phase failure leaks whatever was seeded before the throw. Dogfooding the 0.9.0 upgrade on the umami tutorials, send-a-broadcast's setup seeds an event + ~15 people before the warm-up gotos — a warm-up timeout would orphan all of it. The old pre-#8 cleanup sweep (purgeDemoEvents) masked this; removing it per #8 exposed it.

Today the only safe pattern is hand-rolled try/catch in every setup hook:

async setup(page, ctx) {
  const seeded = await seedEvent(...)
  try { ...warm-up that can throw... }
  catch (err) { await seeded.teardown(); throw err }   // boilerplate every author must remember
}

Most authors won't remember, and the failure is silent (a slowly-filling shared DB).

Suggested direction

Run the teardown chain on the setup-failure path too — wrap adapter.setup + tutorial.setup so that a throw still runs runStepTeardowns(teardownThunks) (and ideally tutorial.teardown / adapter.teardown) before rethrowing. Then ctx.onTeardown registered in setup means what it looks like it means.

Follow-up to #8.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions