fix(showcase/harness): harden browser-pool against PID-ceiling crash-recovery wedge#5185
Merged
Merged
Conversation
… and bounded serve retry Make the long-lived chromium pool survive a pthread/PID-ceiling (`pthread_create: Resource temporarily unavailable`, errno 11) thread- exhaustion storm instead of draining to an empty, permanently-wedged set. - Crash-recovery relaunch backpressure: a transient EAGAIN on relaunch is retried with bounded linear backoff before the entry is evicted, so a thread-exhaustion window that relaxes within seconds recovers in place rather than splicing the entry out of the set. - Self-heal + degraded/recovered alarm: when the set empties mid-life the pool fires an `onDegraded` red alarm (previously only emitted on init() failure — mid-life death was silent) and kicks a background self-heal loop that relaunches a fresh set the moment a launch succeeds, firing `onRecovered`. No manual redeploy required. - Bounded serveNextWaiter transient re-drive: a persistently-transient newContext() on a still-connected browser no longer hot-loops the event loop; it self-reschedules up to a ceiling then leaves the waiter queued for a later release/recovery handoff (mirrors acquire()'s retry-once semantics). - Accounting hardening: generation-token guard on in-flight opens across a recycle, clamped servedContexts rollback on orphan-close, deferred-recycle re-check on non-release teardown paths, and waiter-drain on orphan-by- recycle rollback so freed capacity is served immediately. - orchestrator wires the pool's onDegraded/onRecovered hooks to the shared `system:browser-pool-degraded` red/green capacity-loss signal. Fixes the 2026-06-03 staging incident: the browser pool died from thread exhaustion, the relaunch storm emptied the set, and the harness wedged with no alarm -> 626 D0-red cells until a manual redeploy.
…mium pool Lift the soft nproc limit to the hard ceiling (`ulimit -u $(ulimit -Hu)`) before exec'ing the orchestrator so the legitimate 40-context chromium workload (several hundred OS threads at steady state) has ample thread headroom instead of running near the default ~1024 soft ceiling, where `chromium.launch()` tripped `pthread_create: Resource temporarily unavailable`. `exec` keeps node as PID 1 for correct signal handling; the `|| true` fallback keeps boot resilient when the runtime forbids raising the soft limit (the cgroup pids limit then remains the dominant control).
… and serve-retry paths - STAGING-OUTAGE regressions: degraded alarm fires (not silent) when the set empties from a relaunch storm; self-heal re-inits a fresh set once the kernel relaxes; a waiter queued during the dead window is served by self-heal; a transient relaunch EAGAIN is retried and the entry survives (no eviction, no alarm). - Bounded serveNextWaiter transient re-drive + FIX#7 dead-vs-alive gate propagation to the serve path. - orchestrator: degraded/recovered signal wiring covered. - BUG3 (orphan-by-recycle waiter drain) adapted to the crash-recovery acquire path: an acquire whose in-flight open is orphaned re-enqueues as a waiter; with cap=1 the freed slot goes to the other waiter, so the orphaned acquire settles via its own (now bounded) timeout. The invariant it verifies (freed capacity immediately serves the queued waiter) is unchanged.
jpr5
requested review from
marthakelly,
mme,
ranst91 and
tylerslaton
as code owners
June 3, 2026 17:14
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
Contributor
📣 Social Copy GeneratorGenerate social media copies (Twitter/X, LinkedIn, Blog Post) for this PR using Claude.
|
Contributor
Production Digest-Pinning AuditAll 27 services digest-pinned. Run 2026-06-03 10:14:37 PDT — 0 finding(s). |
5 tasks
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
…race A browser that is mid-`launch()` is not yet in `this.browsers` and has no disconnect handler attached, so it is invisible to every teardown path (shutdown's close pass, recycle, the disconnect handler). Under the self-heal/relaunch storm introduced by the browser-pool hardening cluster (#5174/#5185), a concurrent teardown therefore raced an in-flight launch: the launch escaped teardown accounting entirely — either leaked, or (in production) the browser was closed underneath the still-resolving `chromium.launch()`, which then rejected with `browserType.launch: Target page, context or browser has been closed` (SIGTRAP). The self-heal loop relaunched forever, each relaunch hitting the same race (336 self-heal-launch-failed events in 4 min), wedging the pool. Make launches atomic w.r.t. teardown: `launchBrowser()` registers its launch promise in a new `pendingLaunches` set BEFORE awaiting `rawLaunchBrowser()` and removes it on settle. `shutdown()` drains `pendingLaunches` (alongside `inFlightRecycles`) before its close pass, so it WAITS for every in-flight launch to settle instead of closing the browser mid-launch. The launch seam re-checks `isShutdown` the instant the launch settles: if a shutdown intervened it closes the freshly-launched browser cleanly THEN, exactly once, and throws a shutdown sentinel so the caller (init / recycle relaunch / self-heal) does not register a browser into a pool that is going away. The launch-stagger gate still chains off the raw result, so serialization semantics are unchanged. All #5185 behaviors (backoff, self-heal, degraded/recovered alarms, waiter draining) preserved. Adds a red-green test (FIX#13) reproducing an in-flight launch racing a concurrent shutdown: pre-fix the launching browser escapes teardown (closeCount 0, leaked); post-fix it is drained + closed cleanly exactly once with no close-during-launch rejection.
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
… bash The runtime CMD raised the soft nproc limit via `/bin/sh -c "ulimit -u $(ulimit -Hu) ..."`, but on node:22-bookworm-slim `/bin/sh` is dash, whose builtin `ulimit` does NOT support the `-u` (max-user-processes) flag — it errors `ulimit: Illegal option -u`, which the `2>/dev/null || true` then silently swallows. So #5185's intended PID protection never applied; the soft limit stayed at the inherited default. Run the CMD under `/bin/bash` (present at /usr/bin/bash) instead, whose `ulimit -u` works. Verified against the exact base image under `--ulimit nproc=512:4096`: bash raises the soft limit 512 -> 4096 (the hard ceiling) while dash leaves it untouched and errors. Minimal change — only the interpreter; the command, exec-as-PID-1, and `|| true` fallback are unchanged.
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
…race A browser that is mid-`launch()` is not yet in `this.browsers` and has no disconnect handler attached, so it is invisible to every teardown path (shutdown's close pass, recycle, the disconnect handler). Under the self-heal/relaunch storm introduced by the browser-pool hardening cluster (#5174/#5185), a concurrent teardown therefore raced an in-flight launch: the launch escaped teardown accounting entirely — either leaked, or (in production) the browser was closed underneath the still-resolving `chromium.launch()`, which then rejected with `browserType.launch: Target page, context or browser has been closed` (SIGTRAP). The self-heal loop relaunched forever, each relaunch hitting the same race (336 self-heal-launch-failed events in 4 min), wedging the pool. Make launches atomic w.r.t. teardown: `launchBrowser()` registers its launch promise in a new `pendingLaunches` set BEFORE awaiting `rawLaunchBrowser()` and removes it on settle. `shutdown()` drains `pendingLaunches` (alongside `inFlightRecycles`) before its close pass, so it WAITS for every in-flight launch to settle instead of closing the browser mid-launch. The launch seam re-checks `isShutdown` the instant the launch settles: if a shutdown intervened it closes the freshly-launched browser cleanly THEN, exactly once, and throws a shutdown sentinel so the caller (init / recycle relaunch / self-heal) does not register a browser into a pool that is going away. The launch-stagger gate still chains off the raw result, so serialization semantics are unchanged. All #5185 behaviors (backoff, self-heal, degraded/recovered alarms, waiter draining) preserved. Adds a red-green test (FIX#13) reproducing an in-flight launch racing a concurrent shutdown: pre-fix the launching browser escapes teardown (closeCount 0, leaked); post-fix it is drained + closed cleanly exactly once with no close-during-launch rejection.
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
… bash The runtime CMD raised the soft nproc limit via `/bin/sh -c "ulimit -u $(ulimit -Hu) ..."`, but on node:22-bookworm-slim `/bin/sh` is dash, whose builtin `ulimit` does NOT support the `-u` (max-user-processes) flag — it errors `ulimit: Illegal option -u`, which the `2>/dev/null || true` then silently swallows. So #5185's intended PID protection never applied; the soft limit stayed at the inherited default. Run the CMD under `/bin/bash` (present at /usr/bin/bash) instead, whose `ulimit -u` works. Verified against the exact base image under `--ulimit nproc=512:4096`: bash raises the soft limit 512 -> 4096 (the hard ceiling) while dash leaves it untouched and errors. Minimal change — only the interpreter; the command, exec-as-PID-1, and `|| true` fallback are unchanged.
6 tasks
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
A serveNextWaiter()/openContextOn() that shifted a waiter + reserved a slot BEFORE shutdown could settle its newContext() AFTER shutdown()'s close-pass. shutdown() drains inFlightRecycles + pendingLaunches, but a fire-and-forget serve open is tracked by neither, so the freshly-opened context landed in contextToBrowser/liveContexts on a torn-down pool — a leaked context that is never closed. openContextOn's orphan guard checked generation/recycling/isConnected but NOT isShutdown. Fix: add an isShutdown term to openContextOn's post-await orphan guard (treat shutdown like a recycle — close the just-opened context, roll back the reservation, throw); bail serveNextWaiter before openContextOn when isShutdown; and reject (not re-queue) a shifted waiter / straddling acquire when the open throws under shutdown, so it cannot strand on the already-cleared waiter queue. Preserves all #5185/#5221 behaviors. Adds red-green coverage for the leak (RACE1) and a regression guard for the acquire transient-retry -> concurrent-recycle -> orphan-guard straddle (RACE2), which was verified to keep its reservation accounting balanced via the existing generation guard (no overshoot).
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
A serveNextWaiter()/openContextOn() that shifted a waiter + reserved a slot BEFORE shutdown could settle its newContext() AFTER shutdown()'s close-pass. shutdown() drains inFlightRecycles + pendingLaunches, but a fire-and-forget serve open is tracked by neither, so the freshly-opened context landed in contextToBrowser/liveContexts on a torn-down pool — a leaked context that is never closed. openContextOn's orphan guard checked generation/recycling/isConnected but NOT isShutdown. Fix: add an isShutdown term to openContextOn's post-await orphan guard (treat shutdown like a recycle — close the just-opened context, roll back the reservation, throw); bail serveNextWaiter before openContextOn when isShutdown; and reject (not re-queue) a shifted waiter / straddling acquire when the open throws under shutdown, so it cannot strand on the already-cleared waiter queue. Preserves all #5185/#5221 behaviors. Adds red-green coverage for the leak (RACE1) and a regression guard for the acquire transient-retry -> concurrent-recycle -> orphan-guard straddle (RACE2), which was verified to keep its reservation accounting balanced via the existing generation guard (no overshoot).
9 tasks
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
…profile-dir purge after consecutive launch failures (stop recurring wedge) The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive). Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
jpr5
added a commit
that referenced
this pull request
Jun 4, 2026
…profile-dir purge after consecutive launch failures (stop recurring wedge) The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive). Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the showcase harness's long-lived chromium browser-pool against the OS thread / PID-ceiling exhaustion failure mode that took staging down. The pool now survives a
pthread_create: Resource temporarily unavailable(errno 11) storm, self-heals when the kernel relaxes, alarms loudly on mid-life capacity loss, and is given ample thread headroom in the container.The incident this fixes
On 2026-06-03 the harness container hit the thread/PID ceiling: the long-lived chromium pool (
MAX_CONTEXTS=40, several hundred OS threads at steady state) trippedpthread_create: EAGAINonchromium.launch()/ mid-life context opens. Every relaunch attempt re-threw, so the crash-recovery path spliced each browser out of the set until it emptied. After thatpickLeastLoaded()returned undefined forever, everyacquire()enqueued a waiter that hit the 30s timeout, and the pool sat permanently dead with no alarm (degraded=red only fired oninit()failure, never on mid-life set death) — 626 D0-red dashboard cells until a manual redeploy.The hardening
onDegradedred alarm and kicks a background self-heal loop that relaunches a fresh set the moment a launch succeeds, then firesonRecovered. No manual redeploy. The orchestrator wires these hooks to the sharedsystem:browser-pool-degradedred/green signal.serveNextWaitertransient re-drive — a persistently-transientnewContext()on a still-connected browser no longer hot-loops the event loop; it self-reschedules up to a ceiling, then leaves the waiter queued for a later release/recovery handoff.nproclimit to the hard ceiling (ulimit -u $(ulimit -Hu)) before exec'ing the orchestrator, giving the legitimate 40-context workload thread headroom instead of running near the default ~1024 soft ceiling.MAX_CONTEXTSstays 40 — the constraint was THREADS, not memory.servedContextsrollback on orphan-close, deferred-recycle re-check on non-release teardown paths, and waiter-drain on orphan-by-recycle rollback.This builds cleanly on top of the recently-landed #5174 browser-pool teardown-concurrency hardening (BUCKET-D); both invariant sets are preserved.
Test plan
tsc --noEmit(harness) clean on top of currentmainsystem:browser-pool-degradedred/green signal toggles on an induced set-empty (post-merge)🤖 Generated with Claude Code