envoy-browser rig: screenshots degrade under Brave memory/process accumulation (root-caused, not 'flaky')

[amiller]

Operator: 'flaky' is not acceptable — root-cause and fix it. DONE root-causing; here's the full diagnosis so this is a mechanical fix, not a re-investigation.

Root cause (empirically established, NOT random)

The envoy-browser container runs a single long-lived Brave that ACCUMULATES memory + processes across navigations, and screenshot capture degrades as it does:

• fresh (post-restart): ~1.2 GB / 15 procs → screenshot OK
• after ~5 navigations: ~2.7 GB / 24 procs → screenshot timeout
• saturated: ~10 GB / 19+ procs (hits the container's 10 GB cap) → 'Failed to capture tab: image readback failed', then Brave goes FATAL

Proven: a single screenshot on fresh Brave succeeds with the current flags; the SAME capture fails once accumulated. So it is resource accumulation, not flag-level breakage and not randomness. The 24 procs for ~5 sites strongly suggests the bridge opens a NEW TAB per navigate and never closes them (tab/renderer leak) rather than reusing one target.

The blocker (why a quick fix didn't land)

Brave's launch flags live in /etc/neko/supervisord/brave.conf which is read-only in the running container (baked into the image; sed rename = 'Device or resource busy', cat-redirect = 'Read-only file system'). So --renderer-process-limit cannot be added to the live container — it needs an image rebuild.

Fixes, most-durable first

1. Image rebuild (envoy-browser / browser-box image): add --renderer-process-limit=4; keep --disable-gpu; TEST whether --disable-software-rasterizer should be dropped (paired with --disable-gpu it leaves no rasterizer, though fresh capture works — memory is the primary driver, but verify).
2. Bridge tab hygiene: make navigate REUSE a single target (Page.navigate on the same tab) or close prior tabs, so procs don't accumulate. This is likely the real leak (24 procs / 5 sites).
3. Operational mitigation ALREADY APPLIED: zed:~/paseo-batch/journeys/apps-evidence.sh now restarts Brave fresh at the start of each walk. It helps but a single walk still accumulates enough to fail late captures — insufficient alone; item 1 or 2 is the fix.
4. Architectural answer: RFC 0028 browser pool (RFC 0028: Browser Runtime & Render Pool #65/RFC 0028: browser render pool (isolated/fair/reset leases) #68) — ephemeral, reset, memory-capped per-lease browsers — is the real long-term fix; this rig is the exact case for it.

Acceptance

Three consecutive scheduled /journeys walks each produce 5 real PASS (screenshots present, not OK-NOSHOT/FAIL), with Brave RSS staying under ~3 GB across a full walk. Screenshot the /journeys apps section showing 5 LIVE ✓ with real captures.

Operator steps

Needs write access to the envoy-browser image source (browser-box/envoy repo) to rebuild + redeploy the rig container.

[amiller]

Blocked (verified) — the durable fix is operator-run (image rebuild); it cannot land as a tee-daemon PR. Posting root cause + a correction so the rebuild is one-shot.

Why this loop can't ship it

• tee-daemon has none of the affected code. Exhaustive search (ws-bridge|capture_bridge|brave.conf|neko|--renderer-process-limit|extension navigate|chrome.tabs.create) → 0 hits in this repo; the only prior rig-adjacent PR was RFC RFC 0028: browser render pool (isolated/fair/reset leases) #68 (docs). The fix surface is the envoy bridge + extension + image, none of which live here.
• ~/projects/envoy is not a git repo on this box (no .git/remote), and rebuilding+pushing the neko-envoy-browser image needs ghcr push creds that are absent here. Per box-inventory, that rebuild/redeploy is operator-run (matches the issue's own "Operator steps: Needs write access to the envoy-browser image source").
• A live hot-patch of the running container would "work" but is non-durable (lost on recreate) → a stopgap, which AGENTS.md forbids. So: no faked tee-daemon PR, honest block.

Correction to the issue's hypothesis (fix #2 is already done — don't spend a rebuild on it)

The issue suspects the bridge opens a new tab per navigate ("24 procs / 5 sites → tab/renderer leak"). It does not. In the actually-loaded extension (/usr/share/brave/extensions/envoy/background/service-worker.js, confirmed live), the navigate handler is:

case "navigate": return this.navigate(s, a[0])   // → async navigate(e,t){ await chrome.tabs.update(e,{url:t}) ... }

chrome.tabs.update on the active tab — reuses one target. tabs.create count in the navigate path = 0 (the one tabs.create is the unused createTab method). So tab reuse is already the behavior; fix #2 is a no-op. The 24 procs/5 sites is Brave site-isolation + BFCache + per-origin service-worker processes accumulating across 5 distinct origins in one tab, not a tab leak.

The real lever = fix #1 (launch flags) in the image rebuild — operator-run

Live snapshot just now (container up 20+ min, no flag applied): Brave = 4611 MB / ~42 procs — already past the 3 GB ceiling, with no --renderer-process-limit. Current flags in use: --disable-gpu, --disable-software-rasterizer, --disable-features=DisableLoadExtensionCommandLineSwitch. None of the process-limiting flags are present. The rebuild should add (in /etc/neko/supervisord/brave.conf, which is read-only in the live container — hence image rebuild): --renderer-process-limit=4 (+ consider --process-per-site, --disable-features=BackForwardCache to cut cross-origin process retention).

What I CAN do if useful (cheap, on request)

Hot-patch a candidate flag set onto the live rig, run 3× apps-evidence.sh, report RSS/proc curves to de-risk the rebuild — but I will not present that as the fix. Flagging blocked + removing ready so this tick doesn't re-pick it; operator to un-block when the envoy image source is reachable.

[amiller]

Operational fix LANDED — the rig produces real screenshots again. Root cause confirmed and narrowed: a long-lived Brave accumulates renderer processes (process-per-site) until captureVisibleTab fails; a brave-ONLY restart does NOT fix it (leaves the window unmanaged on X → screenshot times out). The fix that works: full container restart before each walk (docker restart → X + window-manager + Brave brought up in order), which stays 1.1GB across a full 5-app walk. Applied to zed:/paseo-batch/journeys/apps-evidence.sh preamble. Just proven: a full walk captured all 5 apps (real PNGs 38–280KB), /journeys now shows 5 LIVE ✓. The image-rebuild (--renderer-process-limit) + RFC 0028 pool remain the elegant durable fix, but the report now reflects reality reliably.