Phase 14: Run required targets and one VS Code stress validation

[jununfly]

This was generated by AI during triage.

Parent

Parent tracking issue: #154

What to build

Run the canonical Phase 14 formal experiment for the required decision targets, then run at least one VS Code stress validation after the confirmed trigger condition is met. This is the HITL validation slice because it requires maintainer review of generated evidence and follow-up judgment on blockers exposed by the stress target.

Required decision targets are zcodegraph and excalidraw. VS Code is a stress target / optional extended evidence: stress failures do not block required-target decision readiness, but they must be captured and turned into follow-up issues when they expose new blockers.

Acceptance criteria

• Canonical formal experiment is run for zcodegraph and excalidraw.
• Required-target raw artifact is produced.
• Required-target decision summary draft is produced.
• The generated decision summary draft is reviewed by a maintainer.
• If required targets meet the confirmed trigger condition, at least one VS Code stress validation is run.
• VS Code stress raw artifact is produced after trigger conditions are met.
• Stress target failures do not block required-target decision readiness.
• New blockers exposed by stress validation are captured as follow-up issues.

Blocked by

• Phase 14: Generate decision summary draft and companion docs #160

[jununfly]

HITL review update: do not close this issue yet.

Required-only evidence has been produced and reviewed enough to identify the next blockers:

• Required targets were run with the temporary required-only manifest (zcodegraph + excalidraw), without changing the canonical manifest.
• Raw artifact: docs/benchmarks/2026-06-15-rust-indexing-core-phase-14-required-only-rerun2.raw.json
• Decision summary draft: docs/benchmarks/2026-06-15-rust-indexing-core-phase-14-required-only-rerun2-decision-summary-draft.md
• Rust core binary is now available and both required targets completed both arms.
• Both required targets have sufficiency=passed.
• Both required targets still have performance=unavailable because peak RSS is not collected and Rust wall time did not satisfy the configured improvement gate.
• Current corrected classification is failed-required-performance-gate-unmet, not failed-required-arm-unavailable.

Current required-only result:

• zcodegraph: target-failed-performance-gate-unmet
• excalidraw: target-failed-performance-gate-unmet
• experiment: failed-required-performance-gate-unmet

Decision: keep #161 open. The required evidence exists, but the trigger condition for claiming required-target decision readiness is not met, so VS Code stress validation should stay deferred until the performance gate blocker is resolved or the trigger condition is explicitly redefined.

Follow-up issues created:

• Phase 14 follow-up: Collect peak RSS for performance gate evidence #162 Collect peak RSS for performance gate evidence
• Phase 14 follow-up: Investigate Rust indexing wall-time regression #163 Investigate Rust indexing wall-time regression
• Phase 14 follow-up: Add TS/Rust graphStats parity breakdown #164 Add TS/Rust graphStats parity breakdown

This issue should be revisited after those blockers are resolved or after the maintainer explicitly changes the required-target trigger condition.

[jununfly]

HITL diagnosis update after rerun5.

I kept #161 open and added one more instrumentation slice instead of claiming rollout readiness.

What changed in this pass:

• zcodegraph index --engine rust now preserves the Rust core profile and the TypeScript-side finalizeRustIndex() profile into a CLI-emitted profile artifact when ZCODEGRAPH_INDEX_PROFILE_OUT is set.
• scripts/rust-indexing-experiment.mjs now captures that profile into arms.rust.execution.indexProfile and renders it in the decision summary under ## Rust index profile breakdown.
• Added regression coverage in:
  • __tests__/rust-indexing-experiment.test.ts
  • __tests__/rust-index-engine-cli.test.ts
• While validating, the fake JS Rust-core test fixture exposed a Windows spawn EFTYPE path. I fixed Rust core discovery so shebang Node scripts are executed via process.execPath while real binaries still run directly. This preserves packaged/source binary behavior and keeps the test seam portable.

Validation:

• npm run build passed.
• npx vitest run __tests__/rust-indexing-experiment.test.ts __tests__/rust-index-engine-cli.test.ts passed: 32 passed, 1 skipped.

New required-only evidence:

• Raw artifact: docs/benchmarks/2026-06-16-rust-indexing-core-phase-14-required-only-rerun5.raw.json
• Summary draft: docs/benchmarks/2026-06-16-rust-indexing-core-phase-14-required-only-rerun5-decision-summary-draft.md
• Command used --fail-on-required-gate-failure; exit code remained 2, as expected for failed-required-performance-gate-unmet.

Rerun5 result:

• Experiment classification: failed-required-performance-gate-unmet
• Required sufficiency: passed for both zcodegraph and excalidraw
• Required performance: still unavailable/unmet
• Rollout readiness: not claimed

Performance/profile details:

zcodegraph:

• TypeScript total: 9818 ms
• Rust total: 11072 ms
• wallTimeDeltaPct: +12.77%
• peakRssDeltaPct: +4.18%
• Rust index phase profile:
  • sourceScanMs: 1
  • parseExtractionMs: 1381
  • sqliteWriteMs: 2622
  • subprocessStartupHandoffMs: 26
  • frameworkPostExtractMs: 9
  • referenceResolutionMs: 1194
  • dynamicDispatchSynthesisMs: 329
  • dbMaintenanceMs: 17
  • typescriptFinalizationMs: 1551

excalidraw:

• TypeScript total: 25672 ms
• Rust total: 26408 ms
• wallTimeDeltaPct: +2.87%
• peakRssDeltaPct: -9.97%
• Rust index phase profile:
  • sourceScanMs: 5
  • parseExtractionMs: 2283
  • sqliteWriteMs: 4184
  • subprocessStartupHandoffMs: 26
  • frameworkPostExtractMs: 15
  • referenceResolutionMs: 5754
  • dynamicDispatchSynthesisMs: 444
  • dbMaintenanceMs: 30
  • typescriptFinalizationMs: 6246

Diagnosis:

• The remaining Phase 14: Run required targets and one VS Code stress validation #161 blocker is not arm availability and not sufficiency; both required arms complete and produce graph stats.
• The largest remaining measured regression is still the overall Rust CLI index phase.
• The new profile shows the largest Rust-side costs are SQLite writes plus TypeScript-side finalization/reference resolution, especially on excalidraw.
• Because the Rust arm also extracts materially more graph work (many more constants/components/contains/calls/export edges), the current Phase 14 performance gate is still not satisfied and should not be treated as a rollout-ready pass.

Current recommendation:

• Keep Phase 14: Run required targets and one VS Code stress validation #161 open.
• Do not trigger VS Code stress validation yet unless the maintainer explicitly redefines the trigger condition.
• Next closeout path should either optimize/adjust the measured Rust index path or explicitly revise the Phase 14 performance-gate semantics for the "Rust extracts more graph work" case before claiming required-target readiness.

[jununfly]

This was generated by AI during triage.

Phase 15 follow-up plan created

#161 remains open. The rerun5 evidence still classifies the required-only validation as failed-required-performance-gate-unmet, so VS Code stress validation remains deferred unless the trigger condition is explicitly redefined by the maintainer.

Created the follow-up tracking plan for controlled performance gate interpretation:

• Parent: Post-PRD optimization tracker for Rust indexing performance #165
• Phase 15A: Phase 15A: Add graph work profiles and manifest plumbing #166
• Phase 15B: Phase 15B: Run matched-work required-only experiment #167
• Phase 15C: Phase 15C: Interpret gate and decide #161 trigger condition #168

The plan preserves the current A/B structure and raw PRD gate, while adding Rust-side Graph Work Profile control so matched-work evidence can distinguish implementation performance from extra graph work.

[jununfly]

This was generated by AI during triage.

Phase 15B matched-work evidence is now available for #161 review.

Evidence artifacts:

• Raw artifact: docs/benchmarks/2026-06-16-rust-indexing-core-phase-15-required-only-matched-work-rerun1.raw.json
• Decision summary draft: docs/benchmarks/2026-06-16-rust-indexing-core-phase-15-required-only-matched-work-rerun1-decision-summary-draft.md
• Tracking issue: Phase 15B: Run matched-work required-only experiment #167

What changed compared with rerun5:

• The original full-profile rerun5 remains the raw PRD gate evidence and is still classified as failed-required-performance-gate-unmet.
• Phase 15B reran required targets only (zcodegraph, excalidraw) with Rust using manifest-controlled rust.graphWorkProfile = "matched-ts-js".
• Both TypeScript and Rust arms completed for both required targets.
• Sufficiency passed for both required targets.
• The generated classification remains failed-required-performance-gate-unmet because this evidence does not redefine the original raw PRD gate.
• Rollout readiness is not claimed.

Matched-work wall-time evidence:

Target	TypeScript ms	Rust ms	Wall-time delta
zcodegraph	9259	7875	-14.95%
excalidraw	23716	18502	-21.99%

Interpretation for #161:

The raw PRD trigger condition is still unmet unless the maintainer explicitly redefines how matched-work controlled-variable evidence should interact with the gate. The matched-work result is materially better evidence for the causal question: it supports that previous raw A/B performance comparison was distorted by Rust doing different graph work. Final gate interpretation and the #161 next path should be handled in #168.

[jununfly]

This was generated by AI during triage.

Phase 15C HITL gate interpretation is complete in #168.

Decision summary:

• Raw full-profile PRD gate remains unmet and remains the default/full-profile rollout gate.
• The Phase 14: Run required targets and one VS Code stress validation #161 next-step trigger is redefined as a dual-track gate:
  • raw full-profile gate remains unmet;
  • matched-work controlled-variable gate is sufficient to trigger VS Code stress validation.
• Matched-work required-only evidence passed sufficiency and showed Rust faster under matched-ts-js:
  • zcodegraph: TypeScript 9259 ms, Rust 7875 ms (-14.95%)
  • excalidraw: TypeScript 23716 ms, Rust 18502 ms (-21.99%)
• GraphStats deltas are interpretation evidence, not automatic blockers, while sufficiency passes and no blocker-level regression is observed.
• No default/full-profile rollout readiness is claimed.

Follow-up created:

• Phase 15D: Run VS Code matched-work stress validation #169

#169 will run VS Code stress validation under the matched-ts-js profile only. Full-profile VS Code stress remains deferred.

#161 should remain open until #169 produces stress evidence and that evidence is interpreted.