fix(showcase): harden staging→prod promote reliability (snapshot skip + honest verify-prod signal)#5322
Merged
Merged
Conversation
…omote snapshot build_snapshot enumerates every project service and queries each one's serviceInstance. The only guard was `next if inst.nil?` — it handled a NULL result but not a THROWN `GraphQL: ServiceInstance not found` error (a half-deleted service that still appears in the project service list but has no instance in the env). That error bubbled to Railway.run's top-level `rescue GraphQL::Error` and aborted the ENTIRE promote with an opaque exit 2 before any preflight/divergence logic ran (run 27144525566 killed the docs promote this way). Scope the rescue narrowly to ONLY the per-service "ServiceInstance not found" message — log+skip that one service exactly like the nil case — so every other GraphQL failure (auth, rate-limit, schema drift) still propagates fail-loud. Adds red-green coverage: a single thrown not-found is skipped (healthy services still snapshot), while an unrelated GraphQL error still raises.
… never probed When a promote fails, the succeeded-service set is empty, so verify-prod hits its skip branch (`exit 0`). The GitHub job result is therefore `success`, and the notify step rendered `verify-prod=success` in the #oss-alerts Slack message — a misleading green, since prod was never probed. verify-prod now exports a `status` output: `success` after a real probe passes, `skipped` on the empty-CSV skip. notify reads that output (via the new bats-tested verify-prod-display.sh) instead of the raw job result, so the Slack line accurately reads `verify-prod=skipped` vs `success` vs `failure`. A genuine probe failure / contract violation exits non-zero (job result `failure`, status never written), and the display falls back to the job result. Slack formatting is unchanged. Extracts the display mapping into showcase/scripts/verify-prod-display.sh (mirroring promote-fleet.sh) with red-green bats coverage, and adds it to the showcase_validate.yml shellcheck step.
jpr5
requested review from
marthakelly,
mme,
ranst91 and
tylerslaton
as code owners
June 8, 2026 17:08
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📣 Social Copy GeneratorGenerate social media copies (Twitter/X, LinkedIn, Blog Post) for this PR using Claude.
|
Contributor
Production Digest-Pinning AuditAll 28 services digest-pinned. Run 2026-06-08 10:09:06 PDT — 0 finding(s). |
Contributor
|
Size Change: 0 B Total Size: 1 MB ℹ️ View Unchanged
|
7 tasks
jpr5
added a commit
that referenced
this pull request
Jun 8, 2026
A single-service `promote <svc>` ran check_service_set_parity over the FULL staging vs FULL prod fleet and REFUSEd whenever staging carried any service prod lacks. The live staging fleet legitimately contains 13 staging-only services — harness-workers (SSOT-modeled) and 12 starter-* demos — so an otherwise-clean single-service promote (e.g. docs) is blocked with `REFUSE: services in staging not in prod`. Scope the "staging not in prod" REFUSE to the promote TARGET when a single-service promote is in effect (intersect the staging-only set with [target]). The target-absent-from-prod footgun still REFUSEs (target is in the intersection), the "prod not in staging" arm is unchanged, and full-fleet promotes (no --service) retain full strictness. Complements #5322.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two robustness/signal defects in the showcase staging→prod promote, both surfaced while investigating today's
docspromote failure (run 27144525566). Shared theme: harden the promote so one odd service can't abort the whole run, and so the alert never reports a green it didn't earn.Fix #1 — a single thrown
ServiceInstance not foundno longer aborts the whole promoteshowcase/bin/railway·PromoteCommand/build_snapshotbuild_snapshotenumerates every project service and queries each one'sserviceInstance. The only guard wasnext if inst.nil?— it handled a null result but not a thrownGraphQL: ServiceInstance not founderror (a half-deleted service that still appears in the project service list but has no instance in that env). That error bubbled toRailway.run's top-levelrescue GraphQL::Errorand aborted the entire promote with an opaque exit 2 before any preflight/divergence logic ran. In run 27144525566 a single odd/transient service threw this and killed thedocspromote.ServiceInstance not foundfor an individual service exactly like thenilcase: log + skip that one service, keep going.Fix #2 —
verify-prodreportsskipped(not a misleadingsuccess) when it skipped.github/workflows/showcase_promote.yml·verify-prod+notifyWhen the promote fails, the succeeded-service set is empty, so
verify-prodhits its skip branch andexit 0. The GitHub job result is thereforesuccess, and the#oss-alertsSlack line renderedverify-prod=successeven though prod was never probed — a misleading green.verify-prodnow exports astatusjob output:successafter a real probe passes,skippedon the empty-CSV skip.notifyreads that output (not the raw job result) so the Slack line accurately readsverify-prod=skippedvssuccessvsfailure. A real probe failure / contract violation exits non-zero (job resultfailure, status never written) and the display falls back to the job result.Tests (red-green)
showcase/bin/spec/test_snapshot_graphql.rb): a single thrownServiceInstance not foundis skipped (healthy services still snapshot); an unrelated GraphQL error still raises. Verified red before the fix, green after.test_snapshot_ivar_lint.rbline-pins renumbered to match.showcase/scripts/verify-prod-display.sh(mirroringpromote-fleet.sh) so it is unit-testable;showcase/scripts/__tests__/verify-prod-display.batscovers success / skipped / failure / cancelled / fallback. Added the new script to theshowcase_validate.ymlshellcheck step.Test plan
ruby showcase/bin/spec/all_tests.rb— 129 runs, 0 failuresbats showcase/scripts/__tests__/— 18 ok (incl. 6 new)shellcheckon both promote scripts — cleanactionlint(+shellcheck) onshowcase_promote.yml— cleanzizmor --min-severity low(CI config) on changed workflows — clean