Skip to content

Commit d4e7595

Browse files
authored
chore(showcase): include failed job/step + error excerpt in validate Slack alert (CopilotKit#4068)
## Summary - Bare `:x: Showcase validate: failed | View run` forces a click-through to triage. Red alerts must carry triage-ready detail per the oss-alerts policy. - Adds a pre-notify step that extracts the failed step name and first meaningful error line, then interpolates both into the Slack payload. - Companion to CopilotKit#4065 (which quieted routine success noise); this PR makes the remaining failure-only posts actionable at a glance. ## Before / after **Before** ``` :x: Showcase validate: failed | View run ``` **After** (against tonight's run `24598654559`, the validate-parity regression) ``` :x: Showcase validate: failed — Run validate-parity (MUST checks gating): [FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no src/app/demos/hitl-in-chat/ directory | View run ``` ## Extraction approach New `Extract failure details for Slack` step (runs only on `failure() && push && env.SLACK_WEBHOOK != ''`): 1. **Resolve current job ID** via `GET /repos/{repo}/actions/runs/{run_id}/jobs`. Match by `name == "Validate Showcase"` (primary) with a fallback to the first job containing a failed step (rename-drift tolerance). 2. **First failed step name** pulled from the same jobs response. 3. **First error line** via `gh run view --log-failed --job=<id>`. Pipeline: - `awk -F'\t'` to split TSV columns (`job \t step \t timestamp + content`) and strip the leading ISO timestamp + optional BOM from the content column. - `sed` strips ANSI escape codes. - `grep -vE` skips runner header noise (`##[group]`, `shell:`, `env:`, `Run`, blank lines). - `grep -m1 -E` grabs the first line matching `[FAIL]`, `[ERROR]`, `Error:`, `error:`, or `::error`. - `head -c 300` truncates to keep the payload under the ~800-char Slack budget even after JSON escaping. 4. **Writes `failed_step` + `error_excerpt` to `$GITHUB_ENV`** using heredoc delimiters (safe for values containing `=` or newlines). 5. **Never blocks the notify step**: `set +e`, stderr suppressed on all extraction commands, explicit `exit 0`, and safe fallbacks (`"unknown step"` / `"see workflow run for details"`). The notify step's payload now uses `toJSON(format(...))` around all four interpolated values (repo, run_id, failed_step, error_excerpt) so any quotes/backslashes/newlines in extracted content are JSON-encoded defensively — matches the pattern from `showcase_drift-report.yml`. ## Validation - `actionlint .github/workflows/showcase_validate.yml` — no findings in the edited region (lines 360-470). The only reported finding is the pre-existing `depot-ubuntu-24.04-4` label info on line 47, unrelated. - Dry-run of the extraction pipeline against run `24598654559`: ``` gh run view 24598654559 --repo CopilotKit/CopilotKit --log-failed --job=71933395025 \ | awk -F'\t' 'NF>=3 { sub(/^[\xEF\xBB\xBF]?[0-9T:.\\-Z ]+/, "", $3); print $3 }' \ | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' \ | grep -vE '^(##\[|shell: |env: |Run |[[:space:]]*$)' \ | grep -m1 -E '^\[(FAIL|ERROR)\]|^Error:|^error:|^::error' \ | head -c 300 ``` Output: ``` [FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no src/app/demos/hitl-in-chat/ directory ``` Which is exactly the signal that was missing from tonight's bare alert. - `oxfmt --check` and `oxlint` on the edited file: clean. ## Scope - Only `.github/workflows/showcase_validate.yml` is touched. `showcase_deploy.yml` and `showcase_smoke-monitor.yml` are untouched. - Preserves the empty-webhook guard (`env.SLACK_WEBHOOK != ''`) added in CopilotKit#4065 and mirrors that pattern on the new extraction step. ## Test plan - [ ] Merge and wait for the next validate failure on `main` (or temporarily force one on a push-path) to confirm the Slack payload carries the expected `— <step>: <excerpt>` suffix. - [ ] If a subsequent run fails in a step with no matching error marker, confirm the fallback `"see workflow run for details"` renders cleanly rather than an empty string.
2 parents 1633d9f + 9559e15 commit d4e7595

1 file changed

Lines changed: 97 additions & 7 deletions

File tree

.github/workflows/showcase_validate.yml

Lines changed: 97 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -361,21 +361,111 @@ jobs:
361361
# and the PR author's inbox, and we don't want PR-author noise
362362
# pinging the OSS alerts channel. Tradeoff: a broken PR that sneaks
363363
# past review won't alert Slack until after merge.
364+
#
365+
# Extract the failed step name and first meaningful error line so the
366+
# Slack payload is actionable at a glance rather than forcing a
367+
# click-through to the workflow run. Bare "X failed" alerts bury the
368+
# signal; red alerts must carry triage-ready detail per the oss-alerts
369+
# policy. Writes `failed_step` and `error_excerpt` to $GITHUB_ENV for
370+
# consumption by the notify step below.
371+
#
372+
# This step must NEVER fail the job (it runs on failure() already; a
373+
# crash here would compound the original failure with extraction
374+
# noise and could block the notify step). All extraction uses `|| true`
375+
# fallbacks so a malformed jobs response or truncated log still yields
376+
# sane defaults ("unknown" / "see workflow run for details").
377+
- name: Extract failure details for Slack
378+
id: extract
379+
if: failure() && github.event_name == 'push' && env.SLACK_WEBHOOK != ''
380+
env:
381+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
382+
GH_REPO: ${{ github.repository }}
383+
RUN_ID: ${{ github.run_id }}
384+
run: |
385+
set +e # best-effort: never block the notify step below
386+
387+
# --- Find the currently-running job and its first failed step ---
388+
# The jobs API returns every job in the run. We identify *this*
389+
# job by name (matches `jobs.validate.name`) rather than
390+
# job.status=='in_progress', because at this point the step we're
391+
# running hasn't flipped the job state yet in the API. Fall back
392+
# to the first job with a failed step if the name match misses
393+
# (e.g. future rename drift).
394+
jobs_json=$(gh api "/repos/${GH_REPO}/actions/runs/${RUN_ID}/jobs" --paginate 2>/dev/null)
395+
job_id=$(printf '%s' "$jobs_json" | jq -r '
396+
.jobs // []
397+
| map(select(.name == "Validate Showcase"))
398+
| (.[0].id // empty)
399+
' 2>/dev/null)
400+
if [ -z "$job_id" ]; then
401+
job_id=$(printf '%s' "$jobs_json" | jq -r '
402+
.jobs // []
403+
| map(select(.steps // [] | map(.conclusion) | index("failure")))
404+
| (.[0].id // empty)
405+
' 2>/dev/null)
406+
fi
407+
failed_step=$(printf '%s' "$jobs_json" | jq -r --arg id "$job_id" '
408+
.jobs // []
409+
| map(select((.id|tostring) == $id))
410+
| (.[0].steps // [])
411+
| map(select(.conclusion == "failure"))
412+
| (.[0].name // "unknown step")
413+
' 2>/dev/null)
414+
[ -z "$failed_step" ] && failed_step="unknown step"
415+
416+
# --- Pull log and extract first meaningful error line ------------
417+
# `gh run view --log-failed` output is TSV: job\tstep\ttimestamp + content.
418+
# Strip the three leading columns to get the raw step output, strip
419+
# ANSI escape codes, strip any stray BOM, skip runner/group/env
420+
# header noise, then grab the first line matching a recognised
421+
# error marker. Truncate to ~300 chars so the Slack payload stays
422+
# well under the 800-char budget even with escaping overhead.
423+
error_excerpt="see workflow run for details"
424+
if [ -n "$job_id" ]; then
425+
log_excerpt=$(gh run view "$RUN_ID" --repo "$GH_REPO" --log-failed --job="$job_id" 2>/dev/null \
426+
| awk -F'\t' 'NF>=3 { sub(/^[\xEF\xBB\xBF]?[0-9T:.\-Z ]+/, "", $3); print $3 }' \
427+
| sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' \
428+
| grep -vE '^(##\[|shell: |env: |Run |[[:space:]]*$)' \
429+
| grep -m1 -E '^\[(FAIL|ERROR)\]|^Error:|^error:|^::error' \
430+
| head -c 300)
431+
if [ -n "$log_excerpt" ]; then
432+
error_excerpt="$log_excerpt"
433+
fi
434+
fi
435+
436+
# --- Emit to $GITHUB_ENV using heredoc delimiter -----------------
437+
# Heredoc delimiter protects against values that contain `=` or
438+
# newlines breaking the KEY=VALUE format. The delimiter is a
439+
# long random-ish string unlikely to appear in any log line.
440+
{
441+
echo "failed_step<<EOF_FAILED_STEP_b3f2"
442+
printf '%s\n' "$failed_step"
443+
echo "EOF_FAILED_STEP_b3f2"
444+
echo "error_excerpt<<EOF_ERROR_EXCERPT_b3f2"
445+
printf '%s\n' "$error_excerpt"
446+
echo "EOF_ERROR_EXCERPT_b3f2"
447+
} >> "$GITHUB_ENV"
448+
449+
exit 0 # belt-and-suspenders: never propagate a failure
450+
364451
- name: Notify Slack (failure)
365452
if: failure() && github.event_name == 'push' && env.SLACK_WEBHOOK != ''
366453
uses: slackapi/slack-github-action@v2.1.0
367454
with:
368455
webhook: ${{ secrets.SLACK_WEBHOOK_OSS_ALERTS }}
369456
webhook-type: incoming-webhook
370457
# Defensive: wrap dynamic values via toJSON(format(...)) so that
371-
# if github.repository ever contains characters that would break
372-
# the JSON payload (quotes, backslashes, newlines), the value is
373-
# safely JSON-encoded instead of injected as raw text. Matches
374-
# the pattern used in showcase_drift-report.yml. github.run_id is
375-
# numeric so safe on its own, but we wrap both for consistency
376-
# and defense-in-depth.
458+
# if github.repository or the extracted failed_step / error_excerpt
459+
# contain characters that would break the JSON payload (quotes,
460+
# backslashes, newlines), the value is safely JSON-encoded instead
461+
# of injected as raw text. Matches the pattern used in
462+
# showcase_drift-report.yml. github.run_id is numeric so safe on
463+
# its own, but we wrap it for consistency and defense-in-depth.
464+
# env.failed_step and env.error_excerpt are populated by the
465+
# preceding "Extract failure details" step (with safe fallbacks if
466+
# extraction fails).
377467
payload: |
378-
{ "text": ${{ toJSON(format(':x: *Showcase validate*: failed | <https://github.com/{0}/actions/runs/{1}|View run>', github.repository, github.run_id)) }} }
468+
{ "text": ${{ toJSON(format(':x: *Showcase validate*: failed — {0}: {1} | <https://github.com/{2}/actions/runs/{3}|View run>', env.failed_step, env.error_excerpt, github.repository, github.run_id)) }} }
379469
380470
- name: Log (no Slack — webhook unset)
381471
if: failure() && github.event_name == 'push' && env.SLACK_WEBHOOK == ''

0 commit comments

Comments
 (0)