Skip to content

Commit 611a92f

Browse files
jpr5jpr5
authored
fix(showcase): generalize silent-hang watchdog + unbuffered stdout across starters (CopilotKit#4116)
## Summary Generalizes the silent-hang watchdog + unbuffered-stdout shape proven in `showcase/packages/crewai-crews/entrypoint.sh` (PRs CopilotKit#4114 + CopilotKit#4115) to every Bucket-B starter, the Bucket-C spring-ai starter, the generator, and the shared starter template. **Reference shape:** `showcase/packages/crewai-crews/entrypoint.sh` on `origin/main` (unchanged in this PR; we generalize from it). **Notion proposal:** https://www.notion.so/3493aa3818528191bc08f8b2f3fb1e88 **Prior art:** CopilotKit#4114 (watchdog), CopilotKit#4115 (unbuffered stdout) ## What the watchdog does A backgrounded subshell polls the agent's health endpoint every 30s; after 3 consecutive failures (~90s), it `kill -9`s the agent so `wait -n` returns and the container runtime restarts via the normal path rather than a forced `exit` that would bypass logging. ## Per-starter coverage | Slug | Framework | Agent port | Health path | Watchdog applied | |---|---|---|---|---| | ag2 | Python / FastAPI | 8123 | `/health` | Yes | | agno | Python / FastAPI | 8123 | `/health` | Yes | | claude-sdk-python | Python / FastAPI | 8123 | `/health` | Yes | | claude-sdk-typescript | TypeScript / Express | 8123 | `/health` | Yes | | crewai-crews | Python / FastAPI | 8123 | `/health` | Yes (via regen from template; matches reference shape) | | google-adk | Python / FastAPI | 8123 | `/health` | Yes | | langgraph-fastapi | Python / langgraph_cli | 8123 | `/ok` | Yes | | langgraph-python | Python / langgraph_cli | 8123 | `/ok` | Yes | | langgraph-typescript | TS / @langchain/langgraph-cli | 8123 | `/ok` | Yes | | langroid | Python / FastAPI | 8123 | `/health` | Yes | | llamaindex | Python / FastAPI | 8123 | `/health` | Yes | | mastra | TypeScript / mastra dev | 8123 | `/api` | Yes (starter only; package side is single `exec next start` — N/A) | | ms-agent-dotnet | .NET / ASP.NET | 8123 | `/health` | Yes | | ms-agent-python | Python / FastAPI | 8123 | `/health` | Yes | | pydantic-ai | Python / FastAPI | 8123 | `/health` | Yes | | spring-ai | Java / Spring Boot | 8123 (starter) / 8000 (package) | `/health` | Yes; extends the existing 60s startup probe | | strands | Python / FastAPI | 8123 | `/health` | Yes | ## Per-package coverage (hand-applied, not regen-able) Every `showcase/packages/<slug>/entrypoint.sh` that runs a separate agent process received the watchdog + unbuffered block. Python variants also received `python -u` on their uvicorn / langgraph_cli invocations and `awk ... fflush()` log prefixing (replacing the prior `sed` pipe formulation so `$!` correctly captures the agent PID). **N/A:** `packages/mastra/entrypoint.sh` is a single `exec npx next start` — no backgrounded agent to watch. ## Generator + template changes - `showcase/scripts/generate-starters.ts` — new `getAgentHealthPath()` + `getWatchdogBlock()` helpers, plus Spring Boot `/health` 60s startup probe. - `showcase/starters/template/entrypoint.template.sh` — adds `$WATCHDOG_PID` to cleanup, injects `{{WATCHDOG_BLOCK}}`, narrows the final `wait -n` to `$AGENT_PID $NEXTJS_PID` only. ## Out of scope (not touched) - `showcase/packages/crewai-crews/entrypoint.sh` (reference shape; already correct on main). - `showcase/{shell,shell-dashboard,shell-docs}` (separate workstream). - Any agent source (Python / TS / Java / .NET). - Dependency version bumps. ## Test plan - [x] `bash -n` clean on all 32 modified `.sh` files. - [x] `pnpm -C showcase/scripts test` — 1079/1079 tests pass (includes starter-output snapshots). - [x] Watchdog marker present in every modified `packages/<slug>/entrypoint.sh` except mastra (intentional N/A). - [x] `PYTHONUNBUFFERED=1` present in every Python-based package entrypoint. - [ ] CI green before merge. - [ ] Post-merge: observe one deploy per starter on Railway to confirm watchdog doesn't false-positive against a healthy agent. ## Judgment calls (documented) - **`starters/crewai-crews/agent/{requirements.txt,agent_server.py}`**: kept as regen output. These match the current `packages/crewai-crews/` source (shimless, `ag-ui-crewai >=0.2.0`) that landed on main in commit 9379b88. Reverting would re-desync the starter from the upstream source the generator reads. - **`langgraph-typescript` packages side**: classified as Bucket B. The file on `origin/main` backgrounds `@langchain/langgraph-cli dev` on `:8123` and then `exec`s Next.js — the langgraph process can hang independently, so the watchdog applies. - **`mastra` packages side**: classified as N/A. The file on `origin/main` is a single `exec npx next start` with no backgrounded agent. - **`mastra` starter side**: Bucket B (the starter DOES run `mastra dev` on `:8123`). Watchdog probes `/api` (mastra's REST surface; no dedicated health endpoint but returns 200 once the dev server is ready).
2 parents db184eb + 6f7dd1b commit 611a92f

35 files changed

Lines changed: 2133 additions & 332 deletions

File tree

‎showcase/packages/ag2/entrypoint.sh‎

Lines changed: 103 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,108 @@
11
#!/bin/bash
22
set -e
33

4-
# Start agent backend
5-
python -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &
4+
cleanup() {
5+
kill $AGENT_PID $NEXTJS_PID $WATCHDOG_PID 2>/dev/null || true
6+
}
7+
trap cleanup EXIT
68

7-
# Start Next.js frontend (PORT defaults to 10000 for Render)
8-
npx next start --port ${PORT:-10000} &
9+
# Disable Python stdout buffering so the FastAPI/uvicorn agent flushes
10+
# tracebacks and log lines immediately. Without this a silent crash during
11+
# module import can sit in Python's userspace buffer until the process
12+
# exits, by which point the container is already gone.
13+
export PYTHONUNBUFFERED=1
914

10-
# Wait for either process to exit
11-
wait -n
12-
exit $?
15+
echo "========================================="
16+
echo "[entrypoint] Starting showcase package: ag2"
17+
echo "[entrypoint] Time: $(date -u)"
18+
echo "[entrypoint] PORT=${PORT:-not set}"
19+
echo "[entrypoint] NODE_ENV=${NODE_ENV:-not set}"
20+
echo "========================================="
21+
22+
if [ -z "$OPENAI_API_KEY" ]; then
23+
echo "[entrypoint] WARNING: OPENAI_API_KEY is not set! Agent will fail."
24+
else
25+
echo "[entrypoint] OPENAI_API_KEY: set (${#OPENAI_API_KEY} chars)"
26+
fi
27+
28+
# Start agent backend on :8000 with log prefixing so its output is
29+
# distinguishable from Next.js in the Railway log stream.
30+
#
31+
# Belt-and-suspenders log flushing: `PYTHONUNBUFFERED=1` above exports the env
32+
# var, but a child process could in principle un-export or override it. The
33+
# `-u` flag to the Python interpreter forces unbuffered stdout/stderr at the
34+
# interpreter level and is not overridable by user code. Combined with the
35+
# `fflush()` inside the awk pipe below, this guarantees uvicorn request lines
36+
# and tracebacks reach Railway's log stream line-at-a-time rather than
37+
# block-buffered in pipe buffers.
38+
echo "[entrypoint] Starting Python agent on port 8000..."
39+
python -u -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &> >(awk '{print "[agent] " $0; fflush()}') &
40+
AGENT_PID=$!
41+
sleep 2
42+
if kill -0 $AGENT_PID 2>/dev/null; then
43+
echo "[entrypoint] Agent started (PID: $AGENT_PID)"
44+
else
45+
echo "[entrypoint] ERROR: Agent failed to start — exiting"
46+
exit 1
47+
fi
48+
49+
echo "========================================="
50+
echo "[entrypoint] Starting Next.js frontend on port ${PORT:-10000}..."
51+
echo "========================================="
52+
53+
PORT=${PORT:-10000}
54+
# Scope NODE_ENV=production to the Next.js invocation ONLY, not the whole
55+
# container environment. `ENV NODE_ENV=production` at the image level would
56+
# leak into every child process (Python agent, shell, healthchecks). `env`
57+
# prefix binds the value to this single exec.
58+
env NODE_ENV=production npx next start --port $PORT &> >(awk '{print "[nextjs] " $0; fflush()}') &
59+
NEXTJS_PID=$!
60+
61+
echo "[entrypoint] Next.js started (PID: $NEXTJS_PID)"
62+
63+
# Watchdog: Railway deploys of showcase packages have been observed to hit a
64+
# silent agent hang — the Python process stays alive (so `wait -n` never
65+
# fires and the container never restarts) but stops responding on :8000.
66+
# Poll the agent's /health endpoint every 30s; after 3 consecutive failures
67+
# (90s of unreachable agent), kill the agent process so `wait -n` returns
68+
# and Railway restarts the container. We kill the agent (not the whole
69+
# script) first so `set -e` + `wait -n; exit $?` handles the restart
70+
# through the normal path rather than a forced `exit` that would bypass
71+
# logging. Generalized from showcase/packages/crewai-crews/entrypoint.sh
72+
# (PRs #4114 + #4115).
73+
(
74+
FAILS=0
75+
while sleep 30; do
76+
if ! kill -0 $AGENT_PID 2>/dev/null; then
77+
# Agent already dead — wait -n in the main shell will handle it.
78+
break
79+
fi
80+
if curl -fsS --max-time 5 http://127.0.0.1:8000/health > /dev/null 2>&1; then
81+
FAILS=0
82+
else
83+
FAILS=$((FAILS + 1))
84+
echo "[watchdog] Agent health probe failed (count=$FAILS)"
85+
if [ $FAILS -ge 3 ]; then
86+
echo "[watchdog] Agent unresponsive for ~90s — killing PID $AGENT_PID to trigger container restart"
87+
kill -9 $AGENT_PID 2>/dev/null || true
88+
break
89+
fi
90+
fi
91+
done
92+
) &
93+
WATCHDOG_PID=$!
94+
95+
echo "[entrypoint] Watchdog started (PID: $WATCHDOG_PID)"
96+
echo "[entrypoint] All processes running. Waiting..."
97+
98+
wait -n $AGENT_PID $NEXTJS_PID
99+
EXIT_CODE=$?
100+
if ! kill -0 $AGENT_PID 2>/dev/null; then
101+
echo "[entrypoint] Agent (PID: $AGENT_PID) exited with code $EXIT_CODE"
102+
elif ! kill -0 $NEXTJS_PID 2>/dev/null; then
103+
echo "[entrypoint] Next.js (PID: $NEXTJS_PID) exited with code $EXIT_CODE"
104+
else
105+
echo "[entrypoint] A process exited with code $EXIT_CODE"
106+
fi
107+
108+
exit $EXIT_CODE

‎showcase/packages/agno/entrypoint.sh‎

100644100755
Lines changed: 103 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,108 @@
11
#!/bin/bash
22
set -e
33

4-
# Start agent backend
5-
python -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &
4+
cleanup() {
5+
kill $AGENT_PID $NEXTJS_PID $WATCHDOG_PID 2>/dev/null || true
6+
}
7+
trap cleanup EXIT
68

7-
# Start Next.js frontend (PORT defaults to 10000 for Render)
8-
npx next start --port ${PORT:-10000} &
9+
# Disable Python stdout buffering so the FastAPI/uvicorn agent flushes
10+
# tracebacks and log lines immediately. Without this a silent crash during
11+
# module import can sit in Python's userspace buffer until the process
12+
# exits, by which point the container is already gone.
13+
export PYTHONUNBUFFERED=1
914

10-
# Wait for either process to exit
11-
wait -n
12-
exit $?
15+
echo "========================================="
16+
echo "[entrypoint] Starting showcase package: agno"
17+
echo "[entrypoint] Time: $(date -u)"
18+
echo "[entrypoint] PORT=${PORT:-not set}"
19+
echo "[entrypoint] NODE_ENV=${NODE_ENV:-not set}"
20+
echo "========================================="
21+
22+
if [ -z "$OPENAI_API_KEY" ]; then
23+
echo "[entrypoint] WARNING: OPENAI_API_KEY is not set! Agent will fail."
24+
else
25+
echo "[entrypoint] OPENAI_API_KEY: set (${#OPENAI_API_KEY} chars)"
26+
fi
27+
28+
# Start agent backend on :8000 with log prefixing so its output is
29+
# distinguishable from Next.js in the Railway log stream.
30+
#
31+
# Belt-and-suspenders log flushing: `PYTHONUNBUFFERED=1` above exports the env
32+
# var, but a child process could in principle un-export or override it. The
33+
# `-u` flag to the Python interpreter forces unbuffered stdout/stderr at the
34+
# interpreter level and is not overridable by user code. Combined with the
35+
# `fflush()` inside the awk pipe below, this guarantees uvicorn request lines
36+
# and tracebacks reach Railway's log stream line-at-a-time rather than
37+
# block-buffered in pipe buffers.
38+
echo "[entrypoint] Starting Python agent on port 8000..."
39+
python -u -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &> >(awk '{print "[agent] " $0; fflush()}') &
40+
AGENT_PID=$!
41+
sleep 2
42+
if kill -0 $AGENT_PID 2>/dev/null; then
43+
echo "[entrypoint] Agent started (PID: $AGENT_PID)"
44+
else
45+
echo "[entrypoint] ERROR: Agent failed to start — exiting"
46+
exit 1
47+
fi
48+
49+
echo "========================================="
50+
echo "[entrypoint] Starting Next.js frontend on port ${PORT:-10000}..."
51+
echo "========================================="
52+
53+
PORT=${PORT:-10000}
54+
# Scope NODE_ENV=production to the Next.js invocation ONLY, not the whole
55+
# container environment. `ENV NODE_ENV=production` at the image level would
56+
# leak into every child process (Python agent, shell, healthchecks). `env`
57+
# prefix binds the value to this single exec.
58+
env NODE_ENV=production npx next start --port $PORT &> >(awk '{print "[nextjs] " $0; fflush()}') &
59+
NEXTJS_PID=$!
60+
61+
echo "[entrypoint] Next.js started (PID: $NEXTJS_PID)"
62+
63+
# Watchdog: Railway deploys of showcase packages have been observed to hit a
64+
# silent agent hang — the Python process stays alive (so `wait -n` never
65+
# fires and the container never restarts) but stops responding on :8000.
66+
# Poll the agent's /health endpoint every 30s; after 3 consecutive failures
67+
# (90s of unreachable agent), kill the agent process so `wait -n` returns
68+
# and Railway restarts the container. We kill the agent (not the whole
69+
# script) first so `set -e` + `wait -n; exit $?` handles the restart
70+
# through the normal path rather than a forced `exit` that would bypass
71+
# logging. Generalized from showcase/packages/crewai-crews/entrypoint.sh
72+
# (PRs #4114 + #4115).
73+
(
74+
FAILS=0
75+
while sleep 30; do
76+
if ! kill -0 $AGENT_PID 2>/dev/null; then
77+
# Agent already dead — wait -n in the main shell will handle it.
78+
break
79+
fi
80+
if curl -fsS --max-time 5 http://127.0.0.1:8000/health > /dev/null 2>&1; then
81+
FAILS=0
82+
else
83+
FAILS=$((FAILS + 1))
84+
echo "[watchdog] Agent health probe failed (count=$FAILS)"
85+
if [ $FAILS -ge 3 ]; then
86+
echo "[watchdog] Agent unresponsive for ~90s — killing PID $AGENT_PID to trigger container restart"
87+
kill -9 $AGENT_PID 2>/dev/null || true
88+
break
89+
fi
90+
fi
91+
done
92+
) &
93+
WATCHDOG_PID=$!
94+
95+
echo "[entrypoint] Watchdog started (PID: $WATCHDOG_PID)"
96+
echo "[entrypoint] All processes running. Waiting..."
97+
98+
wait -n $AGENT_PID $NEXTJS_PID
99+
EXIT_CODE=$?
100+
if ! kill -0 $AGENT_PID 2>/dev/null; then
101+
echo "[entrypoint] Agent (PID: $AGENT_PID) exited with code $EXIT_CODE"
102+
elif ! kill -0 $NEXTJS_PID 2>/dev/null; then
103+
echo "[entrypoint] Next.js (PID: $NEXTJS_PID) exited with code $EXIT_CODE"
104+
else
105+
echo "[entrypoint] A process exited with code $EXIT_CODE"
106+
fi
107+
108+
exit $EXIT_CODE

‎showcase/packages/claude-sdk-python/entrypoint.sh‎

100644100755
Lines changed: 103 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,108 @@
11
#!/bin/bash
22
set -e
33

4-
# Start agent backend
5-
python -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &
4+
cleanup() {
5+
kill $AGENT_PID $NEXTJS_PID $WATCHDOG_PID 2>/dev/null || true
6+
}
7+
trap cleanup EXIT
68

7-
# Start Next.js frontend (PORT defaults to 10000 for Render)
8-
npx next start --port ${PORT:-10000} &
9+
# Disable Python stdout buffering so the FastAPI/uvicorn agent flushes
10+
# tracebacks and log lines immediately. Without this a silent crash during
11+
# module import can sit in Python's userspace buffer until the process
12+
# exits, by which point the container is already gone.
13+
export PYTHONUNBUFFERED=1
914

10-
# Wait for either process to exit
11-
wait -n
12-
exit $?
15+
echo "========================================="
16+
echo "[entrypoint] Starting showcase package: claude-sdk-python"
17+
echo "[entrypoint] Time: $(date -u)"
18+
echo "[entrypoint] PORT=${PORT:-not set}"
19+
echo "[entrypoint] NODE_ENV=${NODE_ENV:-not set}"
20+
echo "========================================="
21+
22+
if [ -z "$OPENAI_API_KEY" ]; then
23+
echo "[entrypoint] WARNING: OPENAI_API_KEY is not set! Agent will fail."
24+
else
25+
echo "[entrypoint] OPENAI_API_KEY: set (${#OPENAI_API_KEY} chars)"
26+
fi
27+
28+
# Start agent backend on :8000 with log prefixing so its output is
29+
# distinguishable from Next.js in the Railway log stream.
30+
#
31+
# Belt-and-suspenders log flushing: `PYTHONUNBUFFERED=1` above exports the env
32+
# var, but a child process could in principle un-export or override it. The
33+
# `-u` flag to the Python interpreter forces unbuffered stdout/stderr at the
34+
# interpreter level and is not overridable by user code. Combined with the
35+
# `fflush()` inside the awk pipe below, this guarantees uvicorn request lines
36+
# and tracebacks reach Railway's log stream line-at-a-time rather than
37+
# block-buffered in pipe buffers.
38+
echo "[entrypoint] Starting Python agent on port 8000..."
39+
python -u -m uvicorn agent_server:app --host 0.0.0.0 --port 8000 &> >(awk '{print "[agent] " $0; fflush()}') &
40+
AGENT_PID=$!
41+
sleep 2
42+
if kill -0 $AGENT_PID 2>/dev/null; then
43+
echo "[entrypoint] Agent started (PID: $AGENT_PID)"
44+
else
45+
echo "[entrypoint] ERROR: Agent failed to start — exiting"
46+
exit 1
47+
fi
48+
49+
echo "========================================="
50+
echo "[entrypoint] Starting Next.js frontend on port ${PORT:-10000}..."
51+
echo "========================================="
52+
53+
PORT=${PORT:-10000}
54+
# Scope NODE_ENV=production to the Next.js invocation ONLY, not the whole
55+
# container environment. `ENV NODE_ENV=production` at the image level would
56+
# leak into every child process (Python agent, shell, healthchecks). `env`
57+
# prefix binds the value to this single exec.
58+
env NODE_ENV=production npx next start --port $PORT &> >(awk '{print "[nextjs] " $0; fflush()}') &
59+
NEXTJS_PID=$!
60+
61+
echo "[entrypoint] Next.js started (PID: $NEXTJS_PID)"
62+
63+
# Watchdog: Railway deploys of showcase packages have been observed to hit a
64+
# silent agent hang — the Python process stays alive (so `wait -n` never
65+
# fires and the container never restarts) but stops responding on :8000.
66+
# Poll the agent's /health endpoint every 30s; after 3 consecutive failures
67+
# (90s of unreachable agent), kill the agent process so `wait -n` returns
68+
# and Railway restarts the container. We kill the agent (not the whole
69+
# script) first so `set -e` + `wait -n; exit $?` handles the restart
70+
# through the normal path rather than a forced `exit` that would bypass
71+
# logging. Generalized from showcase/packages/crewai-crews/entrypoint.sh
72+
# (PRs #4114 + #4115).
73+
(
74+
FAILS=0
75+
while sleep 30; do
76+
if ! kill -0 $AGENT_PID 2>/dev/null; then
77+
# Agent already dead — wait -n in the main shell will handle it.
78+
break
79+
fi
80+
if curl -fsS --max-time 5 http://127.0.0.1:8000/health > /dev/null 2>&1; then
81+
FAILS=0
82+
else
83+
FAILS=$((FAILS + 1))
84+
echo "[watchdog] Agent health probe failed (count=$FAILS)"
85+
if [ $FAILS -ge 3 ]; then
86+
echo "[watchdog] Agent unresponsive for ~90s — killing PID $AGENT_PID to trigger container restart"
87+
kill -9 $AGENT_PID 2>/dev/null || true
88+
break
89+
fi
90+
fi
91+
done
92+
) &
93+
WATCHDOG_PID=$!
94+
95+
echo "[entrypoint] Watchdog started (PID: $WATCHDOG_PID)"
96+
echo "[entrypoint] All processes running. Waiting..."
97+
98+
wait -n $AGENT_PID $NEXTJS_PID
99+
EXIT_CODE=$?
100+
if ! kill -0 $AGENT_PID 2>/dev/null; then
101+
echo "[entrypoint] Agent (PID: $AGENT_PID) exited with code $EXIT_CODE"
102+
elif ! kill -0 $NEXTJS_PID 2>/dev/null; then
103+
echo "[entrypoint] Next.js (PID: $NEXTJS_PID) exited with code $EXIT_CODE"
104+
else
105+
echo "[entrypoint] A process exited with code $EXIT_CODE"
106+
fi
107+
108+
exit $EXIT_CODE

0 commit comments

Comments
 (0)