Skip to content

Commit 3eeb322

Browse files
jpr5jpr5
authored
fix(showcase): prevent eval auto-trigger by requiring confirmation click (CopilotKit#4543)
## Summary - The "Run Evaluation" link in PR comments was a GET endpoint that dispatched the eval workflow directly. GitHub's link unfurling/preview bot fetches URLs embedded in comments, which caused evals to auto-trigger on every PR push without anyone clicking the link. - Changed the GET endpoint to render a confirmation page (no side effects). The actual dispatch now requires a POST via the form button click. - After dispatch, the response opens in a new tab (via `target="_blank"` on the form) and redirects to the Actions workflow run page where progress can be monitored — instead of redirecting back to the PR. ## Test plan - [ ] Deploy eval-webhook to Railway - [ ] Open a test PR, verify the bot comment link shows the confirmation page (no auto-dispatch) - [ ] Click "Run Evaluation" on the confirmation page, verify eval dispatches and new tab opens to Actions run - [ ] Verify the original tab preserves navigation history
2 parents 2fdbb1c + 6303cd6 commit 3eeb322

1 file changed

Lines changed: 100 additions & 8 deletions

File tree

‎showcase/eval-webhook/src/server.ts‎

Lines changed: 100 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -119,17 +119,74 @@ function verifyTriggerSig(
119119
return crypto.timingSafeEqual(expectedBuf, sigBuf);
120120
}
121121

122+
function validateTriggerParams(
123+
pr: string,
124+
checkRunId: string,
125+
sig: string,
126+
): { valid: true } | { valid: false; status: 400 | 403; message: string } {
127+
if (!pr || !sig) {
128+
return { valid: false, status: 400, message: "Missing parameters." };
129+
}
130+
if (!verifyTriggerSig(pr, checkRunId, sig)) {
131+
return { valid: false, status: 403, message: "Invalid signature." };
132+
}
133+
return { valid: true };
134+
}
135+
122136
app.get("/trigger/eval", async (c) => {
123137
const pr = c.req.query("pr") ?? "";
124138
const checkRunId = c.req.query("check_run_id") ?? "";
125139
const sig = c.req.query("sig") ?? "";
126140

127-
if (!pr || !sig) {
128-
return c.html("<h2>Invalid request</h2><p>Missing parameters.</p>", 400);
141+
const result = validateTriggerParams(pr, checkRunId, sig);
142+
if (result.valid !== true) {
143+
return c.html(
144+
`<h2>${result.status === 400 ? "Invalid request" : "Unauthorized"}</h2><p>${result.message}</p>`,
145+
result.status,
146+
);
129147
}
130148

131-
if (!verifyTriggerSig(pr, checkRunId, sig)) {
132-
return c.html("<h2>Unauthorized</h2><p>Invalid signature.</p>", 403);
149+
return c.html(`
150+
<!DOCTYPE html>
151+
<html>
152+
<head>
153+
<title>Showcase Eval — PR #${pr}</title>
154+
<style>
155+
body { font-family: system-ui; display: flex; justify-content: center; align-items: center; height: 100vh; margin: 0; background: #0d1117; color: #e6edf3; }
156+
div { text-align: center; }
157+
h2 { color: #3fb950; }
158+
button { background: #238636; color: #fff; border: 1px solid #2ea043; border-radius: 6px; padding: 10px 24px; font-size: 16px; cursor: pointer; font-family: system-ui; }
159+
button:hover { background: #2ea043; }
160+
</style>
161+
</head>
162+
<body>
163+
<div>
164+
<h2>Showcase Eval</h2>
165+
<p>PR #${pr}</p>
166+
<form method="POST" action="/trigger/eval" target="_blank">
167+
<input type="hidden" name="pr" value="${pr}" />
168+
<input type="hidden" name="check_run_id" value="${checkRunId}" />
169+
<input type="hidden" name="sig" value="${sig}" />
170+
<button type="submit">Run Evaluation</button>
171+
</form>
172+
</div>
173+
</body>
174+
</html>
175+
`);
176+
});
177+
178+
app.post("/trigger/eval", async (c) => {
179+
const body = await c.req.parseBody();
180+
const pr = String(body["pr"] ?? "");
181+
const checkRunId = String(body["check_run_id"] ?? "");
182+
const sig = String(body["sig"] ?? "");
183+
184+
const result = validateTriggerParams(pr, checkRunId, sig);
185+
if (result.valid !== true) {
186+
return c.html(
187+
`<h2>${result.status === 400 ? "Invalid request" : "Unauthorized"}</h2><p>${result.message}</p>`,
188+
result.status,
189+
);
133190
}
134191

135192
const octokit = getOctokit();
@@ -162,19 +219,54 @@ app.get("/trigger/eval", async (c) => {
162219
`Trigger link: dispatched eval for PR #${pr}, check_run_id=${checkRunId}`,
163220
);
164221

165-
const prUrl = `https://github.com/CopilotKit/CopilotKit/pull/${pr}`;
222+
// Poll for the Actions run URL (up to ~5 seconds)
223+
const fallbackUrl =
224+
"https://github.com/CopilotKit/CopilotKit/actions/workflows/showcase_eval.yml";
225+
let runUrl = fallbackUrl;
226+
const startTime = Date.now();
227+
228+
for (let i = 0; i < 10; i++) {
229+
await new Promise((resolve) => setTimeout(resolve, 500));
230+
try {
231+
const runs = await octokit.actions.listWorkflowRuns({
232+
owner: "CopilotKit",
233+
repo: "CopilotKit",
234+
workflow_id: "showcase_eval.yml",
235+
event: "workflow_dispatch",
236+
per_page: 5,
237+
});
238+
const recent = runs.data.workflow_runs.find((run) => {
239+
const created = new Date(run.created_at).getTime();
240+
return Date.now() - created < 30_000;
241+
});
242+
if (recent) {
243+
runUrl = recent.html_url;
244+
break;
245+
}
246+
} catch {
247+
// ignore polling errors
248+
}
249+
if (Date.now() - startTime > 5000) break;
250+
}
251+
166252
return c.html(`
167253
<!DOCTYPE html>
168254
<html>
169255
<head>
170-
<meta http-equiv="refresh" content="2;url=${prUrl}">
171-
<style>body{font-family:system-ui;display:flex;justify-content:center;align-items:center;height:100vh;margin:0;background:#0d1117;color:#e6edf3;}div{text-align:center;}h2{color:#3fb950;}a{color:#58a6ff;}</style>
256+
<title>Showcase Eval — dispatched</title>
257+
<meta http-equiv="refresh" content="0;url=${runUrl}">
258+
<style>
259+
body { font-family: system-ui; display: flex; justify-content: center; align-items: center; height: 100vh; margin: 0; background: #0d1117; color: #e6edf3; }
260+
div { text-align: center; }
261+
h2 { color: #3fb950; }
262+
a { color: #58a6ff; }
263+
</style>
172264
</head>
173265
<body>
174266
<div>
175267
<h2>Showcase Eval triggered</h2>
176268
<p>PR #${pr} — evaluation dispatched.</p>
177-
<p>Redirecting to <a href="${prUrl}">the PR</a>...</p>
269+
<p>Redirecting to <a href="${runUrl}">the Actions run</a>...</p>
178270
</div>
179271
</body>
180272
</html>

0 commit comments

Comments
 (0)