Skip to content

Commit 8cb84e8

Browse files
AlemTuzlakAlemTuzlak
committed
test(showcase): replicate hitl-in-chat regression spec across all 17 integrations
The hitl-in-chat demo ships in 17 integrations (langgraph-python plus 16 others — mastra, strands, ag2, agno, crewai-crews, langgraph-typescript, langgraph-fastapi, pydantic-ai, llamaindex, langroid, claude-sdk-python, claude-sdk-typescript, ms-agent-python, ms-agent-dotnet, spring-ai, google-adk). All shipped placeholder e2e specs that only checked the chat input was visible — none exercised the actual booking flow. Replace each with the full booking-flow spec written for langgraph-python: 1. The "Schedule a 1:1 with Alice" suggestion renders the time-picker card AND the Tokyo greeting is absent (regression guard against the broad aimock `userMessage: "Alice"` matcher). 2. Picking a slot transitions to the picked-state card and produces a "Booked … Alice" assistant follow-up. 3. The "Book a call with sales" suggestion runs the same flow with the sales attendee. Also add the matching aimock fixture pair for the sales suggestion in feature-parity.json — without it, case 3 would only pass against real OpenAI, not the aimock-backed CI deployments. The pair mirrors the Alice fixture pair: `book_call` toolCall on first turn, confirmation message after the picker resolves. Per-integration coverage matters because each integration has its own framework-specific HITL wiring (`useHumanInTheLoop` binding to the agent, agent-side tool registration, run streaming protocol) that can regress independently of the shared aimock fixture.
1 parent 846a8a8 commit 8cb84e8

18 files changed

Lines changed: 1204 additions & 756 deletions

File tree

‎showcase/aimock/feature-parity.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -651,6 +651,30 @@
651651
]
652652
}
653653
},
654+
{
655+
"match": {
656+
"userMessage": "Please book an intro call with the sales team to discuss pricing.",
657+
"hasToolResult": false
658+
},
659+
"response": {
660+
"toolCalls": [
661+
{
662+
"id": "call_hitl_book_intro_sales_001",
663+
"name": "book_call",
664+
"arguments": "{\"topic\":\"Intro call — discuss pricing\",\"attendee\":\"Sales team\"}"
665+
}
666+
]
667+
}
668+
},
669+
{
670+
"match": {
671+
"userMessage": "Please book an intro call with the sales team to discuss pricing.",
672+
"hasToolResult": true
673+
},
674+
"response": {
675+
"content": "Booked the intro call with the sales team for the time you selected — calendar invite is on its way."
676+
}
677+
},
654678
{
655679
"match": {
656680
"userMessage": "Schedule a 1:1 with Alice next week to review Q2 goals.",

‎showcase/integrations/ag2/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 68 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,100 @@
11
import { test, expect } from "@playwright/test";
22

3-
test.describe("Human in the Loop", () => {
3+
test.describe("HITL in chat — booking flow", () => {
44
test.beforeEach(async ({ page }) => {
5-
await page.goto("/demos/hitl");
5+
await page.goto("/demos/hitl-in-chat");
66
});
77

88
test("page loads with chat input", async ({ page }) => {
99
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
1010
});
1111

12-
test("sends message and gets assistant response", async ({ page }) => {
13-
const input = page.getByPlaceholder("Type a message");
14-
await input.fill("Hello");
15-
await input.press("Enter");
16-
17-
await expect(
18-
page.locator(".copilotKitAssistantMessage").first(),
19-
).toBeVisible({
20-
timeout: 30000,
21-
});
22-
});
23-
24-
test("task request shows step selector with checkboxes", async ({ page }) => {
12+
// Regression: the broad `userMessage: "Alice"` fixture in
13+
// showcase/aimock/feature-parity.json was intercepting this suggestion and
14+
// returning a generic "Nice to meet you, Alice! ... Tokyo" greeting, so the
15+
// book_call HITL flow never fired. The fixture pair added in the same PR
16+
// as this test (full-sentence userMessage match, ordered before the broad
17+
// Alice match) routes the suggestion to a `book_call` toolCall, which is
18+
// what makes the time-picker card render.
19+
test("'Schedule a 1:1 with Alice' suggestion renders the time-picker", async ({
20+
page,
21+
}) => {
2522
const input = page.getByPlaceholder("Type a message");
26-
await input.fill(
27-
"Create a plan with steps to organize a team offsite event",
28-
);
23+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
2924
await input.press("Enter");
3025

31-
// The HITL demo surfaces a single StepSelector card regardless of whether the
32-
// underlying flow is an interrupt hook or a frontend HITL tool — assert on that.
33-
const stepSelector = page.locator('[data-testid="select-steps"]');
34-
await expect(stepSelector.first()).toBeVisible({ timeout: 60000 });
26+
const card = page.locator('[data-testid="time-picker-card"]');
27+
await expect(card).toBeVisible({ timeout: 60000 });
3528

36-
// Should have at least one step item with a checkbox
37-
const stepItems = page.locator('[data-testid="step-item"]');
38-
await expect(stepItems.first()).toBeVisible({ timeout: 5000 });
29+
// The Tokyo greeting starts with "Nice to meet you, Alice" — if any
30+
// assistant message contains that, the broad Alice fixture won and the
31+
// fix has regressed.
32+
await expect(page.getByText(/Nice to meet you, Alice/i)).toHaveCount(0);
3933

40-
// Each step should have a checkbox input
41-
const checkbox = stepItems.first().locator('input[type="checkbox"]');
42-
await expect(checkbox).toBeVisible();
34+
// The card should advertise the booking and the attendee parsed by the
35+
// fixture's toolCall arguments.
36+
await expect(card.getByText(/With Alice/i)).toBeVisible();
4337

44-
// Each step should have descriptive text
45-
const stepText = stepItems.first().locator('[data-testid="step-text"]');
46-
await expect(stepText).not.toBeEmpty();
38+
// At least one selectable time slot is present.
39+
const slots = page.locator('[data-testid="time-picker-slot"]');
40+
expect(await slots.count()).toBeGreaterThan(0);
4741
});
4842

49-
test("step selector shows enabled count and has action button", async ({
43+
test("picking a time slot resolves the HITL with a confirmation", async ({
5044
page,
5145
}) => {
5246
const input = page.getByPlaceholder("Type a message");
53-
await input.fill("Plan the steps to write a quarterly report");
47+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
5448
await input.press("Enter");
5549

56-
const stepSelector = page.locator('[data-testid="select-steps"]').first();
57-
await expect(stepSelector).toBeVisible({ timeout: 60000 });
50+
const slot = page.locator('[data-testid="time-picker-slot"]').first();
51+
await expect(slot).toBeVisible({ timeout: 60000 });
52+
await slot.click();
5853

59-
// Should show a count indicator (e.g., "3/3 selected")
60-
await expect(stepSelector.getByText(/\d+\/\d+\s*selected/)).toBeVisible();
54+
// The picked-state card replaces the slot grid.
55+
await expect(
56+
page.locator('[data-testid="time-picker-picked"]'),
57+
).toBeVisible({ timeout: 10000 });
6158

62-
// Should have either a "Perform Steps" button (interrupt) or Confirm/Reject buttons (HITL)
63-
const actionButton = stepSelector.locator("button").first();
64-
await expect(actionButton).toBeVisible();
59+
// Agent's follow-up confirmation arrives next.
60+
await expect(
61+
page
62+
.locator('[data-role="assistant"]')
63+
.filter({ hasText: /Booked.*Alice/i })
64+
.first(),
65+
).toBeVisible({ timeout: 30000 });
6566
});
6667

67-
test("can approve a step selection and agent continues", async ({ page }) => {
68+
// The other suggestion in the demo. Same HITL flow, different
69+
// attendee/topic. Pinned here so a missing/regressed aimock fixture for
70+
// this suggestion (or a wiring regression in this integration's
71+
// useHumanInTheLoop binding) fails CI loudly instead of silently falling
72+
// through to "the agent rambled at me with no toolCall."
73+
test("'Book a call with sales' suggestion runs the HITL flow end-to-end", async ({
74+
page,
75+
}) => {
6876
const input = page.getByPlaceholder("Type a message");
69-
await input.fill("Plan a trip to mars in 5 steps");
77+
await input.fill(
78+
"Please book an intro call with the sales team to discuss pricing.",
79+
);
7080
await input.press("Enter");
7181

72-
// Wait for step selector
73-
const selector = page.locator('[data-testid="select-steps"]');
74-
await expect(selector).toBeVisible({ timeout: 60000 });
82+
const card = page.locator('[data-testid="time-picker-card"]');
83+
await expect(card).toBeVisible({ timeout: 60000 });
84+
await expect(card.getByText(/Sales team/i)).toBeVisible();
7585

76-
// Find and click the action button (Perform Steps / Confirm)
77-
const actionBtn = page.locator(
78-
'button:has-text("Perform"), button:has-text("Confirm")',
79-
);
80-
await expect(actionBtn.first()).toBeVisible({ timeout: 5000 });
81-
await actionBtn.first().click();
82-
83-
// After approval, the step selector's action button should disappear
84-
// (StepSelector unmounts after onConfirm; StepsFeedback replaces buttons with
85-
// the "Accepted" or "Rejected" banner). Assert on a post-click transition
86-
// that does NOT match the pre-click "N/N selected" counter text.
87-
await expect(actionBtn.first()).not.toBeVisible({ timeout: 10000 });
86+
const slot = page.locator('[data-testid="time-picker-slot"]').first();
87+
await slot.click();
88+
89+
await expect(
90+
page.locator('[data-testid="time-picker-picked"]'),
91+
).toBeVisible({ timeout: 10000 });
92+
93+
await expect(
94+
page
95+
.locator('[data-role="assistant"]')
96+
.filter({ hasText: /Booked.*sales team/i })
97+
.first(),
98+
).toBeVisible({ timeout: 30000 });
8899
});
89100
});

‎showcase/integrations/agno/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 68 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,100 @@
11
import { test, expect } from "@playwright/test";
22

3-
test.describe("Human in the Loop", () => {
3+
test.describe("HITL in chat — booking flow", () => {
44
test.beforeEach(async ({ page }) => {
5-
await page.goto("/demos/hitl");
5+
await page.goto("/demos/hitl-in-chat");
66
});
77

88
test("page loads with chat input", async ({ page }) => {
99
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
1010
});
1111

12-
test("sends message and gets assistant response", async ({ page }) => {
13-
const input = page.getByPlaceholder("Type a message");
14-
await input.fill("Hello");
15-
await input.press("Enter");
16-
17-
await expect(
18-
page.locator(".copilotKitAssistantMessage").first(),
19-
).toBeVisible({
20-
timeout: 30000,
21-
});
22-
});
23-
24-
test("task request shows step selector with checkboxes", async ({ page }) => {
12+
// Regression: the broad `userMessage: "Alice"` fixture in
13+
// showcase/aimock/feature-parity.json was intercepting this suggestion and
14+
// returning a generic "Nice to meet you, Alice! ... Tokyo" greeting, so the
15+
// book_call HITL flow never fired. The fixture pair added in the same PR
16+
// as this test (full-sentence userMessage match, ordered before the broad
17+
// Alice match) routes the suggestion to a `book_call` toolCall, which is
18+
// what makes the time-picker card render.
19+
test("'Schedule a 1:1 with Alice' suggestion renders the time-picker", async ({
20+
page,
21+
}) => {
2522
const input = page.getByPlaceholder("Type a message");
26-
await input.fill(
27-
"Create a plan with steps to organize a team offsite event",
28-
);
23+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
2924
await input.press("Enter");
3025

31-
// The HITL demo surfaces a single StepSelector card regardless of whether the
32-
// underlying flow is an interrupt hook or a frontend HITL tool — assert on that.
33-
const stepSelector = page.locator('[data-testid="select-steps"]');
34-
await expect(stepSelector.first()).toBeVisible({ timeout: 60000 });
26+
const card = page.locator('[data-testid="time-picker-card"]');
27+
await expect(card).toBeVisible({ timeout: 60000 });
3528

36-
// Should have at least one step item with a checkbox
37-
const stepItems = page.locator('[data-testid="step-item"]');
38-
await expect(stepItems.first()).toBeVisible({ timeout: 5000 });
29+
// The Tokyo greeting starts with "Nice to meet you, Alice" — if any
30+
// assistant message contains that, the broad Alice fixture won and the
31+
// fix has regressed.
32+
await expect(page.getByText(/Nice to meet you, Alice/i)).toHaveCount(0);
3933

40-
// Each step should have a checkbox input
41-
const checkbox = stepItems.first().locator('input[type="checkbox"]');
42-
await expect(checkbox).toBeVisible();
34+
// The card should advertise the booking and the attendee parsed by the
35+
// fixture's toolCall arguments.
36+
await expect(card.getByText(/With Alice/i)).toBeVisible();
4337

44-
// Each step should have descriptive text
45-
const stepText = stepItems.first().locator('[data-testid="step-text"]');
46-
await expect(stepText).not.toBeEmpty();
38+
// At least one selectable time slot is present.
39+
const slots = page.locator('[data-testid="time-picker-slot"]');
40+
expect(await slots.count()).toBeGreaterThan(0);
4741
});
4842

49-
test("step selector shows enabled count and has action button", async ({
43+
test("picking a time slot resolves the HITL with a confirmation", async ({
5044
page,
5145
}) => {
5246
const input = page.getByPlaceholder("Type a message");
53-
await input.fill("Plan the steps to write a quarterly report");
47+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
5448
await input.press("Enter");
5549

56-
const stepSelector = page.locator('[data-testid="select-steps"]').first();
57-
await expect(stepSelector).toBeVisible({ timeout: 60000 });
50+
const slot = page.locator('[data-testid="time-picker-slot"]').first();
51+
await expect(slot).toBeVisible({ timeout: 60000 });
52+
await slot.click();
5853

59-
// Should show a count indicator (e.g., "3/3 selected")
60-
await expect(stepSelector.getByText(/\d+\/\d+\s*selected/)).toBeVisible();
54+
// The picked-state card replaces the slot grid.
55+
await expect(
56+
page.locator('[data-testid="time-picker-picked"]'),
57+
).toBeVisible({ timeout: 10000 });
6158

62-
// Should have either a "Perform Steps" button (interrupt) or Confirm/Reject buttons (HITL)
63-
const actionButton = stepSelector.locator("button").first();
64-
await expect(actionButton).toBeVisible();
59+
// Agent's follow-up confirmation arrives next.
60+
await expect(
61+
page
62+
.locator('[data-role="assistant"]')
63+
.filter({ hasText: /Booked.*Alice/i })
64+
.first(),
65+
).toBeVisible({ timeout: 30000 });
6566
});
6667

67-
test("can approve a step selection and agent continues", async ({ page }) => {
68+
// The other suggestion in the demo. Same HITL flow, different
69+
// attendee/topic. Pinned here so a missing/regressed aimock fixture for
70+
// this suggestion (or a wiring regression in this integration's
71+
// useHumanInTheLoop binding) fails CI loudly instead of silently falling
72+
// through to "the agent rambled at me with no toolCall."
73+
test("'Book a call with sales' suggestion runs the HITL flow end-to-end", async ({
74+
page,
75+
}) => {
6876
const input = page.getByPlaceholder("Type a message");
69-
await input.fill("Plan a trip to mars in 5 steps");
77+
await input.fill(
78+
"Please book an intro call with the sales team to discuss pricing.",
79+
);
7080
await input.press("Enter");
7181

72-
// Wait for step selector
73-
const selector = page.locator('[data-testid="select-steps"]');
74-
await expect(selector).toBeVisible({ timeout: 60000 });
82+
const card = page.locator('[data-testid="time-picker-card"]');
83+
await expect(card).toBeVisible({ timeout: 60000 });
84+
await expect(card.getByText(/Sales team/i)).toBeVisible();
7585

76-
// Find and click the action button (Perform Steps / Confirm)
77-
const actionBtn = page.locator(
78-
'button:has-text("Perform"), button:has-text("Confirm")',
79-
);
80-
await expect(actionBtn.first()).toBeVisible({ timeout: 5000 });
81-
await actionBtn.first().click();
82-
83-
// After approval, the step selector's action button should disappear
84-
// (StepSelector unmounts after onConfirm; StepsFeedback replaces buttons with
85-
// the "Accepted" or "Rejected" banner). Assert on a post-click transition
86-
// that does NOT match the pre-click "N/N selected" counter text.
87-
await expect(actionBtn.first()).not.toBeVisible({ timeout: 10000 });
86+
const slot = page.locator('[data-testid="time-picker-slot"]').first();
87+
await slot.click();
88+
89+
await expect(
90+
page.locator('[data-testid="time-picker-picked"]'),
91+
).toBeVisible({ timeout: 10000 });
92+
93+
await expect(
94+
page
95+
.locator('[data-role="assistant"]')
96+
.filter({ hasText: /Booked.*sales team/i })
97+
.first(),
98+
).toBeVisible({ timeout: 30000 });
8899
});
89100
});

0 commit comments

Comments
 (0)