Skip to content

Commit 9845dad

Browse files
AlemTuzlakAlemTuzlak
committed
fix(aimock): re-key HITL confirmations on toolCallId so back-to-back flows work
Bug: in a single chat session, running both HITL booking flows back-to-back (Alice 1:1 → then Sales call without refresh) used to skip the time-picker on the second flow and jump straight to "Booked ..." text. Cause: confirmation fixtures were matched on `hasToolResult: true`, which fires whenever the conversation has ANY tool message in history. After the first flow finished, the second user message short-circuited to a confirmation match before the second flow's toolCall fixture (gated on `hasToolResult: false`) had a chance to fire. The picker never rendered. Fix: re-key the two confirmation fixtures on `toolCallId` (the specific tool_call_id of the matching `book_call` invocation), which only fires when the LAST conversation message is a tool result with that id — exactly the moment we want the confirmation. Drop the `hasToolResult: false` constraint on the toolCall fixtures so they match a fresh user request regardless of prior tool history. Add a back-to-back regression test to all 17 hitl-in-chat specs: walk Alice flow to completion, then sales flow without refresh, assert two `time-picker-card` elements rendered. If the multi-flow regression returns, the second card never appears and the test fails at `toHaveCount(2)`.
1 parent 8cb84e8 commit 9845dad

18 files changed

Lines changed: 896 additions & 16 deletions

File tree

‎showcase/aimock/feature-parity.json‎

Lines changed: 12 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -653,8 +653,15 @@
653653
},
654654
{
655655
"match": {
656-
"userMessage": "Please book an intro call with the sales team to discuss pricing.",
657-
"hasToolResult": false
656+
"toolCallId": "call_hitl_book_intro_sales_001"
657+
},
658+
"response": {
659+
"content": "Booked the intro call with the sales team for the time you selected — calendar invite is on its way."
660+
}
661+
},
662+
{
663+
"match": {
664+
"userMessage": "Please book an intro call with the sales team to discuss pricing."
658665
},
659666
"response": {
660667
"toolCalls": [
@@ -668,17 +675,15 @@
668675
},
669676
{
670677
"match": {
671-
"userMessage": "Please book an intro call with the sales team to discuss pricing.",
672-
"hasToolResult": true
678+
"toolCallId": "call_hitl_book_1on1_alice_001"
673679
},
674680
"response": {
675-
"content": "Booked the intro call with the sales team for the time you selected — calendar invite is on its way."
681+
"content": "Booked the 1:1 with Alice for the time you selected — calendar invite is on its way."
676682
}
677683
},
678684
{
679685
"match": {
680-
"userMessage": "Schedule a 1:1 with Alice next week to review Q2 goals.",
681-
"hasToolResult": false
686+
"userMessage": "Schedule a 1:1 with Alice next week to review Q2 goals."
682687
},
683688
"response": {
684689
"toolCalls": [
@@ -690,15 +695,6 @@
690695
]
691696
}
692697
},
693-
{
694-
"match": {
695-
"userMessage": "Schedule a 1:1 with Alice next week to review Q2 goals.",
696-
"hasToolResult": true
697-
},
698-
"response": {
699-
"content": "Booked the 1:1 with Alice for the time you selected — calendar invite is on its way."
700-
}
701-
},
702698
{
703699
"match": {
704700
"userMessage": "alice"

‎showcase/integrations/ag2/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

‎showcase/integrations/agno/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

‎showcase/integrations/claude-sdk-python/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

‎showcase/integrations/claude-sdk-typescript/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

‎showcase/integrations/crewai-crews/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

‎showcase/integrations/google-adk/tests/e2e/hitl-in-chat.spec.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,56 @@ test.describe("HITL in chat — booking flow", () => {
9797
.first(),
9898
).toBeVisible({ timeout: 30000 });
9999
});
100+
101+
// Regression: the second booking flow in a single chat session used to
102+
// skip the picker entirely and jump straight to "Booked ..." text. Cause
103+
// was the aimock confirmation fixture being keyed on `hasToolResult: true`
104+
// — once the first flow finished, the conversation had a tool message in
105+
// history, so on the second user message the confirmation fixture matched
106+
// before the toolCall fixture (which required `hasToolResult: false`) had
107+
// a chance to fire. Fix re-keyed confirmation fixtures on `toolCallId`
108+
// (matches only when the LAST message is a tool result with that id) and
109+
// dropped the `hasToolResult: false` constraint on the toolCall fixtures.
110+
// This test explicitly walks both flows in one page session — if the
111+
// multi-flow regression returns, the second time-picker never renders and
112+
// this test fails at step 2.
113+
test("both suggestions render the picker when run back-to-back without refresh", async ({
114+
page,
115+
}) => {
116+
const input = page.getByPlaceholder("Type a message");
117+
const card = page.locator('[data-testid="time-picker-card"]');
118+
119+
// Flow 1: Alice
120+
await input.fill("Schedule a 1:1 with Alice next week to review Q2 goals.");
121+
await input.press("Enter");
122+
await expect(card.first()).toBeVisible({ timeout: 60000 });
123+
await page.locator('[data-testid="time-picker-slot"]').first().click();
124+
await expect(
125+
page
126+
.locator('[data-role="assistant"]')
127+
.filter({ hasText: /Booked.*Alice/i })
128+
.first(),
129+
).toBeVisible({ timeout: 30000 });
130+
131+
// Flow 2: sales — same page, no refresh.
132+
await input.fill(
133+
"Please book an intro call with the sales team to discuss pricing.",
134+
);
135+
await input.press("Enter");
136+
137+
// A SECOND picker card must appear. If the regression returns, no new
138+
// card renders and the agent jumps straight to confirmation text.
139+
await expect(card).toHaveCount(2, { timeout: 60000 });
140+
await expect(card.last().getByText(/Sales team/i)).toBeVisible();
141+
142+
// Pick a slot in the new card and verify the sales-specific
143+
// confirmation arrives.
144+
await page.locator('[data-testid="time-picker-slot"]').last().click();
145+
await expect(
146+
page
147+
.locator('[data-role="assistant"]')
148+
.filter({ hasText: /Booked.*sales team/i })
149+
.first(),
150+
).toBeVisible({ timeout: 30000 });
151+
});
100152
});

0 commit comments

Comments
 (0)