Skip to content

test: Add test plans for different Copilot subscription tiers focusing on quota usage. #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ethanyhou merged 3 commits into from
May 19, 2026
Merged

test: Add test plans for different Copilot subscription tiers focusing on quota usage. #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3f04f8a
Add test plans for different Copilot subscription tiers focusing on q…
ethanyhou May 18, 2026
d6575bb
Address comments.
ethanyhou May 19, 2026
fee7d4f
Address comments.
ethanyhou May 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions test-plans/token-based-billing/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Copilot Plan / Quota UI Test Plans

## Overview
Test plans for the plan- and quota-aware UI surfaces in GitHub Copilot for
Eclipse. One file per Copilot plan; each plan exercises the same three
surfaces so cross-plan differences stay easy to compare.

All accounts under test are on token-based billing.

## Surfaces Under Test
1. **Status-bar usage menu** — opened by clicking the Copilot icon in the
Eclipse status bar.
2. **Menu-bar usage menu** — opened from `GitHub Copilot` in the Eclipse menu
bar. Must be visually identical to the status-bar menu.
3. **Quota-warning surfaces in the chat view**
- **Static banner** shown above the chat input area when the user crosses
one of the chat-view quota-usage thresholds (see *Quota-Warning
Thresholds* below).
- **Inline warning widget** shown under a chat turn when a chat request
fails because the quota is exhausted.

The chat-view warning surfaces are driven only by the quotas that affect
chat usage: the **Chat Messages** quota on the Free plan and the
**AI-credit (premium interactions)** quota on every paid plan. The Code
Completions quota does not trigger banners or inline warnings in the chat
view, even though it is shown in the status-bar / menu-bar menus.

## Quota-Warning Thresholds
The static banner is pushed by the language server when the signed-in
account's tracked usage crosses one of these two thresholds:

| Usage crossed | Banner severity / icon |
|---------------|------------------------|
| 75% | Info icon |
| 90% | Warning icon |

Each threshold fires at most once per quota reset period. To re-trigger a
banner for the same threshold, the quota usage on the GitHub quota portal
must first be lowered back below the threshold (or the reset date must pass)
so the language server re-arms the notification.

Per-plan cases below are ordered from **rich quota to poor quota**: usage
**under 75%** (no banner), usage **above 75%** (info banner), and usage
**above 90%** (warning banner). To exercise a specific threshold set usage
on the quota portal a few percentage points **above** the target threshold
(for example, **~76%** to cross 75% and **~91%** to cross 90%) and then send
a chat message.

## Plan → File Map
| Copilot plan | Plan label shown in the menu | File |
|--------------------|-----------------------------------|----------------------------|
| Free | Copilot Free Plan | `TBB-test-free.md` |
| Pro | Copilot Pro Plan | `TBB-test-pro.md` |
| Pro+ | Copilot Pro+ Plan | `TBB-test-pro-plus.md` |
| Max | Copilot Max Plan | `TBB-test-max.md` |
| Business | Business Plan | `TBB-test-business.md` |
| Enterprise | Enterprise Plan | `TBB-test-enterprise.md` |

## Common Preconditions
- Eclipse is running with the GitHub Copilot for Eclipse plugin installed.
- The tester has access to the GitHub quota portal at
`https://github.com/github-copilot/quotas/<accountId>` to adjust usage on the
signed-in account. The menus refresh in real time after a quota change; no
Eclipse restart is required.
- The Copilot chat view is open in Agent mode for chat-related cases.
- Each plan starts with a sign-in case (`TC-<plan>-000`) using an account on
that plan with token-based billing enabled.

## Quota Action Summary
This table describes which overage action the chat-view warning surfaces are
expected to show for each plan.

| Plan | Overage action |
|--------------|-------------------------------------------------------|
| Free | — |
| Pro | Enable Additional Usage / Increase Budget (primary) |
| Pro+ | Enable Additional Usage / Increase Budget (primary) |
| Max | Enable Additional Usage / Increase Budget (primary) |
| Business | — |
| Enterprise | — |

Notes:
- The label switches from "Enable Additional Usage" to "Increase Budget" once
additional paid usage has been enabled for the user.
- The **Upgrade Plan** action (row in the menu, link in the static banner,
button in the inline warning) is **not** plan-driven. It must appear if and
only if the language server reports `canUpgradePlan = true` for the
signed-in account; otherwise it must be absent from every surface. Each
per-plan file starts with a step that records the actual `canUpgradePlan`
value from the language-server log, and all later cases reference that
recorded value.

## How to Read `canUpgradePlan` From the Language-Server Log
1. In Eclipse, open `Window → Preferences → Language Servers`.
2. Tick **Log to console** (and optionally **Log to file**) for the GitHub
Copilot language server, then `Apply and Close`.
3. Open `Window → Show View → Other... → General → Console`.
4. From the Console toolbar dropdown (the open-console icon), choose the
`Eclipse Copilot Language Servers Log` console entry that traces traffic for the GitHub
Copilot language server.
5. Sign in to GitHub (or sign out and back in) so a fresh handshake is logged.
6. Locate the most recent `copilot/quotaChange` notification (sent from the
server). Note the value of the `canUpgradePlan` field in its payload — it
is `true`, `false`, or absent.
- `true` → expect Upgrade Plan everywhere it can appear.
- `false` or absent → expect Upgrade Plan to **not** appear.
155 changes: 155 additions & 0 deletions test-plans/token-based-billing/TBB-test-business.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# Copilot Business Plan (Unlimited Org) — Quota UI

## Overview
This file covers the **Business plan with unlimited premium-interaction
usage** configured by the organization. Under this configuration the usage
menus replace the Monthly Limit row with a single informational message, and
the chat-view quota-warning surfaces stay dormant because there is no
monthly cap to cross.

### Surface-by-surface expectations (unlimited org)

| Surface | Expected behaviour |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Plan label in menu header | `Business Plan` |
| Header row icon | Blue (full) usage icon |
| Monthly Limit row | **Hidden.** Replaced by a disabled message: `You have no monthly limit on AI credits usage set by your organization`. |
| Allowance-reset row | **Hidden.** There is no monthly allowance to reset. |
| Additional-usage status row | **Hidden.** |
| Enable Additional Usage / Increase Budget row | **Hidden.** |
| `Upgrade Plan` row | Present **iff** the language server reports `canUpgradePlan = true` for the signed-in account. Typically `false`. |
| Static banner (chat view) | Not triggered. No 75% / 90% threshold to cross. |
| Inline quota warning under a chat turn | Not triggered by quota exhaustion (no cap). |

### Why the chat-view warning surfaces are not exercised here
With unlimited premium-interaction usage there is no allowance to cross, so
neither the 75% info banner nor the 90% warning banner can fire, and the
quota-exhausted inline warning is unreachable. Banner / inline cases live in
the bounded-org and CFI plans. Any appearance of those surfaces on an
unlimited Business account is a regression.

---

## Test Cases

### TC-CB-U-000: Sign in with an unlimited Business org account, record `canUpgradePlan`

**Type:** `Happy Path`
**Priority:** `P0`

#### Preconditions
- Plugin is installed and Eclipse is signed out.
- A GitHub account that belongs to a **Copilot Business organization
configured with unlimited premium-interaction usage** is available.
- Language-server logging is enabled and the Eclipse Copilot Language Servers
Log console is open (see the README section *How to Read `canUpgradePlan`
From the Language-Server Log*).

#### Steps
1. Open the Copilot status-bar menu and click `Sign in to GitHub`.
2. Complete the device-flow sign-in with the unlimited Business account.
3. Wait for the status-bar icon to switch to the signed-in icon.
4. In the Eclipse Copilot Language Servers Log console, find the most recent
`copilot/quotaChange` notification and:
- **Record `canUpgradePlan`** (`true`, `false`, or absent). Later cases
refer to this as "the recorded `canUpgradePlan` value". For a typical
Business account this is `false`.
- Confirm the payload indicates that the premium-interactions quota is
unlimited. If it is not, the account is not actually on an unlimited
org and the test should be re-run on a correctly provisioned account.

#### Expected Result
- The header row of both menus reads `<username> — Business Plan`.
- The `canUpgradePlan` value and the "unlimited" indication have been
recorded.

#### 📸 Key Screenshots
- [ ] Signed-in state with the `Business Plan` label
- [ ] Language-server log showing `canUpgradePlan` and the unlimited flag

---

### TC-CB-U-001: Status-bar menu — unlimited org informational message

**Type:** `Happy Path`
**Priority:** `P0`

#### Preconditions
- TC-CB-U-000 completed; the signed-in user is on an unlimited Business org.

#### Steps
1. Click the Copilot icon in the Eclipse status bar.

#### Expected Result
1. Header row: `<username> — Business Plan` on top; second row reads
`Copilot Usage` with the **blue/full usage icon**.
2. A **single disabled message row** reading exactly:
`You have no monthly limit on AI credits usage set by your organization`.
3. **No** Monthly Limit row.
4. **No** allowance-reset row (no `Resets today` / `Reset in N days …` text).
5. **No** `Additional usage enabled` / `Additional usage not enabled` status
row, and **no** tooltip on such a row.
6. **No** `Enable Additional Usage` / `Increase Budget` action row.
7. `Upgrade Plan` row — present **iff** the recorded `canUpgradePlan` value
(TC-CB-U-000) is `true`; absent otherwise. For a typical Business account
it must not appear.
8. Standard footer rows (Sign Out, Preferences, etc.) appear as usual below
the usage section.

#### 📸 Key Screenshots
- [ ] Status-bar menu, unlimited Business org (full menu)

---

### TC-CB-U-002: Menu-bar menu mirrors the status-bar menu

**Type:** `Happy Path`
**Priority:** `P0`

#### Steps
1. Open `GitHub Copilot` from the Eclipse menu bar.
2. Compare against the menu captured in TC-CB-U-001.

#### Expected Result
The menu-bar menu shows the same rows in the same order with the same labels,
tooltips, and icons as the status-bar menu. The unlimited-org informational
message must be identical between the two surfaces.

#### 📸 Key Screenshots
- [ ] Menu-bar menu, unlimited Business org

---

### TC-CB-U-003: Chat works without any quota banner or inline warning

**Type:** `Happy Path`
**Priority:** `P0`

#### Preconditions
- TC-CB-U-001 succeeded.
- Copilot chat view is open in Agent mode.

#### Steps
1. Send a normal chat message (e.g. `Hello, what can you do?`) and wait for
the response.
2. Send a second chat message that triggers tool use (e.g. ask to read a file
in the workspace).
3. Inspect the chat view above the input area and under each completed turn.

#### Expected Result
- Both turns complete successfully.
- **No** static banner appears above the chat input (no info-icon banner, no
warning-icon banner, no `Upgrade Plan` link).
- **No** inline warning appears under either turn (no quota-exhausted
message, no `Upgrade Plan` button, no `Enable Additional Usage` button).

#### 📸 Key Screenshots
- [ ] Chat view with two completed turns, no banner and no inline warning

---

## Screenshots Checklist
- [ ] `TC-CB-U-000` Signed-in `Business Plan` label + language-server log payload
- [ ] `TC-CB-U-001` Status-bar menu, unlimited org informational message
- [ ] `TC-CB-U-002` Menu-bar menu mirrors status-bar menu
- [ ] `TC-CB-U-003` Chat view with no banner / no inline warning
Loading
Loading