Add interactive Plotly HTML result dashboards

README.md

@@ -17,7 +17,7 @@ flowchart LR
     A --> S["native Copilot\ncopilot-session/**/events.jsonl"]
     P --> O["jobs/<job>/<run-id>/\nPier result.json + trials"]
     S --> C["Copilot-native analysis\nAIU, tokens, tools, turns"]
-    O --> R["summary.json / summary.md\nshow / inspect / analyze"]
+    O --> R["summary.json / summary.md / summary.html\nshow / chart / inspect / analyze"]
 ```
 
 - **Tasks** are Harbor/Pier task directories: `task.toml`, `instruction.md`, `environment/`,
@@ -47,6 +47,7 @@ uv run copilot-experiments validate
 uv run copilot-experiments run
 uv run copilot-experiments list
 uv run copilot-experiments show --last
+uv run copilot-experiments chart --last --open
 uv run copilot-experiments analyze --last
 ```
 
@@ -106,6 +107,7 @@ uv run copilot-experiments analyze --root examples/tracer_bullet --last
 | `run --resume` | Resume an existing Pier job directory and skip already-completed matching trials. |
 | `list` | List Pier job configs and copyable run selectors. |
 | `show <selector>` / `show --last` | Print a summary for a Pier run (`job` or `job/run`). |
+| `chart <selector>` / `chart --last` | Write an interactive `summary.html` dashboard for a Pier run (`--open`, `--out`, `--cdn`). |
 | `inspect <selector>` | Drill into stored trials by `--agent`, `--task`, and `--trial`. |
 | `analyze <selector>` / `analyze --last` / `analyze --file <events.jsonl>` | Render a rich overview of a selected Copilot session log. |
 
@@ -116,6 +118,7 @@ uv run copilot-experiments analyze --root examples/tracer_bullet --last
 - [`docs/deepswe.md`](docs/deepswe.md) - importing and running DeepSWE tasks through Pier.
 - [`docs/collecting-run-data.md`](docs/collecting-run-data.md) - everything to collect around a Copilot CLI run, including native `events.jsonl`, Pier artifacts, ATIF, and OTel.
 - [`docs/results-format.md`](docs/results-format.md) - Pier `jobs/` layout and derived summaries.
+- [`docs/visualizing-results.md`](docs/visualizing-results.md) - interactive `summary.html` dashboards.
 - [`docs/analysis.md`](docs/analysis.md) - native Copilot session analysis.
 - [`docs/byok-and-local-models.md`](docs/byok-and-local-models.md) - provider env for Copilot CLI.
 - [`docs/adr/`](docs/adr) - architecture decision records.

docs/adr/0021-interactive-html-dashboards.md

@@ -0,0 +1,62 @@
+# 0021. Interactive self-contained HTML result dashboards
+
+- **Status:** Accepted
+- **Date:** 2026-06-20
+- **Deciders:** copilot-experiments maintainers
+
+## Context
+
+The tool could render results only as Rich terminal tables (`show`, `analyze`, `inspect`) and a
+plain `summary.md`. Users asked for high-quality, clean visual outputs of experiment results,
+citing the [DeepSWE leaderboard](https://deepswe.datacurve.ai/) and the
+[GitHub Copilot agentic-harness benchmarking report](https://github.blog/ai-and-ml/github-copilot/evaluating-performance-and-efficiency-of-the-github-copilot-agentic-harness-across-models-and-tasks/)
+as the target look and feel. Those references communicate three things well: task resolution per
+model, token/cost efficiency, and a resolution-vs-cost trade-off with run-to-run variance.
+
+Constraints from our existing ADRs shaped the options:
+
+- The filesystem is the source of truth ([0002](0002-filesystem-is-source-of-truth.md)); a
+  dashboard should be a derived artifact next to `summary.json`, not new authoritative state.
+- Analysis data is separate from rendering ([0006](0006-separate-analysis-data-from-rendering.md));
+  charts must consume the existing summary, not re-derive metrics.
+- We deliberately favored a CLI over a web app for analysis
+  ([0007](0007-cli-rich-analysis-before-web-app.md)); we did not want to (re)introduce a server.
+- Tests stay offline; any dependency must be exercisable without network or Docker.
+
+Candidate approaches: (a) static PNG/SVG via Matplotlib, (b) a served web app, (c) self-contained
+interactive HTML. Static images are easy to embed but not explorable and render poorly across
+screens. A served app reverses ADR 0007. Self-contained HTML keeps the "just files" model while
+giving crisp, interactive, shareable output.
+
+## Decision
+
+We will render each Pier run into a single self-contained `summary.html` dashboard built with
+[Plotly](https://plotly.com/python/), added as a new `charts.py` module.
+
+- Plotly is a **core** runtime dependency, so the `chart` command and the automatic `run`
+  dashboard work with a normal install (`uvx`, `uv tool install`, `pip install`) and no extra to
+  enable. The import is still guarded defensively: if Plotly is somehow unavailable, `run` skips
+  the dashboard and `chart` exits with a clear `ChartError` instead of a bare `ImportError`.
+- `charts.build_dashboard_html(summary)` consumes the existing `summary.json` shape and returns a
+  complete HTML string; `charts.write_dashboard(job_dir)` writes `summary.html` beside the other
+  derived summaries. No new source-of-truth state is introduced.
+- A new `chart` CLI command produces the dashboard on demand (`--last`, `--out`, `--cdn`,
+  `--open`), and `run` emits it automatically after each job.
+- By default plotly.js is embedded for zero-dependency offline viewing; `--cdn` produces a small
+  file that loads plotly.js from the CDN.
+- To feed the charts we populated the previously stubbed variance/coverage aggregates in
+  `pier_results.py` (`std_*`/`cv_*` for AIU and tokens, `mean_resolved_rate`,
+  `resolved_at_k_rate`, per-task `resolved_rate`).
+
+## Consequences
+
+- Runs now yield a crisp, interactive, shareable dashboard with no server, preserving the
+  filesystem-first, CLI-first model. It sits alongside `summary.json` / `summary.md` and is
+  regenerable at any time.
+- `summary.json` gained documented spread and suite-coverage fields, which also benefit `show`
+  and any future consumer; `report.py` key names were corrected to match.
+- We accept a large runtime dependency (Plotly bundles plotly.js). The embedded offline build is
+  ~4.5 MB per file; `--cdn` is available when size matters. Offline tests stay fast by using the
+  `cdn=True` path or monkeypatching the import to assert the defensive `ChartError`.
+- Future chart tweaks live entirely in `charts.py` and read only the summary dict, so rendering
+  stays decoupled from metric derivation per ADR 0006.

docs/results-format.md

@@ -18,6 +18,7 @@ jobs/
       copilot-experiments-run.json
       summary.json          # written by copilot-experiments
       summary.md            # written by copilot-experiments
+      summary.html          # written by copilot-experiments (chart / run)
       <trial-name>/
         config.json
         result.json
@@ -52,6 +53,7 @@ download. `copilot-experiments` adds `copilot-experiments-run.json` to preserve
 | `agent/copilot-otel.jsonl` | Copilot OTel file-exporter output, captured by default for Copilot agent runs unless custom OTel destination settings override it. Useful for per-LLM-call spans with input/output/cache-write/nano-AIU details. |
 | `verifier/reward.txt` / `.json` | Pier verifier reward. Positive reward means solved. |
 | `summary.json` / `summary.md` | Derived agent/task aggregate summary. |
+| `summary.html` | Interactive Plotly dashboard derived from `summary.json` (see [Visualizing results](visualizing-results.md)). |
 
 Pier jobs do not persist per-trial `metrics.json` or `analysis.json` files. Those views are
 derived from `agent/copilot-session/**/events.jsonl` (or `agent/trajectory.json` as a fallback)
@@ -69,6 +71,21 @@ when `show`, `analyze`, or `inspect` runs.
 - Copilot-native token/AIU/tool metrics when native events are available;
 - nullable fallback metrics for non-Copilot agents.
 
+### Variance and suite-coverage aggregates
+
+Each agent entry also carries spread and multi-trial coverage fields, used by the
+`summary.html` dashboard:
+
+| Field | Scope | Meaning |
+| --- | --- | --- |
+| `std_aiu` / `cv_aiu` | agent, task | Population standard deviation and coefficient of variation (`std / mean`) of AIU cost across trials. `null` when no AIU is available; `cv` is `null` when the mean is zero. |
+| `std_total_tokens` / `cv_total_tokens` | agent, task | Same spread measures for total token counts. |
+| `mean_resolved_rate` | agent | Macro-average of per-task `success_rate` (each task weighted equally, regardless of trial count). |
+| `resolved_at_k_rate` | agent | Fraction of tasks solved at least once across their `k` trials (each task's `resolved_rate` averaged). |
+| `resolved` / `resolved_rate` | task | Whether a task was solved at least once (`resolved_rate` is `1.0`/`0.0` per task) and, aggregated per agent, the resolved@k coverage. |
+
+Single-trial tasks report `std_* = 0.0` and `cv_* = 0.0`; empty inputs report `null`.
+
 ## Analyzing a trial
 
 ```bash

docs/visualizing-results.md

@@ -0,0 +1,77 @@
+# Visualizing results
+
+`copilot-experiments` renders each Pier run into a single, self-contained
+`summary.html` dashboard inspired by the [DeepSWE leaderboard](https://deepswe.datacurve.ai/)
+and the [GitHub Copilot agentic-harness benchmarking report](https://github.blog/ai-and-ml/github-copilot/evaluating-performance-and-efficiency-of-the-github-copilot-agentic-harness-across-models-and-tasks/).
+The dashboard is built with [Plotly](https://plotly.com/python/): the charts are interactive
+(hover, zoom, legend toggles) yet the file opens straight from disk with no server.
+
+## Producing a dashboard
+
+```bash
+# Chart the most recent run and open it in a browser.
+uv run copilot-experiments chart --last --open
+
+# Chart a specific run by selector (from `list`).
+uv run copilot-experiments chart <job-name>
+uv run copilot-experiments chart <job-name>/<run-id>
+
+# Write it somewhere else, or use the CDN build (see below).
+uv run copilot-experiments chart --last --out report.html
+uv run copilot-experiments chart --last --cdn
+```
+
+`run` also emits `summary.html` automatically after each Pier job, so a freshly finished
+experiment already has a shareable dashboard next to its `summary.json`.
+
+Plotly ships as a core dependency, so `chart` and the automatic `run` dashboard work out of the
+box with a normal install (`uv sync`, `uvx copilot-experiments`, `uv tool install`, or `pip
+install copilot-experiments`) -- there is no extra to enable.
+
+## What the dashboard shows
+
+The page is derived entirely from `summary.json` (see
+[Results format](results-format.md)), so it stays in sync with `show` and `inspect`:
+
+- **Header + KPI cards** - job identity, status, and headline numbers (agents, tasks, trials,
+  overall success rate, total AIU).
+- **Leaderboard** - agents ranked by success rate, then by cost, with per-agent color that is
+  reused across every chart.
+- **Task resolution** - success rate per agent; grouped by task when the run has more than one
+  task. Higher is better.
+- **Resolution vs. cost** - success rate (y) against average cost per task (x) with ±1σ cost
+  error bars. "Up and to the left" (high success, low cost) is best.
+- **Cost efficiency** - average cost per task per agent with ±1σ spread. Lower is better.
+- **Per-task success** - an agent × task heatmap, shown only when the run has more than one task.
+
+Cost prefers AIU (`avg_aiu` / `std_aiu`) and falls back to token counts
+(`avg_total_tokens` / `std_total_tokens`) when no AIU is available, matching the aggregate fields
+documented in [Results format](results-format.md#variance-and-suite-coverage-aggregates).
+
+## Offline vs. CDN builds
+
+By default the dashboard **embeds** the full plotly.js bundle (~4.5 MB) so it renders with zero
+network access - ideal for archiving a run or emailing it to a colleague. Pass `--cdn` to instead
+reference plotly.js from `https://cdn.plot.ly/`, producing a small file (tens of KB) that needs an
+internet connection to render.
+
+| Build | Flag | File size | Needs network to view |
+| --- | --- | --- | --- |
+| Offline (default) | *(none)* | ~4.5 MB | No |
+| CDN | `--cdn` | ~40 KB | Yes |
+
+## Programmatic use
+
+The dashboard builder is a small public API:
+
+```python
+from copilot_experiments.charts import build_dashboard_html, write_dashboard, plotly_available
+from copilot_experiments.pier_results import build_pier_summary
+
+summary = build_pier_summary(job_dir)
+html = build_dashboard_html(summary, cdn=True)     # returns a complete HTML string
+write_dashboard(job_dir, summary=summary)          # writes <job_dir>/summary.html
+```
+
+`build_dashboard_html` and `write_dashboard` raise `charts.ChartError` (with an install hint) when
+Plotly is unavailable; guard optional call sites with `plotly_available()`.

pyproject.toml

@@ -13,6 +13,8 @@ dependencies = [
     "pyyaml>=6.0",
     "typer>=0.12",
     "rich>=13.7",
+    # Renders the interactive, self-contained HTML result dashboards (`chart` / `run`).
+    "plotly>=5.20",
 ]
 
 [project.scripts]