Add interactive Plotly HTML result dashboards#28
Merged
Conversation
Render each Pier run into a self-contained summary.html dashboard with KPI cards, a ranked leaderboard, and four interactive charts, inspired by the DeepSWE leaderboard and the Copilot agentic-harness report. Good clean, shareable rendering was the goal, so the file opens straight from disk with no server. - New charts.py module (build_dashboard_html / write_dashboard / plotly_available) behind an optional [charts] extra; degrades gracefully with an install hint when Plotly is absent. - New `chart` CLI command (--last/--out/--cdn/--open); `run` now emits summary.html automatically when Plotly is installed. - Populate the previously stubbed variance/coverage aggregates in pier_results.py (std/cv for AIU and tokens, mean_resolved_rate, resolved_at_k_rate, per-task resolved_rate) and fix a CV key mismatch in report.py. - Offline tests (test_charts.py plus aggregate and CLI tests), docs (visualizing-results.md, results-format.md, ADR 0021), and scaffold template updates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The interactive HTML dashboards were gated behind a [charts] optional extra, so common install flows (uvx copilot-experiments, uv tool install --editable .) shipped without Plotly and failed the chart command with an install hint. Move Plotly into the core dependencies and drop the extra; the import stays guarded defensively so a broken install still raises a clear ChartError instead of ImportError. Also update the scaffold template, docs, and ADR 0021 to describe Plotly as a required dependency. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When every agent scored 100 percent, all points landed on the top line and their names (rendered with textposition top center) were cut off by the plot's top edge, plus the outermost labels overflowed the left/right edges. Add headroom above 100 percent, pad the x-range, set cliponaxis=False, and widen the chart's side margins so the labels are always fully visible. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
On-point text labels in the resolution-vs-cost scatter overlapped horizontally when two agents landed at similar costs, because Plotly has no collision avoidance for scatter text. Switch the trace to markers-only and identify agents through the horizontal legend, which wraps cleanly at any width. Drop the redundant on-plot '_ better' annotation (the panel subtitle already says 'up and to the left is better') so it no longer collides with the best point, and relax the now-unneeded x-axis label inset padding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Until now the tool could only show results as Rich terminal tables and a plain
summary.md. There was no crisp, shareable visual output. This adds high-quality interactive dashboards inspired by the DeepSWE leaderboard and the GitHub Copilot agentic-harness benchmarking report, so a finished run can be understood at a glance and passed around without a server.What
Each Pier run now renders into a single self-contained
summary.html: a clean header, KPI cards, a ranked leaderboard, and four interactive Plotly charts:Usage:
runalso emitssummary.htmlautomatically whenever Plotly is installed.Approach and design notes
charts.pymodule is the whole rendering surface (build_dashboard_html,write_dashboard,plotly_available,ChartError). It consumes the existingsummary.jsonshape rather than re-deriving metrics, keeping analysis data separate from rendering.[charts]extra. The dashboard degrades gracefully:runsilently skips it when Plotly is missing, andchartexits with an install hint. Core behavior and all offline tests never require Plotly.--cdnproduces a ~40 KB file that loads plotly.js from the CDN.pier_results.py(std_*/cv_*for AIU and tokens,mean_resolved_rate,resolved_at_k_rate, per-taskresolved_rate) and fixed a CV key mismatch inreport.py. These fields also benefitshowand are documented inresults-format.md.Testing
tests/test_charts.pyplus aggregate and CLI tests. Chart tests use thecdn=Truepath (fast) or monkeypatch the import to assert graceful degradation, so they stay offline.test_cli_validate_checks_pier_configfailure is environment-specific (it assumes Docker is absent, but Docker is installed on this machine) and I confirmed it fails identically without this change.Notes for reviewers
docs/visualizing-results.md,docs/results-format.md, README, and ADR0021-interactive-html-dashboards.md.[charts]extra sochartworks out of the box in generated experiment repos.