spike: Design variant + annotation API shape for score set and variant pages

[bencap]

Context

Blocks: feat: Implement CatVRS transit layer on variant endpoints #744

Variant data transfer is the main performance bottleneck on score set pages. The new data model (AssayedVariant → MappingRecord → Allele) provides richer variant data at all levels (genomic, coding, protein), which makes API shape decisions more consequential.

The API needs to serve two contexts efficiently:

1. Score set page: paginated sparse list for table/chart rendering, with on-demand expansion for inline variant-level visualizations
2. Variant page: full rich single-variant response including alleles at all levels and CatVRS categorical representation for protein-level variants

The design decision produced here is the contract that the CatVRS implementation (#744) builds against.

Data model traversal path

For a given AssayedVariant:

• AssayedVariant → MappingRecord (current) → mapping_record_alleles → Allele rows at applicable levels (filtered by level column)
• For protein-level variants (MappingRecord.assay_level = protein): the associated Allele rows where level = 'coding' are the categorical members for CatVRS
• Annotation data is on per-type annotation tables (VEPAnnotation, GnomADVariant, ClinicalControl) keyed by allele_id

Goal

Produce an ADR or decision comment specifying the concrete API contract for variant data endpoints.

Questions to Resolve

• What fields constitute the minimum viable sparse payload for score set list rendering?
• What is the expansion mechanism for full variant data within the score set page context? (query param, field mask, sub-resource?)
• Does CatVRS expansion live on GET /variants/{id} directly, or a sub-resource?
• How does annotation data surface in responses — inline or by reference?
• What are the acceptable response time targets for each endpoint context?
• How does the response handle variants with alleles at multiple levels (e.g. all three levels for protein-level score sets)?

[bencap]

Spike outcome — variant + annotation API contract

This comment is the decision record for #743: the concrete contract for the score-set/variant pages and the programmatic surface. It deliberately splits into a settled spine and a small set of explicitly-deferred items to be co-designed during the parallel UI + API rework — calling those out is part of the result, not a gap. ((D##) are decision tags carried in the fuller design notes.)

The shape (settled)

• Two kinds of variant, joined by Cat-VRS (D6). The assayed variant (GET /variants/{urn}, MaveDB-native, citable, version-pinned) and the mapped allele (GET /alleles/{digest}, VRS-native, deduplicated, assay-agnostic). The old conflated /mapped-variants is retired. Cat-VRS is the seam: it's anchored on the assayed side and points (by digest) into the mapped substrate — crossing is a pointer walk, not a copy.
• One lean whole-set view backs the score-set page (D63). GET /score-sets/{urn}/variants returns a compact, pre-chewed, served-whole dataset that the table, heatmap, and baseline (score) histogram all compose from — no separate "viz lane." variant_urn is the universal selection/tooltip key; assay_level_digest is the annotation-join bridge. Cursor pagination moves to the programmatic/bulk lane.
• Cat-VRS is spec-pure and uses a star relation model (D54). molecularRepresentation is a clean GA4GH CategoricalVariant; members are every representation, by digest; relations (coordinate_representation_of / encodes / translation_of) are member → defining, carrying score semantics only. Coordinate level is carried by level + ClinGen id, not relations. Materialized as a JSONB skeleton on MappingRecord (D33).
• Annotations ride embedded, keyed by digest (D44–D46). No dedicated annotation endpoint yet; divergent per-member values are never collapsed to one categorical value for a protein variant (D42).
• Temporal as_of is resource-level (D59/D60), one predicate / two modes. Supported on the single content resources (/variants/{urn}, /alleles/{digest}) — reconstruct mode for those, select mode for the immutable VA statements (§8.1). The score-set whole-set view is current-only.
• Supersession is version selection, not a temporal query (D69). Default = current tail; superseded versions are reachable only by explicit selection (their own URN), flagged superseded_by. as_of and include_superseded are distinct, non-conflatable knobs (D73); the per-version URN makes "as_of across a supersession boundary" structurally unrepresentable.

Answers to this issue's questions

• Minimum sparse payload for list rendering → the lean whole-set record: variant_urn, score, a representative (measured-level) consequence, clingen_allele_id, assay_level_digest, and pre-parsed position/ref/alt at both nt and protein levels (drives the DNA/Protein heatmap toggle client-side; D70). Server-parsed — the client stops parsing/translating HGVS.
• Expansion mechanism for full variant data → a query-param-selected variant detail panel on the score-set page today, backed by GET /variants/{urn} (the assayed variant-detail resource). (A dedicated assayed-variant page is proposed — D75.)
• CatVRS expansion: on /variants/{urn} or a sub-resource? → directly on GET /variants/{urn} as the spec-pure molecularRepresentation, assembled on demand from the materialized skeleton. Optional lightweight GET /variants/{urn}/mapping[?as_of=] projection exists for the mapping/Cat-VRS skeleton alone.
• Annotations inline or by reference? → inline/embedded, keyed by VRS digest, joined to the Cat-VRS members by digest (D46). Allele resources are content-addressed and CDN-cacheable (D35).
• Response-time targets → the lean whole-set view (heaviest page load) ≤ 5–10s for the largest score sets (~648k), ~2–4 MB gzipped columnar. Single variant/allele resources bounded by construction. Validated by the O4 sizing experiment (below).
• Variants with alleles at multiple levels → all levels are Cat-VRS members (flat, D48); the lean record carries both nt and protein parsed forms; per-codon {coding, genomic} group by shared ClinGen CA, protein by its own PA.

Explicitly deferred — co-designed in the parallel UI + API rework (D71)

• Annotation-histogram grain/divergence semantics (D63). The effect histogram is variant-grained and solved by the lean record. The clinical-control (ClinVar) and gnomAD histograms are allele-grained and divergent; the choice (representative pick / multi-membership / allele-grained bins) is a product decision, not an API blocker (D72) — the API serves the grain-agnostic primitive (digest-keyed per-allele values + member digests) and the grain is a client-side reduction. ClinVar is already a separate fetch today; VEP follows suit as a secondary route.
• A dedicated assayed-variant page (D75) — the missing symmetric half of the assayed/mapped split (a citable, version-pinned landing page for urn:…#42 carrying the supersession banner). API resource exists; the page is net-new UI.

Defects this design closes

• Superseded data served by default. Direct-by-URN access to a superseded score set's variants/scores/counts/mapped-variants is currently unfiltered — the supersession filter lives only in the search query. The VA-Spec exports have the identical confirmed gap (annotated-variants/* in routers/score_sets.py, /va/* in routers/mapped_variant.py). The selection model (D69) + self-describing superseded_by pointer closes both. See Variant search results from superseded score sets #776.