[v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools)

[ayhammouda]

Context

Parent: #63. Methodology: docs/benchmarks/PUBLIC-BENCHMARK-METHODOLOGY.md (work package 4, real-adapter half — #72 delivered the manifest format; #85 delivered LLM-provider adapters, which are a different axis). Eligibility rules and candidate set (Context7, GitMCP, DeepWiki, Ref.tools) live in the methodology's "Systems Under Test" section.

Status: NOT agent-ready. Filed by the orchestrator per PLAN.md T6(b); Vision must review, pin the competitor versions/endpoints, create the .planning/agent-context/ file (decision 5.14, mirrored as an issue comment), and apply agent-ready before any dispatch.

Goal

Add competitor MCP tool adapters — one per eligible docs MCP — with pinned versions, reproducible invocation, and failure capture, so the maintainer-run live phase can execute every eligible tool cell.

Acceptance criteria (draft — Vision edits before agent-ready)

• One adapter per competitor in benchmarks/adapters/, each recording: pinned version/package/endpoint/commit, invocation transcript, latency, failure category (tool_failure/timeout/mcp_protocol_crash).
• Eligibility screening is encoded: an adapter whose competitor cannot be pinned or reproducibly invoked reports a structured exclusion (methodology honesty rule: "Do not drop failed systems silently"), never a silent skip.
• All unit tests are mock-transport only — zero network calls in CI. Live invocation sits behind the same require_live_environment() guard as [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #85's provider adapters (network at benchmark-runtime is maintainer-controlled live-phase only; the server's own offline-first principle 2.2 is untouched).
• Competitor terms-of-use check is recorded per adapter in the manifest metadata (methodology eligibility rule).
• No README/benchmark claims; Refs #63, never Closes #63.

Scope boundaries

In scope: adapter code, pinning metadata, mocked tests, exclusion reporting. Out of scope: running the live benchmark, scoring, reporting, corpus, README.

Forbidden-territory reminder

Standard §2 list applies; merged tests additive-only; any new third-party runtime dependency is a supervisor-review trigger (justify vs stdlib/httpx).

Validation commands

uv run ruff check src/ tests/
uv run ruff check benchmarks/
uv run pyright src/
uv run pyright benchmarks/
uv run pytest --tb=short -q
uv run pytest tests/benchmarks -q
uv run python-docs-mcp-server doctor

Recovery

If a competitor cannot be made reproducible (no pinnable version, terms forbid benchmarking), stop and comment here with the evidence — exclusion is a maintainer call.

Effort estimate

6–10 hours (four adapters).

Refs #63.

[ayhammouda]

Approved implementation plan + Stage 3 gate record (2026-07-09)

Auditable mirror of PLAN-87-competitor-adapters.md (local, untracked). Codex adversarial review: round 1 no-sign-off (1 blocking, triaged + fixed), round 2 sign-off. Maintainer confirmed A1–A6 as recommended (see the plan's Stage 3 gate record below).

---

PLAN-87 — Competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools)

Produced: 2026-07-09 (plan-review-ship pipeline; Stage 1 output). Planned against main @ 5a518de.
Issue: #87 [v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools) (Refs #63).
Status: awaiting Stage 2 (Codex adversarial review) and the Stage 3 gate (maintainer confirms assumptions A1–A6 = the pre-flight pin/terms confirmation required by the run rules; then context refresh + agent-ready).
Research basis: 5-agent fan-out on 2026-07-09 — four competitor dossiers (live-probed endpoints, npm/registry metadata, ToS text read in full) + repo-seam recon against merged #86/#98/#101 code. Every load-bearing claim below carries its source.

---

0. Scope and hard boundaries

Deliver exactly #87: four adapter modules + guard extensions + dispatch registration + manifest metadata convention + mock-transport tests + a committed template manifest. Nothing else.

• No live network calls anywhere in this change or its tests. Live invocation stays behind the guard; CI never passes it.
• No new dependencies. mcp 1.28.1 is an installed project dependency (uv.lock:297-302) whose client API covers stdio + streamable-HTTP + SSE (verified against .venv: mcp.client.stdio, mcp.client.streamable_http.streamablehttp_client, mcp.client.sse.sse_client; httpx 0.28.1 installed transitively). pyproject.toml/uv.lock must be absent from the diff. Websocket transport is NOT available (extra not installed) — no competitor needs it.
• No merged-test edits. No D6-style sanction exists for this task and none is needed (verified: the guard's unknown-provider test now asserts on "not-a-registered-provider", test_adapters.py:151-152 — registering new providers does not touch it).
• No report.py / model_matrix.py / scoring.py changes. validate_manifest_against_matrix already skips competitor entries that don't declare provider+model (model_matrix.py:227-233) — competitor entries pass through untouched.
• runner.py changes limited to the documented extension seam: one lazily-importing dispatch function + one _ADAPTER_DISPATCH entry per adapter (runner.py:434-444 states "Adding a new dispatchable adapter means adding one entry here").
• Worker prohibitions from PLAN.md (WORKER-PROHIBITIONS block) apply verbatim.

1. Per-competitor dossier summary and design decisions

1.1 Context7 (Upstash)

• Pin (strongest of the four): npm @upstash/context7-mcp@3.2.3 (published 2026-07-06), run as stdio subprocess npx -y @upstash/context7-mcp@3.2.3 (Node ≥18 — live-phase env requirement only, never in tests). Client pins exactly; doc index is hosted and unpinnable → effective pin = 3.2.3 + run date. Docker mcp/context7 is 13 months stale — do not use.
• Transport: stdio (default) via mcp.client.stdio; remote streamable-HTTP https://mcp.context7.com/mcp exists as fallback.
• Auth: optional CONTEXT7_API_KEY (free tier 1,000 calls/mo with account; keyless = lower unspecified limits).
• Tool flow (verified live 2026-07-09): two tools, resolve-library-id + query-docs. Python stdlib coverage confirmed: /python/cpython (48k snippets, versions incl. v3.13.9). Adapter flow: resolve-library-id(libraryName="Python", query=<question>) → query-docs(libraryId=<top id>, query=<question>). Two calls/question, exercising the tool's own routing (methodology "same question to every system"). Library ID hardcoding is a fallback knob, not the default (that would be per-competitor tuning).
• Terms (read in full): client MIT. Upstash ToS §C.8 (April 2025): "You must not perform any benchmark tests or analyses relating to Upstash's Website or Services without express permission of Upstash." The Context7 Addendum (Aug 2025) licenses "evaluation purposes" and sanctions automated MCP access but contains no benchmark clause — so §C.8 stands. Publishing comparative results is forbidden without express permission.
• Decision: build the adapter fully (mock-tested; the client is MIT and evaluation use is licensed). Manifest metadata records terms_check: {verdict: "forbidden-without-permission", clause: "Upstash ToS §C.8", permission: "not-obtained"}. Whether Context7 enters the live-run manifest is the maintainer's call at T13 after seeking permission (support@upstash.com) — assumption A1.

1.2 GitMCP

• Pin (weak): hosted https://gitmcp.io/python/cpython + access date + serverInfo.version (1.1.0 on 2026-07-09), recorded per run. No npm (the npm git-mcp package is an unrelated name-collision project — never reference it), no tags/releases/Docker; self-host needs a Cloudflare account (not clean-clone reproducible). Code reference: idosal/git-mcp@main SHA c487a29.
• Transport: streamable HTTP (verified live; legacy GET/SSE returns 405). mcp.client.streamable_http.streamablehttp_client handles the session-id echo and SSE-framed responses.
• Auth: none.
• Tool flow (verified live): search_cpython_documentation returns no results for stdlib queries (indexes only the repo's README fallback — no llms.txt in cpython); the working path is search_cpython_code(query) → fetch_generic_url_content(url) on the found Doc/**/*.rst (or docs.python.org) URL. Adapter flow: doc-search first (honest: it is the advertised path), then code-search → fetch. This weak-doc-tool finding is a legitimate benchmark result, not an exclusion.
• Terms: Apache-2.0 code; hosted service has no ToS document; README explicitly invites programmatic agent access. Benchmarking permitted. Keep concurrency low (shared GitHub API quota behind it).
• Decision: include, with the weak pin disclosed in manifest metadata — assumption A2.

1.3 DeepWiki (Cognition)

• Pin (weakest): hosted https://mcp.deepwiki.com/mcp + access date + serverInfo.version (2.14.3 on 2026-07-09) + target-wiki last-indexed date (python/cpython: 2026-06-24). No package/image/self-host exists. Third-party npm "deepwiki" packages are NOT Cognition's server — never reference them.
• Transport: streamable HTTP (/mcp; /sse officially deprecated). No auth.
• Tool flow (verified live): ask_question(repoName="python/cpython", question=<verbatim>) — returns an LLM-generated answer (~13.5 s, nondeterministic across identical calls — verified). read_wiki_contents is an all-or-nothing ~1.18 MB blob of a Devin-generated wiki, not official docs. Adapter flow: single ask_question call; 60 s timeout.
• Terms: no DeepWiki-specific ToS; Cognition Platform ToS + AUP (June 2026) contain no benchmark clause; gray zone: AUP competitive-use language, given the publisher is a competing docs tool.
• Decision: include-with-disclosure is the recommendation: manifest metadata flags answer_generation: "llm" (an answer-synthesis system scored inside a retrieval benchmark — the report's per-competitor framing must not hide this) and the competitive-clause note. Maintainer may instead exclude — assumption A3.

1.4 Ref.tools

• Pin: client ref-tools-mcp@3.0.3 (npm, MIT) or Docker mcp/ref-tools-mcp digest sha256:8b5fcfe3...; backend api.ref.tools unpinnable → effective pin = client 3.0.3 + run date. Remote streamable-HTTP https://api.ref.tools/mcp with x-ref-api-key header is vendor-recommended; stdio npx ref-tools-mcp@3.0.3 (env REF_API_KEY) is the legacy path. Adapter default: remote streamable HTTP (no Node needed).
• Auth: API key REQUIRED (signup; free tier = 200 one-time credits; 1 credit per tool call → a 50-question run at 2 calls/question ≈ 100 credits; a full matrix run needs a paid month ~$19).
• Tool flow: ref_search_documentation(query=...) → ref_read_url(url=<top docs.python.org hit>). Verified capability: vendor's own canonical example is a Python-stdlib query. Session-dependent behavior (dedupes repeats within a session; read trimming keyed on session history) → fresh MCP session per question (adapters already instantiate per cell — enforced by design).
• Terms (extracted from SPA bundle, 48k chars, grep-verified): no benchmark/evaluation clause; but §3 "no automated or non-human access", §7 no systematic retrieval "without written permission", subjective disparagement clause, "personal, non-commercial use only" license. Unclear, leaning risky for publication. The systematic-retrieval clause itself names the cure: written permission.
• Decision: build fully (mock-tested); manifest metadata records terms_check: {verdict: "unclear-permission-recommended", ...}; live use needs maintainer key + credits; publication needs the maintainer's permission decision — assumption A4.

2. Shared design

2.1 Adapter shape (mirror of the merged #86 product adapter — the proven template)

Each adapter is a plain class (NOT a ProviderAdapter subclass — that contract models an LLM call; these model retrieval, same as PythonDocsMcpAdapter, python_docs_mcp_adapter.py:9-16):

• benchmarks/adapters/context7_adapter.py — Context7Adapter
• benchmarks/adapters/gitmcp_adapter.py — GitMcpAdapter
• benchmarks/adapters/deepwiki_adapter.py — DeepWikiAdapter
• benchmarks/adapters/ref_tools_adapter.py — RefToolsAdapter

Common contract (all verified against python_docs_mcp_adapter.py):

• Ctor takes endpoint/command overrides + timeout_seconds (30.0 default; 60.0 for DeepWiki) + flow knobs; instantiated fresh per cell by the dispatch fn (gives Ref its fresh-session-per-question requirement for free).
• run(prompt) -> <Adapter>Result(answer: str, tool_calls: list[ToolCallRecord]) — answer = the retrieved content (concatenated tool payloads / ask_question text), tool_calls = full raw request/response records for the transcript (same ToolCallRecord shape: tool, arguments, result, is_error).
• Lazy mcp imports inside the async runner (module import must never require mcp client submodules — established pattern, python_docs_mcp_adapter.py:246-259).
• asyncio.run(asyncio.wait_for(...)); exception mapping exactly: BenchmarkCellFailure re-raised; TimeoutError → "timeout"; other Exception → "mcp_protocol_crash"; tool isError=True → "tool_failure". Known inherited limitation (Codex round-1 finding 2, not fixed here): on failure the runner records tool_calls: None (runner.py:239-247) — the failing call's request/response is carried only in the failure message, not the transcript; changing that is a runner-contract change out of [v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools) #87's scope. Adapters mirror the merged contract exactly.
• Guard-first: the FIRST statement of run() calls the applicable guard (§2.2) — before any subprocess spawn or transport construction. Structural offline proof mirrors test_claude_tokens.py:135-181 (fail-closed stubs).
• Retrieval flows are pure orchestration over a duck-typed session Protocol (same _CallToolSession trick) so tests drive them with scripted doubles, no transport.

2.2 Guard extensions (benchmarks/adapters/guard.py, additive only)

• PROVIDER_API_KEY_ENV += {"ref": "REF_API_KEY", "context7": "CONTEXT7_API_KEY"} (registration point per guard.py:41-45; the unknown-provider merged test is unaffected).
• NEW require_live_competitor(name: str) -> None — two non-secret latches (Codex round-1 finding 3: flag-only gating would weaken the guard posture): (1) BENCHMARK_LIVE_PROVIDERS_ENABLED truthy (same set, same LiveProviderDisabledError), AND (2) new env BENCHMARK_LIVE_COMPETITORS — a comma-separated allowlist that must contain name (e.g. BENCHMARK_LIVE_COMPETITORS=gitmcp,deepwiki). Keyless competitors thus keep the same two-latch shape as keyed providers (global flag + per-target opt-in), with the allowlist standing in for the key. Additive function; existing guard contract and merged tests untouched.
• Mapping: Context7 → require_live_environment("context7") when the manifest entry declares key-mode, else require_live_competitor("context7") (keyless mode is real but rate-limited); GitMCP → require_live_competitor("gitmcp"); DeepWiki → require_live_competitor("deepwiki"); Ref → require_live_environment("ref").

2.3 Dispatch registration (benchmarks/runner.py, the documented seam)

Four lazily-importing dispatch fns (pattern: _python_docs_mcp_stdio_answer, runner.py:410-431) + four _ADAPTER_DISPATCH entries:
"context7", "gitmcp", "deepwiki", "ref-tools". Each reads its config from cell.competitor.raw (endpoint override, key-mode, timeout) and converts ToolCallRecords to transcript dicts exactly as the product adapter does.

2.4 Eligibility screening + structured exclusion (issue criterion 2 — REDESIGNED per Codex round-1 finding 1)

The rejected first design (per-cell BenchmarkCellFailure("tool_failure", "structured-exclusion: ...")) would score excluded competitors as failures: the runner turns every BenchmarkCellFailure into status: failed (runner.py:245), scoring fixes failed cells at 0.0 inside the denominator, and report error-rate stats count them — contaminating exactly the metrics the methodology says exclusions must stay out of ("report the failure reason and exclude it from scored comparisons"). Exclusion is therefore a manifest-load-time refusal, never a cell outcome:

• Manifest metadata convention (Competitor.raw passes every YAML key through untouched, runner.py:73-79; the manifest is snapshotted byte-for-byte into run artifacts): each ACTIVE competitor entry carries
  pin: {kind: npm-version|endpoint-date, value, server_info_version?, code_sha?, access_date},
  terms_check: {verdict: permitted|forbidden-without-permission|unclear-permission-recommended, clause?, checked_on, source_url},
  eligibility: {status: eligible|conditional, conditions?: [...]}.
• Excluded systems live in a top-level exclusions: block (sibling of competitors:), each with id, reason, terms_check, decided_on. The runner's execution path ignores unknown top-level keys, but the byte-for-byte snapshot preserves the block in every run's artifacts — exclusions are documented in the same auditable file, never silently dropped.
• Encoded screening: benchmarks/adapters/eligibility.py exposes validate_competitor_eligibility(raw) (per-entry: pin + terms_check present and well-formed; status ∈ {eligible, conditional}) and validate_manifest_eligibility(manifest_data). The runner's _load_manifest calls the latter: any competitor entry with eligibility.status: excluded (or an entry that belongs in exclusions:) → clean BenchmarkValidationError (exit 2, before any cell executes, message naming the competitor and pointing at the exclusions: block). This is a small additive runner change beyond the dispatch entries, in the runner's established refusal idiom (same class as the non-empty --out refusal) — justified here explicitly so the worker doesn't treat it as scope creep. Adapters' dispatch fns also call validate_competitor_eligibility (defense in depth; unreachable-for-excluded by construction).
• Zero cells ever execute for an excluded competitor → zero contamination of correctness/error-rate/latency metrics. Final include/exclude for the live run stays the maintainer's call when authoring the live manifest (T13); the template (§2.5) prefills the exclusions: block per A1–A4.

2.5 Committed template manifest

docs/benchmarks/competitor-manifest.template.yml — the four competitors with researched pins + terms_check verbatim (clauses quoted), eligibility prefilled per A1–A4, header comment: "TEMPLATE — not a benchmark result; live-run manifests are maintainer-authored copies; per methodology, exclusions must be reported, not hidden." Satisfies the methodology's "Competitor manifest with pinned versions" reproducibility artifact and the issue's terms-recording criterion. NOT at docs/benchmarks/results/ (that path is claim-bearing).

3. File-by-file work list

File	Change
benchmarks/adapters/context7_adapter.py	NEW (~180 lines): stdio (npx pin) + streamable-HTTP modes; resolve→query flow
benchmarks/adapters/gitmcp_adapter.py	NEW (~180): streamable HTTP; doc-search → code-search → fetch-url flow
benchmarks/adapters/deepwiki_adapter.py	NEW (~140): streamable HTTP; single ask_question; 60 s timeout
benchmarks/adapters/ref_tools_adapter.py	NEW (~170): streamable HTTP + x-ref-api-key; search → read_url flow
benchmarks/adapters/eligibility.py	NEW (~80): per-entry + manifest-level eligibility validation (pin/terms/status); exclusion → BenchmarkValidationError
benchmarks/adapters/guard.py	+2 PROVIDER_API_KEY_ENV entries; +require_live_competitor() two-latch gate (additive)
benchmarks/adapters/__init__.py	export new names (additive)
benchmarks/runner.py	+4 lazily-importing dispatch fns, +4 _ADAPTER_DISPATCH entries (documented seam); + one validate_manifest_eligibility call in _load_manifest (refusal idiom, §2.4)
docs/benchmarks/competitor-manifest.template.yml	NEW template manifest (§2.5)
tests/benchmarks/test_competitor_adapters.py	NEW (~550): see §4

Nothing else. Explicitly untouched: report.py, model_matrix.py, scoring.py, corpus.py, all merged tests, pyproject.toml, uv.lock, src/, .github/.

4. Test plan (mock-transport only; zero network in CI)

All in tests/benchmarks/test_competitor_adapters.py (pinned name):

1. Flow tests per adapter via scripted-session doubles (pattern: _ScriptedSession, test_python_docs_mcp_adapter.py:42-58): correct tool sequence + arguments, answer assembly, tool_calls records complete (including the GitMCP empty-doc-search fallback chain and Ref's search→read chaining of the top URL).
2. Failure-category mapping per adapter: isError → tool_failure; timeout → timeout; transport crash → mcp_protocol_crash (monkeypatch the adapter's _run_async, dummy paths — pattern test_python_docs_mcp_adapter.py:240-277).
3. Guard refusals fail closed: for each adapter, every disabled env permutation refuses BEFORE any transport construction. Each adapter routes ALL transport creation through one adapter-local _transport_factory seam (Codex round-1 finding 4: the SDK ships both streamable_http_client and the deprecated streamablehttp_client alias — tests must patch the seam actually used, so make it a single owned symbol); tests monkeypatch that factory (and subprocess/stdio + socket.socket for Context7) to raise AssertionError("network reached") — pattern test_claude_tokens.py:135-181; env names imported from guard constants, never hardcoded.
4. require_live_competitor: refuses without the flag; refuses with flag but name absent from BENCHMARK_LIVE_COMPETITORS; passes only with both latches; key-independent; allowlist parsing tolerant of spaces/case.
5. Eligibility/exclusion: missing pin/terms metadata → per-entry validation error; a manifest whose competitors: list contains eligibility.status: excluded → runner refuses at load with BenchmarkValidationError naming the competitor, exit 2 via the CLI, zero artifacts written (reuse _write_yaml/tmp-dir + CLI-subprocess patterns from test_runner.py); an exclusions: block passes validation and survives byte-for-byte into snapshots/competitor-manifest.yml; the committed template itself parses and validates.
6. Registry integration: manifest entries with the four adapter strings dispatch to the right fns (mocked _run_async), and the template manifest file itself parses + passes eligibility validation.

5. Validation commands (the run's gate; paste full output in PR body)

uv run ruff check src/ tests/
uv run ruff check benchmarks/        # additive until #84 lands in ci.yml
uv run pyright src/
uv run pyright benchmarks/           # additive until #84 lands
uv run pytest --tb=short -q
uv run pytest tests/benchmarks -q
uv run pytest tests/benchmarks/test_competitor_adapters.py -q
uv run python-docs-mcp-server doctor

Plus PR-body proofs: git diff --stat main...HEAD showing pyproject.toml/uv.lock absent; supervisor-review label (expected: total diff >500 lines — precedent #97/#98/#100); Refs #63, Closes #87 only if all criteria met.

6. Labelled assumptions — Stage 3 gate items (maintainer must confirm; these ARE the pre-flight pin/terms confirmations)

• A1 (Context7): pin @upstash/context7-mcp@3.2.3 stdio; terms verdict "forbidden-without-permission" (ToS §C.8) recorded; adapter built now; live inclusion deferred to T13 pending permission request to Upstash. Confirm/override.
• A2 (GitMCP): weak pin (endpoint + date + serverInfo 1.1.0 + code SHA c487a29) accepted with disclosure; include. The verified doc-tool weakness is reported as a finding, not an exclusion. Confirm.
• A3 (DeepWiki): weak pin (endpoint + date + serverInfo 2.14.3 + wiki index date) accepted; include WITH answer_generation: "llm" disclosure + competitive-clause note — or exclude. Recommend include-with-disclosure. Confirm/choose.
• A4 (Ref.tools): pin client 3.0.3 + date; terms "unclear-permission-recommended" recorded; adapter built now; live use needs maintainer account/key/credits; publication needs the permission decision at T13. Confirm.
• A5 (guard design): two new key-env registrations + additive two-latch require_live_competitor() (global flag + BENCHMARK_LIVE_COMPETITORS allowlist) for keyless competitors. Confirm.
• A6 (exclusion semantics): exclusion is a manifest-load-time refusal (runner BenchmarkValidationError, zero cells executed, zero metric contamination) + a documented top-level exclusions: block preserved in run snapshots; live-manifest inclusion is the T13 maintainer decision. Confirm.

7. Out of scope (restated)

Corpus questions; live benchmark calls of any kind; README/PyPI/launch claims; report/scorer/matrix changes; #33/#34; pyproject/uv.lock; obtaining vendor permissions or API keys (maintainer-only, T13); any second competitor repo target beyond python/cpython.

Stage 3 gate record (2026-07-09)

Codex sign-off round 2 (zero new findings; merged-test compatibility of the
runner-refusal design explicitly confirmed). Maintainer (Aymen) confirmed all
six assumptions in-session via AskUserQuestion, each as recommended:
A1 Context7 build-now/permission-before-publishing; A2 GitMCP include with
disclosed weak pin; A3 DeepWiki include with LLM-answer disclosure; A4
Ref.tools build-now/permission+credits at T13; A5 two-latch guard design; A6
manifest-load-time exclusion semantics. These confirmations constitute the
pre-flight pin/terms decision required by PLAN.md before #87 dispatch.
Mirrored as a comment on #87.

Residual review notes

Codex round 1 (no sign-off) — triage: finding 1 (BLOCKING, exclusion-as-cell-failure contaminates scored metrics) accepted → §2.4 redesigned to manifest-load-time refusal + exclusions: block; finding 2 (WRONG-IN-PLAN, failure-transcript claim) accepted → §2.1 corrected, limitation inherited not fixed; finding 3 (NICE-TO-HAVE, flag-only gate) accepted → two-latch require_live_competitor(); finding 4 (NICE-TO-HAVE, transport-symbol patching) accepted → adapter-local _transport_factory seam. Codex confirmations kept: no-dependency claim PASS (mcp 1.28.1 + httpx locked); terms-handling split PASS; scope PASS. Test-plan note honored: mocks do NOT claim to cover live-only behaviors (GitMCP session-id echo, Ref session dedupe, DeepWiki nondeterminism) — those are documented run-time risks in the dossiers, exercised only in the maintainer-run live phase.

[ayhammouda]

Agent context — #87 [v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools)

Status: FINAL for dispatch (refreshed against main @ 5a518de per the T1 timing rule — supersedes any pre-#86 assumptions; #86's registry merged @ 548acd1, guard state includes #101 @ 53749fc)
Auditable copy: this content is mirrored as a comment on issue #87 (.planning/ is gitignored — the issue comment is the copy pre-flight verifies and the dispatch packet links).

1. Roadmap excerpt (why this exists)

• Roadmap §4 v0.5.0: public benchmark harness — "all eight target docs MCPs + no-MCP baseline"; methodology "Systems Under Test" names Context7, GitMCP, DeepWiki, Ref.tools. Decision 5.17: zero comparative claims in public copy until reproducible data exists.
• Governing plan: PLAN-87-competitor-adapters.md (Codex-signed round 2; full text mirrored as a comment on [v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools) #87). Maintainer gate decisions A1–A6 (2026-07-09, also mirrored on [v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools) #87) are BINDING: A1 Context7 built now, publication requires Upstash permission (ToS §C.8); A2 GitMCP in with disclosed weak pin; A3 DeepWiki in with answer_generation: llm disclosure; A4 Ref.tools built now, key/credits/permission at live phase; A5 two-latch guard; A6 manifest-load-time exclusion.

2. Code touch-points (verify against merged main before starting)

• benchmarks/runner.py:439-444 — _ADAPTER_DISPATCH: dict[str, Callable[[BenchmarkCell], _CellDispatchResult]]; add four entries: "context7", "gitmcp", "deepwiki", "ref-tools", each mapping to a lazily-importing dispatch fn (pattern: _python_docs_mcp_stdio_answer, runner.py:410-431 — adapter modules import BenchmarkCellFailure from benchmarks.runner, so dispatch fns MUST lazy-import inside the function body). _CellDispatchResult at runner.py:384-395 (answer: str, tool_calls: list[dict] | None).
• benchmarks/runner.py:200-218 — _load_manifest: add one call to validate_manifest_eligibility(...) (new, §2.4 of the plan): a competitors: entry with eligibility.status: excluded → BenchmarkValidationError before any cell executes; a top-level exclusions: block passes untouched (execution ignores it; the byte-for-byte snapshot at runner.py:568-578 preserves it).
• benchmarks/adapters/python_docs_mcp_adapter.py — THE template: plain class (NOT ProviderAdapter, docstring lines 9-16), lazy mcp imports in _run_async (:246-259), asyncio.run(asyncio.wait_for(...)) (:234), exception mapping BenchmarkCellFailure/TimeoutError→timeout/other→mcp_protocol_crash (:235-244), duck-typed _CallToolSession Protocol (:91-98), ToolCallRecord (:60-67), _tool_result_payload prefers structuredContent (:119-130). On failure the runner records tool_calls: None (runner.py:233-247) — inherited contract, do NOT change it.
• benchmarks/adapters/guard.py:41-45 — PROVIDER_API_KEY_ENV += {"ref": "REF_API_KEY", "context7": "CONTEXT7_API_KEY"}. NEW additive require_live_competitor(name): latch 1 = BENCHMARK_LIVE_PROVIDERS_ENABLED truthy ({"1","true","yes"}, guard.py:47); latch 2 = name ∈ comma-separated BENCHMARK_LIVE_COMPETITORS. Same LiveProviderDisabledError. Guard call is the FIRST statement of every adapter's run().
• mcp 1.28.1 is an installed project dependency (uv.lock:297-302): mcp.client.stdio, mcp.client.streamable_http.streamablehttp_client (+ preferred streamable_http_client at :601), mcp.client.sse.sse_client; httpx 0.28.1 installed. Route ALL transport creation through one adapter-local _transport_factory seam per adapter (tests patch that seam). websocket client is NOT usable (extra absent) — not needed.
• New modules: context7_adapter.py (stdio npx -y @upstash/context7-mcp@3.2.3, flow resolve-library-id→query-docs), gitmcp_adapter.py (streamable HTTP https://gitmcp.io/python/cpython, flow search_cpython_documentation→search_cpython_code→fetch_generic_url_content), deepwiki_adapter.py (streamable HTTP https://mcp.deepwiki.com/mcp, single ask_question, timeout 60 s), ref_tools_adapter.py (streamable HTTP https://api.ref.tools/mcp, header x-ref-api-key, flow ref_search_documentation→ref_read_url), eligibility.py (per-entry + manifest validation). Endpoints/pins/flows/timeouts: exactly per PLAN-87-competitor-adapters.md §1, no improvisation.
• docs/benchmarks/competitor-manifest.template.yml — NEW: four competitors with pins + terms_check verbatim from the plan §1, exclusions: block prefilled per A1–A4, TEMPLATE header comment (not results; live manifests are maintainer-authored).
• benchmarks/adapters/__init__.py — additive exports.

3. Test patterns to follow

• Focused file pinned: tests/benchmarks/test_competitor_adapters.py. Plan §4 is the authoritative test list.
• Scripted-session double: tests/benchmarks/test_python_docs_mcp_adapter.py:42-58; failure-category mapping via _run_async monkeypatch (:240-277); fail-closed env/network pattern tests/benchmarks/test_claude_tokens.py:135-181 (env names imported from guard constants — e.g. PROVIDER_API_KEY_ENV["ref"] — never hardcoded); CLI-as-subprocess + _write_yaml manifest/tmp-dir patterns tests/benchmarks/test_runner.py:37-50, 353-378.
• Zero network in CI: patch each adapter's _transport_factory (plus subprocess/socket.socket for Context7's stdio path) to raise AssertionError("network reached") in every guard-refusal test.

4. Known pitfalls

• npm name traps: git-mcp on npm is an UNRELATED project; third-party "deepwiki" npm packages are NOT Cognition's server. Never reference either.
• Merged tests additive-only; NO sanction exists for this issue (none needed — verified: unknown-provider guard test uses "not-a-registered-provider").
• pyproject.toml/uv.lock must be absent from the diff (mcp SDK + httpx already installed); PR body includes git diff --stat main...HEAD proof.
• No live/network calls ever — not in tests, not "just to check". Node/npx is a live-phase environment note in the template manifest, never a test dependency.
• Supervisor-review expected: total diff >500 lines (precedent [v0.5.0] benchmark corpus — mechanical slice: schema, validator, placeholder fixture (split from #71) #97/[v0.5.0] benchmark adapter — python-docs-mcp-server tool runner (offline stdio) #98/[v0.5.0] benchmark scoring — correctness scorer with manual-adjudication hooks #100) → apply the label + ## Why this triggered supervisor review section.
• Refs #63, never Closes #63; Closes #87 only if every acceptance criterion is met. Branch: agent/87-competitor-adapters.
• validate_manifest_against_matrix (model_matrix.py:227-233) skips entries without provider+model — competitor entries need no matrix changes; do not touch model_matrix.py.

5. Decision log (worker fills in)

• …