🔬 Tutorial DX Eval (alpha.17, ARMED): lock criteria + track 4 tutorial runs + derived backlog

[rickylabs]

Canonical tracker for the 4-tutorial developer-experience evaluation against the published JSR CLI (currently 0.0.1-alpha.10, prod-verified green via e2e-cli-prod). Each tutorial is walked end-to-end by a closely-supervised Claude Opus 4.8 evaluator sub-agent acting as a real developer following the documented path from an empty directory to a running app under .NET Aspire. This issue locks the evaluation criteria and tracks every run plus everything derived from it.

These runs are evaluations, not smoke tests. The bar is PERFECT DX — findings are graded, not pass/fail.

---

Locked evaluation criteria (every run covers all 8 axes)

1. Tutorial validity — does the documented path actually work, start to finish, with no step skipped?
2. Framework stability — can a real developer complete the tutorial (and therefore a legitimate project) on the framework as published?
3. Doc-drift detection — every place the documentation diverges from observed reality.
4. Drift root-cause classification — for each drift, is the fix in the code or in the docs (incorrect/outdated)?
5. DX perfection bar — is the developer experience perfect? Every rough edge, confusing message, or unnecessary step is a finding.
6. Process rethink — concrete ideas to make the scaffold/build/dev loop itself better.
7. New-capability opportunities — plugins, package updates, tools that push DX + enterprise-framework quality further.
8. Documentation polish — findings fold back into all docs (not only the tutorial under test).

Finding classification (every finding is tagged)

• Type: doc-fix · code-fix · new-feature · dx-polish · process
• Severity: blocker (cannot complete) · major (completes but wrong/painful) · minor (cosmetic/wording)
• Evidence: exact command + documented behavior vs observed behavior + file:line of the doc claim.

---

The 4 tutorial runs

#	Track	Focus	Chapters	Status	Run report
1	Storefront	services + durable workflows (catalog, cart contracts, checkout saga, shipping webhook)	6	⬜ not started	—
2	Team Workspace	auth + access control (auth backend, session, 2nd DB, provision job, .withAuthz())	6	⬜ not started	—
3	ERP Sync	jobs, queues & polyglot (triggers, durable jobs, queue+cron, polyglot tasks)	5	⬜ not started	—
4	Live Dashboard	Fresh + SDK stack (contract→SDK, cache-first query, definePage/QueryIsland, durable stream)	6	⬜ not started	—

Runs are executed sequentially (the local Aspire AppHost uses a fixed port — see #138 — so parallel runtime boots would collide), each closely supervised.

Method / environment

• CLI under test: published jsr:@netscript/cli (alpha.10), installed exactly as the docs instruct: deno install --global --allow-all --name netscript jsr:@netscript/cli.
• Canonical docs: the tutorial markdown in docs/site/tutorials/<track>/ on main (source of the live site) — agents cite file:line for every doc-drift finding.
• Runtime: local Deno 2.9 + .NET Aspire + Docker/Postgres; full path including the Aspire boot is exercised where the tutorial calls for it.
• Evaluator surface: Claude Opus 4.8 sub-agents (not CI smoke) so each step is observed and collaborated on.

---

Derived backlog (populated as runs land — each substantial item becomes its own linked issue)

🐛 Code fixes

none yet

✨ Enhancements / process improvements

none yet

🧩 Planned features (plugins / packages / tools)

none yet

📝 Documentation polish

none yet

---

Tracking task: #135. Updated after each run with the run report link and newly-filed derived issues.

[rickylabs]

Run 1 — Storefront · Chapter 1 (Scaffold) — ❌ NOT completable as written

Evaluator: closely-supervised Claude Opus 4.8 sub-agent, real-developer stance, against published jsr:@netscript/cli@0.0.1-alpha.10. Date: 2026-06-27. Full report: .llm/tmp/run/tutorial-eyetest-alpha10--eval/run-1-storefront-ch1.md.

Verdict: Chapter 1 is blocked at command #1 (install) and again at both end-of-chapter verify gates. File generation (scaffold/fmt/git) is solid; everything that proves a running system is broken or mis-documented. 14 findings — 5 blocker · 4 major · 5 minor (code-fix 6 · doc-fix 5 · dx-polish 2 · process 1).

Env: deno 2.9.0, aspire 13.4.6, dotnet 10.0.109, docker 29.5.2. Note: installed CLI is alpha.10 but netscript --version misreports 1.0.0.

🚫 Blockers (5)

• F-1 — install command is dead on arrival. deno install … jsr:@netscript/cli → Could not find version … matches '*'. Every published version is prerelease, latest: null, bare jsr: resolves * (excludes prereleases). Real fix: cut a non-prerelease 0.0.1 so latest populates (unblocks all 4 tutorials). Stopgap: pin @0.0.1-alpha.10 in every doc.
• F-2 — minimum-dependency-age rejects same-day alpha. alpha.10 (published 2026-06-27) is "too new"; required --minimum-dependency-age=0 to install and to deno task check. Any user with an age floor silently can't follow the tutorial.
• F-4 — --dry-run writes a full 153-item tree to disk while printing "No files were written." Non-functional and actively misleading; dry-then-real leaves a polluted tree.
• F-11 — Garnet cache is never scaffolded for --db postgres, yet the chapter promises it and the checklist requires garnet healthy. No garnet|cache|redis token in appsettings.json/apphost.mts. Decision needed: scaffold Garnet by default, or drop it from Ch.1 + checklist.
• F-13 — products service never actually served on :3001. Port is held by a DCP proxy but no Deno process runs services/products/src/main.ts; every probe times out (curl exit 28, 60s+). The chapter's central "running system" claim is false on Windows aspire run.
• F-15 — fresh scaffold fails deno task check with 3 real type errors: service-showcase.ts:58 (QueryOptionsWithInitialData not assignable), :60 (getQueryState missing on QueryClientPort — SDK drift), vite.config.ts:11 (NetScriptVitePlugin not a valid Vite 7 PluginOption). The closing "a clean check means the scaffold is sound" is false out of the box.

⚠️ Majors (4)

• F-3 — netscript --version reports 1.0.0 instead of the installed 0.0.1-alpha.10 (hardcoded; breaks support/triage).
• F-6 — CLI next-steps list db init/generate/seed BEFORE aspire run, contradicting the doc's own callout that DB commands must run after Aspire.
• F-8 — dashboard serves https://localhost:18888; the doc says http (3 refs), and plain http fails outright.
• F-12 — aspire run leaves an orphaned dashboard/DCP/container with no attachable AppHost (aspire stop/MCP report "No running AppHost"); teardown needs manual kills. Tied to the documented aspire run (the canonical agent path is aspire start).
• F-14 — /health is an oRPC route (v1.products.health.check), not a plain GET — the documented curl …/health can't work even when the service is up.

📝 Minors (5)

F-5 dry-run prints one aggregate total, not per-phase · F-7 database/postgres/ workspace absent from the file tour · F-9 aspire restore warns CLI(13.4.6)↔SDK(13.4.4) mismatch, undocumented · F-10 no one-time login token (anonymous dashboard configured) · (F-9 doc note).

Derived backlog

• Code-fix slices (framework — S0: initial public repo genesis #1 priority): F-4 dry-run write-free · F-13/F-12 service-actually-runs + attachable AppHost on Windows aspire run · F-15 scaffold type-soundness (templates + @netscript/sdk getQueryState/QueryOptionsWithInitialData + @netscript/fresh-ui vite plugin type) · F-3 --version from package · F-6 next-steps order · F-14 plain GET /health in scaffold.
• Doc-fix: F-1 stopgap pin · F-8 https · F-5/F-7/F-9/F-10 polish · tutorials' aspire run → aspire start.
• Release/decision (need user): F-1 cut non-prerelease 0.0.1 (the real install fix) · F-2 release aging vs age-floor docs · F-11 Garnet default vs drop.

Program status

Per "fix all → alpha.X → tutorials": remaining chapters (Storefront 2–6 + Workspace/ERP Sync/Live Dashboard) are PAUSED until the Ch.1 blockers are fixed and a clean CLI is republished — running them now would just re-hit the same walls. Eye-tests stay faithful + sequential. This comment + report are the Run-1 deliverable; derived items become their own slices/issues as the fix-train forms.

[rickylabs]

Progress update — alpha.11 docs realignment + polyglot CLI plan (2026-06-27)

Recording all progress on the alpha.11 derived-backlog docs work, plus a grounded plan for polyglot CLI support.

✅ Done — docs realignment (PR #162, branch docs/alpha11-cli-surface)

1. Cache → multi-backend capability (commit b847c6a8, 45 files)

• redis is the default cache backend everywhere; garnet and deno-kv are first-class alternatives via --cache-backend. Garnet kept only where shown as an explicit alternative.
• Real symbols preserved (createGarnetListTransport(), Cache.<backend> keys, GARNET_URI/REDIS_URI, CACHE_PROVIDER).
• aspire-resource-graph diagram regenerated with the default Redis node (byte-clean).

2. Database → polyglot capability (commit 28539811, 27 files)

• NetScript scaffolds a Prisma-backed DB for four engines via --db: postgres | mysql | mssql | sqlite. Postgres is the recommended default (tutorials keep --db postgres); mysql/mssql/sqlite are first-class alternatives. postgres/mysql/mssql → Aspire container; sqlite is file-backed, no container.
• Enrichment (not replacement): anchor pages get a richer --db passage; tutorials gain a swap-engine note; subsystem/connection-mechanics left untouched.
• Lume build green (306 files) for both passes.

Per harness doctrine the authoring lane does not self-certify — per-domain doc validation runs separately in OpenHands (pending).

✅ Ground truth — CLI polyglot DB support is already COMPLETE

Verified end-to-end against the built CLI. All four engines are real, not stubs:

• DbEngine = postgres|mysql|mssql|sqlite; rich per-engine DbEngineProvider (prismaProvider, aspireMethod, supportsContainerMode, defaultImageTag).
• --db <engine> accepts all four; unit tests exercise all four. sqlite correctly supportsContainerMode:false.
• Prisma providers: postgres=postgresql, mysql=mysql, mssql=sqlserver, sqlite=sqlite. Aspire: addPostgres/addMySql/addSqlServer; sqlite skipped (file-backed).
• Literal CLI default is none (no DB); docs frame Postgres as the recommended default.

🔎 Subsystem DB-coupling audit — answering "are the excluded pages really Postgres-locked?"

File-grounded verdict. Only the queue provider is genuinely Postgres-specific; everything else already follows the scaffolded engine.

Subsystem	Coupling	Polyglot effort
Queue / background jobs	HARD-tied to Postgres — raw SQL FOR UPDATE SKIP LOCKED, JSONB::jsonb, TIMESTAMPTZ, NOW(), $N params (not Prisma)	Real work: per-engine adapters. MySQL 8.0+/MSSQL 2019+ feasible; SQLite infeasible (no SKIP LOCKED)
Sagas store	Prisma-agnostic (typed Prisma methods, generic schema)	Zero — runs on all 4 today
Streams	Not DB-coupled (HTTP durable-streams service)	Zero
Auth (better-auth)	Prisma adapter (supports all 4 SQL engines)	Zero
Auth (kv-oauth)	KV-based, zero SQL	Zero

Consequence: the sagas/streams/auth docs I excluded are not actually Postgres-locked — they persist to whatever engine you scaffolded. They MAY get light polyglot framing in a follow-up. The queue provider stays Postgres-specific by design — a genuine framework limitation, not a doc choice.

New arch-debt (record-only): QUEUE-ENGINE-POSTGRES-LOCK — a polyglot queue needs per-engine SQL adapters (SQLite not viable without a polling lock model). Backlog.

📋 Planned — framework slices (harness-gated, WSL Codex; PLAN-EVAL first)

1. E2E matrix coverage — extension-axes.ts DATABASE axis is postgres+mysql only; add mssql+sqlite and parametrize the scaffold runner (sqlite asserts no Aspire container). Keep heavy scaffold.runtime postgres-default; add other engines at the cheaper scaffold/type-check tier.
2. Interactive DB-engine prompt in init — currently --db omitted → silently none. Prompt infra already exists and is unwired (kernel/ports/prompt-port.ts + cliffy-prompt.ts); the slice wires a DB-engine select (default highlighted = postgres), gated off --ci/--yes/--json. --db always wins.

Backlogged: @netscript/prisma-adapter-mysql JSR publish; polyglot queue (above).

Next

• OpenHands per-domain validation of PR Docs: alpha.11 CLI surface + multi-backend cache + polyglot database realignment #162 (cache + DB docs).
• PLAN-EVAL (OpenHands minimax-M3) for the 2 framework slices, then WSL Codex implementation.

🤖 Generated with Claude Code