Design document registry IDs and resolver

[alexeygrigorev]

Design document registry IDs and resolver

Status: pending
Tags: enhancement, process-docs, backend, data, P0
Depends on: #2, #3
Blocks: #6, #8, #9

Scope

Define and implement the first document registry contract for DataOps process documents so work items can link to SOPs, templates, references, and playbooks by stable identifiers instead of brittle paths or raw URLs.

The work includes:

• Define the stable document ID format, alias rules, uniqueness constraints, and expected frontmatter fields for operational documents.
• Add a registry builder that scans imported content/ markdown and returns canonical document metadata needed by the portal and work engine.
• Add resolver behavior for canonical IDs, aliases, repo-relative paths, and wiki-style document references.
• Expose backend metadata and resolve endpoints using the existing docs app/backend conventions after the DTC Operations import is available.
• Add validation that catches duplicate IDs, duplicate aliases, invalid IDs, and broken document references before task/workflow links rely on them.
• Seed or document the minimum ID coverage needed for task-to-process-doc links and the first podcast workflow.

Acceptance Criteria

• A stable document ID convention is documented in the implementation, including allowed characters, namespace/domain guidance, immutability expectations, and examples for SOPs, templates, references, and playbooks.
• The registry scanner reads markdown metadata from content/ and produces canonical records with at least id, aliases, path, title, doc_type, summary, tags, systems, related_docs, and updated_at when available.
• The resolver accepts canonical IDs, aliases, repo-relative paths, and wiki-style references, returning one canonical document record or a clear not-found/ambiguous error.
• Backend API support exists for listing registry metadata and resolving a single document reference, following the imported docs app routing and response style.
• Validation fails on duplicate IDs, duplicate aliases, invalid ID syntax, and broken related_docs or equivalent document references.
• Priority documents needed by task links and the first podcast workflow can be assigned or resolved without relying on raw GitHub URLs.
• Automated tests cover registry scanning, alias resolution, path/wiki reference resolution, duplicate detection, broken reference detection, and API response shape.

Test Scenarios

Scenario: Task opens a process document by stable ID

Given: a task or template references an imported process document by instructionDocId
When: the backend resolves the reference
Then: the response contains the canonical document metadata and current repo-relative path.

Scenario: Old path resolves through an alias

Given: a document has moved and its old path or old ID is listed as an alias
When: the old reference is resolved
Then: the resolver returns the canonical document ID and path without duplicating the document record.

Scenario: Duplicate identifiers are rejected

Given: two markdown documents declare the same ID or alias
When: registry validation runs
Then: validation fails with both conflicting document paths in the error output.

Scenario: Broken document references are visible before release

Given: a document references a missing related document ID
When: registry validation runs in the normal test/CI command
Then: validation reports the missing reference and exits non-zero.

Scenario: Portal can browse document metadata

Given: imported process docs exist in content/
When: the registry metadata endpoint is requested
Then: the response includes stable IDs, titles, doc types, paths, tags, systems, and summaries suitable for frontend search/filter UI.

Out of Scope

• Migrating every existing process document to perfect metadata in this issue.
• Building task UI or workflow UI for selecting documents.
• Implementing task-to-process-doc linking fields beyond the resolver contract; that is tracked in Add task-to-process-doc links #6.
• Defining the full Podcast workflow; that is tracked in Define the Podcast workflow #8 and Build the Podcast end-to-end slice #9.
• Rewriting the imported docs app architecture or replacing the existing search index.

[alexeygrigorev]

Grooming complete. Scope now covers the document ID convention, registry scanner, resolver/API behavior, validation, focused tests, dependencies on #2/#3, and blockers for #6/#8/#9. Labels updated for implementation readiness.

[alexeygrigorev]

Shipped in a5b4a48.

What changed:

• Added a backend document registry with canonical records, generated fallback IDs, stable_id, aliases, paths, related docs, and validation.
• Added /docs/registry and /docs/resolve?ref=... for canonical IDs, aliases, repo paths, visible paths, doc: refs, and wiki refs.
• Wired registry validation into search-index build so duplicate IDs, duplicate aliases, alias conflicts, invalid IDs, and broken related_docs fail in CI.
• Documented the stable ID convention for SOPs, templates, task templates, references, and playbooks.
• Seeded the podcast task-template and create-podcast-document SOP with explicit stable IDs.

Verification:

• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app -> 68 passed.
• uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index -> indexed 357 documents.
• git diff --check passed.
• Tester final verdict: PASS.
• PM acceptance: accepted.
• GitHub Actions: Deploy DataOps V1 Lambda 28297571976 succeeded; Validate DataOps Content 28297571995 succeeded.