Add dataops-knowledge migration scaffold and template validation

[alexeygrigorev]

Add dataops-knowledge migration scaffold and template validation

Status: pending
Tags: enhancement, docs, migration, process-docs, testing, data, P1
Depends on: #47 (closed)
Blocks: Future repository creation, content migration, template conversion, template sync/loading, portal edit commits, and deployed cache refresh issues.

Scope

Build the first safe implementation slice for the future DataTalksClub/dataops-knowledge repository without creating that repository, moving production content, or changing the operator-facing read path.

This issue should add a migration scaffold and validation contract inside DataTalksClub/dataops so later agents have an exact, testable target for the knowledge repository layout and workflow-template YAML migration. The deployed portal, work-engine, and operator UX must continue to use the existing in-repo content/ fallback during this slice.

Expected implementation shape:

• Add a repo-template/scaffold directory for the future dataops-knowledge repository, using a clear name such as templates/dataops-knowledge/, scaffolds/dataops-knowledge/, or another project-consistent path.
• The scaffold must represent the target layout from docs/decisions/dataops-knowledge-repository.md: content/, workflow-templates/, assistant-prompts/, assistant-process/, examples/, images/, indexes/, schemas/, scripts/, and tests/.
• Include concise README/guidance in the scaffold stating that it is a migration target only and is not yet the live source for the portal or work-engine.
• Add strict validation assets for the future workflow-template YAML format, preferably a JSON Schema under the scaffold's schemas/ directory.
• Add a machine-readable migration inventory/manifest that maps every current content/tasks/templates/*.md file to its future workflow-templates/*.yaml target and stable template ID. The manifest must cover exactly the current 11 templates: book-of-the-week, course, maven-ll, newsletter, office-hours, oss, podcast, social-media, tax-report, webinar, and workshop.
• Add a validation command in the existing DataOps validation/tooling stack, preferably under lambda-functions/src/lambda_functions/, that checks the scaffold, schema, and migration manifest from the current repo. It should fail on missing scaffold directories, invalid schema JSON, missing or duplicate template mappings, missing current Markdown template files, missing stable IDs, non-task-template doc types for current templates, and target filenames outside workflow-templates/*.yaml.
• Add focused automated tests for the new validator, including at least one passing fixture and failure coverage for missing template mappings, duplicate target IDs or paths, and invalid target path/schema shape.
• Wire the new validation into the appropriate content/planning CI path so future edits to the scaffold, manifest, schemas, task templates, or validator are checked automatically.
• Preserve the current operator experience: no deployed portal route, docs search, work-engine seed, runtime task creation, or portal edit behavior should start reading from the scaffold or future repo in this slice.

Process Curator requirements:

• Keep stable document IDs as the migration boundary. File paths may change later; current task-template IDs and future template IDs must remain explicit and testable.
• Keep content/tasks/templates/*.md as the transitional canonical source for this issue.
• Do not copy SOPs, images, prompts, assistant process files, examples, or generated indexes into the scaffold except for placeholder README files, .gitkeep-style placeholders, schemas, tests, scripts, or minimal synthetic fixtures needed to validate the scaffold.
• If the implementation includes an example workflow-template YAML file, it must be clearly synthetic or fixture-only unless the issue explicitly explains why copying production template content is safe. Production template conversion belongs to a later issue.

Acceptance Criteria

• A future dataops-knowledge scaffold exists in dataops with the target top-level directories from the accepted ADR and guidance that the scaffold is not yet live production content.
• The scaffold includes a strict workflow-template schema covering the core contract from the ADR: stable id, runtime type, name, schema_version, trigger model, bundle links, phases/stage mapping, tasks with stable task IDs, scheduling offsets or rules, required proof/link declarations, default assignee references by stable role/user ID, instruction_doc_id/source document references, and optional migration source metadata.
• A machine-readable migration manifest maps exactly all current content/tasks/templates/*.md files to future workflow-templates/*.yaml targets with stable IDs, with no duplicates or missing current files.
• A repository validation command checks the scaffold, schema, and manifest and exits non-zero with actionable messages for missing directories, invalid JSON schema, missing/duplicate mappings, missing Markdown template files, invalid template frontmatter, and invalid target YAML paths.
• Automated tests cover the validator success path and the required failure cases without depending on production secrets, GitHub writes, or the future external repository.
• CI validates the new scaffold/schema/manifest/validator on relevant path changes.
• Current portal/content behavior remains unchanged: Lambda/docs search still reads from content/, work-engine runtime templates are not loaded from the scaffold, and portal edit commits are not redirected to a new repository.
• No files are moved out of content/, assistants/, work-engine/, or source repos outside dataops.
• No DataTalksClub/dataops-knowledge repository is created and no GitHub write/token/branch-protection automation is added in this slice.
• Source repos outside dataops, including ../dtc-operations, ../datatasks, and ../podcast-assistant, remain read-only.

Test Scenarios

Scenario: Scaffold defines the future repository without changing runtime behavior

Given the accepted ADR and the current content/ fallback
When the validation command runs against the repository
Then it confirms the future layout, schemas, and migration manifest while the portal and work-engine still use the current in-repo content paths.

Scenario: Current Markdown templates are fully inventoried

Given the 11 current files under content/tasks/templates/
When the migration manifest is validated
Then every current file is mapped to one unique workflow-templates/*.yaml target and one stable template ID, with no missing or duplicate mappings.

Scenario: Bad migration mappings fail clearly

Given a fixture manifest with a missing template, duplicate target path or ID, invalid target path, or missing source Markdown file
When the validator runs in tests
Then it exits with a useful error that tells the next agent what to fix.

Scenario: Workflow-template schema is enforceable before conversion

Given a minimal valid fixture workflow template and invalid fixtures missing required fields
When schema validation runs in tests
Then the valid fixture passes and invalid fixtures fail before any production template conversion begins.

Scenario: Operator UX stays unified during the transition

Given the scaffold exists in the repo
When an operator opens the current portal or a workflow task references process docs
Then the app continues resolving docs from content/ and existing stable document IDs; the scaffold is not exposed as a disconnected second operator tool.

Required Verification

Software Engineer should run and report exact commands and exit codes for:

• git diff --check
• Focused tests for the new validator, for example uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app/test_validate_knowledge_repo.py
• The new validation command against the real repository, using its documented CLI invocation.
• Existing docs/content link validation: uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content
• Search-index build to prove current content remains readable: cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index
• Full docs-app test workflow unless the implementation only adds isolated schema files and a narrowly scoped validator; if narrowed, explain why the focused validator tests plus content validation prove the acceptance criteria.

Tester should rerun the relevant verification independently. Screenshots are not required unless the implementation changes portal UI, routes, or rendered content.

Out of Scope

• Creating DataTalksClub/dataops-knowledge or changing GitHub repository settings, branch protection, repository visibility, secrets, or tokens.
• Moving, copying, or deleting production SOPs, images, prompts, assistant process files, examples, generated indexes, or task templates into a new canonical location.
• Converting the 11 Markdown task templates into production canonical YAML files.
• Implementing work-engine template loading/sync from Git-backed YAML, template version tracking, or source commit tracking.
• Changing Lambda/portal configuration to read from dataops-knowledge.
• Implementing portal edit commits to the knowledge repository.
• Refreshing deployed portal cache/search index from an external knowledge repository.
• Performing the data-safety review for content, images, prompts, assistant knowledge, examples, or private artifacts.
• Migrating runtime state, DynamoDB data, assistant outputs, recordings, transcripts, invoices, receipts, sponsor/client records, or other private/bulky files.
• Modifying ../dtc-operations, ../datatasks, or ../podcast-assistant.

Dependencies

• Decide separate repository strategy for operations documents #47 is the accepted decision baseline. Follow docs/decisions/dataops-knowledge-repository.md, especially the rule to keep content/ in dataops until read/sync/edit/refresh support exists.
• Use docs/repository-structure-recommendation.md, docs/STRUCTURE.md, and docs/README.md for current content structure, frontmatter, stable ID, and repo-meta conventions.
• The next implementation owner is Software Engineer.
• Later issues must handle repository creation, data-safety review, production YAML conversion, work-engine sync/load, portal config, portal edit commits, external-repo CI, deployed cache refresh, assistant knowledge migration, and private artifact handling.

[alexeygrigorev]

PM + Process Curator grooming complete for #59.

I rewrote the issue into an agent-ready first implementation slice: add a dataops-knowledge migration scaffold, workflow-template schema, template migration manifest, validator, tests, and CI wiring inside dataops.

Key grooming decisions:

• Keep the operator UX unified: the portal, search, work-engine, and portal edit behavior stay on the current in-repo content/ fallback for this slice.
• Do not create DataTalksClub/dataops-knowledge, move production content, convert the 11 production templates to canonical YAML, or change runtime read/sync/edit/refresh behavior yet.
• Treat stable IDs and the manifest as the migration boundary for the future repository split.
• Require validation and tests before any later content migration or repository creation work.

Dependencies and sequencing:

• Depends on Decide separate repository strategy for operations documents #47, which is closed and accepted.
• Later issues still need repository creation, data-safety review, production YAML conversion, work-engine sync/load, portal config, portal edit commits, external-repo CI, deployed cache refresh, assistant knowledge migration, and private artifact handling.

Labels updated: removed needs grooming; added enhancement; kept docs, migration, process-docs, testing, data, and P1.

Next owner: Software Engineer.

[alexeygrigorev]

Software Engineer handoff for #59.

Summary:

• Added templates/dataops-knowledge/ migration scaffold with the ADR top-level directories and README guidance that it is not the live source for portal/search/work-engine/edit behavior.
• Added strict workflow-template JSON Schema under templates/dataops-knowledge/schemas/.
• Added indexes/workflow-template-migration-manifest.json mapping exactly the 11 current content/tasks/templates/*.md files to future workflow-templates/*.yaml targets and stable IDs.
• Added lambda_functions.validate_knowledge_repo CLI validation for scaffold directories, schema JSON/shape, manifest coverage, duplicate IDs/source/targets, target path shape, missing source files, stable IDs, and current task-template frontmatter.
• Added focused tests for success, missing mappings, duplicate IDs/paths, invalid target path/missing source, invalid schema shape, and synthetic workflow-template schema validation.
• Wired scaffold validation into content/planning CI path triggers and jobs.

Changed files:

• .github/workflows/validate-dataops-content.yml
• .github/workflows/validate-planning-docs.yml
• scripts/validate_planning_docs.py
• lambda-functions/src/lambda_functions/validate_knowledge_repo.py
• tests/docs_app/test_validate_knowledge_repo.py
• templates/dataops-knowledge/**

Verification:

• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app/test_validate_knowledge_repo.py -> PASS, 8 passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_knowledge_repo --repo-root . --scaffold-root templates/dataops-knowledge -> PASS, scaffold validation passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content -> PASS, docs link validation passed.
• cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index -> PASS, indexed 357 documents.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app -> PASS, 110 passed, 3 skipped.
• uv run --with pytest python -m pytest tests/planning_docs -> PASS, 7 passed.
• git diff --check -> PASS.

Acceptance notes:

• Current runtime behavior remains unchanged. No portal route, docs search, work-engine seed/load path, runtime task creation, or portal edit behavior now reads from templates/dataops-knowledge/.
• No production content, assistant prompts/process files, images, examples, or generated indexes were moved into the scaffold.
• No external DataTalksClub/dataops-knowledge repository was created.
• No source repos outside dataops were modified.
• No commit made.

Remaining gaps for later issues:

• Create the external knowledge repo after data-safety/process approval.
• Convert production templates to YAML.
• Add work-engine sync/load, portal config/edit commits, external-repo CI, deployed cache refresh, assistant knowledge migration, and private artifact handling.

READY FOR TEST

[alexeygrigorev]

PROCESS PASS for #59.

Boundary review:

• templates/dataops-knowledge/ is a migration scaffold only. It contains README placeholders, the workflow-template schema, and the migration manifest; no production SOPs, prompts, images, examples, generated indexes, or converted workflow YAML were moved into it.
• Root content/tasks/templates/*.md remains the transitional canonical source, and the manifest maps exactly the current 11 task templates to future workflow-templates/*.yaml targets with stable IDs.
• The validator checks scaffold directories, schema shape, manifest coverage, duplicate IDs/sources/targets, target path shape, current Markdown source existence, stable frontmatter IDs, and task-template doc type.
• CI wiring validates the scaffold on relevant scaffold/template/validator path changes while current Lambda/docs search still builds from root content/. No portal route, work-engine load path, runtime task creation, portal edit commit behavior, GitHub repo creation, or source-repo modification was introduced.

Verification rerun by Process Curator:

• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app/test_validate_knowledge_repo.py -> PASS, 8 passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_knowledge_repo --repo-root . --scaffold-root templates/dataops-knowledge -> PASS.
• uv run --with pytest python -m pytest tests/planning_docs -> PASS, 7 passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content -> PASS.
• cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index -> PASS, indexed 357 documents.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app -> PASS, 110 passed, 3 skipped.
• git diff --check -> PASS.

No Process Curator fixes were needed. #59 can proceed to PM acceptance after Tester passes.

[alexeygrigorev]

TEST PASS

Tester gate for #59 passes. I reviewed the issue, SWE handoff, changed files, scaffold, validator, schema, manifest, CI wiring, and runtime boundaries.

Evidence:

• git diff --check -> exit 0.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app/test_validate_knowledge_repo.py -> exit 0, 8 passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_knowledge_repo --repo-root . --scaffold-root templates/dataops-knowledge -> exit 0, Knowledge repository scaffold validation passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content -> exit 0, docs link validation passed; anchor validation deferred by tool.
• cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index -> exit 0, indexed 357 documents.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app -> exit 0, 110 passed, 3 skipped.
• uv run --with pytest python -m pytest tests/planning_docs -> exit 0, 7 passed.
• uv run --with pytest python scripts/validate_planning_docs.py -> exit 0, planning docs validation passed.
• uv run --with pyyaml python - <<'PY' ... yaml.safe_load(...) ... PY for .github/workflows/validate-dataops-content.yml and .github/workflows/validate-planning-docs.yml -> exit 0, both workflows parsed with expected jobs. actionlint is not installed locally, so YAML parse was the available static workflow check.

Acceptance criteria verification:

• Scaffold exists at templates/dataops-knowledge/ with ADR top-level directories: content/, workflow-templates/, assistant-prompts/, assistant-process/, examples/, images/, indexes/, schemas/, scripts/, tests/.
• Scaffold README states it is migration-target-only and not the live source for portal, docs search, work-engine templates, or portal edit commits.
• schemas/workflow-template.schema.json is strict (additionalProperties: false) and covers stable id, runtime type, name, schema version, trigger, bundle links, phases/stages, tasks with stable task IDs, schedule offsets/rules, proof/link declarations, role/user assignee refs, instruction/source document IDs, and optional migration source metadata.
• indexes/workflow-template-migration-manifest.json maps exactly the current 11 content/tasks/templates/*.md files to unique workflow-templates/*.yaml targets with stable IDs: book-of-the-week, course, maven-ll, newsletter, office-hours, oss, podcast, social-media, tax-report, webinar, workshop.
• Validator checks scaffold dirs, schema JSON/shape, manifest coverage, duplicates, missing source files, stable IDs, task-template frontmatter, and invalid target YAML paths; focused tests cover success and required failure paths.
• CI wiring added to content/planning workflows and planning-doc workflow path validation.
• Runtime behavior unchanged: git diff --name-only -- content assistants work-engine frontend lambda-functions/src/lambda_functions/docs_app.py lambda-functions/src/lambda_functions/search_handler.py lambda-functions/src/lambda_functions/build_search_index.py lambda-functions/src/lambda_functions/doc_registry.py produced no changed files, and rg found no scaffold/runtime-read references in runtime source outside the new validator.
• No content was moved: git ls-files --deleted content assistants work-engine frontend lambda-functions/src -> 0, and git status --short --untracked-files=all shows only the expected workflow/planning changes plus new validator/tests/scaffold files.
• No DataTalksClub/dataops-knowledge repository creation or GitHub token/branch-protection automation was added.
• Source repos outside dataops were not modified by this work. dtc-operations status is clean; datatasks has an existing dirty e2e/.auth-state.json auth-state change unrelated to this work; /home/alexey/git/podcast-assistant is present but is not a git repository, so no git status is available there.

Screenshots: not required; this issue does not change portal UI, routes, or rendered operator flows.

I also updated the issue acceptance checkboxes to checked per _docs/PROCESS.md.

Ready for Process Curator review. If Process Curator passes, this is ready for PM acceptance.

[alexeygrigorev]

ACCEPTED - PM acceptance for #59.

From the V1/product perspective, this is a safe first implementation slice for the future dataops-knowledge migration:

• The scaffold at templates/dataops-knowledge/ defines the target repository shape and clearly states it is migration-target-only, not the live source for portal/search/work-engine/edit behavior.
• The implementation preserves the unified operator UX: current docs validation and search indexing still read from content/; no portal route, work-engine runtime load path, runtime task creation path, or portal edit commit path is redirected to the scaffold.
• The migration manifest covers exactly the current 11 content/tasks/templates/*.md templates with stable IDs and future workflow-templates/*.yaml targets.
• The schema and validator establish the contract needed before any production YAML conversion or repository split.
• No production SOPs, prompts, images, examples, generated indexes, runtime state, or source-repo files were moved into the scaffold.
• No DataTalksClub/dataops-knowledge repository creation, GitHub write/token automation, or branch-protection automation was added in this slice.

PM spot-check verification:

• git diff --check -> PASS.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_knowledge_repo --repo-root . --scaffold-root templates/dataops-knowledge -> PASS.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app/test_validate_knowledge_repo.py -> PASS, 8 passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content -> PASS.
• cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/dataops-content-search.index -> PASS, indexed 357 documents.

Blockers: none.

Branch may be committed with Closes #59 after ensuring the untracked scaffold, validator, and test files are included in the commit.