Decide separate repository strategy for operations documents

[alexeygrigorev]

Decide separate repository strategy for operations documents

Status: pending
Tags: docs, process-docs, migration, data, research, P0
Depends on: None
Blocks: #25, #33, #34

Scope

Decide and document the repository boundary for DataOps operational knowledge before we move process documents, templates, prompts, or validation workflows out of the application repo.

This is an internal architecture/process planning issue. The output should be an ADR or equivalent decision document plus a concrete migration plan. It should not create repositories, move files, change runtime configuration, or implement sync code.

The decision must use the current V1 direction and repository boundaries as context:

• DataTalksClub/dataops is the product/runtime repo for ops.dtcdev.click, including the portal, Lambda APIs, work-engine, assistant modules, tests, app deployment, and small development seeds.
• content/ is the current transitional home for imported SOPs, references, images, prompts, indexes, and task templates.
• content/tasks/templates/ contains DataTasks workflow templates that encode operational process and must be preserved independently from runtime state.
• assistants/podcast/ is the canonical in-repo Podcast Assistant module. Its code and tests belong with the product repo, while its prompts/process knowledge/template inputs may need a documented knowledge-repo boundary.
• Runtime workflow state, task instances, reminders, artifact metadata, assistant job metadata, and audit events belong in DynamoDB and portable exports, not in the process-doc repository.
• Private or bulky artifacts such as raw uploads, recordings, transcripts, invoices, receipts, generated guest-specific podcast documents, and DynamoDB exports must remain outside public process docs.
• ../dtc-operations, ../datatasks, and ../podcast-assistant remain read-only source systems unless a separate issue explicitly scopes source-repo changes.

The recommended ADR should evaluate whether the long-term knowledge repo is DataTalksClub/dataops-knowledge, DataTalksClub/dataops-docs, or another name, and whether it should be public or private.

Acceptance Criteria

• An internal ADR or decision document is added under docs/ and records the chosen repository boundary, repository name, visibility, ownership, and rationale.
• The ADR explicitly distinguishes app/runtime code in DataTalksClub/dataops, canonical process knowledge in the future knowledge/docs repo, shared/account infrastructure in DataTalksClub/aws-infra, runtime state in DynamoDB, and private/bulky artifacts in external storage such as S3/Dropbox/Google Drive.
• The ADR decides whether the knowledge repo contains only content/ or also task templates, workflow definitions, screenshots/images, process metadata, prompts, assistant process instructions, and lightweight validation/index files.
• The ADR explicitly decides the canonical Git location and preferred format for DataTasks templates currently in content/tasks/templates/, including how those templates stay preserved independently of DynamoDB/work-engine runtime state.
• The ADR documents the assistant boundary for assistants/podcast/: what remains product code in dataops, what knowledge/prompts/templates may move to the process repo, and what generated/private assistant outputs must stay outside Git.
• A migration plan inventories current directories to keep in dataops, move to the knowledge repo, leave as runtime/external artifacts, or defer. The plan must include content/, content/tasks/templates/, content/images/, content/prompts/, content/indexes/, assistants/podcast/process/, assistants/podcast/templates/, and assistants/podcast/knowledge_base/.
• The plan identifies runtime configuration changes needed for Lambda/portal content reads, portal edits, content cache/index refresh, and any GitHub token or repository permission changes.
• The plan describes how portal edits should commit back to the knowledge repo, including branch strategy, review expectations, commit authoring, and rollback/revert behavior.
• The plan describes CI expectations for the knowledge repo, including process-doc validation, template schema validation, internal link/image checks, search-index generation, and how a docs push refreshes the deployed portal cache without requiring an app redeploy.
• The decision includes an issue-tracking model for document work: whether docs issues live in dataops, in the future knowledge repo, or in both with cross-links.
• The decision preserves data-safety/export requirements: canonical templates and process docs are Git-backed, runtime task/workflow state remains exportable separately, and private/generated operational data is not moved into a public docs repo.
• The issue report lists follow-up implementation issues to create after the ADR is accepted, including repository creation, content migration, template sync/loading, portal config changes, docs CI, and cache refresh wiring.

Test Scenarios

Scenario: ADR separates knowledge from runtime state

Given the current V1 plan, imported content, task templates, and assistant module boundaries
When the ADR is reviewed
Then it clearly states which repository owns process knowledge, which system owns runtime task/workflow state, and which private artifacts stay outside Git.

Scenario: DataTasks templates stay recoverable

Given task templates currently live in content/tasks/templates/ and describe repeatable operations
When the boundary decision is applied
Then the templates have a canonical Git-backed home and a documented sync/load path into the work-engine without making DynamoDB the only source of truth.

Scenario: Portal edits have an ownership path

Given an operator edits an SOP or workflow template through the portal
When the future implementation follows the ADR
Then the edit is written to the knowledge repo with the documented branch/review/permission model, not silently mixed into app-code deployment changes.

Scenario: Assistant knowledge is not mixed with private outputs

Given Podcast Assistant has process instructions, templates, knowledge-base files, generated documents, inbox files, and logs
When the migration plan classifies assistant directories
Then reusable process knowledge is separated from assistant product code and private/generated outputs.

Scenario: Docs changes can refresh the live portal safely

Given a process-doc or template change lands in the future knowledge repo
When CI validation passes
Then the plan explains how the deployed portal refreshes content/search/index data without requiring a new Lambda app deployment.

Out of Scope

• Creating DataTalksClub/dataops-knowledge, DataTalksClub/dataops-docs, or any other repository.
• Moving files out of dataops.
• Implementing Lambda/GitHub configuration changes.
• Implementing template sync, cache refresh, docs CI, or portal edit commit behavior.
• Changing ../dtc-operations, ../datatasks, or ../podcast-assistant.
• Migrating runtime state, DynamoDB data, generated assistant outputs, recordings, transcripts, invoices, receipts, or other private/bulky artifacts into Git.
• Using user-facing prose tooling for this internal architecture/process document.

Dependencies

• Use .goal-v1.md as the product goal: the operator experiences one unified workflow-first workspace even if repositories remain separated by ownership.
• Use docs/repository-structure-recommendation.md as the current recommendation baseline, but update the decision if the ADR finds a stronger DataOps-specific boundary.
• Use _docs/import-log.md as the source-state and read-only source-repo policy.
• Use current content/, content/tasks/templates/, work-engine/docs/templates.md, and assistants/podcast/ paths as the inventory baseline.
• Keep the decision compatible with the AI Shipping Labs-derived process in _docs/PROCESS.md: PM grooming, specialist review if needed, Tester verification, PM acceptance, local merge/push, and On-Call monitoring for any later implementation work.

[alexeygrigorev]

Additional requirement: DataTasks templates should be considered part of the process/document repository boundary.

Rationale:

• Templates encode the repeatable operational process, not just app runtime seed data.
• If the app/runtime database is lost or rebuilt, templates should remain preserved in Git.
• Template changes should be reviewable like process document changes.
• The docs/process repo layout should decide where templates live alongside SOPs, playbooks, podcast workflows, and related metadata.

Layout options to evaluate in the ADR:

• content/ for human-readable docs and templates/ for machine-readable task templates.
• processes/<domain>/docs plus processes/<domain>/templates for colocating each workflow family.
• Keep templates as YAML/JSON with generated views in the portal.
• Maintain a migration path from current DataTasks seed templates into Git-backed canonical templates.

Acceptance criteria addition:

• ADR explicitly decides the canonical Git location and format for DataTasks templates.
• The implementation plan includes loading/syncing task templates from Git and preserving them independently of any runtime datastore.

[alexeygrigorev]

PM grooming complete for #47.

I rewrote the issue into the standard DataOps groomed format with Status, Tags, Depends on, Blocks, Scope, Acceptance Criteria, Test Scenarios, Out of Scope, and Dependencies.

Key grooming decisions:

• Kept this as an internal architecture/process planning issue, not implementation.
• Required an ADR under docs/ plus a concrete migration/runtime integration plan.
• Made DataTasks templates a first-class boundary decision because they encode process and must be preserved independently of DynamoDB runtime state.
• Added explicit boundaries for dataops, the future knowledge/docs repo, aws-infra, DynamoDB, external/private storage, and Podcast Assistant code versus knowledge versus generated outputs.
• Excluded repository creation, file moves, runtime config changes, source-repo edits, and user-facing prose tooling.
• Added research and data labels while keeping docs, process-docs, migration, and P0.

Ready for architecture/process implementation.

[alexeygrigorev]

SWE/Architect handoff for #47

Changed files:

• Added docs/decisions/dataops-knowledge-repository.md with the ADR for the future DataTalksClub/dataops-knowledge boundary, visibility, ownership, migration inventory, DataTasks template format/home, assistant boundary, runtime config implications, portal edit commit model, CI expectations, issue tracking model, data-safety/export requirements, and follow-up implementation issues.
• Updated docs/README.md to link the new decisions directory and ADR.

Acceptance criteria status:

• All Decide separate repository strategy for operations documents #47 acceptance criteria are satisfied and checked in the issue body.
• No repositories were created, no files were moved out of dataops, no runtime config was changed, and no source repos outside dataops were edited.

Verification run:

• git diff --check passed.
• ADR section presence check passed for decision, repository boundaries, DataTasks template decision, assistant boundary, migration inventory, runtime config implications, portal edit commit model, CI expectations, issue tracking model, data-safety/export requirements, and follow-up implementation issues.
• Related-doc existence check passed for docs/repository-structure-recommendation.md, docs/v1-runtime-architecture.md, and docs/v1-execution-data-safety.md.
• Docs index link check passed for docs/README.md referencing decisions/dataops-knowledge-repository.md.

Follow-up implementation issues to create after ADR acceptance:

1. Create DataTalksClub/dataops-knowledge with branch protection, ownership, visibility, and initial empty structure.
2. Add knowledge-repo schemas and CI for frontmatter, stable IDs, internal links, image checks, workflow-template validation, prompt validation, and search-index generation.
3. Run data-safety review for content/, content/images/, content/prompts/, content/tasks/templates/, and assistant knowledge/example paths.
4. Migrate content/ process docs and safe images into the knowledge repo.
5. Convert content/tasks/templates/*.md to workflow-templates/*.yaml and generate any needed Markdown views.
6. Implement work-engine template loading/sync from Git-backed YAML with versioning and source commit tracking.
7. Add Lambda/portal configuration for reading from dataops-knowledge, including local development fallback.
8. Implement portal edit commits to the knowledge repository with branch, validation, authoring, and revert behavior.
9. Wire knowledge-repo CI to refresh the deployed portal cache/search index without app redeploy.
10. Classify and migrate reusable Podcast Assistant process knowledge, templates, prompts, and safe knowledge-base summaries.
11. Move Podcast Assistant generated/private outputs to private artifact storage and attach artifact metadata to workflow records.
12. Add admin/status visibility for loaded knowledge repo branch, commit SHA, index build time, and template sync version.

Skipped checks:

• Runtime/backend/frontend/work-engine test suites were skipped because this issue is ADR-only and changed no runtime code, content served by the portal, schemas, or configuration.
• Search-index build was skipped because no content/** files or served docs metadata were changed; docs/ is repo-meta documentation and is not served by the Lambda docs frontend.
• Stylint/user-facing prose tooling was intentionally not run per the issue instructions for internal architecture/process documentation.

Not performed:

• No commit, push, PR, issue close, repository creation, file migration, or runtime configuration change.

[alexeygrigorev]

TEST PASS

Tester verification for #47 completed.

Evidence and commands:

• Read AGENTS.md and _docs/PROCESS.md.
• Read issue: gh issue view 47 --repo DataTalksClub/dataops --json title,state,labels,body.
• Read SWE/Architect handoff: gh api repos/DataTalksClub/dataops/issues/comments/4821917443 --jq '.body'.
• Scoped diff check: git status --short --untracked-files=all showed only M docs/README.md and ?? docs/decisions/dataops-knowledge-repository.md.
• Diff/path check: git diff --name-only plus git ls-files --others --exclude-standard showed only docs/README.md and docs/decisions/dataops-knowledge-repository.md.
• Whitespace check: git diff --check exited 0.
• ADR section check confirmed these headings are present: Decision, Repository Boundaries, Knowledge Repository Contents, DataTasks Template Decision, Assistant Boundary, Migration Inventory, Runtime Configuration Implications, Portal Edit Commit Model, Knowledge Repository CI, Issue Tracking Model, Data Safety And Export Requirements, Follow-Up Implementation Issues, Non-Goals.
• Inventory coverage check confirmed the ADR covers all required named paths: content/, content/tasks/templates/, content/images/, content/prompts/, content/indexes/, assistants/podcast/process/, assistants/podcast/templates/, and assistants/podcast/knowledge_base/.
• Template target check confirmed all current 11 content/tasks/templates/*.md templates have matching proposed YAML targets: book-of-the-week, course, maven-ll, newsletter, office-hours, oss, podcast, social-media, tax-report, webinar, and workshop.
• Link/reference check confirmed docs/README.md references decisions/ and decisions/dataops-knowledge-repository.md, and the ADR related docs exist.
• Repository creation check: gh repo view DataTalksClub/dataops-knowledge --json nameWithOwner,visibility and gh repo view DataTalksClub/dataops-docs --json nameWithOwner,visibility both returned GitHub “Could not resolve to a Repository”, so the candidate repos were not created.
• Source repo boundary check: ../dtc-operations was clean; ../datatasks had an existing unrelated M e2e/.auth-state.json; ../podcast-assistant was not present at the expected adjacent path. I did not modify or revert any adjacent repo.

Acceptance criteria:

• PASS: ADR added under docs/ and records chosen boundary, repo name DataTalksClub/dataops-knowledge, public-by-default visibility with pre-migration data review, ownership, and rationale.
• PASS: ADR distinguishes dataops app/runtime code, future knowledge repo, DataTalksClub/aws-infra, DynamoDB runtime state, and S3/Dropbox/Google Drive private/bulky artifacts.
• PASS: ADR decides the future knowledge repo contains more than content/, including workflow templates, images, process metadata/indexes, prompts, assistant process instructions, schemas/tests/scripts, and validation/index files.
• PASS: ADR chooses DataTalksClub/dataops-knowledge/workflow-templates/*.yaml with strict JSON Schema for DataTasks templates and states DynamoDB is not the source of truth.
• PASS: ADR documents the Podcast Assistant boundary: code/tests remain in dataops, reusable process knowledge/templates/prompts may move after review, generated/private outputs stay outside Git.
• PASS: Migration inventory includes every required path and classifies keep/move/split/defer/external handling.
• PASS: Runtime config implications cover Lambda/portal reads, portal edits, cache/index refresh, GitHub token/App permissions, template sync source, and local fallback.
• PASS: Portal edit commit model covers target repo, branch strategy, review expectations, authoring, and rollback/revert behavior.
• PASS: Knowledge repo CI expectations cover process-doc validation, template schema validation, link/image checks, search-index generation, and deployed cache refresh without app redeploy.
• PASS: Issue tracking model keeps dataops primary, allows knowledge-only issues in the future repo after creation, and requires cross-links/blocking for cross-repo work.
• PASS: Data-safety/export requirements preserve Git-backed docs/templates, separate runtime exports, and keep private/generated operational data out of a public docs repo.
• PASS: Follow-up implementation issues are listed for repo creation, schemas/CI, data-safety review, content migration, template conversion/loading, portal config, portal edit commits, cache refresh wiring, assistant migration, private artifact handling, and admin/status visibility.

Test scenarios:

• PASS: ADR separates process knowledge, runtime state, and private artifacts.
• PASS: DataTasks templates remain recoverable from Git with a documented sync/load path into work-engine runtime records.
• PASS: Portal edits have a documented knowledge-repo commit path instead of app-code deployment changes.
• PASS: Assistant process knowledge is separated from product code and private/generated outputs.
• PASS: Docs/template pushes have a planned CI-to-cache-refresh path without Lambda app redeploy.

Negative/out-of-scope checks:

• PASS: No repository was created.
• PASS: No files were moved out of dataops.
• PASS: No runtime configuration was changed.
• PASS: No source repo outside dataops was modified by this review.
• PASS: Stylint was not run.
• PASS: No commit, push, PR, issue close, or file edit was performed.

UI/screenshots: Not required; no UI changed.

[alexeygrigorev]

ACCEPTED

PM final acceptance for #47 passes from the product and architecture perspective.

Reasons:

• The ADR gives a clear long-term repository strategy: DataTalksClub/dataops remains the product/runtime repository, and the future DataTalksClub/dataops-knowledge repository becomes the canonical home for process knowledge after validation, sync, portal edit, and cache refresh support exist.
• The decision preserves the unified V1 operator UX. It explicitly keeps DataOps as one workflow-first portal while allowing docs, templates, prompts, and assistant process knowledge to move at a different review/release speed from app code.
• DataTasks templates are treated correctly as durable operational process, not runtime state. The ADR chooses DataTalksClub/dataops-knowledge/workflow-templates/*.yaml with strict schema validation, Git-backed recovery, runtime sync/loading, template version tracking, and source commit metadata.
• Runtime state and private artifacts are protected. DynamoDB remains the home for mutable tasks, bundles, reminders, assistant job metadata, audit events, and runtime template records, while raw uploads, transcripts, invoices, generated guest-specific outputs, and bulky/private files stay in private artifact storage rather than public Git.
• The Podcast Assistant boundary is sufficiently clear: assistant code/tests/integration remain in dataops; reusable prompts/process/templates may move only after review; generated and private outputs stay outside Git and are represented through workflow artifact metadata.
• The migration inventory covers the required current paths and classifies what moves, stays, splits, is deferred, or remains external.
• The runtime and portal implications are actionable: Lambda content reads, local fallback, GitHub permissions, template sync, portal edit commits, branch/review behavior, revert handling, cache/index refresh, and admin/status visibility are all called out.
• The follow-up implementation list is concrete enough to drive the next issues for repo creation, schemas/CI, data-safety review, migration, template conversion/loading, portal config, portal edit commits, cache refresh, assistant migration, private artifact handling, and status visibility.

Tester PASS evidence is sufficient for this ADR-only change, and the scoped diff is limited to docs/README.md plus docs/decisions/dataops-knowledge-repository.md. No repo creation, file migration, runtime config change, source-repo edit, commit, push, PR, or issue close is approved or performed in this acceptance step.

[alexeygrigorev]

On-call report for the main push containing #46 and #47.

Commits checked:

• a1f99de Decide DataOps knowledge repository boundary (Closes Decide separate repository strategy for operations documents #47)
• 08de397 Define DataOps design system (Closes Create shared DataOps design system and component tokens #46)

Evidence:

• gh run list --repo DataTalksClub/dataops --limit 20 shows the latest workflow run is Deploy DataOps V1 Lambda for 37bcafb at 2026-06-27T21:24:08Z: https://github.com/DataTalksClub/dataops/actions/runs/28302211699. No runs are listed for a1f99de or 08de397.
• Actions API head-SHA queries for both pushed commits returned no workflow runs.
• Commit check-runs API returned total_count=0 for both pushed commits.
• Changed files were docs-only: Decide separate repository strategy for operations documents #47 changed docs/README.md and docs/decisions/dataops-knowledge-repository.md; Create shared DataOps design system and component tokens #46 changed docs/README.md and docs/design-system.md.
• Current push workflow path filters are for assistants/**, content/**, frontend/**, lambda-functions/**, scripts/**, tests/docs_app/**, work-engine/**, package/lock manifests, and workflow files. They do not include docs/**.

Classification: expected skip/no-op. No CI/deployment infrastructure failure, commit-related code/test failure, content/search validation failure, external account/secret issue, or flake observed.

Current status: #46 and #47 are closed by the pushed commits; no relevant workflows were expected or triggered for these docs-only changes.

Next step: none for on-call. Related report: #46.