Import DTC Operations into dataops

[alexeygrigorev]

Import DTC Operations into dataops

Status: pending
Tags: migration, portal, process-docs, frontend, backend, infra, testing, P0
Depends on: #1, #12
Blocks: #3

Scope

Import the DTC Operations docs app and process-document system from ../dtc-operations into this repo as Phase 1 of _docs/MERGE_PLAN.md.

The implementation should copy the existing source system into the DataOps target structure while preserving the current behavior enough for the follow-up runnable-app issue (#3) to validate it. Include process content, frontend assets, backend/Lambda docs app code, scripts, templates, tests, and required configuration. Rename lambda-functions/ to backend/ during the import if practical; otherwise keep the original directory temporarily and leave a clear transition note for the follow-up work.

Update repo, organization, service, and deployment references from dtc-operations to dataops where required for the imported code to live in this repository. Record source provenance according to the import-log/source-commit policy. Do not modify ../dtc-operations or any other source repo.

Acceptance Criteria

• The ../dtc-operations source commit and chosen import approach are recorded according to the import-log/source-commit policy.
• DTC Operations process content, templates, references, and related docs assets are copied into the intended dataops paths without intentional content rewrites.
• The DTC Operations frontend is copied into frontend/ and remains wired to the imported docs app assets and routes.
• The DTC Operations backend/Lambda docs app code is copied into backend/ or temporarily retained as lambda-functions/ with a clear reason and next-step note.
• Supporting scripts, templates, docs-app tests, Playwright tests, and required configuration files are copied into appropriate dataops paths.
• Repo, organization, service, and deployment references that must point at DataTalksClub/dataops are updated, without adding developer-local absolute paths.
• Baseline validation commands for the imported lint, search, backend, frontend, and Playwright test surfaces are run, or any blocked command is documented with the exact blocker and follow-up owner.
• ../dtc-operations, ../datatasks, and ../podcast-assistant have no working-tree changes from this issue's implementation.

Test Scenarios

Scenario: Source-proven import is present

Given: the DataOps repo and the ../dtc-operations source checkout
When: the import branch is inspected
Then: the imported files exist under the intended DataOps paths and the source commit is traceable from the import notes

Scenario: Imported docs content can be validated

Given: the imported process docs, templates, scripts, and search assets
When: the documented lint and search validation commands are run
Then: they pass, or the issue comment identifies the exact command, failure, and follow-up needed

Scenario: Docs app is ready for runnable verification

Given: the imported frontend and backend docs app code
When: an engineer follows the documented local setup and validation commands
Then: the repo has enough copied code and configuration for #3 to verify startup, search, editing, lint, image handling, and local save behavior

Out of Scope

• Importing DataTasks into work-engine/.
• Importing Podcast Assistant into assistants/podcast/.
• Building the shared operations dashboard or first end-to-end workflow.
• Redesigning or rewriting the DTC Operations frontend.
• Consolidating the TypeScript work engine into the Python backend.
• Production deployment, secrets setup, or live GitHub write verification.
• Modifying ../dtc-operations, ../datatasks, or ../podcast-assistant.

[alexeygrigorev]

Grooming complete. Issue #2 is now scoped for the Phase 1 DTC Operations import, with dependencies, acceptance criteria, test scenarios, and out-of-scope boundaries. Removed needs grooming and added portal/infra/testing coverage labels.

[alexeygrigorev]

PM triage / acceptance decision: #2 is already satisfied by existing work, so I am closing it as completed.

Evidence checked:

• Import implementation exists in commit cde3ce9 (Import v1 portal sources). That commit brought the DTC Operations portal base into this repo, including content/, frontend/, lambda-functions/, scripts/, templates/, tests/, workflows, root config, and README material.
• Current tracked repo contents include the expected imported surfaces: content/ (2126 tracked files), frontend/ (5), lambda-functions/ (24), scripts/ (11), templates/ (3), tests/docs_app/ (7), and tests/frontend/ (3).
• Source provenance is recorded in _docs/import-log.md: DTC Operations source path ../dtc-operations, repo git@github.com:DataTalksClub/dtc-operations.git, source commit e6f7b43f641389fa58a8e027485f5aac18e8ace5, and destination/copy/exclusion notes. That policy/evidence was hardened in commit ab0b290.
• Backend path is intentionally still lambda-functions/. That is allowed by this issue (“or temporarily retained as lambda-functions/”), and _docs/PROCESS.md plus current deploy/test docs use lambda-functions/ as the DataOps Lambda backend path.
• Required DataOps references for the active deployment path are updated: .github/workflows/deploy-dataops-v1.yml deploys dataops-v1, uses the dataops-github-actions-deploy role, and sets DOCS_GITHUB_REPO=dataops. The remaining dtc-operations strings I found are inherited docs/template defaults or compatibility secret/event names, not blockers for this import issue.
• Validation run during this triage:
  • uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app -> 69 passed.
  • cd lambda-functions && uv run --extra search python -m lambda_functions.build_search_index --docs-dir ../content --output ../.tmp/issue-2-triage-search.index -> indexed 357 documents.
  • git -C ../dtc-operations rev-parse HEAD -> e6f7b43f641389fa58a8e027485f5aac18e8ace5.
  • git -C ../dtc-operations status --short -> clean.
  • git status --short --untracked-files=all in dataops -> clean after removing generated local test artifacts.

Acceptance criteria are covered by the imported/current tree and the validation above. Follow-up runtime verification belongs to #3, which this issue originally blocked; #2 no longer needs implementation.