Define import log and source commit policy

[alexeygrigorev]

Define import log and source commit policy

Status: pending
Tags: docs, migration, P0
Blocks: #4, #7

Current Evidence

_docs/import-log.md already exists and records the initial 2026-06-27 import state for DTC Operations, DataTasks, and Podcast Assistant. It includes source paths, source commits/source state, imported locations, exclusions, and purpose notes.

The issue is not fully satisfied yet because the file does not define the reusable import/source-state policy requested in the intake, does not record validation commands, and does not make the expected fields explicit enough for dependent imports (#4 and #7) to follow without inventing their own convention.

Scope

Update _docs/import-log.md into the canonical policy and register for copied source systems in DataTalksClub/dataops.

The implementation should preserve the existing import evidence and add a clear policy that every import or re-import records enough information for reviewers to identify what was copied, from where, at what source revision/state, what was deliberately excluded, and how the import was validated.

For each copied source system, the log should record:

• Source system name.
• Source path and, when applicable, source repository URL or local source identity.
• Source commit hash for Git-backed sources, or explicit non-Git/local source state when no commit exists.
• Destination path(s) in this repo.
• Copied path groups or major copied contents.
• Deliberate exclusions, especially generated, dependency, cache, secret, local runtime, and private output paths.
• Validation commands run after import, including whether each command passed, failed, or was skipped with a reason.
• Follow-up issue references when an import is transitional or will be moved/normalized later.

Keep this issue documentation-only. Do not copy, move, delete, refactor, or re-import source code as part of #12.

Acceptance Criteria

• _docs/import-log.md contains a named policy section that defines the required fields for future source imports and re-imports.
• The policy states that source repos/directories such as ../dtc-operations, ../datatasks, and ../podcast-assistant are read-only unless a specific issue explicitly scopes changes there.
• The policy explains how to record Git-backed sources with git -C <source> rev-parse HEAD and how to record non-Git local directories such as Podcast Assistant.
• The existing DTC Operations entry records source path, source commit, destination path(s), copied path groups, exclusions, validation commands/statuses, and any known follow-up context.
• The existing DataTasks entry records source path, source commit, destination path(s), copied path groups, exclusions, validation commands/statuses, and dependency/follow-up context for Import DataTasks into work-engine #4.
• The existing Podcast Assistant entry records source path, non-Git source state, current destination path, copied path groups, exclusions, validation commands/statuses, and the follow-up context that Import Podcast Assistant into assistants/podcast #7 should move/canonicalize it under assistants/podcast/.
• Validation command entries are explicit enough that a reviewer can distinguish commands that passed from commands that were not run or were blocked by local environment constraints.
• The final document does not claim unverified commands passed.
• No files outside _docs/import-log.md are changed for this issue.

Test Scenarios

• From the repo root, verify the import log exists and contains the policy and all three current source systems:

  test -f _docs/import-log.md
  rg -n "policy|DTC Operations|DataTasks|Podcast Assistant|Validation" _docs/import-log.md

• For Git-backed local sources that are available, verify the recorded commit values can be reproduced:

  git -C ../dtc-operations rev-parse HEAD
  git -C ../datatasks rev-parse HEAD

• For the local non-Git Podcast Assistant source, verify the source-state statement is accurate:

  test -d ../podcast-assistant
  test ! -d ../podcast-assistant/.git

• Review _docs/import-log.md manually and confirm every source entry has source state, destination path(s), copied paths, exclusions, validation commands/statuses, and follow-up references where relevant.

Dependencies

• Local source directories should be available for verification when possible: ../dtc-operations, ../datatasks, and ../podcast-assistant.
• Import DataTasks into work-engine #4 and Import Podcast Assistant into assistants/podcast #7 should continue to treat Define import log and source commit policy #12 as the source of truth for how they record final import/re-import evidence.

Out of Scope

• Importing or re-importing DataTasks.
• Moving Podcast Assistant from podcast-assistant/ to assistants/podcast/; that remains in Import Podcast Assistant into assistants/podcast #7.
• Renaming lambda-functions/, restructuring backend code, or changing runtime paths.
• Adding automated tests beyond lightweight documentation/source-state verification commands.
• Modifying any source repository or directory outside dataops.

[alexeygrigorev]

PM triage complete.

Decision: not already satisfied by current repo evidence, so I groomed #12 instead of closing it.

Evidence reviewed:

• _docs/import-log.md exists and records initial DTC Operations, DataTasks, and Podcast Assistant import/source state.
• The existing log includes source paths, Git commits or non-Git source state, imported locations, exclusions, and purpose notes.
• It does not yet define a reusable import/source-state policy.
• It does not record validation commands/statuses.
• It does not make copied path groups and future re-import requirements explicit enough for dependent issues Import DataTasks into work-engine #4 and Import Podcast Assistant into assistants/podcast #7.
• Import Podcast Assistant into assistants/podcast #7 also needs the policy to distinguish the current transitional podcast-assistant/ import from the target assistants/podcast/ location.

I updated the issue into an implementable documentation-only task with scope, acceptance criteria, test scenarios, dependencies, and out-of-scope boundaries, then removed needs grooming. Remaining labels are docs, migration, and P0.

[alexeygrigorev]

TEST PASS

Tester verification for #12 completed against the current repo/source state.

Evidence:

• test -f _docs/import-log.md: passed.
• rg -n "policy|DTC Operations|DataTasks|Podcast Assistant|Validation" _docs/import-log.md: passed; matched the named policy section, all three source-system entries, and validation sections.
• git -C ../dtc-operations rev-parse HEAD: passed, output e6f7b43f641389fa58a8e027485f5aac18e8ace5, matching the DTC Operations entry.
• git -C ../datatasks rev-parse HEAD: passed, output 9adbc21da3a4db3990a58965950513c618abcda0, matching the DataTasks entry.
• test -d ../podcast-assistant: passed.
• test ! -d ../podcast-assistant/.git: passed, matching the non-Git Podcast Assistant source-state handling.
• git diff --check: passed.
• git diff --name-only -- . ':(exclude)_docs/import-log.md': no output.
• git status --short: only _docs/import-log.md is modified.

Manual/repo checks:

• _docs/import-log.md contains a named import/source-state policy with required fields, read-only source boundary language, Git-backed source commit instructions, and non-Git local source-state instructions.
• DTC Operations, DataTasks, and Podcast Assistant entries each include source state, destinations, copied path groups, exclusions, validation command/status entries, and follow-up context where applicable.
• Destination/path groups listed for all three entries exist in the current dataops checkout.
• Validation statuses distinguish commands that passed from full app suites that were not rerun for this documentation-only issue.

No app suite was run because #12 is documentation-only and no app code changed.

[alexeygrigorev]

ACCEPTED

PM acceptance after Tester pass. _docs/import-log.md now defines the reusable import/source-state policy, keeps the source repos/directories read-only unless explicitly scoped, and records the required source state, destinations, copied path groups, exclusions, validation statuses, and follow-up context for DTC Operations, DataTasks, and Podcast Assistant.

This is acceptable as the source policy needed by #4 and #7. The DataTasks entry gives #4 a source-state convention for re-import/normalization work, and the Podcast Assistant entry clearly identifies the current transitional import plus the #7 move/canonicalization target under assistants/podcast/.

PM spot-checks passed: issue criteria reviewed, Tester pass reviewed, source-state commands reproduce the recorded DTC Operations/DataTasks commits, Podcast Assistant non-Git state is accurate, git diff --check passes, and only _docs/import-log.md is changed.

[alexeygrigorev]

ON-CALL PASS (expected CI skip)

Post-push verification for ab0b2909b222fa351b48b8e1c5d0a5e8f996662c (Define import source-state policy).

Checks:

• git ls-remote origin refs/heads/main points to ab0b2909b222fa351b48b8e1c5d0a5e8f996662c.
• Commit changes only _docs/import-log.md.
• GitHub check-runs API for this SHA returned total_count: 0; gh run list --commit ab0b2909b222fa351b48b8e1c5d0a5e8f996662c returned no runs.
• .github/workflows/deploy-dataops-v1.yml push paths exclude _docs/** and include only workflow/app/content/work-engine/build inputs.
• .github/workflows/validate-dataops-content.yml push paths exclude _docs/** and include only content/** plus the workflow file.

Result: pass. No GitHub Actions run IDs are expected for this docs-only _docs/import-log.md commit because both push workflows were path-filtered out. No CI/deploy failure found, and no on-call fix was needed.