Set up role-agent process for DataOps

[alexeygrigorev]

Set up role-agent process for DataOps

Status: pending
Tags: docs, process-docs, P0
Depends on: None
Blocks: #2, #3

Scope

Set up the DataOps issue-driven role-agent process so work in DataTalksClub/dataops can move from raw intake through grooming, implementation, testing, acceptance, merge, and CI/deploy monitoring.

This includes documenting the repo-specific workflow, creating role-agent instructions for Product Manager, Architect, Process Curator, Assistant Engineer, Software Engineer, Tester, and On-Call Engineer, and ensuring GitHub labels match the pipeline described in _docs/PROCESS.md.

The setup should align with _docs/MERGE_PLAN.md Phase 0 and support the first merge-track issues for importing DTC Operations and keeping the docs app running from the merged repo.

Acceptance Criteria

• _docs/PROCESS.md clearly defines the DataOps issue lifecycle, mandatory gates, role responsibilities, issue format, label taxonomy, merge rules, testing expectations, and human-verification handling.
• Role-agent instruction files exist for Product Manager, Architect, Process Curator, Assistant Engineer, Software Engineer, Tester, and On-Call Engineer, with repo-specific responsibilities and handoff rules.
• Role-agent instructions require agents to read _docs/PROCESS.md before issue work and to respect the source-repo boundary for ../dtc-operations, ../datatasks, and ../podcast-assistant.
• GitHub labels needed by _docs/PROCESS.md exist in DataTalksClub/dataops, including workflow, type, area, and priority labels.
• Initial merge-track issues are filed or confirmed for Phase 0/Phase 1 work, with raw issues carrying needs grooming until the Product Manager grooms them.
• A role agent can inspect an issue and determine the next owner, required gates, dependencies, and expected verification steps without relying on chat context.

Test Scenarios

Scenario: Product Manager grooms a raw issue

Given: a raw GitHub issue with the needs grooming label
When: the Product Manager role agent follows the documented process
Then: the issue is rewritten into the groomed format, receives appropriate labels, records dependencies, and no longer has needs grooming

Scenario: Software Engineer starts implementation after grooming

Given: a groomed implementation issue with dependencies and acceptance criteria
When: the Software Engineer role agent starts work
Then: the agent can identify scope, out-of-scope items, required tests, commit expectations, and handoff requirements from the issue and process docs

Scenario: Merge and deploy monitoring follow the pipeline

Given: an implemented issue that passed tester and Product Manager acceptance
When: the orchestrator merges and pushes the approved branch
Then: the On-Call Engineer role agent knows to monitor CI/deploy status and report or fix failures according to the documented process

Out of Scope

• Importing code or content from ../dtc-operations, ../datatasks, or ../podcast-assistant.
• Implementing portal, backend, frontend, assistant, or work-engine functionality.
• Refactoring the source repositories.
• Replacing local merge workflow with pull requests unless separately requested.

[alexeygrigorev]

Grooming complete: issue body now follows the DataOps process format with status, tags, dependencies, scope, acceptance criteria, test scenarios, and out-of-scope items. Removed needs grooming and set process documentation labels.

[alexeygrigorev]

TEST PASS

Verified issue #1 against the current repo and GitHub state. No app test suite was run because this change is docs/process/agent-instruction only and does not touch app code, content metadata, routing, frontend behavior, or runtime infrastructure.

Commands/evidence:

• sed -n '1,260p' _docs/PROCESS.md and sed -n '261,620p' _docs/PROCESS.md: process doc defines lifecycle, agent workflow, responsibilities, local-only merge rules, mandatory gates, testing expectations, human verification, label taxonomy, temporary-file rules, and DataOps technology/source boundaries.
• rg -n '^(## Issue Lifecycle|## Agents|## Agent Workflow|## Issue Format|## Merging|## Mandatory Steps|## Testing Expectations|## Human Verification|## Labels)' _docs/PROCESS.md: required process sections found, including new ## Issue Format.
• printf '%s\n' product-manager designer architect process-curator assistant-engineer software-engineer tester oncall-engineer | while read r; do test -f ".claude/agents/$r.md" ...: all required role-agent files are present.
• git ls-files --others --exclude-standard .claude/agents: confirms the eight new agent files under .claude/agents/.
• rg -n 'Read _docs/PROCESS.md|Respect source boundaries|\.\./dtc-operations|\.\./datatasks|\.\./podcast-assistant|DataTalksClub/dataops|Do not commit|Do not edit code|PM acceptance|Tester|On-Call|Handoff' .claude/agents: role files are DataOps-specific, require reading _docs/PROCESS.md, include handoff rules, and preserve source repo boundaries.
• gh label list --repo DataTalksClub/dataops --limit 200 --json name --jq '[.[].name] as $labels | ["needs grooming","enhancement","bug","docs","migration","research","portal","process-docs","work-engine","assistant","podcast","frontend","backend","infra","testing","data","design","P0","P1","P2","human"] | map(select($labels | index(.) | not))': returned []; required workflow/type/area/priority/special labels exist, including design.
• gh issue list --repo DataTalksClub/dataops --state all --limit 100 --json number,title,state,labels ...: initial merge-track issues are present, including Import DTC Operations into dataops #2, Keep DTC Operations docs app running from merged repo #3, Import DataTasks into work-engine #4, Import Podcast Assistant into assistants/podcast #7, Define import log and source commit policy #12, and later Phase 0/Phase 1 follow-ups; raw backlog items retain needs grooming.
• gh issue view 2, gh issue view 3, gh issue view 4, gh issue view 7, gh issue view 12: sampled merge-track issues expose status, tags, dependencies, blockers, scope, acceptance criteria, test scenarios, out-of-scope boundaries, and verification commands or expectations, so a role agent can determine next owner/gates/dependencies/verification from issue/process docs.
• git diff --check: passed with no whitespace errors.

Acceptance criteria:

• PASS: _docs/PROCESS.md defines lifecycle, mandatory gates, role responsibilities, issue format, label taxonomy, merge rules, testing expectations, and human verification.
• PASS: role-agent files exist for product-manager, designer, architect, process-curator, assistant-engineer, software-engineer, tester, and oncall-engineer.
• PASS: role instructions require reading _docs/PROCESS.md and respecting ../dtc-operations, ../datatasks, and ../podcast-assistant boundaries.
• PASS: required GitHub labels exist, including design.
• PASS: initial merge-track issues exist/are confirmed; raw backlog items carrying needs grooming are visible.
• PASS: sampled issues plus process docs provide owner, gates, dependencies, and verification without chat context.

[alexeygrigorev]

ACCEPTED

PM acceptance after Tester pass: the role-agent process is usable for DataOps V1 orchestration.

Reasoning:

• _docs/PROCESS.md now defines the issue lifecycle, mandatory gates, role ownership, issue format, labels, merge rules, testing expectations, and human-verification handling clearly enough for role agents and the orchestrator to operate from GitHub issues.
• The eight .claude/agents/*.md role files cover PM, Designer, Architect, Process Curator, Assistant Engineer, Software Engineer, Tester, and On-Call, with repo-specific preflight, source-repo boundaries, and handoff rules.
• Required labels exist in GitHub, including design, and the current backlog has groomed merge-track issues plus raw needs grooming items in the expected state.
• Sampled Phase 0/Phase 1 issues expose next owner, dependencies, acceptance criteria, verification, and out-of-scope boundaries without relying on chat context.

No follow-up required for #1 before commit/merge.

[alexeygrigorev]

On-call post-push verification for 57aa9aa1adae6dee3e16be939c250f433bd05964 (Add DataOps role agent instructions): PASS / expected CI skip.

• origin/main points to 57aa9aa1adae6dee3e16be939c250f433bd05964.
• Changed paths: _docs/PROCESS.md, .claude/agents/*.md.
• Validate DataOps Content push filter only includes content/** and .github/workflows/validate-dataops-content.yml; no match.
• Deploy DataOps V1 Lambda push filter only includes workflow/content/frontend/lambda/scripts/tests/work-engine/package/pyproject/uv files; no match.
• GitHub Actions check: no runs/check-runs for this commit, which matches the path-filter expected skip. No on-call fix needed.

Run IDs: none for this SHA; expected skip by path filters.