Map Open-Source Spotlight workflow to operator-ready process docs and executable task metadata

[alexeygrigorev]

Map Open-Source Spotlight workflow to operator-ready process docs and executable task metadata

Status: pending
Tags: enhancement, docs, process-docs, work-engine, backend, frontend, data, testing, P1
Depends on: #33
Blocks: None

Scope

Map the Open-Source Spotlight (OSS) workflow into the DataOps V1 workflow-first model so an operations manager can run one pre-recorded OSS video from lead selection through YouTube publication and social follow-up without reconstructing the process from scattered docs.

This is not a docs-only cleanup. The workflow must become runnable operator work: tasks with anchor-date offsets, reminders, waiting/follow-up semantics, required proof, bundle links/artifacts, process-doc context, and clear acceptance criteria for done.

Primary files and source material to inspect and update:

• docs/operations-manager-platform-jtbd.md
• work-engine/docs/specs.md
• work-engine/docs/templates.md, section 6. Open-Source Spotlight
• content/tasks/templates/oss.md
• content/media/open-source-spotlight/sops/
• content/media/open-source-spotlight/templates/
• content/media/open-source-spotlight/reference/
• shared video/social SOPs/templates if they are the best in-repo context for OSS tasks, especially YouTube timecodes, playlist, Zoom recording, and social announcement docs
• work-engine/scripts/seed-templates.ts
• existing Podcast workflow mapping in work-engine/scripts/seed-templates.ts as the local pattern for phases, sourceDocIds, per-task instructionDocId, proof requirements, validation, waiting/follow-up semantics, and dashboard states

Implement in DataTalksClub/dataops only. Do not modify ../dtc-operations, ../datatasks, ../podcast-assistant, external Google Docs, Airtable, Zoom, YouTube, Calendly, LinkedIn, X, Slack, or production data.

The current OSS workflow is manual. It starts when a tool/project is selected and an author is being invited; the workflow anchor date is the planned YouTube publication date, usually Wednesday at 17:00 Europe/Berlin according to the current YouTube scheduling SOP. Preserve the current 14 task refs and offsets unless source docs prove a correction is required.

Operator Workflow Requirements

The runnable workflow must let the operator:

• Create an OSS workflow bundle for a tool, author, and target publication date.
• See the next due action on the dashboard and workflow detail without browsing the docs tree first.
• Track outreach replies and scheduling blockers as waiting work with waitingFor, followUpAt, and a note.
• Capture required runtime links and proof: guest email/contact, tool GitHub/repo, Zoom recording or recording source, YouTube video, review/timecodes message, playlist/scheduled status, author publication notice, social announcement proof, and guest share/recommendation follow-up where available.
• Open the relevant SOP/template in context from each task, preferring stable instructionDocId after Extend process docs with stable IDs #33 and keeping instructionsUrl only as fallback/provenance.
• Complete tasks only when required proof exists, using existing task mechanics where possible: requiredLinkName, requiresFile, proofRequirement, validation, comments/external status, and stage transitions.
• Surface due, overdue, waiting, follow-up due, and missing-evidence states on the existing operations dashboard/workflow UI.

Suggested phases:

• lead-outreach: find OSS project/authors, contact via GitHub/LinkedIn/email/community, record author/contact/project links.
• recording-scheduling: agree on time, schedule recording, capture waiting state if Calendly/date confirmation blocks progress.
• recording-intake: record demo, download Zoom recording, upload/create YouTube draft, capture required recording/video link.
• video-production: edit video, add links/timecodes, ask author for review/revisions/links.
• publication: schedule YouTube video for the anchor date/time, add to OSS playlist, notify author of publication date.
• promotion-follow-up: ask guest to share and schedule/post social announcement; close workflow only when public proof and follow-up state are settled.

Task Mapping Requirements

Keep these task refs and due offsets unless the implementation documents a source-backed reason for changing them:

Ref ID	Offset	Phase	Operator action	Required proof / waiting behavior
reach-out-github-authors	-21	lead-outreach	Identify likely maintainers/contributors and start outreach from GitHub or community context.	Capture tool/project link and outreach channel/comment. If no contact or reply, mark waiting for author/maintainer with follow-up date.
reach-out-tool-author	-20	lead-outreach	Send the OSS invitation using the in-repo OSS outreach template.	Required Guest email or contact link; waiting/follow-up if author has not replied.
find-time-calendly	-19	recording-scheduling	Help the author find a time if Calendly does not work.	Comment/external status with proposed or confirmed time; waiting/follow-up for author reply.
schedule-recording	-18	recording-scheduling	Schedule the recording and capture the recording/calendar details.	Comment or external-status proof; include recording/calendar link if available.
record-demo	-14	recording-intake	Record the OSS demo.	Required recording/source proof, or comment if Alexey/author owns the recording handoff.
download-upload-youtube	-13	recording-intake	Download Zoom recording and upload/create YouTube draft.	Required YouTube link; use Zoom/YouTube SOP or reference context.
editing-video	-12	video-production	Edit/review video and prepare it for publication.	External-status/comment proof that edit passed review; missing edit output makes workflow at risk.
add-timecodes-youtube	-11	video-production	Add timecodes/links to the YouTube video using OSS or shared YouTube SOPs.	Comment/external-status proof; include link/timecode note where useful.
ask-authors-review-codes	-10	video-production	Ask authors to review generated timecodes/cuts and send required links.	Waiting/follow-up required if author review or links are pending. Use the OSS revisions/links template.
schedule-youtube-video	0	publication	Schedule YouTube video for the anchor date/time and verify playlist/schedule state.	Required YouTube link and external-status proof; milestone moves bundle to after-event.
tell-author-publish-date	0	publication	Tell the author when the OSS video will be published.	Comment/external-status proof; use the publication-date section of the OSS revisions/links template.
add-to-oss-playlist	+1	publication	Confirm the video is in the Open-Source Spotlight playlist after publication.	External-status/comment proof; missing playlist status is missing evidence.
ask-guest-share-recording	+1	promotion-follow-up	Ask guest to share the recording and recommend other OSS authors.	Comment/external-status proof; waiting/follow-up if guest reply/share is expected.
schedule-social-media	+2	promotion-follow-up	Schedule or publish social announcement for the OSS video.	Required social post proof/link or scheduling confirmation; final task can move bundle to done once no required proof/waiting work remains.

Acceptance Criteria

• content/tasks/templates/oss.md is self-contained and operator-ready: frontmatter follows Extend process docs with stable IDs #33, workflow trigger/anchor/date policy is documented, phases/stages are named, required bundle links are listed, done criteria are explicit, and the task table includes operator action, proof, waiting/follow-up, and contextual doc mapping.
• work-engine/scripts/seed-templates.ts encodes OSS template metadata comparable to the Podcast mapping: sourceDocIds, phases, task-level instructionDocId/instructionStepId where available, systems, proofRequirement, structured validation.operatorAction, validation.completionProof, reminder/dashboard semantics, and waiting/follow-up semantics.
• Existing 14 OSS task refs and offsets are preserved unless a documented source-backed reconciliation note explains a change.
• Required bundle links are operator-visible and include at least Guest email, Tool GitHub, YouTube, plus any additional runtime links needed to prove recording source, author review, social announcement, or playlist/publication state.
• Tasks that produce required links/proof cannot be marked complete without evidence where existing work-engine mechanics support blocking; tasks that require human/external confirmation use a structured proofRequirement or validation fallback instead of silent completion.
• Outreach, date coordination, author review, link requests, guest sharing, and recommendation follow-ups can be represented as waiting with waitingFor, followUpAt, and a note; due follow-ups remain visible on the dashboard.
• Stage behavior is explicit: early tasks remain preparation, scheduled/publication milestones can move the workflow to after-event, and schedule-social-media or an equivalent closure milestone can move the workflow to done only after required proof and waiting work are resolved.
• OSS SOPs/templates/reference docs used by the workflow have stable IDs or clear fallback notes after Extend process docs with stable IDs #33. In-repo docs must be referenced by stable instructionDocId; external Google Docs URLs may remain only as provenance/fallback when no in-repo doc exists.
• related_docs, OSS task-template sourceDocIds, and task instructionDocId values resolve through the document registry after Extend process docs with stable IDs #33.
• work-engine/docs/templates.md remains consistent with the executable OSS template and reconciliation notes.
• The existing bundle/task UI shows OSS next actions, missing proof, waiting/follow-up state, required links/files, and contextual SOP/template links. If existing UI already supports this, add tests proving the seeded OSS metadata appears correctly; only add frontend code if needed.
• Tests cover OSS seed metadata, template instantiation due dates, required proof/link behavior, source doc IDs, instruction doc IDs, waiting/follow-up semantics, stage transitions, and required bundle links.
• Process Curator review is requested during verification for OSS doc mappings and operational usefulness.
• No source repositories outside DataTalksClub/dataops and no external production systems are modified.

Test Scenarios

Scenario: Operator creates an OSS workflow from a selected tool

Given: the OSS template is seeded
When: an operator creates a workflow for tool ExampleDB, author Pat, and anchor date 2026-07-22
Then: the bundle has OSS tags/type, phases, required bundle links, process references, stage preparation, and 14 generated tasks dated from 2026-07-01 through 2026-07-24.

Scenario: Outreach becomes waiting work

Given: the operator has contacted an OSS author and no reply has arrived
When: the operator marks the outreach task waiting with waitingFor: Pat / ExampleDB author, followUpAt, and a note
Then: the task remains unfinished, appears in waiting/follow-up views, and returns as follow-up due when followUpAt arrives.

Scenario: Required YouTube proof blocks publication tasks

Given: download-upload-youtube or schedule-youtube-video has no YouTube URL saved
When: the operator tries to complete the task
Then: completion is blocked or validation names the missing YouTube proof, and the workflow remains at risk.

Scenario: Author review follow-up is not lost

Given: timecodes/cuts have been sent to the author for review
When: the author has not approved or sent links by the follow-up date
Then: ask-authors-review-codes is visible as waiting/follow-up work with the author/link request context.

Scenario: SOPs and templates open in context

Given: the operator opens an OSS task such as outreach, timecodes, schedule YouTube, or ask guest to share
When: they click the task instructions/context action
Then: the relevant in-repo OSS SOP/template or documented fallback instructions link opens from the task context, and unresolved doc IDs show a usable fallback.

Scenario: Publication advances the workflow without hiding follow-up work

Given: the YouTube video is scheduled/published and the author has been notified
When: publication milestone tasks are completed with proof
Then: the workflow stage advances to after-event, while playlist, guest-share, and social tasks remain active until completed.

Scenario: Social follow-up closes only with proof

Given: the OSS video is public
When: the operator completes guest-share and social announcement tasks with required proof/comment and no waiting follow-up remains due
Then: the workflow can move to done, with completed task history retaining proof and timestamps.

Scenario: Stable doc references validate

Given: OSS docs have stable IDs and the OSS template uses sourceDocIds and instructionDocId
When: docs registry/search validation runs
Then: all mapped in-repo process docs resolve by stable ID, aliases remain usable, and missing mappings fail or are explicitly documented as fallback external URLs.

Out of Scope

• Adding new integrations with GitHub, Slack, Calendly, Zoom, YouTube, Airtable, LinkedIn, X, Hootsuite, or email.
• Sending real author messages, scheduling real recordings, uploading real videos, editing YouTube, posting social media, or modifying production/external data.
• Rewriting unrelated webinar/workshop/podcast/video workflow templates except for shared docs referenced by OSS tasks.
• Migrating historical OSS runs, Trello cards, or spreadsheet rows into runtime state.
• Building new reminder UI, dashboard primitives, or proof-gating mechanics beyond using existing fields unless existing UI cannot display the seeded OSS semantics at all.
• Creating a generic workflow designer or changing the broader workflow data model.
• Fixing every placeholder in every OSS source document when the placeholder is not needed to execute the 14-task workflow.
• Editing ../dtc-operations, ../datatasks, or ../podcast-assistant.

Dependencies

• Extend process docs with stable IDs #33 should land first or this issue must explicitly follow the stable-ID policy it defines and preserve aliases for any pre-existing/generated references.
• Add internal link and related-doc validation #34 is useful but not a hard blocker. If Add internal link and related-doc validation #34 has landed, include its documented link/reference validation command. If it has not landed, add focused tests proving OSS sourceDocIds, related_docs, and instructionDocId values resolve through the registry behavior available after Extend process docs with stable IDs #33.
• Existing work-engine task/template fields should be preferred: phases, sourceDocIds, instructionDocId, instructionStepId, systems, validation, proofRequirement, requiredLinkName, requiresFile, stageOnComplete, waitingFor, followUpAt, and bundle links.
• Human-only checks against real author inboxes, Calendly, Zoom, YouTube, Airtable, LinkedIn, X, Slack, or production social channels should be represented as [HUMAN] follow-up criteria or separate issues, not executed by agents for this issue.

Affected Areas

• content/tasks/templates/oss.md
• content/media/open-source-spotlight/**
• shared process docs/templates only when they are directly referenced by OSS task context
• work-engine/scripts/seed-templates.ts
• work-engine/docs/templates.md
• work-engine template/seed/export/instantiation tests
• docs registry/search metadata tests under tests/docs_app/
• frontend/ or work-engine/src/public/ only if existing UI cannot surface OSS proof, waiting/follow-up, required links, or instruction context clearly

Required Verification Commands, Screenshots, And Content Validation

Run from the repo root unless noted:

npm --prefix work-engine test
npm --prefix work-engine run typecheck
npm --prefix work-engine run build

Run E2E when frontend/operator-flow behavior changes, or a focused Playwright spec when the implementation only proves existing UI rendering:

npm --prefix work-engine run test:e2e

Run docs metadata/search checks because stable process-doc IDs, related docs, or content metadata are changed:

uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app

cd lambda-functions
uv run --extra search python -m lambda_functions.build_search_index \
  --docs-dir ../content \
  --output ../.tmp/dataops-content-search.index

If #34 has landed, run its documented docs/reference validation command as well.

Before handoff, include:

git diff --check

If any UI/operator flow changes are made, Tester must capture screenshots under .tmp/screenshots/ for:

• OSS workflow/template card or creation path.
• OSS workflow detail showing phases/tasks, required links/proof, and process-doc context.
• Waiting/follow-up state for an OSS author-review or outreach task.
• Missing-proof blocked completion for a YouTube or social/publication proof task.

Process Curator should validate that the mapped OSS docs are operationally useful: the operator can identify the next action, required inputs, proof needed for completion, and follow-up behavior without reading the whole docs tree.

Blockers

• Current blocker: Extend process docs with stable IDs #33 is open and should define stable process-doc IDs before this issue finalizes OSS sourceDocIds, related_docs, and instructionDocId mappings.
• No external credentials or production writes are required for agent verification. Real YouTube, Zoom, author email, Airtable, Slack, LinkedIn, X, or Calendly checks are out of scope or [HUMAN] follow-ups.