Map Newsletter workflow to operator-ready process docs

[alexeygrigorev]

Map Newsletter workflow to operator-ready process docs

Status: pending
Tags: enhancement, docs, process-docs, work-engine, backend, frontend, data, testing, P1
Depends on: #33
Blocks: None

Scope

Map the Newsletter workflow into an executable DataOps V1 workflow that supports the operations-manager jobs-to-be-done in docs/operations-manager-platform-jtbd.md. This is not a docs-only cleanup. The operator should be able to run the weekly newsletter from the unified operations workspace: see the auto-created workflow, work through due tasks, follow up with sponsors, capture required links/files/proof, open SOPs in context, and close the workflow only when completion criteria are satisfied.

Use these current sources as the implementation inputs:

• .goal-v1.md
• docs/operations-manager-platform-jtbd.md
• work-engine/docs/specs.md
• work-engine/docs/templates.md, section 1. Newsletter
• content/tasks/templates/newsletter.md
• content/overview/reference/newsletter.md
• content/newsletter/**
• relevant finance and social process docs used by newsletter tasks, especially sponsor invoice and newsletter promotional posts
• work-engine/scripts/seed-templates.ts
• existing Podcast workflow mapping in work-engine/scripts/seed-templates.ts and work-engine/tests/seed.test.ts as the local pattern for phases, stable source doc IDs, per-task instruction docs, proof requirements, validation semantics, waiting/follow-up semantics, and dashboard states

Implement the Newsletter mapping in dataops only. Do not modify ../dtc-operations, ../datatasks, ../podcast-assistant, production DynamoDB data, external Google Docs, Mailchimp, Hootsuite, LinkedIn, X, Finom, or spreadsheets.

The Newsletter workflow must keep the existing task refs and timing from the current seed/template unless the source docs prove a correction is required:

Ref ID	Offset	Operator action	Expected proof / closure rule	Context doc mapping
create-sponsorship-document	-14	Create the sponsor content document for the newsletter slot.	Required URL: Sponsorship document.	sop.newsletter.sponsorship.creating-a-document-for-sponsored-content-for-a-newsletter
email-sponsor	-14	Send the sponsor the document, deadline, and Valeriia review context.	Required comment or external status confirming the email was sent; supports waiting/follow-up if sponsor content is not returned.	template.newsletter.send-sponsorship-document-2-weeks-before and/or template.newsletter.communication-with-sponsors
create-mailchimp-campaign	-13	Create or locate the Mailchimp campaign/draft for the issue.	Required URL: Mailchimp newsletter.	Prefer an in-repo Mailchimp draft SOP if present; otherwise map to sop.newsletter.mailchimp.schedule-a-newsletter-on-mailchimp or preserve instructionsUrl with a documented missing-doc note.
fill-sponsored-block	-12	Fill the sponsored block from approved sponsor copy, visual, and CTA, or record that the issue is not sponsored.	Required comment/external status; at risk if sponsored and Sponsorship document is missing or sponsor content is still waiting.	sop.newsletter.sponsorship.fill-in-the-sponsored-block-in-the-newsletter
fill-book-of-the-week-block	-11	Fill or remove the Book of the Week block.	Required comment/external status, including no book this week when applicable.	sop.newsletter.mailchimp.entering-information-in-the-book-of-the-week-block
fill-event-block	-10	Fill event announcement content in the newsletter draft.	Required comment/external status that event block was updated or intentionally skipped.	newsletter overview/reference plus the closest Mailchimp campaign SOP; do not invent a new SOP.
fill-podcast-block	-9	Add the just-published podcast page to the newsletter when applicable.	Required comment/external status; include podcast page URL in the comment or task link when used.	sop.newsletter.mailchimp.add-just-published-podcast-page-to-the-newsletter
fill-article-block	-8	Fill article/community content in the newsletter draft.	Required comment/external status that article block was updated or intentionally skipped.	newsletter overview/reference plus closest Mailchimp campaign SOP; do not invent a new SOP.
schedule-email-newsletter	-1	Schedule the reviewed newsletter in Mailchimp for Monday send time.	Required external status: campaign scheduled; requires Mailchimp newsletter; milestone may move the bundle to announced.	sop.newsletter.mailchimp.schedule-a-newsletter-on-mailchimp
create-invoice	0	Create/send the sponsor invoice for the newsletter placement.	Required file or URL: invoice PDF/link.	sop.finance.bookkeeping.creating-invoices-in-finom or the most accurate existing sponsor-invoice SOP.
send-email-sponsor-publication-live	+1	Notify sponsor that the newsletter promotion is live and share the Mailchimp campaign link.	Required comment/external status; requires Mailchimp newsletter.	template.newsletter.sending-email-on-the-day-of-publication
schedule-sponsorship-linkedin	+2	Schedule or publish the sponsored LinkedIn post and store its link.	Required URL: LinkedIn.	sop.social-media.linkedin.schedule-social-media-posts-with-hootsuite-and-post-about-newsletter-promotional-content or sop.social-media.linkedin.creating-sponsored-content-for-linkedin-post
schedule-sponsorship-twitter	+3	Schedule or publish the sponsored X/Twitter post and store its link.	Required URL: X.	sop.social-media.twitter.schedule-posts-with-twitter-and-post-about-newsletter-promotional-content
add-newsletter-performance	+7	Collect newsletter, LinkedIn, and X performance stats in the tracking spreadsheet.	Required comment/external status; include missing-stat notes when a channel has no clicks/views.	sop.newsletter.mailchimp.getting-campaign-performance-stats and/or sop.newsletter.mailchimp.filling-newsletter-statistics
send-performance-to-sponsor	+7	Send the performance summary to the sponsor.	Required comment/external status; completing it moves the bundle to done.	template.newsletter.newsletter-performance

Implementation should include the following changes where needed:

• Add or correct explicit stable IDs, aliases, related_docs, and metadata for the Newsletter task template and Newsletter workflow-critical SOP/template/reference docs, following Extend process docs with stable IDs #33. Use stable IDs over path-derived IDs in workflow metadata. If Extend process docs with stable IDs #33 lands with different canonical names, use Extend process docs with stable IDs #33's policy and keep the names above as aliases where useful.
• Update content/tasks/templates/newsletter.md so its frontmatter and task table reflect the executable Newsletter mapping: stable source docs, related docs, required bundle links, task proof/closure semantics, and instruction doc IDs rather than Google Docs links as the only process identity.
• Update work-engine/scripts/seed-templates.ts so the Newsletter template has sourceDocIds, phases, per-task instructionDocId/instructionStepId where available, systems, proofRequirement, validation.operatorAction, reminder/dashboard semantics, and waiting/follow-up semantics.
• Keep external instructionsUrl values only as fallback/provenance when no in-repo process doc exists yet. An in-repo SOP/template/reference must be referenced by stable instructionDocId once it exists.
• Preserve the existing automatic trigger: weekly Newsletter workflow creation with triggerSchedule: 0 9 * * 1 and triggerLeadDays: 14.
• Preserve required bundle links and make them operator-visible: Sponsorship document, Mailchimp newsletter, LinkedIn, and X.
• Use existing work-engine task fields where possible: requiredLinkName, requiresFile, proofRequirement, validation, waitingFor, followUpAt, phase, systems, and stageOnComplete.
• Add minimal frontend/operator-flow support only if the existing bundle/task screens cannot show the seeded proof, missing evidence, waiting/follow-up, or SOP context clearly enough for the JTBD acceptance scenarios.
• Update tests so Newsletter has coverage comparable to the Podcast V1 seed coverage already present in work-engine/tests/seed.test.ts.

Newsletter phases should be explicit. Suggested phase IDs:

• sponsor-intake: sponsor document, sponsor email, sponsor waiting/follow-up.
• draft-assembly: Mailchimp draft and content blocks.
• send-prep: final scheduling before Monday send.
• publication: invoice and sponsor live notification.
• promotion: LinkedIn and X sponsor posts.
• performance: stats collection and sponsor performance email.

Acceptance Criteria

• The groomed issue's central product reference, docs/operations-manager-platform-jtbd.md, is reflected in the implementation: Newsletter work is executable from the dashboard/workflow detail and not just browsable in process docs.
• The Newsletter template in work-engine/scripts/seed-templates.ts has stable sourceDocIds, explicit phases, per-task instructionDocId mappings where in-repo docs exist, systems, proofRequirement, and structured validation metadata with operatorAction, completionProof, requiredBundleLinks, reminderSemantics, and dashboardStates.
• Every Newsletter task that needs proof blocks completion through existing task mechanics: sponsorship document URL, Mailchimp campaign URL, LinkedIn URL, X URL, invoice file/link, or required comment/external status as appropriate.
• Sponsor-dependent tasks support waiting/follow-up semantics: the operator can mark the task waiting for sponsor content/review, set followUpAt, see it on the dashboard when due, and move it back to todo/done when the sponsor responds.
• The Newsletter workflow keeps automatic weekly generation with 14 lead days and creates tasks at the expected offsets from the Monday newsletter send date.
• Workflow stage behavior is explicit: scheduling the newsletter can advance the bundle to announced, post-publication work is visible after send, and sending performance to the sponsor completes the workflow with stageOnComplete: done.
• content/tasks/templates/newsletter.md is updated to match the executable template metadata and includes stable related docs rather than only Google Docs links.
• Workflow-critical Newsletter SOP/template/reference docs used by this issue have explicit stable IDs or are documented as missing in-repo docs with fallback instructionsUrl; no task should silently rely on an external Google Docs URL when an in-repo doc exists.
• related_docs, Newsletter task-template sourceDocIds, and task instructionDocId values resolve through the document registry after Extend process docs with stable IDs #33.
• The bundle detail/task UI shows the operator the next action, missing proof, required links/files, waiting/follow-up state, and contextual SOP/template link for Newsletter tasks. If this is already supported, add tests proving the seeded Newsletter metadata appears correctly.
• Tests cover Newsletter seed metadata, instantiation due dates, proof requirements, source doc IDs, instruction doc IDs, waiting/follow-up semantics, stage transitions, and required bundle links.
• Process Curator review is requested during verification for Newsletter doc mappings and operational usefulness.
• No source repositories outside DataTalksClub/dataops are modified.

Test Scenarios

Scenario: Automatic Newsletter workflow creates an operator-ready bundle

Given: the Newsletter template is seeded
When: a bundle is generated for a Monday anchor date, for example 2026-07-20
Then: the workflow has the Newsletter phases, required bundle links, and the 15 current tasks with dates from 2026-07-06 through 2026-07-27 based on the existing offsets.

Scenario: Dashboard shows the next Newsletter action

Given: an active Newsletter bundle has a task due today and required proof is missing
When: the operator opens the dashboard or workflow detail
Then: the task shows the workflow context, the next action, the missing proof label, and a contextual instructions link instead of requiring the operator to browse docs first.

Scenario: Required links block completion

Given: create-sponsorship-document, create-mailchimp-campaign, schedule-sponsorship-linkedin, or schedule-sponsorship-twitter has no corresponding URL saved
When: the operator tries to mark the task done
Then: the app blocks completion and names the missing Sponsorship document, Mailchimp newsletter, LinkedIn, or X link.

Scenario: Invoice proof blocks completion

Given: create-invoice has no uploaded file or invoice URL
When: the operator tries to mark it done
Then: the app blocks completion and asks for invoice proof.

Scenario: Sponsor content waiting is not forgotten

Given: sponsor content has not arrived for a sponsored newsletter slot
When: the operator marks the sponsor task waiting, sets waitingFor: Sponsor content, and sets a follow-up date
Then: the task remains unfinished, appears in waiting/follow-up views, and produces a follow-up reminder when followUpAt is due.

Scenario: SOPs open in context

Given: the operator is editing fill-sponsored-block
When: they click the task instructions/context link
Then: the in-repo SOP for filling the sponsored block opens or resolves by stable instructionDocId, and the operator can return to the Newsletter workflow to complete the task.

Scenario: Newsletter stage changes preserve post-publication work

Given: schedule-email-newsletter is completed
When: the workflow stage changes to announced
Then: publication, promotion, and performance tasks remain active and visible until their own proof/closure criteria pass.

Scenario: Performance report closes the workflow

Given: newsletter performance stats are collected and the performance email is sent to the sponsor
When: send-performance-to-sponsor is completed with required proof/comment
Then: the Newsletter bundle stage becomes done, and the completed task history retains proof and timestamps.

Scenario: Stable doc references validate

Given: Newsletter docs have stable IDs and the Newsletter template uses sourceDocIds and instructionDocId
When: docs registry/search validation runs
Then: all mapped in-repo process docs resolve by stable ID, aliases remain usable, and missing mappings fail or are explicitly documented as fallback external URLs.

Out of Scope

• Rewriting the Newsletter workflow task list beyond mapping the current 15 tasks unless the current source docs prove an implementation bug.
• Adding new external integrations with Mailchimp, Google Sheets, Hootsuite, LinkedIn, X, Finom, or email.
• Sending real sponsor emails, scheduling real campaigns/posts, creating real invoices, or modifying production/external data.
• Moving content/ into a separate knowledge repository.
• Mapping Book of the Week, Tax Report, Open-Source Spotlight, Podcast, Webinar, Workshop, Course, Social Media, Office Hours, or Maven workflows except where an existing Newsletter task links to those docs as context.
• Building a full process quality dashboard or new docs portal navigation.
• Editing ../dtc-operations, ../datatasks, or ../podcast-assistant.

Dependencies

• Extend process docs with stable IDs #33 should land first or this issue must explicitly follow the stable-ID policy it defines and preserve aliases for any pre-existing/generated references.
• Add internal link and related-doc validation #34 is useful but not a hard blocker. If Add internal link and related-doc validation #34 has landed, include its validation command in verification. If it has not landed, add focused tests proving Newsletter sourceDocIds, related_docs, and instructionDocId values resolve through the registry behavior available after Extend process docs with stable IDs #33.
• Existing work-engine fields already support most of the needed semantics. Prefer extending seed metadata and tests over broad data-model changes.
• Human-only checks against real Mailchimp, sponsor inboxes, Hootsuite, LinkedIn, X, Finom, or spreadsheets should be marked [HUMAN] in follow-up issues if needed; they are not required to complete this implementation.

Labels

Use labels: enhancement, docs, process-docs, work-engine, backend, frontend, data, testing, P1.

Remove needs grooming after this body is applied.

Verification Commands

Run work-engine checks because this issue changes executable template metadata and operator workflow behavior:

npm --prefix work-engine test
npm --prefix work-engine run typecheck
npm --prefix work-engine run build

Run E2E when frontend/operator-flow behavior changes, or a focused equivalent if the implementation only proves existing UI rendering:

npm --prefix work-engine run test:e2e

Run docs metadata/search checks because stable process-doc IDs and related docs are changed:

uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app

cd lambda-functions
uv run --extra search python -m lambda_functions.build_search_index \
  --docs-dir ../content \
  --output ../.tmp/dataops-content-search.index

If #34 has landed, run the documented docs/reference validation command, expected to be equivalent to:

uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links \
  --repo-root . \
  --content-root content

Before handoff, include:

git diff --check

[alexeygrigorev]

Groomed as an operator-ready implementation issue. I made the operations-manager JTBD analysis central, scoped Newsletter as an executable V1 workflow rather than a docs-only mapping, added concrete task proof/follow-up/SOP expectations, named #33 as the stable-ID dependency, and removed needs grooming.