Define the first workflow data model

[alexeygrigorev]

Define the first workflow data model

Status: ready for engineering
Tags: enhancement, work-engine, backend, data, process-docs, assistant, testing, P0
Depends on: #5 (closed document registry), #6 (closed task-to-process-doc links), #28 (execution-state storage schema), #48 (closed portable export/data safety)
Blocks: #8, #9, #29, #30, first workflow UI/runtime slices that need a stable workflow contract

Scope

Define the first V1 workflow data model as the reusable contract that turns DataOps from a docs-first portal into a workflow-first operations workspace.

This issue is not the Podcast workflow itself. It defines the shared model that a Podcast, Newsletter, Tax Report, or other workflow can use.

The implementation should:

• Add or update a concise workflow data model document, preferably near the existing V1 schema docs, that names the canonical entities, their required fields, enum values, relationships, and ownership boundary.
• Align work-engine/src/types.ts with that model for workflow definitions/templates, task definitions, runtime workflow bundles, task instances, reminders/follow-ups, proof requirements, docs-in-context fields, artifact references, assistant job references, and audit/history events.
• Keep canonical workflow definitions and reviewed task templates Git-backed under content/; keep mutable workflow runs, tasks, reminders, file/artifact metadata, assistant job metadata, and audit/history state in DynamoDB as described by Define execution-state storage schema #28.
• Preserve existing Design document registry IDs and resolver #5/Add task-to-process-doc links #6 document-link behavior: use sourceDocIds, instructionDocId, instructionStepId, phase, systems, and validation fields rather than inventing another doc-link mechanism.
• Preserve existing Add portable execution-data export and DynamoDB backup safety #48 portable export behavior: extend or verify export mapping only where this model adds durable fields/entities; do not rebuild the export system.
• Make artifact, assistant job, and audit/history relationships explicit enough for Define artifact model and storage policy #29 and Implement assistant job model and lifecycle #30 to implement without changing the core task/bundle/template contract again.
• Add or update focused tests/fixtures that prove one representative workflow definition can instantiate a runtime bundle with tasks, proof requirements, doc context, follow-up fields, artifact/assistant references, stable IDs, and exportable relationship fields.

Acceptance Criteria

• A V1 workflow data model document exists and clearly distinguishes:
  • Git-backed workflow definition/template data;
  • DynamoDB runtime workflow bundle/task/reminder state;
  • external/S3 artifact binaries versus DynamoDB artifact metadata;
  • export/migration records using stable business IDs.
• The model covers workflow definitions with phases/stages, task definitions, offsets from an anchor date, required bundle links, references, default assignee, trigger metadata, and source document IDs.
• The model covers runtime workflow bundles with bundle_id, template_id, title, anchor date, stage, status, references, required bundle links, tags, created/updated timestamps, and stable relationships to tasks/artifacts/assistant jobs/audit events.
• The model covers task instances with task_id, bundle_id, template_id or template_task_ref, assignee, due date, status, waiting/follow-up fields, completion actor/timestamp, proof fields, doc-context fields, comments, tags, and timestamps.
• Task proof semantics are explicit: a task may require a URL, file/artifact metadata, comment, or external status, and a done state must not be valid when required proof is missing.
• Reminder semantics are explicit for due, overdue, follow-up, missing-evidence, recurring, stage-change, and automation-failure notifications, reusing the existing notification/export type names where possible.
• Docs-in-context requirements reuse Design document registry IDs and resolver #5/Add task-to-process-doc links #6 fields and specify fallback behavior for legacy instructionsUrl; no new parallel document registry is introduced.
• Artifact and assistant job links are represented as stable IDs/metadata references only; binaries are not stored in DynamoDB and detailed artifact/job lifecycle remains in Define artifact model and storage policy #29/Implement assistant job model and lifecycle #30.
• Audit/history is modeled as append-only events with actor, entity type, entity ID, action, before/after metadata or summary, and timestamp, even if full event emission is deferred.
• All durable entities have stable application-level IDs and explicit relationship fields suitable for portable export and a future Postgres migration; DynamoDB PK/SK values are not the only relationship source.
• work-engine/src/types.ts, route validation, template instantiation, and export/validation fixtures are consistent with the documented model without regressing existing task/doc-link/export behavior.
• Tests cover the representative workflow fixture, template-to-bundle/task instantiation, waiting/follow-up validation, required proof validation, doc-context propagation, and portable export validation for the model fields touched by this issue.

Test Scenarios

Scenario: Representative workflow definition instantiates cleanly

Given a workflow definition with phases, required bundle links, source docs, task offsets, proof requirements, and doc-context fields
When a bundle is created from the workflow definition using an anchor date
Then the runtime bundle has stable IDs and required link placeholders, and each task has the expected due date, template task ref, bundle ID, proof requirement, doc context, assignee, tags, and status.

Scenario: Completion requires proof

Given a task generated from the workflow model requires a link or file/artifact proof
When the task is marked done without that proof
Then validation rejects the update with a clear missing-proof error and does not record completedAt or completedBy.

Scenario: Waiting work creates follow-up state without closing the task

Given a workflow task is blocked on an external person
When the task is moved to waiting with waitingFor and followUpAt
Then the task remains active, is exportable with waiting fields, and can drive a follow-up-due reminder when the follow-up date arrives.

Scenario: Docs stay contextual

Given a task definition includes instructionDocId, optional instructionStepId, phase, systems, and validation
When the template is saved and instantiated
Then those fields persist and copy to the task, while legacy instructionsUrl remains supported as a fallback only.

Scenario: Portable export remains migration-safe

Given the representative workflow has a bundle, tasks, proof metadata, reminders, and optional artifact/assistant/audit references
When the portable export validator runs
Then exported records use snake_case stable IDs and explicit relationship fields, the manifest marks not-yet-implemented entities explicitly, and no relationship depends only on DynamoDB keys.

Out of Scope

• Defining the actual Podcast workflow content; that remains Define the Podcast workflow #8/Build the Podcast end-to-end slice #9.
• Rebuilding the document registry or task-to-process-doc UI already covered by Design document registry IDs and resolver #5/Add task-to-process-doc links #6.
• Rebuilding the portable export, dry-run import, backup, or restore workflow already covered by Add portable execution-data export and DynamoDB backup safety #48.
• Designing detailed artifact storage policy or assistant job lifecycle; those remain Define artifact model and storage policy #29/Implement assistant job model and lifecycle #30.
• Implementing the full workflow UI, dashboard IA, or visual design.
• Moving binaries into DynamoDB or storing secrets/session tokens in normal exports.
• Migrating to Postgres now.

Dependencies

• Design document registry IDs and resolver #5 and Add task-to-process-doc links #6 are closed and should be treated as existing contracts for document IDs and task/doc links.
• Define execution-state storage schema #28 owns DynamoDB table/storage schema. This issue must stay compatible with Define execution-state storage schema #28 and should not introduce production table lifecycle changes outside that storage-schema work.
• Add portable execution-data export and DynamoDB backup safety #48 is closed and owns the export/backup foundation. This issue may extend field coverage but must not duplicate the export architecture.
• Define artifact model and storage policy #29 and Implement assistant job model and lifecycle #30 depend on this issue for stable task/bundle relationship fields, but they own the detailed artifact and assistant lifecycle decisions.
• Define the Podcast workflow #8/Build the Podcast end-to-end slice #9 should consume this model when defining and building the first Podcast workflow slice.