Migrate active Trello cards into integrated operations-manager work

[alexeygrigorev]

Migrate active Trello cards into integrated operations-manager work

Status: pending
Tags: enhancement, migration, work-engine, frontend, backend, testing, data, P1
Depends on: #15, #29, #48, #50
Blocks: None

Scope

Implement a safe migration path for active Trello cards so they become first-class DataOps operations-manager work, not a separate imported pile. Active Trello cards are the non-template, non-archived cards in the active Trello lists documented in work-engine/docs/data.md: Preparation, Announced, and After event. Each imported card should become an active workflow bundle with normal task instances, due dates, stage, process-doc context, required proof, waiting/follow-up state, artifacts/links where available, audit provenance, and portable export coverage.

Use these current references:

• .goal-v1.md
• docs/operations-manager-platform-jtbd.md
• docs/v1-workflow-data-model.md
• docs/v1-execution-data-safety.md
• work-engine/docs/data.md
• work-engine/docs/templates.md
• work-engine/docs/specs.md
• work-engine/scripts/migrate-data.ts
• work-engine/tests/migrate-data.test.ts
• work-engine/tests/export-portable.test.ts
• work-engine/tests/dry-run-import.test.ts
• content/tasks/templates/
• related process docs in content/**, especially docs that still reference Trello cards or migrated workflow evidence
• source-system references in ../datatasks and ../dtc-operations for comparison only; do not modify source repositories

The implementation should keep the import file-based and local/test safe. It may consume a Trello JSON export fixture or a human-provided export file, but it must not connect to live production Trello, mutate Trello, or perform external writes.

The migration should extend or replace the current active-card path in work-engine/scripts/migrate-data.ts so imported cards use the V1 model already present in the app:

• Bundle: title, anchor date, active status, stage from Trello list, tags, references, bundle links, source/provenance metadata, and related template/process context when known.
• Task: description, date, status, source, template/bundle relation, instructionDocId where resolvable, instructionsUrl as fallback, phase, systems, requiredLinkName, requiresFile, proofRequirement, waitingFor, followUpAt, comments/history notes, completedBy/completedAt when safely known, and import/audit provenance.
• Artifact and bundleLinks: non-Trello attachments and useful card-description links should become either required bundle links or artifact metadata when they are evidence/output links. Do not store binaries or signed/secret URLs in DynamoDB.
• Audit events: create import/audit provenance for each imported bundle/task/artifact when the current implementation supports it, or add omitted-entity/provenance metadata that exports safely until durable audit event creation is wired.
• Notifications: waiting tasks with due follow-ups should participate in the existing follow-up reminder behavior from Generate dashboard reminders from waiting follow-up dates #50.

This should be idempotent. Running the import twice against the same Trello export must not duplicate bundles/tasks/artifacts. Use stable source IDs such as Trello card/checklist/item IDs as provenance and matching keys, while keeping DynamoDB implementation keys out of portable exports.

Acceptance Criteria

• The migration imports only active Trello work by default: non-template, non-closed cards from Preparation, Announced, and After event; archived Done cards and template cards are not imported by the active-card path unless an explicit flag requests a separate mode.
• Imported cards appear as ordinary active workflow bundles in existing Work/Home surfaces, with stage mapped from the Trello list (preparation, announced, after-event) and no separate "Trello import" queue required for daily operation.
• Each checklist item becomes a normal task linked to its bundle, preserving ordering, checklist/phase context, due date when present, completion state when safe, assignee hints where supported, comments/instructions, and a stable templateTaskRef or source item reference.
• Imported tasks use instructionDocId when the linked instruction is resolvable to an in-repo process doc, and preserve instructionsUrl as fallback/provenance when no stable doc ID exists yet.
• Imported task completion semantics are explicit: required links, required files, required artifact proof, required comments, or external-status proof are set where the card/checklist/attachment data indicates evidence is needed; tasks with required proof cannot be completed without that proof.
• Waiting/follow-up semantics are preserved or inferred conservatively for tasks that are blocked on a guest, sponsor, author, speaker, publisher, internal reviewer, freelancer, accountant, or Alexey. A waiting task must have waitingFor, followUpAt, and a short note; uncertain cases remain todo with a migration note instead of silently becoming waiting.
• Non-Trello attachments and useful card-description links become operator-visible bundle links or artifact metadata, with sourceType: migration where artifacts are created. Trello-hosted internal image/card URLs are skipped unless they are needed as provenance and are safe to export.
• Imported data is audit/export safe: no OAuth tokens, API keys, cookies, signed URLs, session data, binary payloads, or private credentials are stored in exported metadata.
• The import supports a dry-run/preview mode that prints or returns counts for cards, bundles, tasks, links/artifacts, waiting tasks, proof requirements, skipped records, unresolved docs, and validation errors without writing data.
• The write mode is idempotent: re-running with the same export updates or skips previously imported records without duplicate bundles/tasks/artifacts, and reports what changed.
• The migration records blockers and unresolved mappings in machine-readable output, including unresolved workflow type, unresolved process doc, missing/invalid dates, unknown assignee, invalid URL, unsafe artifact URL, unsupported proof inference, and duplicate source IDs.
• A fixture-based test covers at least one card from each active stage (Preparation, Announced, After event) and proves the imported bundles/tasks show the correct stage, due dates, process-doc context, waiting/follow-up state, proof requirements, and artifact/link metadata.
• Portable export after a fixture import includes the migrated bundles, tasks, artifacts, notifications, assistant/audit omitted markers or records as appropriate, and validates with validate:export / portable export validation.
• Dry-run import/restore validation can read the exported data and report valid entity counts and relationships without writing production data.
• UI/API behavior remains unified: imported bundles/tasks are visible in today, overdue, waiting/follow-up, active workflows, bundle detail, task proof blockers, and process-doc context using the same paths as native workflows.
• [HUMAN] Before any real production Trello export is used, Alexey or Valeria confirms the export source, export date, included board/lists, and whether any sensitive card content must be redacted.
• [HUMAN] Before any production DynamoDB import write, a human confirms the target environment, backup/export location, dry-run summary, skipped/error report, and rollback/restore plan.

Test Scenarios

Scenario: Dry-run previews active cards without writes

Given: a local Trello JSON fixture with one Preparation card, one Announced card, one After event card, one template card, and one archived Done card
When: the migration runs in dry-run mode
Then: only the three active non-template cards are reported for import, the template and archived cards are skipped with reasons, and no DynamoDB records are written.

Scenario: Active cards become normal workflow bundles

Given: a local Trello JSON fixture with active cards that include labels, descriptions, checklists, due dates, and non-Trello attachments
When: the migration runs in local write mode
Then: the created bundles have active status, mapped stages, tags, references, bundle links/artifacts, and tasks linked by bundleId; they appear through existing bundle/task APIs as ordinary operations-manager work.

Scenario: Task context and proof are preserved

Given: a checklist item contains an instruction markdown link and names a required output such as a Luma, Meetup, YouTube, podcast document, invoice, report, or spreadsheet link
When: the task is imported
Then: the task has instructionDocId if resolvable, instructionsUrl as fallback if not, and the correct required link/file/artifact/comment/external-status proof semantics.

Scenario: Waiting work drives follow-ups

Given: a migrated card has a task that is known to be waiting for a guest/sponsor/freelancer/reviewer and includes or can safely infer a follow-up date
When: the import completes and follow-up notifications are generated
Then: the task remains active with status=waiting, waitingFor, followUpAt, and a note, and it appears in waiting/follow-up views when the follow-up date is due.

Scenario: Ambiguous waiting is not guessed

Given: a migrated checklist item mentions a person but does not clearly represent blocked waiting work or does not provide a safe follow-up date
When: the item is imported
Then: the task remains todo, includes the relevant source note/comment, and the migration report records the unresolved waiting inference for human follow-up.

Scenario: Import is idempotent

Given: the same Trello export has already been imported once
When: the migration runs again in local write mode
Then: no duplicate bundles/tasks/artifacts are created, stable source IDs are used to match existing records, and the summary reports created/updated/skipped counts.

Scenario: Export and restore safety holds after migration

Given: fixture cards have been imported into local work-engine data
When: a portable export is written and validated, then dry-run imported
Then: relationships between bundles, tasks, artifacts, notifications, assistant/audit records or omitted markers validate, no secrets or binaries are exported, and the dry-run import reports insert/update counts without writing data.

Scenario: Imported work is visible in the operator flow

Given: migrated bundles include overdue tasks, waiting follow-ups, missing proof, and process-doc links
When: the operator opens Operations Home and a bundle detail page
Then: the tasks appear in the normal today/overdue/waiting/active-workflow sections, missing proof blocks completion, and process docs open in context without going back to Trello.

Out of Scope

• Connecting to live Trello APIs, mutating Trello cards, archiving Trello cards, or syncing bidirectionally with Trello.
• Importing archived Done Trello history by default.
• Migrating spreadsheet TODOs; that belongs to Migrate spreadsheet TODOs into integrated operations-manager work #42.
• Redesigning the full workflow data model, artifact model, export format, or follow-up reminder system beyond changes required for this migration path.
• Mapping every workflow type to fully curated operator-ready process docs; related workflow mapping issues include Map Newsletter workflow to operator-ready process docs #36, Map Monthly Tax Report workflow to operator-ready docs and metadata #37, Map Book of the Week workflow to process docs #38, Map Open-Source Spotlight workflow to operator-ready process docs and executable task metadata #39, and future workflow issues.
• Creating new external integrations with Luma, Meetup, YouTube, Spotify, Apple Podcasts, Airtable, Mailchimp, Dropbox, Finom, Slack, LinkedIn, X, Google Drive, or email.
• Performing production DynamoDB writes, destructive restores, or external account checks during agent verification.
• Modifying ../dtc-operations, ../datatasks, or ../podcast-assistant.

Dependencies

• Define the first workflow data model #15, Define artifact model and storage policy #29, Add portable execution-data export and DynamoDB backup safety #48, and Generate dashboard reminders from waiting follow-up dates #50 are already closed and provide the V1 workflow model, artifact model, portable export/data-safety contract, and waiting follow-up behavior this issue should use.
• Extend process docs with stable IDs #33 is useful for broader stable process-doc ID coverage but is not a hard blocker. If a process doc cannot be resolved to a stable ID, preserve instructionsUrl and report the unresolved mapping.
• Map Newsletter workflow to operator-ready process docs #36, Map Monthly Tax Report workflow to operator-ready docs and metadata #37, Map Book of the Week workflow to process docs #38, Map Open-Source Spotlight workflow to operator-ready process docs and executable task metadata #39, and similar workflow mapping issues can improve proof/doc mappings, but this migration must still be safe with partial mappings.
• Migrate spreadsheet TODOs into integrated operations-manager work #42 should remain separate because spreadsheet TODO migration has different source shape and standalone/recurring-task semantics.
• Any real production export/import requires human confirmation and should follow docs/v1-execution-data-safety.md backup/export/restore guidance.

Affected Areas

• work-engine/scripts/migrate-data.ts and any helper modules created for Trello active-card import.
• work-engine/tests/migrate-data.test.ts and new fixture-based migration tests.
• work-engine/src/db/*, work-engine/src/export/portable.ts, and route validation only if existing fields cannot store the migration-safe metadata.
• work-engine/src/public/app.js, work-engine/src/pages/index.html, or portal integration tests only if imported work is not visible/actionable through existing operator surfaces.
• Data safety/export docs only if implementation discovers a new durable entity or export rule not already covered by docs/v1-execution-data-safety.md.

Data Safety, Export, And Restore Implications

• Use dry-run first for every real export. Production write mode must not run until a human reviews the dry-run summary, unresolved mappings, unsafe URLs, and skipped records.
• Create an on-demand DynamoDB backup and portable export before production import. Put agent-generated scratch exports under project-local .tmp/exports/ when testing locally.
• Store metadata and stable external/storage references only. Do not store Trello binary attachments, OAuth tokens, API keys, cookies, signed URLs, session values, private credentials, or raw secret-bearing comments.
• Exports must include stable application IDs and explicit relationships, not DynamoDB PK/SK keys as the relationship source.
• After local fixture import, run portable export validation and dry-run import validation to prove relationship integrity, redaction, timestamp parseability, waiting-task requirements, proof requirements, artifacts, notifications, and audit/omitted entities.
• Production restore drills or destructive restore/import checks are [HUMAN] and must not be performed by agents unless explicitly authorized in a later issue.

Blockers

• Real production Trello migration is blocked on [HUMAN] confirmation of the export file, included lists, export date, redaction needs, and target environment.
• Production DynamoDB writes are blocked on [HUMAN] approval of backup/export evidence, dry-run summary, and rollback/restore plan.
• Full instructionDocId coverage may be blocked by unresolved stable process-doc IDs until Extend process docs with stable IDs #33 or workflow-specific mapping issues cover the remaining docs; this should not block a safe import when fallback instructionsUrl and unresolved mapping reports are present.
• If existing work-engine storage lacks a durable place for source provenance/audit events needed for idempotency and export safety, the Architect should review the minimal schema extension before SWE implementation.

Required Verification Commands

Run the work-engine checks because this issue changes migration logic, runtime entities, export safety, and possibly operator behavior:

npm --prefix work-engine test
npm --prefix work-engine run typecheck
npm --prefix work-engine run build

Run focused migration/export commands with a local fixture and project-local export directory. The exact fixture path may differ after implementation, but verification must include dry-run, local write/import, export, export validation, and dry-run restore/import validation:

IS_LOCAL=true npm --prefix work-engine exec -- tsx scripts/migrate-data.ts --dry-run --cards-only --source <fixture-trello-export.json>
IS_LOCAL=true npm --prefix work-engine exec -- tsx scripts/migrate-data.ts --cards-only --source <fixture-trello-export.json>
npm --prefix work-engine run export:data -- .tmp/exports/trello-active-cards-fixture
npm --prefix work-engine run validate:export -- .tmp/exports/trello-active-cards-fixture
npm --prefix work-engine run dry-run:import -- .tmp/exports/trello-active-cards-fixture

If UI/operator surfaces change, run E2E and capture screenshots for Home/workflow detail states with imported tasks:

npm --prefix work-engine run test:e2e

Before handoff, include:

git diff --check

[alexeygrigorev]

PM grooming complete. Key product constraint: active Trello cards should migrate into normal DataOps bundles/tasks in the existing operator flow, with proof, waiting/follow-up, process-doc context, artifact/link metadata, idempotent dry-run/write behavior, and portable export/restore validation. Live Trello access and production import writes are explicitly [HUMAN]-gated.