Add portable execution-data export and DynamoDB backup safety

[alexeygrigorev]

Add portable execution-data export and DynamoDB backup safety

Status: in progress
Tags: backend, data, infra, testing, P0
Depends on: #28 (execution-state schema, resolved)
Blocks: —

Scope

Complete the remaining data-safety gaps for portable execution-data export.
The export command, JSONL format, manifest, validator, redaction rules, and
PITR-enabled tables already exist. This issue adds:

1. A dry-run import path that validates an export and reports what would be
   written without mutating any data.
2. A scheduled export endpoint that triggers a portable export and writes it
   to an output location (local dir or S3 URI), callable as an admin job.
3. A restore drill document covering PITR recovery and portable-export
   validation, with the minimum smoke-check list.

Acceptance Criteria

• A dry-run import command reads an export directory, runs full
  validation, and reports insert/update/skip/invalid counts per entity
  without writing to any database.
• Dry-run import exits non-zero when validation fails, zero when valid.
• Dry-run import reports the total record count it would write.
• A scheduled export route (POST /api/cron/export) triggers a
  portable export and writes it to EXPORT_OUTPUT_DIR (or S3 when
  configured).
• The export route returns the manifest summary (entity counts, output
  location) on success.
• Waiting/follow-up fields (waitingFor, followUpAt) are preserved
  and validated in export/import records.
• Notification type values including follow-up-due are preserved
  and validated.
• A restore drill document explains PITR recovery, on-demand backup,
  portable export, validation, dry-run import, and the smoke-check list.
• Tests cover dry-run import reporting (valid and invalid exports) and
  the scheduled export route.

Test Scenarios

Scenario: Dry-run import of a valid export

Given: a portable export created by writePortableExport
When: dry-run import runs against the export directory
Then: reports the same entity counts as the manifest, exits zero, writes
nothing.

Scenario: Dry-run import of a broken export

Given: an export with a broken task reference and a tampered checksum
When: dry-run import runs against the export directory
Then: reports validation errors, exits non-zero, writes nothing.

Scenario: Scheduled export produces a valid archive

Given: a DynamoDB with seeded execution data
When: POST /api/cron/export runs with EXPORT_OUTPUT_DIR set
Then: returns manifest summary with entity counts, files exist on disk,
and validatePortableExport passes.

Out of Scope

• Real S3 upload implementation (route writes to local dir; S3 URI support
  documented as a follow-up).
• Admin auth/access control for the export endpoint (V1 uses the existing
  cron auth path).
• Multi-table transactional snapshot guarantee (current scan is sequential).
• File body backup (metadata-only export, as documented).
• Actual Postgres migration script.

[alexeygrigorev]

Architecture reference added in commit eb57b6c: docs/v1-runtime-architecture.md now records the data-safety decision. Current deployed V1 does not use DynamoDB yet; #28 should define the execution-state schema and SAM-owned table lifecycle before this issue implements portable export/import validation across all durable runtime entities.

[alexeygrigorev]

Added the data-safety reference in commit 718f0b6: docs/v1-execution-data-safety.md. The doc separates DynamoDB restore safety from portable exports: PITR/on-demand/AWS Backup protect AWS recovery, while JSONL plus manifest exports protect future migration to Postgres or another store. Keeping this issue open for the actual export command/endpoint, validator, and dry-run import.

[alexeygrigorev]

Implemented the first portable execution-data export slice in commit 5a6cb2c.\n\nWhat exists now:\n- export command: npm --prefix work-engine run export:data -- <export-dir>\n- validation command: npm --prefix work-engine run validate:export -- <export-dir>\n- manifest plus JSONL files for current runtime entities: users, tasks, bundles, templates, recurring_configs, files, notifications\n- redaction markers for password hashes and sessions\n- omitted markers for sessions, artifacts, assistant_jobs, audit_events\n- checksum, count, duplicate ID, required field, and relationship validation\n- tests covering successful export, redaction, no DynamoDB keys in JSONL, and broken-reference validation\n\nStill open for this issue: production wiring, S3/export storage decision, admin access control for exports, dry-run import, and expanding coverage when artifacts/assistant jobs/audit events exist.

[alexeygrigorev]

Backup safety update: commit 12d1c6d deployed the V1 DynamoDB execution tables, and PITR is enabled on the durable execution-state tables. This covers AWS-native point-in-time restore for tasks, bundles, templates, users, files, and notifications.

The portable export slice from 5a6cb2c remains the app-level migration path. Still open here: S3/admin export storage, dry-run import, and export coverage for artifacts/assistant jobs/audit events once those entities exist.

[alexeygrigorev]

Additional data-safety audit after deploying the private work-engine Lambda:

Covered now:

• Portable local export exists via npm --prefix work-engine run export:data -- <dir>.
• Export validator exists via npm --prefix work-engine run validate:export -- <dir>.
• Export is JSONL + manifest, strips DynamoDB PK/SK, excludes sessions, and redacts password data.
• Durable deployed DynamoDB execution tables have PITR, SSE, DeletionPolicy: Retain, and UpdateReplacePolicy: Retain.
• Live table checks previously confirmed PITR is enabled on durable V1 tables.

Remaining gaps for this issue:

• No scheduled/offsite export to S3 yet.
• No dry-run import or Postgres restore/import tool yet.
• No app-consistent multi-table snapshot guarantee; current export scans tables one by one.
• File export currently covers metadata. Actual uploaded file bodies need a durable storage/backup story before file deliverables are treated as production records.
• No documented restore drill yet that proves an export can be validated and replayed into a clean environment.

Recommended acceptance additions before closing:

• A scheduled admin export stores encrypted archives in S3 with lifecycle/retention.
• A dry-run import validates and reports what would be written without mutating data.
• A restore drill document covers PITR recovery and portable-export validation.
• File bodies are either stored in durable object storage or explicitly out of V1 production scope.

[alexeygrigorev]

Data-safety scope update from d981f47:

The execution model now has first-class waiting/follow-up task fields:

• status can be waiting.
• Waiting tasks require waitingFor and followUpAt.
• Waiting metadata is preserved when moving a task back to todo or forward to done.

Portable export/import validation for #48 should include these fields for task records, preserve them exactly, and validate that exported/imported waiting tasks have non-empty waitingFor and followUpAt. This matters for future Postgres migration because follow-up state is now operational execution state, not derived UI state.

Related reminder-generation follow-up: #50.

[alexeygrigorev]

Data-safety follow-up from 247a994:

Follow-up reminders are now persisted as notification records with type=follow-up-due, taskId, optional bundleId, and dueAt. Portable notification export now includes notification_type in addition to task_id and due_at.

For #48 restore/import validation, reminder notifications should validate:

• notification_type values that the app understands, including follow-up-due;
• task_id references when present;
• due_at preservation for reminder semantics;
• idempotency keys based on task_id + due_at if replay/import tooling later regenerates reminders.