Add offsite export archives and restore evidence for production data safety

[alexeygrigorev]

Add offsite export archives and restore evidence for production data safety

Status: in progress
Tags: enhancement, backend, work-engine, infra, data, testing, docs, P0
Depends on: #48 (closed)
Blocks: None

Scope

Implement the production-ready offsite archive lane for DataOps V1 execution
data.

#48 already delivered the local portable export, export validation, dry-run
import, scheduled local export route, and restore-drill documentation. This
issue extends that foundation so production execution data can be exported to
durable offsite storage and later proven restorable without relying on Lambda
local files.

Affected areas:

• work-engine/ TypeScript export and cron/admin export path.
• lambda-functions/template.full.yaml and related deployment configuration
  for SAM-owned export storage, environment variables, and least-privilege IAM.
• Restore/export documentation in docs/v1-execution-data-safety.md and
  docs/restore-drill.md.
• Tests for archive writing, S3/offsite storage behavior, restore evidence, and
  production safety gates.

The implementation must stay aligned with the current V1 runtime architecture:
the public Python portal remains the only public entry point, WorkEngineFunction
stays private, runtime execution state stays in SAM-owned DynamoDB tables, and
portable exports remain application-level JSONL snapshots independent of
DynamoDB PK/SK internals.

Acceptance Criteria

• SAM/CloudFormation declares the offsite export storage needed for V1, or
  wires to an explicitly named existing bucket through parameters, with
  server-side encryption, public access blocked, versioning enabled,
  production retain/deletion safety, lifecycle or retention rules, and tags
  suitable for backup selection.
• WorkEngineFunction receives export archive configuration through stack
  parameters/environment variables and has least-privilege IAM for only the
  required export archive prefix/actions. No production bucket, credential,
  account ID, or secret is hardcoded.
• The scheduled/admin export path can write a timestamped portable export
  archive to offsite storage. The archive contains manifest.json and the
  current portable JSONL entity files, including artifacts.jsonl,
  assistant_jobs.jsonl, and audit_events.jsonl when those entities are
  emitted by the current export implementation. Entity omission must remain
  explicit in the manifest when an entity is not implemented.
• Archive object keys include environment and generation time, are stable
  enough for retention/audit review, and avoid leaking private data in the
  key name. The route/command response includes archive URI/key,
  generated_at, schema/export format version, entity counts, and checksum
  summary, but does not return secrets, signed URLs, session tokens, or
  private credentials.
• Export archives preserve the portable export safety rules from
  docs/v1-execution-data-safety.md: no password hashes, live sessions,
  API keys, OAuth tokens, cookies, signed temporary URLs, private
  credentials, raw binary payloads, or DynamoDB-only key dependency in
  normal exports.
• The implementation writes restore evidence for a non-production drill:
  source archive URI/key, app git SHA, export generated_at, manifest
  checksum summary, validation result, dry-run import counts, skipped/invalid
  record counts, target environment, timestamp, and smoke-check checklist
  result. Test and local drill artifacts must live under project-local
  .tmp/exports/ paths.
• A restore/drill command or documented workflow can fetch or read an
  archive, extract the portable export, run
  validate:export, run dry-run:import, and produce the restore evidence
  report without writing production data.
• Production restore/import/write behavior is human-gated: automated cron,
  admin export, validation, and dry-run paths must not mutate production
  DynamoDB tables; any production restore/write action requires an explicit
  human-run command or documented manual approval step before it can write.
• The existing local export path remains usable:
  npm --prefix work-engine run export:data -- <export-dir>,
  validate:export, and dry-run:import still work for local/test archives.
• Documentation explains the offsite archive location/prefix, retention
  expectations, restore evidence format, safe local .tmp/exports/ drill
  path, and the human gate for production restore/write operations.
• [HUMAN] A production operator verifies in AWS that the deployed export
  archive bucket/prefix is encrypted, versioned, private, retained/lifecycled
  as specified, and receives at least one scheduled/admin export archive.
• [HUMAN] Before production execution data is treated as critical, a human
  runs the documented restore drill against a staging or isolated target and
  attaches the restore evidence report to the issue.

Test Scenarios

Scenario: scheduled export uploads an offsite archive

Given: the work-engine has export archive configuration and a mocked or local
S3-compatible storage client.
When: the scheduled/admin export path runs.
Then: it writes one timestamped archive under the configured environment/prefix,
returns archive metadata and manifest summary, and does not expose secrets or
signed URLs.

Scenario: archive contents stay portable and redacted

Given: exported users, tasks, bundles, templates, recurring configs, files,
notifications, artifacts, assistant jobs, and audit events with realistic
relationships and sensitive fields.
When: the archive is extracted and validated.
Then: all emitted entity files pass validate:export, relationship checks pass,
normal exports exclude sensitive fields, and any omitted entity type is listed
explicitly in the manifest.

Scenario: restore evidence is generated without production writes

Given: a previously written archive.
When: the restore drill workflow validates the archive and runs dry-run import.
Then: it writes a restore evidence report with validation status, dry-run
would-write counts, skipped/invalid counts, checksum summary, target environment,
and smoke-check checklist, while making no writes to production DynamoDB tables.

Scenario: production restore remains human-gated

Given: production archive configuration is present.
When: cron/admin export, validation, or dry-run restore code runs automatically.
Then: the code cannot restore, import, overwrite, or delete production execution
records unless a separate explicit human-approved restore/write path is invoked.

Required Verification

• npm --prefix work-engine test

• npm --prefix work-engine run typecheck

• npm --prefix work-engine run build

• Focused tests for archive upload, archive extraction/validation, restore
  evidence generation, and production no-write safety.

• sam validate --template-file lambda-functions/template.full.yaml --lint
  or the repository's equivalent SAM validation command for the touched
  template.

• Docs link validation if restore/data-safety docs are edited:

  uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links \
    --repo-root . \
    --content-root content

• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app
  if Python portal broker, deployment template assumptions, or docs-app behavior
  are touched.

• Attach or summarize the local restore evidence report path under
  .tmp/exports/ in the Tester handoff.

Out of Scope

• Full Postgres or other target-database import tooling.
• Destructive production restore, table replacement, table rename, or production
  data repair automation.
• Binary backup/storage for uploaded files or generated artifacts beyond the
  portable export metadata archive. S3/object-storage backup for artifact
  binaries remains a separate production storage issue.
• UI dashboards for browsing export history, unless a minimal status endpoint is
  needed for verification.
• Changes to source repositories outside DataTalksClub/dataops, including
  ../dtc-operations, ../datatasks, and ../podcast-assistant.

Dependencies

• Add portable execution-data export and DynamoDB backup safety #48 is the closed foundation for the local portable export, validation,
  dry-run import, scheduled local export route, and restore-drill document.
• Implement assistant job model and lifecycle #30, Implement raw intake inbox for operational inputs #31, and Migrate active Trello cards into integrated operations-manager work #41 are not hard blockers for this issue, but any execution
  entities they add or migrate before implementation must be covered by the
  archive or listed explicitly in manifest.json omitted_entities.
• Production AWS verification requires a human with access to the deployed
  dataops-v1 account/stack. Agent work must stop at code, tests, local or
  mocked archive evidence, and documented human checks.

[alexeygrigorev]

Software Engineer handoff for #58.

Summary:

• Added offsite portable export archive support for work-engine exports, using gzip-compressed tar archives containing manifest.json plus the current JSONL entity files.
• Wired scheduled/admin export to archive storage when DATAOPS_EXPORT_ARCHIVE_BUCKET or DATAOPS_EXPORT_ARCHIVE_LOCAL_DIR is configured; responses include archive URI/key, generated time, schema/export format version, entity counts, and checksum summary without credentials or signed URLs.
• Added restore:drill workflow that reads file:// or s3:// archives, extracts to .tmp/exports/, runs validation and dry-run import, and writes restore evidence without production writes.
• Added SAM-owned retained/private/encrypted/versioned archive bucket, work-engine env vars, least-privilege s3:PutObject prefix access, scheduled archive export event, and deploy-role S3 bucket management permissions.
• Updated restore/data-safety docs and README command references.

Changed files:

• work-engine/src/export/archive.ts
• work-engine/scripts/restore-drill.ts
• work-engine/src/routes/cron.ts
• work-engine/src/handler.ts
• work-engine/tests/export-archive.test.ts
• work-engine/tests/scheduled-export.test.ts
• work-engine/tests/handler-cron.test.ts
• work-engine/package.json
• work-engine/README.md
• package.json
• package-lock.json
• lambda-functions/template.full.yaml
• lambda-functions/template.github-actions-dataops.yaml
• tests/docs_app/test_infra_template.py
• docs/v1-execution-data-safety.md
• docs/restore-drill.md

Verification:

• NODE_ENV=test SKIP_AUTH=true node --import tsx --test work-engine/tests/export-archive.test.ts work-engine/tests/scheduled-export.test.ts work-engine/tests/handler-cron.test.ts - passed, 12 tests.
• npm --prefix work-engine test - passed, 683 tests.
• npm --prefix work-engine run typecheck - passed.
• npm --prefix work-engine run build - passed.
• npm --prefix work-engine run export:data -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/issue-58-local-export - passed; emitted empty local portable export with all current entity files.
• npm --prefix work-engine run validate:export -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/issue-58-local-export - passed; valid: true.
• npm --prefix work-engine run dry-run:import -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/issue-58-local-export - passed; valid: true, totalRecords: 0.
• Local archive generation via writePortableExportArchive - passed; archive key execution-exports/local-drill/2026-06-28/dataops-execution-2026-06-28T11-23-34-946Z.tar.gz, local entity counts included tasks: 2.
• npm --prefix work-engine run restore:drill -- --archive file:///home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/issue-58-archive/execution-exports/local-drill/2026-06-28/dataops-execution-2026-06-28T11-23-34-946Z.tar.gz --target-environment local-drill --output-dir /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/issue-58-restore - passed; validation true, dry-run total records 2.
• sam validate --template-file lambda-functions/template.full.yaml --lint - passed.
• sam validate --template-file lambda-functions/template.github-actions-dataops.yaml --lint - passed.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content - passed.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app - passed, 104 passed and 3 skipped.
• git diff --check - passed.

Restore evidence:

• .tmp/exports/issue-58-restore/restore-evidence-2026-06-28T11-23-43-517Z.json
• Extracted archive path: .tmp/exports/issue-58-restore/extracted-2026-06-28T11-23-43-517Z/

Screenshots:

• None. This is backend/infra/docs work with no UI changes.

Remaining gaps / human gates:

• [HUMAN] Production operator must verify in AWS that the deployed archive bucket/prefix is encrypted, versioned, private, retained/lifecycled as specified, and receives at least one scheduled/admin export archive.
• [HUMAN] Before production execution data is treated as critical, a human must run the documented restore drill against staging or an isolated target and attach the restore evidence report.
• No real production AWS writes were performed.

Ready for Tester.

[alexeygrigorev]

ARCHITECTURE PASS for #58.

Reviewed the offsite archive and restore evidence shape across work-engine/src/export/archive.ts, cron/EventBridge invocation, SAM bucket/IAM changes, restore/data-safety docs, and the focused tests. The design keeps the portable archive as an application-level JSONL snapshot, stores it under a stable environment/date key, returns only archive/manifest metadata, and keeps restore evidence on validation plus dry-run import with no production write path. The SAM-owned archive bucket is private, encrypted, versioned, retained, lifecycle-tagged, and the work-engine write permission is scoped to s3:PutObject on the configured archive prefix.

Small review fix applied directly: work-engine/src/export/archive.ts now removes only internally-created temporary export directories after archive creation/upload, including failure paths, so scheduled Lambda exports do not accumulate /tmp/dataops-export-* directories. Caller-provided temp/export dirs remain intact for local drill evidence.

Verification run after the fix:

• NODE_ENV=test SKIP_AUTH=true node --import tsx --test work-engine/tests/export-archive.test.ts work-engine/tests/scheduled-export.test.ts work-engine/tests/handler-cron.test.ts - passed, 12 tests.
• npm --prefix work-engine run typecheck - passed.
• npm --prefix work-engine run build - passed.
• sam validate --template-file lambda-functions/template.full.yaml --lint - passed.
• sam validate --template-file lambda-functions/template.github-actions-dataops.yaml --lint - passed.
• git diff --check - passed.

Remaining gates are the issue's explicit [HUMAN] AWS/deployed-bucket verification and staging/isolated restore-drill evidence. After Tester passes, this can proceed to PM acceptance from the architecture/data-safety perspective.

[alexeygrigorev]

DESIGN CONFIRMATION from #57 acceptance pass:

Current export/data-safety behavior is safe but not yet operator-visible. The Admin surface has maintainer/diagnostic cards but no export, dry-run import, backup, restore, offsite archive, or human-gated production-copy state. The current scheduled export API returns a safe configuration error when EXPORT_OUTPUT_DIR is absent: {"error":"EXPORT_OUTPUT_DIR is not configured"}.

Evidence:

• .tmp/screenshots/issue-57/desktop-portal-admin-data-safety-current-state.png
• .tmp/screenshots/issue-57/mobile-pixel7-portal-admin-data-safety-current-state.png
• .tmp/screenshots/issue-57/export-current-state.json

This remains #58 scope; no duplicate issue filed from #57 for offsite archive/restore evidence.

[alexeygrigorev]

TEST PASS

Tester gate for #58 completed on branch worktree-issue-58 in /home/alexey/git/dataops/.tmp/worktrees/issue-58.

Changed-files review evidence:

• lambda-functions/template.full.yaml declares DataOpsExportArchiveBucket with DeletionPolicy: Retain, UpdateReplacePolicy: Retain, public access block, AES256 encryption, versioning enabled, noncurrent-version lifecycle retention, backup-selection tags, bucket/prefix outputs, and a daily scheduled export event.
• WorkEngineFunction gets DATAOPS_ENV, DATAOPS_EXPORT_ARCHIVE_BUCKET, and DATAOPS_EXPORT_ARCHIVE_PREFIX; IAM grants only s3:PutObject on ${DataOpsExportArchiveBucket.Arn}/${ExportArchivePrefix}/* for archive writes. I found no hardcoded production bucket, account ID, credential, or secret.
• work-engine/src/export/archive.ts, work-engine/src/routes/cron.ts, and work-engine/src/handler.ts implement timestamped tar.gz portable archives with manifest.json plus current JSONL entity files, safe keys shaped as <prefix>/<environment>/<YYYY-MM-DD>/dataops-execution-<generated-at>.tar.gz, response metadata without signed URLs or credentials, mocked S3 upload coverage, local archive coverage, and restore evidence generation.
• restore:drill extracts file:///s3:// archives, runs validatePortableExport, runs dryRunImport, writes evidence under .tmp/exports/, rejects prod/production targets, and performs no import/write/restore action.
• Docs in docs/v1-execution-data-safety.md, docs/restore-drill.md, and work-engine/README.md describe archive location/prefix, retention expectations, restore evidence format, safe .tmp/exports/ drill paths, and human-gated production restore/write behavior.

Commands run and evidence:

• NODE_ENV=test SKIP_AUTH=true node --import tsx --test work-engine/tests/export-archive.test.ts work-engine/tests/scheduled-export.test.ts work-engine/tests/handler-cron.test.ts - exit 0; 12 tests, 12 passed.
• npm --prefix work-engine test - exit 0; 683 tests, 683 passed.
• npm --prefix work-engine run typecheck - exit 0.
• npm --prefix work-engine run build - exit 0.
• npm --prefix work-engine run export:data -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-local-export - exit 0; emitted entity counts tasks: 2 and current entity files including artifacts.jsonl, assistant_jobs.jsonl, and audit_events.jsonl.
• npm --prefix work-engine run validate:export -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-local-export - exit 0; valid: true, errors: [].
• npm --prefix work-engine run dry-run:import -- /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-local-export - exit 0; valid: true, totalRecords: 2, no skipped entities.
• Local archive generation with NODE_ENV=test npx tsx -e ... writePortableExportArchive(...) - exit 0; archive URI file:///home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-archive/archive-store/execution-exports/tester-drill/2026-06-28/dataops-execution-2026-06-28T11-31-40-967Z.tar.gz, archive key execution-exports/tester-drill/2026-06-28/dataops-execution-2026-06-28T11-31-40-967Z.tar.gz, tasks: 1, checksum sha256:6be443ee5a6370b6564a7f1fd68a867a0be8b2617f5a9dcd31035c0464819ac6.
• npm --prefix work-engine run restore:drill -- --archive file:///home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-archive/archive-store/execution-exports/tester-drill/2026-06-28/dataops-execution-2026-06-28T11-31-40-967Z.tar.gz --target-environment tester-drill --output-dir /home/alexey/git/dataops/.tmp/worktrees/issue-58/.tmp/exports/tester-issue-58-restore - exit 0; validationValid: true, dryRunTotalRecords: 1, target tester-drill.
• Restore evidence file: .tmp/exports/tester-issue-58-restore/restore-evidence-2026-06-28T11-31-48-446Z.json. Evidence contains schema dataops.restore-evidence.v1, archive key, app git SHA unknown, export generated_at, manifest checksum summary, validation true, dry-run would-write counts, skipped {}, invalid count 0, target environment tester-drill, smoke checks not_run, and production write gate text.
• Extracted archive path: .tmp/exports/tester-issue-58-restore/extracted-2026-06-28T11-31-48-446Z/; extracted files include all current portable entity files plus manifest.json; manifest lists omitted_entities: ["sessions"] and redactions users.password_hash, sessions.
• sam validate --template-file lambda-functions/template.full.yaml --lint - exit 0; valid SAM template.
• sam validate --template-file lambda-functions/template.github-actions-dataops.yaml --lint - exit 0; valid SAM template.
• uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app - exit 0; 104 passed, 3 skipped.
• uv run --project lambda-functions --extra search python -m lambda_functions.validate_docs_links --repo-root . --content-root content - exit 0; docs link validation passed.
• git diff --check - exit 0.
• gh issue view 58 --repo DataTalksClub/dataops --json body --jq '.body' | rg -n "\\[ \\] .*\\[HUMAN\\]|\\[HUMAN\\]" - exit 0; both [HUMAN] acceptance checks remain unchecked.

Non-gate harness notes:

• I had two local ad-hoc archive-generation harness failures before the successful NODE_ENV=test run: one import/eval mistake and one run without local/test env that failed at AWS credential resolution. These did not perform production writes; the successful verification used local dynalite and file:// archive storage.

Acceptance criteria result:

• Non-human criteria: PASS. Archive storage, least-privilege function wiring, archive contents/key shape/response metadata, portable export safety, restore evidence, restore drill workflow, production no-write/human-gate behavior, local export compatibility, and documentation are verified by code review plus the commands above.
• [HUMAN] AWS deployed bucket/prefix verification: intentionally still unchecked.
• [HUMAN] staging/isolated human restore drill evidence attachment: intentionally still unchecked.

Production safety confirmation:

• No real production AWS writes were performed. AWS interaction in this gate was limited to read-only sam validate; S3 write behavior was covered by a mocked PutObjectCommand unit test and local file:// archive storage. The one accidental no-env ad-hoc probe failed at credential resolution before table setup.

Screenshots: none. This issue changes backend/infra/docs behavior and no UI route or page was changed.

Ready for Architect review. After Architect review, this is ready for PM acceptance from the Tester side.