Deploy private work-engine Lambda behind portal broker

[alexeygrigorev]

Goal

Deploy the DataTasks work-engine as a private runtime service behind the existing DataOps portal so operators get one authenticated experience at ops.dtcdev.click.

The public surface remains the current Python full app Lambda. The work engine is invoked only by the portal backend, not exposed as a second public Function URL.

Decision Inputs

• .goal-v1.md
• docs/v1-runtime-architecture.md
• Execution-state schema issue Define execution-state storage schema #28
• Portable export and backup issue Add portable execution-data export and DynamoDB backup safety #48

Scope

• Add a SAM-managed private WorkEngineFunction for the Node work-engine.
• Add IAM permission for DocsFullAppFunction to invoke WorkEngineFunction.
• Add authenticated /work/api/* broker routes in the Python full app.
• Strip the /work prefix before forwarding requests to the work engine.
• Pass trusted actor/context headers from the authenticated portal session.
• Add a work-engine production auth mode that trusts only the portal broker, gated by environment variables.
• Keep the existing frontend shell as the public UI entry point.

Acceptance Criteria

• GET /work and /work/* serve the portal frontend without bypassing portal auth.
• ANY /work/api/* authenticates in the Python full app, invokes the private work-engine Lambda, and returns status, headers, and JSON bodies correctly.
• Work-engine Lambda has no public Function URL or direct unauthenticated endpoint.
• Production does not use SKIP_AUTH=true.
• Broker forwards method, path, query string, selected headers, and body without leaking session secrets.
• Work-engine receives a stable trusted actor context, initially acceptable as portal-admin if per-user identity is not ready.
• Local development remains possible without AWS Lambda invocation.
• CI covers broker routing, unauthorized access, successful proxied API call, and work-engine health check.
• Deployment template validates with SAM/CloudFormation checks.

Out Of Scope

• Final task dashboard UX.
• Full per-user role model.
• Portable export implementation from Add portable execution-data export and DynamoDB backup safety #48.
• Defining the complete DynamoDB schema from Define execution-state storage schema #28.

Dependencies

• Depends on Define execution-state storage schema #28 for production table names and runtime storage.
• Related to Add portable execution-data export and DynamoDB backup safety #48 because every deployed runtime entity must be exportable.

Test Expectations

• Python tests for /work/api/* auth and broker behavior.
• Node tests for trusted portal auth mode.
• Integration smoke test for /work/health after deploy.
• CI step that builds both the Python full app and the Node work-engine package.

[alexeygrigorev]

Runtime references added in commits eb57b6c and 718f0b6: docs/v1-runtime-architecture.md and docs/v1-execution-state-schema.md. This issue should use those docs as the implementation boundary for the private work-engine Lambda, /work/api broker, env-configured DynamoDB tables, and portal-trusted auth mode.

[alexeygrigorev]

Implemented and deployed in b83f300, with the matching live deploy-role/source update in DataTalksClub/aws-infra@e268211.

What changed:

• Added private SAM-managed WorkEngineFunction (nodejs20.x, dist/handler.handler) with no Function URL.
• Added WorkEnginePortalSecret in Secrets Manager and portal-secret validation between the Python broker and work-engine Lambda.
• Added protected Python full-app broker routes for /work/api/* and /work/health; the broker strips /work, forwards body/query/method, does not forward user auth headers, and maps downstream failures to 502.
• Added work-engine WORK_ENGINE_AUTH_MODE=portal, with normal session auth still required unless the broker secret is valid.
• Added base64 request-body decoding in work-engine for Lambda/file-upload compatibility.
• Wired work-engine DynamoDB table env vars and IAM access to the execution-state tables.
• Updated CI/CD so work-engine/** triggers deploy, Node 20 is pinned in checks and deploy, work-engine tests/typecheck run in CI, SAM builds the Node Lambda, and deploy smoke-invokes the private work-engine health route.

Verification:

• Local: npm test in work-engine passed: 591 tests.
• Local: npm run typecheck and npm run build in work-engine passed.
• Local: uv run --project lambda-functions --extra search --with pytest python -m pytest tests/docs_app passed: 36 tests.
• Local: aws cloudformation validate-template passed for lambda-functions/template.full.yaml, lambda-functions/template.github-actions-dataops.yaml, and aws-infra/sandbox/dataops/template.github-actions.yaml.
• Live GitHub Actions deploy succeeded: https://github.com/DataTalksClub/dataops/actions/runs/28292558606
• Live stack dataops-v1 is UPDATE_COMPLETE.
• Live private Lambda: dataops-v1-WorkEngineFunction-ERlDThL8FGGF, runtime nodejs20.x, state Active, last update Successful.
• Live protected broker check passed: authenticated GET /work/health returned {"status":"ok"}.

Notes:

• The stable portal actor is currently portal-admin unless WORK_ENGINE_PORTAL_USER_ID is set, matching the issue's V1 acceptance criteria.
• The remaining UX work for a unified task dashboard is intentionally outside this infra/broker issue.

[alexeygrigorev]

Closing because the private work-engine Lambda, protected portal broker, CI/CD path, and live smoke checks are deployed and match the acceptance criteria.