chore(infra): add in-deploy CloudFormation drift detection

[mikanmarusan]

Problem

Production stack transitmikanmarusan can silently drift from template.yml when AWS Console edits, hot-fixes, or AWS-side automation modify managed resources. A subsequent sam deploy either clobbers those out-of-band changes without warning, or fails mid-deploy after partial application — the WebACL incident behind PR #39 was an instance of this class.

Decision

Add an in-deploy CloudFormation drift detection step to .github/workflows/deploy-production.yml. The step polls aws cloudformation detect-stack-drift for the stack and hard-fails the deploy when StackDriftStatus != IN_SYNC. Rationale (full discussion at #41 (comment)):

• Drift only matters at deploy time — the moment CFN reconciles. Inline gate is co-located with the action it protects.
• No cron, no auto-opened follow-up Issues, no per-resource exception list. The scheduled-cron + auto-Issue design rejected for being over-engineered for a single-operator personal repo.
• The originating WebACL incident is already structurally fixed twice over (template declares WebACLId since fix(infra): preserve CloudFront WebACL attachment in SAM template #39; workflow has a secret-vs-live ARN comparison). This Issue catches the open-ended class of future drifts, not that specific incident.

Scope

• One new step Detect CloudFormation drift in the deploy job of deploy-production.yml, between SAM Build and Verify WEB_ACL_ARN_PROD secret matches live distribution.
• Polls describe-stack-drift-detection-status with bounded backoff (≤ ~5 min).
• On DETECTION_FAILED → ::error:: with reason → exit 1.
• On DETECTION_COMPLETE + non-IN_SYNC → ::error:: + drifted resources table + remediation hint → exit 1.
• Applies to both dry-run=true and dry-run=false paths.
• Add ::add-mask::$WEB_ACL_ARN_PROD as the first step of the deploy job so describe-stack-resource-drifts output cannot leak the ARN through public workflow logs.

Out of scope

• Scheduled drift detection workflow.
• Auto-remediation.
• Per-resource allowlist of acceptable drift (will be added as a follow-up Issue only if the known CloudFront::Distribution tag-drift false positive surfaces in practice — see aws-cloudformation/cloudformation-coverage-roadmap#901).
• IAM least-privilege scoping of the deploy role (separate Issue chore(infra): scope down gh-actions-deploy-prod role to least privilege #52).

IAM

The existing OIDC role gh-actions-deploy-prod has AWSCloudFormationFullAccess, which covers cloudformation:DetectStackDrift, DescribeStackDriftDetectionStatus, and DescribeStackResourceDrifts. No IAM change required for this Issue.

Pre-merge gate

This PR can only be merged when the production stack transitmikanmarusan is currently IN_SYNC. If it is not, the first production deploy after merge will fail at the new step and all subsequent deploys will be blocked until drift is reconciled. Verify before merge:

DETECTION_ID=$(aws cloudformation detect-stack-drift --stack-name transitmikanmarusan --query 'StackDriftDetectionId' --output text)
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id "$DETECTION_ID"

Repeat until DetectionStatus=DETECTION_COMPLETE and StackDriftStatus=IN_SYNC.

Acceptance

• Deploy to Production with dry-run=true succeeds end-to-end; new step prints Stack is IN_SYNC. and downstream changeset describe runs green.
• Deploy to Production with dry-run=false succeeds end-to-end; new step prints Stack is IN_SYNC. and full deploy + frontend sync + invalidation + both health checks pass.

[mikanmarusan]

Proposal: rewrite to in-deploy drift check (drop cron + auto-issue)

After completing #42 (OIDC migration, #51), the picture here changes enough that the original design — weekly cron + auto-opened GH Issue with diff + WebACL exception list — is worth reconsidering before any implementation work starts.

Why the original design is over-engineered for this repo

1. The originating incident (run 25554289022) is structurally fixed twice over. template.yml now declares WebACLId explicitly (PR fix(infra): preserve CloudFront WebACL attachment in SAM template #39), and deploy-production.yml already runs LIVE_ARN vs SECRET_ARN comparison at deploy time (the Verify WEB_ACL_ARN_PROD secret matches live distribution step). The exact drift class that motivated this Issue cannot recur the same way.
2. CFN DetectStackDrift has known coverage gaps, particularly on CloudFront properties. Without a PoC verifying that CloudFrontDistribution.WebACLId is in the supported set right now, the watcher would provide false sense of security on the very property the Issue cites.
3. The WebACL exception logic re-introduces the bypass it tries to prevent. "Tolerate drift on WebACLId if live ARN matches WEB_ACL_ARN_PROD" silently passes the exact path the original incident used (console-side ACL swap + secret rotation to match).
4. Cron + auto-opened Issues is alert-fatigue prone on a single-operator personal repo. Weekly false-positive drift from CFN (metadata/ordering noise) leads to a muted Issue stream, after which real drift sits unnoticed.
5. Expected drift rate is approximately zero. This is a personal commute board; AWS Console "side quests" against transitmikanmarusan essentially do not happen. The catch rate does not justify standing infrastructure.

Proposed lighter design

Add a single aws cloudformation detect-stack-drift step in .github/workflows/deploy-production.yml, between SAM Build and SAM Deploy (and before the existing dry-run gates). Poll describe-stack-drift-detection-status until DETECTION_COMPLETE, then hard-fail the deploy if StackDriftStatus != IN_SYNC. Print which resources drifted to the workflow log via describe-stack-resource-drifts.

Shape:

• No cron. Drift is checked only at deploy time, which is the only moment it actually matters in this repo's operational profile.
• No auto-opened Issue. Workflow log + a failed run notification is sufficient. If the operator chooses to deploy with drift, they either fix it in the next PR or roll it back from the console.
• No WebACL exception list. The existing Verify WEB_ACL_ARN_PROD secret matches live distribution step is the correct authority on whether the WebACL is intentional; drift detection should sit as an orthogonal layer that does not negotiate with secrets.
• No new IAM identity. The OIDC role gh-actions-deploy-prod already exists. The three CFN drift API actions can be added to its policy.

Dependency

This redesign should ship after #52 (least-privilege role scoping). The three CFN drift permissions:

• cloudformation:DetectStackDrift
• cloudformation:DescribeStackDriftDetectionStatus
• cloudformation:DescribeStackResourceDrifts

…should be folded into the scoped policy as #52 designs it, rather than retrofitted afterwards.

What I'd ask before implementation

• Approval of the redesign (drop cron + auto-issue + exception list).
• The Issue title and body should be rewritten to reflect the new scope before the PR opens, so the spec and the implementation stay aligned.

What stays the same

The Issue's underlying value — "detect AWS-side changes that drift away from IaC" — is intact. Only the mechanism is being scaled down to match the repo's actual operational profile.

[mikanmarusan]

Title and body rewritten per the discussion in #41 (comment). The cron + auto-opened-Issue + WebACL exception list design has been scaled down to a single in-deploy step. Implementation PR follows after a pre-merge drift probe confirms the production stack is currently IN_SYNC.