Provision S3 backup infrastructure for private knowledge repo

[alexeygrigorev]

Provision S3 backup infrastructure for private knowledge repo

Status: in progress
Tags: infra, data, process-docs, P1
Depends on: #81
Blocks: enabling scheduled backups in the future private DataTalksClub/dataops-knowledge repository

Scope

Use ../aws-infra, not the public app repo, to provision the AWS resources needed by the private knowledge-repo S3 backup workflow.

The intended model remains:

• DataTalksClub/dataops is the public app/runtime repo.
• Private DataTalksClub/dataops-knowledge is canonical for operational docs/templates/prompts.
• Private S3 stores daily backup artifacts generated from the private Git repo.

Implement a CloudFormation stack under ../aws-infra/sandbox/dataops that creates:

• a private S3 bucket for dataops-knowledge backups;
• server-side encryption;
• bucket versioning;
• public access blocking;
• lifecycle rules for daily Git archive/bundle backups;
• a GitHub Actions OIDC role assumable by DataTalksClub/dataops-knowledge on main;
• scoped S3 permissions for reading latest/manifest.json and writing backup objects under the configured prefix.

Acceptance Criteria

• Infra source lives in ../aws-infra/sandbox/dataops.
• CloudFormation validates locally.
• Stack is deployed/applied in AWS sandbox account.
• Outputs provide bucket name, prefix, and backup role ARN for future GitHub repository variables.
• Bucket is private, encrypted, versioned, and public access is blocked.
• Backup role trust policy is restricted to DataTalksClub/dataops-knowledge on main.
• S3 role permissions are reasonably scoped to the backup bucket/prefix.
• Infra docs explain how to deploy/update and which variables to set in the private knowledge repo.

Verification

• aws sts get-caller-identity
• aws cloudformation validate-template --template-body file://sandbox/dataops/template.knowledge-backups.yaml --region eu-west-1
• aws cloudformation deploy --stack-name dataops-knowledge-backups --template-file sandbox/dataops/template.knowledge-backups.yaml --region eu-west-1 --capabilities CAPABILITY_NAMED_IAM
• aws cloudformation describe-stacks --stack-name dataops-knowledge-backups --region eu-west-1
• aws s3api get-public-access-block, get-bucket-versioning, get-bucket-encryption, and get-bucket-lifecycle-configuration for the created bucket.

[alexeygrigorev]

Implemented in DataTalksClub/aws-infra commit f54f1da and applied to AWS sandbox.

Stack:

• dataops-knowledge-backups
• Region: eu-west-1
• Account: 817685572750

Outputs for the future private DataTalksClub/dataops-knowledge repo variables:

• DATAOPS_KNOWLEDGE_BACKUP_ROLE_ARN=arn:aws:iam::817685572750:role/dataops-knowledge-backup-writer
• DATAOPS_KNOWLEDGE_BACKUP_BUCKET=dataops-knowledge-backups-817685572750-eu-west-1
• DATAOPS_KNOWLEDGE_BACKUP_PREFIX=dataops-knowledge
• AWS_REGION=eu-west-1

Verification:

• aws sts get-caller-identity: sandbox account 817685572750
• aws cloudformation validate-template --template-body file://sandbox/dataops/template.knowledge-backups.yaml --region eu-west-1: passed
• aws cloudformation deploy --stack-name dataops-knowledge-backups --template-file sandbox/dataops/template.knowledge-backups.yaml --region eu-west-1 --capabilities CAPABILITY_NAMED_IAM: successfully created/updated stack
• aws s3api get-public-access-block: all public access blocks true
• aws s3api get-bucket-versioning: enabled
• aws s3api get-bucket-encryption: AES256
• aws s3api get-bucket-lifecycle-configuration: daily backups transition to STANDARD_IA after 90 days and DEEP_ARCHIVE after 365 days; old latest-pointer versions expire after 365 days
• aws iam get-role: trust is restricted to repo:DataTalksClub/dataops-knowledge:ref:refs/heads/main
• aws iam get-role-policy: permissions scoped to the backup bucket/prefix for reading latest manifest and writing backup objects

Note: we kept the private-repo GitHub Actions backup plan. Lambda remains a fallback if private-repo Actions are not usable.

[alexeygrigorev]

Closed after applying the backup bucket and OIDC role through aws-infra.