Issue Description
Epic link: #53
Add a new step, in data_platform, called data_platform/s3_sync/, that takes the dataset ID and syncs all the raw posts, the preprocessed posts, the features, and the curated data, and uploads to S3. Let's use a bucket "lab_data_integrations_interface" (will need Terraform to define this bucket). Let's use the s3 prefix data/{integration}/{dataset ID}/{raw/preprocessed/features/curated}/ matching what we see locally.
Some things to consider:
- Should we upload all data to S3? Or just newly added ones?
- What if a file has been changed since it was uploaded to S3? Should we re-upload it?
Issue Description
Epic link: #53
Add a new step, in data_platform, called data_platform/s3_sync/, that takes the dataset ID and syncs all the raw posts, the preprocessed posts, the features, and the curated data, and uploads to S3. Let's use a bucket "lab_data_integrations_interface" (will need Terraform to define this bucket). Let's use the s3 prefix
data/{integration}/{dataset ID}/{raw/preprocessed/features/curated}/matching what we see locally.Some things to consider: