Skip to content

Create a pipeline step that uploads data to S3 #56

Description

@mark-torres10

Issue Description

Epic link: #53

Add a new step, in data_platform, called data_platform/s3_sync/, that takes the dataset ID and syncs all the raw posts, the preprocessed posts, the features, and the curated data, and uploads to S3. Let's use a bucket "lab_data_integrations_interface" (will need Terraform to define this bucket). Let's use the s3 prefix data/{integration}/{dataset ID}/{raw/preprocessed/features/curated}/ matching what we see locally.

Some things to consider:

  • Should we upload all data to S3? Or just newly added ones?
  • What if a file has been changed since it was uploaded to S3? Should we re-upload it?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions