[Epic] Implement data ingestion pipeline

[mark-torres10]

Issue Description

Following the plan in https://www.tldraw.com/f/SJAVL-hci-rKA7cw-CRHT?d=v53.113.3300.1898.pqmv-vXMUYTvYWaBYUnl9

High-level tasks:

• Create a batch job that looks for the keywords "trump", "economy" and "iran" and syncs 1,000 posts from those, from Bluesky. This should be just a YAML file (or a series of YAML files). As a side task, add support for .parquet instead of .csv. Create a Bluesky batch job #54
• Add a new step, in data_platform, called data_platform/s3_sync/, that takes the dataset ID and syncs all the raw posts, the preprocessed posts, the features, and the curated data, and uploads to S3. Let's use a bucket "lab_data_integrations_interface" (will need Terraform to define this bucket). Let's use the s3 prefix data/{integration}/{dataset ID}/{raw/preprocessed/features/curated}/ matching what we see locally. Create a pipeline step that uploads data to S3 #56
• Define the orchestration logic to run this as a Prefect DAG. Should run from sync all the way to uploading to S3. Also define a bash script to run the orchestration DAG as a cron job from the root of the repo (this'll be needed for running on HPC). Define orchestration DAG for the data pipeline #57

Next:

• Define Glue partitions + schema for each set of data. Needs to be done in Terraform. Let's set up tables in a {integration}_{stage}_{entity} naming structure (e.g., bluesky_raw_posts, twitter_preprocessed_posts, reddit_curated_comments). Then add a step to the DAG that, once the S3 uploads are done, re-triggers Glue to scan for new partitions. Add Glue tables + partitions #59

Next:

• Experiment with running a big query + returning the .csv in-memory, vs. a presigned URL. Show differences in runtime, memory, and then explain the tradeoffs: Set up presigned URL implementation #60

Along the way, look for opportunities for refactoring + code cleanup (Claude Code isn't particularly the most terse in programming).

[coderabbitai]

🔗 Related PRs

#28 - Add Reddit subreddit post fetch experiment [merged]
#45 - Add mirrorview_scale Reddit ingestion with top-month listing [merged]
#47 - Add Twitter Mirrorview ingestion pipeline [merged]
#48 - Add Twitter preprocess, features, and curate CLIs [merged]
#50 - Scale up Twitter batch collection [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!