Context
s3_upload_status is meant to track whether a run's output has been uploaded to S3, and is used by resume/retry logic (find_resume_run_dir, _retry_pending_uploads, require_all_runs_uploaded, etc.) via metadata.get("s3_upload_status", False).
Initialization is inconsistent across stages:
- Ingestion (
sync_checkpoint.py::build_base_sync_metadata) and features (generate_features/metadata.py::load_or_init_metadata, via the FeatureRunMetadata dataclass default) write s3_upload_status: false to metadata.json immediately when the run starts, before any work happens.
- Preprocessing (
preprocessing/runner.py::save_preprocessed) and curation (curate/runner.py::build_curate_metadata) do not write the key at all on initial save. It's only added — set directly to True — after the S3 upload succeeds.
Practically, both cases are read the same way (.get(..., False)), so consumers aren't broken today. But it's an inconsistency that makes metadata.json files unpredictable to read/debug (sometimes the key is present, sometimes it's just missing), and is a foot-gun for anyone adding new code that checks for the literal key rather than using .get.
Proposed fix
Make save_preprocessed and build_curate_metadata (and any other initial metadata writers) always set "s3_upload_status": False at creation time, matching the ingestion/features pattern. No change needed to the upload-success paths, which already set it to True and flush.
Potential Files to update:
data_platform/preprocessing/runner.py::save_preprocesseddata_platform/curate/runner.py::build_curate_metadata
After the fix, every stage's metadata.json should have s3_upload_status: false present from the moment the run directory is created, regardless of whether the run completes successfully.
Context
s3_upload_statusis meant to track whether a run's output has been uploaded to S3, and is used by resume/retry logic (find_resume_run_dir,_retry_pending_uploads,require_all_runs_uploaded, etc.) viametadata.get("s3_upload_status", False).Initialization is inconsistent across stages:
sync_checkpoint.py::build_base_sync_metadata) and features (generate_features/metadata.py::load_or_init_metadata, via theFeatureRunMetadatadataclass default) writes3_upload_status: falsetometadata.jsonimmediately when the run starts, before any work happens.preprocessing/runner.py::save_preprocessed) and curation (curate/runner.py::build_curate_metadata) do not write the key at all on initial save. It's only added — set directly toTrue— after the S3 upload succeeds.Practically, both cases are read the same way (
.get(..., False)), so consumers aren't broken today. But it's an inconsistency that makesmetadata.jsonfiles unpredictable to read/debug (sometimes the key is present, sometimes it's just missing), and is a foot-gun for anyone adding new code that checks for the literal key rather than using.get.Proposed fix
Make
save_preprocessedandbuild_curate_metadata(and any other initial metadata writers) always set"s3_upload_status": Falseat creation time, matching the ingestion/features pattern. No change needed to the upload-success paths, which already set it toTrueand flush.Potential Files to update:
data_platform/preprocessing/runner.py::save_preprocesseddata_platform/curate/runner.py::build_curate_metadataAfter the fix, every stage's
metadata.jsonshould haves3_upload_status: falsepresent from the moment the run directory is created, regardless of whether the run completes successfully.