Skip to content

Guard against unwritable root-owned jobs/ directories#29

Merged
dbroeglin merged 1 commit into
mainfrom
dbroeglin-guard-root-owned-jobs-dir
Jul 6, 2026
Merged

Guard against unwritable root-owned jobs/ directories#29
dbroeglin merged 1 commit into
mainfrom
dbroeglin-guard-root-owned-jobs-dir

Conversation

@dbroeglin

Copy link
Copy Markdown
Owner

Why

Running a freshly scaffolded experiment can fail with a bare PermissionError: [Errno 13] Permission denied: '.../jobs/<job-name>'.

The Pier Docker backend runs task containers as root and bind-mounts the jobs/ directory, so a completed run can leave jobs/ (and its contents) owned by root. A subsequent run started as your normal user can no longer create a new run directory under it, and Pier surfaces only an opaque PermissionError with no hint at the cause or the fix.

What

  • Add check_jobs_dir_writable() in pier_backend.py. It resolves the run directory Pier will create (jobs/<job-name>[/<run-id>]), walks up to the nearest existing ancestor, and raises PierBackendPreflightError with an actionable sudo chown -R "$(id -u):$(id -g)" <dir> (or rm -rf) remediation when that ancestor is not writable.
  • Wire it into _validate_pier_specs, so both validate and run report a clear jobs dir check and fail fast instead of crashing mid-run.
  • Document the failure mode and remediation in the scaffold README.md.tmpl (new Troubleshooting section) and in docs/collecting-run-data.md.

Notes

  • The writability probe is injectable (writable=) so tests stay offline and do not depend on real uid/permissions (CI often runs as root, where a real chmod-based probe would misbehave).
  • Pier has no container-user remapping knob, so root-owned output can recur; this guard turns the resulting crash into a clear, fixable message.

Testing

  • uv run ruff check . and uv run ruff format --check . clean.
  • uv run pytest -q green, including three new tests covering the writable case, the nearest-existing-ancestor remediation message, and the CLI surfacing the jobs dir check.

The Pier Docker backend runs task containers as root and bind-mounts the
jobs/ directory, so a completed run can leave jobs/ owned by root. A later
run started as the normal user then cannot create a new run directory under
it and Pier fails with a bare PermissionError.

Add check_jobs_dir_writable() to pier_backend.py, which resolves the run
directory Pier will create (jobs/<job-name>[/<run-id>]), finds the nearest
existing ancestor, and raises PierBackendPreflightError with a
chown remediation when it is not writable. Wire it into _validate_pier_specs
so both `validate` and `run` surface a clear "jobs dir" check instead of an
opaque traceback. Document the failure and fix in the scaffold README
template and docs/collecting-run-data.md.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@dbroeglin
dbroeglin merged commit b5f81fc into main Jul 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant