Improve s3fs / fsspec API parity for the built-in S3FileSystem

[laughingman7743]

Background

PyAthena ships its own fsspec.AbstractFileSystem implementation (pyathena/filesystem/s3.py) instead of depending on s3fs, to avoid the s3fs / aiobotocore dependency and version-conflict problems that motivated it (#344, #465, #476, #272).

The trade-off is that users migrating from s3fs occasionally hit behaviors our implementation doesn't cover yet — most recently the transaction small-file bug in #719. This is an umbrella issue to track closing the remaining gaps so the built-in filesystem becomes a more complete drop-in replacement for s3fs (and a more complete AbstractFileSystem).

Status

Complete.

• Done in Add filesystem API parity features to the built-in S3FileSystem #729: directory operations (bucket lifecycle behind opt-in flags), object metadata & attributes (metadata/xattr, tags, ACL), multipart upload management, error translation, and the S3File helpers.
• Done in Complete the filesystem parity checklist: pipe_file, version-aware mode, and fsspec registry DX #730: direct-write pipe_file, version_aware mode + object_version_info, fsspec registry DX, and the async streaming-read verification.
• Out of scope by design: bucket-versioning helpers (infrastructure-level changes, consistent with the bucket-lifecycle decision).

Method

Compared S3FileSystem / S3File against s3fs (s3fs/core.py, main) and fsspec's AbstractFileSystem / AbstractBufferedFile. fsspec defaults that already work via our primitives — glob, walk, du, mv, head/tail, read_block, recursive get/put, multi-cat, pipe — are considered covered and out of scope.

Note: the comparison against s3fs defines the API surface users expect when migrating. It does not mean s3fs's implementation is the reference — see "Approach" below.

Already at parity ✅

• ls / info / exists / find (maxdepth, withdirs) / touch
• cat_file (ranged, concurrent via _fetch_range), put_file, get_file
• cp_file incl. multipart copy, bulk delete (_delete_objects), checksum, sign (presigned URL), created / modified
• multipart upload + transactions / commit / discard (incl. the small-file fix from pyathena s3 fsspec replacement file system not writing when used with transactions & a context manager #719)
• requester_pays, unsigned access, version_id on the read path
• async filesystem (AioS3FileSystem / AioS3File)
• FileNotFoundError translation for 404 / NoSuchKey / NoSuchBucket

Gaps (checklist)

Directory operations

• mkdir / makedirs / rmdir — done in Add filesystem API parity features to the built-in S3FileSystem #729. Design decision: bucket creation/deletion is disabled by default (bucket lifecycle is an infrastructure-level change, not file I/O) and raises PermissionError; opt in via the allow_bucket_creation / allow_bucket_deletion constructor flags. Key prefixes under an existing bucket are a no-op.

Object metadata & attributes

• metadata / getxattr / setxattr (on the fs and on S3File) — done in Add filesystem API parity features to the built-in S3FileSystem #729 (keys round-trip exactly as stored, no underscore/hyphen conversion)
• get_tags / put_tags — done in Add filesystem API parity features to the built-in S3FileSystem #729 (overwrite/merge modes)
• chmod / canned-ACL support (acl= on mkdir) — done in Add filesystem API parity features to the built-in S3FileSystem #729 (objects/buckets, optional recursive with executor-based parallelism)

Multipart upload management

• list_multipart_uploads / clear_multipart_uploads — done in Add filesystem API parity features to the built-in S3FileSystem #729 (paginated, optional key-prefix scoping, returns typed S3MultipartUpload instances)

Versioning

• version_aware mode: version-aware reads (the version observed at open/head time is pinned), ls(versions=True), plus object_version_info returning typed S3ObjectVersion instances. — done in Complete the filesystem parity checklist: pipe_file, version-aware mode, and fsspec registry DX #730
• (optional) bucket-versioning helpers (make_bucket_versioned, rm_versioned_bucket_contents) — not implemented by design: versioning configuration is an infrastructure-level change, consistent with the bucket-lifecycle decision (opt-in flags for bucket create/delete)

Error translation

• Map botocore ClientErrors to the matching OSError subclasses (e.g. 403 → PermissionError), instead of only 404 → FileNotFoundError. — done in Add filesystem API parity features to the built-in S3FileSystem #729 (pyathena/filesystem/s3_errors.py, applied at the S3FileSystem._call chokepoint; mapped by S3 error code first, HTTP status code as fallback)

File-object helpers

• S3File.url / S3File.metadata / getxattr / setxattr — done in Add filesystem API parity features to the built-in S3FileSystem #729
• (perf, optional) direct pipe_file single-PUT (multipart with abort-on-failure for large data; transaction semantics and s3_additional_kwargs preserved) — done in Complete the filesystem parity checklist: pipe_file, version-aware mode, and fsspec registry DX #730
• (verify) async streaming-read behavior of AioS3File — verified in Complete the filesystem parity checklist: pipe_file, version-aware mode, and fsspec registry DX #730: parallel range reads run through S3AioExecutor on the event loop and are covered by the async read tests; fsspec's open_async protocol is intentionally not implemented (the executor-based model is the supported async read path)

Drop-in / DX

• Registry shadowing (raised in pyathena s3 fsspec replacement file system not writing when used with transactions & a context manager #719) — done in Complete the filesystem parity checklist: pipe_file, version-aware mode, and fsspec registry DX #730: registration goes through pyathena.filesystem.register_s3_filesystem, which logs a warning when it overwrites an explicitly registered implementation instead of shadowing silently, and documents how to restore s3fs by re-registering it afterwards

Scope / notes

• Each area can be its own PR with unit tests (mocked, no AWS) plus a couple of integration tests. Keep large-file integration AWS cost bounded and at the filesystem layer (not multiplied across cursor types), as done in Fix empty object write for small files in fsspec transactions #721.
• Not all of these are equally important — suggest prioritizing by what migrators from s3fs actually hit. Help wanted; happy to take these incrementally.

Implementation & contribution notes

Where

• Sync: pyathena/filesystem/s3.py — S3FileSystem (the fs) and S3File (the AbstractBufferedFile).
• Async: pyathena/filesystem/s3_async.py — AioS3FileSystem, and AioS3File(S3File) which inherits _upload_chunk / commit / discard. A fix on S3File usually covers the async path too; add an async test only where the executor (S3AioExecutor) actually differs.

Approach

• Do not mirror s3fs's implementation. PyAthena's filesystem is intentionally its own design: a sync boto3 client with executor-based parallelism and typed response/model classes (S3Object, S3ObjectType, S3StorageClass, S3MultipartUpload, S3Owner, … in pyathena/filesystem/s3_object.py). s3fs holds values in raw dicts and is async-first; neither carries over here. Only the public API surface (method names and behavior users rely on when migrating) aims for compatibility.
• Design from the S3 API semantics and PyAthena's existing patterns. Return typed model classes rather than raw dicts — actively extend s3_object.py when a response needs modeling. Avoid duck-typing (e.g. getattr fallbacks); type parameters explicitly.
• Read fsspec/spec.py (AbstractFileSystem / AbstractBufferedFile) to honor the fsspec contract, and verify each gap against the installed source rather than assumptions. Favor the smallest change that solves the problem over new bespoke state. (Context: Fix empty object write for small files in fsspec transactions #721 fixed the transaction bug with a one-line _upload_chunk return-value change after a more complex first attempt was dropped.)

Tests

• Minimize mocking: real-S3 integration tests are cheap to write here, so prefer them. Mock only what must not run against real AWS (bucket creation/deletion, ACL updates, VersionId request shapes, unrealistic pagination) using the _make_fs helper in TestS3FileSystem (a minimal fs via S3FileSystem.__new__ + mocked _client / _call), or what is pure logic (standalone unit tests, e.g. test_s3_errors.py; S3File write-path tests via the _make_write_file / _make_multipart_write_file helpers in TestS3File). Lock the contract in both directions (regression + anti-simplification).
• Integration (real AWS): put these in TestS3FileSystem / TestAioS3FileSystem. The fs fixture is just S3FileSystem(connect()), so they run once — they are not multiplied across cursor types (pandas/arrow/polars). Parametrize over a small (one-shot PutObject) + one ~10 MiB (multipart) size; block size is 5 MiB (DEFAULT_BLOCK_SIZE = MULTIPART_UPLOAD_MIN_PART_SIZE), so 10 MiB triggers multipart.
• Keep large-file integration cases to ~2-3 uploads total to bound AWS cost; skip a large case when an inherited sync test already covers the path.
• Tests run with pytest -n 8 (xdist): integration tests must not touch shared bucket-wide state (e.g. scope multipart-upload cleanup to a unique key prefix).
• New S3 API calls may need IAM permissions added to the CI OIDC role: scripts/cloudformation/github_actions_oidc.yaml (stack github-actions-oidc-pyathena, deployed with the pyathena AWS profile in us-west-2). Local .env credentials are broader than CI's, so verify CI passes too.
• Run just lint first, then the filesystem tests. PR Add filesystem API parity features to the built-in S3FileSystem #729 is a worked example of all of the above (PR Fix empty object write for small files in fsspec transactions #721 for the write/transaction path).

Kickoff prompt (paste into a new session)

Copy this, fill in the target item, and use it to start an implementation session:

Implement one item from the PyAthena filesystem parity checklist in issue #722.

Target item: <paste one unchecked checkbox item, e.g. "version_aware mode">

Rules:

• Read CLAUDE.md first. The built-in filesystem lives in pyathena/filesystem/s3.py (S3FileSystem, S3File) and pyathena/filesystem/s3_async.py (AioS3FileSystem, AioS3File(S3File) which inherits the write methods — a fix on S3File usually covers async too).
• Do NOT mirror s3fs's implementation — only its public API surface. Design from S3 API semantics and PyAthena's existing patterns: typed model classes in pyathena/filesystem/s3_object.py (extend them when a response needs modeling), no raw-dict returns, no getattr duck-typing. Read fsspec/spec.py from the installed package to honor the fsspec contract, and verify the current gap against the real source, not assumptions.
• Unit tests (no AWS) in tests/pyathena/filesystem/test_s3.py: reuse the _make_fs helper (fs-level, mocked _client/_call) and _make_write_file / _make_multipart_write_file (file-level). Lock the contract (regression + anti-simplification).
• Integration tests in TestS3FileSystem / TestAioS3FileSystem only — these run once and are NOT multiplied across cursor types. Keep large uploads to 2-3 total (block size is 5 MiB), and never touch shared bucket-wide state (tests run with pytest -n 8). If the item calls a new S3 API, add the IAM action to scripts/cloudformation/github_actions_oidc.yaml and deploy the stack.
• Run just format, then just lint (ruff + mypy) and the filesystem tests before finishing.
• Branch off master, commit with a Co-Authored-By trailer, open a draft PR using the repo PR template, and link this issue. PR Add filesystem API parity features to the built-in S3FileSystem #729 is a worked example.

Confirm the chosen item and your plan before implementing.

Spun out of the discussion in #719.

[laughingman7743]

Opened #729 implementing a first batch of the checklist, designed around PyAthena's own architecture (typed model classes such as S3Object / S3MultipartUpload / new S3Owner, sync boto3 client with executor-based parallelism) rather than mirroring the s3fs implementation:

• Error translation — ClientError → standard Python exceptions (403 → PermissionError, etc.) at the _call chokepoint (pyathena/filesystem/s3_errors.py), mapped by S3 error code with HTTP status fallback
• mkdir / makedirs / rmdir — bucket create/delete; key prefixes under an existing bucket are a no-op
• metadata / getxattr / setxattr, get_tags / put_tags, chmod (canned ACLs, incl. acl= on mkdir)
• list_multipart_uploads / clear_multipart_uploads — paginated, returns typed S3MultipartUpload instances
• S3File.url / metadata / getxattr / setxattr
• AioS3FileSystem delegates for all of the above

The PR also fixes a latent dircache bug the new tests exposed: _ls_from_cache raised TypeError when looking up a child path of a cached S3Object entry.

Deferred to follow-ups: version_aware mode (version-aware ls/info/cat + object_version_info), registry-shadowing DX, direct single-PUT pipe_file, and async streaming-read verification.

Note on implementation guidance: the "mirror s3fs" wording in this issue's notes no longer applies — implementations should follow PyAthena's own design (typed response/model classes instead of raw dicts, no s3fs-derived code). Only the public API surface aims for compatibility.