Phase 0 of the Arrow read-path hand-off — build now, server-independent. This is the seam that must exist and be correct before Arrow lands so Arrow is purely additive. Overlaps with #40 (JSON codec) and #41 (ReadResult); this issue tracks the Arrow-readiness constraints on top of that work, not a re-implementation.
The constraint
_codec.py must be format-aware from day one. Structure it as "given a stream of (format, bytes) chunks, produce the requested output" so adding an ARROW_IPC branch later is a new case, not a rewrite. Do not hardcode "bytes are always JSON" anywhere above _codec.
Scope
fetch() returns a ReadResult wrapping the AggregateStream response iterator. Keep results as discrete batches internally (one gRPC response = one batch) — do not eagerly concat into one blob. Batch-wise internals are what let a streaming Arrow reader plug in later (one Arrow batch per chunk).to_pandas() and iteration implemented against JSON batches (current server behaviour): decode bytes → JSON → records → DataFrame. read() ≡ fetch().to_pandas().iter_batches(batch_size=...) — memory-bounded streaming.- The request will gain
ResultFormat format on AggregateStreamRequest (UNSPECIFIED/JSON/ARROW_IPC, unset ⇒ JSON). Phase 0 always sends JSON; leave the seam so Phase 1 flips it.
Why now
The 6 clusters roll out staggered — the same client version will talk to Arrow-capable and JSON-only clusters simultaneously. A format-aware codec is what makes that possible without a rewrite.
Source: planning/pypi-arrow-handoff.md §3, planning/arrow-read-path-decision.md.
Phase 0 of the Arrow read-path hand-off — build now, server-independent. This is the seam that must exist and be correct before Arrow lands so Arrow is purely additive. Overlaps with #40 (JSON codec) and #41 (ReadResult); this issue tracks the Arrow-readiness constraints on top of that work, not a re-implementation.
The constraint
_codec.pymust be format-aware from day one. Structure it as "given a stream of(format, bytes)chunks, produce the requested output" so adding anARROW_IPCbranch later is a new case, not a rewrite. Do not hardcode "bytes are always JSON" anywhere above_codec.Scope
fetch()returns aReadResultwrapping theAggregateStreamresponse iterator. Keep results as discrete batches internally (one gRPC response = one batch) — do not eagerly concat into one blob. Batch-wise internals are what let a streaming Arrow reader plug in later (one Arrow batch per chunk).to_pandas()and iteration implemented against JSON batches (current server behaviour): decodebytes→ JSON → records → DataFrame.read()≡fetch().to_pandas().iter_batches(batch_size=...)— memory-bounded streaming.ResultFormat formatonAggregateStreamRequest(UNSPECIFIED/JSON/ARROW_IPC, unset ⇒ JSON). Phase 0 always sends JSON; leave the seam so Phase 1 flips it.Why now
The 6 clusters roll out staggered — the same client version will talk to Arrow-capable and JSON-only clusters simultaneously. A format-aware codec is what makes that possible without a rewrite.
Source:
planning/pypi-arrow-handoff.md§3,planning/arrow-read-path-decision.md.