Skip to content

v6.x backlog: Parquet bulk-read path via the export service #50

Description

@Pandys

If very large reads (500k+ rows) become a bottleneck, the columnar answer is the export service, not the streaming data plane.

  • Server side (Data API, separate repo): add EXPORT_FORMAT_PARQUET — the format enum documents the extension point (new enum value + Writer in routers/export/serialiser/)
  • Client side: route big reads through CreateExport -> poll GetExport -> SAS URL -> read_parquet, behind the same ReadResult interface
  • Effort: M server / S client. Only start after profiling shows client-side JSON decoding is the dominant cost.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestpriority: lowBacklog / future additionv6v6 rewrite (Major/6)

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions