You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Users sometimes need the full result — 10M rows to open in Excel/pandas — not to view in the browser. Every path we have today buffers the whole result in memory first:
the streaming grid folds every row into tab.result (src/core/stream.js),
the current result-panel Export (exportResult, src/ui/app.js:959) serializes the already-loaded rows into a Blob and triggers a download.
For millions of rows that's gigabytes in RAM and a hung tab. It's also capped — it can only export what the grid already fetched (the row cap from #86).
What is not the obstacle (corrected)
The blocker is not authentication. A same-origin fetch from the current tab carries Authorization: Bearer fine — the app already streams every query that way. The only thing that can't carry the header is a top-level navigation to a ClickHouse URL (window.open('https://ch/?query=…') / <a href>), which sends cookies, not headers. So Export runs a fetch in the current tab, never a navigation — and needs no extra tab or window.
The real constraint is streaming a huge response to disk without buffering it in RAM — the File System Access API (showSaveFilePicker + streaming resp.body to handle.createWritable()), Chromium-only.
Solution
Add an Export button to the editor toolbar (next to Share) that runs the current editor query uncapped and streams the result to a user-chosen file, bypassing the result grid entirely, with progress + cancel. The output format follows the query: if the SQL ends in an explicit FORMAT <x> we stream that format and pick a matching file extension; otherwise we default to TSV.
Settled decisions
Stream to disk via the File System Access API — a hold-back writer over resp.body.getReader() → handle.createWritable(): constant memory regardless of result size. resp.body is already browser-decompressed (fetch handles Content-Encoding), so enable_http_compression=1 stays on (smaller transfer) and the file on disk is plain TSV.
Chromium-only, disabled-with-tooltip elsewhere. Feature-detect showSaveFilePickerand a secure context at button-build time. Where unavailable (Firefox/Safari, or plain-HTTP Chromium), the button renders aria-disabled with a tooltip ("Large export requires Chrome/Edge over HTTPS") — not natively disabled, so the tooltip actually shows. No buffered-Blob fallback — it would defeat the memory goal (consistent with the Chromium-leaning stance, support matrix Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71).
Format follows the query. If the SQL has a trailing FORMAT <x> (via detectSqlFormat), stream that format verbatim and derive the file extension from it (JSON→.json, CSV→.csv, Parquet→.parquet, …). If there's noFORMAT, default to TabSeparatedWithNames (header row, no type row — cleanest for Excel/pandas) with .tsv. Either way the grid / tab.result is never touched by an export.
Its own query_id + AbortController, fully separate from the grid run's app.state.runQueryId / app.state.abortController. Cancel aborts the stream and issues KILL QUERY for the export's id.
Replaces the old result-panel Export (see "Removing the old export"). The freed results-toolbar slot is reserved for a future Expand button (separate issue).
Progress is inline — a small banner (bytes written · elapsed · Cancel) in the current tab. No extra tab/window. (An early draft borrowed the openSchemaView child-tab pattern; dropped — see "Why inline, not a child tab.")
Architecture
Layered so the coverage gate stays at 100% per file: pure logic in src/core/, the fetch in src/net/ behind the injected seam, DOM wiring in src/ui/ behind the injected showSaveFilePicker seam.
1. src/core/ — pure helpers (100% covered)
src/core/format.js — resolve the export SQL + its effective format. Respects a user FORMAT (via the existing detectSqlFormat); only defaults to TSV when absent:
/** * Resolve an editor query for a full export. If it already ends in `FORMAT * <name>` (detectSqlFormat), keep it and report that format; otherwise append * `FORMAT TabSeparatedWithNames`. A trailing `;` is peeled either way. Returns * { sql, format } (empty input → { sql: '', format: 'TabSeparatedWithNames' }; * caller no-ops on empty). Pure. */exportfunctionprepareExportSql(sql){consts=String(sql||'').trim().replace(/;+\s*$/,'').trim();if(!s)return{sql: '',format: 'TabSeparatedWithNames'};constfmt=detectSqlFormat(s);returnfmt
? {sql: s,format: fmt}
: {sql: s+'\nFORMAT TabSeparatedWithNames',format: 'TabSeparatedWithNames'};}
src/core/export.js — map a ClickHouse output format → file { ext, mime } (by family, so the long tail of names is covered), and build the suggested filename:
/** * File extension + MIME for a ClickHouse output format, matched by family so we * don't enumerate all ~90 names. Unknown/pretty-text formats fall back to * `.txt`. `mime` feeds showSaveFilePicker's `accept`. Pure. */exportfunctionformatFileMeta(format){constf=String(format||'');if(/EachRow$/i.test(f)||/^NDJSON$/i.test(f))return{ext: 'jsonl',mime: 'application/x-ndjson'};if(/^JSON/i.test(f))return{ext: 'json',mime: 'application/json'};if(/^CSV/i.test(f))return{ext: 'csv',mime: 'text/csv'};if(/^(TSV|TabSeparated)/i.test(f))return{ext: 'tsv',mime: 'text/tab-separated-values'};if(/^Parquet$/i.test(f))return{ext: 'parquet',mime: 'application/vnd.apache.parquet'};if(/^(Arrow|ArrowStream)$/i.test(f))return{ext: 'arrow',mime: 'application/vnd.apache.arrow.file'};if(/^ORC$/i.test(f))return{ext: 'orc',mime: 'application/octet-stream'};if(/^Avro$/i.test(f))return{ext: 'avro',mime: 'application/octet-stream'};if(/^Native$/i.test(f))return{ext: 'native',mime: 'application/octet-stream'};if(/^(RowBinary|RawBLOB)/i.test(f))return{ext: 'bin',mime: 'application/octet-stream'};if(/^XML$/i.test(f))return{ext: 'xml',mime: 'application/xml'};if(/^Markdown$/i.test(f))return{ext: 'md',mime: 'text/markdown'};if(/^SQLInsert$/i.test(f))return{ext: 'sql',mime: 'application/sql'};return{ext: 'txt',mime: 'text/plain'};// Pretty*, Vertical, Values, unknown}/** * Suggested download filename: sanitized tab name (or a timestamp fallback) + * the format's extension. `now` is injected (Date.now()) for deterministic * tests. Pure. */exportfunctionexportFilename(tabName,now,ext){constbase=String(tabName||'').replace(/[^\w.-]+/g,'_').replace(/^_+|_+$/g,'')||'export-'+newDate(now).toISOString().replace(/[:.]/g,'-');returnbase+'.'+(ext||'tsv');}
src/core/stream.js — find ClickHouse's mid-stream exception block in the retained tail (see "Mid-stream errors" below for the wire format + source refs). Operates on a latin1-decoded string so char index == byte offset (the message may be UTF-8 multibyte, but the block start — the offset we slice on — is always ASCII):
constEXCEPTION_MARKER='__exception__';// ClickHouse WriteBufferFromHTTPServerResponse/** * Find CH's mid-stream exception frame in the retained tail. On a post-headers * failure CH appends (cpp: WriteBufferFromHTTPServerResponse::finalizeImpl): * \r\n__exception__\r\n<tag>\r\n<message>\n<len> <tag>\r\n__exception__\r\n * `tag` is the 16-byte value ALSO sent up front in the X-ClickHouse-Exception-Tag * header — read it from the response and pass it here, so a server-chosen random * tag (never present in data by accident) frames the match with zero false * positives. `tailLatin1` is the retained tail decoded 1 byte→1 char. * Returns { message, cleanBytes } (cleanBytes = byte length of real data before * the frame — what to keep) or null when no frame is present. Pure. * * Legacy fallback (tag == null, servers < 24.11): scan for `\nCode: <n>. * DB::Exception:` and report from there (can't excise as precisely). */exportfunctionfindExceptionFrame(tailLatin1,tag){consts=String(tailLatin1||'');if(tag){constopen='\r\n'+EXCEPTION_MARKER+'\r\n'+tag+'\r\n';conststart=s.indexOf(open);if(start<0)returnnull;constbody=s.slice(start+open.length);constclose=body.indexOf('\r\n'+EXCEPTION_MARKER+'\r\n');// trailerconstraw=close<0 ? body : body.slice(0,body.lastIndexOf('\n',close-1));return{message: utf8(raw).trim(),cleanBytes: start};}constm=/\nCode:\s*\d+\.\s*DB::Exception:[^\n]*/.exec(s);// legacyreturnm ? {message: utf8(m[0]).trim(),cleanBytes: m.index} : null;}// re-decode a latin1 slice back to UTF-8 for display (message text only)constutf8=(latin1)=>newTextDecoder().decode(Uint8Array.from(latin1,(c)=>c.charCodeAt(0)));
2. src/net/ch-client.js — the export request
/** * Issue an uncapped export query and return the raw streaming Response so the * caller can pipe `resp.body` straight to disk. `format` (from prepareExportSql * — the query's own FORMAT, or TSV) is set as `default_format`; the SQL's FORMAT * clause wins when present, so this only matters if none was appended. `query_id` * tags the request so cancel can KILL QUERY it. A failure *before* headers * (non-OK) throws the parsed CH exception; a failure *after* headers is detected * by the caller from the response body frame (findExceptionFrame) + the * X-ClickHouse-Exception-Tag header (CORS-exposed by CH, format-independent — * we leave http_write_exception_in_output_format at its default false). */exportasyncfunctionexportQuery(ctx,sql,{ queryId, signal, format }={}){consturl=chUrl(ctx.origin,{format: format||'TabSeparatedWithNames',params: queryId ? {query_id: queryId} : {},});constresp=awaitauthedFetch(ctx,url,sql,signal);if(!resp.ok)thrownewError(parseExceptionText(awaitresp.text()));returnresp;}
Note: no wait_end_of_query — that would buffer the whole result server-side and defeat the feature (the grid path rejects it for the same reason, ch-client.js:410). chUrl already adds enable_http_compression=1.
constshowSaveFilePicker=env.showSaveFilePicker||(win.showSaveFilePicker ? win.showSaveFilePicker.bind(win) : null);constsecureCtx=env.isSecureContext!=null ? env.isSecureContext : win.isSecureContext;// Fixed for the session (browser + origin don't change) — computed once, no signal.app.canExport=()=>!!showSaveFilePicker&&!!secureCtx;
Export-local state (sibling to the run state, kept separate so an export and a grid run never clobber each other's cancel state):
letexportAbort=null;letexportQueryId=null;// add `exporting: signal(false)` to createState() (src/state.js) — disables the// button while an export is in flight; grid Run stays independent.
The action. Ordering is load-bearing: showSaveFilePicker requires the click's transient activation, which any prior await (a token refresh in ensureConfig/getToken can be a network round-trip > the activation window) would forfeit. So do only synchronous validation, then open the picker, and only after we hold the handle run the async auth/export:
consterrMsg=(e)=>String((e&&e.message)||e);// matches app.js conventionasyncfunctionexportDirect(){if(app.state.exporting.value)return;if(!app.canExport())return;// aria-disabled button; defensive guardconsttab=app.activeTab();if(!tab.sql.trim()){flashToast('Nothing to export',{document: doc});return;}const{ sql, format }=prepareExportSql(tab.sql);// sync — format follows the queryconst{ ext, mime }=formatFileMeta(format);// (1) Picker FIRST — before any await, so the click's transient activation holds.lethandle;try{handle=awaitshowSaveFilePicker({suggestedName: exportFilename(tab.name,Date.now(),ext),types: [{description: format+' data',accept: {[mime]: ['.'+ext]}}],});}catch(e){if(e&&e.name==='AbortError')return;// user dismissed the picker → silent no-opflashToast('Save dialog failed: '+errMsg(e),{document: doc});return;}// (2) Now the awaits are safe — we already own the file handle.awaitensureConfig();if(!(awaitgetToken())){chCtx.onSignedOut();return;}// Dedicated id + abort — never app.state.run*. `uid` drops its prefix when// crypto.randomUUID exists, so prefix explicitly to tag it in system.query_log.exportQueryId='export-'+uid('');exportAbort=newAbortController();app.state.exporting.value=true;constprogress=showExportProgress(cancelExport);// inline banner: bytes + elapsed + Canceltry{constresp=awaitch.exportQuery(chCtx,sql,{queryId: exportQueryId,signal: exportAbort.signal, format });consttag=resp.headers.get('X-ClickHouse-Exception-Tag');// proactive; null on <24.11consterr=awaitstreamToFile(resp,handle,{signal: exportAbort.signal, tag,onProgress: (bytes)=>progress.update(bytes)});if(err)flashToast('Export incomplete — server error mid-stream: '+err,{document: doc});elseflashToast('Export complete',{document: doc});}catch(e){if(!(e&&e.name==='AbortError'))// AbortError = user cancelled → silentflashToast('Export failed: '+errMsg(e),{document: doc});}finally{progress.remove();app.state.exporting.value=false;exportAbort=null;exportQueryId=null;}}
Streaming with a hold-back buffer — the exception frame is at most 16 KiB and always at the very end, so we never commit the last HOLDBACK (32 KiB — 16 KiB max frame + margin) bytes until EOF proves them clean. Bytes only reach disk once they've aged out of the window; at EOF we scan the retained tail and write only the clean prefix, so a mid-stream exception is never written to the file. Memory stays constant (one HOLDBACK buffer). This reads the stream directly rather than via a TransformStream, because the write is conditional (we withhold, inspect, then commit) — a passthrough transform can't un-write:
asyncfunctionstreamToFile(resp,handle,{ signal, tag, onProgress }){constwritable=awaithandle.createWritable();constHOLDBACK=32*1024;// ≥ CH MAX_EXCEPTION_SIZE (16 KiB) + marginconstreader=resp.body.getReader();letheld=newUint8Array(0);letwritten=0;try{for(;;){const{ done, value }=awaitreader.read();if(done)break;if(signal.aborted)thrownewDOMException('aborted','AbortError');constmerged=newUint8Array(held.length+value.length);merged.set(held);merged.set(value,held.length);// Commit everything except the last HOLDBACK bytes; retain the rest.constcommit=Math.max(0,merged.length-HOLDBACK);if(commit>0){awaitwritable.write(merged.subarray(0,commit));written+=commit;onProgress(written);}held=merged.subarray(commit);}// EOF: inspect the retained tail (latin1 → 1 char/byte for byte-accurate slicing).constframe=findExceptionFrame(latin1(held),tag);constclean=frame ? held.subarray(0,frame.cleanBytes) : held;if(clean.length){awaitwritable.write(clean);written+=clean.length;onProgress(written);}awaitwritable.close();returnframe ? frame.message : null;}catch(e){awaitwritable.abort().catch(()=>{});// leave the partial file; report upstreamthrowe;}finally{reader.releaseLock();}}constlatin1=(bytes)=>{lets='';for(constbofbytes)s+=String.fromCharCode(b);returns;};
findExceptionFrame handles both frames (tagged, and the legacy Code: fallback for < 24.11). Note we scan only the 32 KiB tail — never the whole (multi-GB) stream — so it's O(HOLDBACK), not O(result).
Cancel — mirrors the grid cancel() (app.js:709) but on the export's own id/abort:
functioncancelExport(){if(exportAbort)exportAbort.abort();ch.killQuery(chCtx,exportQueryId,sqlString);// best-effort, never throws}
Button + toolbar (app.js:1195–1200) — exportBtn immediately before shareBtn. Use aria-disabled + a .is-disabled class, not the native disabled attribute: a natively-disabled button swallows pointer events, so its title tooltip often never shows — exactly the "why is this greyed out?" case where we most want it. exportDirect's !app.canExport() guard already makes the click a no-op, so the handler stays wired:
constcan=app.canExport();app.dom.exportBtn=h('button',{class: 'tb-btn'+(can ? '' : ' is-disabled'),'aria-disabled': can ? null : 'true',// h() skips null propstitle: can ? 'Export full result to a file (streams to disk, uncapped)'
: 'Large export requires Chrome/Edge over HTTPS',onclick: ()=>app.actions.exportDirect(),},Icon.download(),'Export');consteditorToolbar=h('div',{class: 'ed-toolbar'},app.dom.runBtn,app.dom.fmtBtn,app.dom.explainBtn,app.dom.saveBtn,h('div',{style: {flex: '1'}}),app.dom.exportBtn,app.dom.shareBtn);
.tb-btn.is-disabled gets the greyed/cursor: not-allowed styling (CSS), staying focusable + hoverable so the tooltip and screen-reader announcement work.
Wire exportDirect into app.actions (app.js:1090).
Mid-stream ClickHouse errors (the subtle part)
Once ClickHouse has flushed any data, the HTTP status is already 200. If the query then fails (memory limit, disk, killed replica), CH can't change the status — so !resp.ok only catches pre-header errors. Verified against ClickHouse source (src/Server/HTTP/WriteBufferFromHTTPServerResponse.{h,cpp}, feature since v24.11, present on our 26.3+ targets), the mid-stream path is a structured frame, not loose text:
A 16-byte random tag is generated per request and sent up front in the X-ClickHouse-Exception-Tag response header (finishSendHeaders(), on the first flush → readable by fetch; and it's in Access-Control-Expose-Headers, so cross-origin basic-auth mode reads it too).
On a post-headers failure CH appends, at the very end of the body:
This is format-independent — the key reason we can honor the query's own FORMAT. The in-format error writer (handle_exception_in_output_format, HTTPHandler.cpp:550) fires only when http_write_exception_in_output_format is true and the format supports it; at its default false (confirmed in Settings.cpp), every format — TSV, JSON, CSV, Parquet, … — falls through to cancelWithException, which writes the plain tagged frame above. So we leave the setting at default and one detector covers all export formats.
Detection is therefore reliable, not heuristic. Read the tag from the header, then look for \r\n__exception__\r\n<tag> in the retained tail — a server-chosen random 16-byte tag can't occur in data by accident (zero false positives), and it's plain ASCII so it's findable even inside a binary (Parquet/Arrow) stream. Because the frame is ≤ 16 KiB and always trailing, the hold-back writer (see streamToFile) withholds the last 32 KiB until EOF and writes only the clean prefix — so the exception is never committed to the file, fixing the "pipe-everything-then-notice" flaw where the error text lands in the output. On a hit: toast "Export incomplete — server error mid-stream: …"; the file holds only the clean bytes received before the failure. (For binary formats the file is necessarily truncated/incomplete on a mid-stream failure regardless — the frame is still excised and the error still surfaced.)
Fallback: servers < 24.11 send no tag header → findExceptionFrame(tail, null) scans for the legacy \nCode: <n>. DB::Exception: prefix (less precise excision, but still detected + reported).
Rejected alternatives: wait_end_of_query=1 (buffers the entire result server-side — defeats the feature, and the grid path rejects it for the same reason, ch-client.js:410); a *WithProgress JSON format (reintroduces per-row parsing, kills the straight byte-copy).
⚠️Verify on npm run local before merge: force a mid-stream failure (e.g. SELECT … FROM numbers(1e12) SETTINGS max_result_bytes=… or a max_memory_usage trip) with FORMAT TabSeparatedWithNames, and confirm (a) the X-ClickHouse-Exception-Tag header is present on the 200 response, and (b) the trailing frame matches the bytes above on the target build.
Why inline, not a child tab
An early draft showed progress in a child tab (the openSchemaView pattern). Dropped, because it buys nothing here and costs plenty:
No durability. Like openSchemaView, the current tab owns the fetch + pipeTo (it holds the token). A child tab would be a pure display surface — it owns nothing. Navigate or close the current tab and the export dies regardless, so you can't "close the editor and keep watching in the child." The one thing a separate tab might buy — surviving the opener leaving — isn't real.
Nothing else is gained. "Keep working during export" and "stay visible across query-tab switches" are both satisfied by a non-modal, app-global inline banner.
It costs. A picker-first flow consumes the click's transient activation, so a window.open after the await is normally pop-up-blocked anyway. Plus the openInTab baggage — style injection, theme mirroring, COOP-severed-document handling, beforeunload→cancel — and extra test surface, all for a "bytes · elapsed · Cancel" readout that needs no screen real estate.
So: inline only. A small fixed banner in the current tab; showExportProgress(cancelExport) builds it, progress.update(bytes) refreshes it, progress.remove() clears it. No openWindow, no fallback branch. (A big pan/zoom graph justifies a tab; a progress line does not.)
Removing the old export
The new streaming Export replaces the old result-panel download:
Delete the results-toolbar Export button (src/ui/results.js:435–438).
DeleteexportResult() and its app.actions entry (src/ui/app.js:959, :1100).
KeepexportableResult() — copyResult() still uses it (app.js:941, :946).
toCSV (src/core/export.js) becomes dead → remove it with its tests (unused exported code is a coverage-gate smell). toTSV stays (copy path).
KeepdownloadFile / env.download — file-menu.js still uses them for library/SQL/MD downloads.
Tradeoff (accepted, per Chromium-leaning stance #71): Firefox/Safari (and plain-HTTP Chromium) now have no export path — the button is visibly disabled with a tooltip rather than absent — and the quick "download the rows I'm looking at as CSV" is gone. The disabled-with-tooltip button keeps the feature discoverable.
Browser / context
Export button
Chromium + HTTPS (or localhost)
enabled → stream uncapped result to disk (query's FORMAT, default TSV)
Chromium + plain HTTP
disabled + tooltip
Firefox / Safari
disabled + tooltip
Acceptance
Export button sits in the editor toolbar next to Share; runs the current editor query uncapped and streams to a user-chosen file via showSaveFilePicker.
Format follows the query: no FORMAT → TabSeparatedWithNames + .tsv; explicit FORMAT JSON/CSV/Parquet/… → that format streamed verbatim with a matching extension (.json/.csv/.parquet/…). The picker's suggested name + accept reflect the format.
Memory stays flat for a multi-million-row export (streamed, never fully buffered).
Inline progress shows bytes written + elapsed and can Cancel; cancel aborts the stream and issues KILL QUERY for the export's own query_id (distinct from the grid run's).
The result grid / tab.result is untouched by an export.
A mid-stream server error (after 200) is detected via the __exception__ frame + X-ClickHouse-Exception-Tag, surfaced as "export incomplete", and never written into the file (hold-back writer commits only the clean prefix). (Verify CH's frame on npm run local first.)
The file picker opens before any await (transient activation preserved); auth/config run only after the handle is held.
On Firefox/Safari or non-secure context, the button is aria-disabled with the explanatory tooltip (not natively disabled) and clicking is a no-op.
User dismissing the save-file picker is a silent no-op (no error toast, no export started).
Export uses its own query_id (export-…) reset in finally; cancel aborts + KILL QUERYs it without touching the grid run.
Old result-panel Export removed (results.js, exportResult, toCSV + its tests); copyResult / toTSV / downloadFile retained.
npm test green at the per-file gate: prepareExportSql, formatFileMeta, exportFilename, findExceptionFrame, exportQuery at 100%; showSaveFilePicker / createWritable / ReadableStream seams injected/mocked in the app.js tests.
Test plan (per layer)
core — prepareExportSql: no FORMAT → appends TSV + format: 'TabSeparatedWithNames'; explicit FORMAT JSON/Parquet → SQL kept verbatim + that format; trailing ; peeled; empty → { sql: '', format: 'TabSeparatedWithNames' }. formatFileMeta: JSON→json, JSONEachRow→jsonl, CSV→csv, TSV/TabSeparated→tsv, Parquet→parquet, Arrow→arrow, unknown/Pretty→txt (+ MIME each). exportFilename: sanitizes, timestamp fallback, honors the passed ext. findExceptionFrame: clean tail → null; tagged frame → {message, cleanBytes} with cleanBytes at the frame start; message with a trailing \n; multibyte (UTF-8) message decoded correctly; frame split across the exact HOLDBACK boundary; legacy (tag=null) Code: path; missing/garbled tag → null.
net — exportQuery: sets query_id + default_format, passes signal; non-OK → throws parsed exception; OK → returns the Response (headers intact for the tag read).
ui (app.js) — inject showSaveFilePicker (fake handle whose createWritable captures write/close/abort calls), fetch returning a ReadableStream, and env.isSecureContext. Assert: only clean bytes reach the writable when the stream ends in a tagged frame (exception excised, abort not called); a clean stream writes everything and closes; picker opens before ensureConfig/getToken are awaited; progress updates as bytes age out of the hold-back; exporting toggles and exportQueryId resets in finally; cancel aborts + killQuerys the export id (grid runQueryId untouched); picker AbortError is silent; !canExport() renders aria-disabled + no-op click; tab.result unchanged throughout.
Reconcile forward work: update #68 (this closes the Export track), CHANGELOG.md[Unreleased], README (Export section + support-matrix note), and deploy/http_handlers.xml only if the deployed surface changes (it doesn't — same / endpoint).
Part of #68 (Roadmap to 1.0.0).
Problem
Users sometimes need the full result — 10M rows to open in Excel/pandas — not to view in the browser. Every path we have today buffers the whole result in memory first:
tab.result(src/core/stream.js),exportResult,src/ui/app.js:959) serializes the already-loaded rows into aBloband triggers a download.For millions of rows that's gigabytes in RAM and a hung tab. It's also capped — it can only export what the grid already fetched (the row cap from #86).
What is not the obstacle (corrected)
The blocker is not authentication. A same-origin
fetchfrom the current tab carriesAuthorization: Bearerfine — the app already streams every query that way. The only thing that can't carry the header is a top-level navigation to a ClickHouse URL (window.open('https://ch/?query=…')/<a href>), which sends cookies, not headers. So Export runs afetchin the current tab, never a navigation — and needs no extra tab or window.The real constraint is streaming a huge response to disk without buffering it in RAM — the File System Access API (
showSaveFilePicker+ streamingresp.bodytohandle.createWritable()), Chromium-only.Solution
Add an Export button to the editor toolbar (next to Share) that runs the current editor query uncapped and streams the result to a user-chosen file, bypassing the result grid entirely, with progress + cancel. The output format follows the query: if the SQL ends in an explicit
FORMAT <x>we stream that format and pick a matching file extension; otherwise we default to TSV.Settled decisions
resp.body.getReader()→handle.createWritable(): constant memory regardless of result size.resp.bodyis already browser-decompressed (fetch handlesContent-Encoding), soenable_http_compression=1stays on (smaller transfer) and the file on disk is plain TSV.showSaveFilePickerand a secure context at button-build time. Where unavailable (Firefox/Safari, or plain-HTTP Chromium), the button rendersaria-disabledwith a tooltip ("Large export requires Chrome/Edge over HTTPS") — not nativelydisabled, so the tooltip actually shows. No buffered-Blobfallback — it would defeat the memory goal (consistent with the Chromium-leaning stance, support matrix Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71).FORMAT <x>(viadetectSqlFormat), stream that format verbatim and derive the file extension from it (JSON→.json, CSV→.csv, Parquet→.parquet, …). If there's noFORMAT, default toTabSeparatedWithNames(header row, no type row — cleanest for Excel/pandas) with.tsv. Either way the grid /tab.resultis never touched by an export.query_id+AbortController, fully separate from the grid run'sapp.state.runQueryId/app.state.abortController. Cancel aborts the stream and issuesKILL QUERYfor the export's id.openSchemaViewchild-tab pattern; dropped — see "Why inline, not a child tab.")Architecture
Layered so the coverage gate stays at 100% per file: pure logic in
src/core/, the fetch insrc/net/behind the injected seam, DOM wiring insrc/ui/behind the injectedshowSaveFilePickerseam.1.
src/core/— pure helpers (100% covered)src/core/format.js— resolve the export SQL + its effective format. Respects a userFORMAT(via the existingdetectSqlFormat); only defaults to TSV when absent:src/core/export.js— map a ClickHouse output format → file{ ext, mime }(by family, so the long tail of names is covered), and build the suggested filename:src/core/stream.js— find ClickHouse's mid-stream exception block in the retained tail (see "Mid-stream errors" below for the wire format + source refs). Operates on a latin1-decoded string so char index == byte offset (the message may be UTF-8 multibyte, but the block start — the offset we slice on — is always ASCII):2.
src/net/ch-client.js— the export requestNote: no
wait_end_of_query— that would buffer the whole result server-side and defeat the feature (the grid path rejects it for the same reason,ch-client.js:410).chUrlalready addsenable_http_compression=1.3.
src/ui/app.js— action, seams, streaming, cancelSeams (near the
openWindowseam, ~app.js:69):Export-local state (sibling to the run state, kept separate so an export and a grid run never clobber each other's cancel state):
The action. Ordering is load-bearing:
showSaveFilePickerrequires the click's transient activation, which any priorawait(a token refresh inensureConfig/getTokencan be a network round-trip > the activation window) would forfeit. So do only synchronous validation, then open the picker, and only after we hold the handle run the async auth/export:Streaming with a hold-back buffer — the exception frame is at most 16 KiB and always at the very end, so we never commit the last
HOLDBACK(32 KiB — 16 KiB max frame + margin) bytes until EOF proves them clean. Bytes only reach disk once they've aged out of the window; at EOF we scan the retained tail and write only the clean prefix, so a mid-stream exception is never written to the file. Memory stays constant (oneHOLDBACKbuffer). This reads the stream directly rather than via aTransformStream, because the write is conditional (we withhold, inspect, then commit) — a passthrough transform can't un-write:findExceptionFramehandles both frames (tagged, and the legacyCode:fallback for < 24.11). Note we scan only the 32 KiB tail — never the whole (multi-GB) stream — so it's O(HOLDBACK), not O(result).Cancel — mirrors the grid
cancel()(app.js:709) but on the export's own id/abort:Button + toolbar (
app.js:1195–1200) —exportBtnimmediately beforeshareBtn. Usearia-disabled+ a.is-disabledclass, not the nativedisabledattribute: a natively-disabled button swallows pointer events, so itstitletooltip often never shows — exactly the "why is this greyed out?" case where we most want it.exportDirect's!app.canExport()guard already makes the click a no-op, so the handler stays wired:.tb-btn.is-disabledgets the greyed/cursor: not-allowedstyling (CSS), staying focusable + hoverable so the tooltip and screen-reader announcement work.Wire
exportDirectintoapp.actions(app.js:1090).Mid-stream ClickHouse errors (the subtle part)
Once ClickHouse has flushed any data, the HTTP status is already
200. If the query then fails (memory limit, disk, killed replica), CH can't change the status — so!resp.okonly catches pre-header errors. Verified against ClickHouse source (src/Server/HTTP/WriteBufferFromHTTPServerResponse.{h,cpp}, feature since v24.11, present on our 26.3+ targets), the mid-stream path is a structured frame, not loose text:X-ClickHouse-Exception-Tagresponse header (finishSendHeaders(), on the first flush → readable byfetch; and it's inAccess-Control-Expose-Headers, so cross-origin basic-auth mode reads it too).MAX_EXCEPTION_SIZE = 16 KiB,EXCEPTION_TAG_LENGTH = 16.This is format-independent — the key reason we can honor the query's own
FORMAT. The in-format error writer (handle_exception_in_output_format, HTTPHandler.cpp:550) fires only whenhttp_write_exception_in_output_formatis true and the format supports it; at its default false (confirmed inSettings.cpp), every format — TSV, JSON, CSV, Parquet, … — falls through tocancelWithException, which writes the plain tagged frame above. So we leave the setting at default and one detector covers all export formats.Detection is therefore reliable, not heuristic. Read the tag from the header, then look for
\r\n__exception__\r\n<tag>in the retained tail — a server-chosen random 16-byte tag can't occur in data by accident (zero false positives), and it's plain ASCII so it's findable even inside a binary (Parquet/Arrow) stream. Because the frame is ≤ 16 KiB and always trailing, the hold-back writer (seestreamToFile) withholds the last 32 KiB until EOF and writes only the clean prefix — so the exception is never committed to the file, fixing the "pipe-everything-then-notice" flaw where the error text lands in the output. On a hit: toast "Export incomplete — server error mid-stream: …"; the file holds only the clean bytes received before the failure. (For binary formats the file is necessarily truncated/incomplete on a mid-stream failure regardless — the frame is still excised and the error still surfaced.)Fallback: servers < 24.11 send no tag header →
findExceptionFrame(tail, null)scans for the legacy\nCode: <n>. DB::Exception:prefix (less precise excision, but still detected + reported).Rejected alternatives:
wait_end_of_query=1(buffers the entire result server-side — defeats the feature, and the grid path rejects it for the same reason,ch-client.js:410); a*WithProgressJSON format (reintroduces per-row parsing, kills the straight byte-copy).npm run localbefore merge: force a mid-stream failure (e.g.SELECT … FROM numbers(1e12) SETTINGS max_result_bytes=…or amax_memory_usagetrip) withFORMAT TabSeparatedWithNames, and confirm (a) theX-ClickHouse-Exception-Tagheader is present on the 200 response, and (b) the trailing frame matches the bytes above on the target build.Why inline, not a child tab
An early draft showed progress in a child tab (the
openSchemaViewpattern). Dropped, because it buys nothing here and costs plenty:openSchemaView, the current tab owns thefetch+pipeTo(it holds the token). A child tab would be a pure display surface — it owns nothing. Navigate or close the current tab and the export dies regardless, so you can't "close the editor and keep watching in the child." The one thing a separate tab might buy — surviving the opener leaving — isn't real.window.openafter theawaitis normally pop-up-blocked anyway. Plus theopenInTabbaggage — style injection, theme mirroring, COOP-severed-document handling,beforeunload→cancel — and extra test surface, all for a "bytes · elapsed · Cancel" readout that needs no screen real estate.So: inline only. A small fixed banner in the current tab;
showExportProgress(cancelExport)builds it,progress.update(bytes)refreshes it,progress.remove()clears it. NoopenWindow, no fallback branch. (A big pan/zoom graph justifies a tab; a progress line does not.)Removing the old export
The new streaming Export replaces the old result-panel download:
src/ui/results.js:435–438).exportResult()and itsapp.actionsentry (src/ui/app.js:959,:1100).exportableResult()—copyResult()still uses it (app.js:941,:946).toCSV(src/core/export.js) becomes dead → remove it with its tests (unused exported code is a coverage-gate smell).toTSVstays (copy path).downloadFile/env.download—file-menu.jsstill uses them for library/SQL/MD downloads.Tradeoff (accepted, per Chromium-leaning stance #71): Firefox/Safari (and plain-HTTP Chromium) now have no export path — the button is visibly disabled with a tooltip rather than absent — and the quick "download the rows I'm looking at as CSV" is gone. The disabled-with-tooltip button keeps the feature discoverable.
Acceptance
showSaveFilePicker.FORMAT→TabSeparatedWithNames+.tsv; explicitFORMAT JSON/CSV/Parquet/… → that format streamed verbatim with a matching extension (.json/.csv/.parquet/…). The picker's suggested name +acceptreflect the format.KILL QUERYfor the export's ownquery_id(distinct from the grid run's).tab.resultis untouched by an export.__exception__frame +X-ClickHouse-Exception-Tag, surfaced as "export incomplete", and never written into the file (hold-back writer commits only the clean prefix). (Verify CH's frame onnpm run localfirst.)await(transient activation preserved); auth/config run only after the handle is held.aria-disabledwith the explanatory tooltip (not natively disabled) and clicking is a no-op.query_id(export-…) reset infinally; cancel aborts +KILL QUERYs it without touching the grid run.results.js,exportResult,toCSV+ its tests);copyResult/toTSV/downloadFileretained.npm testgreen at the per-file gate:prepareExportSql,formatFileMeta,exportFilename,findExceptionFrame,exportQueryat 100%;showSaveFilePicker/createWritable/ReadableStreamseams injected/mocked in theapp.jstests.Test plan (per layer)
prepareExportSql: no FORMAT → appends TSV +format: 'TabSeparatedWithNames'; explicitFORMAT JSON/Parquet→ SQL kept verbatim + thatformat; trailing;peeled; empty →{ sql: '', format: 'TabSeparatedWithNames' }.formatFileMeta: JSON→json, JSONEachRow→jsonl, CSV→csv, TSV/TabSeparated→tsv, Parquet→parquet, Arrow→arrow, unknown/Pretty→txt (+ MIME each).exportFilename: sanitizes, timestamp fallback, honors the passed ext.findExceptionFrame: clean tail → null; tagged frame →{message, cleanBytes}withcleanBytesat the frame start; message with a trailing\n; multibyte (UTF-8) message decoded correctly; frame split across the exactHOLDBACKboundary; legacy (tag=null)Code:path; missing/garbled tag → null.exportQuery: setsquery_id+default_format, passessignal; non-OK → throws parsed exception; OK → returns the Response (headers intact for the tag read).app.js) — injectshowSaveFilePicker(fake handle whosecreateWritablecaptureswrite/close/abortcalls),fetchreturning aReadableStream, andenv.isSecureContext. Assert: only clean bytes reach the writable when the stream ends in a tagged frame (exception excised,abortnot called); a clean stream writes everything andcloses; picker opens beforeensureConfig/getTokenare awaited; progress updates as bytes age out of the hold-back;exportingtoggles andexportQueryIdresets infinally; cancel aborts +killQuerys the export id (gridrunQueryIduntouched); pickerAbortErroris silent;!canExport()rendersaria-disabled+ no-op click;tab.resultunchanged throughout.Reconcile forward work: update #68 (this closes the Export track),
CHANGELOG.md[Unreleased], README (Export section + support-matrix note), anddeploy/http_handlers.xmlonly if the deployed surface changes (it doesn't — same/endpoint).