[Bug] Stream scan under CTE may skip NormalizeOlapTableStreamScan

[yujun777]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

When a stream scan is placed inside a CTE, NormalizeOlapTableStreamScan may not run in the old rewrite placement. The stream virtual columns are then left as scan outputs instead of being lowered into parent project aliases, and later rewrite steps treat them as real scan slots.

This breaks hidden stream column resolution in the incremental refresh path and can fail IVM planning.

What You Expected?

Stream scans should be normalized consistently whether they appear directly in the query tree or under a CTE.

How to Reproduce?

A minimal reproduction pattern is:

WITH cte AS (
    SELECT * FROM stream_table
)
SELECT * FROM cte;

In the broken path, the stream scan under the CTE is not lowered into an OLAP/binlog scan plus parent project aliases for the virtual stream columns.

Anything Else?

Related PR: #65468

The fix being proposed preserves LogicalOlapTableStreamScan during deep copy/CTE rewrite and moves NormalizeOlapTableStreamScan into the CTE children rewrite pipeline before OperativeColumnDerive.

[yujun777]

Tracking list: #65265

Fix PR: #65468

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_a53c02f00469
This content is generated by AI for reference only.

Initial triage for #65469:

I checked the live issue, the related PR #65468, and the upstream/master code at base commit e033e2ff8f64da0dc2c36385c5a16a0ecd65b0e4. The report looks code-consistent and should be treated as a likely FE/Nereids rewrite-order bug, not as a user SQL issue.

Why this is plausible:

• BindRelation creates LogicalOlapTableStreamScan for table-stream relations.
• NormalizeOlapTableStreamScan is the rule that lowers a stream scan into base OLAP/binlog scan plans and exposes the stream virtual columns through parent project aliases.
• On current master, NormalizeOlapTableStreamScan is registered in the whole-tree rewrite section before CTE inline/pull-up and inside a notTraverseChildrenOf(LogicalCTEAnchor) scope. A stream scan in the CTE producer subtree can therefore be hidden from that normalize pass. After CTE inline exposes the producer plan, there is no equivalent normalize pass before the CTE child rewrite pipeline reaches OperativeColumnDerive.
• OperativeColumnDerive derives operative slots from LogicalOlapScan.getOutput(). If the stream virtual columns still remain in the scan output, they can be treated as real scan output slots instead of aliases produced by the stream-normalization rule.
• There is also a subtype-preservation issue in the same area: LogicalPlanDeepCopier.visitLogicalCatalogRelation() calls withRelationId() and then withVirtualColumns(). LogicalOlapTableStreamScan overrides withRelationId() on master, but it does not override withVirtualColumns(), so dispatch can fall back to LogicalOlapScan.withVirtualColumns() and return a plain LogicalOlapScan during CTE deep copy/inline.

Assessment of PR #65468:

The proposed direction is appropriate: preserve LogicalOlapTableStreamScan when virtual columns are copied, and move NormalizeOlapTableStreamScan into the CTE children rewrite pipeline after partition pruning and before OperativeColumnDerive. That matches the dependency order implied by the code: normalization needs pruned stream partitions, and operative-column derivation should see the lowered scan/project shape rather than raw stream virtual output slots.

Validation gaps / missing information:

• The issue only gives a SQL shape. To make this reproducible, please add the exact base table DDL, stream object creation, and any required session variables/config such as enable_table_stream and the IVM/MTMV incremental refresh settings.
• Please include the actual IVM planning failure: FE error message or stack trace, the refresh SQL/rewrite SQL if available, and preferably a plan-process or EXPLAIN dump showing the stream virtual columns still present on LogicalOlapTableStreamScan after CTE rewrite.
• PR [fix](fe) Fix stream scan normalization and snapshot/reset pruning #65468 currently changes only FE planner code files. Please add a regression/FE test that creates a real stream-scan path under a CTE and asserts planning succeeds with the stream virtual columns lowered into project aliases. The test should cover the issue pattern, not only a direct stream scan.
• The PR checks were still pending when inspected, with compile/FE UT/performance not completed yet. Do not close this until those finish and the missing regression coverage question is resolved.

Likely impact:

This should mainly affect master table-stream/Nereids planning paths where a stream scan is referenced through a CTE, especially IVM incremental refresh planning. Direct stream scans are less likely to hit this exact placement bug because the current whole-tree normalize pass can see them before CTE handling.