[Discuss] Doris Roadmap 2026

[morningman]

"Scale Intelligence, Accelerate Insights"

Building on 2025's achievements in vector search and indexing capabilities, Apache Doris continues to deepen its AI support in 2026. This roadmap focuses on advancing AI & Hybrid Search capabilities while enhancing query performance, storage efficiency, and data lake integration.

AI & Hybrid Search Innovation:

• Scale vector index to support 10 billion vectors per table with disk-based ANN
• Enhance full-text search with query expressions, scoring, and multi-index support
• Extend hybrid search to Iceberg for unified analytics

Core Enhancements:

• Query engine optimization for complex data types and ETL processing
• Storage improvements for ultra-large tablets and compute-storage separation
• Data lake integration with Iceberg V3 and Paimon support

Roadmap 2025
Roadmap 2024
Roadmap 2023
Roadmap 2022

AI & Hybrid Search

Vector Index

• Implement index-only scan for vector index ([ann] Ann index only scan #57243)
• Implement disk-based ANN (Approximate Nearest Neighbor) for vector index ([feature](ann-index) Support IVF on-disk index type for ANN vector search #61160)
• Optimize compaction policy for vector index
• Enhance vector index capability to support 10 billion vectors in a single table ([feature](be) Implement PQ on-disk vector index with chunk cache #61986)
• Introduce vector index support for Iceberg tables
• Support vector index in merge on write engine. (Support ANN index on MoW table #61994)

Full-Text Search

• Introduce more query expressions: query string and Boolean query ([feature](inverted index) Implement es-like boolean query #58545)
• Implement scoring functionality in the text index ([feature](score) support BM25 scoring in inverted index query_v2 #59847)
• Introduce multi-index support for a single column ([feature](inverted index) support multiple tokenize index in one column #59117)
• Add text index support for Iceberg tables
• Integrate scoring with global lazy materialization

Query Engine

Performance

• Optimize column pruning for complex data types (struct, array, map) ([opt](olap) Support column pruning for struct, array, and map types #57195, [feature](reader) Optimize Complex Type Column Reading with Column Pruning #57204)
• Optimize expression execution for cases such as CASE WHEN and non-const LIKE ([feat](nereids) opt condition expression in filter / join / if /case when #57025)
• Enhance partition pruning capability ([Feature](scan) support runtime partition prune #62589)
• Optimize broadcast join performance
• Implement query condition cache functionality
• Enhance zonemap evaluation to support expressions ([feature](be) Support expression zonemap pruning #63389)

ETL/Incremental Processing

• Enhance spill-to-disk capability to support TPC-DS 10TB workload using 16GB memory ([feat](spill) Support multi-level partition spilling #61212)
• Implement MERGE INTO statement ([feat](load) support merge into #57044, [feature](iceberg) Implements iceberg update delete merge into functionality.  #60482)
• Implement binlog and incremental materialized view functionality ([Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418)
• Implement global query buffer management to reduce memory usage for single queries and make query usage more predictable
• Implement progress bar for long-running queries ([improvement](executor) unify current query runtime statistics and expose task progress #60567)

New Features

• Implement UNNEST functionality ([feature](nereids)support unnest function #58228)
• Implement recursive CTE (Common Table Expression) ([Feature](cte) support recursive cte #58916)
• Implement ASOF join functionality ([feature](join) support ASOF join #59591)
• Introduce Python UDF (User-Defined Function) support ([Feat](udf) Support Python UDF/UDAF/UDTF for Doris #59543)
• Introduce nested variant data type support ([feat](Variant) Support NestedGroup for variant columns with search DSL NESTED clause #60847)
• Enhance function compatibility with Snowflake ([Feature](nereids) Add Snowflake dialect compatibility - Phase 1 & 2 Pure Alias Registration && FE Expression Rewrite #60982)

New DataTypes

• Introduce timestamp with timezone data type ([feature](type)support timestamptz type (#56646) #59399)
• Introduce binary data type

Enhancement

• Unify predicate and expression framework between external tables and internal tables
• Implement short-circuit expression evaluation ([feature](exec) support short circuit evaluation for condition expr #59830)
• Unify local exchange and global exchange, and move local exchange to FE planner ([refactor](local shuffle) Move local exchange planning from BE to FE #63366)

Data Storage

Storage Format

• Optimize compression ratio for string data
• Enhance storage format to support 10k columns in a single file
• Optimize column metadata management for random access
• Optimize nullable column read performance
• Optimize storage for sparse columns in variant data type ([Enhancement](variant) Support mixed subcolumn/doc-value in doc mode #61539)
• Implement partial update functionality for variant sub-fields ([feature](be) Support flexible variant partial update #63182)

Data management

• Enhance tablet management to support ultra-large tablets (100GB+)
• Optimize MOW (Merge-On-Write) import performance for large tablets

File Cache

• Implement table-level cross-compute group synchronized preheating ([feature](cloud) Add table-level event-driven warm up #63832)
• Implement partition time-based TTL (Time-To-Live) support
• Enhance SQL query capability for more granular and reliable cache usage statistics
• Optimize diskless/slow disk scenarios to prevent local disk from becoming a file cache throughput bottleneck
• Implement cache block/allow list policy for fine-grade cache management. ([feature](file-cache) Separate file cache control for OLAP tables and external catalogs #60583)

Compute-Storage Separation

• Implement ultra-fast elastic balance scheduling
• Enhance read-write separation: bind compaction to write compute groups ([feature](cloud) Support compaction read-write separation for cloud mode  #60310)
• Implement distributed cache support for sharing cache across multiple compute groups
• Enhance persistent metadata caching to reduce dependency on metadata service and improve performance

Data Import

• Optimize memory management for large imports with many active tablets that may result in many small files: implement memtable disk spill
• Optimize memory control for scenarios with very large single-row single-column data
• Introduce support for more data import sources, such as AWS Kinesis ([feature](RoutineLoad) Support the Amazon Kinesis #61325)

Data Lakes

Lake Format Performance

• Implement Parquet format Page Cache capability ([feature](vparquet-reader) Implements parquet file page cache. #59307)
• Enable Data Cache by default
• Enhance metadata parsing, planning, and caching for ultra-large scale Iceberg and Paimon ([enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases #59716, [refactor](paimon) Per-catalog Paimon metadata cache with two-level table+snapshot structure #60478)
• Implement Condition Cache for Iceberg and Paimon ([feat](condition cache) Support condition cache for external table #60897)

Materialized View

• Implement snapshot-level incremental refresh for materialized views based on Iceberg and Paimon
• Implement materialized view construction based on Paimon and Iceberg

Data interoperability

• Implement comprehensive Iceberg V3 support ([feature](iceberg)support read iceberg v3 deletion vector.  #59272, [feat](iceberg) Support iceberg v3 row lineage #61398)
• Implement Iceberg data sorting functionality ([feature](iceberg) support insert into iceberg table with sort-order #60540)
• Implement Iceberg Data Rewrite functionality ([feat](iceberg) Implement Iceberg rewrite_data_files action for table optimization and compaction #56413)
• Implement Iceberg Delete/Update functionality ([feature](iceberg) Implements iceberg update delete merge into functionality.  #60482)
• Implement Iceberg/Parquet Variant data type support ([feature](iceberg) Support reading Iceberg variant from Parquet #63192)
• Implement Paimon data write (Support Paimon data write-back: Implement write support for Paimon data. #56005)
• Implement native reader for Paimon MOR (Merge-On-Read) tables (Support Paimon data write-back: Implement write support for Paimon data. #56005)
• Implement Fluss integration ([Feature] Support reading Apache Fluss Table data #59220)
• Implement Paimon Vector and Blob data type support
• Implement standardized Arrow Flight Data Catalog ([Feature](arrow_flight catalog) support arrow_flight catalog #59270)

Metadata Interoperability

• Implement unified permission management for Iceberg REST Catalog ([feature](fe) Support OIDC session credentials for Iceberg REST catalog #63068)
• Implement integration with third-party authentication and authorization systems
• Implement Open Metadata API

Connector Extensibility

• Decouple built-in connectors from FE core int loadable plugins ([Tracking][Catalog] Catalog SPI migration — decouple built-in connectors from FE core into loadable plugins #65185)

Security

• Enhance object storage support for IAM role-based authentication from more cloud vendors ([feat](Iceberg)Rest & S3Table Support Iam-role #60498)

Others

• Refactor all third-party builds to use CMake
• Implement hermetic build support

[zclllyybb]

Very encouraging to see Doris continuing to lead in the field of AI & Hybrid Search! It will bring significant value to actual business.

[Nj-kol]

It would also be great also if you could prioritize adding more AI resource type like Azure Open AI / Mistral

[Nj-kol]

@morningman Is there any plan for backup and recovery ( as part of a broader DR ) in the compute-storage separation mode? This seems to me as an area of pain for Productionization of this mode.

[morningman]

@morningman Is there any plan for backup and recovery ( as part of a broader DR ) in the compute-storage separation mode? This seems to me as an area of pain for Productionization of this mode.

Yes, we are working on "Cluster Snapshot" feature, mainly for backup/restore on Cloud mode

[morningman]

It would also be great also if you could prioritize adding more AI resource type like Azure Open AI / Mistral

Agree!

[Nj-kol]

@morningman Is there any plan for backup and recovery ( as part of a broader DR ) in the compute-storage separation mode? This seems to me as an area of pain for Productionization of this mode.

Yes, we are working on "Cluster Snapshot" feature, mainly for backup/restore on Cloud mode

@morningman That is music to my ears! Sorry to put you on the spot, but any timeline for this feature? We are almost at the cusp of productionization of compute-storage separation mode and desperately need this "Cluster Snapshot" for DR

[Nj-kol]

Hi @morningman , could you please give a tentative timeline when the "Cluster Snapshot" feature would be ready? We are stuck on DR of Compute Storage decoupled mode because of lack of this feature

[gavinchou]

Hi~ @Nj-kol
There is a framework for the snapshot feature for storage-compute mode, you can check the feature with the entrypoint: cloud/src/snapshot/snapshot_manager.h and CloudSnapshotHandler.java
here is the related PRs you can check them out if interested.
Pull requests · apache/doris

BTW, this is a quite complex feature.

[Nj-kol]

@gavinchou I perfectly understand this is quite a complex feature, given the amount of time I have spent recently on this.

But I doubt any Enterprise company would adopt Doris Decoupled mode in production unless the community fixes this feature soon.

I am struggling to justify my one year worth of efforts I spent on Doris when I tell my management that basic DR Features are missing.

I am a strong advocate of Apache Doris,I see huge potential and want the project to succeed. But I think the community needs to prioritise this DR if any large enterprise are to take this project seriously

[morningman]

@gavinchou I perfectly understand this is quite a complex feature, given the amount of time I have spent recently on this.

But I doubt any Enterprise company would adopt Doris Decoupled mode in production unless the community fixes this feature soon.

I am struggling to justify my one year worth of efforts I spent on Doris when I tell my management that basic DR Features are missing.

I am a strong advocate of Apache Doris,I see huge potential and want the project to succeed. But I think the community needs to prioritise this DR if any large enterprise are to take this project seriously

Many commercial companies behind Doris will implements their own Snapshot feature, so we can only implement a open framework so that anyone can implement their owns

[Nj-kol]

@gavinchou I perfectly understand this is quite a complex feature, given the amount of time I have spent recently on this.
But I doubt any Enterprise company would adopt Doris Decoupled mode in production unless the community fixes this feature soon.
I am struggling to justify my one year worth of efforts I spent on Doris when I tell my management that basic DR Features are missing.
I am a strong advocate of Apache Doris,I see huge potential and want the project to succeed. But I think the community needs to prioritise this DR if any large enterprise are to take this project seriously

Many commercial companies behind Doris will implements their own Snapshot feature, so we can only implement a open framework so that anyone can implement their owns

@morningman So are you saying this is "Cluster Snapshot" feature will not be available in the Open Source Doris?

If so, this is highly disappointing. Because this is an ask for a basic feature that is essential for a stable prod deployment.

I would understand more advanced features being available in commerical vendors like VeloDB, but I think lack of even a basic capability does a great disservice to the Open source Doris. I would appeal to the community to kindly reconsider this decision

[morningman]

hi @Nj-kol , it does not mean it will not implemented in open source project. Actually, we welcome everyone to contribute. What i said is we are try to build a more flexible and pluggable framework. So people can implement their owns(include open sourced)