Search before asking
Description
Doris has become a key infrastructure component for near-real-time data warehousing.
As business requirements for data freshness and pipeline flexibility continue to grow, the ability to efficiently and flexibly deliver data to downstream systems, as well as to perform incremental computation based on change data, has become an increasingly urgent core requirement.
Against this background, we plan to introduce Binlog support into the current storage-compute integrated architecture of Doris. With Binlog support, Doris will be able to provide a more standard and complete view of data changes, thereby better supporting downstream subscription, data synchronization, incremental materialized views, and stream-batch integrated computation scenarios.
Overall, the implementation of Binlog can be divided into the following core modules:
- Meta moduleResponsible for the definition, organization, persistence, and management of Binlog-related metadata, including version information, transaction associations, lifecycle management, and maintenance of metadata required for queries.
- Read & Write module
Responsible for the writing, storage, and reading of Binlog data, forming the core data path of the Binlog capability. On the write side, it needs to work together with the existing loading, transaction commit, and Rowset generation flows; on the read side, it needs to provide stable and efficient change querying and consumption capabilities.
- Compaction & Schedule module
Responsible for background organization, merging, scheduling, and cleanup of Binlog data, ensuring that the system can maintain controllable storage costs, stable query performance, and reasonable resource efficiency over the long term.
Notice
No response
Related issues
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
Doris has become a key infrastructure component for near-real-time data warehousing.
As business requirements for data freshness and pipeline flexibility continue to grow, the ability to efficiently and flexibly deliver data to downstream systems, as well as to perform incremental computation based on change data, has become an increasingly urgent core requirement.
Against this background, we plan to introduce Binlog support into the current storage-compute integrated architecture of Doris. With Binlog support, Doris will be able to provide a more standard and complete view of data changes, thereby better supporting downstream subscription, data synchronization, incremental materialized views, and stream-batch integrated computation scenarios.
Overall, the implementation of Binlog can be divided into the following core modules:
Responsible for the writing, storage, and reading of Binlog data, forming the core data path of the Binlog capability. On the write side, it needs to work together with the existing loading, transaction commit, and Rowset generation flows; on the read side, it needs to provide stable and efficient change querying and consumption capabilities.
Responsible for background organization, merging, scheduling, and cleanup of Binlog data, ensuring that the system can maintain controllable storage costs, stable query performance, and reasonable resource efficiency over the long term.
Notice
No response
Related issues
No response
Are you willing to submit PR?
Code of Conduct