Updateable Data-Driven Cardinality Estimator with Bounded Q-error

Abstract

Modern Cardinality Estimators struggle with data updates. This research tackles this challenge within single-table. We introduce ICE, an Index-based Cardinality Estimator, the first data-driven estimator that enables instant, tuple-leveled updates. ICE has learned two key lessons from the multidimensional index and applied them to solve cardinality estimation in dynamic scenarios: (1) Index possesses the capability for swift training and seamless updating amidst vast multidimensional data. (2) Index offers precise data distribution, staying synchronized with the latest database version. These insights endow the index with the ability to be a highly accurate, data-driven model that rapidly adapts to data updates and is resilient to out-of-distribution challenges during query testing. To make a solitary index support cardinality estimation, we have crafted sophisticated algorithms for training, updating, and estimating, analyzing unbiasedness and variance. Extensive experiments demonstrate the superiority of ICE. ICE offers precise estimations and fast updates/construction across diverse workloads. Compared to state-of-the-art real-time query-driven models, ICE boasts superior accuracy (2-3 orders of magnitude more precise), faster updates (4.7-6.9 times faster), and significantly reduced training time (up to 1-3 orders of magnitude faster).