A burst resolution technique for data streams management in the real-time data warehouse

Abstract

Data stream sources are currently emerged with the evolution of traditional data warehouse towards real-time data warehouse. Different solutions have been proposed to extract, transform and load the data streams but investigation is still needed to handle the bursts of incoming data streams. In this paper, we have proposed a flow regulation technique which regulates the fast and time varying bursts of data streams. For this purpose, we have adapted and used the token bucket that is simple and flexible mechanism having little overhead. The objective of this research is to minimize the probability of dropping data streams, synchronize processing power and balancing the load of arriving data streams. An algorithm for the flow regulation technique has been proposed to efficiently regulate the data streams. We have evaluated our technique on synthetic dataset and found that flow regulation technique works well in presence of bursty data streams.