Adaptive load shedding via fuzzy control in data stream management systems

Abstract

Data stream management systems (DSMS) aim to process massive data streams in a timely fashion to support important applications, e.g., financial market analysis. However, DSMS can be overloaded due to large bursts in data stream arrivals and data-dependent query executions. To avoid overloads, we design a new load shedding scheme by applying distributed fuzzy logic control, which is very effective to deal with uncertainties in highly dynamic systems such as DSMS, based on the per-stream backlog and selectivity of each query operator. We have implemented our approach by extending an open source distributed DSMS. The performance evaluation using high-rate Internet traces shows that our approach closely supports a specified backlog bound for each data stream queue, while improving the query processing delay, with little overhead.