Correlation-model-based reduction of monitoring data in data centers

Abstract

Nowadays, in order to observe and control data centers in an optimized way, people collect a variety of monitoring data continuously. Along with the rapid growth of data centers, the increasing size of monitoring data will become an inevitable problem in the future. This paper proposes a correlation-based reduction method for streaming data that derives quantitative formulas between correlated indicators, and reduces the sampling rate of some indicators by replacing them with formulas predictions. This approach also revises formulas through iterations of reduction process to find an adaptive solution in dynamic environments of data centers. One highlight of this work is the ability to work on upstream side, i.e., it can reduce volume requirements for data collection of monitoring systems. This work also carried out simulated experiments, showing that our approach is capable of data reduction under typical workload patterns and in complex data centers.