OLIC: OnLine Information Compression for scalable hosting infrastructure monitoring

Quality-of-service (QoS) management often requires a continuous monitoring service to provide updated information about different hosts and network links in the managed system. However, it is a challenging task to achieve both scalability and precision for monitoring various intra-node and inter-node metrics (e.g., CPU, memory, disk, network delay) in a large-scale hosting infrastructure. In this paper, we present a novel OnLine Information Compression (OLIC) system to achieve scalable fine-grained hosting infrastructure monitoring. OLIC models continuous snapshots of a hosting infrastructure as a sequence of images and performs online monitoring data compression to significantly reduce the monitoring cost. We have implemented a prototype of the OLIC system and deployed it on the PlanetLab and NCSU's virtual computing lab (VCL). We have conducted extensive experiments using a set of real monitoring data from VCL, Planetlab, and a Google cluster as well as a real Internet traffic matrix trace. The experimental results show that OLIC can achieve much higher compression ratios with several orders of magnitude less overhead than previous approaches.