Evidence-based context-aware log data management for integrated monitoring system

Abstract

Integrated monitoring system, enabled with semi-structured datastore, is a promising solution for monitoring SaaS systems. However, according to increasing scale of SaaS systems and their long-term of service operations, the monitoring system has faced the problem in response times of log analysis and storage consumption. Our empirical observation is that the problem is primarily derived from the unselective log processing of semi-structure datastore, whereas there should be heterogeneities in log data that we can take advantage of for efficient log management. Based on this observation, we first attest this insight by investigating the usage patterns of log data in a quantitative manner with an actual dataset of log access histories obtained from a SaaS system serving to enterprise users, and we show that there are heterogeneities in required retention period of logs, response time, and amount of data, depending on log data category and its analysis scenario. Armed with the evidence found from the investigation, we design a methodology of context-aware log management, key features of which are to speculatively pre-cache the log analysis and to proactively archive ones depending on log data category and analysis scenario. Evaluation with a prototype implementation shows that the proposed method reduces the response time and the storage consumption.