Automated Storage Tiering Using Markov Chain Correlation Based Clustering

Abstract

In this paper, we develop an automated and adaptive framework that aims to move active data to high performance storage tiers and inactive data to low cost/high capacity storage tiers by learning patterns of the storage workloads. The framework proposed is designed using efficient Markov chain correlation based clustering method (MCC), which can quickly predict or detect any changes in the current workload based on what the system has experienced before. The workload data is first normalized and Markov chains are constructed from the dynamics of the IO loads of the data storage units. Based on the correlation of one-step Markov chain transition probabilities k-means method is employed to group the storage units that have similar behavior at each point. Such framework can then easily be incorporated in various resource management policies that aim at enhancing performance, reliability, availability. The predictive nature of the model, particularly makes a storage system both faster and lower-cost at the same time, because it only uses high performance tiers when needed, and uses low cost/high capacity tiers when possible.