SwiftER: Elastic Erasure Coded Storage System

Abstract

Over the life-cycle of a data object, it may be difficult to determine a priori how much redundancy to store it with. The desired degree of fault-tolerance may change over time, for instance, because the importance of the data changes, or the storage system environment changes. If the redundancy is achieved using replication, then changing the degree of fault-tolerance would mean adding (or removing) replicas - a reasonably straightforward operation. However, if erasure code is used instead (which is preferable, given the significantly lower storage overhead of erasure codes with respect to fully replicated systems), then, while shrinking redundancy can still be achieved similarly, expanding redundancy becomes non-trivial. A naive approach will require re-coding, which is both network resource and computation heavy. In this paper, we explore the possibility of using network coding techniques, to both distribute computational load, as well as reduce network usage, and in the process, speed-up the process of creating additional redundancy. The contributions of this paper are defining the problem and analyzing the theoretical limits by leveraging on and extending the existing literature on regenerating codes to realize erasure coded redundancy elasticity, propose a framework to realize code instances that are amenable to network coding based elastic expansion of redundancy, and integrate and benchmark one such code instance (which happens to be optimal with respect to the aforementioned established theoretical limit) with OpenStack Swift to demonstrate the practicality and advantages of the proposed approach.