FlexFS: Transparent resilience for GRID storage resources

Abstract

Existing GRID infrastructures rely on explicit user instructions in order to replicate files for the purposes of resiliency. This human-intensive process is inefficient, error prone and, more importantly, makes file replication in GRIDs a cumbersome task. To address this problem, we introduce FlexFS – a fully automated file-system framework that seamlessly plugs into existing GRID structures providing automated file replication and transparent-to-user resilience. FlexFS breaks apart files into blocks and injects resilient information into these blocks through the use of Forward Erasure Correction codes. FlexFS employs a number of methods that facilitate the automated storage and efficient retrieval of the blocks in order to provide I/O throughput similar to that of local hard disks, all in the face of ever-changing utilization and availability of the GRID resources. Compared to currently available GRID replication schemes, FlexFS attains 15% to 230% higher throughput, both for reading and writing files.