pWebDAV: A Multi-Tier Storage System

Abstract

Experiments using the Large Hadron Collider (LHC) currently generate tens of petabytes of reduced data per year, observational and simulation data in the climate domain is expected to reach eXabytes by 2021, and light source experiments are expected to generate hundreds of terabytes per day. At such extreme scale, the substantial amount of concurrency can cause critical contention issue of the I/O system. This study introduces pWebDAV as a heterogeneous, multi-tier storage system. pWebDAV proposes a dynamically coordinated I/O architecture offering overall data flow solutions (remote-local access). The fundamental idea is to implement, for each data transfer, I/O policies on the fly. pWebDAV controls all I/O nodes, participating in the data transfer, directly regardless of the tier. pWebDAV approach can fully utilize the provided I/O & network resources and is able to minimize disk and network contention. The focus in this study is the Metadata node scalability performance.