Hierarchical Storage from NVMe to Tapes

Abstract

Hierarchical storage allows to build both high-performance and high-capacity storage systems by combining various technologies like NVMe, SSDs, HDDs, or tapes. While most existing systems only manage a limited number of levels and technologies, the IO-project aims to implement a deep storage hierarchy in the context of Exascale workloads. This system will be capable of managing most of the existing storage devices, from NVMe to tapes. To achieve this, it leverages two open-source object stores: Motr[1], an innovative object store developed by Seagate and designed for Exascale, and Phobos[2], a tape-capable object store developed by CEA. The resulting system will implement features to allow user and applications to optimize their data placement by specifying "hints" like extended attributes. It will also provide features to migrate data massively according to arbitrary criteria. The system will be accessible through a native API, as well as a S3 gateway.