Utilizing Progressive File Layout Leveraging SSDs in HPC Cloud Environments

With the advent of HPC (High-Performance Computing) Cloud, storage systems based on parallel file systems such as Lustre have become one of important components to support various types of workloads from many users having different storage access patterns. One of the solutions to adapt to such various I/O access patterns is Progressive File Layout (PFL) which can adjust the chunk size and stripe pattern dynamically according to the I/O traffics. However, PFL still has a performance limitation to the small file I/O due to the HDD specific hardware characteristics. In this paper, we propose an optimized Progressive File Layout (PFL) method which can effectively leverage a combination of SSDs and HDDs to support mixed sizes of I/O workloads. By dynamically directing relatively small file I/O operations into the SSD based storage devices, our framework can significantly improve the performance. We have performed a comparative analysis of our PFL on SSDs in Lustre file system with PFL on HDDs and vanilla Lustre configuration with HDDs. Experimental results from various workloads with mixed sizes of I/O threads on our testbed show that the proposed PFL on SSDs can achieve enhanced total I/O throughput by more than six times compared to conventional solutions.