A pyDAOS Approach for Enabling Efficient Data Handling in DAOS

Abstract

High-Performance Computing (HPC) systems require efficient storage solutions to handle vast amounts of data. Distributed Asynchronous Object Storage (DAOS) is designed for such environments, offering scalable and high performance storage. Our study initially explores the capabilities and performance of DAOS using pyDAOS when integrated with a key-value store (KV store). The existing methodologies are analyzed to identify inefficiencies, particularly in handling large I/O operations. This paper proposes an enhanced approach that incorporates chunking for large I/O operations, automated resource selection, and simplified system administration via server discovery and by fetching configuration details. This results show significant improvements in efficiency, scalability, and usability of DAOS using pyDAOS, making it a robust solution for modern data intensive applications.