Taming metadata storms in parallel filesystems with metaFS

Abstract

Metadata performance remains a serious bottleneck in parallel filesystems. In particular, when complex applications start up on many nodes at once, a "metadata storm" occurs as each instance traverses the filesystem in order to search for executables, libraries, and other necessary runtime components. Not only does this delay the application in question, but it can render the entire system unusable by other clients. To address this problem, we present MetaFS, a user-level overlay filesystem that sits on top of an existing parallel filesystem. MetaFS indexes the static metadata content of complex applications and delivers it in bulk to execution nodes, where it can be cached and queried quickly, while relying on the existing parallel filesystem for data delivery. We demonstrate that MetaFS applied to a complex bioinformatics application converts the metadata load placed on a production Panasas filesystem from 1.1 million operations per task to 1.9 MB of bulk data per task, increasing the metadata scalability limit of the application from 66 nodes to 5,000 nodes.