Experience Paper: Danaus: isolation and efficiency of container I/O at the client side of network storage

Abstract

Containers are a mainstream virtualization technique commonly used to run stateful workloads over persistent storage. In multi-tenant hosts with high utilization, resource contention at the system kernel often leads to inefficient handling of the container I/O. Assuming a distributed storage architecture for scalability, resource sharing is particularly problematic at the client hosts serving the applications of competing tenants. Although increasing the scalability of a system kernel can improve resource efficiency, it is highly challenging to refactor the kernel for fair access to system services. As a realistic alternative, we isolate the storage I/O paths of different tenants by serving them with distinct clients running at user level. We introduce the Danaus client architecture to let each tenant access the container root and application filesystems over a private host path. We developed a Danaus prototype that integrates a union filesystem with a Ceph distributed filesystem client and a configurable shared cache. Across different host configurations, workloads and systems, Danaus achieves improved performance stability because it handles I/O with reserved per-tenant resources and avoids intensive kernel locking. Danaus offers up to 14.4x higher throughput than a popular kernel-based client under conditions of I/O contention. In comparison to a FUSE-based user-level client, Danaus also reduces by 14.2x the time to start 256 high-performance webservers. Based on our extensive experience from building and evaluating Danaus, we share several valuable lessons that we learned about resource contention, file management, service separation and performance stability.