Hybrid Host/Network Topologies for Massive Storage Clusters

Abstract

The high demand for large scale storage capacity calls for the availability of massive storage solutions with high performance interconnects. Although cluster file systems are rapidly improving and have the potential to allow extremely large numbers of commodity storage nodes to be pooled into a single large file-system, the number of ports on individual switches has not been increasing as quickly - the largest switches available today support fewer than 2,000 Gigabit Ethernet ports. Our goal, therefore, is to develop a new interconnect topology that can connect hundreds of thousands of nodes and achieve performance comparable to a single switch of equivalent size. At the same time, such a new topology should be readily buildable using inexpensive components. Our proposed architecture exploits the multiple Ethernet ports that are now standard on servers and combines host- based routing and forwarding with network-based switching to allow massively large storage clusters to be built. Simulation results have shown that our proposed design achieves 72% to 90% of the performance of a single switch capable of accommodating all storage nodes, but our approach scales to hundreds of thousands of nodes. Furthermore, we use common off-the-shelf layer-2 switches rather than more expensive models that support layer-3 routing. Finally, our approach is resilient to network faults because it maintains multiple paths between storage nodes.