Performance Boosting Opportunities under Communication Imbalance in Power-Constrained HPC Clusters

This paper provides a detailed message-passing interface (MPI) communication characterization across representative HPC applications. It further evaluates performance and power efficiency improvement opportunities. Specifically, it shows that the traditional approach of active polling while waiting for MPI messages is extremely power inefficient, especially under a constrained cluster-level power budget, where processors can only operate at some percentage of their labeled thermal design power (TDP) due to data center infrastructure limits. To mitigate the communication imbalance among different nodes, one can choose to power gate waiting processes and shift remaining power budget to processes that are in the critical execution paths, a technique we call Gate&Shift. With considerations of overheads from power gating and control-loop, Gate&Shift leads to performance improvement without additional power overhead. Gate&Shift is a reactive scheme that does not require prediction mechanisms. With the aid of real MPI traces and hardware measured power data from an HPC cluster, we show that (1) 1 ms control period for power-shifting is sufficient to achieve most potential performance gains, and (2) for a cluster with processors running at 65% of their labeled TDP, Gate&Shift can achieve 7%, 8.5% and 9% performance improvement for AMR Boxlib, Fill Boundary and Big FFT, respectively.