Evaluating the impact of data layout and placement on the energy efficiency of heterogeneous applications

Heterogeneous compute nodes have become an integral component of today's HPC systems. Recent research has established the importance of data layout and placement on such systems. This paper explores the power and energy aspects of data layout and placement on heterogeneous systems. We present results of an experimental study that evaluates the impact of data layout and placement on candidate HPC node architectures for kernels that exhibit a wide variety of performance characteristics. The results of the study show that data layout and placement can have a significant impact on the energy efficiency of heterogeneous applications. On some platforms, selecting the appropriate layout can yield up to an order-of-magnitude improvement in energy efficiency. The study shows that the conventional approach of using a structure-of-arrays for device-mapped data structures is not always profitable and that in addition to memory divergence, data layout choices are impacted by a variety of factors including arithmetic intensity and task granularity. The results of the study are used to establish a set of energy imperatives to guide data layout and placement across different architectures.