A Massively Parallel Multi-Scale FE2 Framework for Multi-Trillion Degrees of Freedom Simulations

The advent of hybrid CPU and accelerator supercomputers opens the door to extremely large multi-scale simulations. An example of such a multi-scale technique, the FE2 approach, has been designed to simulate material deformations, by getting a better estimation of the material properties, which, in effect, reduces the need to introduce physical modelling at macro-scale level, such as constitutive laws, for instance. Both macro- and micro-scales are solved using the Finite Element method, the micro-scale being resolved at the Gauss points of the macro-scale mesh. As the micro-scale simulations do not require any information from each other, and are thus run concurrently, the stated problem is embarrassingly parallel. The FE2 method therefore directly benefits from hybrid machines, the macro-scale being solved on CPU whereas the micro-scale is offloaded to accelerators. The case of a flat plate, made of different materials is used to illustrate the potential of the method. In order to ensure good load balance on distributed memory machines, weighting based on the type of materials the plate is made of is applied by means of a Space Filling Curve technique. Simulations have been carried out for over 5 trillions of degrees of freedom on up to 2,048 nodes (49,152 CPUs and 12,288 GPUs) of the US DOE Oak Ridge National Laboratory high-end machine, Summit, showing an excellent speed-up for the assembly part of the framework, where the micro-scale is computed on GPU using CUDA.