Exascale CFD in Heterogeneous Systems

Exascale computing has extended the reach of resolved flow simulations in complex, heterogeneous systems far beyond conventional CFD capabilities. As a result, unprecedented pore and micro scale resolution has been achieved in domains that have been traditionally modeled by, and limited to, continuum, effective medium approaches. By making use of computational resources on the new exascale supercomputer, Frontier, at the Oak Ridge Leadership Computing Facility we performed flow simulations that have pushed the limits of domain-to-resolution ratios by several orders of magnitude for heterogeneous media. Our approach is an incompressible, Navier-Stokes CFD solver based on an adaptive, embedded boundary method supported by the Chombo software framework for applied PDEs. The computational workhorse in the CFD application code is an elliptic solver framework in Chombo for pressure-Poisson and viscous, Helmholtz terms that leverages a PETSc-hypre software interface tuned for accelerator-based platforms. We demonstrate scalability of our approach by replicating a unit cylinder packed with microspheres to achieve over 400 billion degrees of freedom simulated. These simulations model domain lengths of over 20 meters with channel volumes of over 400 cm^3 containing millions of packed spheres with 20 micron grid resolution, challenging current understanding of what it means to be a representative elementary volume of the continuum scale in heterogeneous media. We also simulate a range of Reynolds numbers to demonstrate wide applicability and robustness of the approach.