ExaGRyPE: Numerical general relativity solvers based upon the hyperbolic PDEs solver engine ExaHyPE

Abstract

ExaGRyPE describes a suite of solvers and solver ingredients for numerical relativity that are based upon ExaHyPE 2, the second generation of our Exascale Hyperbolic PDE Engine. Numerical relativity simulations are crucial in resolving astrophysical phenomena in strong gravitational fields and are fundamental in analyzing and understanding gravitational wave emissions. The presented generation of ExaGRyPE solves the Einstein field equations in the standard CCZ4 formulation under a 3+1 foliation and focuses on black hole space-times. It employs a block-structured Cartesian grid carrying a higher-order Finite Difference scheme with full support of adaptive mesh refinement (AMR), it facilitates massive parallelism combining message passing, domain decomposition and task parallelism, and it supports the injection of particles into the grid as static data probes or as moving tracers. We introduce the ExaGRyPE-specific building blocks within ExaHyPE 2, and discuss its software architecture and compute-n-feel.

For this, we formalize the creation of any specific astrophysical simulation with ExaGRyPE as a sequence of lowering operations, where a few abstract logical tasks are successively broken down into finer and finer tasks until we obtain an abstraction level which can directly be mapped onto a C++ executable. The overall program logic is fully specified via a domain-specific Python interface, we automatically map this logic onto a more detailed set of numerical tasks, subsequently lower this representation onto technical tasks that the underlying ExaHyPE engine uses to parallelize the application, before eventually the technical tasks in turn are mapped onto small task graphs including the actual astrophysical PDE term evaluations, initial conditions, boundary conditions, and so forth. These can be injected manually by the user, or users might instruct the solver on the most abstract user interface level to use out-of-the-box ExaGRyPE implementations. We end up with a rigorous separation of concerns which shields ExaGRyPE users from technical details and hence simplifies the development of novel physical models. We present the simulations and data for the gauge wave, static single black holes and rotating binary black hole systems, demonstrating that the code base is mature and usable. However, we also uncover domain-specific numerical challenges that need further study by the community in future work.