Adding GPU Acceleration to an Industrial CPU-Based Simulator, Development Strategy and Results

Running multi-million cell simulation problems in minutes has been a dream for reservoir engineers for decades. Today, with the advancement of Graphic Processing Unit (GPU), we have a real chance to make this dream a reality. Here we present our experience in the step-by-step transformation of a fully developed industrial CPU-based simulator into a fully functional GPU-based simulator. We also demonstrate significant accelerations achieved through the use of GPU technology. To achieve the best performance possible, we choose to use CUDA (NVIDIA GPU’s native language), and offload as much computations to GPU as possible. Our CUDA implementation covers all reservoir computes, which include property calculation, linearization, linear solver, etc. The well and Field Management still reside on CPU and need minor changes for their interaction with GPU-based reservoir. Importantly, there is no change to the nonlinear logic. The GPU and CPU parts are overlapped, fully utilizing the asynchronous nature of GPU operations. Each reservoir computation can be run in three modes, CPU_only (existing one), GPU_only, CPU followed by GPU. The latter is only used for result checking and debugging. In early 2019, we prototyped two reservoir linearization operations (mass accumulation and mass flux) in CUDA; both showed very strong runtime speed-up of several hundred times, 1 P100-GPU (NVIDIA) vs 1 POWER8NVL CPU core rated at 2.8 GHz (IBM). Encouraged by this success, we moved into linear solver development and managed to move the entire linear solver module into GPU. Again, strong speed-up of ~50 times was achieved (1 GPU vs 1 CPU). The focus for 2019 has been on standard Black-Oil cases. Our implementation was tested with multiple "million-cell range" models (SPE10 and other real field cases). In early 2020, we managed to put SPE10 fully on GPU, and finished the entire 2000 day time-stepping in ~35 sec with a single P100 card. After that our effort has switched to compositional AIM (Adaptive Implicit Method), with focus on compositional flash and AIM implementation for reservoir linearization and linear solver, both show early promising results. GPU-based reservoir simulation is a future trend for HPC. The development of a reservoir simulator is complex, multi-discipline and time-consuming work. Our paper demonstrates a clear strategy to add tremendous GPU acceleration into an existing CPU-based simulator. Our approach fully utilizes the strength of the existing CPU simulator and minimizes the GPU development effort. This paper is also the first publication targeting GPU acceleration for compositional AIM models.