Scalable Integration of Computational Physics Simulations with Machine Learning

Integration of machine learning with simulation is part of a growing trend, however, the augmentation of codes in a highly-performant, distributed manner poses a software development challenge. In this work, we explore the question of how to easily augment legacy simulation codes on high-performance computers (HPCs) with machine-learned surrogate models, in a fast, scalable manner. Initial naïve augmentation attempts required significant code modification and resulted in significant slowdown. This led us to explore inference server techniques, which allow for model calls through drop-in functions. In this work, we investigated TensorFlow Serving with $\mathbf{gRPC}$ and RedisAI with SmartRedis for server-client inference implementations, where the deep learning platform runs as a persistent process on HPC compute node GPUs and the simulation makes client calls while running on the CPUs. We evaluated inference performance for several use cases on SCOUT, an IBM POWER9 supercomputer, including, real gas equations of state, machine-learned boundary conditions for rotorcraft aerodynamics, and super-resolution techniques. We will discuss key findings on performance. The lessons learned may provide useful advice for researchers to augment their simulation codes in an optimal manner.