Automated regression test method for scientific computing libraries: Illustration with SPHinXsys

Abstract

Scientific computing libraries, whether in-house or open-source, have witnessed enormous progress in both engineering and scientific research. Therefore, it is important to ensure that modifications to the source code, prompted by bug fixing or new feature development, do not compromise the accuracy and functionality that have been already validated and verified. This paper introduces a method for establishing and implementing an automatic regression test environment, using the open-source multi-physics library SPHinXsys as an illustrative example. Initially, a reference database for each benchmark test is generated from observed data across multiple executions. This comprehensive database encapsulates the maximum variation range of metrics for different strategies, including the time-averaged, ensemble-averaged, and dynamic time warping methods. It accounts for uncertainties arising from parallel computing, particle relaxation, physical instabilities, and more. Subsequently, new results obtained after source code modifications undergo testing based on a curve-similarity comparison against the reference database. Whenever the source code is updated, the regression test is automatically executed for all test cases, providing a comprehensive assessment of the validity of the current results. This regression test environment has been successfully implemented in all dynamic test cases within SPHinXsys, including fluid dynamics, solid mechanics, fluid-structure interaction, thermal and mass diffusion, reaction-diffusion, and their multi-physics couplings, and demonstrates robust capabilities in testing different problems. It is noted that while the current test environment is built and implemented for a particular scientific computing library, its underlying principles are generic and can be easily adapted for use with other libraries, achieving equal effectiveness.