Evaluating effectiveness of round-off error compensation with three methods in shallow-water models

Abstract

High-performance computing (HPC) limitations remain a significant bottleneck in the development of numerical models. Mixed-precision techniques, which reduce arithmetic precision to improve speed and memory efficiency, offer a promising solution. However, these methods inevitably introduce increased round-off errors that may destabilize model integrations and require smaller integration steps. This study investigates whether round-off error compensation methods can mitigate such precision-reduced errors. Three widely used methods are evaluated including Gill, Kahan, and Quasi Double-Precision (QDP) within shallow-water models. The suitability of using the double-precision fourth-order Runge-Kutta (RK4-DBL) method as a benchmark is first validated through idealized 1D linear shallow-water model experiments with known analytical solutions. Subsequently, ten perturbed initial-condition experiments are conducted for 2D nonlinear shallow-water model to assess the robustness of each compensation method relative to the RK4-DBL benchmark. When applied to fourth-order Runge-Kutta (RK4) in single precision (RK4-SGL), the Gill, Kahan and QDP methods reduce surface-height root-mean-square (RMSE) errors by approximately one order, four orders, and half an order of magnitude, respectively. In terms of computational cost, runtimes increased by 53%, 4%, and 7% relative to the double-precision reference, respectively. Among these compensation methods, the Kahan method achieves the best performance in both error compensation and computational efficiency, followed by the Gill method. The QDP method, though less effective than the other two, still provides meaningful improvements. Overall, this study demonstrates that these three round-off error compensation methods can improve the accuracy of mixed-precision numerical models while maintaining a reasonable computational cost.