OTBENCHMARK: AN OPEN SOURCE PYTHON PACKAGE FOR BENCHMARKING AND VALIDATING UNCERTAINTY QUANTIFICATION ALGORITHMS

Abstract

. Over the past decade, industrial companies and academic institutions pooled their efforts and knowledge to propose a generic uncertainty management methodology for computer simulation. This framework led to the collaborative development of an open source software dedicated to the treatment of uncertainties, called “OpenTURNS” (Open source Treatment of Uncertainty, Risk’N Statistics). This paper aims at presenting a new Python package, called “ otbenchmark ”, offering tools to evaluate the performance of a large panel of uncertainty quantification algorithms. It provides benchmark classes containing problems with their reference values. Two categories of benchmark classes are currently available: reliability estimation problems ( i.e., estimating failure probabilities) and sensitivity analysis problems ( i.e., estimating sensitivity indices such as the Sobol’ indices). This package can either be used for validating a new algorithm or automatically comparing various algorithms on a set of problems. Additionally, the package provides several convergence and accuracy metrics to compare the performance of each algorithm. To face high-dimensional problems, otbenchmark offers graphical tools to draw multidimensional events, functions and distributions based on cross-cuts visualizations. Finally, to ensure otbenchmark ’s accuracy, a test-driven software development method has been adopted (using, among others, Git for collaborative development, unit tests and continuous integration). Ultimately, otbenchmark is an industrial platform gath-ering problems with reference values of their solutions and various tools to achieve a robust comparison of uncertainty management algorithms.