Reducing false alarms in fault detection: A comparative analysis between conformal prediction and classical methods applied to PCA and autoencoders

Abstract

Setting detection thresholds in data-driven fault detection is a critical challenge, particularly in ensuring a reliable balance between false alarm rate and fault detection capability. Although conformal prediction has been applied to various domains including medicine, finance, and the monitoring of physical systems, its use in industrial fault detection remains underexplored. This study compares conformal prediction methods with classical threshold-setting techniques used in Principal Component Analysis (PCA) and Autoencoder (AE) based fault detection, using extensive experiments on the Tennessee Eastman Process (TEP). The analysis considers conformal prediction strategies, with marginal and conditional validity alongside traditional parametric approaches for PCA and non-parametric methods for AE. The results highlight the sensitivity of false alarm rates to training data availability, with both traditional and marginal conformal methods often exceeding the targeted false alarm risk when training data are limited. In this context, approaches with conditional validity provide a reliable estimation of the uncertainty associated with the false alarm rate. When sufficient training data are available, conditional conformal methods, particularly those based on the Dvoretzky-Kiefer-Wolfowitz (DKW) and Simes adjustments, provide stricter false alarm rate control, systematically remaining below the predefined risk levels. While this comes at the cost of a slight decrease in fault detection rates, the trade-off is particularly relevant in industrial settings where normal operation is overwhelmingly more frequent than fault occurrences. Overall, conformal prediction demonstrates competitive performance compared to analytically established PCA-based thresholds and the widely used Kernel Density Estimation (KDE) for AE-based fault detection.