Comparative Study on the Generalization Ability of Machine Learning and Deep Learning Algorithms for Quality Assessment of Wearable PPG Recordings

Abstract

One of the major challenges in using photoplethysmography (PPG) sensors for heart rate monitoring in real-world settings is ensuring signal quality. This work evaluates and compares quality assessment methods using generic machine learning (ML) and deep learning (DL) pipelines, on a unique and comprehensive framework that includes different sensors, wavelengths, measurement locations, and recording environments. The PPG signals from one proprietary and five publicly available datasets were labeled in terms of quality by comparing the PPG-derived heart rate to a reference heart rate estimated from simultaneous electrocardiograms. Diverse techniques based on common ML classifiers and one- and two-dimensional convolutional neural networks (CNN) were trained on a dataset and tested on the remaining ones. The results showed that several generated models performed comparably to previous studies when they were tested on datasets with similar measurement positions and sensors to the training database. Specifically, reductions in sensitivity, specificity, and F1-score of less than 3% from training to testing were observed on some methods. Contrarily, they reported a notably poorer performance when tested on datasets presenting conditions different from the training. Even the best-performing model, based on the well-known, pre-trained CNN AlexNet, experienced a performance drop of over 20% in that situation. These findings show that the analyzed ML and DL methods lack the ability to generalize across PPG signals captured from diverse environments, sensors, wavelengths, and measurement locations. This suggests that developing case-specific methods might be the shortest path towards reliable PPG quality assessment.