Ensemble-based label adjudication (EBLA): A method for validating continuous physiological measures against observer-dependent reference standards.

Abstract

Comparing a novel physiological measure against an observer-dependent reference standard creates uncertainty when results disagree because errors may arise from either method. We propose ensemble-based label adjudication (EBLA), a computational framework that uses agreement between two imperfect instruments to create a high-confidence training set for adjudicating discordant observations. EBLA was validated using the Wisconsin Breast Cancer Dataset (n = 569) with simulated label noise (10%-30%). Five classifiers were trained on concordant cases using stratified 5-fold cross-validation. In the primary 15%/15% noise scenario, majority-vote EBLA improved accuracy from 0.85 to 0.97, sensitivity from 0.87 to 0.98, and specificity from 0.84 to 0.97. Supermajority and unanimity thresholds did not improve accuracy and excluded 1.4% and 2.5% of cases. EBLA consistently outperformed Cleanlab across all noise conditions. Application to 281 paired Mechanical Ventilation Respiratory Distress Observation Scale (MV-RDOS) assessments and inspiratory effort measurements (PmusPTP) from 45 mechanically ventilated patients increased accuracy from 0.81 to 0.86, primarily by improving specificity from 0.88 to 0.94. EBLA provides a transparent, interpretable framework for sensitivity analysis when validating physiological measurements against imperfect observer-dependent reference standards.