Collaborative human-computer fault diagnosis via calibrated confidence estimation

Most intelligent fault diagnostic studies focus solely on improving accuracy, which implies that decisions are made exclusively by a model. This lacks consideration, both from a safety and ethical perspective. Human-computer collaboration leverages the strengths of both parties to provide more informed and reliable decisions, requiring confidence as a key support. However, deep models typically suffer from miscalibration, i.e., the softmax probability does not represent the true likelihood that the predicted label is correct, motivating many calibration methods, among which confidence penalty (CP) receives attention as a simple method. CP’s performance is highly sensitive to a trade-off parameter and relies on cross-validation tests. However, although the parameter value chosen in this way has better overall performance, it does not outperform the other values in every confidence bin. The way CP penalizes the confidence of all samples with equal strength also makes it difficult to calibrate the confidence of some samples. For this reason, this paper proposes adaptive CP, which can adaptively assign a parameter value to each bin. Furthermore, a novel paradigm of collaborative human–computer fault diagnosis based on the method is established. The experimental results elucidate our motivations for designing the method and demonstrate its superiority.