Classification of carotid plaque vulnerability by neurosurgical residents using ultrasonography in the clinical field

Objective: We aimed to evaluate the accuracy of the classification of carotid plaque vulnerability (unstable vs. stable plaques) by neurosurgical residents based on carotid ultrasonography (US) images. Methods: A total of 405 subjects with 995 images were included in the study. Using a neuroradiologist’s decision as the reference value, the classification results of five reviewers were analyzed. The sensitivity, specificity, and overall accuracy were estimated. Then, a pairwise comparison of the receiver operating characteristic (ROC) curve and precision-recall curve was performed to compare the reviewers’ classification accuracy. Results: The mean age of the subjects was 70.5 years (range, 44–91 years) and 223 (55.1%) were female. The number of unstable and stable plaques was 236 (24.7%) and 749 (75.3%), respectively. The best-balanced classification performance of plaque vulnerability was a sensitivity of 83.7% (95% confidence interval [CI], 78.5%–88.1%), specificity of 69.0% (95% CI, 65.6%–72.3%), and overall accuracy of 72.7% (95% CI, 69.8%–75.4%). The best ROC performance was an area under the curve (AUC) of 0.583 (95% CI, 0.552–0.614). The precision-recall curve also showed low classification accuracy among the reviewers (AUC difference: 0.028; 95% bootstrap CI, 0.007–0.048). Conclusion: The classification accuracy of neurosurgical residents to discriminate plaque vulnerability seen on carotid US images was low in a real-world clinical setting. Thus, it is necessary to develop systems that help to educate and automatically interpret plaque stability.