Evaluation of AI-based nerve segmentation on ultrasound: relevance of standard metrics in the clinical setting.

Abstract

Background: In artificial intelligence for ultrasound-guided regional anaesthesia, accurate nerve identification is essential. The technology community typically favours objective metrics of pixel overlap on still-frame images, whereas clinical assessments often use subjective evaluation of cine loops by physician experts. No clinically acceptable threshold of pixel overlap has been defined for nerve segmentation. We investigated the relationship between these approaches and identify thresholds for objective pixel-based metrics when clinical evaluations identify high-quality nerve segmentation.

Methods: cNerve™ is a deep learning segmentation tool on GE Healthcare's Venue™ ultrasound systems. It highlights nerves of the interscalene-supraclavicular-level brachial plexus, femoral, and popliteal-level sciatic block regions. Expert anaesthesiologists subjectively rated overall segmentation quality of cNerve™ on ultrasound cine loop sequences using a 1-5 Likert scale (1 = poor; 5 = excellent). Objective assessments of nerve segmentation, using the Intersection over Union and Dice similarity coefficient metrics, were applied to frames from sequences rated 5.

Results: A total of 173 still image frames were analysed. The median Intersection over Union for nerves was 0.49, and the median Dice similarity coefficient was 0.65, indicating variable performance based on objective metrics, despite subjective clinical evaluations rating the artificial intelligence-generated nerve segmentation as excellent.

Conclusions: Variable objective segmentation metric scores correspond to excellent performance on clinically oriented assessment and lack the context provided by subjective expert evaluations. Further work is needed to establish standardised evaluation criteria that incorporate both objective pixel-based and subjective clinical assessments. Collaboration between clinicians and technologists is needed to develop these evaluation methods for improved clinical applicability.