External validation of an algorithm to detect vertebral level mislabeling and autocontouring errors

Background and Purpose

This work performs external validation of a previously developed vertebral body autocontouring tool and investigates a post-processing method to increase performance to clinically acceptable levels.

Materials and Methods

Vertebral bodies within CT scans from two separate institutions (40 from institution A and 41 from institution B) were automatically 1) localized and enumerated, 2) contoured, and 3) screened as a means of quality assurance (QA) for errors. Identification rate, contour acceptability rate, and QA accuracy were calculated to assess the tool’s performance. These metrics were compared to those calculated on CTs from the model’s original training dataset, and a post-processing technique was developed to increase the tool’s accuracy.

Results

When testing the model without post-processing on external datasets A and B, accurate identification rates of 83 % and 92 % were achieved for vertebral bodies (C1-L5). Identification rate, contour acceptability rate and QA accuracy were reduced on both datasets compared to accuracies and rates measured on the model’s orginal testing dataset. After algorithm adjustment, identification rate across all vertebrae increased on average by 4 % (p < 0.01) for dataset A and also 4 % on the dataset B (p = 0.01).

Conclusions

A post-processing adjustment within the machine learning pipeline increased performance of vertebral body localization accuracy to acceptable levels for clinical use. External validation of machine learning and deep learning tools is essential to perform before deployment to different insitutions.