Weakly supervised commissioning of externally developed auto-segmentation models and applied to male pelvis MR auto-segmentation

Introduction:

When introducing an auto-segmentation model into clinical practice, assessing the quality of the predicted segmentations and the robustness of the model over a wide range of anatomical variation and/or image quality is difficult. Especially, when the model is provided by an external party and the institution introducing the model does not possess a high-quality dataset to commission the model on.

Materials & Methods:

Assuming that a model is more likely to fail for an atypical case as opposed to a more average one, we propose a methodology that selects cases for commissioning using unsupervised anomaly detection. For this, the model supplier provides a set of image/shape features that correlate with model performance on the training data. Next, the receiving hospital can use these features to train an unsupervised anomaly detector on a large dataset of unlabeled cases and use the anomaly scores to select representative cases for commissioning of the model. Since the anomaly detector is trained on unlabeled data, a large, high-quality, curated dataset is not required on the receiving hospital side.

Results:

Using the proposed approach, the likelihood of selecting atypical edge cases with low segmentation performance was increased, as compared to random selection. For a selection of 20 cases, an increase of 22% was observed.

Conclusions:

The increased performance spread provides a more representative range of expected performance in clinical practice. This approach could be used for model commissioning to increase the confidence that the model performs well over a wide range of expected anatomical variation.