Model utility of a deep learning-based segmentation is not Dice coefficient dependent: A case study in volumetric brain blood vessel segmentation

Cerebrovascular disease is one of the world's leading causes of death. Blood vessel segmentation is a primary stage in diagnosing. Although a few deep neural networks have been suggested to automate volumetric brain blood vessel segmentation, few studies have considered the relevance of the evaluation metrics to diagnosing cerebrovascular disease due to the complicated nature of this task. This study aimed to understand if brain vasculature segmentation using a convolutional neural network (CNN) could meet radiologists' requirements for disease diagnosis. We employed a deeply supervised attention-gated 3D U-Net trained based on the Focal Tversky loss function to extract brain vasculatures from volumetric magnetic resonance angiography (MRA) images. Here we show that our training procedure led to biologically relevant results despite not scoring well using the Dice score, a common metric for algorithm evaluation. We achieved Dice (±SD) = 0.71 ± 0.02 and two radiologists confirmed and validated that our method successfully captured the major blood vessel branches of the circle of Willis (CoW) having biological importance, including internal carotid artery (ICA), middle cerebral artery (MCA), anterior cerebral artery (ACA), and posterior cerebral artery (PCA). Adding radiologists' expert opinions, we could fill this gap that using only the current common evaluation metrics, such as the Dice coefficient, is not enough for brain vessel segmentation assessment. These results suggest the additional value for computational approaches that are designed with end-user stakeholders in mind.