GVI: Guideable Visual Interpretation on medical tomographic images to improve the performance of deep network

Abstract

In medical image analysis, the demand for interpretable deep neural networks is rapidly growing. However, a major challenge is that most existing interpretative methods are applied after training, leading to a lack of integration with the model’s learning process. As a result, these methods often fail to highlight regions within complex medical images critical for decision-making, such as abnormal tissues or lesions, which are essential for accurate diagnoses and treatment planning. This paper introduces Guided Visual Interpretation (GVI), a framework designed to enhance both the performance and interpretability of deep networks. Building on a deep network model with image-level labels, GVI incorporates a small amount of pixel-level annotations combined with attention mechanisms. These mechanisms facilitate visual interpretation through forward propagation, directing the model’s focus to the most relevant regions. By aligning the network’s decision-making with human cognitive processes, GVI improves interpretability. In our study, an attention layer was added after the convolutional layers of a pre-trained classification network. GVI is trained using a mixed supervision approach that integrates pixel-level annotations with a large amount of image-level data. Experimental results on both private and public datasets show that GVI generates visual explanations consistent with human decision-making principles and achieves superior classification accuracy compared to traditional methods. These findings highlight GVI’s potential to improve interpretability and diagnostic performance in critical fields like medical imaging.