Seg-Cam: Enhancing interpretability analysis in segmentation networks

Abstract

Existing interpretability analysis methods face significant limitations when applied to segmentation networks, such as limited applicability and unclear visualization of weight distribution. To address these issues, a novel approach for calculating network layer weights was established for segmentation networks, such as encoder-decoder networks. Rather than processing individual parameters, this method computes gradients based on pixel-level information. It improves the weight calculation model in the Grad-Cam method by removing the constraint that the model’s output layer must be a one-dimensional vector. This modification extends its applicability beyond traditional CNN classification models to include those that generate feature maps as output, such as segmentation models. It also improves the visualization process by calculating the distribution of feature map weights for the specified layer without changing the model architecture or retraining. Utilizing the image segmentation task as the project context, the seg-cam visualization scheme is incorporated into the initial model. This scheme enables the visualization of parameter weights for each network layer, facilitating post-training analysis and model calibration. This approach enhances the interpretability of segmentation networks, particularly in cases where the head layer contains many parameters, making interpretation challenging.