Measuring Modality Utilization in Multi-Modal Neural Networks

Abstract

Multimodal data provides information from different sensor types about the same underlying phenomenon and enhances machine learning performance. However, neural networks trained end-to-end on all the modalities tend to rely mostly on one of the most dominant modalities. The black box nature of neural networks makes it difficult to assess the reliance of the network on various modalities. This work presents a novel modality utilization metric that quantifies the network reliance on different modalities. The proposed metric is validated on NTIRE-21 (classification problem) and MCubeS (image segmentation problem) datasets. The modality utilization metric contributes towards the explainability of multimodal neural networks and offers great utility in the field of multimodal data fusion.