Efficient transfer learning for multi-channel convolutional neural networks

Abstract

Although most convolutional neural networks architectures for computer vision are built to process RGB images, more and more applications complete this information with additional input channels coming from different sensors and data sources. The current techniques for training models on such data, generally leveraging transfer learning, do not take into account the imbalance between RGB channels and additional channels. If no specific strategy is adopted, additional channels are underfitted. We propose to apply channel-wise dropout to inputs to reduce channel underfitting and improve performances. This improvement of performances may be linked to how much new information is brought by additional channels. We propose a method to evaluate this complementarity between additional and RGB channels. We test our approach on three different datasets: a multispectral dataset, a multi-channel PDF dataset and an RGB-D dataset. We find out that results are conclusive on the first two while there is no significant improvement on the last one. In all cases, we observe that additional channels underfitting decreases. We show that this difference of efficiency is linked to complementary between RGB and additional channels.