Tree-based hierarchical fusion network for multimodal finger recognition

Abstract

With digitization comes cyber threats and security vulnerabilities, biometric subject has increasingly evolved from unimodal recognition to more secure and accurate forms of multimodal. However, most existing methods focus on the optimal generation of fusion weighting parameters and the design of models with fixed architecture, and such fixed-architecture fusion methods have difficulties in accurately modeling multimodal finger features with large differences in image distributions. In this paper, a Tree-based Hierarchical Fusion Network (THiFNet) is proposed to fuse features of different modalities by adaptively exploring the common feature space using their interdependencies generated in the convolutional tree. First, in order to extract multi-scale features contained in fingerprint and finger vein images, a Residual Non-Local (Res-NL) backbone network is proposed to compute long-range point-to-point relationships while avoiding the loss of minutiae features extracted by shallow convolutional filters. Further, to adaptively bridge the cross-modal heterogeneity gap, a novel Hierarchical Convolutional Tree (HiCT) is proposed to generate interdependencies between different modalities and within the same modality via channel attention. The primary advantage is that the attention modules used for fusion are dynamically selected by the tree network, modeling a more diverse common feature space and improving accuracy within a limited recognition time. Experimental results on three multimodal finger feature datasets show the framework achieves state-of-the-art results when compared with the other methods.