INTERPRETABLE LEARNING-BASED MULTI-MODAL HASHING ANALYSIS FOR MULTI-VIEW FEATURE REPRESENTATION LEARNING

Abstract

In this work, an interpretable learning-based multi-modal hashing analysis (ILMMHA) model is proposed with appli-cation to multi-view feature representation learning. In the proposed model, a cascade network structure is first utilized to reveal the intrinsically semantic representation of input variables. Then, a multi-modal hashing (MMH) method is integrated with the explored semantic representation, gener-ating an interpretable learning-based model for multi-view feature representation. Since MMH is capable of measuring semantic similarity across multiple variables jointly, it provides a natural link between the explored intrinsically semantic representation and its similarity across multi-modal data/information. Benefiting from integration of the cascade structure and MMH, the ILMMHA model leads to a new multi-view feature representation of high quality. To demonstrate the effectiveness and generic nature of the ILMMHA model, we conduct experiments on the cross-modal based audio-visual emotion and text-image recognition tasks, respectively. Experimental results demonstrate the superiority of the proposed model on multi-view feature representation learning.