Multimodal Knowledge Triple Extraction Based on Representation Learning

Abstract

Knowledge based visual question-answering is an emerging technique that combines computer vision and natural language processing to address image-based questions. This approach requires the model to possess internal reasoning ability and incorporate external knowledge to enhance its generalization performance. Knowledge graphs are commonly employed to integrate various types of knowledge, such as image knowledge, text knowledge, and basic common sense, into visual language question answering models, significantly enhancing their interpretability. However, introducing knowledge into these models often involves retrieval and pre-training methods, which can introduce noise due to the local correlation among knowledge sources and the existence of modality gaps, thereby affecting the model's performance. To mitigate this issue, this paper proposes a multimodal knowledge extraction approach based on distributed representation learning. The approach models inexpressible multimodal facts using explicit triples, considering the semantic gap between visual and textual modalities. Cross-modal representations are obtained through an attention mechanism, resulting in knowledge triplets comprising visual features, cross-modal representations, and text features. By incorporating these knowledge triplets into the visual language question answering model, the task is transformed into pattern matching using knowledge triples. The proposed approach comprehensively considers multiple factors, is not restricted to specific forms of knowledge, and can effectively incorporate a substantial amount of knowledge. Experimental results demonstrate its superiority, with a performance improvement of 0.5% on the OKVQA dataset compared to the baseline model, highlighting its strong generalization ability.