MLLMs-MR: Multi-modal recognition based on multi-modal large language models

Abstract

To address the challenges of recognizing data from different modalities, this study proposes a multi-modal recognition method based on Multi-modal Large Language Models (MLLMs-MR), which can process six distinct data types: images, videos, audio, thermal, point cloud, and event data. Existing methods, such as UniBind, treat language as the central modality and construct a text-centric representation space, effectively reducing the representation imbalance among different modalities and improving recognition accuracy. However, descriptions generated by Multi-modal Large Language Models (MLLMs) are directly used for contrastive learning of text embeddings, which would result in a loss of semantic information from the original input. Furthermore, sole reliance on Large Language Models (LLMs) to be used as embedding centers can lead to category misclassification. To address these issues, we have proposed three key improvements based on UniBind: (1) constructing a category-based knowledge base using MLLMs, effectively reducing irrelevant descriptions; (2) designing fusion embedding center localization, which utilizes LLMs, MLLMs, and basic prompts to enhance the robustness of embedding centers; and (3) proposing a cross-modal attention mechanism that incorporates MLLMs-generated descriptions during training, enabling the model to better learn semantic information from multi-modal data and enhance feature representation. Subsequently, MLLMs-enhanced embeddings are aligned with class labels by contrastive learning to enable the recognition of multi-modal data. Experimental results demonstrate that MLLMs-MR outperforms existing models in multi-modal zero-shot recognition, with a 6.42 % accuracy gain on MSR-VTT. It shows an improvement of 8.19 % during fine-tuning on the ESC 5-fold audio dataset.