Few-Shot Learning Based on Multimodal Information Processing

Few-shot learning aims to develop models with strong generalization capabilities using a small number of training samples. However, most learning methods rely solely on the visual features of a few samples to represent entire categories, leading to poor category representativeness. In contrast, humans can utilize multimodal information to learn category features, thereby making them more representative. Hence, this article emulates the human multimodal learning mechanism by integrating visual features with textual information, thereby facilitating the model’s acquisition of more representative and robust category features. Specifically, this article introduces a novel multimodal fusion mechanism—the visual-semantic fusion selection mechanism (VSFSM)—which comprises a fusion selection module (FS-Module) and a category enhancement module (CE-Module). These two modules collaboratively enhance the model’s classification performance. The FS-Module aligns and fuses semantic information with visual features across both channel and spatial dimensions, performing feature selection and reconstruction. This process not only generates representative category features but also mitigates the impact of noise. The CE-Module guides the model to emphasize category-specific features in the query images, ultimately yielding representative visual-semantic category features while reducing the interference of noise in the query images. Additionally, to better facilitate few-shot learning, this article introduces a novel objective loss function for optimized training. Extensive comparative and ablation experiments conducted on multiple datasets further validate the effectiveness of the proposed method.