ExVQA: a novel stacked attention networks with extended long short-term memory model for visual question answering

Abstract

Visual Question Answering (VQA) has garnered significant attention in recent years due to its potential for broad applications across fields such as medicine, education, and entertainment. However, existing VQA methods still face several limitations, including challenges in handling abstract and complex questions, poor generalization, lack of explainability, and susceptibility to noise and bias. In this study, we propose a novel ExVQA model that leverages Stacked Attention Networks (SANs) and Extended Long Short-Term Memory (xLSTM) for Visual Question Answering. Image features are extracted using Sigmoid loss for Language-Image Pre-training (SigLIP), while question features are represented using the Autoregressive Transformer Decoder model (GPT-Neo) and Extended Long Short-Term Memory networks to facilitate the answer generation process. By utilizing the strengths of SANs and xLSTM, our approach aims to overcome the limitations of previous models and enhance the performance and reliability of VQA systems. Evaluation results on three datasets: PathVQA, VQA-Med 2019 and GQA show that our proposed ExVQA model achieves better performance than existing methods, demonstrating great application potential in the fields of medicine, education and entertainment.