Enriching object-aware image–text highlight information for visual question generation

Abstract

Visual question generation is a challenging task of comprehensively interpreting images and expressing them in natural language. Generating visual questions requiring detailed information depends on identifying key objects and their context when interpreting images with a highlight on the target object. Conventionally, methods rely on global image information or generate captions for the entire image to use as text for question generation. However, these methods often lack focus on target objects and missing key details. In this paper, we propose an object-aware highlighted visual question generation method that enhances question generation by emphasizing target object features in both image and text representations. Our method consists of two key modules: (1) an image feature extraction and transformation module that extracts and highlights relevant object-specific information, and (2) a visual question generation module that uses this highlighted information to generate more specific and contextually enriched questions. We further introduce mutual information loss to enhance the correlation between generated questions and image content. Extensive experiments on K-VQG, VQA v2.0, and OK-VQA show that our method outperforms state-of-the-art models, especially with a 28.25% BLEU score improvement on K-VQG, highlighting its effectiveness.