Understanding document images by introducing explicit semantic information and short-range information interaction

Abstract

Methods on the document visual question answering (DocVQA) task have achieved great success by using pre-trained multimodal models. However, two issues are limiting their performances from further improvement. On the one hand, previous methods didn't use explicit semantic information for answer prediction. On the other hand, these methods predict answers only based on global information interaction results and generate low-quality answers. To address the above issues, in this paper, we propose to utilize document semantic segmentation to introduce explicit semantic information of documents into the DocVQA task and design a star-shaped topology structure to enable the interaction of different tokens in short-range contexts. This way, we can obtain token representations with richer multimodal and contextual information for the DocVQA task. With these two strategies, our method can achieve 0.8430 ANLS (Average Normalized Levenshtein Similarity) on the test set of the DocVQA dataset, demonstrating the effectiveness of our method.