SKFormer: Diagram captioning via self-knowledge enhanced multi-modal transformer

Abstract

Diagram captioning aims to generate sentences with the assistance of key visual objects and relationships. This task is the key basis of several applications like cross-modal retrieval and textbook question answering. Most diagrams consist of simple color blocks and geometric shapes, and the knowledge conveyed is professional and diverse. This not only results in high annotation costs, but also exacerbates the gap between sparse visuals and complex semantics. In this paper, we propose a self-knowledge enhanced multi-modal Transformer denoted as SKFormer, which is based on an encoder–decoder architecture. The encoder includes a perception aggregation graph network PAG, a self-knowledge mining module SKM, and a multi-modal semantic interaction module MMSI. The PAG network takes diagram patches as nodes, and the relationships between nodes as edges, integrating visual perception laws to enhance the visual representation. The SKM utilizes multi-modal LLMs and OCR tools to mine implicit and explicit knowledge in diagrams, while the MMSI is used for the interaction between the visual and semantic contents. The enhanced diagram representation is processed by the decoder to generate sentences, assisting learners in mastering its knowledge. By conducting comprehensive experiments on two datasets, we demonstrate that the SKFormer achieves superior performance over the competitors.