Report is a mixture of topics: Topic-guided radiology report generation

Abstract

Radiologists are in desperate need of automatic radiology report generation (RRG) for alleviating the workload and preventing the inexperienced from making mistakes in diagnosis. From our perspective, each radiology report can be viewed as a mixture of topics, where the topics extend from the disease annotations. Taking into account the abundance of clinical details in radiology reports, harnessing pertinent topic knowledge has the potential to greatly enhance the quality of the generated reports. Hence, we propose a topic-guided radiology report generation framework, which begins by probabilistically inferring the topics of radiographs, followed by the incorporation of related topic graphs and n-grams as expert knowledge. In the process of report generation, each word is generated conditioned on the selected topics. Additionally, we propose a bag-of-words planning, which acts as a novel form of encode–decode stream, providing guidance for report generation. Extensive experimental results on two widely-used radiology reporting datasets (i.e., IU-Xray and MIMIC-CXR) demonstrate that our method outperforms previous state-of-the-art methods. Specially, we introduce an innovative concept in topic-based RRG and clarify its internal functioning mechanism from a probabilistic standpoint.