Label knowledge guided transformer for automatic radiology report generation

Abstract

Background and Objective

The task of automatically generating radiology reports is a key research area at the intersection of computer science and medicine, aiming to enable computers to generate corresponding reports on the basis of radiology images. This field currently faces a significant data bias issue, which causes words describing diseases to be overshadowed by words describing normal regions in the reports.

Methods

To address this, we propose the label knowledge guided transformer model for generating radiology reports. Specifically, our model incorporates a Multi Feature Extraction module and a Dual-branch Collaborative Attention module. The Multi Feature Extraction module leverages medical knowledge graphs and feature clustering algorithms to optimize the label feature extraction process from both the prediction and encoding of label information, making it the first module specifically designed to reduce redundant label features. The Dual-branch Collaborative Attention module uses two parallel attention mechanisms to simultaneously compute visual features and label features, and prevents the direct integration of label features into visual features, thereby effectively balancing the model's attention between label features and visual features.

Results

We conduct experimental tests using the IU X-Ray and MIMIC-CXR datasets under six natural language generation evaluation metrics and analyze the results. Experimental results demonstrate that our model achieves state-of-the-art (SOTA) performance. Compared with the baseline models, the label knowledge guided transformer achieves an average improvement of 23.3% on the IU X-Ray dataset and 20.7% on the MIMIC-CXR dataset.

Conclusion

Our model has strong capabilities in capturing abnormal features, effectively mitigating the adverse effects caused by data bias, and demonstrates significant potential to enhance the quality and accuracy of automatically generated radiology reports.