Building and Validating a Clinical Ultrasound Image Reporting Model

The main job of a radiologist is to understand the essential information hidden in medical images and write diagnostic reports, which are very helpful for subsequent clinical treatment. However, due to the difficulty of interpreting medical images, it requires long-term training. If the training time is short and the experience is insufficient, it will lead to errors in subsequent clinical diagnosis. In addition, the aging population has increased the workload for radiologists, especially with more elderly patients. Therefore, in the case of insufficient workforce and time costs, this study established an Encoder-Decoder architecture for ultrasound image report generation. The Encoder used Faster RCNN to extract lesion-related features from the image, while the Decoder used LSTM to describe the lesion features in words. This approach can effectively assist radiologists in writing diagnostic reports. Faster RCNN and LSTM have shown superior performance in computer vision and natural language processing, but their performance may not be as expected when the dataset is insufficient. Especially, the collection of medical images and reports is difficult, which may result in generated reports that cannot be used. Therefore, this study introduces the idea of prior knowledge, which integrates the lesion organs classified by Faster RCNN and the lesion image features into LSTM to improve the accuracy of describing the lesion organ names and reduce the model’s description errors of organs in small samples, thus increasing the trust of physicians in the report. Finally, in the experimental results, introducing prior knowledge of lesion organ names has a better effect, and the generated reports all contain organ names related to ultrasound images.