Semantic and Visual Enrichment Hierarchical Network for Medical Image Report Generation

Abstract

This paper highlights a novel medical image report generator named SVEH-Net (Semantic and Visual Enrichment Hierarchical Network), which is based on the encoder-decoder framework and attention mechanism. With the consideration of semantic, visual, and tag features fusion, an image feature encoding (IFE) module is introduced to provide global image features for the decoder, and a hierarchical decoder (H-Decoder) which can fusion all semantic and visual features and generate two reports at one time is proposed. In the experiments, our proposed models are evaluated on the Indiana University Chest X-ray radiology report dataset (IU X-ray) and PEIR Gross dataset. On the both two datasets, our model outperforms the state-of-the-art method in BLEU-1/2/3/4, METEOR, and ROUGE-L scores.