Fusing of Medical Images and Reports in Diagnostics of Brain Diseases

Abstract

The combination of MRI images with textual clinical records, has a great potential since the former contains a raw information about study area of the human body, and the latter contains a human integral assessment of the image performed by doctor. In other words, there is a problem of including integral information received from clinicians in medical image processing at the feature fusion level. On the example of the multiple sclerosis diagnosis we study the methods of training deep neural networks to answer the following questions: is it possible to improve the quality of diagnosis of multiple sclerosis by fusing information obtained from a series of MRI images and from texts of medical reports corresponding to these images; what advantages gives an early or a late fusion method respectively in solving this problem? We proposed the end-to-end architecture of the neural network, which, using the "early" information fusion, determines the presence of multiple sclerosis of a patient with a network trust level (accuracy) of 87.5%, compared to the 60% trust level obtained on the same dataset using only MRI images, i.e. without fusion of textual conclusions of radiologists.