Classification of breast lesions using cross-modal deep learning

Automatic detection and classification of lesions in medical images is a desirable goal, with numerous clinical applications. In breast imaging, multiple modalities such as X-ray, ultrasound and MRI are often used in the diagnostic workflow. Training robust classifiers for each modality is challenging due to the typically small size of the available datasets. We propose to use cross-modal transfer learning to improve the robustness of the classifiers. We demonstrate the potential of this approach on a problem of identifying masses in breast MRI images, using a network that was trained on mammography images. Comparison between cross-modal and cross-domain transfer learning showed that the former improved the classification performance, with overall accuracy of 0.93 versus 0.90, while the accuracy of de-novo training was 0.94. Using transfer learning within the medical imaging domain may help to produce standard pre-trained shared models, which can be utilized to solve a variety of specific clinical problems.