Content-based brain tumor retrieval for MR images with joint deep and handcrafted visual features

A novel feature extraction framework is proposed to improve the performance of the brain tumor retrieval system. To extract information that radiologists pay attention to when diagnosing brain tumors, not only features describing the location and layout of the tumor in the brain but also those for texture variations of the tumor region and tumor-surrounding tissues are extracted. The gray level co-occurrence matrix (GLCM) and Fisher vector (FV) are calculated as handcrafted features for the augmented tumor regions. Two deep features are extracted respectively from the whole brain MR images and the augmented tumor regions by fine-tuning the Xception model. Then the handcrafted and deep features are fused together after implementing a feature selection procedure based on roulette wheel selection (RWS) method. Extensive experiments are conducted on the brain CE-MRI dataset. The results show that the proposed system can achieve average mAP of 98.17±0.88% and Prec@10 of 97.56±1.16%, which outperforms the state-of-the-art retrieval systems by a large margin on the same dataset.