Hybrid dermoscopy image classification framework based on deep convolutional neural network and Fisher vector

Abstract

Dermoscopy image is usually used in early diagnosis of malignant melanoma. The diagnosis accuracy by visual inspection is highly relied on the dermatologist's clinical experience. Due to the inaccuracy, subjectivity, and poor reproducibility of human judgement, an automatic recognition algorithm of dermoscopy image is highly desired. In this work, we present a hybrid classification framework for dermoscopy image assessment by combining deep convolutional neural network (CNN), Fisher vector (FV) and support vector machine (SVM). Specifically, the deep representations of subimages at various locations of a rescaled dermoscopy image are first extracted via a natural image dataset pre-trained on CNN. Then we adopt an orderless visual statistics based FV encoding methods to aggregate these features to build more invariant representations. Finally, the FV encoded representations are classified for diagnosis using a linear SVM. Compared with traditional low-level visual features based recognition approaches, our scheme is simpler and requires no complex preprocessing. Furthermore, the orderless representations are less sensitive to geometric deformation. We evaluate our proposed method on the ISBI 2016 Skin lesion challenge dataset and promising results are obtained. Also, we achieve consistent improvement in accuracy even without fine-tuning the CNN.