Hierarchical Feature Map Characterization in Fashion Interpretation

Recent advances in computer vision have been driven by the introduction of convolutional neural networks (ConvNets). Almost all existing methods that use hand-crafted features have been re-examined by ConvNets and achieved state of-the-art results on various tasks. However, how ConvNets features lead to outstanding performance is not completely interpretable to humans yet. In this paper, we propose a Hierarchical Feature Map Characterization (HFMC) pipeline in which semantic concepts are mapped to subsets of kernels based on feature maps and corresponding filter responses. We take a closer look at ConvNets feature maps and analyze how taking different sets of feature maps into account affect output accuracy. We first determine a set of kernels named Generic kernels and prune them from the network. We then extract a set of Semantic kernels and analyze their effects on the results. Generic kernels and Semantic kernels are extracted based on the co-occurrence and energy activation levels of feature maps in the network. To evaluate our proposed method, we design a visual recommendation system and apply our HFMC network to retrieve similar styles to query clothing items on the DeepFashion dataset. Extensive experiments demonstrate the effectiveness of our approach to the task of style retrieval on fashion products.