A sample survey study of poly-semantic neurons in deep CNNs

Abstract

Although deep CNN networks have excellent image classification performance, they do not provide interpretability, and furthermore existing work reveals that these models have complex internals, for example, mysterious polysemantic neurons activate to multiple features. In this work, we analyze the intermediate data of the network dissection paper made by Bau et al. to understand to what extent polysemantic neurons exist. We divide the polysemantic neurons into five types and calculate the percentage of each type by sampling. We find that above 50% neurons identify one concept but there are a quite proportion of neurons that recognize two or more features. This can explain the high classification accuracy and some capacity saving of a deep CNN. By case studies, we draw some conclusions and hypotheses: First, unlike the human visual system, a CNN cannot distinguish detailed features (metaphor: a CNN is like a nearsighted eye). Second, the reason that the CNN is prone to adversarial attacks may be partially due to the polysemantic neurons. Third, polysemantic neurons may partially explain why people wrongly visualize one thing as another in neuroscience.