What is the doggest dog? Examination of typicality perception in ImageNet-trained networks

Abstract

Due to the emergence of numerous model architectures in recent years, researchers finally have access to models that are diverse enough to properly study them from the perspective of cognitive psychology theories, e.g. Prototype Theory. The theory assumes that the degree of membership in a basic-level category is graded. As a result, some concepts are perceived as more central (typical) than others. The most typical category is called a prototype. It can be perceived as the clearest example of a category, reflecting the redundancy structure of the category as a whole. Its inverse is called an anti-prototype. Reasonable perception of prototypes and anti-prototypes is important for accurate projection of the world structure onto the class space and more human-like world perception beyond simple memorization. That is why it is beneficial to study deep models from the perspective of prototype theory. To enable it, we propose 3 methods that return the prototypes and anti-prototypes perceived by deep networks for a specific basic-level category. Additionally, one of our methods allows to visualize the centrality of objects. The results on a wide range of 42 networks trained on ImageNet (Convolutional Networks, Vision Transformers, ConvNeXts and hybrid models) reveal that the networks share the typicality perception to a large extent and that this perception does not lie so far from the human one. We release the dataset with per-network prototypes and anti-prototypes resulting from our work to enable further research on this topic.