Developing Hierarchies for Image Classification Model Evaluation

Classes within computer vision (CV) datasets often exhibit hierarchical structures such as super-subordinate IS-A relations. While some common performance metrics for evaluating CV models such as “top-5 error” ignore hierarchical structure, metrics for hierarchical scoring exist, yet effectiveness for meaningful evaluation is dependent on the ability of the hierarchy to reflect important semantic relationships between classes. Most hierarchical scoring methods reward closeness between prediction and ground truth classes. Such schemes may produce the same score when a child is misclassified as a terrorist as when a car is misclassified as a vehicle or helicopter, ignorant of the different levels of impact of these misclassifications.An approach for developing context-aware hierarchies for use with existing evaluation metrics to reflect the cost of misclassification is needed. The contribution of this paper is to provide a hierarchy construction framework that penalizes misclassifications accordingly given a list of importance ordered categories and a hierarchical scoring method. The framework is demonstrated in a hierarchy selection use case and compared quantitatively against the “top-5 error” metric and a simple super-subordinate relation hierarchical scoring. We qualitatively discuss the efficacy and implications of each approach.