Investigation of Class Separability Within Object Detection Models in Histopathology

Abstract

Object detection is one of the most common tasks in histopathological image analysis and generalization is a key requirement for the clinical applicability of deep object detection models. However, traditional evaluation metrics often fail to provide insights into why models fail on certain test cases, especially in the presence of domain shifts. In this work, we propose a novel quantitative method for assessing the discriminative power of a model’s latent space. Our approach, applicable to all object detection models with known local correspondences such as the popular RetinaNet, FCOS, or YOLO approaches, allows tracing discrimination across layers and coordinates. We motivate, adapt, and evaluate two suitable metrics, the generalized discrimination value and the Hellinger distance, and incorporate them into our approach. Through empirical validation on real-world histopathology datasets, we demonstrate the effectiveness of our method in capturing model discrimination properties and providing insights for architectural optimization. This work contributes to bridging the gap between model performance evaluation and understanding the underlying mechanisms influencing model behavior.