A training-free framework for valid object counting by cascading spatial and semantic understanding of foundation models

Abstract

Object counting aims to estimate the number of specific objects within an image. Though the estimated numeric metrics (e.g., MAE) by current methods are getting close to the ground truth, the validity and reliability of the counting objects are generally neglected. In this paper, we propose a training-free framework, called ValidCounter, to improve the validity of object counting. Two progressive modules are designed, including a spatial location module and a semantic filtering module. Given focused objects, the first module leverages the Segment Anything Model (SAM) to identify the location of objects with similar spatial features. Subsequently, the second module integrates a pre-trained vision-language encoder to compare semantic features between the focused and identified objects for filtering irrelevant ones. Our framework enhances counting accuracy by ensuring each counting object is similar to focused objects both spatially and semantically. Furthermore, as the used models are based on open vocabulary, our framework is class-agnostic and training-free, and comprehensive experiments on different datasets validate its effectiveness. Experimental results on three public datasets demonstrate that the proposed method achieves 7%-12% and 2%-6% improvement in Precision and F1 over the existing training-free approaches while preserving the competitiveness of other performance metrics. Moreover, our method achieves state-of-the-art performance on counting metrics for the FSCD-LVIS unseen-test dataset, along with top-tier effectiveness. Therefore, it can be stated that the proposed method reduces counting errors while enhancing the reliability of the counting source.