Are Synthetic Datasets Reliable for Benchmarking Generalizable Person Re-Identification?

Abstract

Recent studies show that models trained on synthetic datasets are able to achieve better generalizable person re-identification (GPReID) performance than that trained on public real-world datasets. On the other hand, due to the limitations of real-world person ReID datasets, it would also be important and interesting to use large-scale synthetic datasets as test sets to benchmark person ReID algorithms. Yet this raises a critical question: are synthetic datasets reliable for benchmarking generalizable person re-identification? In the literature there is no evidence showing this. To address this, we design a method called Pairwise Ranking Analysis (PRA) to quantitatively measure the ranking similarity, and a subsequent method called Metric-Independent Statistical Test (MIST) to perform the statistical test of identical distributions. Specifically, we employ Kendall rank correlation coefficients to evaluate pairwise similarity values between algorithm rankings on different datasets. Then, after removing metric dependency via PRA, a non-parametric two-sample Kolmogorov-Smirnov (KS) test is performed for the judgement of whether algorithm ranking correlations between synthetic and real-world datasets and those only between real-world datasets lie in identical distributions. We conduct comprehensive experiments, with twelve representative algorithms, three popular real-world person ReID datasets, and three recently released large-scale synthetic datasets. Through the designed PRA and MIST methods and comprehensive evaluations, we conclude that the recent large-scale synthetic datasets ClonedPerson, UnrealPerson and RandPerson can be reliably used to benchmark GPReID, statistically the same as real-world datasets. Therefore, this study guarantees the usage of synthetic datasets for both source training set and target testing set, with completely no privacy concerns from real-world surveillance data. Besides, the study in this paper might also inspire future designs of synthetic datasets, as the resulting p-values via the proposed MIST method can also be used to assess the reliability of a synthetic dataset for benchmarking algorithms.