Benchmarking deep neural representations for synthetic data evaluation

Abstract

Robust and accurate evaluation metrics are crucial to test generative models and ensure their practical utility. However, the most common metrics heavily rely on the selected data representation and may not be strongly correlated with the ground truth, which itself can be difficult to obtain. This paper attempts to simplify this process by proposing a benchmark to compare data representations in an automatic manner, i.e. without relying on human evaluators. This is achieved through a simple test based on the assumption that samples with higher quality should lead to improved metric scores. Furthermore, we apply this benchmark on small, low-resolution image datasets to explore various representations, including embeddings finetuned either on the same dataset or on different datasets. An extensive evaluation shows the superiority of pretrained embeddings over randomly initialized representations, as well as evidence that embeddings trained on external, more diverse datasets outperform task-specific ones.