Predict Inter-photo Visual Similarity via Pre-trained Computer Vision Models

There were three types of photo sets, 200 student life photos, 242 townscapes, and 100 garden landscapes. The student life photos included various photos, including landscapes, objects, people, and food. On the other hand, there were only garden photos in the garden landscapes. There was some variation of photos in the townscapes. The inter-photo visual similarity had been measured manually for each photo set. Dividing the photo set randomly into training and test data parts, the visual similarity was predicted via pre-trained computer vision models (CVMs), such as CNN, Vision Transformer (ViT), and CLIP. The prediction accuracy could be measured by the trace of the correlation matrix between the original and restored three dimensional MDS coordinates aligned by the Procrustes transformation. Three photo sets were used as a benchmark for the predictive power of CVMs. The image features by ViT models pre-trained on ImageNet-21K and by image encoders of CLIP showed high predictive power in any photo set. Combining different CVMs increased the predictive power. The MDS first axis was well restored for any photo set. For the student life photos with a large photo variation, the MDS from the visual similarity and the MDS from the best CVM was found almost the same, so there seemed no need to predict the visual similarity. For the garden landscapes with a small photo variation, the visual similarity prediction using CVMs was not so successful in the second and third axis of MDS.