Towards explainable image composition assessment: a dataset and a model

Abstract

Explainable Image Composition Assessment (Explainable ICA) provides quantitative scores and qualitative explanations. While recent research on ICA has demonstrated remarkable performance in regressing a scalar score for the overall composition perception, we still need a better understanding of how the image compositions, especially their influencing factors, relate to high-level semantics. However, the requirement for a high-quality explainable ICA dataset limits the broad real-world applications like intelligent photography and content recommendation. This deficiency is primarily because the manual annotation is costly, time-consuming, and struggles to capture the inherent aesthetic subjectivity of composition which heavily relies on expert knowledge. In this work, we take a step toward addressing this challenge through the use of multimodal large language models, and construct a large-scale composition description dataset based on the well-designed prompts in a Chain-of-Thought pattern, named CADB Comments. During its development, we ensured diversity across scenes, composition elements, and levels. This dataset comprises 9497 structured textual annotations. Leveraging this dataset, we propose Multimodal Image Composition Assessment (MICA) to bridge quantitative scoring and qualitative explanation, enabling simultaneous output of scores and descriptions. MICA generates descriptions, embeds composition knowledge into visual encoder, and extracts composition representation using the knowledge-enhanced encoder. A composition level contrastive alignment module strengthens correlation between representation and scores via contrastive learning. Extensive experiments on three datasets demonstrate MICA’s competitive performance in explanation, scoring, and classification. The dataset and code are available at https://github.com/dylanqyuan/MICA.