Large Language Models Estimate Fine-Grained Human Color–Concept Associations

Abstract

People reliably associate the meanings of both abstract and concrete words with colors distributed over color space, a phenomenon that influences aspects of visual cognition ranging from object recognition to interpreting information visualizations. Prior research has hypothesized that color–concept associations arise from the cross-modal statistical structure of experience, but it remains unclear whether natural environments contain such structure or whether learning systems can discover it without strong prior constraints. To address these questions, we investigated whether GPT-4, a multimodal large language model, can estimate color–concept association ratings that approximate those made by people. We tested 71 colors spanning perceptual color space and a variety of concepts varying in abstractness. GPT-4 ratings correlated strongly with human ratings across a range of prompting strategies, outperforming prior state-of-the-art methods for automatically estimating color–concept associations from images. In an empirical study assessing people's ability to interpret the meanings of colors in information visualizations, palettes generated from GPT-4's rating data were not only interpretable but, in some cases, more effective than those based on human ratings. Taken together, our results suggest that high-order covariance between language and perception, present in web-scale data, provide sufficient information to learn color–concept associations without initial constraints, and that machine-derived associations can support the optimization of information visualizations for visual communication.