Planning to be incremental: Scene descriptions reveal meaningful clustering in language production

Abstract

How do speakers plan complex descriptions and then execute those plans? In this work, we attempt to answer this question by asking subjects to describe complex visual scenes. We posit that speakers begin planning by organizing the scene into meaningful clusters or groupings of objects. Speakers describe the scene cluster by cluster, allowing for some planning time between each cluster. To test these ideas, in a preregistered study 30 participants described 30 indoor and outdoor scenes while their speech was recorded. Physical distance was calculated by identifying the centroid point of each object and then computing the Euclidean distance between centroid points for every object pair. Semantic distance was calculated using ConceptNet Numberbatch to obtain the semantic similarity between object labels. A clustering algorithm was then applied to establish the appropriate number of clusters per scene and to assign objects to each cluster. We observed that, consistent with our hypothesis, objects separated by shorter physical distances and objects that are semantically more similar were discussed in closer temporal proximity in the verbal descriptions. In addition, word productions that involved jumping from one cluster to another took longer to initiate than those associated with the same cluster. We conclude that speakers address the linearization problem by establishing clusters of objects and using them to facilitate incremental planning. This approach treats multiutterance language production as a type of foraging behavior, where people balance exploration and exploitation.