Category-selective representation of relationships in visual cortex

Abstract

Understanding social interaction requires processing social agents and their relationship. Latest results show that much of this process is visually solved: visual areas can represent multiple people encoding emergent information about their interaction that is not explained by the response to the individuals alone. A neural signature of this process is an increased response in visual areas, to face-to-face (seemingly interacting) people, relative to people presented as unrelated (back-to-back). This effect highlighted a network of visual areas for representing relational information.How is this network organized?Using functional MRI, we measured brain activity of healthy female and male humans (N=42), in response to images of two faces or two (head-blurred) bodies, facing toward or away from each other. Taking thefacing>non-facingeffect as signature of relation perception, we found that relations between faces and between bodies were coded in distinct areas, mirroring the categorical representation of faces and bodies in visual cortex. Additional analyses suggest the existence of a third network encoding relations between (non-social) objects. Finally, a separate occipitotemporal network showed generalization of relational information across body, face and non-social object dyads (multivariate-pattern classification analysis), revealing shared properties of relations across categories. In sum, beyond single entities, visual cortex encodes the relations that bind multiple entities into relationships; it does so in a category-selective fashion, thus respecting a general organizing principle of representation in high-level vision. Visual areas encoding visual relational information can reveal the processing of emergent properties of social (and non-social) interaction which trigger inferential processes.Significance statementUnderstanding social interaction requires representing the actors as well as the relation between them. We show that the earliest, rudimentary representation of a social interaction is formed in visual cortex. Using fMRI on healthy adults, we measured the brain responses to two faces or two (head-blurred) bodies, and found that, beyond representing faces and bodies, the visual cortex represents their relations, distinguishing between seemingly interacting (face-to-face) and non-interacting (back-to-back) faces/bodies. Moreover, we found that information about face and body relations is represented in separate networks, in line with the general organizing principle of categorical representation in visual cortex. The brain network encoding visual relational information may represent emergent properties of interacting people, which underlie the cognitive representation of social interaction.