The role of auditory source and action representations in segmenting experience into events

Abstract

Sounds are generated by interactions between objects in the world and carry information about the sound’s sources and the objects’ sound-generating actions. This dual nature of auditory information poses a problem for defining and investigating auditory object representations in staged theories of perception. In this Review, we describe a framework for separating auditory source and action representations. Auditory source and action representations differ from each other in how they are formed, their relation to prediction, the information they carry, how they are experienced and remembered, and the brain responses associated with them. We also suggest that auditory source and action representations are part of event segmentation: structuring information about the environment and what is happening in it. In real life, auditory scenes are resolved together with other modalities, producing an integrated episodic description of the environment. Thus, event segmentation can guide the integration of information from different modalities and mediate the effects of learned knowledge on auditory scene analysis. We end by discussing how these insights offer important advantages for the development of more comprehensive theories and computational models of sound perception in natural scenes. Sounds are generated by interactions between objects in the world. In this Review, Winkler and Denham describe how representations of sources and actions enable humans to segment complex auditory experience into meaningful units.