The impact of multimodal-multisensory learning on human performance and brain activation patterns

Abstract

The human brain is inherently a multimodal-multisensory dynamic learning system. All information that is processed by the brain must first be encoded through sensory systems and this sensory input can only be attained through motor movement. Although each sensory modality processes different signals from the environment in qualitatively different ways (e.g., sound waves, light waves, pressure, etc.), these signals are transduced into a common language in the brain. The signals are then associated and combined to produce our phenomenology of a coherent world. Therefore, the brain processes a seemingly unlimited amount of multisensory information for the purpose of interacting with the world. This interaction with the world, through the body, is multimodal. The body allows one to affect the environment through multiple motor movements (hand movements, locomotion, speech, gestures, etc.). These various actions, in turn, shape the multisensory input that the brain will subsequently receive. The feedforward-feedback loop that occurs every millisecond among sensory and motor systems is a reflection of these multisensory and multimodal interactions among the brain, body, and environment. As an aid to comprehension, readers are referred to this chapter's Focus Questions and to the Glossary for a definition of terminology. In the following, we begin by delving deeper into how sensory signals are transduced in the brain and how multimodal activity shapes signal processing. We then provide samples of research that have demonstrated that multimodal interactions with the world, through action, facilitate learning. An overview of research on performance measured by overt behavioral responses in adult and developing populations is followed by examples of research on the effects that multimodal learning has on brain plasticity in adults and children. Together, the behavioral and neuroimaging literature underscore the importance of learning through multimodalmultisensory interactions throughout human development.