Transformed visual working memory representations in human occipitotemporal and posterior parietal cortices.

Abstract

Recent fMRI studies reported transformed representations between perception and visual working memory (VWM) in human early visual cortex (EVC). This is inconsistent with the still widely cited original proposal of the sensory account of VWM, which argues for a shared perception-VWM representation based on successful cross-decoding of the two representations. Although cross-decoding was usually lower than within-VWM decoding and consistent with transformed VWM representations, this has been attributed to experimental differences between perceptual and VWM tasks: once they are equated, the same representation is expected to exist in both. Including human participants of both sexes, this study compared target and distractor representations during the same VWM delay period for the same objects, thereby equating experimental differences. Even with strong VWM representations present throughout occipitotemporal cortex (OTC, including EVC) and posterior parietal cortex (PPC), fMRI cross-decoding revealed significant representational differences between distractors (perception) and targets (VWM) in both regions. Similar differences existed between target encoding (perception) and delay (VWM), being greater in OTC than PPC, indicating more invariant target representations in PPC than OTC. As only part of the sensory input is usually task-relevant, sustaining sensory input in VWM without selection/refinement/consolidation is both taxing and unnecessary. Transformed representations, mediated by task goals and associative areas coding task-relevant information (e.g., PPC), can easily account for these and other recent findings. A task-driven transformed account of VWM thus better captures the nature of VWM representation in the human brain (including EVC) than the sensory representations originally proposed by the sensory account of VWM.Significance Statement The original proposal of the sensory account of visual working memory (VWM) argues for a shared representation between perception and VWM in sensory areas. This assumption, however, was not thoroughly tested due to differences in experimental settings in prior studies. Using fMRI cross-decoding and closely matched experimental conditions, this study compared object representations when they were VWM targets and distractors and during the encoding and delay period of VWM. Both comparisons revealed significant representational differences between perception and VWM in human sensory areas. These results are inconsistent with the sensory nature of VWM representations as it is originally proposed. Instead, they support a task-driven transformed account of VWM in which sensory input is selected/refined/consolidated before VWM storage in these areas.