CRCNS: Data-efficient natural image reconstruction to advance visual neuroscience

About This Grant

PROJECT SUMMARY (See instructions): We live in diverse environments featuring dynamic, interacting stimuli that can be viewed from an arbitrary set of perspectives and approached under a variety of goal scenarios. However, modern cognitive and visual neuroscience studies often stand in stark contrast to this reality of our lived experience: typically, participants view isolated, simplistic stimuli while brain activation is recorded and reconstructed or decoded neural representations are quantified. Reconstructing naturalistic visual images from fMRI data presents a challenging task: existing approaches require dozens of hours of 7T fMRI data per participant and extensive compute capability, each of which render these methods inaccessible to traditional cognitive neuroscience labs. Our long-term goal is to develop data-efficient methods to enable use of natural stimuli in cognitive neuroscience studies which characterize neural information content and transformations with cognitive operations. Our objective in the present proposal, which is the next step towards our long-term goal, is to demonstrate the efficacy of natural image reconstruction as an experimental tool in cognitive neuroscience via implementing biologically-inspired optimizations to existing approaches and benchmarking their performance on newly-acquired cognitive task data. Our work involves 3 specific aims: (1) maximize data and computational efficiency of image reconstruction approaches, (2) improve image reconstruction using neuroscience principles, and (3) investigate the impact of cognitive tasks on reconstructions from neural representations of natural images. Across all Aims, we will implement and optimize a novel Al-based image reconstruction approach applied to fMRI data. In Aim 1, we will focus on establishing the efficacy of our approach for training decoding models for new participants using minimal new fMRI data. In Aim 2, we will augment our method by incorporating principles from visual neuroscience as informative priors, like spatial receptive fields estimated from retinotopic mapping and feature-selectivity estimated from functional localizers. In Aim 3, we will test our approach by acquiring several benchmark cognitive neuroscience datasets measuring how reconstructed representations are impacted by visual cognitive task demands including visual attention and working memory. This work is innovative because it enables natural image reconstruction with tractable datasets in cognitive neuroscience labs. This work is significant because it will lead to a new understanding of the neural codes supporting visual perception, attention, and working memory, which are impacted in a host of neurological and psychiatric disorders.