bringing-clinic-home-an-at-home-multi-modal-data-collection-ecosystem

Bringing the Clinic Home: An At-Home Multi-Modal Data Collection Ecosystem to Support Adaptive Deep Brain Stimulation

Adaptive deep brain stimulation (aDBS) shows promise for improving treatment for neurological disorders such as Parkinson's disease (PD). aDBS uses ...

University of California, Berkeley

A key goal in the study of visual processing is to obtain a comprehensive description of the relationship between visual stimuli and neuronal responses. One way to guide the search for models is to use a general nonparametric regression algorithm, such as a neural network. We have developed a multilayer feed-forward network algorithm that can be used to characterize nonlinear stimulus-response mapping functions of neurons in primary visual cortex (area V1) using natural image stimuli. The network is capable of extracting several known V1 response properties such as: orientation and spatial frequency tuning, the spatial phase invariance of complex cells, and direction selectivity. We present details of a method for training networks and visualizing their properties. We also compare how well conventional explicit models and those developed using neural networks can predict novel responses to natural scenes.

Nonlinear V1 responses to natural scenes revealed by neural network analysis.

Studies of the primary visual cortex (V1) have produced models that account for neuronal responses to synthetic stimuli such as sinusoidal gratings. Little is known about how these models generalize to activity during natural vision. We recorded neural responses in area V1 of awake macaques to a stimulus with natural spatiotemporal statistics and to a dynamic grating sequence stimulus. We fit nonlinear receptive field models using each of these data sets and compared how well they predicted time-varying responses to a novel natural visual stimulus. On average, the model fit using the natural stimulus predicted natural visual responses more than twice as accurately as the model fit to the synthetic stimulus. The natural vision model produced better predictions in >75% of the neurons studied. This large difference in predictive power suggests that natural spatiotemporal stimulus statistics activate nonlinear response properties in a different manner than the grating stimulus. To characterize this modulation, we compared the temporal and spatial response properties of the model fits. During natural stimulation, temporal responses often showed a stronger late inhibitory component, indicating an effect of nonlinear temporal summation during natural vision. In addition, spatial tuning underwent complex shifts, primarily in the inhibitory, rather than excitatory, elements of the response profile. These differences in late and spatially tuned inhibition accounted fully for the difference in predictive power between the two models. Both the spatial and temporal statistics of the natural stimulus contributed to the modulatory effects.

Natural stimulus statistics alter the receptive field structure of v1 neurons.

Neurons in cortical area V4 respond selectively to complex visual patterns such as curved contours and non-Cartesian gratings. Most previous experiments in V4 have measured responses to small, idiosyncratic stimulus sets and no single functional model yet accounts for all of the disparate results. We propose that one model, the spectral receptive field (SRF), can explain many observations of selectivity in V4. The SRF describes tuning in terms of the orientation and spatial frequency spectrum and can, in principle, predict the response to any visual stimulus. We estimated SRFs for neurons in V4 of awake primates by linearized reverse correlation of responses to a large set of natural images. We find that V4 neurons have large orientation and spatial frequency bandwidth and often bimodal orientation tuning. For comparison, we estimated SRFs for neurons in primary visual cortex (V1). Consistent with previous observations, we find that V1 neurons have narrower bandwidth than that of V4. To determine whether estimated SRFs can account for previous observations of selectivity, we used them to predict responses to Cartesian gratings, non-Cartesian gratings, natural images, and curved contours. Based on these predictions, we find that the majority of neurons in V1 are selective for Cartesian gratings, whereas the majority of V4 neurons are selective for non-Cartesian gratings or natural images. The SRF describes visual tuning properties with a second-order nonlinear model. These results support the hypothesis that a second-order model is sufficient to describe the general mechanisms mediating shape selectivity in area V4.

Spectral receptive field properties explain shape selectivity in area V4.

Recognizing and responding to a remembered stimulus requires the coordination of perception, working memory, and decision-making. To investigate the role of visual cortex in these processes, we recorded responses of single V4 neurons during performance of a delayed match-to-sample task that incorporates rapid serial visual presentation of natural images. We found that neuronal activity during the delay period after the cue but before the images depends on the identity of the remembered image and that this change persists while distractors appear. This persistent response modulation has been identified as a diagnostic criterion for putative working memory signals; our data thus suggest that working memory may involve reactivation of sensory neurons. When the remembered image reappears in the neuron's receptive field, visually evoked responses are enhanced; this match enhancement is a diagnostic criterion for decision. One model that predicts these data is the matched filter hypothesis, which holds that during search V4 neurons change their tuning so as to match the remembered cue, and thus become detectors for that image. More generally, these results suggest that V4 neurons participate in the perceptual, working memory, and decision processes that are needed to perform memory-guided decision-making.

Working memory and decision processes in visual area v4.

Surface visualizations of fMRI provide a comprehensive view of cortical activity. However, surface visualizations are difficult to generate and most common visualization techniques rely on unnecessary interpolation which limits the fidelity of the resulting maps. Furthermore, it is difficult to understand the relationship between flattened cortical surfaces and the underlying 3D anatomy using tools available currently. To address these problems we have developed pycortex, a Python toolbox for interactive surface mapping and visualization. Pycortex exploits the power of modern graphics cards to sample volumetric data on a per-pixel basis, allowing dense and accurate mapping of the voxel grid across the surface. Anatomical and functional information can be projected onto the cortical surface. The surface can be inflated and flattened interactively, aiding interpretation of the correspondence between the anatomical surface and the flattened cortical sheet. The output of pycortex can be viewed using WebGL, a technology compatible with modern web browsers. This allows complex fMRI surface maps to be distributed broadly online without requiring installation of complex software.

Pycortex: an interactive surface visualizer for fMRI.

Functional MRI studies suggest that at least three brain regions in human visual cortex-the parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA; often called the transverse occipital sulcus)-represent large-scale information in natural scenes. Tuning of voxels within each region is often assumed to be functionally homogeneous. To test this assumption, we recorded blood oxygenation level-dependent responses during passive viewing of complex natural movies. We then used a voxelwise modeling framework to estimate voxelwise category tuning profiles within each scene-selective region. In all three regions, cluster analysis of the voxelwise tuning profiles reveals two functional subdomains that differ primarily in their responses to animals, man-made objects, social communication, and movement. Thus, the conventional functional definitions of the PPA, RSC, and OPA appear to be too coarse. One attractive hypothesis is that this consistent functional subdivision of scene-selective regions is a reflection of an underlying anatomical organization into two separate processing streams, one selectively biased toward static stimuli and one biased toward dynamic stimuli.

Functional Subdomains within Scene-Selective Cortex: Parahippocampal Place Area, Retrosplenial Complex, and Occipital Place Area.

One crucial test for any quantitative model of the brain is to show that the model can be used to accurately decode information from evoked brain activity. Several recent neuroimaging studies have decoded the structure or semantic content of static visual images from human brain activity. Here we present a decoding algorithm that makes it possible to decode detailed information about the object and action categories present in natural movies from human brain activity signals measured by functional MRI. Decoding is accomplished using a hierarchical logistic regression (HLR) model that is based on labels that were manually assigned from the WordNet semantic taxonomy. This model makes it possible to simultaneously decode information about both specific and general categories, while respecting the relationships between them. Our results show that we can decode the presence of many object and action categories from averaged blood-oxygen level-dependent (BOLD) responses with a high degree of accuracy (area under the ROC curve > 0.9). Furthermore, we used this framework to test whether semantic relationships defined in the WordNet taxonomy are represented the same way in the human brain. This analysis showed that hierarchical relationships between general categories and atypical examples, such as  and , did not seem to be reflected in representations measured by BOLD fMRI.

Decoding the Semantic Content of Natural Movies from Human Brain Activity.

It has been argued that scene-selective areas in the human brain represent both the 3D structure of the local visual environment and low-level 2D features (such as spatial frequency) that provide cues for 3D structure. To evaluate the degree to which each of these hypotheses explains variance in scene-selective areas, we develop an encoding model of 3D scene structure and test it against a model of low-level 2D features. We fit the models to fMRI data recorded while subjects viewed visual scenes. The fit models reveal that scene-selective areas represent the distance to and orientation of large surfaces, at least partly independent of low-level features. Principal component analysis of the model weights reveals that the most important dimensions of 3D structure are distance and openness. Finally, reconstructions of the stimuli based on the model weights demonstrate that our model captures unprecedented detail about the local visual environment from scene-selective areas.

Human Scene-Selective Areas Represent 3D Configurations of Surfaces.

Predictive models for neural or fMRI data are often fit using regression methods that employ priors on the model parameters. One widely used method is ridge regression, which employs a spherical multivariate normal prior that assumes equal and independent variance for all parameters. However, a spherical prior is not always optimal or appropriate. There are many cases where expert knowledge or hypotheses about the structure of the model parameters could be used to construct a better prior. In these cases, non-spherical multivariate normal priors can be employed using a generalized form of ridge known as Tikhonov regression. Yet Tikhonov regression is only rarely used in neuroscience. In this paper we discuss the theoretical basis for Tikhonov regression, demonstrate a computationally efficient method for its application, and show several examples of how Tikhonov regression can improve predictive models for fMRI data. We also show that many earlier studies have implicitly used Tikhonov regression by linearly transforming the regressors before performing ridge regression.

Voxelwise encoding models with non-spherical multivariate normal priors.

An integral part of human language is the capacity to extract meaning from spoken and written words, but the precise relationship between brain representations of information perceived by listening versus reading is unclear. Prior neuroimaging studies have shown that semantic information in spoken language is represented in multiple regions in the human cerebral cortex, while amodal semantic information appears to be represented in a few broad brain regions. However, previous studies were too insensitive to determine whether semantic representations were shared at a fine level of detail rather than merely at a coarse scale. We used fMRI to record brain activity in two separate experiments while participants listened to or read several hours of the same narrative stories, and then created voxelwise encoding models to characterize semantic selectivity in each voxel and in each individual participant. We find that semantic tuning during listening and reading are highly correlated in most semantically selective regions of cortex, and models estimated using one modality accurately predict voxel responses in the other modality. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received. Humans can comprehend the meaning of words from both spoken and written language. It is therefore important to understand the relationship between the brain representations of spoken or written text. Here, we show that although the representation of semantic information in the human brain is quite complex, the semantic representations evoked by listening versus reading are almost identical. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received.

The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality.

Experience influences behavior, but little is known about how experience is encoded in the brain, and how changes in neural activity are implemented at a network level to improve performance. Here we investigate how differences in experience impact brain circuitry and behavior in larval zebrafish prey capture. We find that experience of live prey compared to inert food increases capture success by boosting capture initiation. In response to live prey, animals with and without prior experience of live prey show activity in visual areas (pretectum and optic tectum) and motor areas (cerebellum and hindbrain), with similar visual area retinotopic maps of prey position. However, prey-experienced animals more readily initiate capture in response to visual area activity and have greater visually-evoked activity in two forebrain areas: the telencephalon and habenula. Consequently, disruption of habenular neurons reduces capture performance in prey-experienced fish. Together, our results suggest that experience of prey strengthens prey-associated visual drive to the forebrain, and that this lowers the threshold for prey-associated visual activity to trigger activity in motor areas, thereby improving capture performance.

Experience, circuit dynamics, and forebrain recruitment in larval zebrafish prey capture.

Over the past few decades, neuroscience experiments have become increasingly complex and naturalistic. Experimental design has in turn become more challenging, as experiments must conform to an ever-increasing diversity of design constraints. In this article, we demonstrate how this design process can be greatly assisted using an optimization tool known as mixed-integer linear programming (MILP). MILP provides a rich framework for incorporating many types of real-world design constraints into a neuroscience experiment. We introduce the mathematical foundations of MILP, compare MILP to other experimental design techniques, and provide four case studies of how MILP can be used to solve complex experimental design challenges.

Design of complex neuroscience experiments using mixed-integer linear programming.

Semantic information in the human brain is organized into multiple networks, but the fine-grain relationships between them are poorly understood. In this study, we compared semantic maps obtained from two functional magnetic resonance imaging experiments in the same participants: one that used silent movies as stimuli and another that used narrative stories. Movies evoked activity from a network of modality-specific, semantically selective areas in visual cortex. Stories evoked activity from another network of semantically selective areas immediately anterior to visual cortex. Remarkably, the pattern of semantic selectivity in these two distinct networks corresponded along the boundary of visual cortex: for visual categories represented posterior to the boundary, the same categories were represented linguistically on the anterior side. These results suggest that these two networks are smoothly joined to form one contiguous map.

Visual and linguistic semantic representations are aligned at the border of human visual cortex.

Encoding models provide a powerful framework to identify the information represented in brain recordings. In this framework, a stimulus representation is expressed within a feature space and is used in a regularized linear regression to predict brain activity. To account for a potential complementarity of different feature spaces, a joint model is fit on multiple feature spaces simultaneously. To adapt regularization strength to each feature space, ridge regression is extended to banded ridge regression, which optimizes a different regularization hyperparameter per feature space. The present paper proposes a method to decompose over feature spaces the variance explained by a banded ridge regression model. It also describes how banded ridge regression performs a feature-space selection, effectively ignoring non-predictive and redundant feature spaces. This feature-space selection leads to better prediction accuracy and to better interpretability. Banded ridge regression is then mathematically linked to a number of other regression methods with similar feature-space selection mechanisms. Finally, several methods are proposed to address the computational challenge of fitting banded ridge regressions on large numbers of voxels and feature spaces. All implementations are released in an open-source Python package called Himalaya.

Feature-space selection with banded ridge regression.

The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. fMRI was used to record human brain activity while four subjects (two female) read words in four conditions that vary in context: narratives, isolated sentences, blocks of semantically similar words, and isolated words. We then compared the signal-to-noise ratio (SNR) of evoked brain responses, and we used a voxelwise encoding modeling approach to compare the representation of semantic information across the four conditions. We find four consistent effects of varying context. First, stimuli with more context evoke brain responses with higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared with stimuli with little context. Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. In individual subjects, only natural language stimuli consistently evoke widespread representation of semantic information. Third, context affects voxel semantic tuning. Finally, models estimated using stimuli with little context do not generalize well to natural language. These results show that context has large effects on the quality of neuroimaging data and on the representation of meaning in the brain. Thus, neuroimaging studies that use stimuli with little context may not generalize well to the natural regime. Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here, we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuro-imaging data and changes where and how semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.

Article	Year
Nonlinear V1 responses to natural scenes revealed by neural network analysis. Neural networks : the official journal of the International Neural Network Society\| PubMed ID: 15288891	2004
Natural stimulus statistics alter the receptive field structure of v1 neurons. The Journal of neuroscience : the official journal of the Society for Neuroscience\| PubMed ID: 15295035	2004
Spectral receptive field properties explain shape selectivity in area V4. Journal of neurophysiology\| PubMed ID: 16987926	2006
Working memory and decision processes in visual area v4. Frontiers in neuroscience\| PubMed ID: 23550043	2013
Pycortex: an interactive surface visualizer for fMRI. Frontiers in neuroinformatics\| PubMed ID: 26483666	2015
Functional Subdomains within Scene-Selective Cortex: Parahippocampal Place Area, Retrosplenial Complex, and Occipital Place Area. The Journal of neuroscience : the official journal of the Society for Neuroscience\| PubMed ID: 27707964	2016
Decoding the Semantic Content of Natural Movies from Human Brain Activity. Frontiers in systems neuroscience\| PubMed ID: 27781035	2016
Human Scene-Selective Areas Represent 3D Configurations of Surfaces. Neuron\| PubMed ID: 30497771	2019
Voxelwise encoding models with non-spherical multivariate normal priors. NeuroImage\| PubMed ID: 31075394	2019
The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality. The Journal of neuroscience : the official journal of the Society for Neuroscience\| PubMed ID: 31427396	2019
Experience, circuit dynamics, and forebrain recruitment in larval zebrafish prey capture. eLife\| PubMed ID: 32985972	2020
Design of complex neuroscience experiments using mixed-integer linear programming. Neuron\| PubMed ID: 33689687	2021
Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature neuroscience\| PubMed ID: 34711960	2021
Feature-space selection with banded ridge regression. NeuroImage\| PubMed ID: 36334814	2022
Semantic Representations during Language Comprehension Are Affected by Context. The Journal of neuroscience : the official journal of the Society for Neuroscience\| PubMed ID: 36973013	2023

Jack Gallant

Biography

JoVE Journal Publications

Other Publications