Overview
Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel—University of Southern California
Imagine the sound of a bell ringing. What is happening in the brain when we conjure up a sound like this in the "mind's ear?" There is growing evidence that the brain uses the same mechanisms for imagination that it uses for perception.1 For example, when imagining visual images, the visual cortex becomes activated, and when imagining sounds, the auditory cortex is engaged. However, to what extent are these activations of sensory cortices specific to the content of our imaginations?
One technique that can help to answer this question is multivoxel pattern analysis (MPVA), in which functional brain images are analyzed using machine-learning techniques.2-3 In an MPVA experiment, we train a machine-learning algorithm to distinguish among the various patterns of activity evoked by different stimuli. For example, we might ask if imagining the sound of a bell produces different patterns of activity in auditory cortex compared with imagining the sound of a chainsaw, or the sound of a violin. If our classifier learns to tell apart the brain activity patterns produced by these three stimuli, then we can conclude that the auditory cortex is activated in a distinct way by each stimulus. One way to think of this kind of experiment is that instead of asking a question simply about the activity of a brain region, we ask a question about the information content of that region.
In this experiment, based on Meyer et al., 2010,4 we will cue participants to imagine several sounds by presenting them with silent videos that are likely to evoke auditory imagery. Since we are interested in measuring the subtle patterns evoked by imagination in auditory cortex, it is preferable if the stimuli are presented in complete silence, without interference from the loud noises made by the fMRI scanner. To achieve this, we will use a special kind of functional MRI sequence known as sparse temporal sampling. In this approach, a single fMRI volume is acquired 4-5 s after each stimulus, timed to capture the peak of the hemodynamic response.
Procedure
1. Participant recruitment
- Recruit 20 participants.
- Participants should be right-handed and have no history of neurological or psychological disorders.
- Participants should have normal or corrected-to-normal vision to ensure that they will be able to see the visual cues properly.
- Participants should not have metal in their body. This is an important safety requirement due to the high magnetic field involved in fMRI.
- Participants should not suffer from claustrophobia, since the fMRI requires lying in the small space of the scanner bore.
2. Pre-scan procedures
- Fill out pre-scan paperwork.
- When participants come in for their fMRI scan, instruct them to first fill out a metal screen form to make sure they have no counter-indications for MRI, an incidental-findings form giving consent for their scan to be looked at by a radiologist, and a consent form detailing the risks and benefits of the study.
- Prepare participants to go in the scanner by removing all metal from their body, including belts, wallets, phones, hair clips, coins, and all jewelry.
3. Provide instructions for the participant.
- Tell the participants that they will see a series of several short videos inside the scanner. These videos will be silent, but they may evoke a sound in their "mind's ear." Ask the participant to focus on and encourage this auditory imagery, to try to "hear" the sound as best as they can.
- Stress to the participant the importance of keeping their head still throughout the scan.
4. Put the participant in the scanner.
- Give the participant ear plugs to protect their ears from the noise of the scanner and ear phones to wear so they can hear the experimenter during the scan, and have them lie down on the bed with their head in the coil.
- Give the participant the emergency squeeze ball and instruct them to squeeze it in case of emergency during the scan.
- Use foam pads to secure the participants head in the coil to avoid excess movement during the scan, and remind the participant that it is very important to stay as still as possible during the scan, as even the smallest movements blur the images.
5. Data collection
- Collect high-resolution anatomical scan.
- Begin functional scanning.
- Synchronize the start of stimulus presentation with the start of the scanner.
- To achieve the sparse temporal sampling, set the acquisition time of an MRI volume to 2 s, with a 9-s delay between volume acquisitions.
- Present the silent videos via a laptop connected to a projector. The participant has a mirror above their eyes, reflecting a screen at the back of the scanner bore.
- Synchronize the start of each 5-s video clip to begin 4 s after the previous MRI acquisition starts. This will ensure that the next MRI volume is acquired 7 s after the start of the video clip, to capture the hemodynamic activity that corresponds to the middle of the movie.
- Present three different silent videos that evoke vivid auditory imagery: a bell swinging back and forth, a chainsaw cutting through a tree, and a person playing a violin.
- In each functional scan, present each video 10 times, in random order. With each trial lasting 11 s, this will result in a scan 330 s (5.5 min) long.
- Perform 4 functional scans.
6. Data analysis
- Define a Region of Interest (ROI).
- Use the high-resolution anatomical scan of each participant to trace the voxels that correspond to the early auditory cortex (Figure 1). This corresponds to the surface of the temporal lobe, called the planum temporale. Use the anatomical features of each person's brain to create a mask specific to their auditory cortex.
Figure 1: Region of interest tracing. The surface of the planum temporale has been traced on this participant's high-resolution anatomical image, and is shown here in blue. In green is the control mask of the frontal pole. These voxels will be used for MVPA analysis.
- Pre-process the data.
- Perform motion correction to reduce motion artifacts.
- Perform temporal filtering to remove signal drifts.
- Train and test the classifier algorithm.
- Divide the data into training and testing sets. Training data will be used to train the classifier, and the left-out testing data will be used to assess what it has learned. To maximize the independence of training and testing data, leave out data from one functional scan as the testing set.
- Train a Support Vector Machine algorithm on the labeled training data from auditory cortex in each subject. Test the classifier's ability to correctly guess the identity of the unlabeled testing set, and record the classifier's accuracy.
- Repeat this procedure 4 times, leaving out each scan as testing data each time. This type of procedure, in which each section of the data is left out once, is called cross-validation.
- Combine classifier accuracies across the 4 cross-validation folds by averaging.
- Statistical testing
- To determine if the classifier is performing better than chance (33%), we can compare results at the group level to chance. To do this, gather the accuracies for each subject, and test that the distribution is different from chance using a non-parametric Wilcoxon Signed-Rank test.
- We can also ask whether the classifier is performing better than chance for each individual. To determine the probability of a given accuracy level in chance data, create a null distribution by training and testing the MVPA algorithm on data whose labels have been randomly shuffled. Permute the labels 10,000 times to create a null distribution of accuracy values, and then compare the actual accuracy value to this distribution.
- To demonstrate the specificity of the information within the auditory cortex, we can train and test the classifier on voxels from a different location in the brain. Here, we will use a mask of the frontal pole, taken from a probabilistic atlas and warped to fit each subject's individual brain.
Auditory imagery is a process that gives rise to the experience of hearing sounds, even when there are no external auditory stimuli present.
For instance, think about hearing the sound of a cell phone ringing. While information within memory underlies this imaginary event, evidence suggests that an individual’s brain uses the same mechanisms for imagination as those that are involved in the actual perception.
Just upon imagining the ringing, regions within the auditory cortex become activated. However, even though this is true across acoustic stimuli, how sounds are encoded to allow for the detailed processing of distinct sounds—like distinguishing between a doorbell chime and a song playing on the radio—is an important question.
Based on previous work by Meyer and colleagues, this video demonstrates how to combine functional magnetic resonance imaging—fMRI—with presentations of different silent videos to investigate how the brain responds to auditory imagery.
We will also describe how to use a method called multi-voxel pattern analysis—MVPA for short—to predict what subjects have imagined by analyzing patterns of activation obtained during fMRI sessions.
In this experiment, participants lie in an fMRI scanner and are shown a series of silent videos. Each one—whether it’s a rooster crowing, a chainsaw cutting through a tree, or a person playing a piano—evokes distinctive and vivid auditory imagery, and they are asked to imagine the sounds during every single presentation.
The imaging acquisition procedure relies upon sparse temporal sampling, whereby a single fMRI volume is acquired 4 to 5 s after each stimulus is presented. Such timing captures the peak of the hemodynamic response and reduces the likelihood that signals would be masked by scanner noise.
Each imagined sound is expected to induce subtle yet distinctive patterns of neural activity, specifically in the auditory cortex. Here, pattern is the key word: The classical way to analyze these data uses a univariate approach, where the individual voxels—representing some level of activation—are collapsed into a single average.
These values are then compared across sounds and may not produce any significant differences in activation levels.
Instead, using a multivariate analysis, multiple voxels are laid out for each sound and activation levels can be compared collectively, across all voxels—contributing to a unique overall pattern for each imagined sound.
With this multi-voxel pattern analysis, or MVPA, approach, if the patterns are indeed sensitive to specific content, then it’s possible that they could be used to predict the original stimulus. That’s right—MVPA is often referred to as a mind-reading technique!
To achieve this prediction aspect, more intensive processing must be performed after collecting the participants’ data, which are divided into training and testing sets.
Labeled data from the training set are first subjected to machine-learning calculations, specifically, a Support Vector Machine algorithm. This process is used to accurately classify data by recognizing features in the neural patterns that may distinguish the three types of sounds from one another.
After the classifier has learned features to accurately identify the types, it’s presented with unlabeled data from the testing set, and its guesses are then compared to the correct stimuli labels.
In this case, the classification performance serves as the dependent variable—recorded as the classifier’s accuracy—which is also compared to voxels evoked in a different location in the brain, such as the frontal pole.
The classifier is expected to predict the identification of auditory imagery, revealing the importance of MVPA in detecting content-specific activity within the auditory cortex.
Per experimental and safety concerns, verify that all participants are right-handed, have normal or corrected-to-normal vision, no history of neurological disorders or claustrophobia, and do not possess any metal in their body. Also, ensure that they fill out the necessary consent forms.
Before proceeding, explain that they will see several short silent videos in the scanner that may evoke a sound in their mind. Ask them to focus on the imagined sounds, "hear" them as best as they can, and to remain still for the duration of the task.
Now, prepare the participant to enter the scanner. To see these steps in detail, please refer to another fMRI video produced in this collection.
Following preparation, align the participant and send them inside the bore. In the adjacent room, first collect a high-resolution anatomical scan. Then, synchronize the start of the silent video presentation with the start of the functional scan.
To achieve sparse temporal sampling, set the acquisition time of an MRI volume to 2 s, with a 9-s delay in between.
Importantly, coordinate the start of each 5-s video clip to begin 4 s after the previous MRI acquisition starts to capture the hemodynamic activity that corresponds to the middle of the movie.
Present each video 10 times, in random order, generating one scanning session that lasts 5.5 min. Repeat this functional acquisition sequence three more times.
After the four functional scans have been performed, bring the participant out of the scanner, and debrief them to conclude the study.
To define regions of interest, use the high-resolution anatomical scans of each participant and trace voxels on the surface of the temporal lobe that correspond to the early auditory cortex, also known as the planum temporale. In addition, create a mask containing voxels in the frontal lobe, which will be used as the control region.
Then, preprocess the data by performing motion correction to reduce movement artifacts and temporal filtering to remove signal drifts.
Next, divide the data into two sets: training and testing. In one data set, train a classifier—a support vector machine algorithm—making sure to keep data from the two brain regions separate for each subject.
In the other set, assess what the classifier has learned—its ability to correctly guess the identity of unlabeled data—and record the algorithm’s accuracy across runs. Perform this procedure a total of four times, leaving out one functional scan as testing data each time—a process called cross-validation.
To visualize the data, graph the averaged classifier accuracies across the four cross-validation folds for each participant.
Plot these averages for both the primary region of interest—the planum temporale—and the control area—the frontal pole—to compare focal specificity of the classifier, the extent to which a particular area, such as the auditory cortex, is selectively predicted to be involved in auditory imagination.
In this case, run a non-parametric statistic, the Wilcoxon Signed-Rank test, to test performance against chance, which is 33%. Note that the average classifier accuracy in the auditory cortex was 59%, which is significantly different from chance level.
In contrast, the mean performance in the frontal pole mask was 33%, which is not significantly different from chance.
Moreover, notice that classifier performance varied across individuals. After using a permutation test to calculate a new statistical threshold of 42%, see that 19 of 20 subjects had accuracy values significantly greater than this level using voxels from the planum temporale, whereas none had performance greater than chance using voxels from the frontal pole.
Overall, these results imply that MVPA techniques accurately predicted which of the three sounds participants were imagining based on patterns of neural activity. Such predictions were only made within the auditory cortex, suggesting that acoustic content is not represented globally throughout the brain.
Now that you are familiar with how to apply multi-voxel pattern analysis to study auditory imagery, let’s look at how neuropsychologists use multivariate techniques to advance a futuristic approach to mind-reading—the decoding of mental states—in other domains.
Classifiers have been used on fMRI data obtained from the ventral temporal cortex to predict the kinds of objects participants viewed, distinguishing between houses and faces, for example.
Taking this a step further, it’s even possible to predict whether the individual would buy that house or find the person pleasant. As creepy as that sounds, these neuromarketing implications are not far-fetched!
The same approach could also be used to detect either emotional states after watching a show—recognizing that a scary film is indeed terrifying—or even the movie genre; for instance, the frightening flick might engage the amygdala more predictably over a contemplative one that would reliably involve the prefrontal cortex.
In addition, brain-computer interfaces could convert mental states into signals that would enhance communication, in the case of individuals undergoing speech therapy, or movements, for those who’ve suffered from amputation of a limb.
You’ve just watched JoVE’s video on understanding auditory imagery using multi-voxel pattern analysis. Now you should have a good understanding of how to design and conduct the auditory imagery experiment in conjunction with functional neuroimaging, and finally how to analyze and interpret specific patterns of brain activity.
Thanks for watching!
Subscription Required. Please recommend JoVE to your librarian.
Results
The average classifier accuracy in the planum temporale across all 20 participants was 59%. According to the Wilcoxon Signed-Rank test, this is significantly different from chance level of 33%. The mean performance in the frontal pole mask was 32.5%, which is not greater than chance (Figure 2).
Figure 2. Classification performance in each participant. For three-way classification, chance performance is 33%. According to a permutation test, the alpha level of p < 0.05 corresponds to 42%.
The permutation test found that only 5% of the permutations achieved accuracy greater than 42%; thus, our statistical threshold for individual subjects is 42%. Nineteen of the 20 subjects had classifier performance significantly greater than chance using voxels from the planum temporale, while none had performance greater than chance using voxels from the frontal pole.
Thus, we are able to successfully predict from patterns of activity in auditory cortex which of the three sounds the participant was imagining. We were not able to make this prediction based on activity patterns from the frontal pole, suggesting that the information is not global throughout the brain.
Subscription Required. Please recommend JoVE to your librarian.
Applications and Summary
MVPA is a useful tool for understanding how the brain represents information. Instead of considering the time-course of each voxel separately as in a traditional activation analysis, this technique considers patterns across many voxels at once, offering increased sensitivity compared with univariate techniques. Often a multivariate analysis uncovers differences where a univariate technique is not able to. In this case, we learned something about the mechanisms of mental imagery by probing the information content in a specific area of the brain, the auditory cortex. The content-specific nature of these activation patterns would be difficult to test with univariate approaches.
There are additional benefits that come from direction of inference in this kind of analysis. In MVPA we start with patterns of brain activity and attempt to infer something about the mental state of the participant. This kind of "brain-reading" approach can lead to the development of brain-computer interfaces, and may allow new opportunities for communication with those with impaired speech or movement.
Subscription Required. Please recommend JoVE to your librarian.
References
- Kosslyn, S.M., Ganis, G. & Thompson, W.L. Neural foundations of imagery. Nat Rev Neurosci 2, 635-642 (2001).
- Haynes, J.D. & Rees, G. Decoding mental states from brain activity in humans. Nat Rev Neurosci 7, 523-534 (2006).
- Norman, K.A., Polyn, S.M., Detre, G.J. & Haxby, J.V. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn Sci 10, 424-430 (2006).
- Meyer, K., et al. Predicting visual stimuli on the basis of activity in auditory cortices. Nat Neurosci 13, 667-668 (2010).
