Brain Imaging Investigation of the Neural Correlates of Observing Virtual Social Interactions

Summary

This article demonstrates an experimental design in which whole-body animated characters are used in conjunction with functional magnetic resonance imaging (fMRI) to investigate the neural correlates of observing virtual social interactions.

Abstract

The ability to gauge social interactions is crucial in the assessment of others’ intentions. Factors such as facial expressions and body language affect our decisions in personal and professional life alike ¹. These "friend or foe" judgements are often based on first impressions, which in turn may affect our decisions to "approach or avoid". Previous studies investigating the neural correlates of social cognition tended to use static facial stimuli ². Here, we illustrate an experimental design in which whole-body animated characters were used in conjunction with functional magnetic resonance imaging (fMRI) recordings. Fifteen participants were presented with short movie-clips of guest-host interactions in a business setting, while fMRI data were recorded; at the end of each movie, participants also provided ratings of the host behaviour. This design mimics more closely real-life situations, and hence may contribute to better understanding of the neural mechanisms of social interactions in healthy behaviour, and to gaining insight into possible causes of deficits in social behaviour in such clinical conditions as social anxiety and autism ³.

Protocol

1. Stimuli, Task Design, and Experimental Protocol

Our stimuli are created using Poser 7.0 (http://poser.smithmicro.com/poser.html), and they are presented using CIGAL (http://www.nitrc.org/projects/cigal/).

1. The task consists of a series of ten-second animated videos of non-verbal guest-host interactions in a business setting. The subject views the guest being greeted by a host (social interaction condition) or a cardboard cut-out of a host (no social interaction/control condition).
2. The host may display behaviours that are inviting to further social interaction (approach condition), or behaviours that may indicate lack of interest in further interaction (avoid condition).
3. In addition to this basic manipulation of the type of behaviour displayed by the host, other manipulations may also be included. For instance, in some of the trials (typically half) the characters may shake hands, as part of the greeting protocol, whereas in others they do nothing. This manipulation allows investigation of the effect of formal physical touch on behaviour⁴, which may also be expected to have different significance depending on the cultural background of the viewer (e.g., Western vs. East-Asian).
4. Other manipulation may involve the change of the viewer’s perspective from personal (Me) to impersonal (Other). This manipulation aims at tapping into neural networks associated with self-referential processing⁵, and allows identification of responses that are modulated by personal engagement^6,7(e.g., exacerbation of the impact of avoid behaviour, if taken personally). The alternation between these two perspectives can be cued at the beginning of each trial.
5. Videos are followed by rating screens that are asking the subject to rate the host on competence, trustworthiness, and interest in doing business on a 5-point Likert scale (0 = not at all / 4 = very much); these ratings should be counterbalanced across trials, to avoid order effects.
6. Characters in videos are counterbalanced for displayed behaviour (approach vs. avoid), ethnic background (Caucasian vs. Non-Caucasian), shirt colour, and hairstyle. Basic aspects that may influence social interactions (e.g., attractiveness) should also be controlled for (e.g., by off-line ratings of the hosts), to ensure that there are no systematic differences between trial categories, and thus avoid possible confounds (e.g., if hosts displaying avoid behaviour also have overall lower attractiveness scores). Finally, given the evidence that interactions with female-hosts are more effective in influencing decisions (e.g., financial) ⁴, in our current study all hosts were females, but the gender of the hosts can also be counterbalanced across trials – e.g., 50% females vs. 50% males.
7. The experiment is split into runs/blocks of trials, to allow participants time to rest, and avoid massive data loss in case of equipment malfunction. Also, to avoid biased data loss, ideally the conditions/trials should be equally represented in each block, according to each manipulation (e.g., social interaction, gender, shake/no-shake). Run order is also counterbalanced between participants. Each run begins with six seconds of a fixation, to allow stabilisation of the MR signal. An inter-trial interval of 8 seconds follows each movie-trial and ends each run/block.
8. It should be noted that, while compared to experimental designs involving static stimuli the present design has increased ecological validity in identifying neural correlates of social cognition, this approach also poses limitations and challenges. One such challenge is posed by the need to maintain the "background" aspects (e.g., the environment where social interactions occur or the non-relevant aspects of behaviour) as constant as possible, in the context of manipulating the behavioural variables of interest (e.g., gestures distinguishing the targeted social interactions), which may be difficult to accomplish in conditions where dynamic stimuli are used. Nevertheless, we believe that the present approach reaches a reasonable trade-off between the goal of increasing ecological validity of stimuli used in social neuroscience studies and the constraints associated with the involvement of brain imaging tools that require reasonable control of the experimental manipulations to allow valid inferences about the neural correlates of targeted behaviours.

2. Preparing the Subject for the Scan

Subjects are typically recruited on the basis of their age, health, first language, and individual risk factors for MRI scanning, such as metallic joint replacements. However, depending of the purpose of the study, other factors may also be considered, including race/ethnic background, socio-economic status, and history of drug use. All subjects provide written informed consent prior to running the experimental protocol, which is approved by an Ethics Board.

Prior to Entering the scanning room

1. On the day of scanning, participants are asked to complete questionnaires that assess their current state of mind, to ensure that they are not overly anxious or depressed before the experiment ^{8, 9}. These assessments can also be used in conjunction with fMRI data, to investigate how the emotional state at the time of scanning may influence the participants’ responses. Also, in conjunction with post-scanning assessments, these initial evaluations can be used to make sure that the participants’ overall emotional state does not change dramatically as a result of their participation in the experiment. Finally, other behavioural/personality assessments can also be made, to further investigate how individual variations in particular aspects of personality that influence responses in social contexts (e.g., social anxiety, trustworthiness) may influence participants’ responses during the study ^{10, 11}.
2. Prior to the scan, the participant is informed in detail of the scanning procedures, and is given specific instructions for the behavioural task (described below). To avoid discomfort and increased familiarity with the task, the participant is also given an abbreviated practice run for the task.

Entering the scanning room

3. The subject is instructed to lie supine on the scanning bed, with additional cushioning for the head, to ensure comfort during the scan and to minimize movement. To further minimize head movement, the non-adhesive side of a length of tape may be wrapped lightly around the subject’s forehead. If preferred, cushioning may also be placed under the raised knees of the subject, to reduce lower back muscular tension.
4. Subjects are given ear protection (ear plugs) as well as isolation headphones to communicate with the experimenter during the MRI scan.
5. The subject’s right hand is positioned comfortably on the response box, thus allowing the left hand to be used for support or for other measurements (e.g., skin conductance responses). An emergency stop button should also be placed nearby, so that the subject may indicate any urgent need to stop the scanner.
6. Before starting data collection, it is critical to make sure that the subject can see the screen projection clearly, for stimulus presentation, and that the response buttons work properly.

3. Data Recording and Processing

Scanning Parameters

We collected MRI data using a 1.5 Tesla Siemens Sonata scanner for MRI recordings. Our anatomical images were 3D MPRAGE anatomical series (repetition time, TR = 1600 ms; echo time TE = 3.82 ms; number of slices = 112; voxel size = 1 x 1 x 1mm), and functional images consisted of series of 28 functional slices (voxel size = 4x4x4 mm), acquired axially using an echoplanar sequence (TR = 2000 ms; TE = 40 ms; field of view FOV = 256 x 256mm), thus allowing for full-brain coverage.

Data Analysis

We use Statistical Parametric Mapping (SPM2/SPM5) in combination with in-house Matlab-based tools. Pre-processing involves typical steps: quality assurance, TR alignment, motion correction, co-registration, normalization, and smoothing (8 mm³ kernel) ¹².

1. Data analysis combines voxel-wise and region of interest (ROI) approaches to compare brain activity associated with the conditions of interest (e.g., social interaction vs. non-social interaction).
2. Whole-brain voxel-wise analyses produce statistical maps that identify larger networks of brain regions associated with processing that underlies the assessment of social interactions, and ROI analyses allows targeted investigation of the response in specific brain-regions, which are a priori identified as being part of the social cognition network (Figure 1A). ROI analyses are also used to extract the fMRI signal for illustration purposes (Figure 1B). We used an intensity threshold of p=0.001 and an extent threshold of 10 contiguous voxels. In addition to these typical methods involved in data analysis, other ways of analyzing data may also be employed as complementary approaches in studies investigating the neural correlates of social cognition ^{13, 14}.
3. Finally, correlations of brain imaging data with behavioural data (e.g., trustworthiness ratings) and/or scores indexing personality measures (e.g., trait anxiety) can also be performed, to investigate how brain activity co-varies with individual differences in behaviour and personality.

4. Representative Results

Figure 1. Increased activity in the social cognition network in response to observing social interactions. Comparison of social interaction vs. no-interaction/control trials revealed activity in typical social cognition brain regions, including the superior temporal sulcus (STS, a), the lateral and medial prefrontal cortex (mPFC, b & d, respectively), and the amygdala (AMY, c). The "activation maps" are superimposed on high resolution brain images displayed in lateral (left- and right-side panels) and coronal (middle panel) views; the color bars indicate the gradient of t values of the activation maps (based on data from 15 participants), reflecting brain activity time-locked to the onsets of approach/avoid behaviours. The line graphs illustrate the time courses of the fMRI signal, extracted from functional ROIs for each trial type and TR (1 TR = 2 seconds). L = Left; R = Right.

Discussion

The experimental design introduced here allows investigation of the neural correlates of observing and interpreting body language. This design has the potential to advance our knowledge concerning brain mechanisms involved in social interactions, and to extend theoretical models of how we combine perception of different types of body language or social concepts such as trustworthiness to make decisions in interactive social environments ³. Such knowledge can be applied in a variety of personal and business settings, and can improve our understanding of clinical deficits in social interaction. The success of this design depends on proper task manipulation, involvement of ecologically valid stimuli, and careful data collection