Conducting Hyperscanning Experiments with Functional Near-Infrared Spectroscopy

Summary

The present protocol describes how to conduct fNIRS hyperscanning experiments and analyze brain-to-brain synchrony. Further, we discuss challenges and possible solutions.

Abstract

Concurrent brain recordings of two or more interacting persons, an approach termed hyperscanning, are gaining increasing importance for our understanding of the neurobiological underpinnings of social interactions, and possibly interpersonal relationships. Functional near-infrared spectroscopy (fNIRS) is well suited for conducting hyperscanning experiments because it measures local hemodynamic effects with a high sampling rate and, importantly, it can be applied in natural settings, not requiring strict motion restrictions. In this article, we present a protocol for conducting fNIRS hyperscanning experiments with parent-child dyads and for analyzing brain-to-brain synchrony. Furthermore, we discuss critical issues and future directions, regarding the experimental design, spatial registration of the fNIRS channels, physiological influences and data analysis methods. The described protocol is not specific to parent-child dyads, but can be applied to a variety of different dyadic constellations, such as adult strangers, romantic partners or siblings. To conclude, fNIRS hyperscanning has the potential to yield new insights into the dynamics of the ongoing social interaction, which possibly go beyond what can be studied by examining the activities of individual brains.

Introduction

In recent years, neuroscientists have started to study social interactions by recording the brain activities of two or more persons simultaneously, an approach termed hyperscanning¹. This technique opens new opportunities to elucidate the neurobiological mechanisms underlying these interactions. To fully understand social interactions, it may not be sufficient to study single brains in isolation but rather the joint activities of brains of interacting persons². Using different neuroimaging techniques, hyperscanning studies have shown that brain activities of interacting persons or groups synchronize, e.g., while they coordinate their actions³, make music⁴, communicate⁵, engage in classroom activities⁶ or cooperate⁷.

The article presents a protocol for conducting simultaneous recordings with functional near-infrared spectroscopy (fNIRS). Similar to functional magnetic resonance imaging (fMRI), fNIRS measures the hemodynamic response to brain activation. Changes in oxygenated and deoxygenated hemoglobin (oxy-Hb and deoxy-Hb) are calculated based on the amount of diffusively transmitted near-infrared light through tissue⁸. fNIRS is well suited for conducting hyperscanning experiments, especially with children, because it can be applied in less constrained and more natural settings than fMRI. Moreover, it is less prone to movement artifacts than both, fMRI and EEG⁹. In addition, fNIRS data can be acquired at high sampling frequencies (e.g., 10 Hz), thus it highly oversamples the relatively slow hemodynamic response and thereby potentially provides a more complete temporal picture of the brain hemodynamics¹⁰.

This protocol was developed within the study of Reindl et al.¹¹ and has been slightly modified (in particular with respect to the channel placement and bad channel identification) more recently. The aim of the study was to examine synchronized brain activity of parent-child dyads. Using fNIRS hyperscanning, we assessed brain-to-brain synchrony in prefrontal brain areas of children (aged five to nine years) and their parents, mostly mothers, during a cooperative and a competitive computer task. Prefrontal brain regions were targeted as they had been identified as important regions for social interactive processes in previous hyperscanning studies¹. The cooperative and competitive task were originally developed by Cui et al.¹² and recently employed by several previous studies^13,14,15. For the study of Reindl et al.¹¹, the tasks were modified to be suitable for children. Participants were instructed to either respond jointly via button presses in response to a target (cooperation) or to respond faster than the other player (competition). Each child performed each task once with the parent and once with an adult stranger of the same sex as the parent. Within each child-adult dyad, wavelet coherence was calculated for the oxy-Hb signals of corresponding channels as a measure of brain-to-brain synchrony.

This protocol describes the procedures to collect fNIRS hyperscanning data of parent and child during the cooperative and competitive game. The overall procedure, however, is not specific to this research design but is appropriate for different populations (e.g., adult strangers, romantic partners, siblings, etc.) and can be adapted for a number of different experimental tasks. This protocol also outlines one possible analytical procedure, which covers necessary and optional data analysis steps, including fNIRS data preprocessing, bad channel detection, wavelet coherence analysis and validation by random pair analysis.

Protocol

Prior to participation, all parents / children provided informed consent / assent. The study was approved by the ethics committee of the Medical Faculty of RWTH Aachen University.

1. Preparation before the Participant Arrives

1. Prepare NIRS caps.
   1. Choose the cap sizes the same size or slightly larger than the participant’s head circumference.
   2. Cut 15 holes with a diameter of approximately 15 mm each, arranged in a horizontal 3x5 grid, into the forehead area of each of 2 raw EEG caps (see Table of Materials). Make sure that the holes are spaced 30 mm from each other in any direction, that the middle column of holes is located in the center of the forehead, i.e., above the nose, and that the bottom row is located above the eyebrows.
   3. In order to make the caps more comfortable and minimize pressure marks, attach soft foam material (e.g., adhesive window sealing tape or similar flat foam rubber material) at the inner side of the holder grid between the probe sockets and at the edges. Use double-faced adhesive tape or sewing thread if necessary.
   4. Mount an empty 3x5 probe holder grid (see Table of Materials) to each of the modified EEG caps such that the holder grid itself is placed on the inside of the cap and the holder sockets stick in the holes.
      NOTE: The NIRS measurement system (see Table of Materials) has two separate probe sets, use one probe set for each participant.
   5. Gently insert the probes into the appropriate holder sockets on the grids such that only the first ridge of each probe is mounted in the socket, which results in one clicking sound.
   6. Open the probe set monitor window at the NIRS measurement system and select 2 probe sets arranged in a 3x5 grid each, one for the participating child and one for the adult. Ensure that the probe arrangements of the two caps corresponds to the arrangements in the probe set window (i.e., same location of the respective emitter and receiver probe numbers).
2. Prepare the experiment.
   1. Start the NIRS measurement system with laser diodes switched on 30 min. before measuring, such that the system reaches a stable operating temperature.
   2. Set all necessary options at the NIRS measurement system. Make sure that the device is set to event-related measurement and that the RS232 serial input, necessary for receiving triggers from the experimental paradigm, is active.
      Note: The experiment is an adapted version by a paradigm devised by Cui et al.¹², programmed in the non-commercial Psychophysics Toolbox extensions, version 3.0.11¹⁶.
   3. Prepare the experimental paradigm by starting the technical computing software (see Table of Materials) that serves as base for the Psychophysics Toolbox extensions and setting the current directory to the folder that the paradigm is saved in.
   4. Place two chin rests in front of the computer screen to prevent head movements during the experiment.

2. Participant Arrival in the Laboratory

1. Prepare the participants.
   1. Show and explain the experimental setup including the NIRS measurement system to the participants. Always make sure that the participants do not look directly into the laser beam of the NIRS measurement system as this may be harmful to the eye.
   2. Seat the participants next to each other in front of the computer screen. Adjust the height of the chin rests such that both participants sit comfortably.
   3. Instruct the participants and administer practice trials of both the cooperative and the competitive game. Give additional instructions during the practice trials if necessary.
   4. Measure and mark the Fpz point according to the 10-20 system, which is 10% of the distance between nasion and inion, on each participant’s head.
   5. Place the caps with the probes carefully on the participants’ heads, with the laser turned off. Place the front of the cap, including the probe grid, on the participant’s forehead first and then pull down the back of the cap towards the neck. Make sure that the middle probe of the bottom row is placed on Fpz and the middle probe column is aligned along the sagittal reference curve.
   6. Place the fiber strings on the holder arm attached to the NIRS measurement system so that they hang loosely without contact with the participant or chair and that they do not pull on the caps. Use an additional holder (e.g., modified microphone stand or similar) for the second participant if necessary.
   7. Push each probe further into its socket until the small white nose in the center of the top of the probe casing is visible.
      Note: The nose is pushed upwards by a coil spring mechanism as soon as the probe tip touches the participant’s scalp.
   8. Turn the laser on again and test the signal quality by clicking on the Auto Gain button in the probe set monitor window of the NIRS measurement system.
   9. If a channel does not have a sufficient signal (i.e., if it is marked in yellow), gently put the hair underneath the surrounding probe tip aside. If necessary, push the probes further into their sockets but ensure the comfort of the participant. Check whether the signal quality has improved (i.e., the channel is now marked in green) by clicking on the Auto Gain button again.
   10. If step 2.1.9. does not lead to a signal improvement, adjust the signal intensity. If there is too much signal (i.e., if the channel is marked in red), change the signal intensity to low signal intensity by repeatedly clicking on the respective probe’s symbol in the probe set monitor window of the NIRS measurement system. If there is not enough signal (i.e., if the channel is marked in yellow), change the signal intensity to high signal intensity, again by repeatedly clicking on the respective probe’s symbol.
2. Run the experiment
   1. When there are no questions after the practice trials and a good signal quality is ensured, start the experimental paradigm.
   2. Place a towel over the participants’ hands so that they cannot see the hand movements of their respective game partner.
   3. After the experiment, save the data and export the raw light intensity data as a text file by clicking on the Text File Out button. Do not apply any filters in the NIRS measurement system.
   4. Clean all necessary materials (probes, probe holders, chin rests) with ethanol. Wash the caps in a gentle cycle with mild detergent.

3. Data Analysis

1. Data Preprocessing
   Note: There are several non-commercial software packages available for fNIRS data analysis, e.g., HomER¹⁷, NIRS Brain AnalyzIR¹⁸ or SPM for fNIRS¹⁹. The latter was used for the following preprocessing steps. For more information on how to perform these steps, please see the toolbox manual.
   1. Convert the data files to the SPM for fNIRS data format.
   2. Calculate oxy-Hb and deoxy-Hb concentration changes using the modified Beer-Lambert law by pressing the Convert button in the main window. Enter the age of the subject and the distance between source and detector (e.g., 3 cm). Accept the default values for the molar absorption coefficients of oxy-Hb and deoxy-Hb at wavelength (λ) 1 and λ 2 as well as the default values for the differential pathlength factor (DPF) at λ 1 and λ 2.
   3. Preprocess the time series of hemodynamic changes to reduce motion artifacts by selecting the MARA button (for more information on the MARA algorithm see Scholkman et al.²⁰).
   4. Preprocess the time series to reduce slow drifts by selecting the DCT button.
2. Bad channel detection
   Note: Bad channel detection can be performed before and / or after fNIRS data preprocessing. In this protocol, different objective criteria for detecting bad channels and visual inspection are combined. Please note that the proposed list of objective criteria is not exhaustive. For bad channel detection, self-written scripts were used (for the technical computing software see Table of Materials).
   1. Exclude channels in which there is no signal change for several continuous samples, which is indicated by a flat line when plotting the time series.
   2. Calculate the coefficient of variation CV = SD/mean*100 for the raw attenuation data. Exclude channels in which the CV is above a predefined percentage (e.g., 10%; see for instance van der Kant et al.²¹).
   3. Plot the power spectrum of the signal. If there is no heartbeat visible in the signal spectrum around 1 Hz, as indicated by an increased power in this frequency band, exclude the channel from the analysis.
   4. Visually inspect all data before and/or after preprocessing. Decide whether to include the channel based on the objective criteria, described in 3.2.1 – 3.2.3, as well as on subjective visual detection of noisy channels.
3. Brain-to-brain connectivity
   Note: Two different estimate types of brain connectivity can be distinguished: non-directed estimates, which quantify the strength of the connectivity, and directed estimates, which seek to establish statistical evidence for causation from the data²². Here the focus was on the wavelet transform coherence (WTC), a widely applied non-directed estimate for fNIRS brain-to-brain connectivity. Several non-commercial software solutions for the computation of the WTC are available, e.g., one by Grinsted and colleagues²³ or the ASToolbox²⁴, which was used in this protocol for the following steps.
   1. In the AWCO function of the ASToolbox, specify the mother wavelet (e.g., Generalized Morse Wavelet with its parameters beta and gamma), which is used to transform each time series into the time and frequency domain by the continuous wavelet transformation.
   2. Specify the smoothing window type (e.g., Hanning window) and the smoothing window size for the time and scale domain in the AWCO function.
   3. To examine the significance of the WTC coefficients and to calculate their p-values, specify the number of surrogate time series (n ≥ 300) and the ARMA model (e.g., AR (1)) in the AWCO function.
   4. With the parameters specified in steps 3.3.1 to 3.3.3, calculate the wavelet coherence of two corresponding channels (the same channel in two participants).
   5. Choose a frequency band of interest in which the task-related brain-to-brain synchrony is expected to occur based on previous studies and visual inspection of the data (for an alternative approach see Nozawa et al. ²⁵).
   6. Calculate the mean of the WTC coefficients and / or the percentage of significant WTC coefficients in the task-related frequency band for each task block in each channel and for each dyad. Use this value as an outcome measure of brain-to-brain synchrony for further statistical analysis (for more information see Reindl et al.¹¹).
4. Comparison to Random Pairs
   Note: To validate the results, we recommend comparing the WTC of the actual dyads to the WTC of random adult-child pairings, who did not play with each other but performed the same experimental task.
   1. Calculate the WTC, as described in 3.3., for participant pairs who did not play together but performed the same experimental task (i.e., random pairs). Choose the number of random pairs (e.g., 300 for each condition) and calculate the WTC for each random pair.
   2. Compare the coherence of the random and actual pairs to avoid the detection of spurious synchronicity.

Representative Results

Representative data of one parent-child dyad during the cooperative condition are shown in Figure 1. The cooperative task consists of three 30 s rest blocks and two task blocks, with 20 trials each, presented in alternating order. In each trial, participants have to react as simultaneously as possible to a signal to earn a point¹¹.

Figure 1: Hyperscanning data analysis and representative results. Light intensity data is collected in 22 channels (CHs) of two participants. First, bad channels are detected and excluded from further analyses. Afterwards, light intensity data is converted to changes in oxy-hemoglobin (Δ Oxy-Hb) and deoxy-hemoglobin (Δ Deoxy-Hb). Signals are shown for one exemplary parent-child dyad in CH 8 during the cooperative condition. Data is preprocessed by reducing motion artifacts and slow drifts. Afterwards, the wavelet coherence is calculated from the preprocessed oxy-Hb signals. To estimate the significance of each wavelet coherence value, 300 surrogate time series are generated. If the observed wavelet coherence value is higher than 95% of the wavelet coherence values obtained from the surrogate time series at the same point in time and scale, it is regarded as significant. Significant wavelet coherence values are marked by solid lines surrounding the respective areas in the plot. Coherence in the task-related frequency band is depicted within the black box. Please note that the analysis steps and the parameterization depicted in the figure should be understood as an example. The optimal parameterization depends on the data, e.g., different parameters of the MARA algorithm might work best for different types of artifacts²⁰, and there is no gold standard for any of the analysis steps yet. Please click here to view a larger version of this figure.

The results are exemplified for the fNIRS data of channel 8 of both participants of a parent-child dyad. Before preprocessing, raw light attenuation data, received from the fNIRS device, are converted to changes in oxy-Hb and deoxy-Hb for both participants. Next, fNIRS time series are preprocessed to reduce motion artifacts and drifts. Finally, the significant WTC is calculated from the preprocessed oxy-Hb signals of both participants.

Figure 1 illustrates a real valued WTC matrix, which is composed of the coherence coefficients in time and frequency domain (here in period length). The coefficients can range between 0 and 1, with 1 indicating a perfect relationship at a specific time and frequency between both signals²⁴. The coefficients are visualized using a color map ranging from blue (little or no coherence) to red (strong or maximum coherence). Significant coherence values are marked by solid black lines surrounding the respective areas in the plot. The beginning and end of each task block are indicated by vertical dashed lines.

Results show a strong coherence throughout the experiment in a high frequency band, until a period length of ~ 1 s (1 Hz). This likely results from the cardiac rhythms of parent and child. Additionally, results show a strong coherence in a lower frequency band between ~ 2 s and 8 s period length (0.5 - 0.125 Hz). Trial lengths differed due to pseudo-randomized variable cue durations (600 - 1500 ms) and participants' individual reaction times but were around 7 s on average, assuming reaction times of about 1 s. Therefore, coherence in this low frequency range likely reflects a synchronization of brain activities of both subjects during the task.

Discussion

In this protocol, we show how to conduct fNIRS hyperscanning experiments and one possible way to analyze brain-to-brain synchrony, measuring concentration changes of oxy-Hb and deoxy-Hb at frontal brain regions of two subjects simultaneously. FNIRS hyperscanning is relatively easy to apply: a single NIRS device is sufficient to measure brain activities of both subjects by splitting the optodes between them. Thus, no synchronization between different devices is necessary¹. Moreover, since fNIRS does not require strict motion restriction, it is well suited for conducting hyperscanning experiments in a natural environment and in children. In the following, we highlight some critical issues when designing, analyzing and interpreting (fNIRS) hyperscanning experiments, discuss challenges as well as possible solutions.

Experimental Design. One important issue of hyperscanning studies concerns the experimental design. Two participants who complete the same experimental task independently of each other might show similar brain activities, which then might be detected as brain-to-brain synchrony²⁶. To differentiate between brain-to-brain synchrony induced by the experimental task and by the social interaction, appropriate experimental control conditions are necessary. On the one hand, the cooperative and competitive tasks are very well suited because they differ only in the cooperative task component and not in the stimulus material and the participant's motor behavior. On the other hand, less standardized and more natural interactions (e.g., making a puzzle together) might induce more variance in social interactive behavior and might have a greater ecological validity.

Spatial registration of channels. One challenge in fNIRS hyperscanning is measuring hemodynamic activity in corresponding channels. Attaching emitter and detector probes at corresponding locations of two participants' heads does not warrant that activity in two corresponding cortical regions is tapped, as individual brain anatomy is liable to differ across participants. Simultaneously measuring an adult and a child exacerbates this problem by introducing developmental differences on top of anatomical ones. Moreover, with an increasing number of channels, the placement of the channels is less reproducible across subjects because of variability in head shape and size²⁷. One optional accessory to the ETG-4000 is a probe positioning unit which creates probe positions relative to fiducial points on the head in three-dimensional space. These data can then be co-registered to the structural MR image of the participant's brain²⁷. Acquiring MR images and using the positioning unit will enable the experimenter to better control whether activity is actually measured in corresponding brain regions across two participants. Additionally, researchers could partly circumvent this problem by calculating an all-to-all connectivity model, measuring the connection between any two channels of the two participants.

Influence of the systemic physiology. Another important issue is that hemodynamic changes are known to be influenced not only by the effect of the neurovascular coupling, thus neuronal activity, but also by systemic changes, such as changes in heart rate, blood pressure, breathing rate, and autonomic nervous system activity²⁸. Therefore, any synchrony detected in the hemodynamic changes of two cooperating participants may also be attributable to a synchrony of those factors. Previous studies have shown that two interacting partners do indeed synchronize their physiological activities²⁹. Note, however, that in tasks with different experimental conditions which are directly compared to each other, this is only a confounder if physiological coupling is more prominent in one but not the other condition. Nevertheless, it can be helpful to acquire physiological data in hyperscanning studies to enable experimental control of these parameters. Another option, as demonstrated recently by Nozawa et al.²⁵, is to add measurement channels with a short source-detector (S-D) separation (e.g., 1 cm), which are sensitive to the superficial skin blood-flow signal. The corresponding component can then be removed from the fNIRS signal obtained from measurement channels with a regular S-D separation (e.g., 3 cm), thus reducing the influence of physiological confounders. Such a dual or multi-distance approach has been shown to improve the sensitivity to task-enhanced (here: communication-enhanced) brain-to-brain synchrony.

Data Analysis. Hyperscanning results depend on an estimator to quantify brain-to-brain synchrony. In the current study, we calculated the WTC of oxy-Hb signals of corresponding channels as a measure of brain-to-brain synchrony. Wavelet-based methods have the advantage that they consider the oscillatory dynamics of time series in the time-frequency space. The WTC is a non-directed measure calculated from wavelet transformed time series, representing the strength of the relationship between two time series. In future studies, it would be interesting to additionally include directed measures, such as Granger causality, in order to examine which participant "leads" the activity (see for instance Pan et al.¹⁵). Furthermore, while many previous fNIRS-based hyperscanning studies examine brain-to-brain synchrony in only one signal (e.g., oxy-Hb), it is advisable to consider both oxy-Hb and deoxy-Hb (and possibly total-Hb) in order to take full advantage of the fNIRS technique¹⁵.

Limitations. Although fNIRS offers a promising, rapidly growing neuroimaging technique, some technical limitations associated with the device need to be considered when planning such a study (for a recent review see Pinti et al.³⁰). In comparison to EEG and fMRI, fNIRS is more resistant to motion artifacts, yet, it still requires sufficient motion artifact control and detection. There are several potential causes of artifacts. First, some participants tend to move their head abruptly, in particular infants and children, and thereby might pull on the fiber tracts, affecting the optode contact. Developments of new fiberless devices are more robust to movement and thereby allow investigations of active tasks³⁰. The use of a chin-rest can serve as an additional motion artifact control; however, it limits the ability to record brain activities in natural interactions. Second, acquiring an adequate optode contact can be hindered by dark, curly and/ or thick hair of the participant. Placing the optodes can thus be time-consuming and a perfect signal is not always guaranteed. Third, depending on the fNIRS system, wearing optodes for a longer period of time can put pressure on the participant's head, which can be experienced as unpleasant. This does not only limit the recording time of the experiment but might also lead to more movement and artifacts (e.g., smaller children might pull on the cap). In addition to motion artifacts, it is noteworthy that fNIRS provides measures of the cortical surface only. Finally, there are no standardized data analysis guidelines yet. Several toolboxes were developed over the past years and first attempts were made to analyze the effectiveness of various preprocessing techniques (e.g., Brigadoi et al.³¹ and Cooper et al.³²). Moreover, the analytical protocol presented in this article shows one way to analyze fNIRS hyperscanning data. Importantly, the selected parameters of the analysis should be understood as one possible option and not as a standard guideline. Several other analytical protocols for fNIRS hyperscanning have been developed in the last years by different research groups (see for instance Cui et al.¹²; Hirsch et al.³³).

Conclusion. fNIRS hyperscanning is a promising technique to gain further insights into the neurobiological underpinnings of social interactions³⁴. In the future, portable and fiberless NIRS devices may be particularly important when examining brain-to-brain synchrony in natural interactions and moving from the dyad towards larger groups of subjects. Finally, combining different neuroimaging techniques, e.g., EEG-fNIRS, may provide new insights, broadening our understanding of brain-to-brain synchrony.

Materials

Name	Company	Catalog Number	Comments
NIRS measurement system with probe sets and probe holder grids	Hitachi Medical Corporation, Tokyo, Japan	ETG-4000 Optical Topography System	The current study protocol requires an optional second adult probe set for 52 channels of measurement in total as well as two 3x5 probe holder grids.
raw EEG caps	EASYCAP GmbH, Herrsching, Germany	C-SCMS-56; C-SCMS-58	Caps must be provided with holes for NIRS probes by the experimenter. Choose cap size the same size or slightly larger than participant's head circumference.
Technical computing software	The MathWorks, Inc., Natick, MA	MATLAB R2014a (or later versions)	Serves as base for Psychophysics Toolbox extensions (stimulus presentation), SPM for fNIRS toolbox  (fNIRS data analysis), and ASToolbox (WTC computation).