Physiological Correlates of Emotion Recognition

Neuropsychology

Your institution must subscribe to JoVE's Psychology collection to access this content.

Fill out the form below to receive a free trial or learn more about access:

Welcome!

Enter your email below to get your free 1 hour trial to JoVE!





By clicking "Submit", you agree to our policies.

 

Overview

Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel—University of Southern California

The autonomic nervous system (ANS) controls the activity of the body's internal organs and regulates changes in their activity depending on the current environment. The vagus nerve, which innervates many of the internal organs, is an important part of the system. When our brain senses danger, vagal tone is inhibited, leading to a set of changes in the body designed to make us more prepared to fight or flee; for example, our heart rate increases, our pupils dilate, and we breath more quickly. Conversely, when the vagal system is activated, these physiological responses are inhibited, leading to a calmer state. The vagus nerve, then, acts as a kind of "brake" on our arousal. One interesting consequence of this calmer state is that it tends to promote social interaction-when we are not tensed and afraid of our immediate environment we are instead receptive to interacting with others. Poor functioning of this regulatory mechanism, therefore, may be associated with difficulties in social behavior.

One index of autonomic regulation is heart rate variability (HRV). HRV is a measure of how much the gap between one beat and the next varies over time. High HRV means there are continual fluctuations in the heart rate over time, a reflection of successful autonomic regulation. Low HRV means there is consistency of the heart rate over time, a condition associated with poor autonomic regulation.

In this study we will test the hypothesis that increased HRV is associated with more accurate categorization of emotional stimuli.1,2 Following a study by Park et al., we will measure HRV and test its association on a task that measures skill at perceiving facial emotions.3

Cite this Video

JoVE Science Education Database. Neuropsychology. Physiological Correlates of Emotion Recognition. JoVE, Cambridge, MA, (2017).

Procedure

1. Recruit 40 participants.

  1. Participants should have normal or corrected-to-normal vision to ensure they will be able to see the stimuli properly.
  2. Participants should not consume alcohol, caffeine, or other drugs for at least 6 hrs prior to the experiment.
  3. Participants should have no history of neurological, psychiatric, or cardiac disorders.

2. Pre-experiment procedures

Figure 1
Figure 1: Electrode placement. Place the positive electrode below the heart, near the rib cage on the right side of the body. Place the negative above the heart, just below the left collarbone. Place the ground electrode below the heart, near the rib cage on the left side.

  1. Attach three electrodes to the chest to record the heart rate (Figure 1). The electrodes should be pre-gelled. Check to make sure that the gel is not dried.
    1. Fix the positive electrode below the rib cage on the right side.
    2. Fix the negative electrode below the left collarbone forming a diagonal across the heart with the positive electrode.
    3. Fix the ground electrode below the rib cage on the left side.
  2. Record a resting baseline of heart rate variability (HRV) for 5 min.
    1. The electrodes are connected to equipment that amplifies the signal and sends it to a computer for monitoring and recording.
    2. Verify the quality of the measured cardiac signal.
    3. Record the cardiac signal at sample rate of 1000 Hz.

3. Provide instructions for the participant.

  1. Tell the participant that a series of faces will appear on the screen. Their task is to decide if the face is a fearful face or not. They should press the F key on the keyboard if the face is fearful, or the J key if it is not.
  2. Instruct participants to respond as quickly and as accurately as they can.

4. Perform the facial emotion recognition task.

  1. Each trial begins with a fixation cross that remains on the screen for 500 ms.
  2. A face then appears and remains on the screen for 200 ms.
  3. Half of the faces display a fearful expression, and half display a neutral expression. The order of faces is randomized for each participant.
  4. The face is replaced by a fixation cross which remains on the screen for 2 s.
  5. Present 120 trials.

5. Analyze the data.

  1. Calculate the HRV measure from the recorded cardiac signal.
    1. Use automated software to identify the peak of each cardiac cycle known as the R wave.
    2. Visually inspect the data to ensure that each R wave is properly identified and make any adjustments that are necessary.
    3. Calculate the time from each identified R wave to the next, and record these values.
    4. Use specialized software to calculate high-frequency HRV power, which corresponds to the degree of HRV within the 0.15 to 0.4 Hz range.
  2. Analyze behavioral performance on the facial emotion recognition task.
    1. Calculate accuracy separately for fearful and neutral faces.
  3. Analyze the relationship between HRV and emotion recognition.
    1. Compute the correlation between HRV and accuracy at identifying fearful and neutral faces, separately.

Physiological measurements can be used to understand psychological functioning, revealing an interactive relationship between the body and mind.

How fast a person’s heart beats can change depend on their surroundings, and—by extension—their emotional state. For example, someone who senses a shadowy figure following them at night may demonstrate an elevated heart rate due to fear, as well as the physical exertion from running away.

In contrast, when the same individual enters a comfortable environment, such as a gathering at a friend’s house, relatively no danger—except maybe spilling wine on the white carpet—is perceived.

Interestingly, although the speed of the heart can change automatically in response to their own feelings, it can also be influenced by the emotions they distinguish in others—like terror in a man who just broke the host’s favorite lamp.

Here, the very observation of signs of dread in the face of the guilty party—wide eyes, bared teeth, and raised eyebrows—can briefly escalate the heart rate of the observer. Throughout all of these experiences, the individual displayed high heart rate variability, or HRV, representing effective regulation.

In this video, we explore how ranges in cardiovascular capacity—low to high HRV—can be used to predict an individual’s ability to recognize feelings in other people.

We demonstrate how to design and perform an experiment to analyze both heart rate and emotional recognition data.

In this experiment, participants are first outfitted with electrodes around the heart, asked to rest while baseline recordings are taken, and then shown pictures of human faces expressing different emotions, which they must identify as their cardiac signals are continuously recorded.

During the task, a small fixation cross is displayed in the center of a computer screen, followed by a picture of a single human face.

The trick is that this facial image can be of either a calm, neutral expression—for example, someone with a relaxed mouth, eyebrows, and forehead—or one of fear.

When each visual is presented, participants must indicate which feeling it depicts. The idea here is that the overall accuracy of their responses provides a measure of emotional recognition ability.

Every trial ends with the presentation of a final fixation symbol to clearly separate the dependent variables: emotional and cardiac responses across trials types that are shown equally but in random order.

While monitoring heart rate, emphasis is placed on a critical component of the cardiac cycle called the R wave, which corresponds to the large peak during every beat. Heart rate variability—the HRV—is calculated, corresponding to the time between each peak and how it varies over a period of time.

The calculations are processed through a number of mathematical operations, such as fast Fourier transformation, and result in frequency information. Within this whole power spectrum, a certain band, the high frequency range, is of interest since it reflects the state of autonomic regulation.

Specifically, high HRV means that a participant’s heart rate changes continuously, which suggests an index of successful control. In contrast, low HRV equates to consistent values and therefore an association with poor regulation.

Importantly, the HRV measures are compared to the emotion recognition results. Based on previous experience, it is expected that accuracy of identifying feelings will correlate with HRV.

In other words, a participant whose heart rate fluctuates as pictures of different emotions are shown is more likely to correctly distinguish them, suggesting that HRV is a physiological correlate of this identification process.

To begin, greet the participant when they arrive. Verify that they have normal vision, have not consumed any substances, such as caffeine or alcohol, in the last 6 hrs, which could affect their heart rate, and have no history of psychiatric or cardiac disorders.

Then, direct the participant to sit in front of a computer monitor and keyboard. Gather three pre-gelled electrodes, and check that they have not dried out, to ensure that electrical signals from the heart are properly conducted.

Now instruct the participant to affix these electrodes on their chest using an established configuration. Attach the leads, and confirm that they are properly connected to the equipment that will amplify and relay information to the recording computer.

At this point, in the software monitor the cardiac signal to ensure that it’s devoid of noise and artifacts. In addition, verify that the acquisition parameters are correct, and start collecting a resting heart rate for 5 min.

Proceed to explain that pictures of human faces will be shown on the monitor. Emphasize that the participant should quickly push 'F' on the keyboard if they interpret an image as being fearful. In contrast, if the face appears neutral and devoid of emotion, stress that the 'J' key must be used.

Also confirm that the very first fixation cross appears onscreen for 500 ms, and for each trial, the facial image for 200 ms, followed by the final fixation symbol for 2 s. Now, leave the room and continue to record their cardiac signal as they perform 120 emotion recognition trials.

With all of the heart-derived data collected, first navigate the graphical user interface and isolate the R-waves to calculate the time between these identified peaks.

Further process these measures to generate the power spectrum of frequencies, and in particular, record the values in the high-frequency range, which represents the degree of fluctuation within 0.15-0.4 Hz.

Finally, compute the emotion recognition ability of each individual. To do this, first import the data recorded from the key presses, and calculate the average percentage of fearful faces that were correctly identified. Then, repeat this process for neutral images.

To analyze the data, begin by creating a scatter plot where accuracy of fearful face identification is graphed on the y-axis, and HRV values on the x-axis. Include data from all participants in this figure. Then, create a similar plot for neutral face data.

In each, determine the line of best fit and the correlation coefficient. Importantly, the nearer the data points are to this line, the stronger the linear correlation is between emotion recognition and HRV.

Notice that data points were more tightly clustered around the best fit line in the fear recognition graph, compared to its neutral counterpart.

This suggests a positive correlation between HRV and an individual’s ability to distinguish fright; however, no correlation existed between HRV and participants’ capacity to recognize neutral faces, indicating that this relationship is emotion-specific.

Collectively, these results imply a link between the physiological phenomenon of heart rate and social behavior. Specifically, an individual with high HRV may be more likely to recognize emotions in others, and thus excel with social interactions.

Now that you understand how to explore the correlation between autonomic self-regulation and social behavior, let’s look at how researchers are investigating the variability of physiological arousal at a young age and in other cognitive and mental health contexts.

Up until now, we’ve focused on the identification of emotions in adults. However, similar work is being performed in children to investigate factors related to early cardiovascular variation that may predict later health conditions.

For example, researchers measured heart rate while subjecting toddlers to a number of behavioral conditions, including a lullaby during rest, an active jack-in-the-box, tasting sour lemon juice, and audio of an infant crying.

Such experiences resulted in different heart rate outcomes, suggesting that variability can be elicited in young children. Additional research is needed to assess the developmental trajectory of cardiovascular measures and ultimately predict future health outcomes.

In terms of the relationship between HRV and working memory, research has shown that a person with low HRV—whose heart beats at a continuous, even pace despite even frightening surroundings—is more likely to demonstrate poor working memory, and forget the address they are told.

Interestingly, repeated aerobic exercise—like running—can actually improve HRV and memory retention, emphasizing the relationship between body and mind.

Lastly, other work has looked at the whether there is an association between HRV and certain mental health disorders, such as depression.

Specifically, an individual with similarly low HRV may be more likely to exhibit depressive symptoms, like disinterest in activities or withdrawal from social settings. Thus, HRV may serve as a useful, non-invasive biomarker for this social disorder.

You’ve just watched JoVE’s video on the physiological correlates of emotion recognition. By now, you should know how to present socially relevant visual stimuli to participants, and collect and interpret both behavioral and HRV data. In addition, you should understand how scientists are applying HRV to additional contexts, such as how it relates to other cognitive processes like memory.

Thanks for watching!

Results

Performance on the facial emotion recognition task is typically very high; in our data overall accuracy was 92.5%. Participants were more accurate in identifying neutral faces (94.1%) compared with fearful faces (90.9%). Importantly, high frequency HRV power correlated significantly with accuracy in identifying fearful faces (Figure 2). Individuals with high HRV were more accurate in identifying fearful faces (r = 0.36). HRV power did not correlate with accuracy in identifying neutral faces, indicating that the association is emotion-specific.

Figure 2
Figure 2: HRV correlates with facial emotion accuracy. HRV high frequency power correlated with accuracy for fearful faces (left) but not for neutral faces (right). Please click here to view a larger version of this figure.

These data demonstrate an association between individual differences related to activity in the autonomic nervous system and skill for identifying socially relevant emotions in visual stimuli. This finding confirms the link between successful autonomic self-regulation and social behavior; individuals who are more successful in pumping the brakes on their physiological arousal appear to be better at tasks requiring emotional regulation and social interaction.

Applications and Summary

This experiment demonstrates the power of physiological data to provide insight into human cognition. The finding that measurements from the heart can be used to understand psychological functioning reminds us of the intimate connection between the brain and the body. An index of healthy cognitive control and emotion regulation, heart rate variability may serve as a relatively non-invasive biomarker for mental health. For example, low HRV is associated with anxiety disorders4 and depression,5 and also correlates with depression severity. Low HRV may also predict susceptibility to PTSD.6 This simple measure of the autonomic nervous system therefore serves as a window into the emotional health of the brain and body.

References

  1. Appelhans, B.M. & Luecken, L.J. Heart rate variability as an index of regulated emotional responding. Rev Gen Psychol 10, 229-240 (2006).
  2. Thayer, J.F. & Lane, R.D. A model of neurovisceral integration in emotion regulation and dysregulation. J Affect Disord 61, 201-216 (2000).
  3. Park, G., Van Bavel, J.J., Vasey, M.W., Egan, E.J. & Thayer, J.F. From the heart to the mind's eye: cardiac vagal tone is related to visual perception of fearful faces at high spatial frequency. Biol Psychol 90, 171-178 (2012).
  4. Chalmers, J.A., Quintana, D.S., Abbott, M.J. & Kemp, A.H. Anxiety Disorders are Associated with Reduced Heart Rate Variability: A Meta-Analysis. Front Psychiatry 5, 80 (2014).
  5. Kemp, A.H., et al. Impact of depression and antidepressant treatment on heart rate variability: a review and meta-analysis. Biol Psychiatry 67, 1067-1074 (2010).
  6. Gillie, B.L. & Thayer, J.F. Individual differences in resting heart rate variability and cognitive control in posttraumatic stress disorder. Front Psychol 5, 758 (2014).

1. Recruit 40 participants.

  1. Participants should have normal or corrected-to-normal vision to ensure they will be able to see the stimuli properly.
  2. Participants should not consume alcohol, caffeine, or other drugs for at least 6 hrs prior to the experiment.
  3. Participants should have no history of neurological, psychiatric, or cardiac disorders.

2. Pre-experiment procedures

Figure 1
Figure 1: Electrode placement. Place the positive electrode below the heart, near the rib cage on the right side of the body. Place the negative above the heart, just below the left collarbone. Place the ground electrode below the heart, near the rib cage on the left side.

  1. Attach three electrodes to the chest to record the heart rate (Figure 1). The electrodes should be pre-gelled. Check to make sure that the gel is not dried.
    1. Fix the positive electrode below the rib cage on the right side.
    2. Fix the negative electrode below the left collarbone forming a diagonal across the heart with the positive electrode.
    3. Fix the ground electrode below the rib cage on the left side.
  2. Record a resting baseline of heart rate variability (HRV) for 5 min.
    1. The electrodes are connected to equipment that amplifies the signal and sends it to a computer for monitoring and recording.
    2. Verify the quality of the measured cardiac signal.
    3. Record the cardiac signal at sample rate of 1000 Hz.

3. Provide instructions for the participant.

  1. Tell the participant that a series of faces will appear on the screen. Their task is to decide if the face is a fearful face or not. They should press the F key on the keyboard if the face is fearful, or the J key if it is not.
  2. Instruct participants to respond as quickly and as accurately as they can.

4. Perform the facial emotion recognition task.

  1. Each trial begins with a fixation cross that remains on the screen for 500 ms.
  2. A face then appears and remains on the screen for 200 ms.
  3. Half of the faces display a fearful expression, and half display a neutral expression. The order of faces is randomized for each participant.
  4. The face is replaced by a fixation cross which remains on the screen for 2 s.
  5. Present 120 trials.

5. Analyze the data.

  1. Calculate the HRV measure from the recorded cardiac signal.
    1. Use automated software to identify the peak of each cardiac cycle known as the R wave.
    2. Visually inspect the data to ensure that each R wave is properly identified and make any adjustments that are necessary.
    3. Calculate the time from each identified R wave to the next, and record these values.
    4. Use specialized software to calculate high-frequency HRV power, which corresponds to the degree of HRV within the 0.15 to 0.4 Hz range.
  2. Analyze behavioral performance on the facial emotion recognition task.
    1. Calculate accuracy separately for fearful and neutral faces.
  3. Analyze the relationship between HRV and emotion recognition.
    1. Compute the correlation between HRV and accuracy at identifying fearful and neutral faces, separately.

Physiological measurements can be used to understand psychological functioning, revealing an interactive relationship between the body and mind.

How fast a person’s heart beats can change depend on their surroundings, and—by extension—their emotional state. For example, someone who senses a shadowy figure following them at night may demonstrate an elevated heart rate due to fear, as well as the physical exertion from running away.

In contrast, when the same individual enters a comfortable environment, such as a gathering at a friend’s house, relatively no danger—except maybe spilling wine on the white carpet—is perceived.

Interestingly, although the speed of the heart can change automatically in response to their own feelings, it can also be influenced by the emotions they distinguish in others—like terror in a man who just broke the host’s favorite lamp.

Here, the very observation of signs of dread in the face of the guilty party—wide eyes, bared teeth, and raised eyebrows—can briefly escalate the heart rate of the observer. Throughout all of these experiences, the individual displayed high heart rate variability, or HRV, representing effective regulation.

In this video, we explore how ranges in cardiovascular capacity—low to high HRV—can be used to predict an individual’s ability to recognize feelings in other people.

We demonstrate how to design and perform an experiment to analyze both heart rate and emotional recognition data.

In this experiment, participants are first outfitted with electrodes around the heart, asked to rest while baseline recordings are taken, and then shown pictures of human faces expressing different emotions, which they must identify as their cardiac signals are continuously recorded.

During the task, a small fixation cross is displayed in the center of a computer screen, followed by a picture of a single human face.

The trick is that this facial image can be of either a calm, neutral expression—for example, someone with a relaxed mouth, eyebrows, and forehead—or one of fear.

When each visual is presented, participants must indicate which feeling it depicts. The idea here is that the overall accuracy of their responses provides a measure of emotional recognition ability.

Every trial ends with the presentation of a final fixation symbol to clearly separate the dependent variables: emotional and cardiac responses across trials types that are shown equally but in random order.

While monitoring heart rate, emphasis is placed on a critical component of the cardiac cycle called the R wave, which corresponds to the large peak during every beat. Heart rate variability—the HRV—is calculated, corresponding to the time between each peak and how it varies over a period of time.

The calculations are processed through a number of mathematical operations, such as fast Fourier transformation, and result in frequency information. Within this whole power spectrum, a certain band, the high frequency range, is of interest since it reflects the state of autonomic regulation.

Specifically, high HRV means that a participant’s heart rate changes continuously, which suggests an index of successful control. In contrast, low HRV equates to consistent values and therefore an association with poor regulation.

Importantly, the HRV measures are compared to the emotion recognition results. Based on previous experience, it is expected that accuracy of identifying feelings will correlate with HRV.

In other words, a participant whose heart rate fluctuates as pictures of different emotions are shown is more likely to correctly distinguish them, suggesting that HRV is a physiological correlate of this identification process.

To begin, greet the participant when they arrive. Verify that they have normal vision, have not consumed any substances, such as caffeine or alcohol, in the last 6 hrs, which could affect their heart rate, and have no history of psychiatric or cardiac disorders.

Then, direct the participant to sit in front of a computer monitor and keyboard. Gather three pre-gelled electrodes, and check that they have not dried out, to ensure that electrical signals from the heart are properly conducted.

Now instruct the participant to affix these electrodes on their chest using an established configuration. Attach the leads, and confirm that they are properly connected to the equipment that will amplify and relay information to the recording computer.

At this point, in the software monitor the cardiac signal to ensure that it’s devoid of noise and artifacts. In addition, verify that the acquisition parameters are correct, and start collecting a resting heart rate for 5 min.

Proceed to explain that pictures of human faces will be shown on the monitor. Emphasize that the participant should quickly push 'F' on the keyboard if they interpret an image as being fearful. In contrast, if the face appears neutral and devoid of emotion, stress that the 'J' key must be used.

Also confirm that the very first fixation cross appears onscreen for 500 ms, and for each trial, the facial image for 200 ms, followed by the final fixation symbol for 2 s. Now, leave the room and continue to record their cardiac signal as they perform 120 emotion recognition trials.

With all of the heart-derived data collected, first navigate the graphical user interface and isolate the R-waves to calculate the time between these identified peaks.

Further process these measures to generate the power spectrum of frequencies, and in particular, record the values in the high-frequency range, which represents the degree of fluctuation within 0.15-0.4 Hz.

Finally, compute the emotion recognition ability of each individual. To do this, first import the data recorded from the key presses, and calculate the average percentage of fearful faces that were correctly identified. Then, repeat this process for neutral images.

To analyze the data, begin by creating a scatter plot where accuracy of fearful face identification is graphed on the y-axis, and HRV values on the x-axis. Include data from all participants in this figure. Then, create a similar plot for neutral face data.

In each, determine the line of best fit and the correlation coefficient. Importantly, the nearer the data points are to this line, the stronger the linear correlation is between emotion recognition and HRV.

Notice that data points were more tightly clustered around the best fit line in the fear recognition graph, compared to its neutral counterpart.

This suggests a positive correlation between HRV and an individual’s ability to distinguish fright; however, no correlation existed between HRV and participants’ capacity to recognize neutral faces, indicating that this relationship is emotion-specific.

Collectively, these results imply a link between the physiological phenomenon of heart rate and social behavior. Specifically, an individual with high HRV may be more likely to recognize emotions in others, and thus excel with social interactions.

Now that you understand how to explore the correlation between autonomic self-regulation and social behavior, let’s look at how researchers are investigating the variability of physiological arousal at a young age and in other cognitive and mental health contexts.

Up until now, we’ve focused on the identification of emotions in adults. However, similar work is being performed in children to investigate factors related to early cardiovascular variation that may predict later health conditions.

For example, researchers measured heart rate while subjecting toddlers to a number of behavioral conditions, including a lullaby during rest, an active jack-in-the-box, tasting sour lemon juice, and audio of an infant crying.

Such experiences resulted in different heart rate outcomes, suggesting that variability can be elicited in young children. Additional research is needed to assess the developmental trajectory of cardiovascular measures and ultimately predict future health outcomes.

In terms of the relationship between HRV and working memory, research has shown that a person with low HRV—whose heart beats at a continuous, even pace despite even frightening surroundings—is more likely to demonstrate poor working memory, and forget the address they are told.

Interestingly, repeated aerobic exercise—like running—can actually improve HRV and memory retention, emphasizing the relationship between body and mind.

Lastly, other work has looked at the whether there is an association between HRV and certain mental health disorders, such as depression.

Specifically, an individual with similarly low HRV may be more likely to exhibit depressive symptoms, like disinterest in activities or withdrawal from social settings. Thus, HRV may serve as a useful, non-invasive biomarker for this social disorder.

You’ve just watched JoVE’s video on the physiological correlates of emotion recognition. By now, you should know how to present socially relevant visual stimuli to participants, and collect and interpret both behavioral and HRV data. In addition, you should understand how scientists are applying HRV to additional contexts, such as how it relates to other cognitive processes like memory.

Thanks for watching!

A subscription to JoVE is required to view this article.
You will only be able to see the first 20 seconds.

RECOMMEND JoVE