The Emotional Stroop Task: Assessing Cognitive Performance under Exposure to Emotional Content

Summary

The emotional Stroop effect (ESE) is the result of longer naming latencies to ink colors of emotion words than those of neutral words. This report refers to potential sources of confounding and includes a modal experiment that provides the means to control for them.

Abstract

The emotional Stroop effect (ESE) is the result of longer naming latencies to ink colors of emotion words than to ink colors of neutral words. The difference shows that people are affected by the emotional content conveyed by the carrier words even though they are irrelevant to the color-naming task at hand. The ESE has been widely deployed with patient populations, as well as with non-selected populations, because the emotion words can be selected to match the tested pathology. The ESE is a powerful tool, yet it is vulnerable to various threats to its validity. This report refers to potential sources of confounding and includes a modal experiment that provides the means to control for them. The most prevalent threat to the validity of existing ESE studies is sustained effects and habituation wrought about by repeated exposure to emotion stimuli. Consequently, the order of exposure to emotion and neutral stimuli is of utmost importance. We show that in the standard design, only one specific order produces the ESE.

Introduction

Modern life is replete with emotion and stress. Who has avoided the emergency room or (witnessing) a traffic accident? In order to perform efficiently under such stressful situations, it is important to preserve one's composure by focusing on the relevant stimuli. However, research has shown that the emotional valence of the stimulus can affect attention, in particular modulate the speed of processing. In the laboratory, one of the most popular paradigms to study the effect of negative stimuli on performance is the emotional Stroop task. The typical finding is that it takes people longer to name the ink color of emotion words than that of neutral words, the Emotional Stroop Effect (ESE). There are several accounts that attempt to explain the observed slowdown attributing attention³, freezing², or mood³, however it is still a matter of current debate.

The experimental setup of the emotional Stroop task is well known. Words in color are presented singly for view and the participant's task is to name the ink color of each word as quickly and accurately as possible. The words come from two categories of different valence. The first category includes negative words (e.g., DEATH) or words related to a specific psychopathology (e.g., GERMS with obsessive-compulsive patients or BATTLE with post-traumatic stress disorder patients). The second category includes neutral words (e.g., CHAIR). The ESE is the difference in color-naming latency between the emotional and the neutral words. The stimuli can be presented in a single block with emotion and neutral words intermixed in a random fashion or in two separate blocks defined by word category. The slowdown with emotion words is usually more pronounced when the ESE is derived in the blocked design^{4, 5}. Therefore, the block design has become the method of choice for researches of the ESE and it is the method applied in this protocol too (see Figure 1 for an illustration of the emotional Stroop experimental setup).

Figure 1: The Experimental Array in the Emotional Stroop Task: The participant's task is to name color in which the word appears. Please click here to view a larger version of this figure.

It is important to distinguish the ESE from its older namesake, the classic Stroop effect (SE)⁶. In the classic Stroop task, color words such as RED or GREEN are presented in various ink colors and the participant's task is to name the ink color of the words. Despite the shared task — to identify the ink color of words — the SE and the ESE differ. Because the words are color words, the stimuli in the classic Stroop task divide into congruent (the word naming its ink color) and incongruent (word and color conflict) categories. The SE is defined as the difference in color naming performance between congruent and incongruent stimuli. Because the quality of congruity does not apply to stimuli in the ESE — the word CANCER in blue is neither more nor less congruent than the word LECTURE in brown — the SE is not defined in the environment of the ESE. The ESE documents the effect of the emotional valence of the stimuli on performance.

The ESE, just like the SE, has generated voluminous research. In fact, the ESE rivals its namesake in sheer output of experimental studies with both patient and non-patient populations (for a review, see^{1, 7}). The task has been employed with a gamut of pathologies from generalized anxiety (e.g., ^{8, 9}) to trait anxiety (e.g., ^{10, 11}) to obsessive-compulsive disorders (e.g., ^{12, 13}) to depression^{9, 14} to social phobia^{15, 16} to post-traumatic stress disorders (e.g., ^{17, 18}). The ESE has also been studied with unselected populations (e.g., ^{2, 3, 19, 20}), although the effects in heathy participants are not always observed and are often less pronounced. At least a portion of ESE's popularity is attributable to its objective nature as it is not based on self-report and is not intrusive. Furthermore, the emotion words can be selected to tap the specific pathology or current concern of the patient.

Below, we portray the steps required to design and perform an emotional Stroop experiment. Our purpose in this report is to describe in detail an ESE experiment with needed controls. The most important feature of this design is the control it provides against various threats to validity. The main threat treated in this design is that of habituation. Adopting these procedures renders the ESE a valid and reliable means of assessing attention under emotion.

Protocol

The protocol follows the guidelines of Tel-Aviv University Helsinki human research ethics committee.

1. Word Selection and Matching

1. Create lists of words for each word category of interest. For example, make lists of generally emotional words (e.g., HATE, POX), concern relevance words (e.g., RAPE, VICTIM), and neutral words that are preferably orthographic neighbors²¹ of the emotional/concern relevant words (e.g., GATE, BOX). Create lists with as many words as possible as not all of these words will be used; a smaller subset of matched words will later be selected. 
   1. When deemed necessary, verify the valence/emotionality/arousal of the words by a questionnaire rating the words on a rating scale (e.g., from 1 to 7). Attempt to select words that are at the extreme end of the scale. If comparing emotional words with positive words, attempt to include words that are matched on (absolute) emotionality and arousal scores.
2. Write for each word on the lists, its length in characters as well as its estimated frequency in the appropriate language (for English words, use log hyperspace analogue to language (HAL) frequency²²) in order to match the word lists on lexical factors.
   NOTE: This is important as these variables can also affect color naming latency (e.g., ^{21, 23}). Arguably, the most important lexical attributes to control are word frequency, length, and if possible orthographic neighborhood (which naturally intersect with length).

   	Emotional			Neutral
   Word	Length	Frequency	Word	Length	Frequency
   hate	4	10.7	gate	4	9.7
   dead	4	11.2	dear	4	10.29
   poor	4	10.9	pool	4	9.7
   snake	5	8.6	shake	5	8.6
   gloom	5	8.1	bloom	5	8.2
   bomb	4	9.64	comb	4	7.39
   pox	3	7.1	box	3	12.1
   Average	4.1	9.4	Average	4.1	9.4

   Table1: Example of a List of Words Matched on Length, Orthographic Neighborhood, and Average Frequency.
3. Select words by adding pairs each time that match in length and frequency, and that are orthographic neighbors of one another (for example replace one of the emotional word letters with another to form a neutral orthographic neighbor, (see Table 1).
   1. If a complete match is not possible, balance the bias of frequency in the subsequent pairs of words by adding a pair of words with a small gap in frequency in the opposite direction of the gap formed by the previous pair of words. Ensure that the final lists of matched words contain 20 - 50 words in each word category in order to have a minimum of 20 trials in a block to collect sufficient data and avoid repetition of words and hence habituation³.
      NOTE: If many words are difficult to obtain, it is possible to introduce some repetition of words. However try to keep repetitions to a minimum as it may dilute the ESE due to habituation³.
4. Check that the final list is matched as much as possible on all lexical variables (e.g., word frequency, word length). Perform a statistical validation (student's t-test) to confirm that the final lists do not differ significantly from each other on frequency.

2. Preparation of Experimental Design

1. Select the design: Blocked (the stimuli are presented in separate blocks of trials defined by word valence) or mixed (the stimuli are presented in one block in which emotion and neutral words are intermixed in the same list).
   1. Choose a blocked design if seeking to test global effects at the level of the word category.
      NOTE: Effect sizes are usually higher in a blocked design than in a mixed presentation^{4, 5}. Therefore, blocked design is often the favored method of presentation.
   2. Choose a mixed design if seeking to decompose the ESE at the individual word level and for "fast" and "slow" effects, as a negative item can affect not only its own naming (the ESE) but also that of the immediately following item. Consequently, the former is dubbed 'fast effect,' the latter 'slow effect'^{24, 25}.
2. In a blocked experimental design, select a fixed or counterbalanced/randomized order of blocks.
   1. For a counterbalanced order present the two possible order of blocks to different groups of participants:
      Group A: Neutral Block - Emotional Block
      Group B: Emotional Block - Neutral Block
      NOTE: In this balanced design, an ESE is expected to emerge only in the group performing in the first order due to sustained effects in the group performing in the second order (see Figure 4)³. We therefore suggest using a fixed presentation of blocks.
   2. For a fixed order of blocks present first the neutral block then the emotional block, and if desired present an additional new neutral block flowing the emotional block in order to examine sustained effects: Neutral Block 1 - Emotional Block - Neutral Block 2.
      NOTE: In this setup, two effects can be observed. The first is the canonical ESE, calculated as the difference in performance between the emotional block and the first neutral block. A second is a sustained effect, obtained by subtracting the mean latency of the second neutral block from that of the first neutral block. A positive difference indicates the presence of sustained effects brought about by the emotional block.
      1. In order to rule out confounding through training or fatigue, advisably perform an auxiliary experiment with several blocks of neutral items only.
         NOTE: With three blocks of 40 neutral words, no residual fatigue is expected (see Figure 3)³. If there are more than 40 words per block, or more experimental blocks, it may be necessary to control for effects of fatigue by counterbalancing block order (but check for order-of-blocks effects in the statistical analysis).

3. Experimental Programing and Randomization

1. Choose a computer software or programing language to serve as a vehicle to present the stimuli and measure the participant's responses.
   1. Optionally, use the commercially available DirectRT software which is relatively easy to deploy and reliable. See Supplemental Code File for a DirectRT executable excel file as an example for a typical programed ESE experiment. Some additional software packages are SuperLab and E-prime, which are also suitable alternatives for governing the experiment.
2. Select the method of responding: manual or oral. Both types of responding are appropriate.
   1. For keyboard activated responding, use a longer training session in order for the participant to learn the mapping of keys onto the ink colors (of about 20 - 40 trials).
   2. For vocal responding use a shorter training session (of 4 - 8 trials). Set the reaction time to be measured from the outset of the stimulus to the first phoneme said.
      NOTE: Vocal responses pose difficulties at the stage of deciphering the responses (one must listen to the recordings and classify errors). However, a new algorithm that mechanically classifies the vocal recordings can render actual human classification gratuitous²⁶. An advantage of oral responses is that the resulting auditory files can be further analyzed (via other dedicated programs) for vocal parameters associated with emotion³.
3. Choose the ink-colors to be assigned to the words (e.g., the colors blue, green, red, and purple). Use easily discriminable colors against a well contrasted background (e.g., white or grey). If key-press responding is used, use no more than 4 colors, so that the mapping of colors to the keys is easily mastered.
   NOTE: If vocal responding is used, it is possible to use many more colors. Recall though that with a computer algorithm to classify the data, voice-identification errors increase with the number of colors (= responses).
4. Use an easily legible font, and size for the words.
5. Since the preferred approach is to use no repetition of words, assign to each word a single color randomly by the computer program for every participant.
6. Present each word singly around the middle of the computer screen. Optionally, introduce a small amount of spatial uncertainty, so that each word is presented in a random different position approximately 50 pixels around the center (e.g., see Supplemental Code File). This is done in order to discourage participants focusing on a small section of the word (thus circumventing reading).
7. Prepare a short training block to familiarize the participant with the task and the stimuli. For vocal responding, a few trials using the word 'example' in each of the experimental colors may suffice (with computerized identification of data, a whole session is needed to train the voice-identification algorithm); for key-press responding, as many as 20 - 40 trials may be necessary to master the mapping of the colors.
8. Following training, present the experimental blocks (e.g., three blocks of neutral-emotional and neutral words). Introduce short breaks between successive blocks of trials (of say, 30-60 seconds each). Do not allow the next block to begin before the designated interval for the break has elapsed as participants tend to rush through the experiment in order to finish it quickly.
9. Prepare the task instructions. At the start of each block, present the following instruction 'Respond to the ink color of the word as quickly and accurately as possible.' Avoid mentioning word reading, or that the word should be ignored as this request may artificially augment reading the words (see ironic process theory, e.g., ²⁷).
10. Specify the inter-trial or inter-stimulus interval (ISI) between experimental trials of a stimulus and the successive stimulus. Typically use an ISI of 500 msec (in blocked presentation).
    NOTE: Short ISI may promote shorter responses than longer ISI and carry over-effects with emotion items have been reported for ISI up to 1,000 msec²⁴.
11. Optional: Add an anxiety questionnaire at the end of the experiment or in a separate session. This provides for a baseline measure of the participant's anxiety, such as the state-trait anxiety inventory (STAI)²⁸.

4. Subject Selection and Preparation

1. Once programing is completed, recruit participants preferably from the same age-group and background. Participants can enroll for course credit, for pay, or on a voluntary basis.
2. Make sure participants are native speakers of the language used in the study, and do not have any attention deficits or color blindness.
3. Guide the participant to a quiet room in front of a computer. Explain the task instructions and ask the participant to read additional instructions written on the computer screen.

5. Data and Statistical Analysis

1. As a rule, perform reaction time analyses with respect to correct responses only. Also, exclude extreme responses.
   1. Typically, exclude responses faster or smaller than 2.5 SD around the mean. Nevertheless, be careful not to discard more than 5% of the data. Error rates are typically small, but it is still advisable to compare them across conditions and rule out a speed-accuracy tradeoff.
2. With a fixed presentation order, perform planned comparisons (or a student's t-test, if only two blocks are administered) with a designated statistical software such as SPSS or STATISTICA (here instructions are given for STATISTICA). For testing of the ESE, compare the emotion block with the first neutral block. For sustained effects compare the second neutral block (which followed the emotional block) with the first neutral block.
   1. Perform planned comparisons by choosing -Statistics->Advanced Linear/Non-Linear Models->General Linear Models->More results -> Planed comps.
3. When the order of blocks is counterbalanced across participants, perform an ANOVA with Block Valence as a within-subject factor and Order of Blocks as a between-subject factor.
   1. Perform ANOVA by choosing -Statistics->Advanced Linear/Non-Linear Models->General Linear Models.

Representative Results

When blocks follow the neutral-emotion-neutral sequence (e.g., ³), a large ESE of 34 msec is observed via slower responses in the emotion block (mean of 791 msec) than in the first neutral block (mean of 757 msec; see Figure 2). The same group of participants was also fairly sluggish to name the ink color in the second set of (other) neutral words (mean of 778 msec). The 21 msec difference in performance between the two blocks with neutral items documents the presence of the sustained effect of exposure to negative emotional stimuli.

Figure 2: Mean RTs to Name the Ink-color of Singly Presented Words in Three Blocks of Trials with Neutral, Emotion, and More Neutral Items. The blocks with neutral items entail different matched words. Vocal responses were used in this experiment. The error bars depict one standard error around the mean. Please click here to view a larger version of this figure.

In order to verify that there is no flagging of attention or fatigue (especially with a fixed order of blocks), an auxiliary experiment entailing solely blocks of neutral items can be performed. If there is no difference in performance across successive blocks of trials, one can assume that such effects are minimal (see Figure 3).

Figure 3: Mean RTs to Name the Ink-color of Singly Presented Words in Three Blocks of Trials with Neutral Items. Each block entails a different set of matched words. Vocal responses were used in this experiment. The error bars depict one standard error around the mean. Please click here to view a larger version of this figure.

In the typical ESE study in the literature with only two blocks of trials (emotion, neutral), an ESE is only expected to emerge in the group of participants performing first in the neutral block. There is not an ESE in the reversed order of blocks. This should result in an interaction of Block Valence and Block Order in the pertinent ANOVA. Of course, this standard design is not suited to test sustained effects or fatigue and habituation. Therefore, we suggest using designs with a minimum of three blocks, neutral-emotion-neutral (Figure 4).

Figure 4: Mean RTs to Name the Ink Color of Emotion and Neutral Words Presented in Blocks of Different Order. In the Neutral-then-Emotional (NE) group, the block of neutral words preceded the block of emotion words. In the Emotional-then-Neutral (EN) group, the block of emotion words preceded that of neutral words. Vocal responses were used in this experiment. The error bars depict one standard error around the mean. This figure has been modified from³. Please click here to view a larger version of this figure.

Supplemental Code File. Please click here to download this file.

Discussion

The ESE comprises a very simple task: The participant names the ink color of singly presented words. This simple task yields results of both pragmatic and theoretical consequence. The ESE documents the fact that people are sensitive to the emotional valence entailed in stimuli although this feature is completely irrelevant to the task at hand.

The ESE has evolved into an immensely popular method for probing emotion and anxiety with both patient and non-patient populations^{1, 74}. Its appeal can be attributed to its potential as an objective (computer-based)diagnostic tool, free of potential patient-therapist interaction bias. The emotion words can be selected to match the specific pathology or current concern of the patient. Furthermore, the tool is not intrusive nor self-report based. The efficiency of the ESE is firmly established at the group level, but it has yet to be demonstrated at the individual level^{29, 30}. Further studies are needed in order to assess the reliability of the individual patient's ESE and its relation with other known measures of anxiety and other computer based paradigms such as the dot probe³¹. Also, despite the prevalence of the ESE, the precise magnitude of the effect is moot, with reported effect sizes ranging from -1 to 400 msec¹. This is partly due to the use of different settings (e.g., computer/ cards/ oral) or to the specific pathology group tested. However, substantial variability is still evident in studies testing well-defined population groups in similar test settings. One purpose of the current protocol is to present a clear and standardized procedure byway of removing confounding and unwanted variability. This can be achieved by employing critical steps such as avoiding word repetitions, bypassing habituation, and allowing for proper lexical control. Following these guidelines should help researchers collect valid data, draw unbiased conclusions, improve reliability, and aid with comparisons across various emotional Stroop studies.

The importance of the critical steps granted, there are many possibilities for variation. Given the possibility of different research questions, the current protocol may not be optimal for some and departures from the current protocol are possible. For example, if a researcher wishes to examine the effect of vocal emotional interference, modifications of the protocol may be necessary Researchers should decide their preferred method of administration to fit their experimental needs. Variations also apply to the number of blocks and word categories to individual word selection (e.g., controlling for additional lexical variables such as number of syllables) to determining the number of trials (words) within blocks to employing a mixed or blocked design to introducing fixed/randomized/counterbalanced order of blocks to choosing a vocal or keypress responding to choosing the colors or specifying the inter-trial and block intervals. The advantages and disadvantages of each of these considerations are addressed in the relevant protocol steps. Most can be fitted to one's needs and individual preference.

Materials

Name	Company	Catalog Number	Comments
DirectRT software	Empirisoft	http://www.empirisoft.com/directrt.aspx	Software allows easy programing of a custom experiment in a spreadsheet application (e.g., Excel)
Personal computer	Any	https://www.office.com/	At least 256 mb of memory, a graphics card with at least 16 mb of video memory and a processor speed of at least 500 mhz.
A spreadsheet application (e.g., Excel)	Microsoft	Any version	A spreadsheet application capable of editing CSV files
SPSS/STATISTICA software	IBM/Statsoft	http://www-01.ibm.com/software/analytics/spss/  http://www.statsoft.com/	Statistical software