Eye-tracking to Distinguish Comprehension-based and Oculomotor-based Regressive Eye Movements During Reading

Summary

The method was designed to investigate the role of inhibition of return (IOR) in regressive eye movements during reading. The focus is on differentiating between regressions triggered as a result of comprehension difficulty versus those triggered from oculomotor error, including the role of IOR in the two types of regressions.

Abstract

Regressive eye movements are eye movements that move backwards through the text and comprise approximately 10-25% of eye movements during reading. As such, understanding the causes and mechanisms of regressions plays an important role in understanding eye movement behavior. Inhibition of return (IOR) is an oculomotor effect that results in increased latency to return attention to a previously attended target versus a target that was not previously attended. Thus, IOR may affect regressions. This paper describes how to design materials to distinguish between regressions caused by comprehension-related and oculomotor processes; the latter is subject to IOR. The method allows researchers to identify IOR and control the causes of regressions. While the method requires tightly controlled materials and large numbers of participants and materials, it allows researchers to distinguish and control the types of regressions that occur in their reading studies.

Introduction

The method described in this paper was designed to investigate the role of inhibition of return (IOR) in regressive eye movements during reading, focusing on regressions triggered as a result of comprehension difficulty versus those triggered as a result of oculomotor error. Specifically, we investigated whether regressions launched as a result of comprehension difficulty and those launched as a result of oculomotor error are subject to IOR effects.

Regressive eye movements, or regressions, are eye movements that move backwards through the text. Depending on reader and text characteristics, 10-25% of eye movements move backwards¹. This has led researchers to investigate whether IOR effects affect regressive eye movements during natural reading. IOR is an oculomotor effect that results in increased latency to return attention to a target that had been previously attended compared to a target that had not been previously attended². While much of the work done to establish IOR effects has involved non-reading visual attention tasks³, the effect has been extended to reading^4,5.

The work examining regressions and IOR in reading has focused on whether an oculomotor effect such as IOR can influence eye movement control in reading. One study⁵ found evidence of IOR in a reading task. They found that readers spent approximately 30 ms longer on the fixation preceding a regression. This was interpreted as the IOR "cost" - the delay before returning to a previously fixated position. This was supported in regular reading and mindless reading conditions⁶.

Despite evidence that IOR can be found in regular reading situations, it is clear that regressive eye movements do not all have the same underlying cause. Regressions resulting from comprehension difficulty have been well documented^7,8,9,10. Despite the evidence that eye movements during reading are generally guided by cognitive and linguistic factors¹, it is also assumed that sometimes regressions occur in response to low-level oculomotor factors, such as target overshoot¹. It is assumed that on some trials, readers mis-program a saccade and land beyond their intended target word (an overshoot). In this case, a short, corrective regressive saccade may occur so that the unintentionally skipped word can be fixated. Given that two underlying mechanisms - linguistic and oculomotor - have been posited for regressive eye movements, it is not clear whether IOR occurs for both. The current method allows for the measurement of IOR effects when regressive eye movements have been launched as a result of comprehension difficulties and as a result of oculomotor overshoot. Thus, the method allows researchers to distinguish the underlying mechanisms of regressive eye movements, allowing for the evaluation of IOR effects.

The current method takes advantage of the two identified mechanisms triggering regressive eye movements. By designing the materials so that re-reading is likely to be triggered by comprehension difficulty or overshoot, researchers are able to examine the circumstances under which IOR might occur in reading. To encourage re-reading as a result of oculomotor error, we embedded short target words that show high skipping rates of about 50% (adapted from previous research¹¹). Skipped words are often followed by a corrective regressive saccade when the skipping is a result of oculomotor error¹¹. The other set of materials was comprised of sentences that contained semantically ambiguous homographic homophones (e.g., grade: school/incline). The sentences were adapted from an ambiguity study¹² and contained information as to the intended meaning of the homophone that followed the word. Thus, this would increase the chances that readers would re-read for comprehension. The context was consistent with the less-likely meaning of the homophone, making it likely that readers would have to re-read after having initially selected the more frequent, dominant meaning on their first encounter with the target word. The combination of eye-movement monitoring and materials designed to increase regressions makes this method unique in allowing for the examination of regressive eye movements with differing underlying causes.

Understanding the mechanisms underlying regressive saccades and the role that oculomotor factors such as IOR play in them is important to models of eye movement control as well as for understanding the relationship between oculomotor and cognitive control of eye movements. For example, a recent version of the E-Z Reader model of eye movement control employs a 30 ms cost for all regressive eye movements¹³. However, our methodology demonstrated that such a cost only applies to regressions resulting from oculomotor error.

Eye movement measures allow researchers to track the moment-to-moment cognitive processing during reading¹. Recently, models of eye movement control have begun to try to explain the mechanisms underlying regressive eye movements. Since regressions are often launched in relation to comprehension difficulties, any researcher interested in understanding the comprehension processes during reading should attempt to differentiate regressions resulting from oculomotor error versus comprehension processes. This methodology indicates that a cost for regressions is a result only of oculomotor error, serving as a launching point for differentiating between types of regressions. The combination of eye movement measures (regressions, fixation times before regressions) and carefully controlled materials allow for this differentiation.

Protocol

The Institutional Review Boards of Kent State University and Stetson University have approved all methods described here.

1. Eligible Participants

NOTE: The purpose of this research is to understand reading processes in skilled adult readers. Thus, certain eligibility requirements must be met. Such controls ensure that results are directly applicable to a population of skilled adult readers with typical cognitive processes.

1. Recruit participants that meet the following eligibility requirements: must be at least 18 years old, have no reading disabilities (dyslexia, alexia, dysgraphia, etc.), have normal or corrected vision (glasses or contacts are acceptable), and speak English as their native language.

2. Experimental Stimuli

NOTE: Stimuli construction involves the selection of individual target words as well as the creation of sentence contexts in which those target words are embedded.

1. Prepare stimuli for oculomotor corrective regressions.
   1. Select a set of at least 30 target words to be embedded into sentences. The target word (word_n) is the word on which word skipping and regressive eye movements will be measured. Select words from an English language corpus, such as CELEX¹⁴, which provides word frequencies. Select words that are low frequency, with frequencies lower than 30 counts per million. Choose a three-letter content word to facilitate word skipping.
   2. Select the pre-target words (word_n-1) so that the reader is likely to fixate it before skipping or fixating word_n. Again, using an English language corpus, select words that are high frequency, with at least 30 counts per million. Choose a five to seven letter content word to increase the likelihood that it will be fixated and not skipped.
   3. Select the post-target word (word_n+1) so that the reader is likely to fixate it after skipping or fixating word_n. Using the same English language corpus from the previous two steps, select a high frequency preposition that is at least four letters long.
   4. Using the words selected in the previous three steps, create sentence contexts in which these three words will be embedded. Place words_n-1, word_n, and word_n+1 in the middle of the sentence. See step 2.2.6 for some example stimuli.
      ​NOTE: Sentences should be neutral in context without any emotionally charged language. The sentence frames need to be written so that that the overall sentence is grammatical and that words_n-1, word_n, and word_n+1 fit smoothly into the sentence frame. The target word_n should not be predictable from the context that precedes it (see step 2.2.5 for information about norming to ensure that target words are not predictable).
   5. After sentence frames are created, norm the stimuli using a cloze task to ensure that the target word_n is not predictable from the sentence context.
      1. In this task, recruit a separate set of participants.
      2. Have these participants view each sentence context up until the target word, without including the target, and produce a word that they think best fits the sentence.
      3. Let the participants complete this task using pen and paper or enter words on a computer using a program or any other software package for behavioral research.
         NOTE: If the target word is produced by more than 20% of participants, then it is considered predictable from the sentence context and must be replaced with a less predictable word and normed again. In the original experiment¹¹, 100 participants completed the cloze task, and no target word was predicted from the preceding context.
   6. See an example sentence below. In this sentence, green is word_n-1, gem is the three-letter low-frequency unpredictable word_n, and around is word_n+1. In this experiment, the target word was controlled to be low frequency (mean = 10.54 counts per million, standard deviation = 6.97)¹⁴.
      The actress wore a green gem around her neck at the awards show.
      Another set of 10 experimental stimuli are presented below from the original experiment¹⁴.
      He stared at the large urn across the room.
      She was afraid of the black fox until it wagged its tail.
      He tried to grab the small eel from the pond.
      She wanted to see the great zoo inside of the new theme park.
      They walked around the whole gym after their workout.
      The child asked the young elf about working in Santa's shop.
      There was always a short cot under the bed at her cousin's house.
      There was really heavy fog along the coastline this morning.
      He placed his right ski above his left one when he performed the trick.
      She performed her first gig since she wrote some new songs.
2. Prepare stimuli for comprehension-based regressions.
   1. Obtain sentence contexts for this experiment by adapting from a previous study that already has normed sentence frames with ambiguous homophones¹². Download this article and view Appendix A, which provides neutral sentence frames that end with an ambiguous homophone.
   2. Using these sentence frames, write the second half of the sentence that disambiguates the homophone to its subordinate meaning. Determine the subordinate meaning of the target word using the University of South Florida homograph norms¹⁵. For example, the subordinate meaning of the homophone grade is "hill incline" while the dominant meaning is "evaluation scale." Thus, for the sentence "Mike didn't like the grade…" add "…of the hill that they were driving down." This ending makes it so that the ambiguous homophone "grade" is disambiguated to its subordinate meaning by the word "hill." Each sentence should have two short function words between the ambiguous homophone and the words that disambiguates it. Another set of 10 example sentences are presented below¹⁴
      Barbara examined the corn on her foot after it became painful.
      Ben finally noticed the bluff when the trail suddenly ended.
      Betty searched for the file for her nails since they were torn.
      Debbie liked the case that the lawyer decided to take on.
      Everyone noticed the crab when he whined about everything.
      Fred closely examined the table and the chart in the report.
      Harry didn't like the suit that was filed against him.
      He was happy about the press when the wrinkle was removed from his pants.
      Helen looked at the cardinal in the church on Sunday.
      It was a very bad sentence that the judge delivered to the defendant.
      Jen easily saw the ruler and the queen at the palace.

3. Comprehension Check

1. Verify that all participants are reading for comprehension throughout the experiment. Thus, randomly present a series of comprehension questions throughout the experiment. Use simple statements that are followed by a yes or no question to ensure that the participant was comprehending each sentence. Present the comprehension questions separately from the sentence. Eye movement patterns may be different when a participant is not paying close attention to the text. Remove data from any participant who scores less than 80% accuracy on the comprehension questions from data analysis. There should be comprehension questions on at least 10% of trials. Comprehension questions should follow filler sentences, not experimental sentences. An example of four comprehension questions are presented below:
   Sentence:      Christina went to California with her parents last year.
   Question:       Did Christina go to Vermont last year?
   Sentence:      After Mark graduated from college he found a job immediately.
   Question:       Did Mark fail to find a job?
   Sentence:      The telephone rang just as Samantha crawled into bed.
   Question:       Did someone call Samantha last night?
   Sentence:      The tourists regretted not being able to stay longer in Florida.
   Question:       Did the tourists want to stay longer?

4. Eye Tracking Procedure

1. Recruit at least 50 participants for each experiment (oculomotor control and comprehension-based regressions) to ensure sufficient data for analyses; drop trials without regressions. Prepare at least 40 experimental stimuli for each experiment, with at least 10 comprehension questions, and at least 20 filler sentences for each experiment. Each experiment should take approximately 30 minutes to complete.
2. Use an eye-tracker with a sampling rate of 1,000 Hz, which means that eye positions are measured 1,000 times per second. Measure the eye movements from the right eye.
   NOTE: This sampling rate is important as eye movement behavior is extremely fast, and a sample rate of less than 1,000 Hz might not be sensitive enough to capture all cognitive processing.
3. Have the participants seated about 60 cm (24 in.) away from a computer screen. Use a 21 in. screen or other screens for the original experiment¹⁴.
   1. To stabilize head movements, have participants rest their chin on a chin rest. Allow participants to adjust the height of the chin rest and chair until they are comfortable.
   2. Instruct the participants to rest their fingers on a "yes" button and a "no" button that they will use to answer comprehension questions.
4. To calibrate participants' eye positions, use a nine-dot calibration. Instruct the participants to gaze at a white circle in the center of a black screen. Participants will move their eyes to the white circle as it moves around to eight other positions on the screen. Have the participants keep their eyes on the circle until it moves to a new position.
   1. After calibration, validate eye positions. If participants' eye positions are corrected within one degree of visual angle, then ask the participants move on to the reading portion of the study.
5. Have each sentence appear one at a time in random order in the center of the screen. Use any monospaced font such as Courier so that all characters take up the same amount of space on the participant's retina. Intermix filler sentences and comprehension sentences (followed by questions) with experimental sentences.
6. Prior to each sentence, have the participant gaze at a white circle on the left side of the screen that corresponds to the position of the first letter of the first word. When participants are looking at this position, have the experimenter press a button that controls the onset for reading.
   NOTE: This experiment uses self-paced reading, so the participant will press the "yes" button when they have finished reading the sentence. After reading each sentence, the white circle will appear on the screen again and the process will repeat with the next sentence.
7. Perform a drift correct prior to each sentence to ensure that calibration remains accurate. If the eye position is off by more than one degree of visual angle, recalibrate the participant's eye positions.

5. Quantitative Analysis of Data

1. After all data have been collected, open the program DataViewer, and load in all data files in .edf format if using EyeLink software. Delete fixation durations that are fewer than 100 ms or longer than 1,000 ms as these do not reflect linguistic processing.
2. Next, run an Interest Area Report to gather the desired eye movement behavior measures from each word of interest: word_n, and word_n+1. Set interest areas around these items using a delimiter and the default settings for interest area size in Experiment Builder. Select several variables in order to determine the type of regressive eye movement and the presence or absence of an IOR effect:
   IA_SKIP: This variable indicates whether word_n was skipped. This is important to determine whether the regression to word_n is a return or non-return regression.
   IA_REGRESSION_IN: This variable indicates whether a regression was made into word_n. Only analyze trials in which this value is 1, otherwise it is not relevant for IOR analyses because no return eye movement was made to this target.
   IA_REGRESSION_OUT: This variable indicates whether a regression was made out of word_n+1. Only analyze cases in which this variable is 1, which indicates that a return eye movement was made.
   IA_FIRST_RUN_DWELL_TIME: This variable is a measure of how long the reader spent on word_n+1 on the first pass through the sentence. This duration indicates the amount of processing it took before a regressive eye movement was made. This duration will include any potential inhibition time before a regressive eye movement.
3. Now that each trial has been categorized, have the researcher inspect each trial individually to determine whether it counts as an oculomotor- or comprehension-based regressive eye movement. Only inspect trials where IA_REGRESSION OUT of word_n+1 is 1 and where IA_REGRESSION_IN to word_n is 1.
   1. In DataViewer, select the participant's data, and open each trial. Click through each fixation to inspect whether the regression out of word_n+1 was made directly into word_n after it has been fixated or skipped. If so, mark this trial to be included in analyses in the data file. Do not include trials where a regression was made to an earlier part of the sentence or when a regression was made out of word_n+1 after processing other parts of the sentence.
4. After all trials have been marked, analyze data using linear mixed effects models with the languageR and lme4 packages in R Studio¹⁶. Specify random slopes and intercepts for both participants and items. Fixed effects should include skipping of the target word and regressions into the target word.

Representative Results

The results of our previous work using this paradigm¹⁴ resulted in a 17% regression rate in the oculomotor error condition and a 29% regression rate in the comprehension difficulty condition¹⁴. In the oculomotor error condition, 32% of the regressions were to previously fixated words and 68% of regressions were to previously skipped words. The reverse pattern occurred in the comprehension difficulty condition. Of the 29% of regressions, 61% were to previously fixated words whereas 39% were to previously skipped words (See Table 1).

The important variable of interest in these experiments is reading time on word_n+1, or the word from which the regression was launched. An IOR effect would be indicated by longer reading times on word_n+1 prior to the regression when the target word_n was previously fixated compared to when it was skipped. Figure 1 indicated that participants spent significantly more time on word_n+1 prior to a regression to word_n when it was previously fixated (mean = 296 ms, standard error (SE) = 17 ms) than when it was skipped (mean= 253 ms, SE = 14 ms), consistent with an IOR effect. However, no IOR effect was observed in the comprehension difficulty condition. In this condition, reading times on word_n+1 were no different when a regression was made to word_n when it was previously fixated (mean = 296 ms,SE = 10 ms) than when it was previously skipped (mean = 291 ms, SE = 8 ms).

Figure 1: Reading times on word_n+1 prior to making a regression to word_n. This figure represents the reading times on word_n+1 from which the regression was launched. The regression was either made to a word that was previously fixated or previously skipped. Regression to the previously fixated words are expected to be longer if there is an IOR effect, demonstrating the latency to move the eyes back to a previously fixated target. Values are mean ± SE (error bars). Please click here to view a larger version of this figure.

Oculomotor Error		Comprehension Difficulty
17% Regression Rate		29% Regression Rate
Previously Fixated	Previously Skipped	Previously Fixated	Previously Skipped
32%	68%	61%	39%

Table 1: Probability of making a regression to a previously skipped or fixated target word. This table breaks down the probability of making each type of regression. The overall regression rates for the two types are presented on the top lines. The bottom line represents the percentage of times that the eyes returned to a previously fixated or previously skipped word.

Discussion

The current research provides a method for distinguishing between two different types of regressive eye movements during reading - those that are based on comprehension difficulty and those that are based on oculomotor error. The data provide evidence that a low-level attentional process, IOR, may depend on the type of regression. It was found that IOR only occurs for oculomotor-based regressions, but not for comprehension based regressions¹⁴. Thus, it is important to distinguish between different types of regressive eye movements when drawing conclusions about cognitive processing during reading.

While this protocol uses a commonly accepted and straightforward eye-tracking methodology, the critical steps are in creating the stimuli. It is important to control as many aspects of the stimuli as possible so that the researcher can accurately discriminate the two different types of regressive eye movements. It appears as though using biased ambiguous homophones that lead the reader to select the contextually incorrect meaning of the word results in comprehension-based regressive eye movements. In contrast, using short, low frequency, content words results in oculomotor corrective regressive eye movements. One modification that can be made to this design is using better control of the word from which the regression is launched. For example, in the comprehension-based regression stimuli, we were unable to control the length or frequency of the word from which the regression was launched in the original study¹⁴. In the analyses, we included the length and frequency of that word as covariates; however, it would be best to control the words as tightly as possible so that any variability can be attributed to IOR and not to other linguistic processing.

Given the design of the experiment, researchers are only able to analyze a small subset of the data. In the original experiment, only 15% of trials involved a regression that was made to the target word for oculomotor error and only 24% of trials involved a regression that was made to the target word for comprehension difficulty¹⁴. All other trials were discarded from analyses, as they were not relevant to the research question. Thus, a limitation of this design is that it requires large sample sizes of participants and a large number of items in the experiment. A typical sample size used in reading studies may not provide enough power to detect differences when they exist.

Future research could replicate and expand on these findings by investigating whether low skill and high skill readers vary in the degree to which they make different types of regressions. It is already known that beginning readers make more regressive eye movements than more skilled readers¹⁷, and that low-skill readers make more regressive eye movements than high skill readers¹⁸. However, it is not yet known whether there are differences between low skill and high skill readers on the IOR latency and whether they differ on the likelihood of making the two types of regressions. It is reasonable to expect that low skill readers would be more likely to make comprehension-based regressions than high skill readers would. However, there may be no differences in their probability of making oculomotor-based regressions as these rely less on the linguistic processing skills that distinguish high skill and low skill readers.

Materials

Name	Company	Catalog Number	Comments
Eyelink 1000 Plus	SR Research		Video-based eye tracker
Experiment Builder Software	SR Research		Software to build eye tracking experiments
Data Viewer	SR Research		Software to retireve eye tracking measures