Efficiently Recording the Eye-Hand Coordination to Incoordination Spectrum

Summary

Cerebral injury can damage both ocular and somatic motor systems. Characterization of motor control post-injury affords biomarkers that assist in disease detection, monitoring, and prognosis. We review a method to measure eye-hand movement control in health and in pathologic incoordination, with look-and-reach paradigms to assess coordination between eye and hand.

Abstract

The objective analysis of eye movements has a significant history and has been long proven to be an important research tool in the setting of brain injury. Quantitative recordings have a strong capacity to screen diagnostically. Concurrent examinations of the eye and upper limb movements directed toward shared functional goals (e.g., eye-hand coordination) serve as an additional robust biomarker-laden path to capture and interrogate neural injury, including acquired brain injury (ABI). While quantitative dual-effector recordings in 3-D afford ample opportunities within ocular-manual motor investigations in the setting of ABI, the feasibility of such dual recordings for both eye and hand is challenging in pathological settings, particularly when approached with research-grade rigor. Here we describe the integration of an eye tracking system with a motion tracking system intended primarily for limb control research to study a natural behavior. The protocol enables the investigation of unrestricted, three-dimensional (3D) eye-hand coordination tasks. More specifically, we review a method to assess eye-hand coordination in visually guided saccade-to-reach tasks in subjects with chronic middle cerebral artery (MCA) stroke and compare them to healthy controls. Special attention is paid to the specific eye- and limb-tracking system properties in order to obtain high fidelity data from participants post-injury. Sampling rate, accuracy, permissible head movement range given anticipated tolerance and the feasibility of use were several of the critical properties considered when selecting an eye tracker and an approach. The limb tracker was selected based on a similar rubric but included the need for 3-D recording, dynamic interaction and a miniaturized physical footprint. The quantitative data provided by this method and the overall approach when executed correctly has tremendous potential to further refine our mechanistic understanding of eye-hand control and help inform feasible diagnostic and pragmatic interventions within the neurological and rehabilitative practice. 

Introduction

A critical element of the neurological function is eye-hand coordination or the integration of ocular and manual motor systems for the planning and execution of combined function towards a shared goal, for example, a look, reach and grab of the television remote. Many purposeful tasks depend on visually guided actions, such as reaching, grasping, object manipulation and tool use, which hinge on the temporally and spatially coupled eye and hand movements. Acquired brain injuries (ABI) cause not only limb dysfunction but also ocular dysfunction; more recently, there is also evidence pointing to the dysfunction of eye-hand coordination¹. Coordinated eye-hand motor control programs are susceptible to insult in neurological injuries from vascular, traumatic and degenerative etiologies. These insults may cause a breakdown between any of the indispensable relationships needed for the integrated and rapid motor control^2,3,4,5,6. Many studies on the manual motor function have been completed and have leveraged visual guidance as a core pillar of the paradigm without a method or protocol in place to analyze eye movements concurrently.

In ABI, conspicuous motor deficits are often detected during the bedside clinical examination. However, concurrent ocular motor impairments and complex impairments involving the integration of sensory and motor systems may be subclinical and necessitate objective recording to be identified^{7,8,9,10,11,12,13,14,15,16}. Ocular-manual motor coordination depends on a large and interconnected cerebral network, highlighting the need for a detailed study. An eye-hand coordination evaluation with dual objective recordings provides an opportunity to assay both cognitive and motor function in multiple populations, including healthy controls and subjects with a history of brain injury, thus providing insight into cerebral circuitry and function³.

While saccades are ballistic movements that can vary in amplitude depending on task need, studies have shown dependencies between saccade and hand movement during visually guided action^17,18,19,20. In fact, recent experiments have demonstrated that control systems for both movements share planning resources^21,22. The motor planning hub for eye-hand coordination lies in the posterior parietal cortex. In a stroke, there are well-known deficits in motor control; hemiparetic patients have been shown to generate inaccurate predictions given a set of neural commands, when asked to perform visually guided hand movements, using either the more affected (contralateral) or less affected (ipsilateral) limb^{23,24,25,26,27,28,29}. Furthermore, eye-hand coordination and related motor control programs are susceptible to insult following neurological injuries, decoupling the relationships, temporally and spatially, between effectors³⁰. Objective recordings of eye and hand control are paramount to characterizing the incoordination or degree of coordination impairment and improves the scientific understanding of eye-hand motor control mechanism in a functional context.

Although there are many studies of eye-hand coordination in healthy controls^{17,31,32,33,34}, our group has advanced the field by our setting of neurological injury, for instance during stroke circuitry assessment, have investigated the spatial and temporal organization of hand movements, often in response to visually displayed spatial targets. Studies that have expanded the objective characterization to eye and hand have almost exclusively focused on the performance capacity to record both effectors post-stroke or in pathologic settings; the described protocol enables robust characterization of ocular and manual motor control in unconstrained and natural movements. Here we describe the technique in an investigation of visually-guided saccade-to-reach movements in subjects with chronic middle cerebral artery (MCA) stroke relative to healthy controls. For the simultaneous recording of saccade and reach, we employ concurrent eye and hand motion tracking.

Protocol

1. Participant

1. Recruit control participants older than 18 years, without a history of neurological dysfunction, significant eye injury, significant depression, major disability and/or electrical implants.
2. Recruit Stroke participants older than 18 years, with a history of brain injury in the middle cerebral artery (MCA) distribution, have the ability to complete the Fugl-Meyer Scale, maintain a full range of eye movements^35,36, have the ability to perform pointing tasks, and without the history of additional neurological dysfunction, significant eye health comorbidity, significant depression, major disability and/or electrical implants.
3. Ask participants to sign a consent form approved by the Institutional Review Board of New York University’s School of Medicine.
4. Participant Screening (for detailed exclusion criteria please see Rizzo et al³⁷)
   1. Take history and perform clinical examinations as discussed below.
      1. Assess the cognitive state of participants with Mini Mental State Examination (MMSE)³⁸.
      2. Perform neurological examination.
      3. Examine extraocular muscles and eye movements.
         1. Ask the participants to follow the researcher’s finger with their eyes while keeping their head in one position. Draw an imaginary H letter in front of them and make sure that your finger moves far enough out and up/down, assessing center, up, down, left, right, down/left, down/right, up/left, and up/right.
         2. Ask the participants to follow and maintain the gaze on an object moved slowly through their visual field to assess smooth pursuit. Cover a distance of approximately 24 inches and using a pencil as a target, sweep back and forth slowly in horizontal and vertical directions, repeating each three times.
         3. Ask participants to look as fast as possible between 2 targets that are placed 24 inches apart to assess saccades. Use a pencil and a pen as targets and direct gaze to the targets in a back and forth manner three times horizontally and vertically.
         4. Ask the participants to fixate on an object as it moves slowly towards to their eyes to assess convergence, centering the target, a pencil, on the bridge of their nose. Following this procedure, repeat the test by bringing the same target from the nose back out to the starting position (divergence).
         5. Ask the patient to cover one eye and look at the researcher’s nose. Move the hand out of the patient’s visual field and then bring it in, wag the finger slowly and ask the patient to let the researcher know when the hand comes back into view, repeat this for upper left, upper right, lower left, and lower right quadrants.
            NOTE: When the patient covers their right eye, cover the left eye, and vice versa.
      4. Assess the visual impairment by a visual-motor Integration test.
      5. Assess the visual acuity by Snellen chart^39,40.
      6. Assess the visual field with confrontation and if in question, perform Goldman or Humphrey visual field testing^41,42.
      7. Assess hemi-spatial neglect via line bisection test and the single letter cancellation test⁴³.
      8. Quantify the extent of disability via 25-item National Eye Institute Visual Functioning Questionnaire (NEI-VFQ-25) and a 10-item supplement survey⁴⁴.

2. Preparation for the experiment and the physical configuration of equipment

1. Equipment:
   1. Choose an eye tracker
      1. Choose an eye tracker that is capable of head-mounted use (to avoid interference with desk-based reach movements) high spatial resolution (≤0.1^o) and high temporal resolution (≥250 Hz).
      2. Record the binocular eye movement with the eye tracker at a sampling rate of 250 Hz (sampling eye position every 4 ms) tracking both pupil and corneal reflection.
   2. Choose of a limb tracker
      1. Choose a limb tracker that can map the movement in the x, y, z position, ³ 0.08 cm accuracy, Latency ³ of 3.5 ms.
   3. Choose a laptop capable of running a customized script that controls real-time integration of data acquired from two systems and co-registering the signals in real-time (Table of Materials).
   4. Choose a display monitor capable of integrating with the chosen laptop and that is large enough to support one-to-one correspondence between monitor and tabletop reach space
   5. Define a rectangle identical in size to the display monitor on a table surface between the participant and the display monitor, to use as a functional reaching space for experimental work.
2. Set up preparation:
   1. Set up a table with the height adjustable chair.
   2. Place a display monitor 40 cm from the far edge of the table (Table of Materials).
   3. Place a tabletop board (reaching surface) with the 1-1 ratio dimension with the display monitor.
   4. Set up the limb tracker by mounting the electromagnetic source under the table (Table of Materials).
   5. Set up the eye tracker, host PC (Table of Materials).
      1. Attach four infrared (IR) illuminators to four corners of the monitor using straps.
      2. Set the eye tracker configurations from eye tracker setup options screen.
         1. Select 13-point calibration from the pre-set configuration of the eye tracker.
         2. Select the high saccade sensitivity to detect small saccade.
         3. Select Pupil-CR mode to record both pupils and cornea.
         4. Select a sampling rate at 250 Hz.
3. Participant physical preparation
   1. Seat participants on a height-adjustable chair at the table with the computer display.
   2. Position the participant 60 cm away from the display monitor (Table of Materials).
   3. Fix the motion sensor (Table of Material) to the distal aspect of the index finger of the hand of the to be tested arm (dominant arms for controls, and both arms in participants with stroke)
   4. Place the eye tracker on the participants’ headband and adjust the headband and cameras (Table of Materials).
      1. Fitting the headband
         1. Adjust the tightness and position of the headband (using headband knobs) so that the front pad is in the center of the forehead and the side pads above the participant’s ears.
         2. Make sure that the headband camera is in the center of the forehead and over the bridge of the nose.
         3. Ask participants to raise their eyebrows, and if the headband moves, refit it higher or lower on the forehead.
      2. Adjust the camera and corneal illuminator position. Ask the participants to look at the display monitor.
         1. From the camera screen, select the head camera image, verify that it shows four large spots from the IR markers that are positioned in the center of the head camera image. If they are not in the center, adjust accordingly.
         2. From the camera setup screen, select one eye at the time. Adjust the two eye cameras by lowering and raising the eye camera handle till the pupil of the eye is in the center of the camera image
         3. Focus the eye camera by rotating the lens holder.
         4. Set the pupil threshold by pressing the Auto Threshold button on the camera setup screen.
         5. Perform the same adjustment for the other eye.
4. Calibration
   1. Calibrate the limb tracker output to reaching surface using a 9-point calibration, ask the participant to place their sensor attached finger on reaching surface (tabletop) locations as displayed on the monitor screen.
   2. Calibrate the eye tracker, ask participants to look at the calibration target that appears as a blue dot and maintain fixation until the next dot appears on the screen
      NOTE: Calibration targets appear in 13 randomly selected positions on the screen
   3. Calibrate the eye tracker at least twice per session, first one at the start of the experiment and at its halfway point.

3. Experiment

1. Ask participants to move their finger onto the start position, covering the start circle on the screen with the finger-indicator dot (red dot), while fixating (eye) the start position on the screen.
   NOTE: The start position is a correspondent location of the fixation point (blue dot) displays on the center of the screen (Figure 1a). The position of the finger is represented as 4 mm radius red dot on the screen.
2. Require participants to maintain finger position on the start circle for 150 ms until the target appears.
3. Ensure that participants fixate the start position until they hear a beep sound (“go beep”). (Figure 1)
   NOTE: The duration between target appearance and the go signal is randomized between 250 to 750 ms to prevent anticipation of the go signal.
4. Instruct participants to move both their eyes and fingertip quickly and accurately to the designated target as they hear the beep sound (Figure 1)
   1. Designated target appears 1 cm radius white circle
5. Instruct participants to touch the tabletop location at the position of the virtual target as displayed on the screen by lifting the hand and finger and re-connecting the fingertip and tabletop
   1. Make sure participants make a pointing movement by lifting the hand and finger rather than dragging the hand and finger on the tabletop.
   2. Display the end location of the reach as a red dot, following reach completion.
   3. Determine reach completion by a combination of low-velocity (<5% peak) and 3 mm z-plane threshold.
6. Ask participants to perform a series of familiarization trials before starting data acquisition.
7. Start data acquisition after participants touched 5 of the last 10 targets successfully.
8. Ask participants to perform a series of look and reach trials as they were instructed during familiarization trials.
   1. Have participants perform a total of 76 trials.
9. Have control participants perform the experiment with their dominant hand.
10. Whenever it is possible, have participants with stroke perform the experiment with both the hands, more-affected and less-affected.
11. Participants complete the entire experiment with at least one hand.

Figure 1. Schematic view of setup and experiment. (a) Schematic representation of display monitor and reaching surface during a trial. (b) Sequencing of actions within visually-guided reach. First Fixation (F) appears. The target (T) appears after a randomized length of time. The ‘go’ signal occurs as auditory beep sound (signified by the light-grey vertical bar) after an unpredictable time interval (concurrent offset of F) following by target appearance. Hand (H) and Eye (E) movements follow the go-signal. Please click here to view a larger version of this figure.

Representative Results

Thirty participants participated in the research study. There were 17 participants in the control cohort, and 13 participants in the stroke cohort. Two participants couldn’t finish the whole experiment, so their data were excluded from the analysis.

Demographics and Questionnaire Assessments

Table 1 shows the clinical and demographic characteristics of the representative stroke cohort.

Mean unweighted VFQ scores were 91.33 ± 13.01 in stroke participants, versus 94.87 ± 4.87 in healthy controls (p=0.203, ns). Mean scores of the 10-item supplement were 95 ± 11.57 in stroke participants, versus 96.27 ± 6.64 in healthy controls (p=0.375, ns). Mean scores for the composite and 10-item supplement were 92.36 ± 12.18 in stroke participants, versus 95.12 ± 4.65 in healthy controls (p=0.244, ns). Stroke participants had a mean Fugl-Meyer score of 55.54 ± 13.33, with a range of 30-66.

Eye and Hand Movements Durations and Latencies

In Figure 2 saccade and reach latencies, measured as the duration between the go signal and movement onset, are plotted. Stroke participants made the initial (primary) saccades significantly earlier in both less-affected and more-affected sides, comparing to healthy control participants (p<.05) (more-affected hand: 0.082 s, CI: [0.052 0.112]; less-affected hand: 0.106 s, CI: [0.08 0.132]; control saccade onsets: 0.529 s, CI: [0.514 0.543]). Compare to control, stroke participants made remarkably early initial saccade to target but there were no significant differences between control reach onsets and less-affected or more-affected reach onsets in stroke participants (less-affected hand: 0.545 s, CI: [0.521 0.568]; more-affected hand: 0.60 s, CI: [0.567 0.632]; control reach onsets: 0.556 s, CI: [0.544 0.568]). Latency between the initial saccade and reach onset, which represents a temporal decoupling in stroke participants, was greater in in both the more-affected and less-affected hand, a 519 ms (CI: [476 562]) and a 439 ms (CI: [404 474]) separation respectively in stroke, versus a minimal separation of 27 ms (CI: [8.5 45]) in controls (all p<.05). Stroke participants not only made the longest duration reaches (computed as the difference between movement onset and termination) with their more-affected side (604 ms, CI: [587 622]) but also increased their average reach time on the less-affected side (546 ms, CI: [537 555] vs 352 ms, CI: [348 356]) (all p<.05).

Eye Movements Frequency

We examined the interval between the initial saccade onset and reach onset, which was minimal in healthy controls and significantly longer in stroke participants in the less- and more- affected side. We noticed differences in the number of saccades that were made during this period. The number of saccades produced by stroke participants regardless of the limb they used, was more than healthy controls. We plotted the number of secondary saccades made by participants in histograms (Figure 3). Healthy controls in 90% of trials made a single saccade and sustained fixation at the target until they completed the reach. In sharp contrast, this pattern was generated in 50% of trials (z = 32.2, p<.05) for those with stroke and the remainder made multiple saccades. (Figure 3). Figure 4 shows an example of such saccade traces.

Spatial Errors of the Eye and Hand Movements

With regard to the amplitude from movement endpoint to target center (movement error), stroke participants had increased reach errors in both less and more affected hands relative to healthy controls (control: 9.3 mm, CI: [9.0 9.5]; less-affected arm: 19.2 mm, CI: [18.4 20.0]; more-affected arm: 21.4 mm, CI: [20.5 21.4]) (Figure 5; all p<.05). Along with the increase in reach errors, saccade endpoint errors increased greatly as shown in Figure 5 (control: 18.3 mm, CI: [17.9 18.7]; less-affected arm: 36.4 mm, CI: [35.2 37.6]; more-affected arm: 41.6 mm, CI: [40.3 43.0]; all p<.05).

Arm Motor Impairment and Eye-Hand Latency Decoupling Correlation

The Fugl-Meyer score was used to assess arm motor impairment. It was expected that temporal decoupling in stroke participants would correlate with arm motor impairment severity, but our results demonstrated that it was statistically insignificant for the less (r = -0.64, ns) and more affected (r = -0.34, ns) arms.

ID	Age	Sex	H / H a	Stroke	Chronicity (yrs)	Fugl-Meyer Score c
	(yrs)			Characteristics b
1	78	M	R/L	R MCA distribution	2	66
2	61	F	R/L	R MCA distribution	7	66
3	34	M	R/R	L MCA distribution	1.7	66
4	39	F	R/R	L MCA distribution	1.4	45
5	70	M	R/R	L MCA distribution	2.8	58
6	60	F	R/L	R MCA distribution	2.6	30
7	73	M	R/L	R MCA distribution	6	58
8	51	F	R/L	R MCA distribution	12.2	30
9	60	M	R/R	L MCA distribution	4.4	63
10	39	M	R/L	R MCA distribution	4.7	47
11	70	M	R/L	R MCA distribution	2	66
12	47	F	R/R	L MCA distribution	1.5	61
13	65	F	R/R	L MCA distribution	0.7	66
Avg	57.5				3.8	55.5
(SD)	-14.3				-3.2	-13.3

TABLE 1. Stroke Clinical Characteristics.
^a “H/H” = Handedness / Hemiparesis: Handedness (assessed through Edinburgh Inventory) / Hemiparesis Laterality

^b “Stroke Features”: lesion location obtained from medical history with participant and/or family members serving as historian; region and laterality cross-validated for consistency with examination findings
^c “Fugl-Meyer Score”: a summation of the Upper Extremity Score [total possible 66], which reflects the extent of post-stroke motor impairment.

Figure 2. Saccade and Reach Latencies Saccade onsets (indicated by blue circles) occur significantly earlier in the stroke participants, while there were no significant differences between control reach onsets (indicated by green circles) and stroke participants (indicated by green circles) (with a slight delay on the more-affected side). Latency between the initial saccade and reach onset is indicated with a light grey bar. (onsets: circles, terminations: squares) (error bar: 95% Confidence Interval) Please click here to view a larger version of this figure.

Figure 3. Histograms of the number of saccades in addition to the primary saccade. The upper histogram shows, control participants overwhelmingly make a primary saccade only. There were either no additional saccades beyond the primary saccade or contain a single secondary saccade in about 96% trials. The lower histogram shows stroke participants, make up to five secondary saccades in the same 96% of trials. Please click here to view a larger version of this figure.

Figure 4. Figure shows, random raw saccade trace from two control participants and two stroke participants. Two samples (unfiltered, raw) eye (blue) and hand (green) traces from control participants (left column), and stroke participants (right column) are plotted in screen mm to allow for simultaneous plotting of eye and hand traces. In two stroke participants trials, multiple eye movements are made before they complete the reach, as opposed to control participants trials that make a single saccade at or close the time of the reach. Please click here to view a larger version of this figure.

Figure 5. Average endpoint error by participant grouping and/or arm Green bars indicate average reach error, and blue bars indicate average saccade (primary)  error. Two-sample t-tests were performed. Please click here to view a larger version of this figure.

Discussion

The advent of eye and hand tracking systems as available tools for objectively exploring the characteristics of ocular-manual motor systems has accelerated research studies, enabling a nuanced recording approach for an essential task in daily activities – eye-hand coordination. Many natural action-dependent tasks are visually guided and depend on vision as a primary sensory input. Gaze is programmed through ocular motor commands which point central vision at key spatial targets; this information is pivotal and assists in acquiring hand goals. The key is that coordinated eye-hand behavior must be executed efficiently and accurately. For example, deciding to grab a coffee cup will result in a rapid eye movement to the handle, a terminal fixation, the acquisition of pivotal environmental detail for index finger placement and prehension, all in temporally synchronized series. Following movement initiation, visual feedback of the upper limb is crucial for online error monitoring and correction.

Assessment of eye-hand coordination with our distinct methodology indicates that stroke hampers the coordination of eye-hand movement control. Stroke participants with MCA-injury reveal both less accurate saccades and reach (in both less/more-affected sides) relative to healthy controls; there also appears to be stark decoupling between the primary saccade onset and reach onset in both less-/more-affected sides. While impairments of eye and hand movement contribute separately to functional compromise, there appears to be a specific deficit in eye-hand coordination, which may amplify reaching errors and further compromise neurologic function; this occurs when these separate effector systems fail to coordinate toward a single synchronous behavior. One potential explanation may lie in the additional computational load of executing dual eye-hand movements and the related interference effects^46,47,48,49. Experimental paradigms that require eye-hand movement co-registration permit scientists to systematically probe dual tasks; this is especially relevant to pathologic populations that have known difficulties with such tasks, regardless of the combination (cognitive-motor, motor-motor, etc)^50,51,52.

Eye and upper limb movements are sensitive markers of cerebral injury and myriad applications exist diagnostically, prognostically, and therapeutically^{53,54,55,56,57,58,59}. Eye movements and their relationships to limb movements create an even greater ‘window’ into the brain than previously thought. Aside from direct impairment in eye movement function, deficits in eye movement compensation in response to hand movement impairment is a new area rife with scientific opportunity. Once further characterized, eye-hand coordination will be capable of shedding light on multiple applications and motivate further studies to understand its full implications for functional movement control, translating mechanistic insight into clinical knowledge. The key to eye-hand control research is the robust methodology and vigorous protocols that enable one to assay such physiology concurrently and with high fidelity.

Despite the benefits delineated here, there are still methodological limitations present. As described in the methods section, participants are instructed to fixate the target as it appears on a display monitor and to make a concurrent reach on a tabletop positioned immediately in front of the workstation. This requires a transformation of spatial information from the monitor to the tabletop and adds an extra cognitive step. While this cognitive challenge is identical to the transformation one makes during computer work, translating information from the screen to the work station or into mouse-keyboard ‘space’, a more naturalistic task would use a translation-free paradigm. Regardless, robust 3-D hand tracking with objectively characterized eye recordings allow one to probe integrated motor control that revolves around multi-effector coordination. In addition, the present approach affords an opportunity to assess the eye-hand control aspects critical to interaction with the computer interface in real-time.

While quantitative dual-effector recordings in 3-D afford robust opportunities within ocular-manual motor investigations in the setting of ABI, the feasibility of such dual recordings for both eye and hand is challenging, particularly in a pathological setting when executed with research-grade rigor. Efforts have attempted to combine eye and hand tracker to assess eye and hand physiology, but the data output is often unstable ⁶⁰. When these instabilities seen in healthy populations are taken into account and juxtaposed with the technical calibration and recording issues in participants with pathology, the data becomes less and less useful. Hence, it is pragmatic to leverage a method and paradigm, as described here. Accordingly, eye position calibration is completed in the depth plane of interest, eye-specific stimuli are displayed at this single distance and gaze measurement fidelity is subsequently robust. At other distances, the eye’s view is no longer aligned, and characterization is limited to 3-D recordings of hand position^61,62. The quintessential study of the eye and hand in the pathologic setting will be best achieved with custom software that permits multi-depth calibrations, integrated hardware, a central computer or host system for signal co-registration, and a protocol similar to the one aforementioned.

Materials

Name	Company	Catalog Number	Comments
27.0" Dell LED-Lit monitor	Dell	S2716DG	QHD resolution (2560 x 1440)
ASUS ROG G750JM 17-Inch	AsusTek Computer Inc
Eye Link II	SR-Research	500 Hz binocular eye monitoring 0.01 º RMS resolutions
Matlab	MathWorks
Polhemus MicroSensor 1.8	Polhemus		240 Hz, 0.08 cm accuracy