Evaluating Tests of Cognition using a Computerized Touch-Sensitive Tablet, Eye Tracking, and Functional Magnetic Resonance Imaging

Summary

A protocol for simultaneous recording of visuomotor behavior and brain activity during standard paper-based cognitive tests using an MRI-compatible tablet and eye-tracking technology alongside functional MRI, towards improving the usage of such tests. Preliminary results are presented from a young healthy adult performing a Trail-Making test.

Abstract

Paper-based tests of cognition (such as the Trail-Making test, or TMT) have long been used in clinical and research settings to evaluate how the healthy or impaired brain supports behavioral performance. Despite widespread use, the neural correlates of such tests are poorly understood, and the tests have sensitivities and specificities that are less than desired. To address these shortcomings, a multi-modal research protocol is proposed that simultaneously combines novel tablet technology, eye tracking, and functional magnetic resonance imaging to explore the relationships between kinematic and visual behavior and neural activity associated with cognitive test performance. Protocol rationale, step-by-step methodology, and results from a representative participant are provided to demonstrate protocol validity and to illustrate the potential of exploring the kinematic, visual, and neural correlates of a representative test of cognition. The current protocol can expand the limits of existing clinical MRI neuroscience research, with implications for the future diagnosis and management of various cognitive disorders.

Introduction

Tests of Cognition (ToC) were first popularized in the 20^th century to investigate and characterize normal and abnormal or pathological cognitive behavior. Since their emergence, these tests have become widely adopted in research and clinical settings¹. Many ToC were developed with simple response formats, such as speaking or writing/drawing using a pen and paper. As an example of the latter category, the Trail-Making Test (TMT) is a widely used representative ToC that is favored due to its sensitivity to cognitive impairment². Comprised of two parts, TMT-A (numbers only) and TMT-B (numbers and letters), the test requires participants to use a pen to connect (link) 25 characters that are pseudo-randomly arranged on the page, in ascending sequential (and in the case of TMT-B, also alternating) order (i.e., TMT-A: 1-2-3-4-5-6…; TMT-B: 1-A-2-B-3-C…). To assess cognitive performance on the TMT, time to completion and errors are tabulated and compared to normative values, based on age range and education status². The TMT is thought to recruit and assess complex cognitive processes, including task switching, visual search, memory, visuomotor control, and attention—all of which are important aspects of executive frontal lobe function^1,3.

The TMT exhibits high sensitivity among ToC, but in terms of diagnoses, its poor specificity is well recognized as a limitation⁴. In general, sensitivity and specificity concerns are a drawback to the application and validity of ToC, particularly in clinical settings⁴. The traditional recourse to alleviate this concern has been to administer ToC in “test batteries” (often including the TMT) to improve discrimination between cognitively impaired and cognitively intact groups. However, test batteries are time-consuming, costly, and require considerable expertise to administer and analyze⁵. These logistical concerns, in turn, led to the development of “cognitive assessment” tools: substantially streamlined (and increasingly, computerized) test batteries for rapid administration in resource-limited settings (e.g., medical clinics), at the cost of some of the sensitivity and specificity gain. One example of such a tool is the Montreal Cognitive Assessment (MoCA)⁶.

Computerized assessments, such as the adapted MoCA, have been successfully validated through comparison to pen and paper analogs⁷, and to test batteries of ToC⁸. Yet fundamental limitations remain with all of these behavioral testing tools, including insufficient differentiation between appropriate and erroneous performance, focus on test scores for the entire test rather than intra-test effects, and limited insight into the various behavioral strategies and associated brain activity that underpin ToC performance^4,9. However, these limitations may be overcome through research that combines detailed behavioral recordings, intra-task behavioral evaluation¹⁰, and functional neuroimaging (e.g., electroencephalography¹⁰, functional near-infrared spectroscopy¹¹, and functional magnetic resonance imaging¹²).

Functional magnetic resonance imaging (fMRI) generates high-resolution images of brain activity by mapping hemodynamic response as a proxy for neural activation. Although expensive, the superior spatial resolution of fMRI over electroencephalography (EEG) and functional near-infrared spectroscopy allows for the localization of activity throughout the whole brain. Accordingly, the present work describes a novel administration method for ToC using the TMT as a representative example, which pairs fMRI with detailed, continuous, and simultaneous behavioral recording using computerized MRI-compatible tablet and eye-tracking systems. This multi-modal protocol offers greatly enhanced evaluation of the relationship between cognitive task performance and neural activity estimated by fMRI, useful to improve understanding of existing ToC and possibly providing insight for the development of enhanced ToC in the future.

Before providing a detailed description of the experimental setup to acquire tablet, eye-tracking, and fMRI data simultaneously, it is helpful to summarize the conceptual layout and approach (Figure 1). For MRI-compatibility and ergonomic reasons, the tablet system is slightly different from commercially available tablets. Popular tablets have a transparent touch-sensitive screen mounted on top of a computer display, enabling the user to look directly at the tablet and to receive visual input that seamlessly includes their stylus-based writing and drawing responses. In the present scenario, there is no computer display under the touch-sensitive screen. This design avoids the need for complex computer display electronics to operate safely in the intense magnetic field at the center of the magnet bore and without negatively impacting MR images. From an ergonomic perspective, space in the magnet bore is also rather limited, making it impractical for a research participant to view their hand directly while writing and drawing.

The experimental setup thus has participants perform tablet interactions on a support stand at their waist, while all visual information (test stimuli, stylus responses, video of their hand manipulating the stylus) is integrated together for viewing at the rear opening of the magnet bore through a mirror. The visual information is displayed on a rear projection screen using a commercially available, MRI-compatible projector (details provided below). Similarly, a commercially available eye-tracking system (details also provided below) is mounted in the rear magnet bore for rapid video recording of eye movements through the same mirror. The projector, screen, and eye-tracking apparatus must be arranged carefully so that they do not physically interfere with one another. Last, power and data connections to and from the tablet, projector, and eye-tracking system are made using various shielded cables, passing through the “penetration panel” of the radiofrequency shield that protects the magnet room and MRI system from surrounding electromagnetic interference. The data cables are under computer control, shown conceptually in Figure 1 as a single device under operator control in the MRI console area (distinct from the computer console used to operate the MRI system). As described below, multiple computers are involved in the present experimental setup.

Tablet system

The custom-built, computerized tablet system is comprised of MRI-compatible components (touch-sensitive surface, adjustable elevated support platform, force-sensitive stylus, projector system)¹², including a video camera with a 4.3 mm lens (designated the “TabletCam” in the lab) and a custom light-emitting diode (LED) illuminator¹³, enabling administration of ToC and recording of naturalistic writing or drawing responses within the magnet bore during fMRI (Figure 2A,B). Located in the console area, two linked computers are used for system control: one associated with receiving and processing video data from the video camera (“Tablet Video Camera computer”) and the other for test administration, delivery of visual stimuli, logging of tablet data, and creation of a video file consisting of the time-dependent administered visual stimuli superimposed with stylus writing and drawing responses (“Stimulus/Response computer”; Figure 2C). The two-computer approach is chosen for unimpeded real-time performance of each set of latency-sensitive functions; modularity for research requiring different configurations (e.g., different tablet-based behavioral tasks, optional use of the video camera); and ease of compatibility (the only requirement is a compatible video output format).

The tablet system has been used previously in several fMRI studies of ToC, which all suggest its strong ecological validity¹⁴. The optional video camera is added to the original tablet configuration to provide the participant with visual feedback of hand position (VFHP) during task performance, in an interactive augmented reality (AR) environment, enabling viewing of task stimuli as well as stylus responses and hand movements superimposed in real time¹³ (Figure 2D). In the original implementation of the video camera data processing¹³, the hand and stylus were isolated from each video frame using a skin color detection algorithm, with the stylus implemented in red to fall within the red-green-blue (RGB) distribution for skin color. More recently, a “blue screen” approach has been adopted for its simplicity and other advantages. A blue backdrop is created by covering the touch-sensitive surface of the tablet with blue painter’s tape. It is then possible to segment the hand and stylus from the backdrop in each video frame based on the substantially different color distribution of the tape. At the same time, this process also enables the creation of a binary mask with a value of “one” at every location occupied by the hand or stylus, and “zero” elsewhere. The stimulus/response video and camera video are then superimposed by creating frames consisting of a) stimulus/response video data everywhere that a given mask equals zero, and b) camera (hand and stylus) video data everywhere that the given mask equals one. The painter’s tape has the additional benefit of introducing extra friction when the stylus tip is moved across the stylus surface, closer to the experience of writing with a pen or pencil on paper, in comparison to the low-friction “plastic on plastic” feel when the tape is removed. Overall, the resulting interactive AR environment further enhances the ecological validity of the tablet design, while reducing reliance on proprioception to execute fine motor movements (as occurs when VFHP is absent)^13,15.

The tablet setup is used in conjunction with an MRI-compatible projector (Figure 2E) and a custom rear projection screen at the rear of the magnet bore. Participants view the screen through an angled mirror mounted on the head coil. Using a fingertip or stylus (which also includes a sensor to record contact force), the participant interacts with the touch-sensitive surface mounted on the support platform, which is positioned at the waist and is adjustable for each individual. Analog tablet signals pass through an electromagnetic interference (EMI) filter at the radiofrequency penetration panel, are transformed to touch data (surface location and force data) by a tablet interface box outside the magnet room, are logged and interpreted for graphical representation of touch responses on the Stimulus/Response computer, then are merged with visual stimuli and segmented hand and stylus video; and are presented to the participant using the projector.

TMT block design

The TMT is administered in a fixed block design consisting of alternating periods of TMT-A and TMT-B task performance, and of visual fixation to a central, black crosshair displayed on a white background. The overall task design was adapted from existing TMT literature^1,16,17,18, where TMT-A involves linking circled numbers (1 to 25) pseudo-randomly distributed across the screen, in ascending order. Similarly, TMT-B involves linked circled numbers (1–13) and letters (A-L) in an alternating and ascending fashion. The visual fixation condition is included so that brain activity associated with TMT-A, and separately with TMT-B, can be analyzed as a statistical contrast been the activations of interest and that of a simple, stable condition with low cognitive demand. Due to the inherently low signal-contrast-to-noise ratio observed in fMRI experiments, each behavioral condition (TMT-A, TMT-B, visual fixation) is repeated in multiple trials, enhancing the statistical power to detect brain activity when the collective fMRI data are analyzed. The TMT plots for each trial are adapted from standard TMT layouts by either rotating the stimulus distribution by 180°, swapping number-only stimuli and number-letter stimuli, or both—thus minimizing visual and motor confounds due to differences in character and number distribution on the TMT-A and TMT-B plots¹⁸.

The present experimental and training tasks are implemented in commercially available stimulus presentation software for behavioral and neuroimaging research, for execution on the Stimulus/Response computer. Practically, the TMT is administered in two “runs”, each 4 min:50 s in duration. Each run consists of an initial 10 s block of resting fixation, followed by two trials of TMT-A task (40 s), resting fixation (20 s), TMT-B task (60 s), and resting fixation (20 s) (Figure 3). At the beginning of each run, participants are given instructions that mirror those used in standardized paper TMT testing^16,17,18,19: connect the circles from “Begin” to “End” as fast and as accurately as possible, without lifting the stylus from the touch-sensitive surface. Unlike conventional paper TMT administration, the test administrator (a member of the research lab) does not stop and subsequently re-initiate TMT performance in the event that the participant makes errors. Instead, participants are instructed simply to continue to the next corresponding character link in the sequence. This modification eliminates any data analysis confounds associated with stopping and restarting eye-tracking and fMRI data collection within a given TMT trial. However, this then necessitates the implementation of error detection and categorization methods after the data are collected (see the protocol and discussion sections). In addition, the test administrator visually monitors the stylus responses in real time during TMT performance to record whether any errors were made, and to ensure that the touch-sensitive surface remains well-calibrated. In cases of tablet calibration errors and other hardware errors (e.g., power or equipment failure), the test administrator also decides whether to repeat the current TMT data acquisition run, possibly including recalibration of the touch-sensitive surface, or to stop and exclude use of the participant data in the subsequent analysis.

Eye tracking

When the human visual system processes a scene, such as during TMT performance, ballistic eye movements (saccades) are preceded and followed by periods of temporal stability (fixations)²⁰. An MRI-compatible high-speed eye-tracking system is thus used in the present context to perform long-range monocular eye tracking of fixations and saccades with infrared illumination (910 nm wavelength) and 1 kHz sampling frequency (Figure 4A). From the position of the eye-tracking camera under the projection display, the eye of the participant is localized in the head coil mirror (Figure 4B-D). Note that the product head-coil mirror shipped with the MRI system was replaced by a front-surface mirror provided by the eye-tracker manufacturer, to enable high-quality tracking. The pupil is detected using a standard centroid-fitting algorithm that tracks corneal reflection (Figure 4D), and the following metrics are measured: fixations, saccades, as well as blink rate and pupil size, two additional quantities associated with cognitive processing (see Discussion). A trigger pulse emitted by the MRI system at the start of fMRI is used to time-synchronize the brain activation recordings with a) the TMT task stimulus delivery and stylus responses (as controlled by the Stimulus/Response computer); and b) the eye-tracking data with TMT performance. To facilitate data analysis, the eye-tracking data are additionally “time-stamped” to provide labels associated with key events during the experiment, including the start and end times of each TMT-A and TMT-block in a given run.

An additional lab member is primarily responsible for the eye-tracking setup with the participant, eye-tracking calibration, and real-time visual inspection of eye-tracking data acquisition. Calibration and validation of the eye-tracking system is performed prior to the first TMT run (Figure 4E), and in a "drift-checking” procedure between the first and second TMT runs, to ensure consistency of results while accounting for possible slight changes in head position (see Protocol below for exact specifications and sequence). The calibration consists of a nine-point eye-tracking test, with the participant required in each case to fixate at a target in the center of the display, followed successively by eight different peripheral targets, in pseudo-random order. For validation, the participant tracks the same nine targets again, and the calibration model is used to estimate the gaze position. This enables a set of error measurements to be collected, constituting the difference between the estimated gaze and the actual target location. Spatial error is reported in degrees of visual angle on test completion. The initial calibration and validation are acceptable if the average error is <0.5^o and the maximum error is <1.0^o, corresponding to the “GOOD” grading provided by the eye-tracking software. Other categories with successively worse errors are graded as, for example, “FAIR”, “POOR”, or “FAILED”, necessitating recalibration and validation. The lab member can also check for outlier errors, which may indicate a mis-fixation at one point, or systematic error patterns that suggest a setup issue with the eye tracker. Between runs, the drift-checking procedure consists of performing a validation test with fixation at the central target only. A successful check (maximum error < 2.0^o) permits the second TMT run to proceed; otherwise, the lab member must perform calibration followed by validation until the average error is <1.0^o, and the maximum error is <2.0^o. All error values are logged for later evaluation. The standard settings of the eye-tracking system software are used to categorize the eye-tracking data into saccades and fixations. Saccades are classified by the following detection thresholds: motion 0.1^o; velocity 30^o/s; and acceleration 8,000^o/s. All other eye-tracking data are classified as fixations.

Neuroimaging

A 3-Tesla MRI system is used with a 64-channel head coil to obtain high-quality neuroimaging data. Anatomical acquisition begins with a high resolution, three-dimensional, sagittal T1-weighted magnetization-prepared rapid gradient echo (MPRAGE) sequence (repetition time/echo time/inversion time/flip angle TR/TE/TI/FA=2,500 ms/4.37 ms/1,100 ms/7^o, generalized auto-calibrating partially parallel acquisitions (GRAPPA) factor 2, 256 x 256 matrix, 192 slices, 1 mm isotropic voxels, 3 min:45 s imaging time). An indirect measurement of brain activity is then obtained by fMRI of blood oxygenation level-dependent (BOLD) signal contrast arising from neurovascular coupling²¹. For fMRI, the typical T2*-weighted BOLD acquisition uses echo-planar imaging (EPI, TR/TE/FA = 1,750 ms/30 ms/40^o, slice acceleration 2, phase acceleration 2, 80 x 80 matrix, 60 slices, 2.5 mm isotropic voxels, 165 time points, 4 min:49 s imaging time). Two such fMRI runs are conducted for TMT (described above).

Protocol

Testing and development of the experiment protocol occurred through volunteer participants who each provided their free written informed consent to participate in the study. This study has been reviewed and approved by the Research Ethics Board (REB) at the Sunnybrook Health Sciences Centre, Toronto, Canada.

1. Experimental procedure

NOTE: Steps 1–5 occur prior to participant setup on the patient table of the MRI system. The pertinent MRI system locations consist of the console area, the magnet room, and the adjacent equipment room. Console area computers and connections at the penetration panel are shown in Figure 5.

1. General setup
   NOTE: The protocol is described for the specific MRI system and laboratory environment used by the co-authors at Sunnybrook Research Institute. Protocol variations may be required for other MRI systems and environments. See the Table of Materials for a full list of hardware and software. Different versions of the touch-sensitive tablet have been made available to researchers based on their local site conditions.
   1. Prepare the tablet for visual feedback of hand position (VFHP).
   2. Ensure that the tablet is securely attached to its frame and that the MRI-compatible tablet video camera is attached.
   3. Apply fresh blue tape to the tablet surface, ensuring that the entire touch surface area is covered, with no major creases that may interfere with drawing or skew the calibration. Remove excess tape from the edges of the tablet surface. 
2. Tablet system setup (console area)
   1. At the equipment-room side (ERS) of the radiofrequency (RF) penetration panel, plug in the tablet video camera power adapter and connect it to the camera filter box. 
   2. Connect the Bayonet Nut Coupling (BNC) video cable from the filter box to the hand video input of the Tablet Video Camera computer.
   3. Connect a 9-pin D-subminiature connector (DB9) extension cable from the tablet interface box to the filter on the ERS of the RF penetration panel.
   4. Once the Stimulus/Response and Tablet Video Camera computers are running, connect the two universal serial bus (USB) cables from the interface box to the Stimulus/Response computer, and connect the tablet interface box to power.
   5. Use a high-definition multimedia interface (HDMI) cable to connect the Stimulus/Response computer display output to the Tablet Video Camera computer stimulus/response video input.
   6. To send the processed tablet video camera display to the fMRI Projection System, connect a video graphics array (VGA) cable between the two devices. Switch on the MRI-compatible Projector.
   7. Connect the USB response box (URB) BNC to the MRI trigger pulse output system. Plug the USB end of the cable into the Stimulus/Response computer just prior to starting the fMRI experiment.
3. Tablet system setup (magnet room)
   1. Bring the tablet, stylus, tablet link (DB9), and tablet video camera link cables into the magnet room. 
   2. Connect the tablet link and tablet video camera link cables from the tablet system to the magnet-room side (MRS) of the RF penetration panel.
      NOTE: Ensure that there are no kinks or loops in any of the MRS cables, as this can potentially cause RF heating.
   3. Secure the tablet system to the patient table by sliding the MRI-compatible tablet clips into the rails of the patient table, two clips per side. 
   4. Place the MRI-compatible projector behind the back end of the magnet, approximately 1 m away from the magnet bore. Mount the MRI-compatible rear projection screen inside the magnet bore, approximately 2 m away from the projector (see Figure 4B,C).
4. Eye-tracking system setup (magnet room, without participant)
   NOTE: Detailed Long Range Mount MRI Installation instructions are provided in the Eye-tracking system (see the Table of Materials) Installation Guide. Positioning of the eye-tracking camera in the magnet room should adopt Eye-tracking system recommendations for component placement and wiring in an MRI environment, which may vary by site (Eye-tracking system Installation Guide - Long Range Mount Installation - MRI Installation pg. 47-57)²².
   1. Place the MRI-compatible eye-tracking camera inside the magnet bore, between the projector screen and the edge of the bore, so that the camera mount is flush with the outer edge of the bore. Secure the camera system to the bore by adjusting the plastic screws on the camera mount.
   2. Connect the fibre optic (FO) cable to the MRI-compatible eye-tracking camera. Route the FO cable outside to the console area through the waveguide at the console to connect to the MRI-unsafe eye camera interface.
   3. Bring the eye-tracker power cable into the magnet room, connect the DB9 end to the penetration panel filter, and connect the other cable end to the MRI-compatible eye-tracking camera and illuminator. Remove the camera lens cap.
      NOTE: The DB9 end of the power cable may be MR-unsafe; securely connect this end to the penetration panel right away once brought into the MR environment, while maintaining maximum distance from the magnet. Additionally, keep the FO cable and eye-tracker power cable away from one another and away from any other cables on the floor of the magnet room, to avoid possible entanglement and signal interference.
5. Eye-tracking system setup (console area, without participant)
   1. At the ERS of the penetration panel, connect the eye-tracker power adapter to an outlet and to the corresponding DB9 filter port.
   2. To capture the Stimulus/Response computer triggers on the eye-tracking computer, connect their parallel ports with a DB25 cable.
   3. For communication between the eye-tracking system and the Tablet Video Camera computer, connect the two via a Category-5e (CAT5e) ethernet network cable. Power up the eye-tracking computer.
6. Participant setup (within magnet room)
   1. Prepare the patient table with the 64-channel head coil and ask the participant to lie supine on the table with their head as far into the coil as possible. To prevent movement, add padding around the head for a secure fit. Use the landmark laser to verify that the head is centered within the head coil.
   2. Adjust the head coil mirror position until the participant has a clear and unobstructed view of the rear projection screen.
   3. Place the tablet mount over the waist of the participant such that the touch-sensitive surface is at a comfortable position to facilitate writing and drawing maneuvers.
   4. Place the tablet stylus in the dominant hand of the participant and ask them to hold the stylus as if they were holding a pen. Ask the participant to touch all four corners of the touch surface with the stylus, to assess comfort. Adjust the tablet position and add padding under the elbow as needed to minimize strain or obstruction.
   5. Once a comfortable position is achieved, tightly secure the tablet system to the patient bed using the Velcro straps. Slowly move the participant and tablet system into the magnet bore with care. Make sure that the tablet system does not bump the edge of the bore, and that the tablet cables do not tangle (Figure 2A).
7. Eye-tracking software setup (console area and magnet room)
   NOTE: All software setup performed on the Tablet Video Camera computer or the Stimulus/Response computer is achieved by research lab members using the appropriate keyboard strokes and mouse clicks.
   1. On the Tablet Video Camera computer, open the Video camera.exe program. While the system initializes, wait for the Settings dialog to appear, and press OK using the computer mouse.
      NOTE: At this point, the participant should be able to see full-screen video feedback of their hand position/stylus (Figure 2D).
   2. On the Tablet Video Camera computer, open the Screen Recorder program.
   3. Create a new screen capture session for the eye-tracking data from the participant using their participant ID.
   4. Follow the Eye-tracking system User Manual’s recommendations to configure pupil and corneal reflection thresholds, and to Calibrate and Validate the eye-tracking camera (Eye-tracking system User Manual - Tutorial: Running an Experiment pg. 81 - 91)²³.
      1. Adjust the eye-tracking camera view of the right eye of the participant by toggling between different camera views, focusing the lens, and adjusting the illuminator.
      2. Once acceptable pupil threshold and corneal reflection (CR) values are configured, record the values, and proceed with 9-point calibration (press C).
      3. Validate the calibration (press V). Record the average and maximum validation angle values before proceeding to the fMRI experiment. If suboptimal calibration results are achieved (FAIR or POOR), repeat calibration/validation until GOOD results are achieved, corresponding to an average error of <0.5^o and a maximum error of <1.0^o (Figure 4D,E).
8. Tablet calibration
   1. Use the Stimulus/Response computer to calibrate the tablet touch surface.
   2. Open ELO 3-point calibration to begin tablet calibration.
   3. Instruct the participant to use the stylus to touch and release the three targets that appear on the screen, consecutively, within the time limits.
   4. Once calibration is complete, open the referenced graphics editing application (see the Table of Materials) and instruct the participant to draw freely to confirm that the stylus is tracking properly. Repeat steps 8.1–8.4 as needed.
      NOTE: Frequent jerks or jumps in tablet response graphics suggest that the stylus is not tracking well and requires recalibration.
9. Training protocol
   1. To familiarize the participant with writing and drawing on the tablet interface, ask them to follow the guiding prompts through a self-paced training task from an essential tremor study²⁴. This includes the participant signing their name and performing the Fahn-Tolosa-Marin Tremor task, which consists of drawing spiral and horizontal lines between increasingly narrow guidelines.
   2. To familiarize the participant with the TMT, guide them through a self-paced training task consisting of simplified versions of TMT-A and TMT-B, with only 12 items. Following this training, guide them through full-sized, alternate versions of TMT-A and TMT-B, with the items rearranged, using the same timing as the experimental task. Monitor participant performance to ensure that the tablet remains well calibrated and that the participant is executing the TMT task according to the prompts.
10. Experimental paradigm
    NOTE: This workflow implements the TMT block design described above.
    1. Start the Eye tracker recording. On the Tablet Video Camera computer, select Start Recording in the Screen Recorder program.
    2. On the Stimulus/Response computer, open the TMT-Run1_slow.ebs2 E-Prime (E-Run) script file.
    3. Make the final connection to the trigger output of the MRI system: plug the URB to the Stimulus/Response computer.
    4. Input the participant ID and the session number when prompted by the E-Run script.
    5. Give the participant verbal instructions for completing the TMT using the MRI system intercom (Figure 6). Confirm that the participant is ready to proceed.
    6. The E-Run script will present the participant with TMT instructions. Execution of the first run of TMT-A, TMT-B, and visual fixation conditions will begin once a trigger pulse is sent from the MRI system at the start of fMRI via the URB.  
    7. Monitor the eye-tracker data during the run to ensure that the signal is stable (one lab member). Additionally, monitor the TMT performance (stylus responses) of the participant to ensure that they are following the instructions given and that there are no issues with setup (i.e., unreliable video projection, poorly tracking stylus, etc.; second lab member). Have the second lab member also note the presence of any performance errors for TMT-A or TMT-B, and the trial number.
    8. Once the run has ended, stop the eye recording and perform a Drift Correction, following the recommendations of the Eye-tracking system User Manual (pg. 91-92)²³. If the drift-check results in error < 2.0°, proceed. If the error is ≥2.0, perform calibration/validation until the average error is <1.0° and the maximum error <2.0°.
    9. For Run 2, restart the eye recording session, and open the E-Run script file TMT-Run2_slow.ebs2 on the Stimulus/Response computer. Enter the same participant ID and session number as for Run 1. Repeat the task instructions (Figure 6). Again, the trigger pulse will initiate the task once fMRI has begun. As for the first TMT run, have the second lab member note the presence of any TMT performance errors.
    10. Once the experiment is complete, complete one final eye-tracking validation (step 7.4.3) and record the average and maximum error values. Then, click File | Close on the eye-tracking software to export the data. Take the participant out of the magnet, and begin equipment take-down.
11. Equipment take-down and data saving
    1. The TMT data will be automatically saved on the Stimulus/Response computer in the same folder as the TMT scripts.
    2. Eye tracking data will be saved once the recording session is closed.
    3. In the SR Research Screen Recorder program on the Tablet Video Camera computer, navigate to File and select Close – this will transfer the files from the Eye Tracking computer to the Tablet Video Camera computer.
       NOTE: Simply quitting the program window will not result in the proper transfer/saving of experimental data.
    4. Once data transfer is complete, shut off all computers and store equipment.

2. Analysis

1. Participant
   1. To demonstrate the protocol and its potential impact, tablet-based TMT, eye-tracking, and fMRI data were collected from a vounteer participant (a healthy, right-handed, 22-year-old female) with no reported history of neurological, psychological, or writing disorders.
2. Tablet kinematic metrics
   1. Analyze raw kinematic tablet data (stylus position in x,y coordinates) using custom scripts written in MATLAB available on GitHub²⁵. Raw data are processed using the custom script NPTF2F_CompleteAnalysis.m, which calls additional custom scripts: NPTF2F_RemoveErrors.m; NPTF2F_SpeedData.m; NPTF2F_SignalData.m; getAverageForce.m; getTotalDistance.m; sigfilt1.m; spikeRemoval.m; and zeroX.m. To run NPTF2F_CompleteAnalysis.m, input participant identification, the date of data collection, and pulse sequence order (EPI/INI or INI/EPI), where INI denotes inverse imaging²⁶.
      NOTE: The TMT-related fMRI data collection at the authors’ institution can be run in either imaging mode, with EPI chosen here (see Neuroimaging above). The INI fMRI acquisition records brain activity with higher temporal resolution and is beyond the scope of the present work. On running the script, analysis proceeds in multiple sections. Sections 0 and 1 populate the MATLAB Workspace and read and store data from input text files, respectively.
      1. Section 2 asks the user to input the number of Total, Correct, and Incorrect Links from visual analysis of TMT-A trial performances. Ensure that visual analysis errs on the side of leniency; if the participant did not make contact with a circle but a clear attempt was made in the direction of the circle, count the link as Correct. Similarly, if the participant ‘overshot’ a circle and made contact with a neighboring circle while rerouting the stylus to the next correct circle, do not count this as an additional (and incorrect) link.
         NOTE: The current scope of analysis only examines fully correct trials or the correct links made within a trial. Section 3 allows for the removal of linking errors in each trial. No removals are necessary in the present case because the participant made no linking errors.
      2. Wait for Section 4 to calculate statistics from trial data by calling the NPTF2F_SpeedData() function.
      3. Wait for Section 5 to call NPTF2F_SignalsData().
      4. Observe Section 6 outputting tablet kinematic data in a format suitable for further data processing (16 trials x 15 parameters).
3. Aggregate data to quantify performance features and descriptive statistics on a per-trial basis.
   1. Determine the completion time as the time taken for the participant to reach the final sequence character from the start of the TMT trial, with an upper limit set by the maximum block durations of 40 s (TMT-A trials) or 60 s (TMT-B trials).
   2. Calculate speed (pixels per second, [px/s]) as the change in x,y coordinates (as a function of stylus movement) over time. The touch panel active area is 129 mm x 97 mm, and the stimulus display area is 103 mm x 77 mm (1,024 x 768 pixels, 9.0° x 6.7° visual angle, not including the surrounding area in the live video showing the tablet and the participant’s hands).
   3. Considering the possibility of ceiling effects resulting from fixed block durations (i.e., failure to complete TMT-A or TMT-B within the maximum time duration), calculate another metric, seconds per link (SPL)¹⁵, by dividing completion time (seconds) by the number of links (correct stylus responses making connections between two items). 
      NOTE: Higher SPL values indicate slower linking performance and vice versa.
   4. Use the eye-tracking screen recording video file to confirm overall task completion and note any erroneous behavior (e.g., incorrect linking, stylus lifting).
      NOTE: The participant in this case had no erroneous TMT performance.
   5. Use the mean, first, and third quartile speed values to differentiate linking and non-linking periods for each trial, as described below.
   6. Define linking periods (speed values above the first quartile) by rapid acceleration to peak speed values followed by a deceleration of similar magnitude.
   7. Define speeds below the first quartile as non-linking periods, typified by visual search behavior prior to purposeful linking behavior.
      NOTE: These linking and non-linking behaviors, and their neural correlates, have been recently characterized in a study of tablet-based TMT performance in young adults during electroencephalography¹⁰.
   8. Use linking and non-linking periods to determine linking duration (average time spent connecting a link, [ms]), and non-linking duration (average time spent looking for the next connection, [ms]), respectively.
   9. Calculate the total distance (D) of stylus responses during a trial in pixels as another index of inter-trial variability. Calculate the average percentage extra distance traveled (EDT) for each trial, expressed as a percentage of the optimal (shortest) path.
   10. Calculate distance per link (DPL, px/link) as the average distance travelled to form one link in each trial.
   11. Calculate the average force (arbitrary units, [au]) across linking and non-linking periods only, omitting data between trials. 
4. Eye-tracking metrics
   1. View and process the eye-tracking data on a per-trial basis, using the native software for the eye-tracking system (see the Table of Materials).
   2. Proof-of-concept and potential are demonstrated for eye-tracking data time-averaged separately for entire TMT-A and TMT-B performance conditions. Parse and separate the data from the continuous data stream recorded for each run, based on the time-stamped trigger codes generated by the Stimulus/Response computer, denoting the start and end of each TMT-A and TMT-B task block within the eye-tracking EDF data files.
   3. Report descriptive statistics, including saccade count, fixation count, fixation time (ms), fixation percentage, blink count, blink rate (blinks/s), and pupil size (in arbitrary units [au]).
      NOTE: Specific definitions for each parameter are listed in Table 1. The statistics related to fixation and saccades are produced via report generators built into the software using default threshold and amplitude values.
5. Statistical reporting
   1. Given the proof-of-concept nature of the experiment, involving a single research participant, perform simple statistical testing without correction for multiple comparisons. Calculate average tablet and eye-tracking metrics for TMT-A and TMT-B across the two experiment runs (totaling four instances of each test condition).
   2. For each tablet and eye-tracking metric, use a paired two-tailed t-test to evaluate whether statistically significant differences exist between the two TMT parts (TMT-B versus TMT-A).
6. Neuroimaging data
   1. Generate proof-of-concept fMRI maps of brain activity using analysis of functional neuroimaging (AFNI) freeware²⁷, which is widely adopted in the research community.
      NOTE: A script detailing the specific imaging analysis pipeline and parameter choices is provided on GitHub²⁵. Briefly, the sequence of AFNI image processing pipeline steps to assess for brain activity at each volume element (voxel) location in the brain is as follows:
      1. Concatenate the fMRI data from the two TMT runs.
      2. Perform preprocessing steps prior to activation map generation, including voxel-wise corrections for spikes (outliers) in fMRI signal amplitude as a function of time, physiological effects relating to respiration and cardiac pulsation²⁸, image slice acquisition time, and motion.
      3. Align T1-weighted anatomical MRI data to a standard brain atlas template^29,30 with a nonlinear warping procedure.
      4. Apply the warp parameters to the fMRI data.
      5. Spatially filter the fMRI data using a 5 mm full-width at half maximum (FWHM) Gaussian kernel.
      6. Divide the fMRI time-course at each voxel by the mean value and then multiply by 100 to rescale the fMRI signals to percentage units.
      7. Enter the fMRI data into a general linear model (GLM) including boxcar waveforms representing active times during TMT-A and TMT-B task blocks (derived from tablet data) convolved with a canonical hemodynamic response function, plus regressors for low frequency fluctuations, motion and motion derivatives, and physiological regressors to remove residual effects of the cardiac and respiratory cycles.
      8. Calculate initial maps corresponding to the brain activation (beta coefficients from the voxel-wise GLM analysis) for a) the average TMT-A plus TMT-B performance versus fixation; and b) the average TMT-B – TMT-A performance. Report each map at p < 0.0005 and then, apply a cluster size threshold to correct for multiple comparisons at p < 0.05.

Results

Using the eye-tracking screen recording file, representative performances of TMT-A and TMT-B at a single point in time in the augmented reality environment are shown in Figure 7A,B, respectively. TMT-A and TMT-B performances (blue line) and gaze data (red line) over successive 2.5 s intervals are shown in Figure 7C,D, respectively. This time interval was chosen for ease of visualizing several successive instances of linking behavior in a single graphic. A shorter time interval simply shows one link (or none), whereas a longer time interval shows more links and clutter and is more difficult to interpret visually. Upon inspecting Figure 7C,D, in particular, it is evident that for the first several seconds of performing TMT-A and TMT-B, the participant visually searches and encodes for the first links to make before they move the stylus. There are also indications that throughout TMT-A and TMT-B performance for the time intervals shown, gaze (and visual searching behavior) precedes the appropriate stylus linking movements.

Table 1 summarizes the participant's average kinematic and eye-tracking metrics for TMT performance across all trials (four instances of TMT-A, four instances of TMT-B, across two separate runs). Completion times for TMT-B (31.3 s ± 6.0 s) trended greater than for TMT-A (24.0 s ± 5.7 s) (p = 0.06). This is consistent with the more complex mental processing that is required to perform TMT-B. The average speed of link drawing performance was not significantly slower for TMT-A (0.35 ± 0.04 px/ms) than for TMT-B (0.36 ± 0.13 px/ms) (p = 0.91), whereas SPL trended greater for TMT-B (1.31 ± 0.25 s) than for TMT-A (1.00 ± 0.24 s) (p = 0.06). Average linking period durations were not significantly different (702 ± 299 ms (TMT-B) and 729 ± 221 ms (TMT-A) (p = 0.92)), nor were non-linking period durations (576 ± 451 ms (TMT-B) and 260 ± 29 ms (TMT-A) (p = 0.23)). Total distance (D) was not significantly different for TMT-B (10,300 ± 1,270 px) compared to TMT-A (10,600 ± 1,930 px) (p = 0.52). The percent extra distance traveled (EDT) relative to the shortest possible distance was 27.1 ± 7.1% for TMT-A and 24.2 ± 6.3% for TMT-B (p = 0.59). Distance per link (DPL) for TMT-A was 442 ± 80 px/link and 429 ± 53 px/link for TMT-B (p = 0.52). Stylus force trended slightly higher on average for TMT-B (9.3 ± 1.8) than for TMT-A (5.5 ± 3.5) (p =0.11). No errors were made during either task condition. Collectively, these results are consistent with the interpretation that there is significant variation in motor performance over both TMT-A and TMT-B, such that any possible differences between the two TMT parts due to cognitive complexity in average speed of link drawing, linking period duration, non-linking period, D, EDT, DPL and stylus force are obscured in analysis at the single-participant level by the pseudo-random presentation of the stimuli on the display. As expected, however, the trend in greater SPL for TMT-B compared to that for TMT-A agrees well with the findings for completion time, reflecting the strong correlation between the two metrics.

The eye-tracking data demonstrated a trend toward slightly more saccades in TMT-B (90 ± 24) than in TMT-A (71 ± 22) (p = 0.10). The analogous results for fixations were almost identical, given that saccades and fixations are strongly interrelated. The average fixation time in TMT-A was 308 ± 40 ms, whereas the average fixation time in TMT-B was 314 ± 32 ms (p = 0.32). The average percentage of time spent in a fixation (fixation %) for TMT-A was 90.0 ± 2.3%, significantly different from the value of 88.7 ± 2.1% for TMT-B (p = 0.01). The blink count per trial was significantly higher in TMT-B (5.0 ± 2.6) than in TMT-A (2.0 ± 1.2) (p = 0.04). When accounting for the difference in average completion time between tests, blink rate was still significantly larger for TMT-B (0.15 ± 0.06 blinks/s) compared to TMT-A (0.08 ± 0.05 blinks/s) (p = 0.03), as might be expected for the former task, as it is more cognitively demanding. The average pupil size remained very similar across conditions (1,588 ± 140 for TMT-A; 1,648 ± 59 for TMT-B) (p = 0.29).

When analyzing brain activity during both task conditions (TMT-A and TMT-B versus visual fixation), significant widespread positive activation was observed, along with several negatively activated clusters (which tended to be smaller). The top 25 clusters by size included positive activation in portions of medial and lateral cerebellum, left precuneus, superior and inferior parietal lobules, left middle occipital gyrus, precentral gyri, left postcentral gyrus, left superior frontal gyrus, right superior occipital gyrus, supplementary motor areas, left middle cingulate cortex, right supramarginal gyrus, left middle frontal gyrus, and right calcarine gyrus. A subset of these activations is shown in representative images in Figure 8. Negative activation was present in the angular gyri, left superior frontal gyrus, middle temporal gyrus, right inferior parietal gyrus, right superior temporal gyrus, right postcentral gyrus, right supramarginal gyrus, left inferior frontal gyrus (pars orbitalis), right paracentral lobule, and right precentral gyrus. For the TMT-B versus TMT-A contrast, however, no significant positive or negative activations were observed. As mentioned in the Discussion (see below), these collective fMRI observations are consistent with previous fMRI results obtained in the laboratory.

Figure 1: Conceptual diagram of the experimental apparatus. (A) The computer monitor is used to control the apparatus and cognitive test administration, and to visualize results, as performed by (B) the computer. Power, control, and data recording cables pass through the (C) radiofrequency penetration panel. The key apparatus includes the (D) computerized MRI-compatible tablet consisting of a touch-sensitive surface and stylus, light-emitting diode illuminator, and "Tablet Video camera" video camera capturing hand and stylus movements. (E) A reflective mirror mounted on the head coil enables eye-tracking of the participant, lying on (F) the patient table of the MRI system, using (G) a remote video-recording system. The mirror also enables the participant to view test stimuli, tablet responses, and associated hand/stylus movements on (H) a rear projection screen as presented by (I) a remote MRI-compatible projection system. Please click here to view a larger version of this figure.

Figure 2: Tablet setup. (A) Tablet layout on the patient table with a volunteer participant. (B) Close-up of the tablet, mount, and stylus (yellow) in two different orientations showing the arrangement of the "tablet video camera" and light-emitting diode illuminator. (C) Tablet Video Camera and Stimulus/Response computers for controlling the tablet system, from the MRI console area. (D) Representative view of the augmented reality environment while a participant performs TMT-A. The red dot indicates the instantaneous gaze position and is not shown to the participant. (E) MRI-compatible projection system for presentation of an augmented reality environment to the participant, on the rear projection screen. The screen is mounted in the magnet bore and is not seen in this view; see Figure 4 for a clear depiction. Abbreviation: TMT = Trail-Making test. Please click here to view a larger version of this figure.

Figure 3: Trail-Making Test schematic. Timing diagram of TMT administration during fMRI. Top: Timing diagram indicating the duration of TMT-A, TMT-B, and fixation blocks, administered in each of two runs. Bottom: Sample image displays of each condition. Note that trials of TMT-A and TMT-B involve different stimulus patterns for each trial so that participants do not perform based on spatial memory. All visual fixation trials involve the same image display. Abbreviations: TMT = Trail-Making Test; fMRI = functional MRI; TMT-A = part A; TMT-B = part B; Fix = visual fixation. Please click here to view a larger version of this figure.

Figure 4: Eye-tracker setup. (A) Image of MRI-compatible eye video camera, illuminator, and mount. (B) Image from the front magnet bore opening, showing the spatial relationship of the eye-tracking apparatus to the tablet, head coil and mirror, and projection screen. (C) Image from the front magnet bore opening with tablet and head coil removed, showing the relationship between the projector and projection screen used with the tablet, and the eye-tracking camera and illuminator. (D) Eye-tracker software environment showing video recording of a participant in a large field of view and a cropped and zoomed field where corneal reflections are detected to enable eye tracking, and the pupil is detected for pupil diameter recording. (E) Example screenshots during eye-tracker calibration and validation. Please click here to view a larger version of this figure.

Figure 5: Console and penetration panel setup. (A) MRI console area showing the four monitors used in the experiments. From left to right: eye-tracker; tablet video camera; tablet stimulus/response; and the MRI system console. (B) Image of the magnet-room side of the penetration panel showing all pertinent hardware connections. (C) Analogous connections on the equipment-room side. Abbreviation: BNC = bayonet nut coupling. Please click here to view a larger version of this figure.

Figure 6: Verbal task instructions. The training task involves participants using a tablet and stylus to practice drawing a smooth line between guidelines, familiarizing them with the device before the cognitive test. The TMT consists of two parts: Part A requires connecting numbered circles in ascending order, while Part B alternates between numbers and letters in ascending order. Abbreviation: TMT = Trail-Making Test. Please click here to view a larger version of this figure.

Figure 7: TMT performance. Time samples of (A) TMT-A performance and (B) TMT-B performance in augmented reality from the perspective of the participant. The red dot in each image represents the gaze point. The images are frames from the eye-tracker screen recording video file; note that the participant cannot see the gaze point during test performance. (C,D) Successive 2.5 s time intervals of TMT-A and TMT-B performance (blue lines), including time-dependent gaze data (red lines), respectively. Saccades are evident as thin red lines, whereas "knots" are also evident where the gaze does not move rapidly, indicating fixations. Abbreviation: TMT = Trail-Making Test. Please click here to view a larger version of this figure.

Figure 8: fMRI activation maps. Activation (fMRI signal contrast) for (TMT-A and TMT-B) versus fixation. Slice positions are 14 mm apart at the indicated Z coordinates in stereotactic atlas space. The color bar represents the percentage BOLD signal contrast in significantly activated areas, with positive values demonstrating higher than baseline activation. Abbreviations: fMRI = functional MRI; TMT = Trail-Making Test; L = Left; R = Right; BOLD = blood oxygenation level-dependent. Please click here to view a larger version of this figure.

PARAMETER	DEFINITION	TMT A	TMT B	P-VALUE (2-TAILED,  PAIRED)
Completion Time (s)	Average time (in seconds) taken to complete each trial	24.0 (± 5.7)	31.3 (± 6.0)	0.06
Speed (px/ms)	Average speed (in pixels per millisecond) of the stylus' movement throughout each trial	0.35 (± 0.04)	0.36 (± 0.13)	0.91
Seconds per Link, SPL (s/Link)	Average time (in seconds) taken to complete each link in each trial	1.00 (± 0.24)	1.31 (± 0.25)	0.06
Linking Duration (ms)	Average time (in milliseconds) spent connecting each link throughout each trial	729 (± 221)	702 (± 299)	0.92
Non-Linking Duration (ms)	Average time (in milliseconds) spent looking for the next connection throughout each trial	260 (± 29)	576 (± 451)	0.23
Total Distance (px)	Average distance (in pixels) the stylus travelled in each trial	10600 (± 1930)	10300 (± 1270)	0.52
Extra Distance Travelled, EDT (%)	The average extra distance travelled for each trial, expressed as a percentage of the optimal (shortest) path possible	27.1 (± 7.1)	24.2 (± 6.3)	0.59
Distance per Link, DPL (px/Link)	The average distance (in pixels) travelled to form one link in each trial	441 (± 80)	429 (± 53)	0.52
Force (abritrary units)	Average force (in arbitrary units) exerted on the tablet screen in each trial	5.5 (± 3.5)	9.3 (± 1.8)	0.11
Saccade Count	Average number of saccades in each trial	71 (± 22)	90 (± 24)	0.10
Fixation Count	Average number of fixations in each trial	71 (± 22)	90 (± 24)	0.09
Fixation Time (ms)	Average time (in milliseconds) of each fixation in each trial	308 (± 40)	315 (± 32)	0.32
Fixation Percentage (%)	Average percentage of time spent in a fixation throughout each trial	90.0 (± 2.3)	88.7 (± 2.1)	0.01
Blink Count	Average number of blink in each trial	2.0 (± 1.2)	5.0 (± 2.6)	0.04
Blink Rate (Blinks/s)	Average number of blinks performed per second throughout each trial	0.08 (± 0.05)	0.15 (± 0.06)	0.03
Pupil Size (abritrary units)	Average pupil size throughout each trial	1588 (± 140)	1648 (± 59)	0.29

Table 1: Summary statistics for tablet kinematic metrics and eye-tracking metrics, tabulated for performance of TMT-A and TMT-B by a young healthy adult female. Definitions of each metric are given with standard deviations shown in brackets. Metrics shown in italics involve averaging across linking performance in each trial and then subsequently averaging over all trials, for TMT-A and for TMT-B, respectively. P values are listed for two-tailed, paired t-testing of differences in metric values between TMT-A and TMT-B. P values shown in bold indicate significant effects for two-tailed testing at p < 0.05. Italics = average of averages for each test. Bolded = Passes two-Tailed, Paired test.

Discussion

The present work showcases a comprehensive protocol for simultaneously acquiring eye-tracking and fMRI data during the performance of tablet-based ToC. The following discussion first evaluates various aspects of the protocol, then focuses on the results shown for a representative participant. Future applications of the protocol are also mentioned throughout.

The protocol was carefully designed over multiple years based on lengthy experience gained in developing the tablet system and in conducting fMRI research involving either the tablet or eye tracking (but without combining the latter two components). In particular, all calibration-related steps ensure that the data obtained accurately reflect participant performance. Tablet calibration at the start of the session ensures that the stylus (and cursor) track writing and drawing behavior accurately in the augmented reality display, despite any change in the camera view that may have occurred during handling. To ensure that head motion does not significantly confound results, eye-tracking calibration and drift correction are implemented and validated based on the manufacturer's recommendations and available system software, in addition to continuous monitoring of the eye-tracking datastream throughout trials. Improper or omitted calibration for either the tablet or the eye-tracking system may generate biased results. The tablet and eye-tracking results presented here, however, and others generated in the lab, suggest that data of excellent quality can be obtained in healthy adults. Extra data processing may be required in the future in the case of other study populations, such as the elderly or patients with neurological or psychiatric conditions. For example, data may need to be excluded from analysis due to intermittent periods of excessive head motion (as determined from the motion estimates obtained in section 2.6.1.2 of the protocol). Initial portions of data in the first run may also need to be excluded due to learning or habituation effects (persistent even after initial training), although their timecourse would also be interesting to characterize in future research, and may provide an additional mechanism to distinguish TMT performance in these populations from that of young healthy adults.

Trigger pulses are important to the protocol, enabling time-synchronized recording of the tablet, eye-tracking, and fMRI datastreams. Whereas the fMRI signal is based on BOLD hemodynamic responses, which typically vary on the timescale of seconds, the eye-tracking and tablet kinematic data show meaningful content in the 10-100 ms range. Time-synchronization of the collective data set thus provides a unique opportunity to study mechanisms of perception, cognition, and action during TMT performance with unprecedented temporal detail. Initial investigations could characterize the association between brain activity in specific brain regions and eye-tracking parameters temporally averaged over TMT-A and TMT-B trials. For a group of participants, this would enable investigation of possible associations between the activity of a given brain region and each eye-tracking parameter, using simple linear regression and calculation of correlation coefficients. Exploring whether additional features of spatiotemporal activation can be resolved in the fMRI data using the rapid fluctuations in the tablet and eye-tracking data is also of interest in the future. Emerging work is showing that the fMRI data acquisition parameters can be adjusted to measure BOLD signals with much finer sampling; for example, a sampling period of 100 ms with INI fMRI has resulted in improved detection of brain activation dynamics³¹. Recent work investigating the tablet-based TMT using EEG has also shown that intra-task linking and non-linking periods are associated with different spatial patterns of frequency band power¹⁰, motivating the use of the protocol to search for similar fMRI signal associations. Recognizing the hemodynamic response underlying fMRI signals is much slower than the time scale of saccades and fixations, however, first steps in this direction will likely involve characterizing potential differences in TMT-A and TMT-B performance involving behavior that occurs early versus late in the linking sequence (with the latter particularly challenging in TMT-B); and potential differences for links that are challenging to perform versus those that are less challenging, based on visual inspection of the eye-tracking and kinematic data.

The protocol includes a training module that enables participants to become familiar with making tablet-based responses and performing the linking movements necessary to perform the TMT. Such training (including future modifications tailored to other tasks or other ToC under study) is designed to develop proficiency in those who have had little experience interacting with computer tablets, such as some elderly individuals, and those who may face challenges in this communication mode due to brain dysfunction. The augmented reality environment, including VFHP from the tablet video camera video feed, enables tablet interactions with high ecological validity but does not provide an experience that is completely identical to typical writing and drawing using a pen and paper. For example, the participant must make their responses while lying down in the magnet and viewing computer graphics, including a disembodied presentation of their hand without the normal, natural proprioceptive input and body-centered spatial coordinates. Whereas future studies can be contemplated that explore the consequences of manipulating the latter two factors, the present anecdotal evidence suggests that with simple training, healthy individuals rapidly and easily become proficient at using this tablet technology, such that learning effects in tablet-based fMRI studies can be neglected after a short training module.

The current protocol can be used in the future, with fMRI performed during the training module, to provide quantitative scientific evidence in support or against the latter statement. (In previous tablet-based fMRI studies of the TMT that did not include the training, neuroimaging data from the first trial of TMT-A and TMT-B were discarded to avoid learning effects^10,19.) It will also be interesting to explore the tablet-based and ToC learning effects in various patient populations (such as those with cognitive impairment), which may require the training module to be enhanced. In other investigations outside of the magnet, the training module could also be adapted to serve as a useful screening tool, permitting patient participants who are non-compliant with instructions or who are otherwise unable to perform tasks adequately to be excluded from imaging studies.

As a last point of discussion related to the training task, it is important to note that functional neuroimaging of ToC is usually limited by the noisy nature of brain activation signals and the need to analyze lengthy time series data over multiple task repetitions to obtain statistically significant brain activation maps³². This procedure is at odds with the typical presentation of ToC, in which the test is administered once. As the capabilities of functional neuroimaging modalities improve in the future (e.g., by performing fMRI at ultrahigh magnetic fields of 7 T and above), it may be possible to compare brain activation from a single-trial test of cognition to that obtained from multiple trials. However, at present, it has been shown that multi-trial tablet-based TMT performance has reasonable convergent validity with performance of the actual pen-and-paper test¹⁵.

Although designed to facilitate evaluations of ToC with fMRI, the protocol is inherently flexible and modifiable to accommodate wide-ranging research goals. For example, the tablet video camera was specifically added to enable VFHP to bolster ecological validity, but may be excluded if it is not required, or turned on and off for different task conditions (such as in studies exploring the integration between visual, proprioceptive, and motor processing). Additionally, the tablet can easily be used synchronously with the eye-tracking system in a non-MRI environment solely for behavioral testing, or with other functional neuroimaging modalities such as EEG, functional near-infrared spectroscopy, and positron emission tomography. Hardware modifications might be required in the case of studies involving magnetoencephalography (MEG), to suppress the magnetic fringe field of the tablet to well below femtoTesla at the MEG magnetic field sensors. Depending on experimental needs, the protocol can also be augmented to include other sensory stimulus presentation and response recording equipment. For example, this could include MRI-compatible headphones to present auditory stimuli, and button boxes to log button-press responses, ultimately permitting the brain activation signals from arbitrary ToC to be compared to those generated by block or event-related design tasks more typically adopted by the functional neuroimaging community. Other protocol changes could be made to account for motor or visual impairment in various patient populations. For example, additional control tasks could be added that include simple drawing movements (such as repetitively linking two stimuli together with much less cognitive demand), enabling the contribution of motor impairment to overall TMT performance to be estimated (i.e., by examining the brain activation contrasts (TMT-A versus rest; simple drawing versus rest; TMT-A versus simple drawing; and similarly for TMT-B). The number of required linkages in TMT-A and TMT-B could be lowered to reduce the possibility of muscle fatigue. Visual impairment could be accommodated by presenting larger visual stimuli or stimuli with stronger display contrast. However, additional fMRI of control groups would have to be undertaken with such modifications to provide an unbiased assessment of the brain activity of patients versus controls.

Despite its robustness, the protocol could undergo several improvements. In particular, it is somewhat labor-intensive to execute: the use of three or more lab personnel (including one technologist to operate the MRI system) is desirable to achieve high efficiency during equipment set-up and take-down, and during data collection (one individual to monitor the tablet computers and one to monitor the eye-tracking computer). With two trained staff at our site, 10 min before and after MRI are currently required for setup and takedown, although these times could be shortened by involving another lab member to assist. In the future, a time-efficiency gain could be achieved by "preconfiguring" certain hardware components and making more efficient use of equipment carts for easier transport and establishing cable connections. Permanent installation (partial or full) in the MRI suite would be the easiest option if space and equipment availability permit.

Next, the protocol was demonstrated by obtaining the representative tablet, eye-tracking, and fMRI results from one young healthy adult volunteer. The results largely met with expectations, as described below, but at the outset, it should be emphasized that the values obtained for the various behavioral metrics and activated brain areas have been statistically evaluated at the within-participant level and do not account for the mean and variability at the group level. Future multi-modal testing of a large cohort of healthy individuals will be required to obtain group-level information as “normative” data, which ultimately can be compared to the results obtained from analogous testing of patient populations with brain dysfunction. Sample-size calculations for such studies will likely be driven by the low contrast-to-noise ratio of fMRI signals, as well as the cost of acquiring such data. Some tools are available in the scientific literature for fMRI sample size estimation³². With this proviso, the present narrative primarily focuses on briefly interpreting the trends and significant effects observed.

The participant demonstrated slightly greater completion time and greater non-linking duration for TMT-B compared to TMT-A, replicating previous tablet-based findings and consistent with established TMT performance on paper^2,18,33. These findings may reflect the need for more time to process, search, and identify the next correct target in TMT-B versus TMT-A, considering that TMT-B is thought to be more mentally challenging. No errors were recorded for either task condition, and all TMT trials were completed within the allocated time, consistent with standard TMT completion by young, educated, healthy adults². The SPL value was greater for TMT-B than TMT-A as expected, given that both TMT-B and -A have the same number of total links, and the TMT-B completion time was longer. Despite increased visual search complexity in TMT-B, slightly greater D and EDT values were observed in TMT-A. Both metrics were newly developed for the present work, so no specific comparisons can be made to reports in previous tablet-based TMT literature. However, it is speculated that the slower performance in TMT-B might have altered the position of the individual on the “speed-accuracy” trade-off³⁴ plot in relation to their faster performance in TMT-A—thus leading to more accurate linking with associated decreased D and EDT values. This interpretation needs to be confirmed in future testing.

The eye-tracking metric results for this participant are intriguing. A slightly greater number of saccades and fixations, blink count, and blink rate were found when the participant performed TMT-B compared to TMT-A. Greater saccade and fixation counts may indicate increased visual search effects across the visual stimuli in condition B. Supporting this possibility, previous work has shown that both counts increase as the mental cost increases to process a more complex search array³⁵. The increased blink count and blink rate for TMT-B compared to those for TMT-A may represent increased cognitive control for the former task condition. Interestingly, many studies support that the rate of spontaneous eye blinks (and blink count within a fixed trial duration, as studied here) are useful proxies of dopamine activity³⁶. Dopamine is an important neurotransmitter involved in learning, working memory, and goal-oriented behavior³⁷ all of which underpin successful TMT performance and are required to a greater extent in TMT-B compared to TMT-A. Numerous studies investigating spontaneous and task-evoked eye-blinks show that both metrics are sensitive to modulations of cognitive control³⁸. Last, a very similar average pupil size was observed for both TMT parts, suggesting that the participant was able to perform both parts with similar levels of mental effort without straining their processing capacity³⁸. These interpretations are again consistent with the literature on TMT performance² and that the participant performed both parts efficiently without errors. Future work will be required to investigate the detailed gaze characteristics associated with intra-task TMT behavior. Such work will be extremely interesting, providing means to evaluate the extent that visual search behaviors a) precede tablet responses; b) are altered for links that are challenging to perform versus those that are easy to perform due to the spatial distribution of number and letter stimuli, and c) are altered when TMT performance errors are made.

Regarding the topic of errors in TMT performance, error logging and quantification will be an important aspect of future research that falls outside the present proof-of-concept study of a high-performing young healthy adult. The present protocol is limited to the logging of TMT performance errors at the time of data acquisition but can easily be augmented to include the number of errors made for TMT-A and TMT-B trials, as well as statistical measures of central tendency and variation for a given participant, based on manual evaluation of digitized video files of stylus interactions. Beyond this, a rubric is required for the categorization of types of TMT performance errors. Once sufficient error data are accumulated by manual inspection, it should also be possible to develop artificial intelligence methods to detect and classify errors accurately, making the process of evaluating errors much less time-intensive.

Neuroimaging analysis revealed significant widespread activation (for both TMT-A and TMT-B tasks analyzed together versus the rest condition) in regions of the brain, including those responsible for visual processing, motor function, and sensory perception and integration. Activation of these regions resembles fMRI activation seen in previous TMT neuroimaging studies^15,19. As a simple example of activation associated with motor function, the contralateral (left) pre-central gyrus hand region was positively activated by the right-handed motor response, and there was also a small cluster of ipsilateral negative activation (not shown in Figure 8), characteristic activation patterns for primary sensorimotor regions during task-relevant movement^39,40. Even with a relatively conservative threshold and correction, the strength of the fMRI activation for this participant suggests that the task is a good probe of visuo-motor function, including in the cerebellum and the midbrain. However, specific conclusions about the brain regions supporting TMT performance should not be drawn from the data for this single participant, which is included only for demonstration. Note also that the lack of observed activity for the TMT-B versus TMT-A contrast was not surprising for a single participant. This particular contrast is known to be "weak", typically requiring analysis of fMRI data from a larger sample group as well as a carefully optimized image-processing pipeline for reliable detection of activation signals⁴¹. These latter points emphasize again that the present neuroimaging work demonstrates proof of concept in experimental design, fMRI recording, and analysis, but future studies will be required involving one or more groups of participants (e.g., individuals with neurological disease and healthy controls) to obtain results that are generalizable at the population level.

It is important to emphasize that the metrics developed for this protocol (to quantify TMT-related tablet and eye-tracking responses, and brain activation during fMRI) are not exhaustive. Instead, they build off experience conducting tablet-based TMT-fMRI studies and fMRI studies involving eye-tracking over the past years. The tablet and eye-tracking metrics are not necessarily independent and may have certain co-dependencies, suggesting that a multivariate analysis of their association with the TMT-fMRI data would be beneficial, for example, using the method of partial least squares⁴². In the future, new metrics that quantify aspects of the gaze path would be useful, as part of characterizing intra- and inter-individual variability in correct test performance (and in errors), including between groups of healthy individuals as well as patients. The expectation is that such work will reveal substantial gains in TMT sensitivity and specificity for discriminating patients from controls using the tablet-based TMT, eye tracking, and fMRI data and associated quantitative metrics, in comparison to standard pen-and-paper TMT administration and standard TMT scoring. If this prediction is correct, there will also be opportunities to explore whether discrimination can be improved even more through various artificial intelligence approaches and through the development of completely new, modern ToC, using the insight gained from this overall research program.

In conclusion, a novel multi-modal protocol is presented for assessing human performance of ToC using computerized tablet technology, eye tracking, and fMRI. When compared to related but simpler research protocols^20,43,44,45, the present protocol is thought to be more informative due to the inclusion of tablet technology with high ecological validity coupled with eye tracking, while maintaining an ergonomic and efficient study design. The protocol provides the opportunity for seamless correlation of task performance, neural activity, and eye-movement metrics in different multivariate and machine-learning frameworks to explore the neural underpinnings of ToC. Pilot data, involving a representative young healthy adult performing the tablet-based TMT, are very promising. The protocol thus opens the door to a large program of research that includes developing a much more nuanced understanding of the neural underpinnings of ToC, as well as investigating the potential to use existing and newly developed ToC coupled with eye tracking and functional neuroimaging for much more sensitive and specific characterization of patients with different brain dysfunctions, in comparison to healthy individuals.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
3T MRI System with 64-channel head coil	Siemens Healthineers (Erlangen, GER)	Magnetom Prisma	Records fMRI data.
Electromagnetic Interference Filter	Spectrum Control Inc. (Fairview, PA, USA)	56-705-005-LI	Passes tablet and stylus signals from the magnet room to the tablet interface box.
Eye-tracker Software	SR Research Ltd. (Ottawa, ON, CAN)	EyeLink Explorer (version 4.3.1, 64 bit)	Enables eye-tracker data visualization and processing.
Graphics Editing Application	Microsoft Inc. (Redmond, WA, USA)	Paint	Used to familiarize participants with tablet writing and drawing.
MATLAB	MathWorks Inc.  (Natick, MA, USA)	R2022a	Used to analyze kinematic tablet data and perform statistical analyses.
MRI-compatible Eye Tracker	SR Research Ltd. (Ottawa, ON, CAN)	EyeLink 1000 Plus	Records eye-tracking data during fMRI.
MRI-compatible Projector	Avotec, Inc. (Stuart, FL, USA)	Silent Vision	Presents augmented reality visual stimuli to the participant.
MRI-compatible Tablet Components (including touch-sensitive surface, adjustable elevated support platform, force-sensitive stylus, light-emitting diode illuminator)	Not applicable	Not applicable	Custom-designed and assembled in the lab. See references 12, 13 for details.
Stimulus Presentation Software	Psychology Software Tools (Sharpsburg, PA, USA)	E-Prime, version 2.0	Software for developing and administering all tablet-based training and task implementations.
Stimulus/Response Computer	Not applicable	Not applicable	Multi-component design. See reference 13 for details.
Touch-sensitive Surface Driver Application	ELO Touch Solutions Inc. (Milpitas, CA, USA)	Single Touch Driver	Used to calibrate the touch-sensitive surface when participants perform touch-to-target tasks.
Trigger and Response Device	Rowland Institute (Cambridge, MA, USA)	Rowland USB Response Box	Used to time-synchronize tablet-based tasks, eye-tracking and fMRI data streams.
Video Camera	MRC Instruments GmbH (Heidelberg, GER)	12M-i	Records video of hand and stylus interactions on the tablet touch-sensitive surface.
Video Camera Computer	Not applicable	Not applicable	Multi-component design. See reference 13 for details.