Combining Computer Game-Based Behavioural Experiments With High-Density EEG and Infrared Gaze Tracking

Procedures for recording high-density EEG and gaze data during computer game-based cognitive tasks are described. Using a video game to present cognitive tasks enhances ecological validity without sacrificing experimental control.

This video demonstrates a procedure to obtain simultaneous gaze tracking and high density EEG recordings from subjects during performance of engaging and ecologically valid cognitive tasks embedded in a computer game. First, electrodes are affixed to the subject. The gaze tracking camera is then calibrated.

The subject plays the game while EEG and Gaze data are recorded. Event related potentials and event related spectral perturbations are obtained that are revoked by game Events. The main Advantage of this technique over existing methods, for example, stimulus presentation packages like ePrime or presentation, is that embedding experiments within a computer game relieves performance anxiety by moving away from an explicit testing context and helps subjects remain motivated and engaged with tasks throughout a long experimental session.

Encompassing a broad array of cognitive tasks. These qualities are particularly important for pediatric and clinical populations in whom attention and motivation can be obstacles to complex or lengthy experimental sessions. My name is Keith Yoder and I'm a research assistant in the Beaumont Laboratory.

Assisting me today will be David Post, a lab technician and UAUA graduate student in the Department of Human Development. We will be using a 128 channel bios, semi active two measurement system, and an iLink 1000 infrared gaze tracker from SR Research Before the combined computer game and EEG experiment can begin, the electrodes should be balanced to ensure that they have small offsets in the range of plus or minus 20 microvolts an hour before the subject arrives. For the EEG experiment, balance the electrodes by soaking them for five to 10 minutes in a salt bath consisting of one teaspoon of salt per liter of distilled water.

Do not leave the electrodes in any liquid for longer than 10 minutes at a time. After removing the electrodes from the salt bath, use a spray bottle filled with distilled water to rinse them just before the subject arrives. Switch on the gaze tracking camera converter boxes and computers for simultaneous gaze tracking and EEG, be sure to use the 16 millimeter long lens in the gaze tracking camera.

The setup uses four computers, one dedicated gaze tracking computer, one dedicated EEG acquisition computer, one dedicated stimulus presentation computer, and one computer for video acquisition and data analysis. Two converter boxes. Manipulate VGA outputs from the gaze tracking computer and the stimulus presentation computer, and send a spliced video signal to the video acquisition computer.

In this way, the video acquisition computer can record the game screen seen by the subject overlaid with a cursor representing current gaze fixation and a timestamp. This overlay video is saved to a disc file on the video acquisition computer. Now let's see the EEG Setup.

Begin setting up for the EEG experiment by measuring the circumference of the subject's. Head around the brow and inion and select an appropriately sized electrode cap. Place the cap on the subject's head ensuring that the tag is outside the cap resting on the subject's neck.

Reposition the cap until the A one electrode is centered with respect to the nasion inion and left pinna, right pin es and the cap midline lies along the midline of the subject's head. The nasion is the depression between the nose and the brow. The inion is the bony bump on the midline at the back of the head, and the pinna is the flap of cartilage extending at the anterior side of the opening of the ear canal.

When the cap is in position, apply an adhesive ring to the plastic housing of electrodes. Ex five and ex six. Align the opening of the ring with the electrode pellet.

Remove the paper backing from the adhesive ring and cover the electrode contact with conductive gel. Place EX six on the subject's right mastoid and EX five on the subject's left mastoid. Use a blunt syringe to place conductive gel in every electrode holder in the cap.

Wiggle the syringe tip to part the subject's hair. Then simultaneously press the plunger down and pull the syringe away from the head as though you were decorating a cake. Fill until the gel is flushed with the top of the plastic housing.

Be sure not to overfill. It is better to have too little gel than too much excess gel can bleed between electrode sites causing electrode bridging. If electrodes become bridged, there's nothing for it.

But to remove the cap, have the subject wash and dry their hair, and begin again with the plugins of the electrode leads hanging over the shoulder and the sensor ends in one hand gently place each electrode in its corresponding holder within the electrode cap. Grasp only the plastic housing of the electrodes, and be very careful not to crimp the wires. It is critical to not touch the electrode tips.

Contact with skin, clothing, or any material other than conductive gel will degrade the quality of electrodes. Place conductive gel on X one through X four and use adhesive rings to attach them to the subject's face. Place X one and X two about one centimeter horizontally from the subject's left and right outer canthi respectively.

Place X three and DX four about one centimeter below the middle of the subject's left and right eyes on the zygomatic bones. Gently collect the electrode leads behind the subject and loosely wrap the C-M-S-D-R-L lead around the others. To create a ponytail, place Velcro ties at the top and bottom of the ponytail to hold the leads in place and use medical tape to affix the ponytail to the back of the subject shirt.

Be certain to allow enough slack in these leads for the subject's neck to flex forward comfortably. Now prepare the GSR electrodes by applying a potassium chloride solution to each electrode. Use medical tape to affix GSR electrodes to the second and third fingers on the subject's non-dominant hand or whichever hand will not be being used during the experiment.

To signal behavioral responses with all the electrodes set up, have the subject sit in a non reclining stationary chair. In front of the stimulus presentation monitor, plug all electrodes into the converter box and check the EEG acquisition computer to make sure that the electrode offsets are within plus or minus 40 microvolts. If an electrode has a very large offset or is noisy, gently remove the electrode from the cap.

Insert more gel into the housing and replace the electrode. If a problem persists in a single electrode, or if a single electrode becomes contaminated with pulse artifact, that channel should be interpolated from surrounding electrodes. After the recording has finished, Set up gaze tracking by first affixing a target sticker above the septics eyebrow medial to the eye.

Launch the pop-up calibration application on the stimulus presentation computer. Begin a new session and change the type dropdown menu to CMD. After entering the appropriate command, press the send button to turn on event logging from the popup calibration application launch camera setup.

Position the camera so that the target sticker and subject's I are centered to auto threshold. Press the A key on the keyboard for the stimulus presentation computer. Adjust the focus by turning the camera lens until the I to be tracked is clear.

Begin calibrating the gaze tracking system to the subject by pressing C on the keyboard for the stimulus presentation computer. A dot will appear in nine different locations on the screen in random order. Subject should look at the dot when it appears, rather than anticipating the location of the next dot.

To validate the calibration, press the V key. The dots will appear again in random order. If the calibration was good, the subject gaze will map onto each validation dot.

If calibration fails, ensure that pupil and corneal reflection thresholds are appropriate. Pupil and corneal thresholds can be adjusted from the camera setup screen on the island host computer in the upper left of the screen, clicking on the pupil and corneal up and down arrows increases and decreases the thresholds respectively. Ensure that the sampling rate is set to 500 hertz by clicking on the 500 Hertz button.

Minimize the pop-up calibration application on the stimulus presentation computer and begin recording gaze data by right clicking on the iLink icon in the task bar and selecting Start recording. Start video recording on the video acquisition computer using the Black Magic Media Express software. Then start EEG recording on the EEG acquisition computer by clicking the pause button.

Now on the stimulus presentation, computer launched the game as polus is a custom built game designed to ensure ecological validity Without sacrificing experimental control in one game, subjects must rescue a cargo ship that is under attack. A go no-go task is implemented by introducing wormhole, out of which appear either a friend who helps the player or an Enemy who harms the player.Players. Also perform a dot motion coherence task by steering a spaceship Through drifting star fields during another game.

Subjects must use a radar scope to collect asteroids in a modified Posner task, the four sectors in which asteroids might appear flicker at different fundamental frequencies. Changes in the spectral content of the EEG can then assess attention based on the amplitudes of the frequencies that tag each sector. Another game asks subjects to help make a galactic treaty by identifying the emotions depicted in faces.

Each of five faces must be placed on empty spots on a gameboard to create a row or column of three faces expressing the same emotion. A fourth game implements the Sally Anne test of Theory of mind players. Watch the Captain Sally as she places his cargo on one of four planets.

The role of Anne is played by a space pirate who moves the cargo to another planet. Finally, an admiral observes both the captain and the pirate and relays information about the pirate's actions to the captain. Players must then plot intercept courses for the captain and for the pirate inferring where each will seek the cargo, and thereby implementing first order and second order theory of mind in contrasts between several conditions.

Recent results in human computer interaction point to the power of the game context to establish and to maintain motivation in tasks that otherwise might not seem engaging in this way. Multiple cognitive tasks in many repeated trials can be performed during a single experimental session without fatiguing the subject. This is especially important because in order to obtain sufficient trials for EEG averaging subjects should accumulate about a hundred trials of each stimulus condition For auditory stimuli, swap out the passive speakers on the stimulus presentation computer for powered speakers.

Then plug in and switch on the amplifier. Set the volume to level 60. This is necessary to achieve an 80 decibel SPL maximum sound level when the subject has finished playing the games.

Exit the game and stop recording data by pressing the stop button within the graphical interface. On each of the video acquisition gaze tracking and EEG acquisition computers. Turn off the EEG converter box and disconnect all of the leads from the box.

Remove the adhesive rings and place them into warm distilled water. Use a spray bottle filled with distilled water to remove any gel left on the electrodes and use warm tap water and soap to remove gel from the electrode cap. Now we'll show some representative EEG results shown.

Here is an event related spectral perturbation plot obtained from a frontal midline electrode fz. In this game, present enemy corresponds to the appearance of an enemy ship or a go condition and present friendly corresponds to the appearance of a friendly ship or a no-go condition during the no-go condition. Typically, developing children demonstrated significantly greater gamma band EEG power than children with an autism spectrum condition.

When attempting this procedure, it's important to avoid touching the electrode contacts, crimping the wire leads, or using too much gel.