Measuring the Switch Cost of Smartphone Use While Walking

Summary

This study design measures the task-switching cost of using a smartphone while walking. Participants undergo two experimental conditions: a control condition (walking) and a multitasking condition (texting while walking). Participants switch between these tasks and a direction determining task. EEG data as well as behavioral measures are recorded.

Abstract

This paper presents a study protocol to measure the task-switching cost of using a smartphone while walking. This method involves having participants walk on a treadmill under two experimental conditions: a control condition (i.e., simply walking) and a multitasking condition (i.e., texting while walking). During these conditions, the participants must switch between the tasks related to the experimental condition and a direction determining task. This direction task is done with a point-light walker figure, seemingly walking towards the left or the right of the participant. Performance on the direction task represents the participant’s task-switching costs. There were two performance measures: 1) correct identification of the direction and 2) response time. EEG data are recorded in order to measure the alpha oscillations and cognitive engagement occurring during the task switch. This method is limited in its ecological validity: pedestrian environments have many stimuli occurring simultaneously and competing for attention. Nonetheless, this method is appropriate for pinpointing task-switching costs. The EEG data allow the study of the underlying mechanisms in the brain that are related to differing task-switching costs. This design allows the comparison between task switching when doing one task at a time, as compared to task switching when multitasking, prior to the stimulus presentation. This allows understanding and pinpointing both the behavioral and neurophysiological impact of these two different task-switching conditions. Furthermore, by correlating the task-switching costs with the brain activity, we can learn more about what causes these behavioral effects. This protocol is an appropriate base for studying the switching cost of different smartphone uses. Different tasks, questionnaires, and other measures can be added to it in order to understand the different factors involved in the task-switching cost of smartphone use while walking.

Introduction

Because both smartphone penetration and the tendency to multitask are increasing, it is important to understand the impact smartphone use while walking has on attention. The literature has demonstrated repeatedly that task switching comes with a cost¹, including smartphone use while walking. Studies have found that using a smartphone while walking can be distracting and dangerous^2,3,4. These dangers have been linked to the attentional impairments of doing such a task^3,4,5,6,7. Due to the complex nature of the pedestrian environment, studying it in an experimental context that is ecologically valid can be problematic. Nonetheless, conducting such studies in actual pedestrian environments can come with complications of their own because many extraneous variables can come into play, and there is a risk of harm to the participant due to distractions. It is important to be able to study such a phenomenon in a relatively safe environment that remains as realistic as possible. In this article, we describe a research methodology that studies the task-switching cost of texting while walking, while both increasing the validity of the task and mitigating the potential risks involved.

When using a smartphone while walking, individuals are forced to switch from the smartphone tasks to walking and environment-related tasks. Hence, in order to study such a phenomenon, we found it pertinent to frame this method within the literature on multitasking, specifically focused on the task switching paradigm. In order to do this, the task switching paradigm was used¹, having participants switch between a pre-stimulus task and a post-stimulus task. One of the two pre-stimulus tasks involved multitasking, while the other one did not. In the post-stimulus task, participants had to respond to a stimulus whose perception is influenced by divided attention⁸. Moreover, experimental laboratory studies that try to be as ecologically valid as possible have often used virtual pedestrian environments to understand the attentional impact of smartphone use while walking^4,9. Nonetheless, in order to capture the underlying neurophysiological mechanisms, we chose to focus on the specific task-switching reaction to one stimulus to minimize the number of stimuli participants had to react to. In this way, we can pinpoint more precisely the task-switching cost coming purely from switching attention away from the smartphone and towards the stimulus. With our study design, we use behavioral measures (i.e., task-switching cost) and neurophysiological data to better understand the attentional impairments found during pedestrian smartphone use.

During a task-switching experiment, participants usually performed at least two simple tasks pertaining to a set of stimuli, with each task requiring a different set of cognitive resources referred to as a “task-set”¹. When individuals are forced to switch between tasks, their mental resources need to adapt (i.e., inhibition of previous task-set and activation of the current task-set). This “task-set reconfiguration” process is believed to be the cause of the task-switching cost¹. The task-switching cost is usually determined by observing the differences in either the response time and/or the error rate between trials where participants switch between tasks and those where they do not¹⁰. In our experiment, we had three task-sets: 1) responding to a point-light walker stimulus; 2) texting on a smartphone while walking; and 3) simply walking. We compared the switch cost between two different conditions: 1) simply walking prior to responding to the stimulus, and 2) walking while texting prior to responding. In this way, we captured the cost of multitasking on a smartphone prior to switching the task and were able to directly compare it to the non-multitasking switch cost of simply walking before the appearance of the visual stimulus. Because the smartphone used in this study was of a specific brand, all participants were screened prior to the experiment to be sure they knew how to properly use the device.

In order to simulate a realistic experience representative of the pedestrian context, we decided to use a point-light walker figure as a visual stimulus, representing a human form walking with a 3.5° deviation angle towards the left or the right of the participant. This figure is made up of 15 black dots on a white background, with the dots representing the head, shoulders, hips, elbows, wrists, knees, and ankles of a human (Figure 1). This stimulus is based on biological motion, which means that it follows the pattern of movement that is typical of humans and animals¹¹. Furthermore, this stimulus is more than ecologically valid; it requires complex visual processing and attention in order to be analyzed successfully^12,13. Interestingly, Thornton et al.⁸ found that proper identification of the point-like walker’s direction is greatly impacted by divided attention, making it suitable as a performance measure when studying task-switching costs when multitasking. Participants were asked to verbally state the direction the figure was walking. The appearance of the walker was always preceded by an auditory cue that signaled its appearance on the screen.

Performance on the point-light walker task and neurophysiological data allowed us to determine the attentional impact of both conditions and help determine what caused them. Performance was measured by looking at the error rates and response times when determining the direction of the point-light walker figure. In order to understand the underlying cognitive and attentional mechanisms involved in the attentional impairments we found with the performance measure, we assessed the participants' neurophysiological data using the EEG actiCAP with 32 electrodes. EEG is an appropriate tool in terms of temporary precision, which is important when trying to see what causes poor performance at specific moments in time (e.g., the appearance of the point-light walker figure), although artefacts may be present in the data due to movements. When analyzing the EEG data, two indexes are particularly relevant: 1) alpha oscillations; and 2) cognitive engagement. Research has found that alpha oscillations may represent working memory control as well as active inhibition of task-irrelevant brain circuits^14,15,16,17. By comparing the alpha oscillations at baseline levels with those occurring with the stimulus presentation^18,19, we obtained the alpha ratio. With this ratio, we determined the event-related changes that could be underlying the attentional impairment observed when texting while walking. With regards to cognitive engagement, Pope et al.²⁰ developed an index where beta activity represents increased arousal and attention, and alpha and theta activity reflect decreases in arousal and attention^21,22. This analysis was done to determine whether increased engagement prior to the appearance of the stimulus would complicate the task set reconfiguration required in order to respond to the walker figure.

With the methodology described in this paper, we seek to grasp the underlying mechanisms that impact task-switching performance in participants engaged in multitasking episodes. The walking condition represents a non-multitasking task-switch performance that is compared to a multitasking task-switch performance (i.e., texting while walking). By measuring the roles of task-set inhibition and task-set activation, we sought to better understand the switch costs that occur when texting while walking. It is relevant to note that the original study was done in an immersive virtual environment²³ but was later replicated in an experimental room (see Figure 2) with a projector displaying the walker figure on a screen in front of the participant. Because this virtual environment is no longer available, the protocol was adapted to the current experimental room design.

Protocol

Before beginning the data collection, it is important to receive all the necessary ethical research approval for human participants. This should be done through the appropriate review boards and/or human participants review committees.

This protocol was approved and certified by the ethics board from HEC Montréal for the Tech3Lab research facility.

1. Preparation of the visual stimulus

1. Create the experimental template for the visual stimulus with a visual experiment presentation software, such as E-prime. Create one for the practice trial (six trials) and one for the experimental conditions (22 trials).
2. Open the E-prime software and go to the structure window, where the logic of the experiment can be created.
   1. Double click on SessionProc (the timeline for sequencing the order of appearance of the E-objects).
   2. Drag the TextDisplay object from the toolbox into the SessionProc line.
      1. Double click the TextDisplay object that has been inserted into the SessionProc and write the study instructions: “When you hear the auditory cue, please raise your head and indicate aloud which direction the walker is going, either towards your left or towards your right. The experiment will begin shortly.”
      2. Click on the Property Pages icon on the top of the TextDisplay window. Click on the Common tab and change the Name box setting to instructions. Click on the Duration drop down menu and select Infinite. Click on the Duration/Input tab and choose Add, select Keyboard and press OK. Click on OK again to exit the Property Pages. This ensures that the instructions stay on the screen until you press to start the experiment.
   3. In SessionProc, now drag and drop a List object into the SessionProc line (place it after the instructions). Double click on the List object. In the Procedure column write “Left-Trial”, press Enter and click YES to the pop-up window asking to create a new procedure. When the next pop-up window asks to make this the default value click NO.
   4. Double click the List object in the SessionProc line. Click on the green button named Add Attribute. Name the attribute as: correct response. Click OK.
   5. Click on the blank space in the column correct response and write L (this is to signal that this list object if for the walker going towards the left).
   6. Return to SessionProc and click on the new object that was created called Left-Trial.
   7. Go to the SessionProc and double click the Left-Trial object.
      1. Drag and drop the InLine object into the Left-Trial line and rename it.
         1. SelectITI. Double click on the InLine object and write the following code:
            Dim nRandom As Integer
            nRandom = Random (16500, 17500)
            c.SetAttrib “ITIDur”, nRandom
         2. The code presents the walker stimulus at intervals of time between 16,500 ms and 17,500 ms.
   8. Double click Left-Trial. Drag and drop a Slide object into the Left-Trial line. Rename it Waiting, this object will be a blank screen that appears between the visual stimuli for the amount of time determined by the code in step 1.2.7.
   9. Double click on the Slide object.
      1. Click on the Sub-Object called SlideText and click somewhere in the slide to place the object there.
      2. Remove the existing text from that image.
      3. Click Sub-Object Property Pages.
      4. In the Frame tab set both the width and height to 100%. Click OK.
   10. Click the Property Pages and go to the Duration/Input tab. Type in duration the following value: [ITIDur].
   11. Double click Left-Trial and drag and drop a SoundOut object onto the Left-Trial line.
       1. Double click the SoundOut object.
       2. Under Filename select the appropriate sound cue file directory.
       3. Change the Buffer size to 1,000 ms.
       4. Click OK.
   12. Go back to Left-Trial and drag and drop a Slide object into the Left-Trial line and rename it Walker Left.
       1. Double click this new object.
       2. Add a SlideMovie sub-object by clicking the sub-object and then clicking the slide.
       3. Click Sub-Object Property Pages and under Filename select the directory of the video file of the left walker.
       4. Set Stop After Mode to OffsetTime.
       5. Click on Stretch and chose YES.
       6. Set End Movie Action to Terminate.
       7. Click on the Frame tab and set the width and the height to 100%.
       8. For the Position, set both X and Y position to 50%.
       9. Finally, set the Border Colour to white.
       10. Click OK.
       11. Click on the Property Pages of the Slide object.
           1. Click on the Duration/Input tab.
           2. Set the Duration to 4,000. Set PreRelease to 500.
           3. Click OK.
   13. Repeat this whole procedure (i.e., from steps 1.2.3–1.2.9) for the right trial. Name the procedure Right-Trial. When following the procedure, change only the correct response (i.e., to R instead or L) and the video file. Use the video file directory for the right walker.
3. Double click the SessionProc.
   1. Drag and drop and Slide object to the SessionProc line
   2. Double click this object and add a SlideText Sub-Object.
   3. Write Pause as the text.
   4. Again, go into the Sub-Object Property Pages and in the Frame tab make the width and height 100%. Make the Position for X and Y 50%.
   5. Click OK.
4. Double click on the List object that was already created.
   1. Click on the Property Pages of the List object.
   2. In the Selection tab set the Order to Random and click OK.
   3. In the Weight column insert the following numbers:
      1. Practice: enter the number 3 in both the Left-Trial row and the Right-Trial row.
      2. Experiment: enter the number 11 in both the Left-Trial row and the Right-Trial row.
5. At the top of the window click the icon Generate to create an executable script file. Save it to the desktop for easy access. This is the file that will be run during the experiment.
   1. Save the practice trial as “Practice” and the experimental trials as “Experiment”.
   2. Test the script created by clicking on the Run icon.
6. In the E-studio folder, an E-run file will be created. Both the files created (one for the practice trial and one for the experimental trials) can be placed in a folder on the computer’s desktop. To run the visual experiment, simply click on the appropriate icon.
7. Once the visual stimulus’ experimental templates are created attempt to display them with the projector.
   1. With the projector settings, modify the height of the walker figure and make sure that it is centered directly in front of where the participant would be standing on the treadmill.
   2. With a measuring tape, measure the height of the walker directly on the projector screen. Calculate the distance between the screen and the eyes of a person standing on the treadmill in order for the stimulus to cover 25° of visual angle, and move the treadmill accordingly. To calculate the necessary distances, one may use the following website: http://elvers.us/perception/visualAngle/

2. Setup of the laboratory environment

1. Turn on the four recording computers, the EEG amplifier, the projector, the treadmill, the speakers, and the smartphone.
2. Set up the recording equipment.
   1. Open the synchronization software with the specific subroutine created for the study with markers at 10 s.
      1. The synchronization software sends a pulse that appears in the form of a marker and light pulse in the EEG and video recordings every 10 s.
   2. Turn on the video recording software. The cameras should automatically turn on as well. If not, manually turn them on.
   3. Open and set up the EEG recording software for the participant.
   4. Open the folder containing the visual stimulus executable script file made with the visual experiment presentation software.
   5. Prepare the EEG setup and materials according to the procedures suggested by the manufacturers.
   6. Delete the conversation from the previous participant from the smartphone.
   7. Place a new bottle of water next to the participant’s resting chair.

3. Participant preparation

1. Welcome the participant into room 1 and tell them briefly about the study duration and the compensation.
2. Ask participants to remove their jewelry (e.g., earrings, piercings, necklaces), glasses, smartphone, and any content in their pockets, place these in a bin, and put it in a locker.
3. Ask participants to get rid of any chewing gum they might be eating and ensure that they have eaten prior to beginning the experiment.
4. Ensure that the participant is wearing comfortable walking shoes and have them double knot their shoelaces to guarantee the participant’s safety during the experiment.
5. Have the participant read and sign the consent form.
   1. Read the following script and have the participant sit so they can read and sign the consent form:
      
      “Here is a consent form stating that you agree to participate in this study. Read it carefully and sign it. Do not hesitate if you have any questions.”
6. Take the participant to the designated participant preparation room, room 3, where the EEG cap is to be set up.
7. Read the prepared script that explains the flow of the experimental process:
   
   ​"You may notice that every so often I might read a text. This is done to ensure that all participants receive identical instructions. In this study, we are interested in how people interact with a stimulus in front of them while sending text messages and walking at a moderate speed. For about 40 minutes, you will text [name of research assistant] that you met earlier with this smartphone [show the smartphone]. While you are texting you will hear a sound from time to time. This sound will be followed by an image of a walking character. Your task is to raise your head to the screen here [point to the screen] and to indicate aloud whether the character is walking towards your right OR your left. You will not be asked to do anything else. I will write down your answers. Note that in all blocks there are two choices for an answer (right and left), so it is impossible that there will, for example, only left or right as a single choice. The direction the character comes from is totally random. After dictating your answer, you simply continue to text [name of research assistant]. It is important not to turn around when you answer or if you want to talk to me because you could be destabilized and fall. Keep your head forward. I will be behind this mirror here [point to the glass] throughout the duration of the experiment. Do you have any questions?"
8. Measure the participant’s head circumference for the EEG electrode cap. For this experiment an EEG actiCap with 32 preamplified electrodes was used.
   1. Choose the appropriate size EEG cap, place it on a foam head for support, and place all the electrodes in their proper location.
   2. Remeasure the participant’s head circumference to determine the starting point of the cap by using the 10-20 reference system.
   3. Place the cap on the participant’s head starting from the front and hold it in position while pulling it backwards. Make sure the cap is placed properly.
   4. Connect the EEG cap’s cables to the EEG control box.
   5. Show the gel applicator to the participant so they can see that it is not sharp and allow them to touch it if they so desire. Read the following script:
      
      “Here is the applicator and the tip that I will use to put the gel on the EEG cap that you have on your head. You can touch it; it does not hurt. The tip is just short enough that it never touches your head.”
   6. Turn on the EEG electrode box so that all the electrode lights turn red.
   7. Activate the electrodes by first moving the hair out of the way and then applying the gel to each electrode: start with the ground electrode and then the reference electrode. Once these two electrodes turn green add the remaining electrodes.
   8. Place the gel until all the electrode sensors turn green.
   9. Measure the impedance on the control box.
   10. Disconnect the cables from the control box and connect them to the move adapter (i.e., the adapter kit that wirelessly transmits the data back to the control box).
   11. Place the adapter kit in a fanny pack and ask the participant to attach it around their waist, with the cables and the adapter kit placed towards the participant’s back.
   12. Go back to the computer room (room 4) and check the impedance of each electrode.
   13. Verify that the data quality is satisfactory by visually inspecting the signal on the EEG software’s monitor screen. If necessary, fix the problematic electrodes.
9. Take the participant into the experimental room (i.e., room 2).
10. Have the participant stand on the treadmill and attach the treadmill safety key to the participant.
11. Turn on the treadmill to a speed of 0.8 mph and have the participant walk for 2 min so they get familiarized with the speed. During these 2 min, remind the participant about the instructions:
    
    “For about 40 minutes, you will text [name of research assistant] with a smartphone. While you are texting, you will hear a sound from time to time. This sound will be followed by an image of a walking character. Your task is to raise your head to the screen at that time and indicate aloud, in your opinion, whether the character is walking towards your right or your left. You will not be asked to do anything else. I will write down your answers. After stating your answer, you simply continue to text [name of research assistant]. It is important to always give an answer. If you are not sure, tell us your best guess. Do not turn around when you give your answer or if you want to talk to me because you could be destabilized and fall. Keep your head forward. There are four parts to the experiment, two where you text [name of research assistant] while walking, and two where you are just walking. Each part lasts about 12 minutes and there is a break of 2 minutes between each part. Do you have any questions?”

4. Practice trial

1. Give the participant the smartphone.
2. Tell the participant that they will be doing a practice trial.
3. Click on the stimulus’ executable script file for the Practice trials. Enter the participant number and begin the trial.
4. Have the participant practice responding to the visual stimuli while partaking in a texting conversation with the research assistant. This practice session will last 3 min.
5. Once the session begins, follow the texting conversation script created for the study.
6. Write down the participant's answer to each stimulus appearance on a spreadsheet template.
7. After the 3 min, have the participant sit on a chair and drink some water. During this time adjust the treadmill speed to 0.4 mph.
8. Remind the participant of the study instructions.

5. Data collection

1. Setup
   1. Go to the workflow sheet to choose the condition order for the current participant. Two orders are possible: In order A, trials 1 and 3 use the texting condition, while trials 2 and 4 use the control condition. In order B, trials 1 and 3 use the control conditions and trials 2 and 4 use the texting conditions. During each trial the visual stimulus appears 22 times.
   2. Make sure that all the recording software is ready to be started in synchrony.
   3. Turn on all the recording software (e.g., EEG, video)
   4. Have the participant get back on the treadmill and slowly increase the speed back to 0.8 mph.
   5. Turn on the visual stimulus program and start running it.
   6. Read the trial’s instructions depending on the experimental condition.
      1. Run the stimulus’ executable script file for the Experiment trials. Enter the participant number and the code chosen for the specific conditions. Begin the trial.
2. Control condition
   1. Ensure that the smartphone is out of the participant’s field of vision during this task.
   2. Instruct the participant to simply walk on the treadmill and respond to the visual stimulus every time it appears by answering "left" or "right":
      
      “For this task, you will simply have to walk on the treadmill. From time to time you will hear a sound. This sound will be followed by an image of a walking character. Your task is to raise your head to the screen then and indicate aloud, in your opinion, whether the character is moving to your right or your left. You will not be asked to do anything else. I will write down your answers myself. After dictating your answer, you simply continue walking. It is important to always give an answer. If you are not sure, tell us your best guess. Do not turn around when you give your answer or if you want to talk to me because you could be destabilized and fall. Keep your head forward. Start when I give you the signal. Do you have any questions?”
   3. Signal to the participant that the trial is about to begin and start the visual stimulus trial.
   4. Write down the participant's response every time they answer to the visual stimulus. When a participant fails to respond leave the field blank.
   5. At the end of the trial, have the participant sit down and drink some water.
   6. During these breaks, keep running all the recording software and leave the treadmill on at a speed of 0.4 mph.
   7. After the break get the participant back on the treadmill and as they walk, gradually increase the speed back to 0.8 mph.
3. Texting condition
   1. While the participant is walking on the treadmill hand them the smartphone.
   2. Instruct the participant to text as they would naturally (e.g., using one hand or two hands) while walking on the treadmill and respond to the visual stimulus every time it appears by answering "left" or "right":
      
      “For this task, you will text [name of research assistant] with a smartphone. On the smartphone, open the message application. Then select the conversation saying "Hello". You will have to actively participate in a texting conversation. While you are texting you will hear a sound from time to time. This sound will be followed by an image of a walking character. Your task is to raise your head to the screen here and indicate, in your opinion, whether the character is moving to your right or your left. You will not be asked to do anything else. I will write down your answers myself. After dictating your answer, you simply continue texting. It is important to always give an answer. If you are not sure, tell us your best guess. Do not turn around when you give your answer or if you want to talk to me because you could be destabilized and fall. Keep your head forward. Start when I give you the signal. Do you have any questions?”
   3. Signal to the participant that the trial is about to begin and start the visual stimulus trial.
   4. Instruct the participant to have a texting conversation while walking on the treadmill. Instruct them to also respond to the visual stimulus every time it appears by answering "left" or "right".
   5. Have the research assistant follow the conversation script and keep the conversation going throughout the condition.
   6. Write down the participant's response every time they answer to the visual stimulus. When a participant fails to respond leave the field blank.
   7. At the end of the trial, take the smartphone from the participant and have the participant sit down and drink some water.
   8. During these breaks, keep running all the recording software and leave the treadmill on at a speed of 0.4 mph.
   9. After the break get the participant back on the treadmill and as they walk, gradually increase the speed back to 0.8 mph.

6. End of data collection

1. At the end of the experimental manipulation have the participant turn off the treadmill. Have the participant sit down and drink some water.
2. Remove the EEG cap and take the participant to a shower where they can wash their hair if they so choose.
3. Give the participant their compensation and thank them for their participation. Ensure that the participant leaves with their copy of the consent form and that they retrieve all their personal items.

Results

This study protocol was originally conducted with 54 participants, each responding to 88 direction trials. Half of those trials occurred when participants were simply walking prior to the stimulus presentation; the other half occurred when the participants were texting while walking prior to the stimulus presentation.

Behavioral results
Performance on the point-light walker’s direction represents task-switching costs, with lower performance representing higher task-switchi...

Discussion

A critical choice when using the protocol would be ensuring the quality of the neurophysiological data. There is an inherent complication to using a tool like EEG during movement, because excessive movement can create a lot of noise in the data. It is therefore important to consider, prior to the data collection, how the data will be prepared to remove as many artefacts as possible without modifying the actual signal. Nonetheless, it is still quite likely that there will be higher rates of data exclusion because particip...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
The Observer XT	Noldus		Integration and synchronization software: The Noldus Observer XT (Noldus Information Technology) is used to synchronize all behavioral, emotional and cognitive engagement data.
MediaRecorder	Noldus		Audio and video recording software
FaceReader	Noldus		Software for automatic analysis of the 6 basic facial expressions
E-Prime	Psychology Software Tools, Inc.		Software for computerized experiment design, data collection, and analysis
BrainVision Recorder	Brain Vision		Software used for recording neuro-/electrophysiological signals (EEG in this case)
Analyzer			EEG signal processing software
Qualtrics	Qualtrics		Online survey environment
Tapis Roulant	ThermoTread GT Office Treadmill
Syncbox	Noldus		Syncbox start the co-registration of EEG and gaze data by sending a Transistor-Transistor Logic (TTL) signal to the EGI amplifier and a keystroke signal to the Tobii Studio v 3.2.
Move2actiCAP	Brain Vision		Add-on for a digital wireless system for EEG
iPhone 6s	Apple
iMessage	Apple
iPad	Apple