Virtual Reality Experiments with Physiological Measures

Raphael P. Weibel; Jascha Grübel; Hantao Zhao; Tyler Thrash; Dario Meloni; Christoph Hölscher; Victor R. Schinazi

doi:10.3791/58318

Behavior

Virtual Reality Experiments with Physiological Measures

Published: August 29, 2018 doi: 10.3791/58318

Raphael P. Weibel¹, Jascha Grübel¹, Hantao Zhao¹, Tyler Thrash^1,2,3, Dario Meloni¹, Christoph Hölscher¹, Victor R. Schinazi¹

¹Chair of Cognitive Science, ETH Zürich, ²Geographic Information Visualization and Analysis, University of Zürich, ³Digital Society Initiative, University of Zürich

Summary

Virtual reality (VR) experiments can be difficult to implement and require meticulous planning. This protocol describes a method for the design and implementation of VR experiments that collect physiological data from human participants. The Experiments in Virtual Environments (EVE) framework is employed to accelerate this process.

Abstract

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and laboratory experiments, respectively. VR is especially useful for geographic visualizations and investigations of spatial behavior. In spatial behavior research, VR provides a platform for studying the relationship between navigation and physiological measures (e.g., skin conductance, heart rate, blood pressure). Specifically, physiological measures allow researchers to address novel questions and constrain previous theories of spatial abilities, strategies, and performance. For example, individual differences in navigation performance may be explained by the extent to which changes in arousal mediate the effects of task difficulty. However, the complexities in the design and implementation of VR experiments can distract experimenters from their primary research goals and introduce irregularities in data collection and analysis. To address these challenges, the Experiments in Virtual Environments (EVE) framework includes standardized modules such as participant training with the control interface, data collection using questionnaires, the synchronization of physiological measurements, and data storage. EVE also provides the necessary infrastructure for data management, visualization, and evaluation. The present paper describes a protocol that employs the EVE framework to conduct navigation experiments in VR with physiological sensors. The protocol lists the steps necessary for recruiting participants, attaching the physiological sensors, administering the experiment using EVE, and assessing the collected data with EVE evaluation tools. Overall, this protocol will facilitate future research by streamlining the design and implementation of VR experiments with physiological sensors.

Introduction

Understanding how individuals navigate has important implications for several fields, including cognitive science¹^,²^,³, neuroscience⁴^,⁵, and computer science⁶^,⁷. Navigation has been investigated in both real and virtual environments. One advantage of real-world experiments is that navigation does not require the mediation of a control interface and thus may produce more realistic spatial behavior. In contrast, virtual reality (VR) experiments allow for more precise measurement of behavioral (e.g., walking trajectories) and physiological (e.g., heart rate) data, as well as more experimental control (i.e., internal validity). In turn, this approach can result in simpler interpretations of the data and thus more robust theories of navigation. In addition, neuroscience can benefit from VR because researchers can investigate the neural correlates of navigation while participants are engaged in the virtual environment but cannot physically move. For computer scientists, navigation in VR requires unique developments in processing power, memory, and computer graphics in order to ensure an immersive experience. Findings from VR experiments can also be applied in architecture and cartography by informing the design of the building layouts⁸ and map features⁹ to facilitate real-world navigation. Recently, advances in VR technology combined with a dramatic decrease in its cost have led to an increase in the number of laboratories employing VR for their experimental designs. Because of this growing popularity, researchers need to consider how to streamline the implementation of VR applications and standardize the experiment workflow. This approach will help shift resources from implementation to the development of theory and extend the existing capabilities of VR.

VR setups can range from more to less realistic in terms of displays and controls. More realistic VR setups tend to require additional infrastructure such as large tracking spaces and high-resolution displays¹⁰. These systems often employ redirected walking algorithms in order to inject imperceptible rotations and translations into the visual feedback provided to users and effectively enlarge the virtual environment through which participants can move¹¹^,¹². These algorithms can be generalized in that they do not require the knowledge of environmental structure¹³ or predictive in that they assume particular paths for the user¹⁴. Although most research on redirected walking has used head-mounted displays (HMDs), some researchers employ a version of this technique with walking-in-place as part of a large projection system (e.g., CAVEs)¹⁵. While HMDs can be carried on the head of the participant, CAVE displays tend to provide a wider horizontal field of view¹⁶^,¹⁷. However, less infrastructure is needed for VR systems using desktop displays¹⁸^,¹⁹. Neuroscientific research has also employed VR systems in combination with functional magnetic resonance imaging (fMRI) during scanning²⁰, in combination with fMRI after scanning²¹^,²², and in combination with electroencephalography (EEG) during recording²³^,²⁴. Software frameworks are needed in order to coordinate the variety of displays and controls that are used for navigation research.

Research that incorporates VR and physiological data poses additional challenges such as data acquisition and synchronization. However, physiological data allows for the investigations of implicit processes that may mediate the relationship between navigation potential and spatial behavior. Indeed, the relationship between stress and navigation has been studied using desktop VR and a combination of different physiological sensors (i.e., heart rate, blood pressure, skin conductance, salivary cortisol, and alpha-amylase)²⁵^,²⁶^,²⁷^,²⁸. For example, van Gerven and colleagues²⁹ investigated the impact of stress on navigation strategy and performance using a virtual reality version of a Morris water maze task and several physiological measures (e.g., skin conductance, heart rate, blood pressure). Their results revealed that stress predicted navigation strategy in terms of landmark use (i.e., egocentric versus allocentric) but was not related to navigation performance. In general, findings from previous studies are somewhat inconsistent regarding the effect of stress on navigation performance and spatial memory. This pattern may be attributable to the separation of the stressor (e.g., the cold pressor procedure²⁶, the Star Mirror Tracing Task²⁵) from the actual navigation task, the use of simple maze-like virtual environments (e.g., virtual Morris water maze²⁶, virtual radial arm maze²⁸), and differences in methodological details (e.g., type of stressor, type of physiological data). Differences in the format of collected physiological data can also be problematic for the implementation and analysis of such studies.

The Experiments in Virtual Experiments (EVE) framework facilitates the design, implementation, and analysis of VR experiments, especially those with additional peripheral devices (e.g., eye trackers, physiological devices)³⁰. The EVE framework is freely available as an open-source project on GitHub (https://cog-ethz.github.io/EVE/). This framework is based on the popular Unity 3D game engine (https://unity3d.com/) and the MySQL database management system (https://www.mysql.com/). Researchers can use the EVE framework in order to prepare the various stages of a VR experiment, including pre- and post-study questionnaires, baseline measurements for any physiological data, training with the control interface, the main navigation task, and tests for spatial memory of the navigated environment (e.g., judgments of relative direction). Experimenters can also control the synchronization of data from different sources and at different levels of aggregation (e.g., across trials, blocks, or sessions). Data sources may be physical (i.e., connected to the user; see Table of Materials) or virtual (i.e., dependent on interactions between the participant's avatar and the virtual environment). For example, an experiment may require recording heart rate and position/orientation from the participant when that participant's avatar moves through a particular area of the virtual environment. All of this data is automatically stored in a MySQL database and evaluated with replay functions and the R package evertools (https://github.com/cog-ethz/evertools/). Evertools provides exporting functions, basic descriptive statistics, and diagnostic tools for distributions of data.

The EVE framework may be deployed with a variety of physical infrastructures and VR systems. In the present protocol, we describe one particular implementation at the NeuroLab at ETH Zürich (Figure 1). The NeuroLab is a 12 m by 6 m room containing an isolated chamber for conducting EEG experiments, a cubicle containing the VR system (2.6 m x 2.0 m), and a curtained area for attaching physiological sensors. The VR system includes a 55" ultra-high definition television display, a high-end gaming computer, a joystick control interface, and several physiological sensors (see Table of Materials). In the following sections, we describe the protocol for conducting a navigation experiment in the NeuroLab using the EVE framework and physiological sensors, present representative results from one study on stress and navigation, and discuss the opportunities and challenges associated with this system.

Subscription Required. Please recommend JoVE to your librarian.

Protocol

The following protocol was conducted in accordance with guidelines approved by the Ethics Commission of ETH Zürich as part of the proposal EK 2013-N-73.

1. Recruit and Prepare Participants

Select participants with particular demographics (e.g., age, gender, educational background) using a participant recruitment system or mailing list (e.g., UAST; http://www.uast.uzh.ch/).
Contact selected participants by e-mail. In this e-mail, remind the participants of the session time and requirements. Let the participants know that they must wear a loose-fitting top (for blood pressure monitoring), refrain from alcohol for 12 h before the experiment, and refrain from several other activities (i.e., caffeine, smoking, eating, and exercise) for 3 h before the experiment.

2. Prepare the Experiment and Physiological Devices Using EVE

Before each experimental session, start the computer, the experimenter monitor, and the testing monitor.
Ensure that the room fan, the thermometer and the humidity monitor are on.
Switch on the machine measuring the electrodermal activity (EDA) and the electrocardiography (ECG; e.g., PowerLab from ADInstruments). See Table of Materials.
Open the EDA/ECG software (EVE currently supports Labchart from ADInstruments) and create a new settings file. Select a sampling rate of 1,000 Hz and the appropriate number of channels (e.g., one for EDA and one for ECG). Save this settings file and re-save a version with a different name for each experimental session.
For the EDA electrodes, perform an open-circuit zero (i.e., without the electrodes attached to anything) to obtain a baseline measure of system conductivity.
Ensure that the control interface (e.g., joystick) is connected to the computer.
On the experimenter monitor, open the executable Unity file for the experiment.
1. Open the "Experiment Settings" menu in EVE, and enter the experiment parameters (e.g., participant ID number, physiological measurement file, experimental condition, room temperature and humidity).
2. Click "Start Experiment".

3. Experimental Procedure

Introduction and consent procedure
1. Pick up the participant at the agreed meeting location and guide him/her to the laboratory.
2. Indicate that the session will take approximately 90 min and ask the participant to store his or her watch and/or mobile phone.
3. Ask the participant to sit in the experimental chair and explain the experimental procedure according to the prepared verbal script.
4. Ask the participant to read the information sheet and sign the informed consent form.
Connection of EDA and ECG sensors
1. Clean the index finger and the ring finger of the non-dominant hand with a wet tissue without soap. Ensure that they are dry and connect the two EDA electrodes to the medial phalanges.
2. Clean the skin on the chest where the ECG electrodes will be placed with a wet cloth.
3. Place the white, black, and red electrodes on the participant's body between the ribs according to Figure 2. Place the white electrode on the upper right abdomen (UR), and the black electrode on the upper left abdomen (UL). Place the red electrode on the lower left abdomen (LL). Ensure that the three electrodes are not directly over a rib.
4. Connect the three color-coded ECG wires to the corresponding electrodes attached to the participant's body.
Pre-experiment questionnaires
1. Provide the participant with a keyboard and a mouse that will be used to answer the questionnaires (e.g., demographic questions, the first part of the Short Stress State Questionnaire, the Santa Barbara Sense of Direction Scale), and inform him or her that they will be asked a series of questions on the computer.
2. Inform the participants that they can ask the experimenter questions regarding the questionnaires at any time.
3. Close the two side walls of the cubicle while the participant is completing the questionnaires.
Preparations for physiological measurement. These steps can be conducted while the participant is completing the questionnaires.
1. Inform the participant that the experimenter will now prepare the physiological devices.
2. Check that the electrodes are attached to the correct locations.
3. Attach the blood pressure cuff to the non-dominant arm.
4. Provide instructions to the participant regarding the accurate measurement of blood pressure. Tell the participant to minimize arm and body movements, keep the blood pressure cuff at heart level, and maintain an upright posture with his or her feet flat on the floor.
5. Connect the two EDA wires to the electrodes on the fingers.
6. Turn off the light above the monitor and dim all other overhead lights to the lowest setting.
7. Hand the joystick to the participant and ensure that the mouse is off the screen of the testing monitor.
8. Zero the EDA channel in order to obtain a measure of an individual's starting level of skin conductance.
9. In the EDA/ECG software, open the "Bio Amp" dialog box. Choose the signal range in which the heart beat signal covers around one third of the preview window (5 mV in most cases).
10. Start recording with the EDA/ECG software, and check whether a signal is visible in the EDA/ECG software window on the experimenter monitor.
11. Start blood pressure recording by pressing the appropriate button in the blood pressure machine.
12. Switch to the open Unity program, and press "Start Measurements". A fixation cross should appear.
Joystick training and baseline video
1. Ask the participant to watch and follow the training video that instructs him or her how to use the joystick.
2. Ask the participants to complete the training maze in order to practice using the joystick. In this training maze, participants are instructed to follow the arrows that indicate a route and collect floating gems.
3. If the experiment includes sound, place the headphones on the participant's head.
4. Ask the participant to watch the baseline nature video without moving. This video is used to account for a baseline measurement of the participant's physiological data during the subsequent analysis.
Navigation task
1. Ensure that the participants have read the instructions regarding the to-be-completed navigation task. Inquire as to whether the participant has any questions before the navigation task begins. Tell the participant that they should not ask questions during the navigation task.
2. Ask the participant to press the trigger on the joystick when he or she is ready to begin the navigation task.
Final physiological measures and detachment of physiological sensors
1. Wait until the system has completed the final blood pressure measurement.
2. Stop recording EDA and ECG by pressing the stop button in the EDA/ECG software.
3. Remove the blood pressure cuff.
4. Remove the EDA electrodes from the participant.
5. Ask the participants not to remove the ECG electrodes until the end of the experiment.
6. Remove the joystick and headphones.
Post-experiment questionnaires
1. Provide the participant with a keyboard and a mouse for the post-experiment questionnaires (e.g., the second part of the Short Stress State Questionnaire, the Self-Assessment Manikin, the Simulator Sickness Questionnaire).
2. Inform the participants that they will be asked another series of questions on the computer and that he or she can ask questions if necessary.
End of the experimental session
1. Inform the participant that the experimental part is now finished. Thank her or him for participating in the experiment.
2. Tell the participant that he/she can now remove the ECG electrodes.
3. Pay out the participants and ask them to sign the printed receipt.
4. Ask if the participant has any questions regarding the purpose of the experiment, and escort him or her outside of the experimental room.

4. After Each Experimental Session

Open the "Evaluation" menu in EVE in order to conduct experiment diagnostics (e.g., replay trajectories), and save the physiological measurement files in the EDA/ECG software.
In the "Evaluation" menu in EVE, press the "Add Event Marker" button to mark events in the physiological measurement files. This step is critical for the analysis of the physiological data in terms of particular experimental phases.
Save the EDA/ECG file in the physiological measurement file in the EDA/ECG software.
Export the experimental data for backup using the evertools package.
Switch off the EDA/ECG machine and clean the EDA electrodes with alcohol pads.
Mark that the participant showed up in the participant recruitment system.

Subscription Required. Please recommend JoVE to your librarian.

Representative Results

From each participant in the NeuroLab, we typically collect physiological data (e.g., ECG), questionnaire data (e.g., the Santa Barbara Sense of Direction Scale or SBSOD³¹), and navigation data (e.g., paths through the virtual environment). For example, changes in heart rate (derived from ECG data) have been associated with changes in stress states in combination with other physiological³² and self-report measures³³. Our system allows for different types of questionnaires to be presented such as the Short Stress State Questionnaire³⁴ and the SBSOD³¹ . The SBSOD is a self-report measures of spatial ability that is often correlated with navigation behavior in real and virtual, large-scale, environments³⁵. In addition, navigation data can be used to infer participants' spatial decision-making (e.g., hesitations, navigation efficiency) in different stressful contexts³⁶.

A representative study investigated the effect of stress on the acquisition of spatial knowledge during navigation. We tested 60 participants (29 women and 31 men; mean age = 23.3) individually during a 90 min session. During the navigation task of each session, participants were placed into one of two groups (i.e., stress and no stress) and completed three learning and testing phases while EDA and ECG data were continuously recorded. The learning phases involved finding a set of four locations (Figure 3) with the aid of a map that could be triggered using a button on the joystick. The testing phases involved navigating to each of these locations in a particular order with a timer visible. For only the stress group, participants were also penalized monetarily for the amount of time required to find these locations. This monetary pressure was the only manipulation of stress in the present study.

As predicted, the physiological data from this experiment indicated higher arousal for the stress group than the no stress group in terms of heart rate, t(58) = 2.14, se = 1.03, p = .04, but not in terms of EDA, t(58) = -0.68, se = 0.02, p = .50 (Figure 4). In addition, there was a negative correlation between SBSOD score and the time required to find the four goal locations during the learning phase, r(58) = -0.40, p = .002, but not in the testing phase, r(58) = -0.25, p = .057. According to the visualized trajectories, the participants in the stress group appeared to be less distributed in the virtual environment. Together, these results suggest that higher arousal and spatial ability may be related to more efficient navigation behavior.

Figure 1: Photographs of the NeuroLab at ETH Zürich. (a) View of the experimenter and participant during testing. The experimenter can monitor the participant's progress in real-time. (b) Closeup view of the participant navigating through the virtual environment while physiological data is collected. Please click here to view a larger version of this figure.

Figure 2: Diagram representing the placement of the three ECG electrodes. This figure has been modified form FOAM (Free Open Access Meducation)³⁷which is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The image has been changed to highlight the electrodes necessary for a 3-electrode system. These electrodes should be placed between the ribs on the upper right abdomen (UR), the upper left abdomen (UL) and the lower left abdomen (LL) Please click here to view a larger version of this figure.

Figure 3: Screenshots from a navigation experiment in the NeuroLab. (a) Screenshot from the joystick training video. Participants were asked to reproduce the movements of the joystick from the video in the top-right corner. (b) Screenshot from the joystick training maze. Participants moved through a maze by following floating arrows and collecting gems. (c) Screenshot from the learning phase of the navigation task. Participants could press the trigger on the joystick in order to call a map of the virtual environment. A list of target locations was displayed on the right side of the screen. (d) Screenshot from the testing phase of the navigation task. Participants were asked to find the same locations in a particular order while a moving clock and moving reward were visible. Please click here to view a larger version of this figure.

Figure 4: Representative results from one navigation experiment in the NeuroLab using physiological devices and the EVE framework. (a) A graph representing the relationship between group (stress group in aquamarine and control group in salmon pink) and mean heart rate (after corrections for baseline values³⁸). Mean heart rate is significantly higher for the stress group than the control group. (b) A scatter plot representing the relationship between SBSOD score and time spent learning (in black) and testing (in gray). There is a significant negative relationship between SBSOD score and time spent learning and a similar trend for time spent testing. (c) A map of the virtual environment that displays aggregated path data from the stress (aquamarine) and control (salmon pink) groups. Darker coloring indicates that a higher proportion of paths taken along that route are from a particular group. For empty areas, the proportion of paths taken was similar for the two groups. Goal locations are also marked with black dots. As shown, the stress group was more likely than the control group to move along more direct paths between the goal locations. Please click here to view a larger version of this figure.

Subscription Required. Please recommend JoVE to your librarian.

Discussion

In the present paper, we described a protocol for conducting experiments in VR with physiological devices using the EVE framework. These types of experiments are unique because of additional hardware considerations (e.g., physiological devices and other peripherals), the preparatory steps for collecting physiological data using VR, and data management requirements. The present protocol provides the necessary steps for experimenters that intend to collect data from multiple peripherals simultaneously. For example, the use of physiological devices requires cleaning and attaching the electrodes to specific locations on the participant's body (e.g., the chest and fingers) in such a way as to not interfere with other peripherals (e.g., the joystick). The timing of such steps must account for the potential drift in the physiological signals and the appropriate window within which the data is reliable. The experimenter's consideration of timing is also critical for preparatory steps within each experimental session. For example, participants must complete a baseline phase (e.g., watching a nature video) in order for the experimenter to account for individual differences in physiological reactivity, as well as a training phase with the control interface in order for the experimenter to disentangle their ability to maneuver from their spatial decision-making in VR¹⁶^,¹⁷. In addition, the synchronization and storage of these data increase in complexity with the number of data sources. The EVE framework described in the present protocol provides a solution for studies with several data sources in VR. In addition, the flexibility of the EVE framework allows researchers to modify the experimental design according to their research questions and add new peripherals such as eye trackers and electroencephalography.

However, there are some limitations to this approach. First, working with the EVE framework requires some knowledge of computer science and basic programming skills. Second, the interpretation of physiological data is based on a long tradition of empirical research that must be considered during the design and analysis of these types of studies. Knowledge of this literature is critical given that physiological data can be easily misinterpreted (e.g., confusing stress and arousal). Third, many experiments in VR are susceptible to criticisms regarding external validity with respect to the virtual environment and control interface. For example, desktop VR often employs handheld joysticks and does not provide realistic proprioceptive feedback during walking. Compared to the studies in real environments, virtual environments tend to lead to the underestimation of distances³⁹ and less precision in spatial updating without proprioceptive feedback (without physically turning)⁴⁰. However, distance estimation and turn perception in VR can be improved with explicit visual feedback⁴¹^,⁴².

Previous research has demonstrated that experiments in VR can still reproduce realistic spatial¹⁸^,³⁹ and social³⁶^,⁴³^,⁴⁴ behavior. In addition, VR allows for greater experimental control and systematic variations that would be difficult in real-world scenarios⁴⁵. Frameworks such as EVE can also facilitate the development of a research program using VR by providing opportunities for reproducing and extending previous work. For example, researchers can slightly modify an existing experiment to include additional questionnaires or a different trial structure. A few additional advantages of the EVE framework are efficient data management, the availability of online tutorials, and the potential for others to contribute to its development. Indeed, the EVE framework is available for free as an open-source project that encourages collaboration.

Ongoing studies in this laboratory are investigating the impact of environmental features on the perception and physiological responses of participants with different socioeconomic backgrounds and the influence of congested environments on the physiological responses of participants immersed in a virtual crowd. In the future, this protocol may incorporate multi-user, networking technology that will allow participants at different physical locations to interact virtually. Finally, the EVE framework is currently being extended to include data analysis packages beyond simple diagnostics and the visualization of spatial data.

Subscription Required. Please recommend JoVE to your librarian.

Disclosures

The authors have nothing to disclose.

Acknowledgments

The virtual environment was kindly provided by VIS Games (http://www.vis-games.de) to conduct research in virtual reality.

Materials

Name	Company	Catalog Number	Comments
Alienware Area 51 Base	Dell	210-ADHC	Computation
138 cm 4K Ultra-HD LED-TV	Samsung	UE55JU6470U	Display
SureSigns VS2+	Philips Healthcare	863278	Blood Pressure
PowerLab 8/35	AD Instruments	PL3508	Skin Conductance
PowerLab 26T (LTS)	AD Instruments	ML4856	Heart Rate
Extreme 3D Pro Joystick	Logitech	963290-0403	HID

DOWNLOAD MATERIALS LIST

References

Gallistel, C. R. The Organization of Learning. , MIT Press. Cambridge, MA. (1990).
Waller, D., Nadel, L. Handbook of Spatial Cognition. , American Psychological Association. Washington D.C. (2013).
Denis, M. Space and Spatial Cognition: A Multidisciplinary Perspective. , Routledge. Abingdon, Oxon. (2017).
Epstein, R. A., Patai, E. Z., Julian, J. B., Spiers, H. J. The cognitive map in humans: spatial navigation and beyond. Nature Neuroscience. 20, 1504 (2017).
O'Keefe, J., Nadel, L. The Hippocampus as a Cognitive Map. , Clarendon Press. Oxford. (1978).
Kuipers, B. J. Modelling spatial knowledge. Cognitive Science. 2, 129-153 (1978).
Heppenstall, A. J., Crooks, A. T., See, L. M., Batty, M. Agent-Based Models of Geographical Systems. , Springer. Netherlands. Dordrecht. (2012).
Kuliga, S. F., Thrash, T., Dalton, R. C., Hölscher, C. Virtual reality as an empirical research tool - Exploring user experience in a real building and a corresponding virtual model. Computers, Environment and Urban Systems. 54, 363-375 (2015).
Credé, S., Fabrikant, S. I. Let's Put the Skyscrapers on the Display-Decoupling Spatial Learning from Working Memory. Proceedings of Workshops and Posters at the 13th International Conference on Spatial Information Theory (COSIT 2017). , 163-170 (2018).
Hodgson, E., Bachmann, E. R., Vincent, D., Zmuda, M., Waller, D., Calusdian, J. WeaVR: a self-contained and wearable immersive virtual environment simulation system). Behavior Research Methods. 47 (1), 296-307 (2015).
Nilsson, N., et al. 15 Years of Research on Redirected Walking in Immersive Virtual Environments. IEEE Computer Graphics and Applications. , 1-19 (2018).
Razzaque, S., Kohn, Z., Whitton, M. C. Redirected walking. Proceedings of EUROGRAPHICS. , 105-106 (2001).
Hodgson, E., Bachmann, E. Comparing Four Approaches to Generalized Redirected Walking: Simulation and Live User Data. IEEE Transactions on Visualization and Computer Graphics. 19 (4), 634-643 (2013).
Nescher, T., Huang, Y. -Y., Kunz, A. Planning redirection techniques for optimal free walking experience using model predictive control. 2014 IEEE Symposium on 3D User Interfaces (3DUI). , 111-118 (2014).
Razzaque, S., Swapp, D., Slater, M., Whitton, M. C., Steed, A. Redirected walking in place. Eurographics workshop on virtual environments. , 123-130 (2002).
Meilinger, T., Knauff, M., Bulthoff, H. Working Memory in Wayfinding-A Dual Task Experiment in a Virtual City. Cognitive Science: A Multidisciplinary Journal. 32 (4), 755-770 (2008).
Grübel, J., Thrash, T., Hölscher, C., Schinazi, V. R. Evaluation of a conceptual framework for predicting navigation performance in virtual reality. PLOS ONE. 12 (9), 0184682 (2017).
Weisberg, S. M., Schinazi, V. R., Newcombe, N. S., Shipley, T. F., Epstein, R. A. Variations in Cognitive Maps: Understanding Individual Differences in Navigation. Journal of experimental psychology. Learning, memory, and cognition. , (2014).
Wiener, J. M., Hölscher, C., Büchner, S., Konieczny, L. Gaze behaviour during space perception and spatial decision making. Psychological research. 76 (6), 713-729 (2012).
Hassabis, D., Chu, C., Rees, G., Weiskopf, N., Molyneux, P. D., Maguire, E. A. Decoding Neuronal Ensembles in the Human Hippocampus. Current Biology. 19 (7), 546-554 (2009).
Maguire, E. A., Nannery, R., Spiers, H. J. Navigation around London by a taxi driver with bilateral hippocampal lesions. Brain. 129, 2894-2907 (2006).
Marchette, S. A., Vass, L. K., Ryan, J., Epstein, R. A. Anchoring the neural compass: coding of local spatial reference frames in human medial parietal lobe. Nature neuroscience. 17 (11), 1598-1606 (2014).
Vass, L. K., et al. Oscillations Go the Distance: Low-Frequency Human Hippocampal Oscillations Code Spatial Distance in the Absence of Sensory Cues during Teleportation. Neuron. 89 (6), 1180-1186 (2016).
Sharma, G., Gramann, K., Chandra, S., Singh, V., Mittal, A. P. Brain connectivity during encoding and retrieval of spatial information: individual differences in navigation skills. Brain Informatics. 4 (3), (2017).
Richardson, A. E., VanderKaay Tomasulo, M. M. Influence of acute stress on spatial tasks in humans. Physiology & Behavior. 103 (5), 459-466 (2011).
Duncko, R., Cornwell, B., Cui, L., Merikangas, K. R., Grillon, C. Acute exposure to stress improves performance in trace eyeblink conditioning and spatial learning tasks in healthy men. Learning & memory (Cold Spring Harbor, N.Y.). 14 (5), 329-335 (2007).
Klopp, C., Garcia, C., Schulman, A. H., Ward, C. P., Tartar, J. L. Acute social stress increases biochemical and self report markers of stress without altering spatial learning in humans. Neuro endocrinology letters. 33 (4), 425-430 (2012).
Guenzel, F. M., Wolf, O. T., Schwabe, L. Sex differences in stress effects on response and spatial memory formation. Neurobiology of Learning and Memory. 109, 46-55 (2014).
van Gerven, D. J. H., Ferguson, T., Skelton, R. W. Acute stress switches spatial navigation strategy from egocentric to allocentric in a virtual Morris water maze. Neurobiology of Learning and Memory. 132, 29-39 (2016).
Grübel, J., Weibel, R., Jiang, M. H., Hölscher, C., Hackman, D. A., Schinazi, V. R. EVE: A Framework for Experiments in Virtual Environments. Spatial Cognition X: Lecture Notes in Artificial Intelligence. , 159-176 (2017).
Hegarty, M., Richardson, A. E., Montello, D. R., Lovelace, K., Subbiah, I. Development of a self-report measure of environmental spatial ability. Intelligence. 30, 425-447 (2002).
Ziegler, M. G. Psychological Stress and the Autonomic Nervous System. Primer on the Autonomic Nervous System. , 189-190 (2004).
Michaelis, J. R., Rupp, M. A., Montalvo, F., McConnell, D. S., Smither, J. A. The Effect of Vigil Length on Stress and Cognitive Fatigue. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 59 (1), 916-920 (2015).
Helton, W. S. Validation of a Short Stress State Questionnaire. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 48 (11), 1238-1242 (2004).
Wolbers, T., Hegarty, M. What determines our navigational abilities. Trends in Cognitive Sciences. 14 (3), 138-146 (2010).
Moussaïd, M., et al. Crowd behaviour during high-stress evacuations in an immersive virtual environment. Journal of The Royal Society Interface. 13 (122), (2016).
Cadogan, M. Lead positioning. , Available from: https://lifeinthefastlane.com/ecg-library/basics/lead-positioning/ (2017).
Wilder, J. The law of initial value in neurology and psychiatry. The Journal of Nervous and Mental Disease. 125 (1), 73-86 (1957).
Loomis, J., Knapp, J. Visual Perception of Egocentric Distance in Real and Virtual Environments. Virtual and Adaptive Environments. , 21-46 (2003).
Richardson, A. R., Waller, D. The effect of feedback training on distance estimation in virtual environments. Applied Cognitive Psychology. 19 (8), 1089-1108 (2005).
Klatzky, R. L., Loomis, J. M., Beall, A. C., Chance, S. S., Golledge, R. G. Spatial updating of self-position and orientation during real, imagined, and virtual locomotion. Psychological Science. 9, 293-298 (1998).
Bakker, N. H., Werkhoven, P. J., Passenier, P. O. Calibrating Visual Path Integration in VEs. Presence: Teleoperators and Virtual Environments. 10 (2), 216-224 (2001).
Thrash, T., et al. Evaluation of control interfaces for desktop virtual environments. Presence. 24 (4), (2015).
Kinateder, M., Warren, W. H. Social Influence on Evacuation Behavior in Real and Virtual Environments. Frontiers in Robotics and AI. 3, 43 (2016).
Loomis, J. M., Blascovich, J. J., Beall, A. C. Immersive virtual environment technology as basic research tool in psychology. Behavior Research Methods, Instruments & Computers. 31 (4), 557-564 (1999).

Behavior