Audio-based Environment Simulator (AbES) is virtual environment software designed to improve real world navigation skills in the blind.
Audio-based Environment Simulator (AbES) is virtual environment software designed to improve real world navigation skills in the blind. Using only audio based cues and set within the context of a video game metaphor, users gather relevant spatial information regarding a building’s layout. This allows the user to develop an accurate spatial cognitive map of a large-scale three-dimensional space that can be manipulated for the purposes of a real indoor navigation task. After game play, participants are then assessed on their ability to navigate within the target physical building represented in the game. Preliminary results suggest that early blind users were able to acquire relevant information regarding the spatial layout of a previously unfamiliar building as indexed by their performance on a series of navigation tasks. These tasks included path finding through the virtual and physical building, as well as a series of drop off tasks. We find that the immersive and highly interactive nature of the AbES software appears to greatly engage the blind user to actively explore the virtual environment. Applications of this approach may extend to larger populations of visually impaired individuals.
Finding one’s way in an unfamiliar environment presents as a significant challenge for the blind. Navigating successfully requires an understanding of the spatial relationships that exist between one’s self and objects in the environment1,2. The mental representation that describes surrounding space is referred to as a spatial cognitive map3. Blind individuals can gather relevant spatial information regarding their surrounding environment through other sensory channels (such as hearing) allowing for the generation of an accurate spatial cognitive map for the purposes of real world navigation tasks4,5.
Considerable interest has arisen regarding the educative potential of virtual environments and action video games as a means to learn and master skills6-9. Indeed, many strategies and approaches have been developed for the blind for this purpose (see 4,10-12). We have developed Audio-based Environment Simulator (AbES); a user-centered audio-based virtual environment that allows for simulated navigation and exploration of an existing physical building. Drawing from original architectural floor plans, a virtual rendering of a modern two-story building (located at the Carroll Center for the Blind; Newton, MA) was generated with the AbES software (Figures 1A and B). AbES incorporates an action game metaphor with a premise designed to promote full exploration of the building space. Using simple key strokes and spatialized sound cues, users navigate and explore the entire building to collect a maximum number of jewels hidden in various rooms. Users must avoid roving monsters that can take them away and hide them elsewhere in the building (Figure 1C).
We demonstrate that interacting with AbES allows a blind user to generate an accurate spatial cognitive map of a target building based on auditory information acquired within the context of an action game metaphor. This is confirmed by a series of post-training behavioral performance tests designed to assess the transfer of acquired spatial information from a virtual environment to a real-world and large scale indoor navigation task (see Figure 2 for overall study design). Our results show that blind users are able to successfully navigate throughout a building for which they were previously unfamiliar, despite the fact that at no time were they informed of the overall purpose of the study, nor were they instructed to recall the spatial layout of the building while playing the game.
1. Participant Demographics
This is an on-going study that recruits blind male and female participants aged between 18-45 years. All participants are legally blind of early onset (documented prior to the age of 3) and of varying ocular etiologies. None of the study participants were previously familiar with the spatial layout of the target physical building.
2. Preparation and Familiarization with AbES
3. Training and Game Play with AbES (3 Sessions Each Lasting 30 min for a Total of 1.5 hr)
4. Assess Virtual Navigation Task Performance
5. Assess Physical Navigation Task Performance
6. Assess Physical Drop off Task Performance
Results from three early blind participants (aged between 19 and 22 years) are shown (see Table 1 for participant characteristics). In summary, all three participants showed a high level of success on all three navigation tasks following game play with the AbES software. This was confirmed by the performance scores (group mean and individual) on all three behavioral tasks (see Figure 6). The percentage correct performance for the virtual (mean: 90%) followed by the physical (mean: 88.7%) navigation tasks illustrates a high level of success and comparable performance for both tasks (Figure 6A). Performance on the drop off experiments suggests that participants often selected the shortest route possible to exit the building (mean score: 3.0) (Figure 6B). Finally, the average time taken to navigate to target is shown for all three navigation tasks is shown in Figure 6C. Virtual navigation time (assessed first) was typically longer (mean: 137.3 sec) than physical (73.8 sec) navigation performance. The shorter mean navigation times observed in the drop off task (mean: 37.3 sec) are consistent with the fact that participants were likely to choose the shortest possible path to exit the building.
Assessing individual results from one representative study participant and navigation route on all three tasks assessed revealed that virtual navigation from a start to end point located on the first floor took 79 sec (Figure 7A; path shown in yellow). Assessment of performance on the same path in the physical building took 46 sec (Figure 7B). Assessment of drop off task performance illustrates that the participant took the shortest path possible (scoring 3 points and a taking navigation time of 48 sec) (Figure 7C).
subject | age (years) | etiology of blindness | level of visual function |
1 | 22 | retinopathy of prematurity | residual (light perception) |
2 | 19 | Peters anomaly; bilateral retinal detachment; end stage glaucoma | profound (no light perception) |
3 | 19 | retinopathy of prematurity | residual (light perception) |
Table 1. Participant characteristics.
Figure 1. Virtual environment rendered in AbES. A) original two-story building floor plan. The building includes 23 rooms and a series of connecting corridors as well as 3 separate entrances and 2 stairwells. Given the existing spatial layout, there are multiple route possibilities to enter and exit the building, B) virtual rendering of target building in AbES, C) objects encountered while playing AbES in game mode. Click here to view larger figure.
Figure 2. Overall Study Design. All participants undergo a fixed training and game play period with AbES followed by a series of navigation assessments (always in sequential order). Assessments of performance include virtual, physical, and drop off navigation tasks.
Figure 3. AbES keystrokes.
Figure 4. Training and game play with AbES. A) Participants sit at a computer terminal wearing a blindfold and stereo headphones. B) Photo of an investigator with a study participant.
Figure 5. Summary of navigation task assessments. A) Data capture from virtual navigation path assessment. The start and end points are read to the participant and the next path is loaded automatically after completion. The path taken (shown in yellow) and time to target are collected automatically by the software. B) Investigator assesses performance in a physical navigation task. Timing (using a stopwatch) commences with the participant’s first step and ends when the participant reports arriving to the target end point. C) Sample route and scoring strategy for drop off navigation task. There are three exit doors and thus multiple possible routes to exit the building. Based on the starting point, the path taken (shown in yellow) is scored. Three (3) points are given for using the shortest exit, followed by 2 and 1 point (a score of zero indicates unable to find an exit). Click here to view larger figure.
Figure 6. Summary results from navigation task assessments. Results (group means and individual results from 10 tested navigation routes) from 3 representative participants in the study are shown. A) Percentage correct performance for the virtual followed by the physical navigation tasks. B) Performance results (average number of points) on drop off tasks. C) Average time taken to navigate to target is shown for all three navigation assessments. Click here to view larger figure.
Figure 7. Individual results from navigation task assessments. Representative results are shown from one study participant on all three navigation tasks assessed. A) virtual navigation (path shown on yellow). B) assessment of performance on the same path in the physical building. C) assessment of a drop off task illustrates that the participant took the shortest path possible. The alternative potential paths (yellow dotted lines) and score value relative to the given starting point are also shown. Click here to view larger figure.
Supplmental Movie 1. Supplementary video of annotated video game play. Video sequence showing a player (yellow moving icon) entering a room located on the first floor where a jewel is hidden. Spatialized sounds (left and right channel) allow the player to orient and identify the location of objects (e.g. doors and obstacles) in their surrounding environment. Once a jewel is found, the player exits the building and they must avoid roving monsters (red moving icons). The player then continues to explore the building (first and second floors) to find more hidden jewels. Click here to view supplemental movie.
We describe an interactive audio-based virtual environment simulator designed to improve general spatial awareness and navigation skills in the blind. We demonstrate that interacting with AbES provides accurate cues that describe the spatial relationships between objects and the overall layout of the target environment. Blind users can generate accurate spatial cognitive maps based on this auditory information and by interacting with the immersive virtual environment. Furthermore, interacting with AbES within the context of a game metaphor demonstrates that spatial cognitive constructs can be learned implicitly and rather simply through causal interaction with the software. As demonstrated in this initial phase of the study, the interactive and immersive nature of the game can improve the individual’s spatial awareness of a new environment, provide a platform for creating an accurate spatial cognitive map, and may reduce the insecurity associated with independent navigation prior to arriving at an unfamiliar building.
Typically, individuals with visual impairment can gain functional independence through orientation and mobility (O&M) training. It is important however that training strategies remain flexible and adaptable so that they can be applied to novel and unfamiliar situations and tailored to a person’s own strengths and weaknesses so as to address their particular challenges, needs and learning strategies. The creative use of interactive virtual navigation environments such as AbES may provide for this flexibility and supplement current O&M training curriculum. This software represents an adjunctive strategy that not only draws upon the benefits of high motivational drive, but also provides for a testing platform to carry out more controlled and quantifiable studies to test and validate the effectiveness of these training approaches.
Current and future investigations will include a large-scale study where participants are randomized to differing methods of training (e.g. gaming compared to direct serial route learning) and navigation (i.e. route finding) performance will be compared. We will also investigate differences between early and late blind as well as the relationship between additional factors of interest including age and gender.
Finally, given the apparently engaging nature of this combined virtual environment and gaming approach, it would also be of interest to investigate the potential benefit of AbES on navigation skill development in blind individuals beyond the profile described here. For example, the largest (and fastest growing) segment of visual impairment is in the aging population and current trends are expected to increase13. Thus, it would seem highly relevant to explore the effectiveness of this approach for the non-visual acquisition of spatial information supporting navigation skills in this demographic group. Given that AbES is a computer based approach, it is difficult to speculate at this time on its effectiveness on non-digital natives. Along similar lines, developing AbES in manner that would be amenable to individuals with residual vision (i.e. low vision) could also be worthwhile. Given that the majority of individuals who are legally blind fall under this category13, training in virtual environments prior to actual physical travel may also be of benefit to plan routes and avoid difficulties associated with trying to access information in an unfamiliar environment. In this direction, current work is aimed at developing AbES features such as zooming (i.e. high magnification) and high a contrast display to support individuals with low vision.
The authors have nothing to disclose.
The authors would like to thank Rabih Dow, Padma Rajagopal, Molly Connors and the staff of the Carroll Center for the Blind (Newton MA, USA) for their support in carrying out this research. This work was supported by the NIH/NEI grant: RO1 EY019924.
Name of Equipment | Comments |
---|---|
Laptop computer | Laptop used exclusively for training participants and collecting data |
Stereo Head phones (fully enclosed circumaural design) | Worn by all participants during training |
Blindfold | Worn by all participants during training and testing |