Virtual Prism Adaptation Therapy: Protocol for Validation in Healthy Adults

Sungmin Cho; Won-Seok Kim; Seo Hyun Park; Jihong Park; Nam-Jong Paik

doi:10.3791/60639

Medicine

Virtual Prism Adaptation Therapy: Protocol for Validation in Healthy Adults

Published: February 12, 2020 doi: 10.3791/60639

Sungmin Cho*¹, Won-Seok Kim*², Seo Hyun Park², Jihong Park², Nam-Jong Paik²

¹Delvine Inc., ²Department of Rehabilitation Medicine, Seoul National University College of Medicine, Seoul National University Bundang Hospital

* These authors contributed equally

Summary

This experimental protocol demonstrates the use of virtual prism adaptation therapy (VPAT) in healthy adults and the association between VPAT and functional near infrared spectroscopy to determine the effect of VPAT on cortical activation. Results suggest that VPAT may be feasible and could induce similar behavioral adaptation as conventional prism adaptation therapy.

Abstract

Hemispatial neglect is a common impairment after stroke. It is associated with poor functional and social outcomes. Therefore, an adequate intervention is imperative for the successful management of hemispatial neglect. However, the clinical use of various interventions is limited in real clinical practice. Prism adaptation therapy is one of the most evidence-based rehabilitation modalities to treat hemispatial neglect. To overcome any possible shortcoming that may occur with prism therapy, we developed a new system using immersive virtual reality and depth-sensing camera to create a virtual prism adaptation therapy (VPAT). To validate the VPAT system, we designed an experimental protocol investigating the behavioral errors and changes in cortical activation via the VPAT system. Cortical activation was measured by functional near infrared spectroscopy (fNIRS). The experiment consisted of four phases. All four included clicking, pointing or rest applied to right-handed healthy people. Clicking versus pointing was used for investigating the cortical region related with the gross motor task, and pointing with VPAT versus pointing without VPAT was used for investigating the cortical region associated with visuospatial perception. The preliminary results from four healthy participants showed that pointing errors by the VPAT system was similar to the conventional prism adaptation therapy. Further analysis with more participants and fNIRS data, as well as a study in patients with stroke may be required.

Introduction

Hemispatial neglect, which affects the ability to perceive the contralateral hemispatial visual field, is a common impairment after stroke¹^,². Although rehabilitation after hemispatial neglect is important, due to its association with poor functional and social outcomes, rehabilitation is often underutilized in real clinical practice³^,⁴.

Among the various existing rehabilitation approaches suggested for hemispatial neglect, prism adaptation (PA) therapy has proven effective for recovery and improvement in hemispatial neglect in patients with subacute or chronic stroke⁵^,⁶^,⁷^,⁸. However, conventional PA is underutilized due to several drawbacks⁹^,¹⁰. These include 1) high cost and time requirement due to the prism lens needing to be changed to adjust to the degree of deviation; 2) the need to set up additional materials to be pointed at and to mask the hand trajectory; and 3) PA can only be used by patients who can sit and control their head position.

A recent study reproducing the adaptation effects in the virtual reality (VR) environment reported that it may be possible for the virtual prism adaptation therapy (VPAT) to have different effects depending on the subtypes of neglect¹¹. It was also suggested that cortical activation for PA may vary according to brain lesions¹². However, little is known about the cortical activation pattern seen in VR-induced PA.

To overcome these obstacles and promote the use of PA in a clinical setting, we developed a new PA therapy system using an immersive VR technology called virtual prism adaptation therapy (VPAT), via the use of a depth-sensing camera. We designed an immersive VR system with the ability to provide visual feedback about the position of a virtual limb to promote spatial realignment¹³. Using this immersive VR technology, which mimicked the effect of conventional PA, we designed an experiment to validate the VPAT system in healthy participants.

By conducting our visualized experimental protocol, we investigated whether the new VPAT system can induce behavioral adaptation, similar to conventional PA. Additionally, we would like to explore whether the VPAT system can induce the activation in the cortical regions associated with visuospatial perception or recovery of hemispatial neglect after stroke.

Subscription Required. Please recommend JoVE to your librarian.

Protocol

All procedures were reviewed and approved by the Seoul National University Bundang Hospital Institutional Review Board (IRB). To recruit healthy participants, posters were used to advertise around the hospital.

1. Experimental set-up

Participant recruitment
1. Perform the subject screening process using the following inclusion criteria: 1) healthy, between 18 and 50 years old; 2) right-handed, assessed by Edinburgh handedness inventory¹⁴; 3) able to wear the head mount display for VR and to detect objects within VR; and 4) no history of diseases affecting the brain, such as stroke, Parkinson's disease, or traumatic brain injury.
  NOTE: These criteria were designed to screen participants with the ability to participate in the experiment and regulate factors affecting the results.
2. Recruit participants and provide a detailed explanation of the entire study and expected clinical issues. Consent must be obtained prior to inclusion.
Experimental system
NOTE: A customized VPAT system using an immersive VR system and depth-sensing camera was used. Functional infrared spectroscopy (fNIRS) was simultaneously used to investigate the cortical activation. VPAT and fNIRS were linked together for the experiment (Figure 1).
1. VPAT system
  NOTE: The VPAT system consists of a head mount display for VR implementation, a hand tracking sensor that can recognize hand gestures for intuitive input by the user, and a hardware push button. The overall composition is shown in Figure 1.
  1. Make sure that the hand tracking sensor is not tilted in front of the head mount display.
  2. Check that the reference camera for the VR system is properly installed on top of the front monitor.
  3. Secure the push button in a location near the hand to be used by the participant for the experiment.
  4. Run the software to make sure there are no errors.
    NOTE: The virtual environment was implemented to match the actual environment as close as possible. The task was performed through hand pointing within the virtual environment and button input through the push button.
2. fNIRS
  1. Use a commercial fNIRS system including a personal computer (PC), 31 optodes (15 light sources and 16 detectors), textile EEG caps, and data recording software.
3. Linkage between VPAT system and fNIRS (Figure 1).
  1. Use the remote keyboard control software using TCP/IP communication to synchronize the starting event in the VPAT system with the timing of recording in the fNIRS system.
  2. Use the remote command key in the computer to start fNIRS recording.

2. Experimental set-up (Figure 2)

fNIRS measurement setting
1. Place the participant in a chair with his/her back in a straight posture, about fifteen centimeters away from the table. Confirm that the participant's hand does not hit the table when reaching out.
2. For fNIRS cap setting, select the cap size according to the participant's head circumference. Place the cap so that the vertex (Cz) is located at the intersection of the midpoint between the inion and nasion and the midpoint between the left preauricular and right preauricular areas. Display the montage on the screen and connect 15 sources and 24 detectors to the montage. If necessary to improve the gain from the light source, use conductive gel after hair preparation and insert the optode. Have the participant wear a retaining cap.
  NOTE: The study used three different sizes of textile EEG caps with circumferences of 54, 56, and 58 cm.
3. For software setting (calibration, etc.), run the fNIRS system software and load the neglect montage.
4. Let the montage be displayed on the screen and set 15 sources and 24 detectors according to the montage (Figure 3).
5. Press the calibrate button. If "Lost" is displayed on the screen, repeat the hair preparation, and then recalibrate.
VPAT system setting
1. Connect the HMD, reference camera, and Leap motion camera, and push the button connecting the computer to set up the VPAT system.
2. Mount the virtual reality head-mounted display (VR HMD) on the participant's head over the cap for fNIRS. Make sure to avoid movement of the cap.
3. Run the VPAT software. Enter the participant's information (name abbreviation, age, handedness) and press the "Start" button.
4. Confirm the visualization of the virtual hand in the display. Proceed with a two-step calibration (i.e., screen calibration and target distance calibration).
5. Instruct the participant to watch the red cross mark (+) in the center, then press the "r" key to calibrate the screen.
  NOTE: Screen calibration places the virtual space in front of the user's visual range by recentering the coordinate system.
6. Instruct the participant to point to the target (i.e., ball) with his or her right hand, then press the "O" key to calibrate the hand position.
  NOTE: In our study, the object that the participant had to target was a white ball on a pink stick that came down from the top of the view. Target distance calibration places the target within the reach of the user. This is used to correctly position the target during the experiment.
7. After calibration, press the "w" key to begin the experiment.
VPAT and fNIRS linkage setting
1. Use the event synchronization software to enter the trigger for analysis into fNIRS and connect VPAT to fNIRS.
2. For time synchronization between VPAT and fNIRS, connect the computers with the two systems to the same network, and then synchronize them through the self-produced key transferring program.
3. After connecting through the IP and Port inputs of both computers, start the experiment session via the "w" key in the VPAT program. The event synchronization software is executed automatically, and triggers during execution are automatically transferred to fNIRS and saved.
4. After the experiment, obtain the software auto-termination and VPAT data. Then stop the VPAT and fNIRS system software.
  NOTE: The participants must return their hands to their original position after pointing during the VPAT experiment.

3. Experiment to validate VPAT system

Block designed experiment with fNIRS recording (Figure 4)
1. After completing the set-up process in step 2, confirm the participant's readiness to start the experiment.
2. Start the VPAT system without the prism mode and instruct the participant to point to the target in the VR system immediately for familiarization with the procedure.
3. Each phase consists of blocks for pointing, clicking, or resting (Figure 4). Again, instruct the participant to click on the button or point to the target in the VR system with their right index finger as quickly as possible.
4. Start the experiment with four phases simultaneously with fNIRS recording by clicking the start key.
  NOTE: During the pointing task, the white ball had to be touched within a fixed time.
  1. Instruct the participants to point, click, or rest when the appropriate icon appears.
    NOTE: During the task, pointing and clicking were indicated by an icon directly above the white ball and right side of the timer bar. The time to perform the task was indicated by the timer bar as shown in Figure 2.
  2. Tell the participant to touch the target that appears on the left or right side within 3 s. For the clicking block, instruct the participant to press the push button.
    NOTE: The target set containing the white ball was located at a distance of -10° or 10° from the participant's center, obtained by calibration. The target set appeared randomly on the right or left side. According to the experimental design, the target appeared for 3 s, then disappeared, and then regenerated to a new position.
  3. Ensure that the participant performs the same way when the phase is switched.
    NOTE: In the pointing task, Virtual Prism Adaptation Mode showed a deviation of 10° or 20° to the left side of the imaginary hand in the VR space relative to the participant's head. Zero degrees indicated that the positions of the virtual hand and the actual hand coincided.
    NOTE: The experiment (Figure 4) consists of a total of four phases, with each phase consisting of pointing and clicking or rest alternately (Phase 1 and 4 were pointing and clicking, and phase 2 and 3 were pointing and resting).

4. Data analysis

Pointing error analysis
NOTE: The data were stored from the moment the experimenter pressed the start button "w". The data were automatically stored at about 60 Hz every frame through the VPAT software. The phase name, elapsed time, and virtual index finger position were stored over time. The pointing error was the angle value between the target and index finger, centered on the participant's head position.
1. Classify the pointing task data by phases (pre-VPAT, VPAT 10°, VPAT 20°, post-VPAT).
2. Classify the data of the pointing task and the clicking task in the data of each phase (phase 1 and 4).
3. Classify the data by sub-phase in units of 30 s according to each phase and each type of task.
4. Extract the median value of 10 trial error (pointing error) values from the index finger position data for median pointing error analysis.
5. Use the repeated measures analysis of variance test (ANOVA) to analyze the difference between each phase.
  NOTE: In the case of hand tracking using the Leap motion sensor, outliers were due to occlusion or false detection of the hand posture. With the exception of false hand position data, the median value was used to find the representative pointing error value in the sub-phase.
fNIRS data processing
1. Launch the fNIRS analysis software and load the raw data file and probe information.
2. Perform a marker setting process by editing the event record to verify each condition during the experiment.
3. Carry out data preprocessing by deleting the experimentally irrelevant time intervals, remove artifacts, such as steps and spikes, and apply frequency filters to exclude experimentally irrelevant frequency bands.
  NOTE: All data sets were filtered with a 0.01 Hz high-pass filter and a 0.2 Hz low-pass filter to remove instrumental or physiological noise contributions.
4. Specify wavelengths by entering the value of the peak illumination wavelengths (i.e., 760 and 850 nm). Use a physical distance of 3 cm between the source and detector for channel.
5. Select the baseline field, which refers to the time period that corresponds to a baseline wherein participants are typically resting quietly.
  NOTE: We selected the baseline field as the full-time course of the data set, which was the default setting.
6. Compute the time series of hemodynamic states to finish the preprocessing from the filtered data.

Subscription Required. Please recommend JoVE to your librarian.

Representative Results

Data from four healthy participants (1 man and 3 women) were used as representative results. A pointing error is shown in Figure 5A, with the averages of median value of 10 trials in the sub-phase of each pointing task lasting 30 s. Values on average for the median pointing errors in the first block of each phase were 0.45 ± 0.92 (pre-VPAT), 4.69 ± 3.08 (VPAT 10°), 5.43 ± 2.22 (VPAT 20°), and -5.17 ± 1.60 (post-VPAT). The trend of pointing error change was statistically significant (p = 0.001) via the repeated measures ANOVA. A pointing error for each subject is presented in Figure 5B, illustrating the adaptation during the VPAT phase and post-prismatic adaptation (negative pointing error).

Figure 1: Experimental setting with VPAT and fNIRS linkage system. VPAT = virtual prism adaptation therapy; fNIRS = functional near infrared spectroscopy. This figure was previously published by Kim et al.¹⁵ Please click here to view a larger version of this figure.

Figure 2: The subject performing the experiment with VPAT and fNIRS system. VPAT = virtual prism adaptation therapy; fNIRS = functional near infrared spectroscopy. Please click here to view a larger version of this figure.

Figure 3: Montage containing 54 channels by arranging 15 light sources (red circles) and 24 detectors (blue circles) at intervals of 3 cm. Space between the nearest sources and the detector constituted one channel, which is represented as yellow circles with a number. Please click here to view a larger version of this figure.

Figure 4: Experimental design. VPAT = virtual prism adaptation therapy; Pt = pointing; Cl = clicking; Re = resting. Please click here to view a larger version of this figure.

Figure 5: Pointing errors in each block. (A) Average value graph of subject's median pointing error in each block. This figure was previously published by Kim et al.¹⁵ (B) Median pointing error in each block by each subject. The counterclockwise direction (i.e. left from the target) is the positive value. Please click here to view a larger version of this figure.

Subscription Required. Please recommend JoVE to your librarian.

Discussion

This study implemented the prism adaption therapy using a translated hand movement in a VR environment. It investigated whether the deviation implemented was causing angle overshooting and behavioral adaptation, as in conventional prism adaption therapy.

In the median pointing error result (Figure 5) and the first pointing error result, the pointing error changed significantly when the phase was switched. Although some hand-recognition errors were eliminated, there may still be false detection. The use of a median value to eliminate systematic error, such as false tracking, showed that the average pointing error results were lower than expected. Post-prismatic adaptation was constantly shown in each subject (Figure 5B). These results showed similar behavioral adaptation to the conventional prism adaptation therapy.

There were some problems in the experiment. False detection of the hand occurred frequently in the pointing task. In some cases, even though the hand reached the target during pointing, the virtual hand was not tracked due to a Leap motion recognition error. In addition, because the participants were wearing HMD in the clicking task, it was difficult for them to locate the push button and the experimenter had to provide continuous assistance. The weight of the HMD and its long-term application could also cause pain in the area that comes into contact with the fNIRS optode. Therefore, there were times when the HMD was lifted or the participants themselves were holding the HMD.

If we overcome the shortcomings of the system and consolidate the results of the experiment through more data analysis, including fNIRS data, it could potentially be used in the treatment of visuospatial neglect. In addition, game-friendly contents can be applied to present an immersive and fun treatment modality. Nonetheless, further study with a more advanced VPAT system proving clinical efficacy in stroke patients with visuospatial neglect is needed.

Several previous studies have reported motion sickness induced by the use of Immersive VR, or head-mounted VR sets¹⁶. Motion sickness is reported to be infrequent if VR is implemented in seated positions¹⁷. Motion mismatch can also cause motion sickness, but it can be reduced by independently configuring the background in the virtual environment¹⁸^,¹⁹. In this system, only the hand deviation angle caused motion mismatch, which should have less impact on motion sickness overall.

Participants in this experiment were normal adults, so there were no consistent problems. However, to be used as therapeutic treatment for stroke patients, the above issues need to be considered, and virtual prism therapy protocols need to be taken into account, such as taking breaks during treatment or the length of treatment time.

Subscription Required. Please recommend JoVE to your librarian.

Disclosures

Won-Seok Kim, Sungmin Cho, and Nam-Jong Paik have a patent entitled "Method, system and readable recording medium of creating visual stimulation using virtual model", number 10-1907181, which is relevant to this work.

Acknowledgments

This study was supported by the Seoul National University Bundang Hospital Research Fund (14-2015-022) and by the Ministry of Trade Industry & Energy(MOTIE, Korea), Ministry of Science & ICT(MSIT, Korea), and Ministry of Health & Welfare(MOHW, Korea) under Technology Development Program for AI-Bio-Robot-Medicine Convergence (20001650). We would like to thank Su-Bin Park, Nu-Ri Kim and Ye-Lin Jang for helping to prepare and proceed with the video shooting.

Materials

Name	Company	Catalog Number	Comments
EASYCAP	Easycap	C-SAMS	Platform to accommodate fNIRS optodes
Leap Motion 3D Motion Controller	Ultrahaptics	FBA_LM-C01-US	Hand detection device attached HMD
Leap Motion VR Developer Mount for VR Headset	Ultrahaptics	VR-UAZ
Matlab R2015a	Mathworks		Programming language running with NIRStar
NIRScout	Medical Technology LLC	NSC-CORE	fNIRS system
nirsLAB v201605	Medical Technology LLC		Software for analyzing data collected with NIRScout
NIRStar 14.1	Medical Technology LLC		NIRScout Acquisition Software
Occulus Rift DK2	Occulus		VR HMD
PowerMate USB Multimedia Controller	Griffin Technology	NA16029	Push Button in task
SuperLab 5.0	Cedrus Corp.		Synchronize the stimulus presentations allied to NIRScout

DOWNLOAD MATERIALS LIST

References

Appelros, P., Karlsson, G. M., Seiger, A., Nydevik, I. Neglect and anosognosia after first-ever stroke: incidence and relationship to disability. Journal of Rehabilitation Medicine. 34 (5), 215-220 (2002).
Buxbaum, L., et al. Hemispatial neglect subtypes, neuroanatomy, and disability. Neurology. 62 (5), 749-756 (2004).
Jehkonen, M., et al. Visual neglect as a predictor of functional outcome one year after stroke. Acta Neurologica Scandinavica. 101 (3), 195-201 (2000).
Jehkonen, M., Laihosalo, M., Kettunen, J. Impact of neglect on functional outcome after stroke–a review of methodological issues and recent research findings. Restorative Neurology and Neuroscience. 24 (4-6), 209-215 (2006).
Mizuno, K., et al. Prism adaptation therapy enhances rehabilitation of stroke patients with unilateral spatial neglect: a randomized, controlled trial. Neurorehabilitation and Neural Repair. 25 (8), 711-720 (2011).
Shiraishi, H., Yamakawa, Y., Itou, A., Muraki, T., Asada, T. Long-term effects of prism adaptation on chronic neglect after stroke. NeuroRehabilitation. 23 (2), 137-151 (2008).
Yang, N. Y., Zhou, D., Chung, R. C., Li-Tsang, C. W., Fong, K. N. Rehabilitation interventions for unilateral neglect after stroke: a systematic review from 1997 through 2012. , (2013).
Rossetti, Y., et al. Prism adaptation to a rightward optical deviation rehabilitates left hemispatial neglect. Nature. 395 (6698), 166-169 (1998).
Barrett, A., Goedert, K. M., Basso, J. C. Prism adaptation for spatial neglect after stroke: translational practice gaps. Nature Reviews Neurology. 8 (10), 567-577 (2012).
Maxton, C., Dineen, R., Padamsey, R., Munshi, S. Don't neglect 'neglect'-an update on post stroke neglect. International Journal of Clinical Practice. 67 (4), 369-378 (2013).
Gammeri, R., Turri, F., Ricci, R., Ptak, R. Adaptation to virtual prisms and its relevance for neglect rehabilitation: a single-blind dose-response study with healthy participants. Neuropsychol Rehabilitation. , 1-14 (2018).
Saj, A., Cojan, Y., Assal, F., Vuilleumier, P. Prism adaptation effect on neural activity and spatial neglect depend on brain lesion site. Cortex. 119, 301-311 (2019).
Redding, G. M., Wallace, B. Generalization of prism adaptation. Journal of Experimental Psychology: Human Perception and Performance. 32 (4), 1006-1022 (2006).
Caplan, B., Mendoza, J. E. Encyclopedia of Clinical Neuropsychology. , Springer. 928 (2011).
Kim, W. S., Paik, N. J., Cho, S. 2017 International Conference on Virtual Rehabilitation (ICVR). , IEEE. 1-2 (2017).
Munafo, J., Diedrick, M., Stoffregen, T. A. The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects. Experimental Brain Research. 235 (3), 889-901 (2017).
Merhi, O. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. , SAGE Publications Sage CA. Los Angeles, CA. 2618-2622 (2018).
Duh, H. B. L., Parker, D. E., Furness, T. A. Proceedings of 9th International Conference on Human-Computer Interaction. , Citeseer. New Orleans, LA, USA. 5-10 (2018).
Prothero, J. D., Draper, M. H., Parker, D., Wells, M. The use of an independent visual background to reduce simulator side-effects. Aviation, Space, and Environmental Medicine. 70, 3 Pt 1 277-283 (1999).

Medicine