A Gesture- and Voice-controlled Virtual Reality System for Immersive Home Design: System Design and User Experience Analysis

Summary

This paper presents a virtual reality system controlled by gestures and voice for immersive home design, which improves accuracy, usability, and accessibility through the multimodal fusion of gesture and speech inputs. Experimental results indicate shorter task times, enhanced placement accuracy, and improved user satisfaction compared to traditional practice.

Abstract

Gesture- and voice-based interaction in VR has shown promise in various applications; however, existing systems are hindered by their emphasis on navigation tasks, dependence on two-handed gestures, lack of fine-grained precision, and testing on small, homogeneous populations. These limitations limit applicability to intricate, design-centric workflows. To address these limitations, we introduce a Gesture- and Voice-Controlled Virtual Reality System for immersive home design, which incorporates the multimodal fusion of hand-tracked gesture input and natural language instructions for precise furniture placement, material selection, and spatial adjustment. Three aspects of novelty in the system are (i) adaptive modes of interaction to support one-handed usage, numeric voice commands, and coarse-to-fine adjustment of precision; (ii) multimodal fusion architecture merging gesture, speech, and context information for sub-centimeter precision; and (iii) task-aware user experience evaluation framework tailored to home design processes. The development process follows a systematic pipeline, from requirements analysis and design of gesture and voice lexicons, to multimodal intent recognition, VR scene assembly with snapping and alignment, and iterative testing. Experimental results on a wide variety of participant populations demonstrate a 30% decrease in placement error and substantial improvements in usability and workload ratings over controller-based interaction. The results suggest the potential of multimodal VR for creating accessible and engaging home design experiences.

Introduction

Virtual Reality (VR) is an interactive computer-generated environment that produces a three-dimensional experience in which users can see, interact with, and communicate with technology as if they were actually present. In the context of this study, virtual reality is regarded as a multimodal interaction environment, rather than merely a visual display technique. Latest VR systems integrate visual engagement with gesture recognition, spatial senses, and natural voice input, allowing users to operate simulated items with high precision. This comprehension is consistent with current VR research that prioritizes immersion, engagement, and natural user experiences. Its ap....

Protocol

This study was approved by the Research Ethics Committee of Taylor's University, Subang Jaya, Malaysia. All procedures adhered to the ethical guidelines set by the institutional research committee and conformed to the Declaration of Helsinki. Prior to data collection, informed consent was obtained from each participant. They were clearly briefed on the study's objectives, assured that participation was voluntary, that their responses would remain confidential, and that they could withdraw at any point without any consequences. Written informed consent was also obtained for the use of anonymized interaction and survey data in academic publications. No personally identi....

Results

The designed multimodal VR home design system was validated with three fundamental datasets: EgoGesture (dynamic gestures), IPN Hand (ongoing hand gestures), and Fluent Speech Commands (voice commands with intent-slot labels). Collectively, the three datasets offered a comprehensive training and evaluation base, addressing both gesture recognition and voice-based semantic comprehension. The validation establishes that gesture-voice integration significantly enhances task accuracy and reduces workload compared to controll.......

Discussion

The results of this research advance the technology of the base paper, which mainly showed the merits of immersive VR navigation for accessibility by combining gesture and voice modalities for design tasks. The multimodal system performed better on all objective and subjective performance measures, with shorter task completion times, higher placement accuracy, and lower error rates than controller-based and hybrid approaches. Moreover, measures of usability like SUS, NASA-TLX, and IPQ noted greater user satisfaction, dec.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
EgoGesture Dataset	Nanyang Technological University	DOI: 10.1109/TMM.2018.2808769	Large-scale egocentric hand gesture dataset for dynamic gesture recognition training.
Fluent Speech Commands Dataset	Fluent.ai / University of Alberta	DOI: 10.48550/arXiv.1804.04363	Dataset for voice intent and slot-based semantic understanding for ASR/NLU training.
HaGRID Dataset (optional)	Sber AI / Open Source	v1.1	Optional dataset used for static hand gesture detector pretraining.
Igroup Presence Questionnaire (IPQ)	igroup.org	N/A	Used for measuring immersion and presence in VR environment.
IPN Hand Dataset	Universidad Veracruzana	DOI: 10.1109/CVPRW50498.2020.00241	Dataset for continuous, natural hand gesture recognition and segmentation tasks.
Matplotlib / Seaborn	Open Source	Latest Stable	Used for visualization of performance metrics and evaluation results.
MediaPipe Hands	Google Research	Open-source (GitHub)	Used for real-time hand and gesture tracking for gesture input recognition.
Mozilla Common Voice (optional)	Mozilla Foundation	Version 17	Optional dataset for pretraining ASR model on general speech data.
Multimodal Fusion Layer	Custom algorithm	N/A	Fuses gesture and voice input streams based on confidence-weighted arbitration.
NASA-TLX Workload Assessment	NASA Human Factors	N/A	Used for cognitive workload analysis of participants.
Natural Language Understanding (NLU) Module	Custom (based on BERT / RoBERTa)	N/A	Used to extract semantic intents and slots (action, object, parameter) from transcribed voice input.
NumPy / Pandas / OpenCV	Open Source	Latest Stable	Used for numerical operations, data preprocessing, and video frame manipulation.
Python Programming Language	Python Software Foundation	Version 3.10+	Used for implementing ML models (TCN, ASR, NLU, fusion).
PyTorch Deep Learning Framework	Meta AI	Version 2.0+	Used for building and training gesture recognition (TCN) and NLU models.
RGB Camera	Logitech / RealSense	Logitech C920 or Intel RealSense D435	Used for hand gesture capture and motion recognition input.
System Usability Scale (SUS) Questionnaire	Standard Evaluation Tool	N/A	Used for assessing subjective usability of VR system.
Temporal Convolutional Network (TCN)	Custom implementation (PyTorch / TensorFlow)	N/A	Used for dynamic gesture recognition from time-series skeletal data.
Unity 3D Engine	Unity Technologies	Version 2022.3 LTS	Used for developing immersive VR environment with interactive furniture, material editing, and real-time rendering.
VR Headset	Oculus (Meta) / HTC Vive	Oculus Quest 2 or HTC Vive Pro	Used for immersive visualization and spatial tracking during VR interaction.
Whisper ASR Model	OpenAI	Whisper (base model)	Automatic Speech Recognition engine for converting speech to text.
Workstation	Custom-built PC	Intel Core i7 / NVIDIA RTX 3070 / 32 GB RAM	Used for real-time processing, training, and inference of gesture and voice models.