Photorealistic Learned Landscapes for Augmented Reality

Summary

This paper presents a system for photorealistic 3D reconstruction using 360-degree images, Gaussian Splatting, and virtual reality integration. The approach can be applied to different applications such as education, simulating learning environments; construction, to simulate works off-sites and retrieve metrics; or healthcare, to train autistic people in daily-life tasks.

Abstract

The ability to create photorealistic 3D reconstructions of real-world environments from standard color images has numerous applications in fields such as healthcare, education, and industry. This paper presents a novel system that integrates advanced techniques such as structure from motion, incremental scene registration, and Gaussian Splatting to reconstruct detailed 3D models. These models are seamlessly integrated into virtual reality environments to enable immersive interaction. The pipeline begins with the capture of 360-degree images using a matrix-based pattern for comprehensive coverage. The images are processed using COLMAP to estimate camera poses and generate a sparse point cloud. Gaussian Splatting refines the reconstruction by optimizing the positions, shapes, and appearance of the points to produce highly realistic 3D environments. These models are then rendered in Unity, enabling interaction by VR headsets and supporting additional features such as avatars driven by large language models. One use case demonstrates the versatility of the system. In healthcare, the approach creates controlled, familiar virtual spaces for therapy, particularly beneficial for people with autism spectrum disorders. The immersive environments and interactive avatars provide a safe and comfortable environment for personalized therapy. This system offers significant advantages, including high visual fidelity, ease of image acquisition, and immersive user experiences. However, challenges remain, such as the computational cost of Gaussian Splatting and its reliance on accurate camera pose estimation. By overcoming these limitations, this method has the potential to transform applications in various domains, ranging from therapeutic interventions to industrial design and cultural preservation.

Introduction

The ability to recreate and interact with 3D landscapes has become critical in many different areas of modern technology applications. Traditional techniques for creating virtual representations of real locations sometimes rely on labor-intensive, rigid models and manual measurements¹. Advanced computer vision techniques have emerged as viable options for automated 3D reconstruction and interaction in order to overcome these limitations².

The development of complex 3D models from 360-degree photographs is a notable feature of this progress. Accurate spatial information can be obtained by pinpointing the exact areas in which the photos were taken using state-of-the-art tools such as COLMAP³. The process is further enhanced by Gaussian Splatting⁴, which allows complicated settings to be recreated in 3D with good quality.

Incorporating these 3D models into Virtual Reality (VR) systems opens up new avenues for exploration and engagement beyond reconstruction. To increase engagement, users can measure real-world objects such as doors and tables, move around virtual space, and even add digital objects and avatars. These avatars can produce dynamic and interactive responses using Large Language Models (LLMs), resulting in immersive experiences⁵.

This work presents a methodology to create three-dimensional representations of a real-world environment using regular color images and display them within a virtual reality setup. The system relies on general-purpose structure-from-motion and Gaussian splatting pipelines. The system is suitable for the creation of further applications in construction or healthcare.

The proposed pipeline produces visually realistic and appealing three-dimensional representations of real environments, which is its main advantage. In addition, it uses color images, which are very convenient compared to other approaches that require costly sensors such as LiDARs or structured light cameras⁶. The rationale behind the proposal is that the ability to easily create visually appealing three-dimensional representations has many applications. It can be used in construction for quality control, in healthcare to treat autistic people in a known and controlled environment, or in entertainment to include an accurate representation of a real location in a video game.

The development of systems for 3D reconstruction and interaction with virtual environments has advanced significantly in recent years, driven by the integration of computer vision, machine learning, and virtual reality technologies². The key components and methodologies relevant to this work are outlined below.

Accurate 3D reconstruction relies on image processing to extract spatial and geometric information. Traditional methods such as Structure from Motion (SfM)⁷ and Multi-View Stereo (MVS)⁸ have been widely used to estimate camera poses and generate dense point clouds. Tools such as COLMAP are recognized for their robustness in performing these tasks, combining SfM and MVS to produce highly accurate 3D models from image datasets. However, limitations remain in dealing with complex lighting, textures, and highly dynamic environments.

Recent advancements include Gaussian Splatting⁹, which uses a point-based representation to produce photorealistic reconstructions with greater efficiency than traditional mesh-based methods. Gaussian Splatting enables smoother visualization and more accurate spatial representation, particularly in scenes with intricate detail.

Once the 3D models have been reconstructed, transforming them into VR-compatible formats is crucial for immersive interaction. The use of VR systems has expanded beyond entertainment into areas such as education, healthcare¹⁰, and industrial design. Modern VR frameworks enable seamless navigation, interaction with digital objects, and enhanced realism, making them suitable for applications requiring high user engagement.

The integration of interactive features such as virtual objects and avatars significantly enhances the functionality and depth of virtual reality (VR) systems. Avatars powered by Large Language Models (LLMs) facilitate realistic conversational interactions, enabling dynamic and adaptive user experiences. In particular, these technologies have shown potential in therapeutic contexts, enabling tailored interactions based on user behavior or emotional state. For example, this study¹¹ developed a VR application using avatars as chatbots powered by LLMs to train autistic individuals in vocational communication skills, highlighting the adaptability of LLM-powered avatars in therapy. In addition, LLMs were embedded in augmented reality environments to promote inclusion and engagement, further supporting the therapeutic potential of LLM-driven avatars.

Virtual environments are increasingly being used in therapeutic settings, particularly for individuals with Autism Spectrum Disorder (ASD). Studies show that familiar and controlled virtual spaces can reduce anxiety and increase engagement during therapy sessions¹¹. Avatars in these environments can act as facilitators, delivering therapy in a way that feels safe and accessible to the patient.

Despite progress, challenges remain, including the computational cost of 3D reconstruction, maintaining high fidelity in virtual environments, and ensuring smooth integration between systems. However, the continued development of more efficient algorithms and hardware, as well as advances in AI-driven tools, provides opportunities for further innovation in this area. This paper presents a novel system for 3D reconstruction and interaction with virtual environments using 360-degree images. The approach integrates techniques such as image localization using COLMAP, high-quality reconstruction using Gaussian splatting, and VR compatibility for immersive exploration. By incorporating digital objects and interactive avatars powered by LLMs, the system enables applications in personalized therapy for individuals with ASD and construction project verification. These tools provide a comprehensive framework for creating adaptive virtual environments that enhance both therapeutic and industrial practices. Figure 1 shows the proposal pipeline.

Figure 1: System pipeline. Diagram of the proposed pipeline for photorealistic reconstruction of the environment composed of the real environment, the 360° images, the projections obtained, the SfM process, the Gaussian Splatting process, and the integration with Virtual Reality. Please click here to view a larger version of this figure.

Protocol

1. Image capture

1. Place the 360° camera on a tripod that features adjustable height. Select a series of positions within the environment to be scanned, following a square mesh pattern where each edge is spaced 1.5 m apart.
2. Capture images at three different heights at each selected point of the mesh: approximately 0.4 m, 1.2 m, and 2 m.
3. Convert the 360° images into equirectangular format images. Use the Ista360 app, for example. Select the Image, press the Export button, choose Export 360 photo mode, and export it as a 2:1 ratio image.
4. Extract 16:9 format perspective images from each equirectangular image with a 90° horizontal field of view. Use the following horizontal angles for all images: 0, 45, 90, 135, 180, 225, 270, 315. For images taken at a height of 0.4 m, use vertical angles 0, 50. For images taken at a height of 1.2 m, use vertical angles -50, 0, 50. For images taken at a height of 2 m, use vertical angles -50, 0. For the extraction process, use the Equi2Pers.py Python script (Supplementary Coding File 1; Figure 2).

Figure 2: Equirectangular images to projections. Diagram of how an equirectangular image is converted to a cube-map projection. Please click here to view a larger version of this figure.

2. Camera pose estimation with COLMAP

1. Create a new COLMAP project by clicking File > New Project, then specify the path to the images and create a new database.
2. Extract features for each image by clicking Processing > Feature Extraction, selecting PINHOLE as a camera model, and sharing for all images. Leave the remaining parameters as default. In areas with a lot of texture, max_num_features can be reduced to increase the method's efficiency.
3. Perform feature matching by clicking Processing > Feature Matching, using the default parameters.
4. Compute SfM by clicking Reconstruction > Start Reconstruction to obtain the camera positions and orientations, using the default COLMAP parameters.
5. Minimize the reprojection errors using bundle adjustment by clicking Reconstruction > Bundle Adjustment.
6. Generate a dense 3D representation of the scene by clicking Reconstruction > Dense Reconstruction, with outputs including camera poses and reconstructed points.

3. Photorealistic 3D scene reconstruction using Gaussian splatting (GS)

1. Execute the train.py file with the parameters -s, -m, and -r, where -s specifies the COLMAP project path, -m defines the output path for Gaussian Splatting (if not specified, a default one is generated), and -r rescales the images, typically set between 2 and 4.
   NOTE: The file train.py is in the Gaussian Splatting official repository https://github.com/graphdeco-inria/gaussian-splatting.
2. Locate the .ply file generated by Gaussian Splatting in the output path for later use in Unity.

4. Rendering in virtual reality with Unity and Gaussian splatting

1. Connect the VR headset to the computer. This method depends on the VR headset used.
2. Use Unity Hub to create a 3D project with version 2022.3.44f1. Install Unity version 2022.3.44f1 if necessary. Go to Projects > New Project, select version 2022.3.44f1 and the 3D (Core) template, set a project name and location, and click Create Project to generate it with default settings.
3. Install a plugin to manage VR headset and simplify application development, including tasks such as accessing the joystick and eye tracking. Do this through the Unity Asset Store using the Package Manager by clicking Window > Package Manager.
4. Use the UnityGaussianSplatting plugin to generate an asset from the Gaussian Splatting output. Do this through the Unity Asset Store using the Package Manager. To generate this asset, go to Tools > GaussianSplats > Create GaussianSplatAsset and provide the .ply file generated by Gaussian Splatting.
5. Use the UltraLeap plugin to improve hand detection, enhancing user interaction. To install this, go to Unity Asset Store > Package Manager and click Window > Package Manager.
6. Use the whisper.unity plugin to transcribe audio captured by the VR headset's microphone. Install this as described in step 4.5.
7. Use an LLM model to generate responses. Install the LLMUnity plugin to enable the use of this LLM. Install this as described in step 4.5.
   1. Create an empty GameObject and add a LLM script, then click on the download model button to select a default model. For the characters, add an empty object and attach the LLMCharacter script to it, where the name and role can be specified.
8. Use the Meta - Voice SDK to generate audio from the response produced by LLM. To achieve this, install a Text-to-Speech plugin through the Unity Asset Store using the Package Manager by clicking Window > Package Manager.
9. Use VR headsets to enable immersive interaction.

Results

In this section, representative results, strong points, and limitations of the proposed pipeline are discussed. The results of the application of the proposed method are as follows. Figure 3 shows the camera position (in red) and the dense point cloud. Groups of cameras with common origins represent projections from the same equirectangular image.

Discussion

This study presents a novel methodology for photorealistic 3D reconstruction and virtual environment interaction using 360-degree images and advanced rendering techniques. The critical steps, challenges, and implications of the proposed system are discussed below.

Critical steps
One of the key steps in the proposed pipeline is image capture. Using a 360-degree camera positioned at various heights and locations, a comprehensive dataset is generated to maximize coverage of...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
GPUs NVIDIA GeForce RTX 2080	NVIDIA	TU104-400A	Device for image processing and reconstruction
Insta360 X4	Insta360	CINSABMA-H	Data acquisition device for reconstruction
Meta Quest III	Meta	899-00582-01	Device for visualizing and interacting with the reconstruction
Memory Card Evo Plus 128 GB	Samsung	MB-MC128KA/EU	memory card for 360° photo storage