Photorealistic Learned Landscapes for Augmented Reality

Summary

This paper presents a system for photorealistic 3D reconstruction using 360-degree images, Gaussian Splatting, and virtual reality integration. The approach can be applied to different applications such as education, simulating learning environments; construction, to simulate works off-sites and retrieve metrics; or healthcare, to train autistic people in daily-life tasks.

Abstract

The ability to create photorealistic 3D reconstructions of real-world environments from standard color images has numerous applications in fields such as healthcare, education, and industry. This paper presents a novel system that integrates advanced techniques such as structure from motion, incremental scene registration, and Gaussian Splatting to reconstruct detailed 3D models. These models are seamlessly integrated into virtual reality environments to enable immersive interaction. The pipeline begins with the capture of 360-degree images using a matrix-based pattern for comprehensive coverage. The images are processed using COLMAP to estimate camera poses and generate a sparse point cloud. Gaussian Splatting refines the reconstruction by optimizing the positions, shapes, and appearance of the points to produce highly realistic 3D environments. These models are then rendered in Unity, enabling interaction by VR headsets and supporting additional features such as avatars driven by large language models. One use case demonstrates the versatility of the system. In healthcare, the approach creates controlled, familiar virtual spaces for therapy, particularly beneficial for people with autism spectrum disorders. The immersive environments and interactive avatars provide a safe and comfortable environment for personalized therapy. This system offers significant advantages, including high visual fidelity, ease of image acquisition, and immersive user experiences. However, challenges remain, such as the computational cost of Gaussian Splatting and its reliance on accurate camera pose estimation. By overcoming these limitations, this method has the potential to transform applications in various domains, ranging from therapeutic interventions to industrial design and cultural preservation.

Introduction

The ability to recreate and interact with 3D landscapes has become critical in many different areas of modern technology applications. Traditional techniques for creating virtual representations of real locations sometimes rely on labor-intensive, rigid models and manual measurements¹. Advanced computer vision techniques have emerged as viable options for automated 3D reconstruction and interaction in order to overcome these limitations².

The development of complex 3D models from 360-degree photographs is a notable feature of this progress. Accurate spatial information can be obtained by pinpo....

Protocol

1. Image capture

1. Place the 360° camera on a tripod that features adjustable height. Select a series of positions within the environment to be scanned, following a square mesh pattern where each edge is spaced 1.5 m apart.
2. Capture images at three different heights at each selected point of the mesh: approximately 0.4 m, 1.2 m, and 2 m.
3. Convert the 360° images into equirectangular format images. Use the Ista360 app, for example. Select the Image, press the Export button, choose Export 360 photo mode, and export it as a 2:1 ratio image.
4. Extract 16:9 format perspe....

Results

In this section, representative results, strong points, and limitations of the proposed pipeline are discussed. The results of the application of the proposed method are as follows. Figure 3 shows the camera position (in red) and the dense point cloud. Groups of cameras with common origins represent projections from the same equirectangular image.

Discussion

This study presents a novel methodology for photorealistic 3D reconstruction and virtual environment interaction using 360-degree images and advanced rendering techniques. The critical steps, challenges, and implications of the proposed system are discussed below.

Critical steps
One of the key steps in the proposed pipeline is image capture. Using a 360-degree camera positioned at various heights and locations, a comprehensive dataset is generated to maximize coverage of.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
GPUs NVIDIA GeForce RTX 2080	NVIDIA	TU104-400A	Device for image processing and reconstruction
Insta360 X4	Insta360	CINSABMA-H	Data acquisition device for reconstruction
Meta Quest III	Meta	899-00582-01	Device for visualizing and interacting with the reconstruction
Memory Card Evo Plus 128 GB	Samsung	MB-MC128KA/EU	memory card for 360° photo storage