Name: Photorealistic Learned Landscapes for Augmented Reality (Video) | JoVE Journal
Uploaded: 2025-06-27T13:20:00.000+00:00
Duration: 6 min 54 s
Description: Learn photorealistic 3D reconstruction from 360-degree images using Gaussian splatting and COLMAP for immersive VR applications. This method enables therapy, education, and industrial validation with enhanced visual realism.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This paper is part of the grant PID2022-138453OB-I00 funded by MICIU/AEI/ 10.13039/501100011033 and by ERDF A way of making Europe.

1. Image capture
<ol>
	<li>Place the 360° camera on a tripod that features adjustable height. Select a series of positions within the environment to be scanned, following a square mesh pattern where each edge is spaced 1.5 m apart.</li>
	<li>Capture images at three different heights at each selected point of the mesh: approximately 0.4 m, 1.2 m, and 2 m.</li>
	<li>Convert the 360° images into equirectangular format images. Use the Ista360 app, for example. Select the Image, press the Export button, choose Export 360 photo mode, and export it as a 2:1 ratio image.</li>
	<li>Extract 16:9 format perspective images from each equirectangular image with a 90° horizontal field of view. Use the following horizontal angles for all images: 0, 45, 90, 135, 180, 225, 270, 315. For images taken at a height of 0.4 m, use vertical angles 0, 50. For images taken at a height of 1.2 m, use vertical angles -50, 0, 50. For images taken at a height of 2 m, use vertical angles -50, 0. For the extraction process, use the Equi2Pers.py Python script (Supplementary Coding File 1; Figure 2).</li>
</ol>
<img alt="Equirectangular image to cube map projection; panoramic transformation method; diagram." class="xfigimg" src="/files/ftp_upload/68386/68386fig2.jpg" data-altupdatedat="1788304762000" data-alt="Figure 2"> 
Figure 2: Equirectangular images to projections. Diagram of how an equirectangular image is converted to a cube-map projection. <a href="https://www.jove.com/files/ftp_upload/68386/68386fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
2. Camera pose estimation with COLMAP
<ol>
	<li>Create a new COLMAP project by clicking File &gt; New Project, then specify the path to the images and create a new database.</li>
	<li>Extract features for each image by clicking Processing &gt; Feature Extraction, selecting PINHOLE as a camera model, and sharing for all images. Leave the remaining parameters as default. In areas with a lot of texture, max_num_features can be reduced to increase the method's efficiency.</li>
	<li>Perform feature matching by clicking Processing &gt; Feature Matching, using the default parameters.</li>
	<li>Compute SfM by clicking Reconstruction &gt; Start Reconstruction to obtain the camera positions and orientations, using the default COLMAP parameters.</li>
	<li>Minimize the reprojection errors using bundle adjustment by clicking Reconstruction &gt; Bundle Adjustment.</li>
	<li>Generate a dense 3D representation of the scene by clicking Reconstruction &gt; Dense Reconstruction, with outputs including camera poses and reconstructed points.</li>
</ol>
3. Photorealistic 3D scene reconstruction using Gaussian splatting (GS)
<ol>
	<li>Execute the train.py file with the parameters -s, -m, and -r, where -s specifies the COLMAP project path, -m defines the output path for Gaussian Splatting (if not specified, a default one is generated), and -r rescales the images, typically set between 2 and 4. 
	NOTE: The file train.py is in the Gaussian Splatting official repository https://github.com/graphdeco-inria/gaussian-splatting.</li>
	<li>Locate the .ply file generated by Gaussian Splatting in the output path for later use in Unity.</li>
</ol>
4. Rendering in virtual reality with Unity and Gaussian splatting
<ol>
	<li>Connect the VR headset to the computer. This method depends on the VR headset used.</li>
	<li>Use Unity Hub to create a 3D project with version 2022.3.44f1. Install Unity version 2022.3.44f1 if necessary. Go to Projects &gt; New Project, select version 2022.3.44f1 and the 3D (Core) template, set a project name and location, and click Create Project to generate it with default settings.</li>
	<li>Install a plugin to manage VR headset and simplify application development, including tasks such as accessing the joystick and eye tracking. Do this through the Unity Asset Store using the Package Manager by clicking Window &gt; Package Manager.</li>
	<li>Use the UnityGaussianSplatting plugin to generate an asset from the Gaussian Splatting output. Do this through the Unity Asset Store using the Package Manager. To generate this asset, go to Tools &gt; GaussianSplats &gt; Create GaussianSplatAsset and provide the .ply file generated by Gaussian Splatting.</li>
	<li>Use the UltraLeap plugin to improve hand detection, enhancing user interaction. To install this, go to Unity Asset Store &gt; Package Manager and click Window &gt; Package Manager.</li>
	<li>Use the whisper.unity plugin to transcribe audio captured by the VR headset's microphone. Install this as described in step 4.5.</li>
	<li>Use an LLM model to generate responses. Install the LLMUnity plugin to enable the use of this LLM. Install this as described in step 4.5.
	<ol>
		<li>Create an empty GameObject and add a LLM script, then click on the download model button to select a default model. For the characters, add an empty object and attach the LLMCharacter script to it, where the name and role can be specified.</li>
	</ol>
	</li>
	<li>Use the Meta - Voice SDK to generate audio from the response produced by LLM. To achieve this, install a Text-to-Speech plugin through the Unity Asset Store using the Package Manager by clicking Window &gt; Package Manager.</li>
	<li>Use VR headsets to enable immersive interaction.</li>
</ol>

<ol><li>Zollmann, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+Reality+for+Construction+Site+Monitoring+and+Documentation%5BTitle%5D)+AND+%22Proceedings+IEEE%22%5BJournal%5D)">Augmented Reality for Construction Site Monitoring and Documentation.</a> Proceedings IEEE. 102 (2), 137-154 (2012).</li>
<li>Zhou, L., Wu, G., Zuo, Y., Chen, X., Hu, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+comprehensive+review+of+vision-based+3d+reconstruction+methods%5BTitle%5D)+AND+%22Sensors%22%5BJournal%5D)">A comprehensive review of vision-based 3d reconstruction methods.</a> Sensors. 24 (7), 2314(2024).</li>
<li>Structure-from-motion revisited. Schonberger, J. L., Frahm, J. M. IEEE Conf Comp Vis Pattern Recog, , 4104-4113 (2016).</li>
<li>Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3d+gaussian+splatting+for+real-time+radiance+field+rendering%5BTitle%5D)+AND+%22ACM+Trans.+Graph%22%5BJournal%5D)">3d gaussian splatting for real-time radiance field rendering.</a> ACM Trans. Graph. 42 (4), 139-141 (2023).</li>
<li>Embedding large language models into extended reality: Opportunities and challenges for inclusion, engagement, and privacy. Bozkir, E., et al. Proc 6th ACM Conf Conversat User Interf, , 1-7 (2024).</li>
<li>Remondino, F., El-Hakim, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Image-based+3D+modelling%3A+A+review%5BTitle%5D)+AND+%22Photogrammet+Record%22%5BJournal%5D)">Image-based 3D modelling: A review.</a> Photogrammet Record. 21 (115), 269-291 (2006).</li>
<li>Özyeşil, O., Voroninski, V., Basri, R., Singer, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+survey+of+structure+from+motion%5BTitle%5D)+AND+%22Acta+Numerica%22%5BJournal%5D)">A survey of structure from motion.</a> Acta Numerica. 26, 305-364 (2017).</li>
<li>Structure from motion using full spherical panoramic cameras. Pagani, A., Stricker, D. IEEE Int Conf Comp Vision Workshops, , 375-382 (2011).</li>
<li>Dalal, A., Hagen, D., Robbersmyr, K. G., Knausgård, K. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Gaussian+Splatting%3A+3D+Reconstruction+and+Novel+View+Synthesis%3A+A+Review%5BTitle%5D)+AND+%22IEEE+Access%22%5BJournal%5D)">Gaussian Splatting: 3D Reconstruction and Novel View Synthesis: A Review.</a> IEEE Access. 12, 96797-96820 (2024).</li>
<li>Zhang, M., Ding, H., Naumceska, M., Zhang, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Virtual+Reality+Technology+as+an+Educational+and+Intervention+Tool+for+Children+with+Autism+Spectrum+Disorder%3A+Current+Perspectives+and+Future+Directions%5BTitle%5D)+AND+%22Behav+Sci%22%5BJournal%5D)">Virtual Reality Technology as an Educational and Intervention Tool for Children with Autism Spectrum Disorder: Current Perspectives and Future Directions.</a> Behav Sci. 12 (5), 138(2023).</li>
<li>Failla, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Virtual+reality+for+autism%3A+unlocking+learning+and+growth%5BTitle%5D)+AND+%22Front+Psychol%22%5BJournal%5D)">Virtual reality for autism: unlocking learning and growth.</a> Front Psychol. 15, 1417717(2024).</li>
<li>Huang, B., Yu, Z., Chen, A., Geiger, A., Gao, S. 2D Gaussian Splatting for Geometrically Accurate Radiance. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aHuang%2C+B+%22arXiv%22">arXiv</a>. , (2024).</li>
<li>Zhang, Y., et al. Evaluating Human Perception of Novel View Synthesis: Subjective Quality Assessment of Gaussian Splatting and NeRF in Dynamic Scenes. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aZhang%2C+Y+%22arXiv%22">arXiv</a>. , (2025).</li>
</ol>

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

This study presents a novel methodology for photorealistic 3D reconstruction and virtual environment interaction using 360-degree images and advanced rendering techniques. The critical steps, challenges, and implications of the proposed system are discussed below.
Critical steps 
One of the key steps in the proposed pipeline is image capture. Using a 360-degree camera positioned at various heights and locations, a comprehensive dataset is generated to maximize coverage of...

The ability to recreate and interact with 3D landscapes has become critical in many different areas of modern technology applications. Traditional techniques for creating virtual representations of real locations sometimes rely on labor-intensive, rigid models and manual measurements1. Advanced computer vision techniques have emerged as viable options for automated 3D reconstruction and interaction in order to overcome these limitations2.
The development of complex 3D models from 360-degree photographs is a notable feature of this progress. Accurate spatial information can be obtained by pinpointing the exact areas in which the photos were taken using state-of-the-art tools such as COLMAP3. The process is further enhanced by Gaussian Splatting4, which allows complicated settings to be recreated in 3D with good quality.
Incorporating these 3D models into Virtual Reality (VR) systems opens up new avenues for exploration and engagement beyond reconstruction. To increase engagement, users can measure real-world objects such as doors and tables, move around virtual space, and even add digital objects and avatars. These avatars can produce dynamic and interactive responses using Large Language Models (LLMs), resulting in immersive experiences5.
This work presents a methodology to create three-dimensional representations of a real-world environment using regular color images and display them within a virtual reality setup. The system relies on general-purpose structure-from-motion and Gaussian splatting pipelines. The system is suitable for the creation of further applications in construction or healthcare.
The proposed pipeline produces visually realistic and appealing three-dimensional representations of real environments, which is its main advantage. In addition, it uses color images, which are very convenient compared to other approaches that require costly sensors such as LiDARs or structured light cameras6. The rationale behind the proposal is that the ability to easily create visually appealing three-dimensional representations has many applications. It can be used in construction for quality control, in healthcare to treat autistic people in a known and controlled environment, or in entertainment to include an accurate representation of a real location in a video game.
The development of systems for 3D reconstruction and interaction with virtual environments has advanced significantly in recent years, driven by the integration of computer vision, machine learning, and virtual reality technologies2. The key components and methodologies relevant to this work are outlined below.
Accurate 3D reconstruction relies on image processing to extract spatial and geometric information. Traditional methods such as Structure from Motion (SfM)7 and Multi-View Stereo (MVS)8 have been widely used to estimate camera poses and generate dense point clouds. Tools such as COLMAP are recognized for their robustness in performing these tasks, combining SfM and MVS to produce highly accurate 3D models from image datasets. However, limitations remain in dealing with complex lighting, textures, and highly dynamic environments.
Recent advancements include Gaussian Splatting9, which uses a point-based representation to produce photorealistic reconstructions with greater efficiency than traditional mesh-based methods. Gaussian Splatting enables smoother visualization and more accurate spatial representation, particularly in scenes with intricate detail.
Once the 3D models have been reconstructed, transforming them into VR-compatible formats is crucial for immersive interaction. The use of VR systems has expanded beyond entertainment into areas such as education, healthcare10, and industrial design. Modern VR frameworks enable seamless navigation, interaction with digital objects, and enhanced realism, making them suitable for applications requiring high user engagement.
The integration of interactive features such as virtual objects and avatars significantly enhances the functionality and depth of virtual reality (VR) systems. Avatars powered by Large Language Models (LLMs) facilitate realistic conversational interactions, enabling dynamic and adaptive user experiences. In particular, these technologies have shown potential in therapeutic contexts, enabling tailored interactions based on user behavior or emotional state. For example, this study11 developed a VR application using avatars as chatbots powered by LLMs to train autistic individuals in vocational communication skills, highlighting the adaptability of LLM-powered avatars in therapy. In addition, LLMs were embedded in augmented reality environments to promote inclusion and engagement, further supporting the therapeutic potential of LLM-driven avatars.
Virtual environments are increasingly being used in therapeutic settings, particularly for individuals with Autism Spectrum Disorder (ASD). Studies show that familiar and controlled virtual spaces can reduce anxiety and increase engagement during therapy sessions11. Avatars in these environments can act as facilitators, delivering therapy in a way that feels safe and accessible to the patient.
Despite progress, challenges remain, including the computational cost of 3D reconstruction, maintaining high fidelity in virtual environments, and ensuring smooth integration between systems. However, the continued development of more efficient algorithms and hardware, as well as advances in AI-driven tools, provides opportunities for further innovation in this area. This paper presents a novel system for 3D reconstruction and interaction with virtual environments using 360-degree images. The approach integrates techniques such as image localization using COLMAP, high-quality reconstruction using Gaussian splatting, and VR compatibility for immersive exploration. By incorporating digital objects and interactive avatars powered by LLMs, the system enables applications in personalized therapy for individuals with ASD and construction project verification. These tools provide a comprehensive framework for creating adaptive virtual environments that enhance both therapeutic and industrial practices. Figure 1 shows the proposal pipeline.
<img alt="Virtual reality process diagram with real world to VR via 360 images, COLMAP, Gaussian splatting." class="xfigimg" src="/files/ftp_upload/68386/68386fig1.jpg" data-altupdatedat="1788304762000" data-alt="Figure 1"> 
Figure 1: System pipeline. Diagram of the proposed pipeline for photorealistic reconstruction of the environment composed of the real environment, the 360° images, the projections obtained, the SfM process, the Gaussian Splatting process, and the integration with Virtual Reality. <a href="https://www.jove.com/files/ftp_upload/68386/68386fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table><tbody><tr><td>GPUs NVIDIA GeForce RTX 2080</td><td>NVIDIA</td><td>TU104-400A</td><td>Device for image processing and reconstruction&amp;nbsp;</td></tr><tr><td>Insta360 X4</td><td>Insta360</td><td>CINSABMA-H</td><td>Data acquisition device for reconstruction</td></tr><tr><td>Meta Quest III</td><td>Meta</td><td>899-00582-01</td><td>Device for visualizing and interacting with the reconstruction</td></tr><tr><td>Memory Card Evo Plus 128 GB</td><td>Samsung</td><td>MB-MC128KA/EU</td><td>memory card for 360&amp;deg; photo storage</td></tr></tbody></table>

photorealistic learned landscapes for augmented reality

Photorealistic Learned Landscapes for Augmented Reality

The ability to create photorealistic 3D reconstructions of real-world environments from standard color images has numerous applications in fields such as healthcare, education, and industry. This paper presents a novel system that integrates advanced techniques such as structure from motion, incremental scene registration, and Gaussian Splatting to reconstruct detailed 3D models. These models are seamlessly integrated into virtual reality environments to enable immersive interaction. The pipeline begins with the capture of 360-degree images using a matrix-based pattern for comprehensive coverage. The images are processed using COLMAP to estimate camera poses and generate a sparse point cloud. Gaussian Splatting refines the reconstruction by optimizing the positions, shapes, and appearance of the points to produce highly realistic 3D environments. These models are then rendered in Unity, enabling interaction by VR headsets and supporting additional features such as avatars driven by large language models. One use case demonstrates the versatility of the system. In healthcare, the approach creates controlled, familiar virtual spaces for therapy, particularly beneficial for people with autism spectrum disorders. The immersive environments and interactive avatars provide a safe and comfortable environment for personalized therapy. This system offers significant advantages, including high visual fidelity, ease of image acquisition, and immersive user experiences. However, challenges remain, such as the computational cost of Gaussian Splatting and its reliance on accurate camera pose estimation. By overcoming these limitations, this method has the potential to transform applications in various domains, ranging from therapeutic interventions to industrial design and cultural preservation.

In this section, representative results, strong points, and limitations of the proposed pipeline are discussed. The results of the application of the proposed method are as follows. Figure 3 shows the camera position (in red) and the dense point cloud. Groups of cameras with common origins represent projections from the same equirectangular image.
<img alt="3D point cloud reconstruction, data visualization, red segments h...

This paper presents a system for photorealistic 3D reconstruction using 360-degree images, Gaussian Splatting, and virtual reality integration. The approach can be applied to different applications such as education, simulating learning environments; construction, to simulate works off-sites and retrieve metrics; or healthcare, to train autistic people in daily-life tasks.

The ability to create photorealistic 3D reconstructions of real-world environments from standard color images has numerous applications in fields such as ...

photorealistic-learned-landscapes-for-augmented-reality

Research

JoVE Journal

Engineering

Source:Carly Fassler1, Liyu Huang1, Kavita Prasad1, Jiayi Xu2, Guansen Tong2, Jie Ying Wu2, Michael C. Topf1,31Department of Otolaryngology – Head and Neck Surgery, Vanderbilt University Medical Center.2Computer Science Department, Vanderbilt University.3Vanderbilt University School of Engineering, Vanderbilt University.This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

This study presents an experimental paradigm for a usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

This protocol offers a guide to implement infrared marker tracking for free-moving phantoms (e.g., organs) and holographic visualization using Augmented Reality. Additionally, it outlines a setup for preclinical validation of holographic navigation systems using electromagnetic tracking on free-moving phantoms.

Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation

Organic photovoltaic (OPV) materials are inherently inhomogeneous at the nanometer scale. Nanoscale inhomogeneity of OPV materials affects performance of photovoltaic devices. In this paper, we describe a protocol for quantitative measurements of electrical and mechanical properties of OPV materials with sub-100 nm resolution.

Concurrent Quantitative Conductivity and Mechanical Properties Measurements of Organic Photovoltaic Materials using AFM

We report on a process of in situ alteration of HF treated Si (001) surface into a hydrophilic or hydrophobic state by irradiating samples in microfluidic chambers filled with H2O2/H2O solution (0.01%-0.5%) or methanol solutions using pulsed UV laser of a relative low pulse fluence.

Selective Area Modification of Silicon Surface Wettability by Pulsed UV Laser Irradiation in Liquid Environment

Because of the importance and extensive use of palladium, gold and cobalt metals in high-technology equipment, their recovery and recycling constitute an important industrial challenge. The metal recovery system described herein is a simple, low-cost means for the effective detection, removal, and recovery of these metals from the urban mine.

Detection and Recovery of Palladium, Gold and Cobalt Metals from the Urban Mine Using Novel Sensors/Adsorbents Designated with Nanoscale Wagon-wheel-shaped Pores

We describe a technique for inscribing identical fiber Bragg gratings into each core of a multicore fiber. This is achieved by introducing an additional surface into the optical path to mitigate lensing by the curved surface of the fiber cladding.

Writing Bragg Gratings in Multicore Fibers

Experimental methods for investigation of solid state cooling processes and characterization of elastocaloric material properties of Shape Memory Alloys (SMA) are presented. A custom-built test rig has been designed for controlling and comprehensive monitoring of elastocaloric cooling processes. Furthermore, it provides a validation platform for thermomechanically coupled modeling approaches.

Experimental Methods for Investigation of Shape Memory Based Elastocaloric Cooling Processes and Model Validation

We demonstrate the precise manipulation of individual organic molecules on a metal surface with the tip of a scanning probe microscope driven in 3D by the experimenter's hand using a motion capture system and fully immersive virtual reality goggles.

Hand Controlled Manipulation of Single Molecules via a Scanning Probe Microscope with a 3D Virtual Reality Interface

Learn photorealistic 3D reconstruction from 360-degree images using Gaussian splatting and COLMAP for immersive VR applications. This method enables therapy, education, and industrial validation with enhanced visual realism.

Photorealistic Learned Landscapes for Augmented Reality

Explore More Videos