Photorealistic Learned Landscapes for Augmented Reality

Our research focuses on creating photorealistic 3D reconstructions from 360 degree images to build immersive virtual environments. It explores how this can support therapy, education, and industrial validation. Common approaches include extraction from motions, and multiview exterior, often using tools like COLMAP. Recent methods like offer faster, more visually realistic results for immersive applications.

Challenge includes the computational costs of passionate plotting, hardly low test stream, and inserting a graphic camera post estimation, which directly affect the cost and quality. We address the lack of totalistic 3D custodian system that interact slowly into B arc and support dynamic usage right interacting with accurate sensor.

[Narrator] To begin, place a 360 degree camera on a tripod that has adjustable height. Select a series of positions within the environment for scanning, using a square mesh pattern where each edge is spaced 1.5 meters apart. At each mesh point, capture images at three different heights of approximately 0.4 meters, 1.2 meters, and two meters. Convert the 360 degree images to equi-rectangular format, using a tool like the Insta 360 app. Select the image, press the export button. Choose export 360 photo mode and export it as a two to one ratio image. Use the Equi2Pers.py script to extract 16 to nine format perspective images from each equi-rectangular image with a 90 degree horizontal field of view. Apply horizontal angles of zero, 45, 90, 135, 180, 225, 270, and 315 degrees, and vertical angles based on height. Next, click on file a new project to create a new COLMAP project. Specify the path to the images and create a new database. Click on processing, followed by feature extraction to extract features for each image. Select pinhole as the camera model and share all the images. Leave remaining parameters as default compute structure for motion by clicking on reconstruction and start reconstruction to obtain the camera positions and orientations, using the default COLMAP parameters. Click on reconstruction and choose bundle adjustment to minimize the reprojection errors. Now generate a dense 3D seen representation by choosing dense reconstruction with outputs, including camera poses and reconstructed points. For photo realistic 3D seen reconstruction, using Gaussian splatting, execute the train.py script, using the parameters minus S, minus M, and minus R. Locate the generated .py file within the specified output directory for subsequent import into Unity. Connect the virtual reality headset to the computer, following the specific instructions for the headset model used. Use Unity Hub to create a 3D project with version 2022.44f1. Navigate to projects, click new project. Select the 3D built-in render pipeline template. Set the project name and location and click create project. To manage the VR headset and simplify development tasks, install a plugin from the Unity Asset Store via the package manager by clicking window and package manager. Use the Unity Gaussian splatting plugin to convert the Gaussian splatting output into a usable asset. Improve hand tracking by installing the ultra elite plugin via the package manager from the Unity Asset Store. Transcribe audio from the VR headset microphone, using the whisper.unity plugin. Install it using package manager. Enable response generation, using a large language model by installing the LLM Unity plugin. Install it through the package manager as demonstrated earlier. Generate speech from LLM generated responses, using the meta voice SDK. Install a text to speech plugin from the Unity Asset Store via the package manager by clicking window and package manager. Finally, use the VR headset to experience and interact with the immersive environment. Groups of camera positions derived from shared equi-rectangular origins were used to generate dense point clouds for seen reconstruction, revealing a consistent spatial mapping of capture angles. The proposed method, using Gaussian splatting, produced a photorealistic reconstruction, closely resembling the real environment. Users could effectively interact with the reconstructed environment through virtual reality, maintaining immersion and spatial awareness, with visuals presented inside the headset matching the room setup. Familiar and unfamiliar virtual environments were developed with therapeutic goals in mind, based on feedback from professional therapists. A virtual agent was rendered in the reconstructed space, allowing users to engage in realistic interactive scenarios through VR, with the agent appearing as a lifelike figure in the headset view. Virtual reconstructions replicated specific viewpoints, accurately, when based on input images, but deviations in perspective resulted in noticeable rendering limitations. Compared to COLMAP point cloud output, Gaussian splatting produced more visually continuous and lifelike reconstruction, suitable for real-time interaction, albeit with reduced metric precision.