Integrating Terrestrial Laser Scanning, Generative AI, and Mixed Reality for Digital Heritage Reconstruction

Summary

This replicable protocol demonstrates a workflow for digital heritage reconstruction. It integrates Terrestrial Laser Scanning to capture site geometry, generative AI to reconstruct lost cultural elements from archival sources, and Mixed Reality to visualize these synthesized elements in situ.

Abstract

The rapid acceleration of urbanization in developing regions has precipitated a dual crisis for architectural heritage: the physical degradation of structures and the intangible dissipation of cultural memory. Traditional conservation methodologies, which rely predominantly on static documentation, such as photography and manual surveying, are increasingly insufficient for capturing the complex spatial and temporal dimensions of these historic sites. This study introduces a novel, combinatorial methodological framework designed to bridge the gap between rigorous archiving and dynamic public engagement. We detail a workflow that synergizes three distinct technologies: Terrestrial Laser Scanning (TLS) to capture the tangible geometric attributes of the building with millimeter-level accuracy; Conditional Generative Artificial Intelligence (CGAI) to generate historically informed visual representations of lost cultural elements (specifically, traditional temple fair structures) based on archival references; and Augmented Reality (AR) via holographic headsets to superimpose these layers into a unified Mixed Reality (MR) experience. The protocol describes the complete pipeline from data acquisition strategies and point cloud processing to AI-driven model generation and final deployment. This approach yields a dynamic digital simulacrum that successfully overlays high-precision spatial data with interpretive cultural visualizations, offering a technically feasible and scalable workflow for the digital preservation and augmented exhibition of urban heritage landmarks.

Introduction

The preservation of architectural heritage constitutes a critical imperative within the contemporary discourse of urban planning and cultural sustainability^1,2. As cities undergo rapid modernization and digitization, the tension between development and conservation becomes increasingly acute³. This phenomenon is particularly evident in the Central Plains of China, a region that serves as a cradle of civilization yet faces the relentless pressure of urban expansion^4,5. The protection of cultural heritage extends beyond the mere maintenance of physical relics; it encompasses the safeguarding of historical context, the transmission of cultural significance, and the preservation of the authenticity of the heritage site⁶. Heritage conservation is not a passive act of archiving but an active strategy for cultural rejuvenation, shaping the identity of nations and regions while enabling future generations to maintain a continuous connection with their ancestral spiritual world⁷. Furthermore, the effective preservation of these sites fosters a multidimensional cultural ecology that interacts dynamically with the global economy, enhancing the region’s image on an international stage⁸.

Despite the recognized importance of heritage conservation, current research and funding often disproportionately favor prominent, monumental sites such as the Shaolin Temple or the historic centers of Luoyang and Kaifeng⁹. This leaves regionally significant but less globally visible landmarks, such as the City God Temple of Zhengzhou, in a state of marginalization¹⁰. Constructed during the Ming Dynasty, the City God Temple, as shown in Figure 1, represents a singular example of well-preserved religious architecture in Zhengzhou. Historically, it functioned not merely as a religious sanctuary but as a vibrant nexus of social, economic, and folk activities—most notably the Temple Fair¹¹. These fairs were the lifeblood of the temple, integrating sacrifice, commerce, and entertainment into a cohesive cultural expression¹². However, the encroachment of modern commercialism and urban rezoning has led to the cessation of these traditional fairs, resulting in a conservation dilemma. While the physical skeleton of the temple remains, the intangible spirit—the noise, the visual vibrancy, and the social interaction—has largely dissipated. This loss of place (space plus social function) underscores the urgent need for conservation methods that can restore both the tangible structure and the intangible atmosphere^13,14. Therefore, the protocol detailed in this study is explicitly scoped and designed for heritage sites characterized by this specific dichotomy: locations where the physical architectural structures remain structurally intact, but the associated ephemeral cultural context and intangible artifacts have been lost.

Figure 1. Photograph of the City God Temple of Zhengzhou showing the current state of the entrance facade. Please click here to view a larger version of this figure.

The deployment of digital technology in cultural heritage conservation offers a transformative solution to this dilemma. The principle of conservation demands maintaining the chronological value of the material ontology⁶. Traditional analog methods, such as hand measurement and 2D photography, often fail to document the complex geometries of ancient architecture comprehensively and are powerless to restore lost ephemeral scenes. Consequently, the field has shifted towards “Digital Twinning,” a concept originally derived from aerospace engineering, which involves creating a virtual replica of a physical entity that spans its lifecycle¹⁵. In the context of heritage, a digital twin serves as a precise, unalterable archive that supports diagnosis, prediction, and interaction¹⁶.

Terrestrial Laser Scanning (TLS) has emerged as the foundational technology for constructing the geometric layer of these digital twins. TLS utilizes LiDAR (Light Detection and Ranging) to capture millions of data points per second, creating a high-density point cloud that records every fissure, tilt, and texture of a building with millimeter-level accuracy¹⁷. This technology is non-contact, making it ideal for fragile structures, and highly efficient compared to total station surveying. Recent applications have demonstrated the efficacy of TLS in complex scenarios: Marsella et al.¹⁸ utilized TLS to assess the structural safety of St. Peter’s Basilica in the Vatican, identifying minute deformations invisible to the naked eye. Similarly, Barontini et al.¹⁹ developed a Historic Building Information Modeling (HBIM) framework that integrates TLS data for preventive conservation. The ability of TLS to diagnose technical conditions—such as wall cracking, foundation settlement, and floor deflection—provides a scientific basis for physical restoration interventions^20,21.

However, while TLS excels at recording the extant physical reality, it cannot restore the vanished historical reality. A point cloud is accurate, but it is static and devoid of historical context. To reconstruct the missing historical scenes—such as the bustling atmosphere of the temple fair—this study turns to the emerging field of Conditional Generative Artificial Intelligence (CGAI). Unlike traditional manual 3D modeling, which is labor-intensive and relies heavily on the subjective interpretation of the modeler, CGAI (e.g., Stable Diffusion or Tencent Huiyuan 3D) utilizes deep learning algorithms to generate visual content based on textual descriptions²². By analyzing vast datasets of historical architectural styles and textures, CGAI can synthesize or reconstruct plausible historical elements—lanterns, banners, temporary stalls—based on archival text and visual data. This represents a paradigm shift from purely recording what exists to simulating what historically existed, using data-driven logic to fill the gaps in the physical archive²³.

The final challenge lies in the dissemination of this digital simulacrum. A digital twin stored on a server has limited social value if it is not accessible to the public. Augmented Reality (AR) and Mixed Reality (MR) technologies bridge the physical and virtual worlds, allowing the digital twin to be superimposed onto the physical site²⁴. Unlike Virtual Reality (VR), which isolates the user in a completely synthetic environment, AR maintains the connection to the physical heritage site while enhancing it with digital information. Platforms like Fologram and devices like the Microsoft HoloLens 2 enable the projection of high-fidelity 3D models into the real-world coordinate system, facilitating a direct dialogue between the visitor and the reconstructed history²⁵. This immersive approach has been shown to significantly enhance visitor engagement and understanding, transforming passive viewing into active exploration²⁶.

While current literature often treats digital conservation technologies in isolation, the integrated protocol proposed in this article offers distinct methodological novelty. First, unlike standard TLS documentation workflows that yield highly accurate but culturally sterile geometric archives, our method introduces dynamic semantic context. Second, while photogrammetry-based reconstructions excel at texturing extant physical surfaces, they are structurally incapable of regenerating completely lost cultural artifacts that lack physical remnants. Finally, in contrast to existing AR heritage visualization systems that frequently rely on manually crafted, generic 3D assets or isolate users in VR, this workflow utilizes CGAI to generate site-specific, historically informed visualizations directly anchored to millimeter-accurate TLS data. This fills a critical gap by detailing a rigorous, reproducible protocol. To resolve conceptual ambiguities and establish a clear methodological rationale, we explicitly define the core components of this workflow. Documented geometry refers to the highly precise, static 3D spatial replica of the existing physical architecture; TLS is required here to capture this undeniable empirical reality with non-contact precision. In contrast, AI-generated interpretive reconstruction denotes the visual representations of lost cultural artifacts (e.g., festival decorations) derived from archival text; CGAI is required for this step because it synthesizes probabilistic, historically informed interpretations rather than absolute physical truths. Finally, MR is required for in-situ visualization, serving as the experiential medium that seamlessly superimposes the interpretive cultural layers onto the documented physical geometry in the real world. By structuring our approach around these defined concepts, this combinatorial workflow not only provides a technical route not only provides a technical route for the digitization of urban cultural heritage in the Central Plains but also offers a new theoretical model for how we define “preservation” in the digital age—moving beyond the preservation of matter to the revitalization of memory^27,28.

Protocol

1. Historical data collection and site analysis

1. Initiate the project by conducting a comprehensive archival review. Retrieve historical records, photographs, architectural drawings, and textual descriptions related to the target heritage site (in this case, the City God Temple of Zhengzhou) from local archives, temple management offices, and relevant cultural heritage bureaus.
2. Analyze the textual data specifically to identify descriptions of lost ephemeral elements, such as temple fair decorations, temporary stalls, banners, and specific lighting arrangements (e.g., “red lanterns,” “dragon dances”).
3. Conduct a preliminary field investigation during daylight hours to understand the site topography and document the current architectural preservation state (Figure 2). Document the layout of the complex, noting the central axis, the location of the Main Hall, Bell and Drum Towers, and potential obstacles such as ancient trees or narrow corridors that may obstruct laser scanning lines of sight.

Figure 2. Current view of the temple courtyard captured during the preliminary site investigation. Please click here to view a larger version of this figure.

4. Develop a detailed scanning station strategy / deployment plan. Identify specific coordinate locations for the Terrestrial Laser Scanner (TLS) to ensure 360° coverage of all structures. Plan for a minimum of 12 stations, with the flexibility to expand to 18 stations based on site complexity.
5. Obtain necessary administrative permissions to access the site during non-public hours (typically early morning 07:00–08:30 or late afternoon 17:00–18:30) to minimize data noise caused by pedestrian traffic.

2. Terrestrial laser scanning (TLS) acquisition

1. Deploy a high-precision phase-shift terrestrial laser scanner. Verify that the battery charge is sufficient for the duration of the scan and that the SD storage card has adequate capacity.
2. Set up the scanner tripod at the first pre-determined station point (Start with the Main Hall/Zone A). Adjust the tripod legs to ensure stability on the stone pavement.
3. Level the scanner using the built-in bubble level or the digital inclinometer on the device interface. Ensure the inclination is less than 0.5° to guarantee the verticality of the point cloud data.
4. Configure the scanning parameters via the onboard touchscreen interface:
   1. Set the Resolution to 1/4 or 1/5 (depending on time constraints), ensuring a point spacing of approximately 6 mm at 10 m.
   2. Set the Quality to 3x or 4x to reduce noise.
   3. Enable Color capture (Integrated Camera) to overlay Red, Green, Blue (RGB) texture onto the geometry.
   4. Ensure the scanning range covers 0.6 m to 120 m with a ranging accuracy of ±2 mm.
5. Execute a pilot scan of a small section (e.g., the Bell Tower) to verify lighting conditions. Check the preview for overexposure in outdoor areas or underexposure in shadowed eaves.
6. Proceed with the full scanning sequence. For each station, trigger the scan and retreat from the scanner’s field of view to avoid casting shadows or appearing in the data (ghosting).
7. Move the scanner to the next station, ensuring a minimum overlap of 30–40% with the previous station. This overlap is critical for the cloud-to-cloud registration process later.
8. Capture high-resolution reference photographs using an external Digital Single-Lens Reflex (DSLR) camera at each station to supplement the scanner’s internal camera, especially for detailed carvings or calligraphic plaques.

3. Point cloud data processing

1. Transfer the raw scan data (.fls format) from the SD card to a high-performance workstation running processing software.
2. Import the raw scans into the software project workspace.
3. Apply pre-processing filters to the raw data: Use the Stray Point Filter to remove dust and flying insects.
4. Use the Dark Scan Point Filter to remove erroneous data from highly reflective or absorptive surfaces.
5. Execute automatic registration. Allow the software to align the scans based on geometric feature recognition and GPS data.
6. Inspect the registration report. Verify that the mean tension error (average distance between corresponding points) is below 4mm. If the error exceeds the threshold, perform manual registration. Select three distinct common points (e.g., corners of a building, pillars) in two overlapping scans to force alignment.
7. Apply the Moving Object Filter to automatically detect and remove transient noise such as pedestrians or birds that moved through the scene during scanning.
8. Perform Color Balancing (Multi-exposure adjustment) to equalize the brightness between the bright sky and the dark temple interiors.
9. Create a project point cloud and export the final model as a structured .E57 or .OBJ file. Ensure the file size is manageable (e.g., downsample to a grid spacing of 5mm if the file exceeds 10GB).

4. Conditional generative AI (CGAI) modeling

1. Identify the specific historical elements to be reconstructed based on the analysis in Step 1.2 (e.g., traditional temple fair stalls, lanterns).
2. Utilize a selected CGAI platform.
3. Construct precise text prompts by directly translating semantic descriptors and historical records found in archival sources (e.g., the Annals of Zheng County) into visual parameters. For example, input: "Traditional Chinese Ming Dynasty red lantern, glowing, hanging from eaves, high detail" or "Wooden market stall, ancient style, cloth banner, 3D texture."
4. Execute the generation process. Typically, produce a batch of 20–30 iterations for each historical element to ensure a sufficient sample size of stylistic variations and structural possibilities.
5. Convene a formal validation panel comprising heritage historians and domain experts before integrating any generated assets into the final MR visualization. Select historically plausible outputs by cross-referencing the AI batches against surviving visual archives and established historical consensus.
6. Discard any results that exhibit anachronistic features or structural impossibilities. Iterate the text prompts based on the expert panel's explicit feedback to refine the historical accuracy of the models.

5. Integration and digital twin construction

1. Open the 3D modeling environment.
2. Import the processed TLS point cloud (.OBJ) into the 3D modeling software.
3. Import the CGAI-generated assets.
4. Manually align and position the CGAI assets within the point cloud environment. Place the lanterns under the eaves and position the stalls in the courtyard space according to the historical layout analysis.
5. Use the software’s modeling tools to clean up any mesh intersections or scale discrepancies between the point cloud and the AI models.
6. Texture the combined model. Apply the High Dynamic Range (HDR) images from the scanner to the point cloud mesh and ensure the AI assets have appropriate material properties (roughness, metallicity).
7. Export the fully integrated Digital Twin model as a .GLTF (Graphics Language Transmission Format) file. This format is optimized for web and AR applications.

6. Augmented reality (AR) deployment

1. Install MR deployment plugin for the 3D modeling software on the workstation.
2. Install the companion app on the Mixed Reality headset.
3. Connect both the workstation and the MR headset to the same local 5GHz Wi-Fi network to ensure low-latency data transmission.
4. Launch the MR plugin in the modeling software to generate a session QR code.
5. Don the MR headset and launch the companion app. Scan the QR code to synchronize the session.
6. Evaluate on-site environmental conditions prior to deployment. Strictly schedule MR visualization sessions during early morning or late afternoon to avoid strong infrared interference from the noon sun, which severely degrades the headset's spatial tracking capabilities and causes model drift.
7. Perform on-site model alignment and establish spatial registration strategies:
   1. Deploy physical tracking markers (e.g., ArUco markers or QR codes) at stable anchor points in the real world, such as the stone base of the main incense burner. We explicitly recommend marker-based tracking over purely spatial environment mapping to counteract infrared interference from direct sunlight, ensuring model stability under challenging daytime lighting and acting as a robust fallback if natural spatial anchors fail.
   2. Utilize the headset's optical sensors to detect the markers and automatically lock the virtual model into the correct coordinate system, using manual hand gestures in the headset only for minor rotational fine-tuning until the digital point cloud perfectly overlaps the physical structure.
8. Lock the model position.
9. Walk through the site to verify registration stability. Ensure the alignment accuracy threshold is strictly maintained, tolerating a maximum visual drift of only 2–5 cm during active user movement to preserve the immersive illusion.

Results

The terrestrial laser scanning campaign, executed in November 2024, utilized 18 scan stations to achieve comprehensive coverage of the Main Hall, Entrance Plaza, and Apse zones. As detailed in Table 1, the scanning parameters were strictly calibrated—specifically utilizing a resolution setting of 1/4 and a quality setting of 4x—to yield a point distance of approximately 6.1 mm at a 10-meter range. This specific calibration was essential to balance high data density with on-site acquisition efficiency. Co...

Discussion

This study successfully demonstrates a combinatorial methodological framework for the digital renaissance of architectural heritage. By integrating Terrestrial Laser Scanning (TLS), Conditional Generative AI (CGAI), and Augmented Reality (AR), we have established a workflow that transcends the limitations of traditional static preservation, as summarized in the methodological framework (Figure 8).

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
DSLR camera	High-resolution reference photography	N/A
FARO Focus3D S120	Terrestrial Laser Scanner (TLS)	FARO Technologies
FARO SCENE 2019	Point cloud data processing software	FARO Technologies
Fologram	MR development plugin and headset application	Fologram
High-performance workstation	Data processing and rendering	N/A
Microsoft HoloLens 2	Mixed Reality (MR) headset	Microsoft
Rhino 8	3D modeling environment	Robert McNeel & Associates
SD storage card	Raw scan data storage	N/A
Stable Diffusion	Conditional Generative AI (CGAI) model	Stability AI
Tencent Huiyuan 3D	Conditional Generative AI (CGAI) platform	Tencent