A 3D Digital Model for the Diagnosis and Treatment of Pulmonary Nodules

Summary

The objective of this study is to develop a novel 3D digital model of pulmonary nodules that serves as a communication bridge between physicians and patients and is also a cutting-edge tool for pre-diagnosis and prognostic evaluation.

Abstract

The three-dimensional (3D) reconstruction of pulmonary nodules using medical images has introduced new technical approaches for diagnosing and treating pulmonary nodules, and these approaches are progressively being acknowledged and adopted by physicians and patients. Nonetheless, constructing a relatively universal 3D digital model of pulmonary nodules for diagnosis and treatment is challenging due to device differences, shooting times, and nodule types. The objective of this study is to propose a new 3D digital model of pulmonary nodules that serves as a bridge between physicians and patients and is also a cutting-edge tool for pre-diagnosis and prognostic evaluation. Many AI-driven pulmonary nodule detection and recognition methods employ deep learning techniques to capture the radiological features of pulmonary nodules, and these methods can achieve a good area under-the-curve (AUC) performance. However, false positives and false negatives remain a challenge for radiologists and clinicians. The interpretation and expression of features from the perspective of pulmonary nodule classification and examination are still unsatisfactory. In this study, a method of continuous 3D reconstruction of the whole lung in horizontal and coronal positions is proposed by combining existing medical image processing technologies. Compared with other applicable methods, this method allows users to rapidly locate pulmonary nodules and identify their fundamental properties while also observing pulmonary nodules from multiple perspectives, thereby providing a more effective clinical tool for diagnosing and treating pulmonary nodules.

Introduction

The global incidence of pulmonary nodules is variable, but it is generally estimated that about 30% of adults have at least one pulmonary nodule visible on chest radiographs¹. The incidence of pulmonary nodules is higher in specific populations, such as heavy smokers and those with a history of lung cancer or other lung diseases. It is important to note that not all pulmonary nodules are malignant, but a thorough evaluation is necessary to rule out malignancy². The early detection and diagnosis of lung cancer are crucial for improving survival rates, and regular screening with low-dose computed tomography (LDCT) is recommended for high-risk individuals. Many AI-driven pulmonary nodule detection and recognition methods^3,4,5,6,7 employ deep learning techniques to capture the radiological features of pulmonary nodules, and these methods can achieve good area under the curve (AUC) performance. However, false positives and false negatives remain a challenge for radiologists and clinicians. The interpretation and expression of features from the perspective of pulmonary nodule classification and examination are still unsatisfactory. At the same time, the 3D reconstruction of pulmonary nodules based on LDCT has gained increasing attention as a digital model for various types of nodules.

The 3D reconstruction of pulmonary nodules is a process that generates a 3D representation of a small growth or lump in the lung. This process typically involves the application of medical image analysis techniques that leverage both medical expertise and data intelligence approaches. The resulting 3D digital model offers a more detailed and accurate depiction of the nodule, enabling the improved visualization and analysis of its size, shape, and spatial relationship with the surrounding lung tissues^8,9,10,11,12. Such information can aid in the diagnosis and monitoring of pulmonary nodules, particularly those suspected of being cancerous. By facilitating more precise analysis, the 3D reconstruction of pulmonary nodules has the potential to enhance the accuracy of diagnosis and inform treatment decisions.

Maximum intensity projection (MIP) is a popular technique in the field of 3D reconstruction of pulmonary nodules and is used to create a 2D projection of a 3D image^8,9,10,11,12It is particularly useful in the visualization of volumetric data extracted from digital imaging and communications in medicine (DICOM) files scanned by CT. The MIP technique works by selecting the voxels (the smallest units of 3D volume data) with the highest intensity along the viewing direction and projecting them onto a 2D plane. This results in a 2D image that emphasizes the structures with the highest intensity and suppresses those with lower intensity, which makes it easier to identify and analyze relevant features^9,10,11,12. However, MIP is not without limitations. For example, the projection process can result in a loss of information, and the resulting 2D image may not accurately represent the 3D structure of the underlying object. Nevertheless, MIP remains a valuable tool for medical imaging and visualization, and its use continues to evolve with advances in technology and computing power¹¹.

In this study, a successive MIP model to visualize pulmonary nodules is developed that is easy to use, user-friendly for radiologists, physicians, and patients, and allows the identification and estimation of the properties of pulmonary nodules. The primary advantages of this processing approach include the following aspects: (1) eliminating false positives and false negatives arising from pattern recognition, which enables a focus on assisting physicians to obtain more comprehensive information on the location, shape, and 3D size of pulmonary nodules, as well as their relationship to the surrounding vasculature; (2) enabling specialist physicians to attain professional knowledge of the characteristics of pulmonary nodules even without the assistance of radiologists; and (3) enhancing both communication efficiency between physicians and patients and prognosis evaluation.

Protocol

NOTE: During the data preprocessing stage, the original DICOM data must be sorted and intercepted to ensure compatibility with various devices and consistent results. Adequate adjustable capacity must be reserved for intensity processing, and a continuous 3D perspective is essential for observation. In this protocol, a methodical description of the research approach is provided, detailing a case involving an 84-year-old female patient presenting with pulmonary nodules. This patient provided informed consent for her diagnosis via digital modeling and authorized the utilization of her data for scientific research purposes. The model reconstruction function is derived from the PulmonaryNodule software tool (see the Table of Materials for details). Ethical clearance was obtained from The Ethics Committee of Dongzhimen Hospital, affiliated to Beijing University of Chinese Medicine (DZMEC-KY-2019.90).

1. Data collection and preparation

1. LDCT data for the detection of pulmonary nodules
   NOTE: The differences seen in the parameters values do not depend on the research method used.
   1. Acquire patient consent for the acquisition of DICOM data. Transfer all data to the designated working directory.
   2. Identify the data directory with the highest number of scanning layers and the thinnest layer thickness to optimize the accuracy based on the file information. Generally, the more DICOM scan files a patient has, the thinner the scan layer thickness.
   3. By implementing the Dicominfo function and using the DICOM files as the function parameters, obtain the slice thickness and pixel spacing parameters in the MATLAB environment. These parameters are essential for setting the 3D volume display rate. For the example data utilized in this study, the slice thickness was 1 mm, the pixel spacing was 0.5 mm, and a total of 200 layers were scanned.
2. Correcting the sorting of the scanned data
   NOTE: The sequence of every image should be sorted for volume construction.
   1. Read the location data of every image by using the function Dicominfo. Obtain the location by inputting info.SliceLocation into the MATLAB workspace.
   2. Implement the SliceLocation function to store the location array for a variable, and make a plot of it (Figure 1).
   3. By using the Data Tips button at the top-right of the GUI, add a data tip to the plot on the point that represents the maximum location value of the normal sequence (the top location of the patient imaging; Figure 1).
   4. Sort all the images, and extract the images starting at 1 to the maximum location value by implementing the function VolumeResort.
   5. Store the volumes of the valid images with the sorted index, which will be useful for tracing back to the important nodules.
3. Inspecting the thoracic volume
   NOTE: Having a well-defined data-storing structure makes follow-up work more convenient.
   1. Implement the VolumeInspect function to show three views of the constructed volume. Drag the crosshair intersection up and down in the coronal axis to quickly browse all the images in the horizontal axis (Figure 2).
   2. Move the crosshair to the horizontal axis to browse all the images in the coronal axis. The crosshair is in the same spatial coordinates in the 3D volume; therefore, moving it on one axis will change the location of the images in the other two axes.
   3. For the VolumeInspect function, use the default intensity window for the lung in the GUI. Adjust the actual filter performance by holding the left mouse button and dragging in the axis.

2. Digital model for horizontal 3D reconstruction

NOTE: The 3Dlung_Horizon subprocess performs a thorough examination of pulmonary nodules from a horizontal perspective.

1. Implement the Build_3Dlung_Horizon function in MATLAB workplace to reconstruct the 3D digital model of the pulmonary nodules under the filter window of the lung, and then open the GUI to check the horizontal 3D model (Figure 3).
2. Unlike in step 1.3.2, the GUI in Figure 3 is a continuous 3D lung structure in which various types of pulmonary nodules and their relative spatial relationships with the lung tissue can be seen clearly. When dragging the scroll bar on the GUI with the mouse, the continuous 3D lung structure can be observed.
3. The upper-right corner of the GUI in Figure 3 provides icons for zooming in, zooming out, returning to the global view, and marking the coordinates of the pixel selected. Use the zoom function to observe the local features of the lesions and output relevant 3D structural output pictures. Use the Mark pixel coordinates button to calculate the distance between two points in order to measure the size of the nodules.
4. The default color bar is the jet colormap, which means that blue to red represents values from low to high. Right-click Color Bar in the pop-up menu to select the common gray colormap and reset the whole GUI.
5. If the filter window is not satisfied, use the left mouse button to drag up and down in the middle of the figure to adjust the window level. Drag left and right to adjust the window width, and the corresponding accurate filtering range will be displayed on the color bar.

3. Constructing a 3D digital model for any specific nodule

NOTE: The slice number is a parameter of the function 3D_Nodules, which reconstructs a 3D digital model that can be viewed from every perspective.

1. To determine the slice number, as in Figure 3, check at the top-right of the scroll bar; in Figure 3, the slice number is 70. Use the function Build_3D_Nodules with two parameters, including the slice number and the thoracic volume created in step 1.3, to reconstruct a 3D digital model for specific nodules. This is a user-defined model, as the input slice number is variable and depends on the user.
2. If the Build_3D_Nodules function is executed correctly, the user can check the pulmonary nodule located in a certain slice number from various perspectives in the pop-up GUI (Figure 4). To do this, perform the following actions:
   1. Press and hold the left mouse button, as in the center of Figure 4, and drag it in any direction to change the perspective of the pulmonary nodules. It should be noted that the observation angle should take into account the anatomical considerations and try to show both the medical characteristics of the pulmonary nodules and the relationship between the nodules and the surrounding tissues.
3. Use the zoom and move icons in the upper-right corner, as done in Figure 3. Additionally, by rolling the middle mouse button, the user can continuously zoom in or zoom out of the view of the model.
4. The GUI in Figure 4 shows the coordinate indication of the model in the lower-left corner, where the positive direction on the z-axis is the scanning direction in the horizontal position. Implement the screenshot tool provided by the operating system to save the required 3D projection of the nodules.

4. Digital model of a coronal 3D reconstruction

NOTE: The Build_3Dlung_Coronal subprocess is executed to evaluate pulmonary nodules from an alternative coronal perspective, thus aiding clinicians and patients in developing a more precise and inclusive understanding of the location and attributes of the nodules.

1. Implement the Build_3Dlung_Coronal function in the MATLAB workplace to reconstruct the 3D digital model of pulmonary nodules under the filter window of the lung, and then open the GUI, as prepared by the function, to check the coronal 3D model (Figure 5).
2. The GUI in Figure 5 shows a continuous coronal 3D lung structure in which various types of pulmonary nodules and their relative spatial relationships with the lung tissue can be seen clearly. Drag the scroll bar on the GUI with the mouse to observe the continuous coronal 3D lung structure.
3. The upper-right corner of the GUI, as shown in Figure 5, also provides icons for zooming in, zooming out, returning to the global view, and marking the coordinates of the pixel selected. Use these functions to observe the local features of the lesions and to generate relevant 3D structural pictures. Mark the pixel coordinates to calculate the distance between two points, which is often used to measure the size of the nodules.
4. The default color bar is the jet colormap, in which the colors from blue to red represent values from low to high. Right-click the color bar in the pop-up menu to select the common gray colormap and reset the whole GUI.
5. If the filter window is not appropriate, use the left mouse button to drag up and down in the middle of the figure to adjust the window level; drag left and right to adjust the window width, and the corresponding accurate filtering range will be displayed on the color bar.

5. Output 3D video for dominant pulmonary nodules

NOTE: Converting the optimal 3D digital model of a pulmonary nodule into a dynamic 3D video enables physicians and patients to better comprehend the condition and make accurate judgments, which is especially critical for formulating effective treatment plans.

1. In the workspace, prepare the required 3D digital model, and pre-visualize the relative spatial relationships between the pulmonary nodules and the lung tissue to be displayed from various angles (Figure 3 and Figure 4).
2. In this study, Adobe Captivate 2019 was used to record all the GUI interaction processes. To begin, open the software, and create a new screen recording project. Turn the camera off, and the red screen recording range box will pop up for recording only the screen operation. In this study, the version 5.1 GUI was included in the box. Click the recording button to operate the GUI, and generate a digital video file of the screen recording.
3. After recording the dynamic display of the pulmonary nodules, return to the operating environment of the software by clicking on the icon in the taskbar.
4. By using the video publishing feature, save the recorded dynamic video of the 3D digital model of the pulmonary nodules. Click on File > Distribute and configure the file storage path. Name the file, and save the desired digital video file.

Representative Results

To make the method applicable to a wider range of devices, the stacking order of each scan needs to be reorganized based on the internal coordinates of the DICOM file system (Figure 1) to generate the correct 3D volume (Figure 2). Based on the accurate volume data, we utilized algorithmic continuous reconstruction of the patient's lung horizontal and coronal MIPs (Figure 4 and Figure 5) for the precise diagnosis and treatment of the patient's pulmonary nodules.

DICOM data from different devices are usually not sorted in the correct order from low to high along the anatomical position in the patient. However, for model reconstruction, each image must be sorted in a low-to-high forward order. Figure 1 not only shows the typical distribution of DICOM sequence positions but also for GUI interaction necessary to determine the position boundaries of the image sequence. This step is an important part of data preparation in the model reconstruction process.

Figure 2 essentially represents the three views of the 3D volume of the entire lung in the axial, coronal, and sagittal planes. If the previous data preparation and the volume calculation are correct, the lung images in each view can be viewed, as in Figure 2. This GUI also allows for window-level filtering by dragging the mouse to view the images at different window levels. The underlying volume, as shown in Figure 2, serves as the data basis for 3D model reconstruction.

Figure 3 shows the continuous 3D reconstruction results in the axial view. In this GUI, the physician can observe the patient's lung structure continuously along the axial view, quickly locate pulmonary nodules, and observe the relationship between the nodules and the surrounding lung tissue. As shown in Figure 3, operations such as the local magnification of nodule views, the recovery of the window level, and the marking of pixel positions can also be performed. The color bar displays colors corresponding to the different intensity values in the image.

The GUI shown in Figure 4 provides physicians with the ability to observe the interested pulmonary nodules from any perspective.

Figure 5 shows the continuous 3D reconstruction results in the coronal view. In this GUI, the physician can observe the patient's lung structure continuously along the coronal view, quickly locate pulmonary nodules, and observe the relationship between the nodules and the surrounding lung tissue. As shown in Figure 5, operations such as the local magnification of nodule views, the recovery of the window level, and the marking of pixel positions can also be performed. The color bar displays colors corresponding to the different intensity values in the image.

Figure 1: The image location plot according to the file name sequence. The figure shows the typical distribution of DICOM sequence positions and indicates the GUI interaction necessary to determine the position boundaries of the image sequence. Please click here to view a larger version of this figure.

Figure 2: The GUI of three views of the pulmonary 3D volume. The figure represents the three views of the 3D volume of the entire lung in the axial, coronal, and sagittal planes. If the previous data preparation and the volume calculation are correct, lung images in each view can be viewed. Please click here to view a larger version of this figure.

Figure 3: The GUI for checking the pulmonary nodules from a horizontal view. The figure shows the continuous 3D reconstruction results in the axial view. The physician can observe the patient's lung structure continuously along the axial view, locate pulmonary nodules, and observe the relationship between the nodules and the surrounding lung tissue. The colors in the color bar correspond to different intensity values in the image. Please click here to view a larger version of this figure.

Figure 4: Checking the 3D digital model of a specific pulmonary nodule. The GUI allows the user to observe the pulmonary nodules of interest from any perspective. Please click here to view a larger version of this figure.

Figure 5: The GUI for checking the pulmonary nodules from a coronal view. The figure shows the continuous 3D reconstruction results in the coronal view. In this GUI, the physician can observe the patient's lung structure continuously along the coronal view, quickly locate pulmonary nodules, and observe the relationship between the nodules and the surrounding lung tissue. The colors in the color bar correspond to different intensity values in the image. Please click here to view a larger version of this figure.

Discussion

Different LDCT devices have significant differences in the DICOM image sequences they output, especially in terms of the file system management. Therefore, to reconstruct the key 3D digital model of a pulmonary nodule in the later stages of the protocol, the data preprocessing step is particularly important. In the data preparation and preprocessing stage (step 1.2.2), the sequence z-axis coordinate can be correctly sorted by using the sequence shown in Figure 1, which can also be used to properly arrange the correct image order required for modeling and generate the correct 3D volume for the subsequent modeling work. The 3D reconstructions (step 2.1 and step 4.1) in the horizontal and coronal axes provide physicians and patients with double checks of pulmonary nodules from the two most commonly used perspectives. Detecting nodules and presenting their spatial characteristics and relationships with the lung tissue, especially with the pulmonary arteries, and using software tools are crucial for disease diagnosis and treatment plan formulation. In terms of doctor-patient communication, a good 3D dynamic video (step 5.2) is an excellent communication tool that supports patients to understand their own condition and prognosis.

When discussing the clinical and research scenarios of this study, an important issue that must be addressed is how to provide a clear and comprehensive understanding of pulmonary nodules under treatment to both specialized physicians and patients. Typically, patients bring examination results from different devices and historical periods to their doctors, and in the absence of support from a radiologist, physicians need to make accurate judgments about the size, location, and characteristics of the patient's pulmonary nodules in order to provide appropriate treatment plans. Patients, on the other hand, need to effectively understand and track the development status and treatment effects of their own lung nodules. Therefore, whether the case requires understanding CT data from different devices and times, bridging the gap between radiologists and specialized physicians, or assisting better doctor-patient communication, this study provides an ideal solution and technical means to address these scenarios.

Although current tools have shown outstanding performance in 3D modeling and feature visualization of pulmonary nodules, there is still room for evolution and improvement. First, a GUI with mutual indexing between 2D tri-views and 3D models could be developed to facilitate cross-validation of the two perspectives. Secondly, continuous 3D models from arbitrary viewing angles are also worth further development. Thirdly, integrating the long-term tracking and management of patient treatment plans and prognoses is also an important direction for evolution, as this would be useful for a complete medical imaging solution in the field of pulmonary nodules.

Due to the requirement of machine learning to produce a large number of consistent medical image sequence samples of different types of pulmonary nodules¹³ and the considerable scale of AI computing power, it is not yet possible to recognize and automatically classify pulmonary nodules based on 3D volume features^14,15. This is a research direction that will continue to be focused on in the later stages of this work.

The significance of this study lies in providing a continuous 3D digital model for the diagnosis and treatment of pulmonary nodules. Physicians and patients can better understand the condition and make more rational judgments by observing the features of nodules from various perspectives in 3D space, which is also of important reference value for the prognosis evaluation of the nodules. On the basis of existing work, deep machine learning could be introduced to classify lung nodules from a more multidimensional perspective. Through the use of clinical treatment cases, the efficacy of drugs and other treatments could be tracked using this method to provide an increasingly accurate quantitative basis for the prognosis evaluation of pulmonary nodules.

Materials

Name	Company	Catalog Number	Comments
MATLAB	MathWorks	2022B	Computing and visualization
Tools for Modeling	Intelligent  Entropy	PulmonaryNodule V1.0	Beijing Intelligent Entropy Science & Technology Co Ltd. Modeling for CT/MRI fusion