Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Summary

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Abstract

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Introduction

The treatment of intracranial aneurysms is a challenging aspect of neuro-interventional surgery, necessitating precise surgical planning to ensure optimal patient outcomes. In recent years, virtual reality (VR) technology has become a promising tool for enhancing surgical planning by providing surgeons access to immersive, patient-specific anatomical models in a virtual 3D environment^{1,2,3,4,5,6,7,8}. This article presents a comprehensive protocol for the use of medical imaging and segmentation, 3D modeling, VR surgical planning, and idealized virtual roadmap generation to aid in surgical planning for the treatment of aneurysms.

The combination of these steps culminates in a virtual surgical planning approach, allowing physicians to immerse themselves in a virtual environment and gain a comprehensive understanding of a patient's unique anatomy prior to a surgical procedure. This immersive approach empowers surgeons to explore optimal positioning and simulate various procedural scenarios. Recording these scenarios can provide insight into the placement of real-world surgical equipment, such as C-arm positioning.

In addition to positioning angles, it is also possible to measure anatomy in a virtual environment using measurement tools designed for 3D space. These measurements can provide insight into the correct sizing and shape of the device to be used in an intracranial aneurysm case⁹.

This protocol presents a comprehensive process that seamlessly combines medical imaging, image segmentation, VR model preparation, and virtual surgical roadmap generation to enhance the surgical planning process. Using a combination of leading-edge technologies, this protocol provides opportunities to save valuable time in the operating room¹⁰, as well as a boost to surgeon confidence and understanding of complex surgical cases^11,12,13.

Protocol

De-identified human DICOMs or DICOMs for patient care are used in accordance with institutional guidelines for patient care, the Health Insurance Portability and Accountability Act of 1996 (HIPAA), and collaboration with the Institutional Review Board (IRB) when appropriate.

1. Segment patient-specific anatomy

1. Acquisition of medical scans
   1. Begin the segmentation process with a physician or surgeon ordering medical scans. These scans are a part of the standard patient care protocol and do not introduce extra procedures.
   2. If the physician knows they will request segmentation, ensure they ask for thinly sliced data sets to be exported from the MRI or CT scanner. In most cases, these thin slices are less than 1 mm thick; however, this resolution can vary between scanners. Take CTA scans with contrast in order to ensure proper segmentation of vasculature and blood pools.
      1. For data acquisition, acquire MRI 3D sequence with the following suggested parameters: run in axial, ensure slice thickness and space between slices of 0.625 mm or smaller, zero spacing. Acquire CT 3D series with the following suggested parameters: slice scanner in helical mode, slice thickness and space between slices of 0.625mm, e.g., Neuro: Kvp of 120, Smart mA range of 100-740, rotation speed at .5ms, or Cardiac: Kvp of 70, Smart mA range of 201-227 (smart MA mode 226), rotation speed at 0.28 ms¹⁴. Follow the institution's helical parameters for each body part.
         NOTE: The 3D sequence should be acquired so that in the reconstruction, there is a near isotropic resolution in the axial, coronal, and sagittal planes. In most cases, the thinnest slices should be acquired. The 3D sequence is run in addition to the institution's standard protocol for imaging. However, it is run at the same time, so there is minimal additional work, radiation exposure, and expense for the clinical team and patient.
2. Ask the physician to request the segmentation of the model, specifying what anatomy will be the central focus for the segmentation process (A physician or surgeon usually completes this step).
3. Download scan data and save it locally.
   1. If more than one scan was taken, ensure that DICOM datasets from the scan are compared to determine which scan set has the thinnest slice setting and best contrast, as this will provide the highest resolution 3D models when segmented.
   2. Once the best image set is determined, download it from the image database for segmentation, anonymize, or leave the data as is with protected health information (PHI). This protocol will work with an anonymized DICOM.
4. Import the DICOM dataset into the segmentation software.
   NOTE: The following instruction set uses terms specific to the Materialise Mimics segmentation software. While the Materialise Suite is subscription-based software, there are open-source alternatives such as 3DSlicer. Tool names and terminology may vary across other segmentation tools.
5. Create a rough mask of the target anatomy, such as bone, blood pool, aneurysm, etc.
   1. Under the SEGMENT tab, select New Mask tool.
   2. Set the upper and lower threshold boundaries by clicking and dragging both to capture as much of the relevant target anatomy as possible while limiting the capture of surrounding tissues. Click and drag the boundaries within the threshold tool or input the desired Hounsfield Unit (HU).
   3. While setting thresholds, crop to a specific area of the scan to avoid excess selection of surrounding tissue. The upper and lower threshold boundaries vary greatly depending on scan type, sequence type, amount of contrast, and patient.
   4. Click OK to finalize the rough mask.
6. Use other tools within the SEGMENT tab to remove unnecessary portions of the mask or add missing tissue as needed.
   1. Use the Region Grow tool to separate all voxels of the mask directly connected to a voxel selected by the user; use Edit Mask to add or remove voxels in the mask via both the 2D and 3D windows; use Multiple Slices Edit to add or remove voxels through interpolation between slices farther apart; and use Fill Holes or Smart Fill to fill holes of a user-defined size within the mask.
   2. Continue refining the mask making use of tools in the tab until the 2D to 3D interpolation is as accurate as possible.
7. Repeat steps 1.5 and 1.6 for all target anatomies.
8. Consult a physician regarding the completed segmentation to ensure accuracy.
   1. Show the completed masks to a physician to ensure important anatomy has not been omitted and excess anatomy has not been included. In most cases, consult the physician who requests the segmentation for quality control. The physician ensures the portion of the DICOM highlighted by the mask on each slice is as accurate as possible (see Figure 1).
9. Export segmentation for further processing.
   1. Convert finalized masks to parts by using the Calculate Part tool located in the Project Management menu on the right side.
   2. Export calculated parts to 3D files by right-clicking on The Part and selecting Export STL.

2. Prepare the model for virtual reality

1. Create a new Blender project and remove the default scene elements. Press the a key to highlight all visible elements, and then x followed by Enter to remove them from the scene.
   NOTE: Blender is a free and open-source modeling software. While other modeling software may be able to accomplish the same tasks, the terminology used in this step will be specific to Blender.
2. Import the anatomy files through File > Import > Stl (.stl).
3. Align the patient's anatomy with the world's origin.
   1. Select all patient anatomy in order to maintain relative positioning. To do this, press the a key after all files are imported.
   2. Use the Move and Rotate tools to align the anatomy with the world's origin. Ensure the patient's nose is aligned with one axis, with the perpendicular axis contacting the ear area and the top of the skull, respectively. Use orthographic views, which can be activated with the widget in the top right corner of the blender interface.
4. Import the VR protractor and align it to the patient's anatomy. This protractor has been specially designed by the OSF engineering team to aid in acquiring C-arm angles in VR, based on angles in 3D space.
   1. Import the protractor .stl file from Supplementary File 1.
   2. Align (0,0) on the protractor, represented by the longest measurement mark, with the patient's nose. Orient the gap in the protractor arms toward the patient's feet.
   3. Scale the protractor accordingly. For most cases, scale the protractor quite small in order to ensure ease of measurement after scaling in VR. In the case of aneurysms, try to scale the protractor in a way that it lies just outside of the aneurysm area.
5. Align anatomy origins to the world origin.
   1. Right-click click in the main viewport and select Snap > Cursor to World Origin. This ensures the 3D cursor is aligned with the global origin.
   2. Select all visible models with the a key.
   3. Right-click, click in the viewport and select Set Origin > Origin to 3D Cursor. This aligns the 3D origin of all models to the same point, ensuring they will line up and scale properly when imported to VR.
6. Add texture or color to the models as desired for better distinction in VR.
   1. This is an optional step. Select the individual .stl files, then click the Material Properties tab on the right-hand side of the screen. Under this tab, the base color can be adjusted to the desired color. Repeat this step for each object to add color.
7. Export the finalized model as a single. glb/.gltf file. Ensure that none of the Limit to options under the Include tab of the export window are selected.
   NOTE: The. glb/.gltf file format reflects the file type required for use in the VR software, as well as for upload to the NIH 3D library. Other export types may be needed for different software.

3. Training medical professionals in virtual reality

NOTE: The following instructions are written to be used with the Enduvo digital classroom software. While it may be possible to use other 3D viewing software, the ability to move models, place cameras, and record physician positioning are some features that make this software ideal for this procedure. Different VR headsets, controllers, and software combinations may have different controls.

1. Create a new lesson.
   1. Import the .gltf file exported in step 2.7 in the lesson creation menu. The software may produce a message stating: The file type (GLB) that you are attempting to upload is not currently fully supported. Ignore this message and click the Confirm button.
2. Open the lesson in VR for finalization.
   1. Using the transparency menu accessed by pressing down on the thumbpad or joystick of one controller, hide all models except the target anatomy. The aneurysm should be the only model visible.
3. Place the surgeon or physician in VR and allow them some time to familiarize themselves with the 3D space and functions and the anatomy in the lesson.
4. Once the surgeon is comfortable with the anatomy, begin recording.
   1. Start the record function using either the virtual button in the VR space or the record button on the secondary monitor.
   2. Let the surgeon rotate the target anatomy to find preferred viewing angles for both anteroposterior (AP) and lateral fluoroscopy views. After finding a preferred angle, ask the surgeon to pause briefly and state they have found a preferred angle and whether the current viewing angle is AP or lateral.
   3. After all preferred angles are found, stop recording with either the VR button or external monitor.

4. Generation of fluoroscopy roadmap in VR

1. Acquire fluoroscopy analogs using recorded surgeon placement.
   1. Place an image mimicking the gray hue background of a fluoroscopy image behind the model in the virtual space. Use the select button on the controller, often the trigger on the backside of the controller, to manipulate the image as needed. This creates a consistent background color which makes it easier to view the anatomy and is more representative of fluoroscopy.
   2. Place the camera in line with the surgeon's view at the time declared as a preferred viewing angle, ensuring the camera is pointed roughly at the center of the target anatomy. The surgeon will appear as a set of floating glasses and two controllers in VR.
   3. Capture a 2D snapshot with the camera in the desired position. Repeat the step for each preferred angle.
2. Acquire C-arm angles using the surgeon's movements and protractor.
   1. Pause the recorded lesson when the surgeon declares a preferred viewing angle.
   2. Click the Trackpad to open the quick menu and select the On/Off check box to unhide the protractor attached to the model.
   3. Use the controller's grab button to select and manipulate a pointer or straightedge in line with the surgeon's viewpoint, also passing through the origin of the protractor.
   4. Step back from the model and view the angles from the orthographic viewpoints corresponding with the C-arm motions.
   5. For a neurology case with the patient's nose oriented to 0° on all C-arm axes, take the AP angles from the sagittal and axial planes. Take the lateral angles from the coronal and axial planes. In both cases, the axial plane corresponds with the right and left angles of the C-arm, while the sagittal and coronal planes correspond with the cranial and caudal angles.
   6. Repeat steps above for each preferred angle.

Results

Following the presented protocol, virtual surgical roadmaps can be generated for both the AP and lateral fluoroscopy views. These roadmaps are created by placing a camera at the viewpoint of the surgeon in VR to capture their ideal AP and lateral views while also placing a colored background behind the target anatomy to better replicate a fluoroscopy image. The VR protractor is used at this point to record the angle from which the surgeon is viewing the target anatomy, recorded as right, or left anterior oblique (RAO/LAO - camera offset to the patient's right or left, respectively), and cranial or caudal anterior (CRA/CAA - camera offset to toward the patient's head or feet, respectively)¹⁵. When developing this process, retrospective cases were used to provide the ability to compare angles measured in VR with the actual angles used on the C-arm machines in surgery. Three different retrospective cases were selected for this process, each case having been treated with a different surgical device. The diversity of these three cases shows the versatility of the presented protocol. The surgeon was asked to find preferred AP and lateral angles without referencing the C-arm angles used during the procedure, and the VR measurements were then compared to these pre-existing C-arm positions.

In case 1, the declared preferred AP viewing angle was measured in VR as 16° CRA, 12° RAO. The actual measurements used in surgery for this case were 11° CRA and 13° RAO. The maximum error among these measurements is 5° on the cranial/caudal axis. Figure 2A shows the surgeon's declared AP view in virtual reality, followed by Figure 2B, which shows the actual angle used in surgery as seen in VR, and Figure 2C, which shows the surgical fluoroscopy image. Comparing the three images shows the VR images to be extraordinarily similar to the actual fluoroscopy image at the same angle.

The lateral view of the same case displayed one of the many challenges of this process due to the 3D model being inadequately reviewed. Because of this faulty review, there were some extraneous vessels segmented that, according to the surgeon, inhibited their view of the aneurysm in VR and are not connected to the target anatomy and, as such, are not accurately reflected in VR. These discrepancies were a result of miscommunication in the required target anatomy during the quality control session with the physician. These discrepancies can be seen in Figure 2D-F, which shows the surgeon's declared lateral, the VR representation based on surgical fluoroscopy angles, and the actual fluoroscopy images from left to right, respectively. With the exception of extraneous vessels, the surgeon's declared AP view closely resembles the actual fluoroscopy image, despite the measurements taken being 6° and 26° off in the coronal and axial planes, respectively. The replication of actual measurements in VR, as shown in Figure 2E, also depicts a similar view to the real fluoroscopy shown on the right of Figure 2F, with the main discrepancy being the anomalous extra vessels. This case employed a less reliable manual placement of the protractor tool, which may account for the slight difference in measurement. Future cases employ a protractor that is bound to the anatomy in order to ensure maximum accuracy of angle measurements taken in VR.

In cases 2 and 3, the views selected to be optimal in VR were not representative of the views used in the actual procedure. This was a consequence of the initial placement of models in VR being a blinded study. It is important to note the surgeon expressed that fluoroscopy procedures can have multiple acceptable treatment angles, and there is not necessarily a correct angle. For the purpose of comparison, images were taken in VR from the reported surgical angles. Figure 3 shows the VR AP view in Figure 3A and the surgical AP view in Figure 3B. In Figure 3, a similar comparison can be made between lateral views in Figure 3C,D for case 2. For case 3, Figure 4 shows the AP comparison Figure 4A,B, as well as the lateral comparison Figure 4C,D. The similarities between the VR and fluoroscopy images of these cases further demonstrate the ability of VR to be used in surgical planning.

An important benefit of this protocol is the improvement of surgical planning by leveraging 3D models in a VR environment. A previous study on the effectiveness of VR in surgical planning for complex oncological cases showed that roughly 50% of cases that employed the use of VR altered the surgical approach from the plan made using only 2D datasets⁹. VR has also been proven useful in the surgical planning process for liver tumor resection^16,17, as well as procedures involving head and neck pathology¹⁸. The surgeon participating in the creation of this protocol stated that : in VR I can see [the anatomy] so much better, showing the benefit of VR for endovascular neurosurgical applications.

Figure 1: Screenshot from within segmentation software. The screenshot shows the highlighted anatomy based on Masks. Please click here to view a larger version of this figure.

Figure 2: Case 1. (A) Anteroposterior view of Case 1 as placed by the operating surgeon in VR. (B) Anteroposterior view of Case 1 in VR based on angle measurements taken during surgery. (C) Anteroposterior fluoroscopy view captured during surgery. (D) Lateral view of Case 1 as placed by the operating surgeon in VR. (E) Lateral view of Case 1 in VR based on angle measurements taken during surgery. (F) Lateral fluoroscopy view captured during surgery. Please click here to view a larger version of this figure.

Figure 3: Case 2. (A) Anteroposterior view of Case 2 in VR based on angle measurements taken during surgery. (B) Anteroposterior fluoroscopy view of Case 2 captured during surgery. (C) Lateral view of Case 2 in VR based on angle measurements taken during surgery. (D) Lateral fluoroscopy view of Case 2 captured during surgery. Please click here to view a larger version of this figure.

Figure 4: Case 3. (A) Anteroposterior view of Case 3 in VR based on angle measurements taken during surgery. (B) Anteroposterior fluoroscopy view of Case 3 captured during surgery. (C) Lateral view of Case 3 in VR based on angle measurements taken during surgery. (D) Lateral fluoroscopy view of Case 3 captured during surgery. Please click here to view a larger version of this figure.

Supplementary File 1: A 3D model of protractors was developed and used for the protocol in STL file format. Please click here to download this File.

Discussion

3D modeling was introduced to medical workflows with the advent of 3D printing technologies^{2,3,4,6,7,9,11}, but VR affords novel applications of 3D technology beyond a physical 3D object. Efforts to replicate anatomy and scenarios in a virtual world allow for personalized medical practice on individual patients^{1,2,3,4,9,11,13,16}. This work demonstrates the expansive capability of creating new pre-surgical simulations in a digital world with minimal effort.

Throughout the presented protocol, there are several steps that are critical to the success of a case. The most important factor in producing adequate results with proper resolution is acquiring the correct medical imaging. The presented process does not require additional scans on the patient, using the standard CTA scan that is scheduled for every intracranial aneurysm case. Most scanners will store scans for a short time, depending on the scanner model and health system protocol, allowing the imaging technician to upload the acquired thin slices of the scans typically less than 1 mm thick slices are often not stored longer than a few days due to the storage size. These thin slices allow for greater detail and the inclusion of smaller anatomy, such as blood vessels. After segmentation has taken place, physician quality control must be completed to ensure the 3D models generated represent the patient anatomy as accurately as possible in future steps. Quality control of all models should be a part of the segmentation process, minimizing the potential for propagation of error throughout the remainder of the protocol. Quality control includes blood vessel borders and segmenting the aneurysm separately from the surrounding vessels, similar to how it would present with contrast. Quality control with a physician is of utmost importance as the physician holds the entirety of the responsibility for the accuracy of the models, especially if the models are to be used in further decision-making of patient's treatment. In some circumstances, it may be feasible or practical for the physician to complete the segmentation step themselves.

The next important step in the protocol is maintaining spatial model alignment while integrating the protractor measurement tool. Blender has proved to be an extremely helpful tool for this step as it allows for the combination of multiple STL file types into one combined file with multiple layers, each of which is spatially aligned and can be colored or textured for added clarity. Additionally, during this step, the protractor STL is added so that angle data can be gathered in VR. This protractor model was specifically developed using a computer aided design (CAD) tool, SolidWorks. Taking advantage of high-precision dimensioning tools within the software, an arc with tic marks denoting every 5° in all three axes was created. The protractor also has crosshairs denoting the true center of that model and allowing for alignment to the center of the patient's anatomy. There is also a large bar within the model signifying (0,0) and is to be aligned with the patient nose. Also, it is important to note that this was done manually and could have increased the error percentage. Alignment is of utmost importance to ensure the accuracy of all potential angle measurements. Once properly aligned, the model is ready for VR, where recording of the physician placement of the model allows for future determination of the angles at which the model has been placed. During the recording, everything within the virtual space is recorded in reference to one another, most importantly the physician's point-of-view (POV) and the models' movements and rotations. Taking full advantage of this recording and the pause feature, a straight edge is placed from the physician's POV through the protractor model's crosshair, and measurements can be observed in a manner remarkably similar to the use of an actual protractor.

This methodology does have some limitations. One such limitation is that there is not necessarily a single correct orientation for the aneurysm when viewing it in fluoroscopy. This led to multiple validation attempts simply due to the different viewing angles. This limitation can be viewed as a possible benefit from the perspective that with additional familiarity that comes from manipulating the 3D model, it is possible that the physician will find an optimal view as compared to the current method of determining angles within the operating suite. Another potential limitation of this protocol is that it is possible to determine a viewing angle in VR that would not actually be possible for the C-arms to get to. This limitation would be taken into account and known by the physician in VR so specifications could be made if this became part of surgical planning. Another limitation, proving the importance of the quality control step, is that in some instances, vessels that are distal of the aneurysm, in reality, are not seen as prominently in fluoroscopy procedures as they would be if included in the model in VR. This can force the physician to be mindful of a vessel that would not necessarily be in the way during the procedure in VR, leading to a suboptimal viewing angle being generated in VR. In segmentation, it is possible to segment out the majority of the blood vessels and the area of interest; the interventionalist could choose to toggle between models of vessels to ensure there would be no additional vessels in their viewing angle, the use of contract minimizes this risk as well.

The development of a 3D model protractor and a protocol that can provide angle measurements in multiple axes within VR holds immense importance and promises a wide array of potential applications. The benefits could prove to be multifaceted, potentially enhancing various industries from architecture and engineering to manufacturing and military applications. However, as shown in this protocol, its true potential shines in the realm of healthcare, directly within the surgical planning portions of patient care. Surgeons can utilize this tool to meticulously assess and plan all types of procedures by being able to visualize and measure angles directly in VR. This technique is similar to work done for cardiac catheterization¹⁹. One direct benefit of knowing particular angles pre-procedure is the significant reduction in the need for a full 360-degree spin during fluoroscopy, a commonly employed imaging technique during aneurysm repair. By determining the angles required to mimic the virtual surgical roadmap, the surgeons can position the equipment more accurately, thus minimizing the radiation exposure to the patient. This not only contributes to patient safety by minimizing risks associated with radiation exposure but also streamlines the surgical procedure. With reduced time spent on fluoroscopy adjustments, surgical teams can operate more efficiently, ultimately leading to shorter procedure times.

Recent advancements in 3D modeling and virtual reality technology allow medical staff to avoid improvisational thinking during surgeries by obtaining a deep understanding of a patient's internal anatomy prior to operation in all but the most urgent cases^{1,2,3,4,6,9,11,13,16}. If time allows, medical staff should leverage the use of medical image segmentation and VR diagnostics to further their understanding of the case prior to placing the patient on the operating table. This will ultimately lead to a better understanding of each unique patient, as well as reduced surgery time and time under anesthesia.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
3D Slicer		N/A	Open source segmentation software
Blender		N/A	Open-source CAD software that can import and edit organic models created through segmentation
Enduvo	Enduvo	N/A	A proprietary VR viewer built for education, and our VR viewer of choice
McKesson PACS Change Healthcare Radiology Solution	McKesson	N/A	Any Picture Archiving and Communication System should be suffiecient, McKessen is simply our PACS software solution of choice.
Mimics	Materialise	N/A	Segmentation software
Quest	Oculus	N/A	Virtual Reality Headset
Steam VR	Steam	N/A	Computer to headset connection software.
VR capable computer			See Steam VR for minimal requirements.
VR-STL-Viewer	GitHub	N/A	A open-source VR viewer capable of importing and viewing .stl and can be used, however we cannot guarantee all functionalities mentioned in this paper will be available