Augmented Reality-Based Visualization of a Resected Tumor Specimen

Abstract

Source:
Carly Fassler¹, Liyu Huang¹, Kavita Prasad¹, Jiayi Xu², Guansen Tong², Jie Ying Wu², Michael C. Topf<sup>1,3

1</sup>Department of Otolaryngology – Head and Neck Surgery, Vanderbilt University Medical Center.

²Computer Science Department, Vanderbilt University.

³Vanderbilt University School of Engineering, Vanderbilt University.

This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal.

Protocol

All procedures involving sample collection have been performed in accordance with the institute's IRB guidelines.

NOTE: A schematic detailing the protocol workflow can be seen in Figure 1.

1. 3D Scanning

1. Obtain a 3D Scan of an oncologic resection specimen per the previously described 3D scanning protocol7.
   1. Set up the commercially available desktop 3D scanner in proximity to the pathology lab where specimens are typically taken.
   2. Gently rinse, and pat dry the resected specimen. Then, take 2D photographs of its anterior and posterior surfaces.
   3. Place the specimen on the scanner turntable, ensuring the turntable is protected from any bodily fluids with a thin plastic sheet.
   4. Using 3D scanner software and a laptop computer, obtain a 3D scan of the specimen's anterior and posterior surfaces.
   5. Using the 3D scanner software, use three-point data cross-registration to align the two sets of scan data.
   6. Once the scan resembles the 3D structure of the ex vivo specimen, proceed to step 1.2.
2. Click Save Your Scan and save the completed scan from the 3D scanning software as a .obj file and a .jpeg file.

2. Augmented reality environment setup

1. Open the desktop AR software. Open the Augmented Reality VISE package.
2. Upload the 3D model produced in step 1.2 as a .obj file and the texture data as a .jpeg file into the software by dragging and dropping the files from the desktop into the Project Assets box.
3. Put on the Head Mounted Display (HMD).
4. Open the menu in the HMD environment by raising the non-dominant wrist. When a square icon appears, tap it by pointing the finger at the icon and look for a menu to appear.
5. Select File | Show All Apps | open the Holographic Remoting application on the HMD.
6. Connect the HMD to the software by selecting Mixed Reality | Remoting | Holographic Remoting for Play Mode in the top toolbar.
7. Input the IP address of the HMD, which is displayed when the Holographic Remoting application is open.
   NOTE:  If not displayed within the app, the IP address can also be found in the settings or through the voice command "What's My IP Address".
8. Ensure the HMD and the device running the software are both connected to the same Wi-Fi network. Hit the triangle Play button in the top center. The HMD headset will now display Rendering.
9. Drag the 3D object model into the left Hierarchy bar. Open the dropdown menu for the object. Drag the default into the Adjust Pivot popup window under Replace Object. Click on Apply Selected Objects.
10. Open the Medical_Object dropdown in the Hierarchy menu and select anchor before selecting Move Medical_Object's pivot here in the Adjust Pivot popup.
11. Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup. Select Apply Selected Texture File.
12. Hit the Play button in the top center of the screen once more.

3. Entering virtual reality

1. Utilize the voice command Open to initialize the hologram.
2. If the object is not seen or if it is ever lost from view, use the voice command Reposition to bring the object back to the center of view.
3. The object will render in anatomical size. Use the voice command Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size.
4. Pinch with the pointer finger and thumb to resize, move, and rotate the object in the same way a physical object is manipulated.

4. Utilizing voice commands

1. Manipulate the object hands-free using the following voice commands:
   1. Right: Rotate the object to the right.
   2. Left: Rotate the object to the left.
   3. Bigger: Make the object bigger.
   4. Smaller: Make the object smaller.
   5. Increase transparency: Increase the object's transparency.
   6. Decrease transparency: Decrease the object's transparency.

5. AR-guided re-alignment of the specimen

NOTE: 3D realignment of the hologram can be seen in Figure 2.

1. Manipulate the 3D specimen hologram seen through the HMD using the thumb and pointer finger.
   1. Grasp the object by putting the thumb and pointer finger together in the real environment where the hologram is seen in the virtual environment.
   2. Let go of the object by opening the thumb and pointer finger in the real environment once the 3D hologram is placed where desired in the virtual environment.
2. Utilize the voice command Lock to secure the hologram in place and Unlock to manipulate the object once more.
3. Based on the anatomic orientation of the hologram, realign the virtual specimen into the resection bed by following steps 5.1-5.2. When the hologram is realigned as much as possible with the resection bed as viewed through the HMD, use the voice command Lock to lock the specimen in place.