Augmented Reality-Based Visualization of a Resected Tumor Specimen

Place a resected tumor specimen on a 3D scanning table connected to a computer. Scan the anterior and posterior surfaces to create a 3D image.

Once this 3D image matches the physical specimen, open it in augmented reality or AR software.

Wear a head-mounted display or HMD and connect it wirelessly to the AR software.

This allows visualization of the 3D image through the HMD device as a virtual hologram of the specimen.

Align and anchor the hologram to ensure spatial accuracy within the AR environment, then apply textures to enhance realism.

Use a voice command to open the hologram in a virtual environment and, with hand gestures, reposition, rotate, and adjust the size.

Alternatively, use voice commands for hands-free adjustments.

This augmented reality-based visualization generates a tumor hologram that aids in precise margin identification at the surgery site.

To begin, gently pat dry a rinsed, resected oncologic specimen. Take 2D photographs of the anterior and posterior surfaces of the specimen.

Place the specimen on a 3D scanner's plastic-covered turntable. Scan the specimen's anterior and posterior surfaces to obtain a 3D image.

Use three-point data cross-registration to align the two sets of scan data. When the scan results resemble the 3D ex vivo specimen, click on Save Your Scan to save the scan as a .obj and a .jpeg file.

Next launch the Augmented Reality software and open the Augmented Reality VISE package. Drag and drop the .obj and .jpeg files from the desktop into Project Assets box.   

Put on the Head Mounted Display or HMD and raise the non-dominant wrist to open the menu in the HMD environment. When a square icon appears, point the finger at the icon and tap.

When the menu appears, click on File followed by Show All Apps and open the Holographic Remoting application. In the top toolbar, select Mixed Reality then press Remoting and Holographic Remoting for Play Mode.

Input the IP address of the HMD, then hit Play after ensuring that the HMD and device are connected to the same WiFi network. The headset should now display Receiving….

Drag the 3D object model into the left Hierarchy bar, open the dropdown menu, and drag the default into the Adjust Pivot popup window under Replace Object.

Now click on Apply Selected Objects then open the Medical_Object dropdown in the Hierarchy menu and choose anchor. Choose Move Medical_Object's pivot here in the Adjust Pivot popup.

Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup, select Apply Selected Texture File, and hit the Play button.

Now use the voice commands Open to initialize the hologram, Reposition to bring the object into the center of the view if it is not visible, Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size.

Pinch with the pointer finger and thumb to resize, move, and rotate the object. Control the object hands-free using voice commands Right or Left to rotate, Bigger or Smaller to resize, and Increase or Decrease transparency to adjust its transparency.

To realign the hologram, place the thumb and pointer finger together to grasp the object, then realign it against the resection bed, and release the object in the desired virtual spot. Finally, use the voice commands Lock or Unlock to secure the hologram in its place.