Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation

The scope of this study at the Princess Maxima Center for Pediatric Oncology is to develop and validate an augmented reality system. This system should accurately align holograms of moving organs. One current experimental challenge is validating that the hologram remains accurately aligned with the real-time position of a moving organ.

Currently, augmented reality validation techniques have only been described for rigid anatomical structures like bones. However, our protocol offers the advantage that it can be used to validate augmented reality for moving organs as well. To begin, open the 3D computer-aided design software and create a new file.

Select the Solid tab, and click Create Sketch to start a new design for an infrared marker. Add three or four small circles with a diameter of three millimeters by pressing Center Diameter Circle. Using the Line tool, connect the vertices of the triangle to the midpoints of the opposite sides and draw lines connecting the circles to calculate the center point.

At the center point, draw a circle using Center Diameter Circle, then draw rectangles connecting this center circle with each of the smaller circles using the two-point Rectangle tool. Extrude the center circular base and connecting rectangles to a thickness of two millimeters. Extrude the smaller circles to a thickness of five millimeters.

Press Create, then select Thread, and add threads to the three cones using an ISO Metric Profile to fit 6.4 millimeter infrared reflective spheres. Using the 3D Print or Export function, export the final model as an object file. Within the 3D computer-aided design software, select Measure to measure the x, y, and z-coordinates of the infrared reflective spheres relative to the center point.

Measure the locations of the center points of each small circle in correlation to the center of the entire shape. Launch the game development software. Import the IRTrackingOrgans_HoloLens project file and open the project.

Using a text editor, open the JavaScript Object Notation file saved in the Assets or StreamingAssets folder. Adapt the file to define the custom infrared marker using the previously recorded coordinates and following the default format. In the DINO Unity tab, select the ToolManager, click ResearchModeController, followed by JSON file and Parent transform, then click Create Objects Apply JSON Setting.

Import the created 3D infrared marker model. Select the patient-specific 3D model and change its transform coordinates in the Inspector window to match the position of the spawned markers in the scene. Then drag the patient-specific 3D model into the scene to insert it.

Transform the patient's 3D model to align the infrared marker to its surface. Position the infrared marker close to the center of the model to reduce positional error from the lever effect. Now, connect the patient scene to a button in the menu screen to allow for multiple case selections.

Navigate to Go to Assets, Scenes, and Menu scene. In the Hierarchy window, go to NearMenu4x2, then to ButtonCollection, and select the relevant button. In the Inspector window, go to Basic Events and under MenuScript.

LoadScene, type in the name of the patient scene. Create or obtain a 3D model of a kidney phantom with realistic anatomical structures. Import the 3D model into a 3D CAD modeling software.

Then use the Solid, Create, and Hole functions to integrate five registration pivot points on the side of the model. Set the Hole Type to Simple, Hole tap Type to Simple, Drill Point to Angle, Height to 0.5 millimeters, and Diameter to 4.0 millimeters. To fixate the electromagnetic reference sensor, create a cylinder with a hole and integrate it into the kidney model.

Start a new sketch and use Center Diameter Circle to draw a circle and an inner circle with a diameter of 2.8 millimeters. Extrude the outer circle by 16.5 millimeters. Then go to Modify, followed by Combine.

Select both the 3D kidney model and the cylinder, choose Join, and confirm by clicking OK.Then use the Export or 3D Print function to export the final integrated model. Next, use a flexible or semi-flexible filament, such as thermoplastic polyurethane, to print the kidney phantom following the procedure described previously. Place the field generator of the electromagnetic tracking system directly beneath the printed kidney phantom.

Remove all ferromagnetic objects from the surrounding environment to prevent electromagnetic field inhomogeneities. Then connect the electromagnetic sensor and the electromagnetic pointer to the tracking system. Attach the electromagnetic reference sensor to the 3D model by fixing it securely inside the cylinder using glue.

In 3D Slicer, import the 3D kidney model containing the pivot points. Use the Fiducial Registration Wizard. Select Place a control point, and digitally assign the registration landmarks.

To perform landmark registration in 3D Slicer, use the electromagnetic pointer to pinpoint the physical landmark points. Press Place a control point at each physical location to register them in the software. Then calculate the rigid linear registration transform by pressing Update.

Now, apply the calculated registration transform to the 3D model to link it with the electromagnetic reference sensor. Move the physical model and confirm that the digital version in 3D Slicer follows its motion. Launch the holographic display device, and open the holographic application configured earlier.

Then navigate to the correct patient-specific 3D model that is currently visualized in 3D Slicer. Now, attach the infrared marker to the specified location using glue, ensuring the fitted 6.4 millimeter infrared reflective spheres are in place as guided by the preoperative planning. Use the electromagnetic pointer to digitally identify the target points as seen through the holographic visualization.

Save the resulting set of EM sensor coordinates. Calculate the error by comparing the saved target coordinates to the actual placed landmarks to validate the accuracy of the holographic visualization. Across all participants, the Point Localization Error, or PLE, showed a median value of 8.74 millimeters, with individual measurements ranging from 2.78 to 13.20 millimeters.

Surgeon 2 consistently achieved the lowest PLE measurements, including the two most accurate localizations at 2.78 and 3.48 millimeters. The largest localization error was observed during the third measurement by Surgeon 3 with a PLE of 13.20 millimeters. This protocol will assist others in deploying holographic projects and accurately validating their augmented reality system in a preclinical setting.

Our surgical research group will soon start with the automated holographic tracking for multiple pediatric surgical cases. The movable organs are tracked based on machine learning algorithms and RGB camera feeds.