Name: Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation (Video) | JoVE Journal
Uploaded: 2025-09-02T14:15:00.000+00:00
Duration: 10 min 25 s
Description: Discover infrared marker tracking for holographic navigation using electromagnetic tracking and 3D Slicer. This protocol validates augmented reality alignment with moving organs in preclinical settings.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

We thank Hisham Iqbal for his expertise and support in establishing infrared marker tracking using the HoloLens 2, based on the open DINO-DLL repository.

This study followed the guidelines of our institution and was not subjected to the Medical Research Involving Human Subjects Act (WMO). Therefore, it was not required to obtain informed consent from the participants.
1. Preparing hardware and software packages for 3D modeling and AR application deployment
<ol>
	<li>Download and install the following software programs on a personal computer (PC) running Microsoft Windows 10.
	<ol>
		<li>Download Unity Hub v3.11.1 and Unity v. 2019.4.22f1 from&#160;https://unity.com/download. Include Visual Studio 2019 during the installation of Unity 2019.4.22f1. Link to manual: https://docs.unity3d.com/2019.4/Documentation/Manual/index.html</li>
		<li>Download MeshMixer v. 3.5.0 from https://apps.autodesk.com/FUSION/en/Detail/Index?id=4108920185261935100&#38;appLang=en&#38;os=Win64, the link to manual is https://help.autodesk.com/view/MSHMXR/2019/ENU/</li>
		<li>Download 3DSlicer v. 5.6.2 from https://download.slicer.org/, the link to manual is https://slicer.readthedocs.io/en/latest/</li>
		<li>Download Autodesk Fusion v. 2.0.21508 fromhttps://www.autodesk.com/products/fusion-360/personal, the link to manual is https://help.autodesk.com/view/fusion360/ENU/</li>
		<li>Download Bambu Studio v. 01.09.07.52 from https://bambulab.com/en/download/studio, the link to manual is https://wiki.bambulab.com/en/studio-handy</li>
	</ol>
	</li>
</ol>
2. Designing and printing custom infrared markers
<ol>
	<li>Design infrared markers in a 3D Design Software as described below.
	<ol>
		<li>Open the 3D Computer-aided design (CAD) software (see Table of Materials) and create a new file.</li>
		<li>Select the SOLID tab and click Create Sketch to begin sketching a new design for an infrared marker.</li>
		<li>Add three or four small circles with a diameter of 3 mm by pressing Center Diameter Circle. These circles serve as attachment points for screws.</li>
		<li>Calculate the center point of the infrared marker by connecting the vertices of the triangle to the midpoints of the opposite sides. Press Line and connect all circles by drawing lines from one side to the point of a circle.</li>
		<li>Create a circle as the base of the infrared marker on the center point using Center Diameter Circle. Use a 3-point Rectangle to draw rectangles that connect the center circle with each of the three or four smaller circles.</li>
		<li>Extrude the circular base and rectangles to a thickness of 2 mm, and the small circles to 5 mm.</li>
		<li>Add a thread to each of the three cones using an ISO Metric Profile (e.g., M3 &#215; 0.5, 6g, Right Hand) to accommodate 6.4 mm infrared reflective spheres by pressing Create and then Thread.</li>
		<li>Export the model as an object (OBJ) file using the 3D Print or Export function.</li>
		<li>Within 3D CAD software, measure the XYZ coordinates of the infrared reflective spheres in correlation to the center point by selecting Measure. Measure the locations of the center points of the circles in correlation to the center point of the shape. Use these coordinates in step 4.1.2.</li>
	</ol>
	</li>
	<li>Print 3D markers as described below.
	<ol>
		<li>Import the exported stereolithography (.STL) file of the infrared marker into software suitable for the 3D printer by dragging it into the scene (see Table of Materials).</li>
		<li>Configure the slicing parameters, including the layer height (0.08 mm is the smallest possible layer height to avoid inaccurate printing of the thread for the infrared reflective sphere) by pressing Quality &#62; Layer Height. Add support to the design by pressing Support &#62; Enable Struct.</li>
		<li>Export the slicing file to the 3D printer by clicking Slice All, export the file by clicking Export all Slice File, and print the 3D model using a 3D printer (see Table of Materials) with polylactic acid filament (e.g., PLA; see Table of Materials).</li>
	</ol>
	</li>
</ol>
3. Preparing the 3D patient-specific model of the kidney
<ol>
	<li>Model segmentation
	<ol>
		<li>Open the 3D segmentation software (see Table of Materials) and import the patient MRI/CT data using the Import DICOM files.</li>
		<li>Go to the Segment Editor, choose the appropriate Source volume, and create a new segmentation by clicking Add.</li>
		<li>Select manual or semi-automatic segmentation based on imaging modality.</li>
		<li>For manual segmentation, use the Paint and Erase tool to segment the tumor and surrounding relevant structures in each slice.</li>
		<li>For semi-automatic segmentation, consider using options like Threshold with the appropriate threshold range for the specific structure, and Scissors to segment out the irrelevant structures.</li>
		<li>In the Data screen, select the Created Segmentation and then right-click to go to the Export Visible Segments to Models button. Ensure that the Eye option is selected on the right side of the screen.</li>
		<li>Export the STL files of the models by clicking Save and save the files as a .STL file.</li>
	</ol>
	</li>
	<li>Post-processing of patient-specific model
	<ol>
		<li>Import the STL file into a mesh editor (see Table of Materials) and reduce the number of triangles by selecting 3D model &#62; Edit and then reduce by a percentage that reduces the triangles without deforming the visual aspect of the 3D model, and press Accept.</li>
		<li>Ensure that the target points are visually represented within the 3D model of the holographic application for further validation. Press Add spheres and place them on the 3D model.</li>
		<li>Export the 3D models to an OBJ file format by pressing File &#62; Export. Ensure that the 3D model has approximately 100,000 polygons by selecting the model and reduce the polygons by pressing Edit &#62; Reduce. Higher polygon counts necessitate more operations from the Graphics Processing Unit, so reducing the number of polygons in the scene can substantially decrease the render time.</li>
	</ol>
	</li>
</ol>
4. Preparing the holographic application
<ol>
	<li>Configure the IRTrackingOrgans_HoloLens Project as described below.
	<ol>
		<li>Launch the game development software (see Table of Materials) and import the IRTrackingOrgans_HoloLens project and open it.</li>
		<li>Adapt the JavaScript Object Notation (JSON) file using a text-editor, following the default format, to implement a custom infrared marker based on the coordinates measured in step 2.1.10. The JSON file is saved in Assets/StreamingAssets.</li>
		<li>Go to the DINO Unity tab, select the ToolManager &#62; ResearchModeController &#62; JSON file and Parent transform, and click Create Objects &#38; Apply JSON Setting.</li>
		<li>Import the virtual infrared marker 3D model as an asset that was created in step 1.1.</li>
		<li>Transform the virtual infrared marker 3D model to the position of the spawned markers in the scene by selecting the model and changing the transform coordinates in the inspector window.</li>
		<li>Insert a patient-specific 3D model into the scene by selecting and dragging it into the scene.</li>
		<li>Transform the patient 3D model to the correct place, so that the infrared marker touches the surface of the 3D model. Position the infrared marker close to the center of the 3D object to minimize inaccuracies due to the lever effect.</li>
	</ol>
	</li>
	<li>Connect the scene with a patient selection menu
	<ol>
		<li>For practical use and selection of multiple cases, connect the patient scene to a button in the menu screen. Go to Assets &#62; Scenes &#62; Menu scene.</li>
		<li>In the Hierarchy window, go to NearMenu4x2 and ButtonCollection and then the relevant button.</li>
		<li>In the Inspector window, go to Basic Events and under MenuScript.LoadScene type in the name of the patient scene.</li>
	</ol>
	</li>
	<li>Prepare the HMD for first-time deployment 
	NOTE: This section is only necessary if the application is deployed for the first time.
	<ol>
		<li>Log in to the HMD (see Table of Materials) device and set the device in Research Mode. Go to Settings &#62; Update &#38; Security &#62; For Developers &#62; Turn on Developer Features and Device Discovery.</li>
		<li>Pair the HMD to a PC (Wi-Fi or USB-C). If this is the first time connecting, follow these steps: Find the HMD&#39;s IP address in the developer tab, fill in the IP address in a web browser to connect to the Device Portal, and pair the device by generating a PIN and filling in the PIN.</li>
	</ol>
	</li>
	<li>Build and deploy the application to an HMD
	<ol>
		<li>Add the scenes to the build by going to File &#62; Build Settings, adding the scene in the following order: Menu &#62; Tracking Scene by pressing Add Open Scenes.</li>
		<li>Build the project using the Universal Windows Platform, Target Device HoloLens, and Architecture x64. Click Build and select a build map.</li>
		<li>Open the build file (.sln) with Visual Studio 2019 and change the platform to ARM64. Then open Properties by right clicking on the .sln File in the Solution Explorer and inside Debugging, type in the IP-address of the HMD under Machine Name.</li>
		<li>Deploy the application to the HMD by selecting Debug &#62; Start without Debugging.</li>
		<li>Launch the HMD and open the holographic application. Subsequently, navigate to the patient Menu screen and select the Appropriate Case to initiate holographic visualization and guidance.</li>
	</ol>
	</li>
</ol>
5. Validation of holographic visualization of moving organs
<ol>
	<li>Semi-deformable phantom printing
	<ol>
		<li>Create or obtain a 3D model of a kidney phantom with realistic anatomical structures.</li>
		<li>Import the 3D model into a 3D CAD modelling software and integrate five registration pivot points onto the side of the model by using Solid &#62; Create &#62; Hole &#62; with settings Hole Type: Simple, Hole tap Type: Simple, Drill Point: Angle, Height: 0.5 mm, and Diameter: 4.0 mm.</li>
		<li>Integrate a cylinder with a hole into the 3D model to fixate the EM reference sensor for further validation steps.
		<ol>
			<li>Create a Sketch with a circle and an inner circle with a diameter of 2.8 mm using Center Diameter Circle. Extrude the outer circle by 16.5 mm.</li>
			<li>Combine the cylinder with the 3D model by Modify &#62; Combine &#62; Select 3D model and cylinder &#62; Join &#62; OK.</li>
		</ol>
		</li>
		<li>Export the 3D model using the Export or 3D print function.</li>
		<li>Use a flexible or semi-flexible filament, such as TPU, (see Table of Materials) to print the kidney phantom according to the procedure described in step 2.2.</li>
	</ol>
	</li>
	<li>3D Slicer set-up using EM-tracking system
	<ol>
		<li>See the extensive 3D Slicer and SlicerIGT tutorial (https://www.slicerigt.org/wp/user-tutorial/) for setting up an EM-system with 3D-Slicer.&#160; 
		NOTE: This section of the protocol presumes that the setup of 3DSlicer, EM-tracking configuration, and connection is well-understood and correctly set up.</li>
		<li>Place the field generator of the EM tracking system (see Table of Materials) directly beneath the phantom. Remove all ferromagnetic materials from the environment to avoid electromagnetic field inhomogeneities.</li>
		<li>Connect the EM sensor (see Table of Materials) and the EM pointer (see Table of Materials) to the EM-tracking system. Ensure that the transforms of these tools are accurately visualized in 3DSlicer.</li>
		<li>Attach the EM-reference sensor (e.g., NDI Aurora 6DOF Cable Tool) to the 3D model by securing it in the cylinder with glue.</li>
		<li>In 3D Slicer, import the 3D model with pivot-points and allocate the landmark points digitally using the Fiducial Registration Wizard &#62; Place a control Point.</li>
		<li>Perform landmark registration by pinpointing the landmark points in real-life with the EM pointer, press Place a control point and register them in 3DSlicer. Calculate the rigid linear registration transform by pressing Update.</li>
		<li>After registration, apply the registration transform to the 3D model to establish a link between it and the EM-reference sensor. Subsequently, if the 3D model is physically moved, its digital counterpart in 3DSlicer should move; accordingly, confirm this visually by observing the movements.</li>
	</ol>
	</li>
	<li>Holographic validation
	<ol>
		<li>Launch the device and open the holographic application of step 4.4.5. Subsequently, navigate to the correct patient 3D model, which is also visualized in 3DSlicer.</li>
		<li>Fixate the infrared marker to the correct place using glue, with the fitted 6.4 mm infrared markers (see Table of Materials), as visualized by the preoperative planning.</li>
		<li>Use the EM pointer to digitally pinpoint the target points based on holographic visualization. Save the set of EM sensor coordinates.</li>
		<li>Calculate the error in locating the target landmarks compared to the placed landmarks to quantitatively validate the holographic visualization.</li>
	</ol>
	</li>
</ol>

<ol><li>Bortot, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Image-guided+cancer+surgery%3A+a+narrative+review+on+imaging+modalities+and+emerging+nanotechnology+strategies%5BTitle%5D)+AND+%22J+Nanobiotechnol%22%5BJournal%5D)">Image-guided cancer surgery: a narrative review on imaging modalities and emerging nanotechnology strategies.</a> J Nanobiotechnol. 21 (1), 155(2023).</li>
<li>Liu, Q., Luo, D., Cai, S., Li, Q., Li, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Circumferential+resection+margin+as+a+prognostic+factor+after+rectal+cancer+surgery%3A+A+large+population-based+retrospective+study%5BTitle%5D)+AND+%22Cancer+Med%22%5BJournal%5D)">Circumferential resection margin as a prognostic factor after rectal cancer surgery: A large population-based retrospective study.</a> Cancer Med. 7 (8), 3673-3681 (2018).</li>
<li>Cost, N. G., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Pathological+review+of+Wilms+tumor+nephrectomy+specimens+and+potential+implications+for+nephron+sparing+surgery+in+Wilms+tumor%5BTitle%5D)+AND+%22J+Urol%22%5BJournal%5D)">Pathological review of Wilms tumor nephrectomy specimens and potential implications for nephron sparing surgery in Wilms tumor.</a> J Urol. 188 (4 Suppl), 1506-1510 (2012).</li>
<li>Taghavi, K., Sarnacki, S., Blanc, T., Boyer, O., Heloury, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+rationale+for+nephron-sparing+surgery+in+unilateral+non-syndromic+Wilms+tumour%5BTitle%5D)+AND+%22Pediatr+Nephrol%22%5BJournal%5D)">The rationale for nephron-sparing surgery in unilateral non-syndromic Wilms tumour.</a> Pediatr Nephrol. 39 (4), 1023-1032 (2024).</li>
<li>Mezger, U., Jendrewski, C., Bartels, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Navigation+in+surgery%5BTitle%5D)+AND+%22Langenbecks+Arch+Surg%22%5BJournal%5D)">Navigation in surgery.</a> Langenbecks Arch Surg. 398 (4), 501-514 (2013).</li>
<li>Herrlich, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Instrument-mounted+displays+for+reducing+cognitive+load+during+surgical+navigation%5BTitle%5D)+AND+%22Int+J+Comput+Assist+Radiol+Surg%22%5BJournal%5D)">Instrument-mounted displays for reducing cognitive load during surgical navigation.</a> Int J Comput Assist Radiol Surg. 12 (9), 1599-1605 (2017).</li>
<li>Portnoy, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Three-dimensional+technologies+in+presurgical+planning+of+bone+surgeries%3A+current+evidence+and+future+perspectives%5BTitle%5D)+AND+%22Int+J+Surg%22%5BJournal%5D)">Three-dimensional technologies in presurgical planning of bone surgeries: current evidence and future perspectives.</a> Int J Surg. 109 (1), 3-10 (2023).</li>
<li>Meulstee, J. W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Toward+Holographic-Guided+Surgery%5BTitle%5D)+AND+%22Surg+Innov%22%5BJournal%5D)">Toward Holographic-Guided Surgery.</a> Surg Innov. 26 (1), 86-94 (2019).</li>
<li>Glas, H. H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+Reality+Visualization+for+Image-Guided+Surgery%3A+A+Validation+Study+Using+a+Three-Dimensional+Printed+Phantom%5BTitle%5D)+AND+%22J+Oral+Maxillofacial+Surg%22%5BJournal%5D)">Augmented Reality Visualization for Image-Guided Surgery: A Validation Study Using a Three-Dimensional Printed Phantom.</a> J Oral Maxillofacial Surg. 79 (9), 1943.e1-1943.e10 (2021).</li>
<li>Fitski, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((MRI-Based+3-Dimensional+Visualization+Workflow+for+the+Preoperative+Planning+of+Nephron-Sparing+Surgery+in+Wilms%27+Tumor+Surgery%3A+A+Pilot+Study%5BTitle%5D)+AND+%22J+Healthcare+Eng%22%5BJournal%5D)">MRI-Based 3-Dimensional Visualization Workflow for the Preoperative Planning of Nephron-Sparing Surgery in Wilms' Tumor Surgery: A Pilot Study.</a> J Healthcare Eng. 2020 (1), 8899049(2020).</li>
<li>van der Zee, J. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Virtual+Resection%3A+A+New+Tool+for+Preparing+for+Nephron-Sparing+Surgery+in+Wilms+Tumor+Patients%5BTitle%5D)+AND+%22Curr+Oncol%22%5BJournal%5D)">Virtual Resection: A New Tool for Preparing for Nephron-Sparing Surgery in Wilms Tumor Patients.</a> Curr Oncol. 29 (2), 777-784 (2022).</li>
<li>Fida, B., Cutolo, F., di Franco, G., Ferrari, M., Ferrari, V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+in+open+surgery%5BTitle%5D)+AND+%22Updates+Surg%22%5BJournal%5D)">Augmented reality in open surgery.</a> Updates Surg. 70 (3), 389-400 (2018).</li>
<li>Moreta-Martinez, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+in+computer-assisted+interventions+based+on+patient-specific+3D+printed+reference%5BTitle%5D)+AND+%22Healthc+Technol+Lett%22%5BJournal%5D)">Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference.</a> Healthc Technol Lett. 5 (5), 162-166 (2018).</li>
<li>van Doormaal, T. P. C., van Doormaal, J. A. M., Mensink, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Clinical+Accuracy+of+Holographic+Navigation+Using+Point-Based+Registration+on+Augmented-Reality+Glasses%5BTitle%5D)+AND+%22Oper+Neurosurg+%28Hagerstown%29%22%5BJournal%5D)">Clinical Accuracy of Holographic Navigation Using Point-Based Registration on Augmented-Reality Glasses.</a> Oper Neurosurg (Hagerstown). 17 (6), 588-593 (2019).</li>
<li>Zhao, Z., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+technology+in+image-guided+therapy%3A+State-of-the-art+review%5BTitle%5D)+AND+%22Proc+Inst+Mech+Eng+H%22%5BJournal%5D)">Augmented reality technology in image-guided therapy: State-of-the-art review.</a> Proc Inst Mech Eng H. 235 (12), 1386-1398 (2021).</li>
<li>Doughty, M., Ghugre, N. R., Wright, G. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmenting+Performance%3A+A+Systematic+Review+of+Optical+See-Through+Head-Mounted+Displays+in+Surgery%5BTitle%5D)+AND+%22J+Imaging%22%5BJournal%5D)">Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery.</a> J Imaging. 8 (7), 203(2022).</li>
<li>Vávra, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Recent+Development+of+Augmented+Reality+in+Surgery%3A+A+Review%5BTitle%5D)+AND+%22J+Healthcare+Eng%22%5BJournal%5D)">Recent Development of Augmented Reality in Surgery: A Review.</a> J Healthcare Eng. 2017, 4574172(2017).</li>
<li>Iqbal, H., Baena, F. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Semi-Automatic+Infrared+Calibration+for+Augmented+Reality+Systems+in+Surgery%5BTitle%5D)+AND+%222022+IEEE%2FRSJ+Int+Conf+Intelligent+Robots+Syst+%28IROS%29%22%5BJournal%5D)">Semi-Automatic Infrared Calibration for Augmented Reality Systems in Surgery.</a> 2022 IEEE/RSJ Int Conf Intelligent Robots Syst (IROS). , 4957-4964 (2022).</li>
<li>Zhao, W., Jiang, D., Queralta, J. P., Westerlund, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Multi-Scale+Supervised+3D+U-Net+for+Kidneys+and+Kidney+Tumor+Segmentation%5BTitle%5D)+AND+%22arXiv%22%5BJournal%5D)">Multi-Scale Supervised 3D U-Net for Kidneys and Kidney Tumor Segmentation.</a> arXiv. , (2020).</li>
<li>Elloumi, N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((CT+Images+Segmentation+Using+a+Deep+Learning-Based+Approach+for+Preoperative+Projection+of+Human+Organ+Model+Using+Augmented+Reality+Technology%5BTitle%5D)+AND+%22Int+J+Comp+Intel+Appl%22%5BJournal%5D)">CT Images Segmentation Using a Deep Learning-Based Approach for Preoperative Projection of Human Organ Model Using Augmented Reality Technology.</a> Int J Comp Intel Appl. 22 (02), 2350006(2023).</li>
<li>von Haxthausen, F., Moreta-Martinez, R., Pose Díez de la Lastra, A., Pascau, J., Ernst, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((UltrARsound%3A+in+situ+visualization+of+live+ultrasound+images+using+HoloLens+2%5BTitle%5D)+AND+%22Int+J+Comput+Assist+Radiol+Surg%22%5BJournal%5D)">UltrARsound: in situ visualization of live ultrasound images using HoloLens 2.</a> Int J Comput Assist Radiol Surg. 17 (11), 2081-2091 (2022).</li>
<li>Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery. Haouchine, N., et al. 2014 IEEE Int Conf Robotics Automat, , 4121-4126 (2014).</li>
</ol>

The authors have nothing to disclose.

The presented protocol outlines an approach to real-time organ tracking and holographic visualization, including validation for moving tumors and organs in a phantom setting. Leveraging infrared marker-based tracking with an HMD, this method has the potential to better maintain correct anatomical holographic overlays during the manipulation of moving organs. Despite its application for tracking the kidney, this method could also be explored in other clinical fields, such as open surgery for the liver or lungs, with minimal adjustments in the segmentation protocol. Secondly, infrared markers could potentially be used with various surgical techniques, such as laparoscopic surgery. In such cases, infrared reflective stickers could be detected on laparoscopic images to track organs, tools, or anatomical landmarks. However, the application of this technique to other clinical fields or surgical techniques may introduce errors due to variations in deformability of the target organ or technical limitations, such as a limited field of view, which require validation in preclinical experiments. 
Step 1 of the protocol is focused primarily on setting up the necessary hardware and software tools. This setup requires numerous applications and steps, so it's crucial that all software packages are installed correctly, including any necessary extensions, to avoid deployment issues downstream. It is not anticipated that different versions of software will cause issues, even though the combination of the game development software and the integrated development environment is crucial. 
In step 2, the process of creating customized infrared markers is described. This step becomes particularly important if the tracking will be utilized for other applications. The flexibility to modify the shape of the infrared marker ensures its potential suitability for diverse preclinical applications. Furthermore, users can explore various design options to improve the infrared marker's adherence to the surface of an organ and enhance the accuracy of tracking the infrared marker. Additionally, testing multiple infrared marker diameters can lead to improved detection over distances exceeding 60 cm. 
In step 3, patient-specific 3D modeling based on medical imaging is described. Accurate segmentation of the kidney and tumor is crucial, as it directly influences the accuracy of surgical guidance. Poor segmentation may result in misleading visualizations that compromise surgical precision19. Secondly, this step is the most time-consuming. Integrating fully automatic segmentation methods can expedite the protocol, reducing the necessity for manual and semi-automatic adjustments while ensuring precise anatomical segmentation20. Optimizing the polygon count is crucial for achieving optimal AR rendering performance. If this optimization is not performed, the performance of the HMD is significantly compromised.
In step 4, the configuration of the holographic application is outlined, following the implementation of DINO-DLL. One critical aspect is proper alignment between the infrared marker positions and the holographic anatomical models, as this affects manual registration accuracy. Especially, the lever effect should be minimized to prevent inaccuracy at further distances from the center of the infrared marker. Further improvements could include implementing additional registration methods. Moreover, the current system exhibits an acceptable tracking rate for continuous visualization, which aligns with literature21. Thirdly, further improvements should involve implementing a Kalman filter to reduce noise in the infrared marker tracking data, thereby eliminating the jitter of holographic visualization. 
In step 5, the framework provides a holographic validation method utilizing EM tracking. This protocol is useful for validating the accuracy of holograms in a phantom setting, as it provides a quantitative assessment of holographic accuracy for moving organs. A crucial step here is the precise integration of EM-tracking sensors within semi-deformable 3D-printed phantoms. Users must ensure accurate calibration of the EM sensors and landmark registration in 3D Slicer. If validation errors occur, re-registration or removing any metallic objects could enhance the validation accuracy. To further validate the clinical feasibility, ex vivo organs could be employed to more accurately simulate the surgical tissue22. 
This protocol serves as a comprehensive guide for researchers aiming to implement AR solutions for organ tracking and validate these systems in phantom experiments. Additionally, it provides a broadly applicable validation setup that can be easily utilized across different clinical scenarios, particularly for validating AR methods for moving organs. Given the complexity of deploying holographic applications, this framework facilitates the transition from conceptual AR-based solutions to preclinical validation.

In surgical oncology, accurately identifying the tumor location and relation to adjacent healthy tissues is crucial for achieving complete tumor resection while preserving healthy tissue1. Incomplete resections can lead to local recurrence and a decreased survival rate2,3, while excessive tissue removal may impair function and quality of life4. Surgical navigation systems hold promise in improving radical resections while preserving healthy tissue, by providing surgeons with intraoperative guidance that can potentially lead to improved clinical outcomes5. However, conventional surgical navigation systems typically present two-dimensional (2D) anatomical information on screens positioned outside the surgical field. This approach forces surgeons to mentally correlate the displayed 2D information with the patient's actual three-dimensional (3D) anatomy, increasing cognitive load6. Although recent advancements in 3D modeling provide surgeons with a better understanding of the tumor relation with surrounding anatomical structures7, this information is still visualized outside the operating area, maintaining the switching focus problem6,8. These limitations of surgical navigation systems can contribute to potential errors in the use of surgical navigation and potentially lead to suboptimal surgical outcomes9. 
To overcome the above-mentioned limitations, augmented reality (AR) has emerged as a promising solution by visualizing anatomical structures and resection borders in 3D on patient10,11. By superimposing preoperative 3D models, which are segmented based on Magnetic Resonance Imaging (MRI) or Computed Tomography (CT) data, anatomy can be visualized. In systematic reviews, the potential benefits of AR for open surgery in adolescent patients were highlighted12, and preliminary work in this field demonstrates the feasibility of patient-specific 3D guides equipped with visual markers for automatic registration13. Van Doormaal et al. developed a navigation system with an AR device by using a point-based registration and a pointer with an image target for neurosurgery14. They assessed the developed AR application in the operating room on patients before surgery and in a phantom experiment, which showed a fiducial registration error of 7.2 mm and 4.4 mm, respectively.
Despite the promising progress, these registration systems are often rigid, lacking real-time tracking of target organs, and so, there remains a need for real-time tracking of organ movement15,16. This particularly holds true for moving organs, which are manipulated during surgery, such as the kidney and liver, which can result in inaccurate guidance, the need for reregistration, which takes considerable time, and potential harm to healthy tissue or incorrect resections17. To further address these issues, a novel AR system based on an application presented by Iqbal et al. to incorporate infrared markers for continuous organ tracking was developed18. This development allows the AR overlay to dynamically adapt to real-time changes in organ position, thereby maintaining spatial accuracy and potentially enhancing surgical precision. By combining rigid registration with dynamic infrared marker-based tracking, this system offers significant advancement towards achieving accurate, real-time holographic guidance in surgery. 
This protocol presents an infrared marker-based AR navigation and preclinical validation system for a Head-Mounted Display (HMD). We aim to develop and validate a real-time augmented reality navigation system to maintain accurate holographic overlays of moving organs in a preclinical setting. First, the protocol provides a description of how a holographic application is prepared that uses infrared markers with a dimension of 32 mm (width) x 15 mm (length) x 6 mm (height) to track phantom organs in real-time, maintaining the overlay of 3D holograms independent of movement. We use a kidney phantom, printed with thermoplastic polyurethane filament (TPU), as an example model for a moving organ. Secondly, it gives an overview of how to design and print custom infrared markers and how to integrate these markers into the holographic visualization application. This allows other researchers and clinicians to adapt the application to other preclinical phantom scenarios that involve the simulation of open surgery and moving organs. Finally, a validation method based on electromagnetic tracking offers quantitative measurements to compute accuracy, offering preclinical validation of holographic guidance in phantom experiments. A limitation of this methodology is the absence of an automated registration procedure, which constrains the accuracy of this system. However, this approach assists users in determining the appropriateness of their developed AR technique for their clinical application.

<table><tbody><tr><td>3D Slicer (v5.6.2)</td><td>Slicer</td><td>N/A</td><td>Medical image segmentation software and necessary for electromagnetic tracking validation</td></tr><tr><td>6.4 mm (1/4&quot;) M3 Markers Infrarec Markers</td><td>OptiTrack</td><td>N/A</td><td>Infrared reflective spheres which should be attached to marker for tracking&amp;nbsp;</td></tr><tr><td>Autodesk Fusion 360 (v2.0.21508)</td><td>Autodesk</td><td>N/A</td><td>CAD software for designing infrared markers and phantoms</td></tr><tr><td>Bambu Studio (v01.09.07.52)</td><td>Bambu Lab</td><td>N/A</td><td>3D printing slicing software for Bambu 3D printers</td></tr><tr><td>Bambu X1 Carbon</td><td>Bambu Lab</td><td>N/A</td><td>3D printer used for infrared markers and phantom models</td></tr><tr><td>HoloLens 2</td><td>Microsoft</td><td>N/A</td><td>Augmented Reality Head-Mounted Display for AR visualization</td></tr><tr><td>IRTrackingOrgans_HoloLens&amp;nbsp;</td><td>Open source</td><td>N/A</td><td>Unity-based application supporting IR marker tracking</td></tr><tr><td>MeshMixer (v3.5.0)</td><td>Autodesk</td><td>N/A</td><td>Used for mesh editing and polygon reduction</td></tr><tr><td>NDI Aurora</td><td>Northern Digital Inc.</td><td>N/A</td><td>Electromagnetic tracking system for validation</td></tr><tr><td>NDI Aurora 6DOF Cable Tool</td><td>Northern Digital Inc.</td><td>N/A</td><td>Sensor for registering movement of phantom organs</td></tr><tr><td>NDI Aurora 6DOF Probe</td><td>Northern Digital Inc.</td><td>N/A</td><td>Used to identify landmark locations on the phantom</td></tr><tr><td>Polylactic Acid Filament</td><td>Any Manufacturer</td><td>N/A</td><td>Filament for printing rigid parts like infrared markers</td></tr><tr><td>Thermoplastic Polyurethane Filament</td><td>Any Manufacturer</td><td>N/A</td><td>Semi-flexible filament for printing deformable kidney phantom</td></tr><tr><td>Unity Hub (v3.11.1) and Unity (v2019.4.22f1)</td><td>Unity Technologies</td><td>N/A</td><td>Game development software for AR application development and deployment</td></tr><tr><td>Visual Studio 2019</td><td>Microsoft</td><td>N/A</td><td>Required IDE for Unity integration and deployment</td></tr></tbody></table>

technical approach for infrared tracking for soft tissue navigation with a holographic head-mounted display and preclinical validation

Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation

Augmented Reality (AR) has the potential to enhance surgical guidance by superimposing three-dimensional (3D) anatomical information directly onto the patient during surgical procedures. However, the practical implementation of AR encounters significant challenges, particularly in accurately tracking organs that move freely during surgical manipulation. Consequently, reliable organ tracking methods are necessary to maintain precise holographic overlays intraoperatively. Preclinical validation of holographic visualizations regarding accuracy poses additional challenges, requiring experimental protocols for quantitative assessment. This protocol addresses these two challenges: it describes a comprehensive approach for developing AR visualization applications using custom-made infrared markers for real-time organ tracking using a Head-Mounted Display (HMD), and it provides a validation framework leveraging electromagnetic (EM) tracking to validate holographic accuracy in phantom experiments. This work outlines step-by-step guidance for creating patient-specific 3D models from medical imaging, designing and manufacturing custom infrared markers, integrating these markers into an AR application for an HMD, and deploying them for surgical navigation. Additionally, it details a validation procedure by using EM-tracking to quantitatively measure the precision of holographic visualizations in semi-deformable kidney phantoms. Therefore, this protocol both facilitates real-time organ tracking and establishes a preclinical validation methodology. Implementing real-time organ tracking could enhance surgical guidance for free-moving organs by accurately overlaying holograms, potentially leading to improved surgical accuracy and better patient outcomes.

A kidney phantom was used to demonstrate the performance of the infrared-tracking system for organ tracking and to validate the holographic validation setup in moving organs. The complete workflow is outlined in Figure 1.
First, the kidney was semi-automatically segmented based on MRI data using the thresholding tool in 3DSlicer. The resulting 3D model was exported and imported into 3D CAD software to reduce the polygon count. A second model was saved, and five target points were integrated into this model using the sphere tool (Figure 2). This model was used for the technical validation of the holographic display. The first version of the model, without target points, was imported into Autodesk Fusion. Five pivot points were integrated into this model, and the cylinder was integrated to facilitate the EM sensor. Using 3D slicing software, the 3D model was prepared for 3D printing. TPU with a print density of 8% was used to create a minimally flexible kidney surface.
A standardized infrared marker was designed, 3D-printed, and fitted with infrared reflective spheres (6.4 mm diameter). From this infrared marker, the coordinates of the infrared marker were measured in correlation to the center point. Inside the game development software application, the JSON file containing the coordinates of the infrared marker was imported. Secondly, the 3D model of the kidney was imported, with target points for validation purposes. Also, for visualization purposes, the infrared marker model was imported and translated to the position of the points implemented by the JSON file. The 3D model was transformed to the center of the infrared marker (Figure 3), and additional shaders were applied. After integrating the patient menu scene, the application was deployed on the HMD.
Based on the placement of the IR-markers, the holographic 3D model is visualized on the kidney inside a pediatric abdominal phantom using the HMD (Figure 4). It had a tracking rate of 11.6 Hz. However, for distances exceeding 60 cm, the HMD loses the ability to track the infrared markers. Secondly, the continuous tracking and noise in the infrared-marking tracking causes the holographic overlay to flicker, resulting in inaccurate visualization.
For validation purposes, the EM-tracking system was connected to 3D Slicer through the Plus Server. An EM sensor was placed on the phantom kidney for tracking (Figure 2). After point-based registration, the 3D model was registered with a median accuracy of 0.59 mm, which proved to be an accurate method for validating holographic accuracy (Figure 5). The median Point Localization Error was 8.74 mm (Interquartile Range: 6.38 - 10.85), based on input from three surgeons (Table 1).
The implementation of this AR tracking and visualization system involves a protocol that spans approximately 45-60 min. An experienced technical physician with 2 years of experience executed the entire protocol once to determine the duration of the individual steps of the protocol. Notably, certain steps are only necessary to be executed once. The essential steps for each patient include segmentation, model integration in the game development software, and scene configuration. Segmenting anatomical structures in patient-specific cases requires relatively more time due to the multiple anatomical structures involved, but segmenting the renal parenchyma and tumor can be completed within 30 min. Integrating the segmented 3D models into the application and aligning them with the infrared marker takes approximately 5 min of manual adjustments. Connecting the correct scene requires no more than 5 min. The game development project build time varies depending on hardware specifications but typically takes around 3 min, followed by approximately 10 min for deployment onto the HoloLens 2. Overall, excluding the validation setup, this protocol demonstrates a method for moving organ tracking in preclinical settings.
<img alt="Preoperative MRI to 3D kidney model; holographic visualization in surgical procedure diagram." class="xfigimg" src="/files/ftp_upload/68607/68607fig1.jpg" data-altupdatedat="1762509930000" data-alt="Figure 1"> 
Figure 1: Schematic overview of the workflow. The workflow shows steps that are required per patient in a phantom setting, including the preoperative phase, holographic, and intraoperative phases. The pre-operative phase consists of segmenting (see step 3) pre-operative medical imaging. Preparation of the holographic application consists of virtually planning the infrared marker placement on the 3D model (see step 4). In the intraoperative phase, the surgeons can select the correct patient and fix the infrared marker for holographic visualization and continuous tracking. <a href="https://www.jove.com/files/ftp_upload/68607/68607fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Preoperative IR marker and EM sensor placement diagram for surgical planning and alignment." class="xfigimg" src="/files/ftp_upload/68607/68607fig2.jpg" data-altupdatedat="1762509930000" data-alt="Figure 2"> 
Figure 2: Overview of kidney phantoms used in the validation methodology. Left: a 3D hologram of the kidney with the target points and virtual placement of the infrared marker. Middle: 3D phantom with integrated EM sensor and pivot-points for registration. Right: 3D printed phantom, with the infrared marker and cylinder for the EM sensor, used for the validation procedure. <a href="https://www.jove.com/files/ftp_upload/68607/68607fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="3D modeling interface, kidney structure in Unity, spatial design, debugging visual tools." class="xfigimg" src="/files/ftp_upload/68607/68607fig3.jpg" data-altupdatedat="1762509930000" data-alt="Figure 3"> 
Figure 3: Preparation of the holographic application in the game development software. The kidney model is transformed into an infrared marker. Secondly, shaders are applied to the kidney and to the target points. <a href="https://www.jove.com/files/ftp_upload/68607/68607fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="IR-guided surgical navigation setup; tools with IR-marker, IR-pointer for position tracking." class="xfigimg" src="/files/ftp_upload/68607/68607fig4.jpg" data-altupdatedat="1762509930000" data-alt="Figure 4"> 
Figure 4: Holographic visualization of the phantom experiment. Left: Placement of the infrared marker on the kidney. Right: Holographic visualization of target points in the correct order (1 to 5). Displacement of the holographic visualization is caused by the jitter in the infrared marker tracking. <a href="https://www.jove.com/files/ftp_upload/68607/68607fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="3D Slicer system diagram with EM-tracker, highlighting optical and pointer frames for surgical navigation." class="xfigimg" src="/files/ftp_upload/68607/68607fig5.jpg" data-altupdatedat="1762509930000" data-alt="Figure 5"> 
Figure 5: Set-up from the EM-tracking validation protocol for holographic visualization of moving organs. Green, Red, and Blue visualize the transformation of the necessary EM-tools for validation. Yellow and Green visualize the transformation regarding the Head-Mounted Display (HMD). <a href="https://www.jove.com/files/ftp_upload/68607/68607fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Participant</td>
			<td>Measurement</td>
			<td>GT-X (mm)</td>
			<td>GT-Y (mm)</td>
			<td>GT-Z (mm)</td>
			<td>Point-X (mm)</td>
			<td>Point-Y (mm)</td>
			<td>Point-Z (mm)</td>
			<td>PLE (mm)</td>
		</tr>
		<tr>
			<td>Surgeon 1</td>
			<td>1</td>
			<td>-67.02</td>
			<td>7.88</td>
			<td>297.50</td>
			<td>-76.72</td>
			<td>8.97</td>
			<td>295.49</td>
			<td>9.97</td>
		</tr>
		<tr>
			<td></td>
			<td>2</td>
			<td>-46.77</td>
			<td>4.78</td>
			<td>249.67</td>
			<td>-55.71</td>
			<td>-0.26</td>
			<td>243.61</td>
			<td>11.91</td>
		</tr>
		<tr>
			<td></td>
			<td>3</td>
			<td>-3.21</td>
			<td>-12.36</td>
			<td>244.46</td>
			<td>-9.99</td>
			<td>-3.03</td>
			<td>244.83</td>
			<td>11.54</td>
		</tr>
		<tr>
			<td></td>
			<td>4</td>
			<td>-15.06</td>
			<td>1.16</td>
			<td>273.72</td>
			<td>-20.00</td>
			<td>2.71</td>
			<td>272.70</td>
			<td>5.27</td>
		</tr>
		<tr>
			<td></td>
			<td>5</td>
			<td>-39.00</td>
			<td>5.40</td>
			<td>281.25</td>
			<td>-46.82</td>
			<td>6.91</td>
			<td>277.75</td>
			<td>8.70</td>
		</tr>
		<tr>
			<td>Surgeon 2</td>
			<td>1</td>
			<td>-67.02</td>
			<td>7.88</td>
			<td>297.50</td>
			<td>-63.60</td>
			<td>8.02</td>
			<td>292.12</td>
			<td>6.38</td>
		</tr>
		<tr>
			<td></td>
			<td>2</td>
			<td>-46.77</td>
			<td>4.78</td>
			<td>249.67</td>
			<td>-45.94</td>
			<td>2.73</td>
			<td>246.98</td>
			<td>3.48</td>
		</tr>
		<tr>
			<td></td>
			<td>3</td>
			<td>-3.21</td>
			<td>-12.36</td>
			<td>244.46</td>
			<td>-5.43</td>
			<td>-10.70</td>
			<td>244.27</td>
			<td>2.78</td>
		</tr>
		<tr>
			<td></td>
			<td>4</td>
			<td>-15.06</td>
			<td>1.16</td>
			<td>273.72</td>
			<td>-11.87</td>
			<td>0.80</td>
			<td>267.51</td>
			<td>7.00</td>
		</tr>
		<tr>
			<td></td>
			<td>5</td>
			<td>-39.00</td>
			<td>5.40</td>
			<td>281.25</td>
			<td>-35.54</td>
			<td>5.82</td>
			<td>273.28</td>
			<td>8.70</td>
		</tr>
		<tr>
			<td>Surgeon 3</td>
			<td>1</td>
			<td>-67.02</td>
			<td>7.88</td>
			<td>297.50</td>
			<td>-62.97</td>
			<td>7.87</td>
			<td>287.43</td>
			<td>10.85</td>
		</tr>
		<tr>
			<td></td>
			<td>2</td>
			<td>-46.77</td>
			<td>4.78</td>
			<td>249.67</td>
			<td>-44.59</td>
			<td>-0.42</td>
			<td>242.70</td>
			<td>8.96</td>
		</tr>
		<tr>
			<td></td>
			<td>3</td>
			<td>-3.21</td>
			<td>-12.36</td>
			<td>244.46</td>
			<td>2.23</td>
			<td>-20.32</td>
			<td>253.48</td>
			<td>13.20</td>
		</tr>
		<tr>
			<td></td>
			<td>4</td>
			<td>-15.06</td>
			<td>1.16</td>
			<td>273.72</td>
			<td>-10.73</td>
			<td>1.33</td>
			<td>266.14</td>
			<td>8.74</td>
		</tr>
		<tr>
			<td></td>
			<td>5</td>
			<td>-39.00</td>
			<td>5.40</td>
			<td>281.25</td>
			<td>-34.95</td>
			<td>5.93</td>
			<td>271.74</td>
			<td>10.35</td>
		</tr>
	</tbody>
</table>
Table 1: For each measurement, the ground truth (GT) coordinates of the target landmarks, their corresponding point location coordinates, and the PLE measured for all surgeons are provided.

This protocol offers a guide to implement infrared marker tracking for free-moving phantoms (e.g., organs) and holographic visualization using Augmented Reality. Additionally, it outlines a setup for preclinical validation of holographic navigation systems using electromagnetic tracking on free-moving phantoms.

Augmented Reality (AR) has the potential to enhance surgical guidance by superimposing three-dimensional (3D) anatomical information directly onto the ...

The scope of this study at the Princess Maxima Center for Pediatric Oncology is to develop and validate an augmented reality system. This system should accurately align holograms of moving organs. One current experimental challenge is validating that the hologram remains accurately aligned with the real-time position of a moving organ.Currently, augmented reality validation techniques have only been described for rigid anatomical structures like bones. However, our protocol offers the advantage that it can be used to validate augmented reality for moving organs as well. To begin, open the 3D computer-aided design software and create a new file.Select the Solid tab, and click Create Sketch to start a new design for an infrared marker. Add three or four small circles with a diameter of three millimeters by pressing Center Diameter Circle. Using the Line tool, connect the vertices of the triangle to the midpoints of the opposite sides and draw lines connecting the circles to calculate the center point.At the center point, draw a circle using Center Diameter Circle, then draw rectangles connecting this center circle with each of the smaller circles using the two-point Rectangle tool. Extrude the center circular base and connecting rectangles to a thickness of two millimeters. Extrude the smaller circles to a thickness of five millimeters.Press Create, then select Thread, and add threads to the three cones using an ISO Metric Profile to fit 6.4 millimeter infrared reflective spheres. Using the 3D Print or Export function, export the final model as an object file. Within the 3D computer-aided design software, select Measure to measure the x, y, and z-coordinates of the infrared reflective spheres relative to the center point.Measure the locations of the center points of each small circle in correlation to the center of the entire shape. Launch the game development software. Import the IRTrackingOrgans_HoloLens project file and open the project.Using a text editor, open the JavaScript Object Notation file saved in the Assets or StreamingAssets folder. Adapt the file to define the custom infrared marker using the previously recorded coordinates and following the default format. In the DINO Unity tab, select the ToolManager, click ResearchModeController, followed by JSON file and Parent transform, then click Create Objects Apply JSON Setting.Import the created 3D infrared marker model. Select the patient-specific 3D model and change its transform coordinates in the Inspector window to match the position of the spawned markers in the scene. Then drag the patient-specific 3D model into the scene to insert it.Transform the patient's 3D model to align the infrared marker to its surface. Position the infrared marker close to the center of the model to reduce positional error from the lever effect. Now, connect the patient scene to a button in the menu screen to allow for multiple case selections.Navigate to Go to Assets, Scenes, and Menu scene. In the Hierarchy window, go to NearMenu4x2, then to ButtonCollection, and select the relevant button. In the Inspector window, go to Basic Events and under MenuScript.LoadScene, type in the name of the patient scene. Create or obtain a 3D model of a kidney phantom with realistic anatomical structures. Import the 3D model into a 3D CAD modeling software.Then use the Solid, Create, and Hole functions to integrate five registration pivot points on the side of the model. Set the Hole Type to Simple, Hole tap Type to Simple, Drill Point to Angle, Height to 0.5 millimeters, and Diameter to 4.0 millimeters. To fixate the electromagnetic reference sensor, create a cylinder with a hole and integrate it into the kidney model.Start a new sketch and use Center Diameter Circle to draw a circle and an inner circle with a diameter of 2.8 millimeters. Extrude the outer circle by 16.5 millimeters. Then go to Modify, followed by Combine.Select both the 3D kidney model and the cylinder, choose Join, and confirm by clicking OK.Then use the Export or 3D Print function to export the final integrated model. Next, use a flexible or semi-flexible filament, such as thermoplastic polyurethane, to print the kidney phantom following the procedure described previously. Place the field generator of the electromagnetic tracking system directly beneath the printed kidney phantom.Remove all ferromagnetic objects from the surrounding environment to prevent electromagnetic field inhomogeneities. Then connect the electromagnetic sensor and the electromagnetic pointer to the tracking system. Attach the electromagnetic reference sensor to the 3D model by fixing it securely inside the cylinder using glue.In 3D Slicer, import the 3D kidney model containing the pivot points. Use the Fiducial Registration Wizard. Select Place a control point, and digitally assign the registration landmarks.To perform landmark registration in 3D Slicer, use the electromagnetic pointer to pinpoint the physical landmark points. Press Place a control point at each physical location to register them in the software. Then calculate the rigid linear registration transform by pressing Update.Now, apply the calculated registration transform to the 3D model to link it with the electromagnetic reference sensor. Move the physical model and confirm that the digital version in 3D Slicer follows its motion. Launch the holographic display device, and open the holographic application configured earlier.Then navigate to the correct patient-specific 3D model that is currently visualized in 3D Slicer. Now, attach the infrared marker to the specified location using glue, ensuring the fitted 6.4 millimeter infrared reflective spheres are in place as guided by the preoperative planning. Use the electromagnetic pointer to digitally identify the target points as seen through the holographic visualization.Save the resulting set of EM sensor coordinates. Calculate the error by comparing the saved target coordinates to the actual placed landmarks to validate the accuracy of the holographic visualization. Across all participants, the Point Localization Error, or PLE, showed a median value of 8.74 millimeters, with individual measurements ranging from 2.78 to 13.20 millimeters.Surgeon 2 consistently achieved the lowest PLE measurements, including the two most accurate localizations at 2.78 and 3.48 millimeters. The largest localization error was observed during the third measurement by Surgeon 3 with a PLE of 13.20 millimeters. This protocol will assist others in deploying holographic projects and accurately validating their augmented reality system in a preclinical setting.Our surgical research group will soon start with the automated holographic tracking for multiple pediatric surgical cases. The movable organs are tracked based on machine learning algorithms and RGB camera feeds.

technical-approach-for-infrared-tracking-for-soft-tissue-navigation

Research

JoVE Journal

Medicine

Source:Carly Fassler1, Liyu Huang1, Kavita Prasad1, Jiayi Xu2, Guansen Tong2, Jie Ying Wu2, Michael C. Topf1,31Department of Otolaryngology – Head and Neck Surgery, Vanderbilt University Medical Center.2Computer Science Department, Vanderbilt University.3Vanderbilt University School of Engineering, Vanderbilt University.This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

Source: Kato, D. et. al., Evaluation and Manipulation of Neural Activity Using Two-Photon Holographic Microscopy. J. Vis. Exp. (2022)This video demonstrates the two-photon imaging technique combined with optogenetics using holographic microscopy. Using a spatial light modulator, this technique enables precise neuronal stimulation expressing the red-light-sensitive protein and calcium imaging with minimal photodamage. This approach provides insights into the functional connectivity and...

Two-Photon Holographic Microscopy to Study Neural Activity and Connectivity in a Mouse Model

Near-infrared fluorescence (NIRF) imaging may improve therapeutic outcome of breast cancer surgery by enabling intraoperative tumor localization and evaluation of surgical margin status. Using tissue-simulating breast phantoms containing fluorescent tumor-simulating inclusions, potential clinical applications of NIRF imaging in breast cancer patients can be assessed for standardization and training purposes.

Tissue-simulating Phantoms for Assessing Potential Near-infrared Fluorescence Imaging Applications in Breast Cancer Surgery

We provide herein a detailed description of the experimental protocol for imaging with a newly developed hand-held optoacoustic (photoacoustic) system for three-dimensional functional and molecular imaging in real time. The demonstrated powerful performance and versatility may define new application areas of the optoacoustic technology in preclinical research and clinical practice.

Universal Hand-held Three-dimensional Optoacoustic Imaging Probe for Deep Tissue Human Angiography and Functional Preclinical Studies in Real Time

Accurate and efficient visualization of invasive medical devices is extremely important in many ultrasound-guided minimally invasive procedures. Here, a method for localizing the spatial position of a needle tip relative to the ultrasound imaging probe is presented.

Three-Dimensional Ultrasonic Needle Tip Tracking with a Fiber-Optic Ultrasound Receiver

We developed a two-photon holographic microscope that can visualize, assess, and manipulate neural activity using high spatiotemporal resolution, with the aim of elucidating the pathogenesis of neuropsychiatric disorders that are associated with abnormal neural activity.

Evaluation and Manipulation of Neural Activity Using Two-Photon Holographic Microscopy

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

Transcranial ultrasound stimulation (TUS) is an emerging non-invasive neuromodulation technique that requires careful planning of acoustic and thermal simulations. The methodology describes an image processing and ultrasound simulation pipeline for efficient, user-friendly, streamlined planning for human TUS experimentation.

Pipeline for Planning and Execution of Transcranial Ultrasound Neuromodulation Experiments in Humans

This protocol describes the method of neuronavigated electrode placement for focal, transcranial direct current stimulation (tDCS) administered during functional magnetic resonance imaging (fMRI).

Neuronavigated Focalized Transcranial Direct Current Stimulation Administered During Functional Magnetic Resonance Imaging

Here, we describe a standardized protocol for motor mapping using nTMS combined with diffusion tensor imaging (DTI)-based reconstruction of the corticospinal tract (CST). The protocol is reproducible, clinically feasible, and readily integrable into routine clinical workflows, providing a robust and valuable framework for motor pathway assessment, neuroplasticity research, and rehabilitation planning.

A Standardized Protocol for Functional Motor Mapping Using Navigated Transcranial Magnetic Stimulation

Discover infrared marker tracking for holographic navigation using electromagnetic tracking and 3D Slicer. This protocol validates augmented reality alignment with moving organs in preclinical settings.

Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation (Video) | JoVE Journal

Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation

Explore More Videos