Name: Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone (Video) | JoVE Journal
Uploaded: 2020-01-02T13:00:00.000+00:00
Duration: 9 min 26 s
Description: Learn to build an augmented reality smartphone app for visualizing patient-specific 3D anatomical models using 3D-printed reference markers. This accessible method requires no prior medical imaging or software development experience.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This report was supported by projects PI18/01625 and PI15/02121 (Ministerio de Ciencia, Innovaci&#243;n y Universidades, Instituto de Salud Carlos III and European Regional Development Fund "Una manera de hacer Europa") and IND2018/TIC-9753 (Comunidad de Madrid).

<ol><li>Coles, T. R., John, N. W., Gould, D., Caldwell, D. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Integrating+Haptics+with+Augmented+Reality+in+a+Femoral+Palpation+and+Needle+Insertion+Training+Simulation%5BTitle%5D)+AND+%22IEEE+Transactions+on+Haptics%22%5BJournal%5D)">Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation.</a> IEEE Transactions on Haptics. 4 (3), 199-209 (2011).</li>
<li>Pelargos, P. E., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Utilizing+virtual+and+augmented+reality+for+educational+and+clinical+enhancements+in+neurosurgery%5BTitle%5D)+AND+%22Journal+of+Clinical+Neuroscience%22%5BJournal%5D)">Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery.</a> Journal of Clinical Neuroscience. 35, 1-4 (2017).</li>
<li>Abhari, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Training+for+Planning+Tumour+Resection%3A+Augmented+Reality+and+Human+Factors%5BTitle%5D)+AND+%22IEEE+Transactions+on+Biomedical+Engineering%22%5BJournal%5D)">Training for Planning Tumour Resection: Augmented Reality and Human Factors.</a> IEEE Transactions on Biomedical Engineering. 62 (6), 1466-1477 (2015).</li>
<li>Uppot, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Implementing+Virtual+and+Augmented+Reality+Tools+for+Radiology+Education+and+Training%2C+Communication%2C+and+Clinical+Care%5BTitle%5D)+AND+%22Radiology%22%5BJournal%5D)">Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care.</a> Radiology. 291, 182210(2019).</li>
<li>Pratt, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Through+the+HoloLensTM+looking+glass%3A+augmented+reality+for+extremity+reconstruction+surgery+using+3D+vascular+models+with+perforating+vessels%5BTitle%5D)+AND+%22European+Radiology+Experimental%22%5BJournal%5D)">Through the HoloLensTM looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels.</a> European Radiology Experimental. 2 (1), 2(2018).</li>
<li>Rose, A. S., Kim, H., Fuchs, H., Frahm, J. -M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Development+of+augmented-reality+applications+in+otolaryngology-head+and+neck+surgery%5BTitle%5D)+AND+%22The+Laryngoscope%22%5BJournal%5D)">Development of augmented-reality applications in otolaryngology-head and neck surgery.</a> The Laryngoscope. , (2019).</li>
<li>Zhou, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Robot-Assisted+Surgery+for+Mandibular+Angle+Split+Osteotomy+Using+Augmented+Reality%3A+Preliminary+Results+on+Clinical+Animal+Experiment%5BTitle%5D)+AND+%22Aesthetic+Plastic+Surgery%22%5BJournal%5D)">Robot-Assisted Surgery for Mandibular Angle Split Osteotomy Using Augmented Reality: Preliminary Results on Clinical Animal Experiment.</a> Aesthetic Plastic Surgery. 41 (5), 1228-1236 (2017).</li>
<li>Heinrich, F., Joeres, F., Lawonn, K., Hansen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Comparison+of+Projective+Augmented+Reality+Concepts+to+Support+Medical+Needle+Insertion%5BTitle%5D)+AND+%22IEEE+Transactions+on+Visualization+and+Computer+Graphics%22%5BJournal%5D)">Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion.</a> IEEE Transactions on Visualization and Computer Graphics. 25 (6), 1(2019).</li>
<li>Deng, W., Li, F., Wang, M., Song, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Easy-to-Use+Augmented+Reality+Neuronavigation+Using+a+Wireless+Tablet+PC%5BTitle%5D)+AND+%22Stereotactic+and+Functional+Neurosurgery%22%5BJournal%5D)">Easy-to-Use Augmented Reality Neuronavigation Using a Wireless Tablet PC.</a> Stereotactic and Functional Neurosurgery. 92 (1), 17-24 (2014).</li>
<li>Fan, Z., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+interactive+surgical+visualization+system+using+mobile+spatial+information+acquisition+and+autostereoscopic+display%5BTitle%5D)+AND+%22Journal+of+Biomedical+Informatics%22%5BJournal%5D)">3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display.</a> Journal of Biomedical Informatics. 71, 154-164 (2017).</li>
<li>Arnal-Burró, J., Pérez-Mañanes, R., Gallo-del-Valle, E., Igualada-Blazquez, C., Cuervas-Mons, M., Vaquero-Martín, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Three+dimensional-printed+patient-specific+cutting+guides+for+femoral+varization+osteotomy%3A+Do+it+yourself%5BTitle%5D)+AND+%22The+Knee%22%5BJournal%5D)">Three dimensional-printed patient-specific cutting guides for femoral varization osteotomy: Do it yourself.</a> The Knee. 24 (6), 1359-1368 (2017).</li>
<li>Vaquero, J., Arnal, J., Perez-Mañanes, R., Calvo-Haro, J., Chana, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+patient-specific+surgical+printing+cutting+blocks+guides+and+spacers+for+open-+wedge+high+tibial+osteotomy+%28HTO%29+-+do+it+yourself%5BTitle%5D)+AND+%22Revue+de+Chirurgie+Orthop%C3%A9dique+et+Traumatologique%22%5BJournal%5D)">3D patient-specific surgical printing cutting blocks guides and spacers for open- wedge high tibial osteotomy (HTO) - do it yourself.</a> Revue de Chirurgie Orthopédique et Traumatologique. 102, Supplement 131(2016).</li>
<li>De La Peña, A., De La Peña-Brambila, J., Pérez-De La Torre, J., Ochoa, M., Gallardo, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Low-cost+customized+cranioplasty+using+a+3D+digital+printing+model%3A+a+case+report%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Low-cost customized cranioplasty using a 3D digital printing model: a case report.</a> 3D Printing in Medicine. 4 (1), 1-9 (2018).</li>
<li>Kamio, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Utilizing+a+low-cost+desktop+3D+printer+to+develop+a+%22one-stop+3D+printing+lab%22+for+oral+and+maxillofacial+surgery+and+dentistry+fields%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Utilizing a low-cost desktop 3D printer to develop a "one-stop 3D printing lab" for oral and maxillofacial surgery and dentistry fields.</a> 3D Printing in Medicine. 4 (1), 1-7 (2018).</li>
<li>Punyaratabandhu, T., Liacouras, P., Pairojboriboon, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Using+3D+models+in+orthopedic+oncology%3A+presenting+personalized+advantages+in+surgical+planning+and+intraoperative+outcomes%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Using 3D models in orthopedic oncology: presenting personalized advantages in surgical planning and intraoperative outcomes.</a> 3D Printing in Medicine. 4 (1), 1-13 (2018).</li>
<li>Wake, N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Patient-specific+3D+printed+and+augmented+reality+kidney+and+prostate+cancer+models%3A+impact+on+patient+education%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Patient-specific 3D printed and augmented reality kidney and prostate cancer models: impact on patient education.</a> 3D Printing in Medicine. 5 (1), 1-8 (2019).</li>
<li>Barber, S. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+Reality%2C+Surgical+Navigation%2C+and+3D+Printing+for+Transcanal+Endoscopic+Approach+to+the+Petrous+Apex%5BTitle%5D)+AND+%22OTO+Open%3A+The+Official+Open+Access+Journal+of+the+American+Academy+of+Otolaryngology-Head+and+Neck+Surgery+Foundation%22%5BJournal%5D)">Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex.</a> OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation. 2 (4), (2018).</li>
<li>Witowski, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+and+three-dimensional+printing+in+percutaneous+interventions+on+pulmonary+arteries%5BTitle%5D)+AND+%22Quantitative+Imaging+in+Medicine+and+Surgery%22%5BJournal%5D)">Augmented reality and three-dimensional printing in percutaneous interventions on pulmonary arteries.</a> Quantitative Imaging in Medicine and Surgery. 9 (1), (2019).</li>
<li>Moreta-Martínez, R., García-Mato, D., García-Sevilla, M., Pérez-Mañanes, R., Calvo-Haro, J., Pascau, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+in+computer-assisted+interventions+based+on+patient-specific+3D+printed+reference%5BTitle%5D)+AND+%22Healthcare+Technology+Letters%22%5BJournal%5D)">Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference.</a> Healthcare Technology Letters. , (2018).</li>
</ol>

The authors have nothing to disclose.

AR holds great potential in education, training, and surgical guidance in the medical field. Its combination with 3D printing opens may open new possibilities in clinical applications. This protocol describes a methodology that enables inexperienced users to create a smartphone app combining AR and 3DP for the visualization of anatomical 3D models of patients with 3D-printed reference markers.
In general, one of the most interesting clinical applications of AR and 3DP is to improve patient-to-physician communication by giving the patient a different perspective of the case, improving explanations of specific medical conditions or treatments. Another possible application includes surgical guidance for target localization, in which 3D-printed patient-specific tools (with a reference AR marker attached) can be placed on rigid structures (i.e., bone) and used as a reference for navigation. This application is especially useful for orthopedic and maxillofacial surgical procedures, in which bone tissue surface is easily accessed during surgery.
The protocol starts with section 1, describing the workstation set-up and software tools necessary. Section 2 describes how to use 3D Slicer software to easily segment target anatomies of the patient from any medical imaging modality to obtain 3D models. This step is crucial, as the virtual 3D models created are those displayed in the final AR application.
In section 3, 3D Slicer is used to register the 3D models created in the previous section with an AR marker. During this registration procedure, patient 3D models are efficiently and simply positioned with respect to the AR marker. The position defined in this section will determine the hologram relative position in the final app. It is believed that this solution reduces complexity and multiplies the possible applications. Section 3 describes two different options to define the spatial relationships between the models and AR markers: "visualization" and "registration" mode. The first option, "visualization" mode, allows the 3D models to be positioned anywhere with respect to the marker and displayed as the whole biomodel. This mode provides a realistic, 3D perspective of the patient&#39;s anatomy and allows moving and rotating of the biomodels by moving the tracked AR marker. The second option, "registration" mode, allows attachment and combining of a marker adaptor to any part of the biomodel, offering an automatic registration process. With this option, a small section of the 3D model, including the marker adaptor, can be 3D-printed, and the app can display the rest of the model as a hologram.
Section 4 provides guidelines for the 3D printing process. First, the user can choose between two different markers: the "dual color marker" and "sticker marker". The whole "dual color marker" can be 3D-printed but requires a dual extruder 3D printer. In case this printer is not available, the "sticker marker" is proposed. This is a simpler marker that can be obtained by 3D-printing the cubic structure, then pasting the images of the cube with sticker paper or sticker glue. Furthermore, both markers were designed with extensible sections to perfectly fit in a specific adaptor. Thus, the marker can be reused in several cases.
Section 5 describes the process to create a Unity project for AR using the Vuforia software development kit. This step may be the hardest portion for users with no programming experience, but with these guidelines, it should be easier to obtain the final application that is presented in section 6. The app displays the patient&#39;s virtual models over the smartphone screen when the camera recognizes the 3D-printed marker. In order for the app to detect the 3D marker, a minimum distance of approximately 40 cm or less from the phone to the marker as well as good lighting conditions are required.
The final application of this protocol allows the user to choose the specific biomodels to visualize and in which positions. Addtionally, the app can perform automatic patient-hologram registration using a 3D-printed marker and adaptor attached to the biomodel. This solves the challenge of registering virtual models with the environment in a direct and convenient manner. Moreover, this methodology does not require broad knowledge of medical imaging or software development, does not depend on complex hardware and expensive software, and can be implemented over a short time period. It is expected that this method will help accelerate the adoption of AR and 3DP technologies by medical professionals.

AR and 3DP are technologies that provide increasing numbers of applications in the medical field. In the case of AR, its interaction with virtual 3D models and the real environment benefits physicians in regards to education and training1,2,3, communication and interactions with other physicians4, and guidance during clinical interventions5,6,7,8,9,10. Likewise, 3DP has become a powerful solution for physicians when developing patient-specific customizable tools11,12,13 or creating 3D models of a patient&rsquo;s anatomy, which can help improve preoperative planning and clinical interventions14,15.
Both AR and 3DP technologies help to improve orientation, guidance, and spatial skills in medical procedures; thus, their combination is the next logical step. Previous work has demonstrated that their joint use can increase value in patient education16, facilitating explanations of medical conditions and proposed treatment, optimizing surgical workflow17,18 and improving patient-to-model registration19. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this work is to describe a step-by-step methodology that enables the use of AR and 3DP by inexperienced users without the need for broad technical knowledge.
This protocol describes how to develop an AR smartphone app that allows the superimposing of any patient-based 3D model onto a real-world environment using a 3D-printed marker tracked by the smartphone camera. In addition, an alternative approach is described to provide automatic registration between a 3D-printed biomodel (i.e., a 3D model created from a patient&rsquo;s anatomy) and the projected holograms. The protocol described is entirely based on free and multi-platform software.
In previous work, AR patient-to-image registration has been calculated manually5 with surface recognition algorithms10 or has been unavailable2. These methods have been considered somewhat limited when an accurate registration is required19. To overcome these limitations, this work provides tools to perform accurate and simple patient-to-image registration in AR procedures by combining AR technology and 3DP.
The protocol is generic and can be applied to any medical imaging modality or patient. As an example, a real clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. The first step describes how to easily segment the affected anatomy from computed tomography (CT) medical images to generate 3D virtual models. Afterward, positioning of the 3D models is performed, then the required tools and models are 3D-printed. Finally, the desired AR app is deployed. This app allows for the visualization of patient 3D models overlaid on a smartphone camera in real-time.

<table><tbody><tr><td>3D Printing material: Acrylonitrile Butadiene Styrene (ABS)</td><td></td><td></td><td>Thermoplastic polymer material usually used in domestic 3D printers.</td></tr><tr><td>3D Printing material: Polylactic Acid (PLA)</td><td></td><td></td><td>Bioplastic material usually used in domestic 3D printers.</td></tr><tr><td>3D Slicer</td><td></td><td></td><td>Open-source software platform for medical image informatics, image processing, and three-dimensional visualization</td></tr><tr><td>Android</td><td>Alphabet, Inc.</td><td></td><td>Android is a mobile operating system developed by Google. It is based on a modified version of the Linux kernel and other open source software, and is designed primarily for touchscreen mobile devices such as smartphones and tablets.</td></tr><tr><td>Autodesk Meshmixer</td><td>Autodesk, Inc.</td><td></td><td>Meshmixer is state-of-the-art software for working with triangle meshes. Free software.</td></tr><tr><td>iPhone OS</td><td>Apple, Inc.</td><td></td><td>iPhone OS is a mobile operating system created and developed by Apple Inc. exclusively for its hardware.</td></tr><tr><td>Ultimaker 3 Extended</td><td>Ultimaker BV</td><td></td><td>Fused deposition modeling 3D printer.</td></tr><tr><td>Unity</td><td>Unity Technologies</td><td></td><td>Unity is a real-time development platform to create 3D, 2D VR &amp;amp; AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering &amp;amp; more. Free software.</td></tr><tr><td>Xcode</td><td>Apple, Inc.</td><td></td><td>Xcode is a complete developer toolset for creating apps for Mac, iPhone, iPad, Apple Watch, and Apple TV. Free software.</td></tr></tbody></table>

combining augmented reality and 3d printing to display patient models on a smartphone

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

The protocol was applied to data from a patient suffering from distal leg sarcoma in order to visualize the affected anatomical region from a 3D perspective. Using the method described in section 2, the portion of the affected bone (here, the tibia and fibula) and tumor were segmented from the patient's CT scan. Then, using the segmentation tools from 3D Slicer, two biomodels were created: the bone (section of the tibia and fibula) (Figure 1A) and tumor (Figure 1B).
Next, the two 3D models were positioned virtually with respect to the marker for optimal visualization. Both modes described in section 3 were followed for this example. For visualization mode, the models were centered in the upper face of the marker (Figure 2). For registration mode, the marker adaptor was positioned in the bone (specifically, the tibia [Figure 3]). Then, a small section of the tibia was selected to be 3D-printed with a 3D marker adaptor (Figure 4). An Ultimaker 3 extended 3D printer with PLA material was used to create the 3D-printed markers (Figure 5A, B), marker holder base (Figure 5C) for the "visualization" mode, and section of the tibia for "registration" mode (Figure 5D). Figure 5E shows how the marker was attached to the "visualization" mode 3D-printed base. Figure 5F shows the attachment with the "registration" mode 3D-printed biomodel. Finally, Unity was used to create the app and deploy it on the smartphone.
Figure 6 shows how the app worked for "visualization" mode. The hologram was accurately located in the upper part of the cube as previously defined. Figure 7 shows the application for "registration" mode, in which the app positioned the complete bone model on top of the 3D-printed section. The final visualization of the holograms was clear and realistic, maintained the real sizes of the biomodels, and positioned accurately. When using the smartphone application, the AR marker needs to be visible by the camera for the app to correctly display the holograms. In addition, the light conditions in the scene must be of good quality and constant for proper marker detection. Bad light conditions or reflections on the marker surface hinder the tracking of the AR marker and cause malfunctioning of the app.
The time required to create the app depends on several factors. The duration of section 1 is limited by the download speed. Regarding anatomy segmentation (section 2), factors affecting segmentation time include complexity of the region and medical imaging modality (i.e., CT is easily segmented, while MRI is more difficult). For the representative example of the tibia, approximately 10 min was required to generate both 3D models from the CT scan. Biomodel positioning (section 3) is simple and straightforward. Here, it took approximately 5 min to define the biomodel position with respect to the AR marker. For the 3D printing step, the duration is highly dependent on the selected mode. The "dual color marker" was manufactured at high quality in a period of 5 h and 20 min. The "sticker marker" was manufactured in a period of 1 h and 30 min, plus the time required to paste the stickers. The final step for app development can be time-consuming for those with no previous experience in Unity, but it can be easily completed following the protocol steps. Once the AR markers have been 3D-printed, the development of an entirely new AR app can be performed in less than 1 h. This duration can be further reduced with additional experience.
<img alt="Bone structure analysis, 3D model highlighting fracture location and healing, educational diagram." class="xfigimg" src="/files/ftp_upload/60618/60618fig01.jpg" data-altupdatedat="1788282455000" data-alt="Figure 1"> 
Figure 1: Representation of 3D models created from a CT image of a patient suffering from distal leg sarcoma. (A) Bone tissue represented in white (tibia and fibula). (B) Tumor represented in red. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="3D diagram of forearm bone and green shape, labeled, showing spatial orientation analysis." class="xfigimg" src="/files/ftp_upload/60618/60618fig02.jpg" data-altupdatedat="1788282455000" data-alt="Figure 2"> 
Figure 2: Results showing how "visualization" mode in 3D Slicer positions the virtual 3D models of the bone and tumor with respect to 3D-printed marker reference. The patient 3D models (A) are positioned above the upper face of the marker cube (B). <a href="https://www.jove.com/files/ftp_upload/60618/60618fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Static equilibrium diagram; femur tibia positioning; force vector analysis; mechanical model." class="xfigimg" src="/files/ftp_upload/60618/60618fig03.jpg" data-altupdatedat="1788282455000" data-alt="Figure 3"> 
Figure 3: Results showing how "registration" mode in 3D Slicer positions the virtual 3D models of the bone and tumor (A) with respect to 3D-printed marker reference (B). The marker adaptor is attached to the bone tissue model. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="3D bone model with fixation device, diagram of orthopedic engineering, fracture stabilization." class="xfigimg" src="/files/ftp_upload/60618/60618fig04.jpg" data-altupdatedat="1788282455000" data-alt="Figure 4"> 
Figure 4: Small section of the bone tissue and 3D marker adaptor. The two components are combined then 3D-printed. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="3D printed orientation markers on cubes, for navigation precision; six configurations." class="xfigimg" src="/files/ftp_upload/60618/60618fig05.jpg" data-altupdatedat="1788282455000" data-alt="Figure 5"> 
Figure 5: 3D printed tools required for the final application. (A) "Two color cube marker" 3D-printed with two colors of materials. (B) "Sticker cube marker" 3D-printed, with stickers pasted. (C) Marker base cube adaptor. (D) Section of the patient's bone tissue 3D model and marker cube adaptor. (E) "Sticker cube marker" placed in the marker base cube adaptor. (F) "Two color cube marker" placed in the marker adaptor attached to the patient's anatomy. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig05large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Augmented reality visualization of bone structure on a smartphone, experiment setup for anatomy study." class="xfigimg" src="/files/ftp_upload/60618/60618fig06.jpg" data-altupdatedat="1788282455000" data-alt="Figure 6"> 
Figure 6: App display when using "visualization" mode. The patient's affected anatomy 3D models are positioned above the upper face of the 3D-printed cube. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig06large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Augmented reality application displaying 3D model overlay on physical object, educational demo." class="xfigimg" src="/files/ftp_upload/60618/60618fig07.jpg" data-altupdatedat="1788282455000" data-alt="Figure 7"> 
Figure 7: AR visualization when using "registration" mode. The 3D-printed marker enables registration of the 3D-printed biomodel with the virtual 3D models. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig07large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

combining-augmented-reality-3d-printing-to-display-patient-models-on

Research

JoVE Journal

Medicine

Source:Carly Fassler1, Liyu Huang1, Kavita Prasad1, Jiayi Xu2, Guansen Tong2, Jie Ying Wu2, Michael C. Topf1,31Department of Otolaryngology – Head and Neck Surgery, Vanderbilt University Medical Center.2Computer Science Department, Vanderbilt University.3Vanderbilt University School of Engineering, Vanderbilt University.This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

Using modern plastic extrusion and printing technologies, it is now possible to quickly and inexpensively produce physical models of X-ray CT data taken in a laboratory.  The three -dimensional printing of tomographic data is a powerful visualization, research, and educational tool that may now be accessed by the preclinical imaging community.

3D Printing of Preclinical X-ray Computed Tomographic Data Sets

The protocol describes the fabrication of fully colored three-dimensional prints of patient-specific, anatomical skull models to be used for surgical simulation. The crucial steps of combining different imaging modalities, image segmentation, three-dimensional model extraction, and production of the prints are explained.

Multicolor 3D Printing of Complex Intracranial Tumors in Neurosurgery

This method demonstrates a voxel-based 3D printing workflow, which prints directly from medical images with exact spatial fidelity and spatial/contrast resolution. This enables the precise, graduated control of material distributions through morphologically complex, graduated materials correlated to radiodensity without loss or alteration of data.

Voxel Printing Anatomy: Design and Fabrication of Realistic, Presurgical Planning Models through Bitmap Printing

A complex revision hip arthroplasty was performed using a custom-made implant and mixed reality technology. According to the authors' knowledge, this is the first report of such procedure described in the literature.<br...

The Use of Mixed Reality in Custom-Made Revision Hip Arthroplasty: A First Case Report

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

A novel method for 3D scanning and virtual mapping of cancer resections is proposed with the goal of improving communication among the multidisciplinary cancer care team.

Enhanced Communication of Tumor Margins Using 3D Scanning and Mapping

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Here, we describe a protocol for constructing a heart model from scratch based on computed tomography and present it to medical students using three-dimensional (3D) printing and mixed reality technology to learn anatomy.

Mixed Reality Technology and Three-Dimensional Printing in Teaching: Heart Anatomy as an Example

Learn to build an augmented reality smartphone app for visualizing patient-specific 3D anatomical models using 3D-printed reference markers. This accessible method requires no prior medical imaging or software development experience.

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Explore More Videos