the-use-mixed-reality-custom-made-revision-hip-arthroplasty-first

The Use of Mixed Reality in Custom-Made Revision Hip Arthroplasty: A First Case Report

The technology of 3D printing and visualization of anatomical structures is rapidly growing in various fields of medicine. A custom-made implant and mixed ...

mixed-reality-technology-three-dimensional-printing-teaching-heart

Mixed Reality Technology and Three-Dimensional Printing in Teaching: Heart Anatomy as an Example

Mixed reality technology and three-dimensional (3D) printing are becoming more and morecommon in the field of medicine. During the COVID-19 pandemic and ...

AGH University of Kraków

MedApp S.A.

AGH University of Science and Technology

AGH University of Krakow

MedApp S. A.

The aim of this study was to evaluate the usefulness of different methods of speech signal analysis in the detection of voice pathologies. Firstly, an initial vector was created consisting of 28 parameters extracted from time, frequency and cepstral domain describing the human voice signal based on the analysis of sustained vowels /a/, /i/ and /u/ all at high, low and normal pitch. Afterwards we used a linear feature extraction technique (principal component analysis), which enabled a reduction in the number of parameters and choose the most effective acoustic features describing the speech signal. We have also performed non-linear data transformation which was calculated using kernel principal components. The results of the presented methods for normal and pathological cases will be revealed and discussed in this paper. The initial and extracted feature vectors were classified using the k-means clustering and the random forest classifier. We found that reasonably good classification accuracies could be achieved by selecting appropriate features. We obtained accuracies of up to 100% for classification of healthy versus pathology voice using random forest classification for female and male recordings. These results may assist in the feature development of automated detection systems for diagnosis of patients with symptoms of pathological voice. 

Voice data mining for laryngeal pathology assessment.

Knowledge about tumor bed localization and its shape analysis is a crucial factor for preventing irradiation of healthy tissues during supportive radiotherapy and as a result, cancer recurrence. The localization process is especially hard for tumors placed nearby soft tissues, which undergo complex, nonrigid deformations. Among them, breast cancer can be considered as the most representative example. A natural approach to improving tumor bed localization is the use of image registration algorithms. However, this involves two unusual aspects which are not common in typical medical image registration: the real deformation field is discontinuous, and there is no direct correspondence between the cancer and its bed in the source and the target 3D images respectively. The tumor no longer exists during radiotherapy planning. Therefore, a traditional evaluation approach based on known, smooth deformations and target registration error are not directly applicable. In this work, we propose alternative artificial deformations which model the tumor bed creation process. We perform a comprehensive evaluation of the most commonly used deformable registration algorithms: B-Splines free form deformations (B-Splines FFD), different variants of the Demons and TV-L optical flow. The evaluation procedure includes quantitative assessment of the dedicated artificial deformations, target registration error calculation, 3D contour propagation and medical experts visual judgment. The results demonstrate that the currently, practically applied image registration (rigid registration and B-Splines FFD) are not able to correctly reconstruct discontinuous deformation fields. We show that the symmetric Demons provide the most accurate soft tissues alignment in terms of the ability to reconstruct the deformation field, target registration error and relative tumor volume change, while B-Splines FFD and TV-L optical flow are not an appropriate choice for the breast tumor bed localization problem, even though the visual alignment seems to be better than for the Demons algorithm. However, no algorithm could recover the deformation field with sufficient accuracy in terms of vector length and rotation angle differences.

Improving oncoplastic breast tumor bed localization for radiotherapy planning using image registration algorithms.

This article proposes a novel framework for the locally-oriented evaluation of segmentation algorithms (LEFMIS). The presented approach is robust and takes into account local inter/intra-observer variability and the anisotropy of medical images. What is more, the framework makes it possible to distinguish types of error locally. These features are crucial in the context of cancer image data. The proposed framework is based on use of the signed anisotropic Euclidean distance transform and the distance projection. It can be used easily in many different applications with or without additional expert outlines (both inter- and intra-observer variability). The performance of the proposed framework is depicted using both artificial and kidney cancer CT data with experts' manual outlines. In the article, in the case of artificial data, it is presented that the manual outlines dispersion is symmetric in relation to the truth border. The effectiveness of the selected segmentation algorithm was analysed in the context of kidney cancer using computed tomography data. For the calculated local inter-observer variability, 80.11% of the surface points generated by the kidney segmentation algorithm are within one expert outline standard deviation and 97.96% are within five. An error distribution shift in the direction of type I error equivalent was also observed. Finally, the significance of the local estimation of error type differences is presented. The article shows the greater usefulness and flexibility of the proposed framework in comparison to the state-of-the-art methods. The exemplary usage of the LEFMIS with or without inter-/intra-observer variability is also presented.

LEFMIS: locally-oriented evaluation framework for medical image segmentation algorithms.

Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98% of all landmarks and their mean landmark registration accuracy (TRE) was 0.44% of the image diagonal. The challenge remains open to submissions and all images are available for download.

ANHIR: Automatic Non-Rigid Histological Image Registration Challenge.

Devices and systems secured by biometric factors became a part of our lives because they are convenient, easy to use, reliable, and secure. They use information about unique features of our bodies in order to authenticate a user. It is possible to enhance the security of these devices by adding supplementary modality while keeping the user experience at the same level. Palm vein systems are based on infrared wavelengths used for capturing images of users' veins. It is both convenient for the user, and it is one of the most secure biometric solutions. The proposed system uses IR and UV wavelengths; the images are then processed by a deep convolutional neural network for extraction of biometric features and authentication of users. We tested the system in a verification scenario that consisted of checking if the images collected from the user contained the same biometric features as those in the database. The True Positive Rate (TPR) achieved by the system when the information from the two modalities were combined was 99.5% by the threshold of acceptance set to the Equal Error Rate (EER).

Contact-Free Multispectral Identity Verification System Using Palm Veins and Deep Neural Network.

The use of multiple dyes during histological sample preparation can reveal distinct tissue properties. However, since the slide preparation differs for each dye, the tissue slides are being deformed and a nonrigid registration is required before further processing. The registration of histology images is complicated because of: (i) a high resolution of histology images, (ii) complex, large, nonrigid deformations, (iii) difference in the appearance and partially missing data due to the use of multiple dyes. In this work, we propose a multistep, automatic, nonrigid image registration method dedicated to histology samples acquired with multiple stains. The proposed method consists of a feature-based affine registration, an exhaustive rotation alignment, an iterative, intensity-based affine registration, and a nonrigid alignment based on modality independent neighbourhood descriptor coupled with the Demons algorithm. A dedicated failure detection mechanism is proposed to make the method fully automatic, without the necessity of any manual interaction. The described method was proposed by the AGH team during the Automatic Non-rigid Histological Image Registration (ANHIR) challenge. The ANHIR dataset consists of 481 image pairs annotated by histology experts. Moreover, the ANHIR challenge submissions were evaluated using an independent, server-side evaluation tool. The main evaluation criteria was the target registration error normalized by the image diagonal. The median of median target registration error is below 0.19%. The proposed method is currently the second-best in terms of the average ranking of median target registration error, without statistically significant differences compared to the top-ranked method. We provide an open access to the method software and used parameters, making the results fully reproducible.

Multistep, automatic and nonrigid image registration method for histology samples acquired using multiple stains.

The purpose of this study was to investigate the potential of a combination of 3D mixed-reality visualization of medical images using CarnaLife Holo (MedApp, Poland) system as a supporting tool for innovative, minimally invasive surgery/irreversible electroporation-IRA, Nano-Knife), microwave ablation (MWA)/for advanced gastrointestinal tumors. Eight liver and pancreatic tumor treatments were performed. In all of the patients undergoing laparoscopy or open surgery volume and margin were estimated by preoperative visualization. In all patients, neoplastic lesions were considered unresectable by standard methods.

3D mixed-reality visualization of medical imaging data as a supporting tool for innovative, minimally invasive surgery for gastrointestinal tumors and systemic treatment as a new path in personalized treatment of advanced cancer diseases.

Breast-conserving surgery requires supportive radiotherapy to prevent cancer recurrence. However, the task of localizing the tumor bed to be irradiated is not trivial. The automatic image registration could significantly aid the tumor bed localization and lower the radiation dose delivered to the surrounding healthy tissues. This study proposes a novel image registration method dedicated to breast tumor bed localization addressing the problem of missing data due to tumor resection that may be applied to real-time radiotherapy planning. We propose a deep learning-based nonrigid image registration method based on a modified U-Net architecture. The algorithm works simultaneously on several image resolutions to handle large deformations. Moreover, we propose a dedicated volume penalty that introduces the medical knowledge about tumor resection into the registration process. The proposed method may be useful for improving real-time radiation therapy planning after the tumor resection and, thus, lower the surrounding healthy tissues' irradiation. The data used in this study consist of 30 computed tomography scans acquired in patients with diagnosed breast cancer, before and after tumor surgery. The method is evaluated using the target registration error between manually annotated landmarks, the ratio of tumor volume, and the subjective visual assessment. We compare the proposed method to several other approaches and show that both the multilevel approach and the volume regularization improve the registration results. The mean target registration error is below 6.5 mm, and the relative volume ratio is close to zero. The registration time below 1 s enables the real-time processing. These results show improvements compared to the classical, iterative methods or other learning-based approaches that do not introduce the knowledge about tumor resection into the registration process. In future research, we plan to propose a method dedicated to automatic localization of missing regions that may be used to automatically segment tumors in the source image and scars in the target image.

Semi-Supervised Deep Learning-Based Image Registration Method with Volume Penalty for Real-Time Breast Tumor Bed Localization.

Mixed reality (MR) holograms can display high-definition images while preserving the user's situational awareness. New MR software can measure 3D objects with gestures and voice commands; however, these measurements have not been validated. We aimed to assess the feasibility and accuracy of using 3D holograms for measuring the length of coronary artery bypass grafts.

Feasibility and accuracy of real-time 3D-holographic graft length measurements.

Exact biopsy planning and careful execution of needle injection is crucial to ensure successful procedure completion as initially intended while minimizing the risk of complications. This study introduces a solution aimed at helping the operator navigate to precisely position the needle in a previously planned trajectory utilizing a mixed reality headset. A markerless needle tracking method was developed by integrating deep learning and deterministic computer vision techniques. The system is based on superimposing imaging data onto the patient's body in order to directly perceive the anatomy and determine a path from the selected injection site to the target location. Four types of tests were conducted to assess the system's performance: measuring the accuracy of needle pose estimation, determining the distance between injection sites and designated targets, evaluating the efficiency of material collection, and comparing procedure time and number of punctures required with and without the system. These tests, involving both phantoms and physician participation in the latter two, demonstrated the accuracy and usability of the proposed solution. The results showcased a significant improvement, with a reduction in number of punctures needed to reach the target location. The test was successfully completed on the first attempt in 70% of cases, as opposed to only 20% without the system. Additionally, there was a 53% reduction in procedure time, validating the effectiveness of the system.

Mixed Reality Biopsy Navigation System Utilizing Markerless Needle Tracking and Imaging Data Superimposition.

For many years, teaching of anatomy has been based on traditional forms of teaching, but innovative solutions are currently being implemented on a large scale around the world. The COVID-19 pandemic and distance learning have influenced the development of new technologies in teaching.

Anatomical theater or full digitization? Students' assessment and preferences in the field of anatomy teaching.

The estimation of vessels' centerlines is a critical step in assessing the geometry of the vessel, the topological representation of the vessel tree, and vascular network visualization. In this research, we present a novel method for obtaining geometric parameters from peripheral arteries in 3D medical binary volumes. Our approach focuses on centerline extraction, which yields smooth and robust results. The procedure starts with a segmented 3D binary volume, from which a distance map is generated using the Euclidean distance transform. Subsequently, a skeleton is extracted, and seed points and endpoints are identified. A search methodology is used to derive the best path on the skeletonized 3D binary array while tracking from the goal points to the seed point. We use the distance transform to calculate the distance between voxels and the nearest vessel surface, while also addressing bifurcations when vessels divide into multiple branches. The proposed method was evaluated on 22 real cases and 10 synthetically generated vessels. We compared our method to different state-of-the-art approaches and demonstrated its better performance. The proposed method achieved an average error of 1.382 mm with real patient data and 0.571 mm with synthetic data, both of which are lower than the errors obtained by other state-of-the-art methodologies. This extraction of the centerline facilitates the estimation of multiple geometric parameters of vessels, including radius, curvature, and length.

Vessel Geometry Estimation for Patients with Peripheral Artery Disease.

The DECODE-3DViz pipeline represents a major advancement in the web-based visualization of large-scale medical imaging data, particularly for peripheral artery computed tomography images. This research addresses the critical challenges of rendering high-resolution volumetric datasets via WebGL technology. By integrating progressive chunk streaming and level of detail (LOD) algorithms, DECODE-3DViz optimizes the rendering process for real-time interaction and high-fidelity visualization. The system efficiently manages WebGL texture size constraints and browser memory limitations, ensuring smooth performance even with extensive datasets. A comparative evaluation against state-of-the-art visualization tools demonstrates DECODE-3DViz's superior performance, achieving up to a 98% reduction in rendering time compared with that of competitors and maintaining a high frame rate of up to 144 FPS. Furthermore, the system exhibits exceptional GPU memory efficiency, utilizing as little as 2.6 MB on desktops, which is significantly less than the over 100 MB required by other tools. User feedback, collected through a comprehensive questionnaire, revealed high satisfaction with the tool's performance, particularly in areas such as structure definition and diagnostic capability, with an average score of 4.3 out of 5. These enhancements enable detailed and accurate visualizations of the peripheral vasculature, improving diagnostic accuracy and supporting better clinical outcomes. The DECODE-3DViz tool is open source and can be accessed at https://github.com/mohammed-abo-arab/3D_WebGL_VolumeRendering.git .

ArticleTotal : 2	Year
Publication title Cited by 7	2022
Publication title Cited by 1	2025

Article	Year
Voice data mining for laryngeal pathology assessment. Computers in biology and medicine\| PubMed ID: 26471193	2016
Improving oncoplastic breast tumor bed localization for radiotherapy planning using image registration algorithms. Physics in medicine and biology\| PubMed ID: 29293469	2018
LEFMIS: locally-oriented evaluation framework for medical image segmentation algorithms. Physics in medicine and biology\| PubMed ID: 29999495	2018
ANHIR: Automatic Non-Rigid Histological Image Registration Challenge. IEEE transactions on medical imaging\| PubMed ID: 32275587	2020
Contact-Free Multispectral Identity Verification System Using Palm Veins and Deep Neural Network. Sensors (Basel, Switzerland)\| PubMed ID: 33036259	2020
Multistep, automatic and nonrigid image registration method for histology samples acquired using multiple stains. Physics in medicine and biology\| PubMed ID: 33197906	2021
3D mixed-reality visualization of medical imaging data as a supporting tool for innovative, minimally invasive surgery for gastrointestinal tumors and systemic treatment as a new path in per... Journal of cancer research and clinical oncology\| PubMed ID: 34110490	2022
Semi-Supervised Deep Learning-Based Image Registration Method with Volume Penalty for Real-Time Breast Tumor Bed Localization. Sensors (Basel, Switzerland)\| PubMed ID: 34198497	2021
Feasibility and accuracy of real-time 3D-holographic graft length measurements. European heart journal. Digital health\| PubMed ID: 38264694	2024
Mixed Reality Biopsy Navigation System Utilizing Markerless Needle Tracking and Imaging Data Superimposition. Cancers\| PubMed ID: 38791972	2024
Anatomical theater or full digitization? Students' assessment and preferences in the field of anatomy teaching. Folia morphologica\| PubMed ID: 39387399	2024
Vessel Geometry Estimation for Patients with Peripheral Artery Disease. Sensors (Basel, Switzerland)\| PubMed ID: 39409481	2024
DECODE-3DViz: Efficient WebGL-Based High-Fidelity Visualization of Large-Scale Images using Level of Detail and Data Chunk Streaming. Journal of imaging informatics in medicine\| PubMed ID: 39953258	2025

Andrzej Skalski

Biography

JoVE Journal Publications

Other Publications