Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Summary

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Abstract

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Introduction

AR can superimpose the 3D model in a real-world environment. The technological development towards AR has made a paradigm shift in many fields, namely education¹, medical², manufacturing³, and entertainment⁴. AR technology, along with ultra-reliable low-latency communication, proves its inevitable role in the medical field. From the learning stage of human anatomy to surgical guidance, the stages of learning can be visualized with AR-powered software^5,6 and hardware. AR provides a crucial and reliable solution to the medical practitioner in a surgical environment^7,8.

Aortic valve stenosis is the heart valve disease, which is most common among mankind⁹. The root cause of the disease is poor food habits and irregular routines of day-to-day life. The symptom and result of the disease is the narrowing of the heart valve, followed by a reduction in the blood flow. This problem needs to be addressed before any damage takes place to the human heart. Thus, the heart is overburdened to process the blood flow. So, before any damage happens, surgery needs to be done, which, owing to technological developments in recent days, can also be done using the TAVR procedure. The procedure can be adopted based on the condition of the heart and other body parts of patients. In this TAVR^10,11, the catheter is inserted to replace the damaged valve in the heart. However, placing the catheter position¹² to replace the valve is tedious for the practitioner. This idea motivated us to design an automated surgery model based on AR^13,14, which helps the surgeon to precisely position the valve during the replacement process. Moreover, the surgery can be performed by a motion mapping algorithm, which maps the surgeon's movement captured from a remote location to the robotic arm.

In the existing work^15,16,17, the visualization of the TAVR¹⁸ procedure is monitored through fluoroscopy. Hence, it is difficult to analyze the heart valve and tedious to find the replacement location. This sets up a barrier to positioning the catheter in the human heart. In addition, the remote motion is mapped to the surgical field to make the process automated. However, to overcome the research gap, we propose an automated robotic-based surgery for valve replacement using AR-assisted technology.

The protocol is a generic model that can be applied to all surgical environments. In the beginning stage of the work, 2D images are captured all around the surgical environment with the fullest spatial resolution of the largest degree of freedom. This means that enough images are captured for 3D reconstruction¹⁹ purpose, followed by motion mapping through hand gesture tracking²⁰.

Protocol

1. Surgical environment

1. Design a surgery environment as shown in Figure 1. Make sure the environment has an object-carrying platter, a robotic arm, and two side-hanging arms, one to hold a camera placeholder and the other to have a consistent white background along with the weighing module for balance.
2. Develop two drivers, one for the snapshot of the live surgical environment, as mentioned in steps 2.1 to 2.10, and the other to control the revolving mechanism that supports 360° surveillance, as mentioned in steps 3.1 to 3.4.
3. Before implementing the above two modules, enable Bluetooth of the mobile device and the laptop, which serve as the surgeon's HoloLens emulator.
4. Pair the devices for uninterrupted image transmission.

2. Setting up the driver to control the two hanging arms

1. Make sure that the hanging arms are controlled by a stepper motor with the arrangement as shown in Figure 2 for a flawless revolution of 360°.
2. Connect the motor to the microcontroller board using the TB 6600 driver. To run the motor, install the microcontroller IDE from the browser.
3. Click the Download button to download the software. Then, in the microcontroller IDE, go to File > Open a New Sketch to write the code.
4. Make sure to connect the microcontroller board to interface with the new sketch through a dedicated connection port, say COM 4. Check the Com Port and verify that it shows the microcontroller board.
5. Check the hardware switch settings of the stepper motor driver TB 6600. Ensure that the settings are such that the current flow is 2 A, which can be attained by setting SW4 ON and SW5 and SW6 OFF.
6. Ensure the switch positions of SW1, SW2, and SW3 are set so that the micro-step is 1/8 steps to attain the revolution steps as per the requirement. Ensure the settings are SW1 OFF, SW2 ON, and SW3 OFF in TB6600.
7. Connect RTC 3231 with the microcontroller to have real global time synchronization. Make sure that the revolution step size is 12° and that the motor step increment is triggered only when the real-time unit, i.e., the seconds read from the RTC module, is odd in number.
8. Connect the 5 V pin of the microcontroller board to the RTC VCC and the microcontroller's GND to the RTC's GND.
9. Connect the SCL pin of RTC to the A0 pin and the SDA to the A1 pin of the microcontroller. This module can ensure a step size of 12°, making 30 steps in one revolution. Ensure that the step increment happens every odd second. Let this software module drive the stepper motor²¹.
10. Verify the setup is working correctly by running the code, which is available on the GitHub page: https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.
11. Download Android Studio to develop the automatic camera app. Ensure the system requirements are met, then download the software from the website.

3. Developing a driver for mobile-based scene surveillance and image transmission as a client module

1. Develop a camera application in the Android operating system that can take snapshots every 2 s, especially when the seconds are odd numbers.
2. Connect the mobile phone with the system. In Android Studio, click New > New Project and choose Empty Views Activity. Click on Next to develop an Android code, which is available at https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.
3. Ensure that the app captures the images automatically and sends them to a remote device consistently.
4. Transmit snapshots from the mobile app immediately after taking the snapshot to the paired device, i.e., to the remote surgeon's system, through Bluetooth.
   NOTE: Make sure the modules mentioned in sections 2 and 3 run in time synchronization, one for every even number of seconds and the other for every odd number of seconds.

4. Developing a client module to receive and handle surveillance images

1. Open the server module, which is a graphical user interface.
2. Enter the VVID port number in the text field VVID, whose default value is 94f39d29 7d6d 437d 973b fba39e49d4ee.
3. Click on Create Socket to create and bind the socket. Click on Connect to establish a connection with the mobile app.
4. Click on Capture to capture and save the scene surveillance images in the local folder
5. Enter the local folder name in the field folder name if it needs to be other than the default one mentioned.

5. Operating the robotic arm

1. Let the client module include a robotic arm as well. Design the arm to have a rotational movement in its base, shoulder, elbow, wrist, and fingers.
2. Make sure that MG 996R servos are used for governing the rotational movement at the base, shoulder and elbow. Ensure that the SG 90 servo motor is used to control the rotational movement at the elbow and fingers.
3. Compile the code given in https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR in the microcontroller IDE to drive the robotic arm based on the commands received from the remote surgeon.

6. 3D reconstruction for augmented reality

1. Read two images at a time in a sequence, one by one, from the local folder to obtain the possible overlapping (as the images are collected in close proximity, there will be an overlapping between the consecutive images) between them.
2. Design a tertiary filter as per the requirement of the Directional Intensified Feature Description using the Tertiary Filtering²² (DITF) algorithm to obtain the gradient and orientation.
3. Extract the features using the DITF method²², as shown in Figure 3.
4. Reconstruct 3D images from the collected features using SFM²³.

7. Hand gesture recognition at the surgeon's location

1. Facilitate the surgeon to inspect the 3D reconstructed image features by enabling him/her to visualize the environment from all perspectives by providing hand gesture-based rotation and zoom in/out of reconstructed features.
2. Normalize and map the distance between the tip of the surgeon's thumb and the index finger of the right hand into a corresponding angle of rotation. Let the normalization be in such a way that the bare minimum distance corresponds to 0° and the maximum to 180°.
3. Transmit the hand gesture control, through Bluetooth, to the remote surgery environment as well for the rotation of the object platter, which makes it revolve on its axis as the 3D reconstructed features revolve at the surgeon's end.
4. Find the distance between the tip and thumb of the surgeon's left hand to control the movement of the fingers of the robot arm.
5. Measure the angle of elevation from the spatial distance between the tip of the thumb and index fingers of the surgeon's left hand with respect to an imaginary x-y ground plane to determine the elevation angle. Map this angle into an elevation angle that the robot arm can make with the x-y plane.
6. Find the azimuth angle that the hand of the surgeon makes with that of the virtual y-z plane. Identify these angles through hand gesture-based recognition.
7. Map the distance, elevation, and azimuth angles to control the robot's finger movement and arm rotation, which both correspond to elevation and azimuth angles.
8. Let the surgeon inspect the reconstructed features by zooming and rotating. Let the surgeon transmit commands to the robot arm to perform surgery from a remote location.
9. Make sure that the surgery commands are transmitted as a control string of sequence starting with a string concurrence followed by the values to control the platter rotation and robotic arm controller. Let [θ_b, θ_s, θ_e, θ_w, θ_f] be the angle of the vector that consists of values, each corresponding to the control signal corresponding to the base, shoulder, elbow, wrist, and finger of the robot arm.
   NOTE: The GitHub link provides the code to enable hand gesture control in the surgical field. https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.

Results

The protocol was tested with the heart phantom model. Figure 2 shows the expected setup for the live surveillance of the surgical field with the help of spatially distributed cameras. The distributed cameras, as shown in Figure 2, help to increase the spatial resolution of the field for effective 3D reconstruction. However, realizing the physical placement of those cameras in various spatial locations involves complexity. So, we have optimized the setup design and come up with the solution of spatially self-oriented camera arrangement revolved by a NEMA motor driven by a TB6600 driver. This structure is feasible and is implemented in the model and the same is shown in Figure 1.

In Figure 1, the automatic camera snapshots followed by their transmission through Bluetooth protocol are governed by an Android code. The module is organized in such a way that the capturing takes place once in every odd second, and their transmission is done once in every even second, as mentioned in section 3. The microcontroller module that interfaces the NEMA motor takes care of rotating the structure once every even second so that enough time is guaranteed for unblurred photo capturing. So, in total, 30 photos are taken over 360° of rotation, and they are transmitted using Bluetooth.

Also, in Figure 1, the 3D reconstructed view of DITF descriptors of the surgical environment is shown. It is to be noted that these descriptor-based reconstructions can be inspected by the surgeons by zooming and rotation through hand motion mapping. Also, these motions are sent to the surgical environment to imitate the surgery so that the catheter can be inserted in the real field. The motion mapping is done in such a way that they are mapped into six different angles to control the robot parts, namely, base, shoulder, elbow, wrist, gripper, and finger. These angles are denoted in vector form as [θ_b, θ_s, θ_e, θ_w, θ_f]. In the given vector, the value θ_p corresponds to motion mapping to rotate the platter of the human object. The same θ_p is used to orient the descriptor displayed in the HoloLens emulator at the surgeon's side as well.

Figure 3 shows the DITF features of the phantom model, which is the essential step for 3D reconstruction. Based on the extracted features, the correspondences are identified in image matching. The correspondence between the reference images and various results of 45° rotation is shown in Figure 4. The correspondence with different colors clearly indicates the effectiveness of identifying and matching similar features even when the images are in different view orientations.In Figure 5, the accuracy of motion mapping is included, which indicates that when the distance between the two fingers is low, the accuracy is high. However, when the distance between fingers increases, the accuracy starts decreasing.

On the other hand, the time taken for the model to process the data is essential in AR. Therefore, this parameter is included for the validation of the proposed model, the time delay to process the image is measured, and the results are verified with existing algorithms such as Oriented FAST and rotated BRIEF (ORB)²⁴ and Boosted Efficient Binary Local Image Descriptor (BEBLID)²⁵. The results show that the DITF surpasses the existing models, such as ORB and BEBLID, in terms of latency, as shown in Figure 6. In addition to that, the 3D reconstruction model is validated with the reconstruction error, and the figure shown in Figure 7 indicates the histogram is narrow, which implies the reconstruction error is minimal; it shows that the reconstruction of the proposed algorithm is verified and validated. Figure 8 shows the output of the 3D reconstruction for the proposed model. It reflects the clarity in visualization, and the quantitative results of the image are also verified using the plots. These results prove that the proposed model extracted all the necessary features with the rotation transformation to reconstruct the 3D model. Therefore, the remote expert can have precise visualization and control of the surgical field.

Figure 1: Implementation of hardware setup for Transcatheter Aortic Valve Replacement automated surgery using Augmented Reality. (A) Surgical Field with a live monitoring system. (B) Augmented Reality-based Visualization. Please click here to view a larger version of this figure.

Figure 2: Expected model of the surgical environment. The model shown is equipped with live surveillance of the surgical environment using spatially distributed camera sensors. Please click here to view a larger version of this figure.

Figure 3: Model of a human heart placed in surgery testbed and its extracted features using feature descriptor algorithm. (A) Input heart phantom model. (B) Directional intensified feature description using tertiary filtering feature extraction of heart phantom Model. Please click here to view a larger version of this figure.

Figure 4: Feature matching between two image features. (A) Directional intensified feature description using tertiary filtering feature extraction for heart phantom model. (B) Directional intensified feature description using tertiary filtering feature extraction of 45° rotated heart phantom model. The lines indicate the correspondence between the features of similar images at two different orientations. The line color indicates the different chosen features. Please click here to view a larger version of this figure.

Figure 5: Accuracy of hand gesture tracking. The figure shows percentage accuracy. The sample number (N) is chosen to be 500. The motion mapping is done 500 times, out of which the number of correct maps is found. Then, the accuracy is calculated as the ratio between the correct mapping and the total number of samples. Please click here to view a larger version of this figure.

Figure 6: Latency of directional intensified feature description using tertiary filtering with Oriented FAST and rotated BRIEF and Boosted Efficient Binary Local Image Descriptor Algorithm. The latency, which is the time taken to extract the features from an image, is plotted. We chose N as 500, which means the time delay calculation is done over 500 times and then averaged. The procedure is done for three algorithms and plotted. Please click here to view a larger version of this figure.

Figure 7: Reconstruction error distribution of the proposed model. Reconstruction error is the error between the reconstructed 3D image from its feature and the original image. The error plot is a histogram that indicates the count of occurrences of a particular event. The count is maximum when the error is zero, and it decays on either side. This is the desired outcome. The figure indicates that the variance of the histogram (Normal density) is less, so the error from the mean (0) is not very much spread in either direction. Please click here to view a larger version of this figure.

Figure 8: 3D reconstruction of heart model. Each colored dot indicates the 3D reconstructed point from the corresponding feature. As of now, the feature reconstruction algorithm normalizes the color, and with that, other features are colored. Please click here to view a larger version of this figure.

Discussion

In an existing work¹⁵, X-ray and CT scans are examined to locate the catheter in the heart. However, AR TAVR replacement establishes a new possibility in TAVR¹⁸ surgical procedure by the implementation of an automated model using 3D reconstruction. As mentioned in the protocol section this work has five stages to design. The first stage of DITF²², mentioned in section 6, which we proposed in our previous work²², is enhanced in a way that embeds the model in medical applications. The second stage of 3D reconstruction is the challenging task of constructing the 3D model from 2D points. We tested the model with various input images to validate the model. SFM²³ is adopted to reconstruct the images. The correspondence of the image is measured with the help of DITF²², along with a calibration matrix. The next stage is hand gesture tracking; the interface of the hand gesture control in 3D reconstruction is another challenging task in real time. The scaling control and rotation variation are embedded with hand gestures for easy access to the 3D models by the remote expert. The fourth stage is Mobile app creation, which is stated in section 3 for the automatic image capturing. This app loads the data to the remote expert as a 3D model. The final stage is hardware setup, which is demonstrated using the heart phantom model with a robot arm, which is automated to insert the catheter to replace the injured valve of the heart, as shown in Figure 1.

The developed method is toward performing the remote surgery from the scene surveillance of the surgical environment using augmented reality and feature extraction. The significant contributions proposed in this article are not seen in the literature as per our best knowledge. The developed method has low latency to reconstruct the 3D image from the features. Also, the method has very low error between the reconstructed features and the original image. This latency and the error are shown in Figure 6 and Figure 7, respectively. The figures also show that the other algorithms, namely ORB²⁴ and BEBLID²⁵ are substandard than the proposed method.

The challenges that we faced in the development of the method is the Bluetooth protocol-based transmission between the mobile App and the remote server. As the scene capturing and transmission must be completed every 2 seconds, it demanded ultrahigh-speed communication. It often ended up in a broken socket, and it required a lot of exception handling and parallel routine management for smooth processing. In addition to that, we faced some issues during the development and implementation of the method.

However, there are some troubleshooting techniques to solve them. Bluetooth communication involves choosing the correct channel or VVID port number. Also, switching the Bluetooth device at either end is to be ensured before proceeding any further. Regarding the microprocessor-based routine to drive the robot arm and the hanging rotation arm, the port numbers for serial communication are to be ensured. Motor driver connection, considering the NEMA and servo motor is to be ensured properly. If not done properly, the NEMA motor rotates in an undesired fashion, even if just the connection to the driver is made wrong. The leads of two coils of the NEMA motor need to be identified properly; otherwise, the connection with the driver may end up rotating in an undesired fashion or even damage the motor. The design of a coupler to support the hanging of a rotating arm, which weighs around 1 kg, was a challenge. To address the issue, we drilled a hole of 1 mm diameter in the shaft of the motor. Balancing either side of the arm, including the weight of a mobile device, is also a challenge; otherwise, the structure may drag on the ground surface, which will lead to tear down or pull down of the structure from the motor shaft. Proper screwing in the base, shoulder, elbow, wrist, and fingers of the robot arm needs to be ensured; if not, the parts might fall when it rotates and lifts the weight. The parallel running of two modules should be taken care of by setting an appropriate Python environment, with different threads running in parallel for smooth working.

The results conclude that the proposed Automatic TAVR¹⁸ model is an effective method for surgical guidance in AR. This proposed model is flexible to apply this prototype to any surgery in the medical field as per the medical expert guidance. Learning-based models may improve the 3D reconstruction. To reconstruct the 3D model, multiple views with good lighting images need to be given as input; however, this can be addressed in the future. Moreover, in the future, we would like to develop a 5G communication-based transmission to ensure low latency and ultra-reliable communication for faultless and smooth operation. Also, we would like to develop our own AR device instead of an emulator currently used. This can improve the visualization of the 3D model in a live surgical environment.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
android IDE	software	https://developer.android.com/studio	software can be downloaded from this link
Arduino Board	Ardunio Uno	Ardunio Uno	Microcontroller for processing
arduino software	software	https://www.arduino.cc/en/software.	software can be downloaded from this link
Human Heart phantom model	Biology Lab Equipment Manufacturer and Exporter	B071YBLX2V(8B-ZB2Q-H3MS-1)	light weight model with 3parts to the deep analysis of heart.
mobile holder	Humble universal monopoad holder	B07S9KNGVS	To carry the mobile in surgical field
pycharm IDE	software	https://www.jetbrains.com/pycharm/	software can be downloaded from this link
Robot arm	Printed-bots	B08R2JLKYM(P0-E2UT-JSOU)	arm can be controlled through control signal.it has 5 degree of freedom to access.
servo motor	Kollmorgen Co-Engineers Motors	MG-966R	high-torque servo motor,servo pulses ranging from 500 to 2500 microseconds (µs), with a frequency of 50Hz to 333Hz.
servomotor	Kollmorgen Co-Engineers Motors	SG-90R	1.8 kg-cm to 2.5 kg-cm load can be applied to SG-90R servo.
Stepper Motor	28BYJ-48	28BYJ-48	Steper motor, 5V DC, 100 Hz frequency, torque 1200 Gf.cm
Stepper Motor	Nema 23	Nema	Steper motor, 9V - 42 V DC, 100 Hz frequency