Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery

Summary

The human depth perception of 3D stereo videos depends on the camera separation, point of convergence, distance to, and familiarity of the object. This paper presents a robotized method for rapid and reliable test data collection during live open-heart surgery to determine the ideal camera configuration.

Abstract

Stereo 3D video from surgical procedures can be highly valuable for medical education and improve clinical communication. But access to the operating room and the surgical field is restricted. It is a sterile environment, and the physical space is crowded with surgical staff and technical equipment. In this setting, unobscured capture and realistic reproduction of the surgical procedures are difficult. This paper presents a method for rapid and reliable data collection of stereoscopic 3D videos at different camera baseline distances and distances of convergence. To collect test data with minimum interference during surgery, with high precision and repeatability, the cameras were attached to each hand of a dual-arm robot. The robot was ceiling-mounted in the operating room. It was programmed to perform a timed sequence of synchronized camera movements stepping through a range of test positions with baseline distance between 50-240 mm at incremental steps of 10 mm, and at two convergence distances of 1100 mm and 1400 mm. Surgery was paused to allow 40 consecutive 5-s video samples. A total of 10 surgical scenarios were recorded.

Introduction

In surgery, 3D visualization can be used for education, diagnoses, pre-operative planning, and post-operative evaluation^1,2. Realistic depth perception can improve understanding^3,4,5,6 of normal and abnormal anatomies. Simple 2D video recordings of surgical procedures are a good start. However, the lack of depth perception can make it hard for the non-surgical colleagues to fully understand the antero-posterior relationships between different anatomical structures and therefore also ....

Protocol

The experiments were approved by the local Ethics Committee in Lund, Sweden. The participation was voluntary, and the patients' legal guardians provided informed written consent.

1. Robot setup and configuration

NOTE: This experiment used a dual-arm collaborative industrial robot and the standard control panel with a touch display. The robot is controlled with RobotWare 6.10.01 controller software and robot integrated development environment (IDE) RobotStudio 2019.5²⁵. Software developed by the authors, including the robot application, recording application, and postprocessing scripts, a....

Results

An acceptable evaluation video with the right image placed at the top in top-bottom stereoscopic 3D is shown in Video1. A successful sequence should be sharp, focused, and without unsynchronized image frames. Unsynchronized video streams will cause blur, as shown in the file Video 2. The convergence point should be centered horizontally, independent of the camera separation, as seen in Figure 9A,B. When the robot transitions between the posi.......

Discussion

During live surgery, the total time of the experiment used for 3D video data collection was limited to be safe for the patient. If the object is unfocused or overexposed, the data cannot be used. The critical steps are during camera tool calibration and setup (step 2). The camera aperture and focus cannot be changed when the surgery has started; the same lighting conditions and distance should be used during setup and surgery. The camera calibration in steps 2.1-2.4 must be carried out carefully to ensure that the heart .......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
2 C-mount lenses (35 mm F2.1, 5 M pixel)	Tamron	M112FM35	Rated for 5 Mpixel
3D glasses (DLP-link active shutter)	Celexon	G1000	Any compatible 3D glasses can be used
3D Projector	Viewsonic	X10-4K	Displays 3D in 1080, can be exchanged for other 3D projectors
6 M2 x 8 screws			To attach the cXimea cameras to the camera adaptor plates
8 M2.5 x 8 screws			To attach the circular mounting plates to the robot wrist
8 M5 x 40 screws			To mount the robot
8 M6 x 10 screws with flat heads			For attaching the circular mounting plate and the camera adaptor plates
Calibration checker board plate (25 by 25 mm)			Any standard checkerboard can be used, including printed, as long as the grid is clearly visible in the cameras
Camera adaptor plates, x2			Designed by the authors in robot_camera_adaptor_plates.dwg, milled in aluminium.
Circular mounting plates, x2			Distributed with the permission of the designer Julius Klein and printed with ABS plastic on an FDM 3D printer. License Tecnalia Research & Innovation 2017. Attached as Mountingplate_ROBOT_SIDE_ NewDesign_4.stl
Fix focus usb cameras, x2 (5 Mpixel)	Ximea	MC050CG-SY-UB	With Sony IMX250LQR sensor
Flexpendant	ABB	3HAC028357-001	robot touch display
Liveview			recording application
RobotStudio			robot integrated development environment (IDE)
USB3 active cables (10.0 m), x2			Thumbscrew lock connector, water proofed.
YuMi dual-arm robot	ABB	IRB14000