A Novel Dual-Modal Deep Learning Approach for Real-Time Removal of Hepatic Fluorescence in Indocyanine Green-Guided Laparoscopic Cholecystectomy

Summary

This protocol presents a dual-modal deep learning approach for real-time hepatic fluorescence removal during indocyanine green-guided laparoscopic cholecystectomy (ICGLC).

Abstract

Indocyanine green-guided laparoscopic cholecystectomy (ICGLC) improves biliary visualization but is often hindered by hepatic fluorescence contamination, which interferes with anatomical structures. This study aims to develop and evaluate a novel dual-modal deep learning framework to automatically detect and remove hepatic fluorescence contamination in real-time during ICGLC procedures. A dataset of 33,123 dual-modality surgical frames was constructed from 48 patients who underwent elective ICGLC. Fluorescence and white-light images were fused and annotated. Several deep learning models were compared, and a DeepLabV3-based mid-fusion network was selected. Morphological post-processing and phased training strategies were implemented to enhance segmentation accuracy and generalizability. The proposed model achieved a Dice coefficient of 0.838 and recall of 0.863 on the final test set. In subjective evaluations, 10 senior surgeons consistently rated the AI-processed videos as clearer, with markedly improved bile duct visualization and reduced visual fatigue. The model demonstrated real-time performance at 0.018 seconds per frame. This study presents the first real-time AI solution for hepatic fluorescence removal in ICGLC. The dual-modal deep learning model significantly enhances visual clarity, offering potential to improve surgical safety, operational efficiency, and training effectiveness. Future prospective studies are warranted to assess the clinical impact on operative outcomes.

Introduction

Cholelithiasis is a prevalent gastrointestinal disorder, consistently showing high incidence rates globally across various populations, affecting over 4% of the global population^1,2,3,4. Laparoscopic cholecystectomy (LC) is extensively regarded as the 'gold standard' for treating cholelithiasis due to its advantages such as minimal intraoperative trauma, mild postoperative pain, favorable cosmetic results, and short hospital stays^5,6,7. In the United States alone, general surgeons perform between 750,000 and 1,000,000 LCs annually. However, the complication rate of LC exceeds that of traditional open surgery⁸, with the occurrence of serious complications like biliary duct injury (BDI) being 2 to 3 times higher^9,10. Research indicates that BDI during laparoscopic procedures frequently arises from technical errors in the anatomical dissection phase, with anatomical misidentifications being particularly prevalent (up to 85%), often involving misjudgments regarding the common bile duct (CBD) or variations in the anatomy of the right posterior hepatic duct^11,12.

To decrease the prevalence of surgical complications in contemporary practice, an array of sophisticated methods and advanced technologies have been developed, including critical view of safety (CVS), intraoperative cholangiography, and near-infrared fluorescence (NIRF) technology^5,8. The CVS is increasingly regarded as a standard method for identifying gallbladder structures^5,13,14, requiring the removal of connective tissue in the cystohepatic triangle and dissection of the lower third of the gallbladder from the cystic plate, to free a clear view of the cystic duct and cystic artery, thus preventing vascular and biliary injuries during laparoscopic cholecystectomy^11,15.

Indocyanine green (ICG)-based real-time intraoperative NIRF cholangiography was another proven solution, which was first clinically reported in 2008¹⁶. ICG is metabolized exclusively by the liver, binds to plasma proteins, and emits a peak light wavelength of about 830 nm when irradiated with near-infrared light¹⁷. This characteristic allows for the visualization of liver tumors and the biliary system, showing promising applications in enhancing surgical safety and reducing operation times^18,19,20. Studies comparing the combined use of CVS and NIRF against CVS alone indicated benefits for obese patients and cases with severe inflammation, not only shortening the surgery time but also reducing the incidence of BDI^21,22. However, its clinical applications encounter several limitations, including the use of various imaging devices²³, and delayed ICG injection or excessive liver tissue absorption that result in high liver fluorescence intensity and contaminating the field of view, thereby obscuring crucial biliary system imaging²⁴. If auxiliary technologies could be applied intraoperatively to remove liver fluorescence contamination while preserving correct biliary fluorescence imaging, it could help optimize the surgical view, further enhancing the safety and smoothness of the surgery.

Artificial intelligence (AI) technology is a promising tool for optimizing fluorescence. AI in medical imaging and surgical operations has been a research hotspot in recent years, achieving numerous technological breakthroughs and clinical applications^25,26,27. Existing studies and applications of AI deep learning technology in laparoscopic surgery videos were often based on white-light single modality, including the recognition of various anatomical landmarks such as the recurrent laryngeal nerve, kidney edges, blood vessels, and the gallbladder^28,29,30,31. Currently, there is a lack of research on the real-time removal of liver fluorescence contamination during fluorescence-assisted LC. Dual-modal image segmentation, which has been widely applied in industry, may help fill this gap. This study aims to explore applications in dual-modal image segmentation within ICGLC scenarios, and to develop and validate training methods for real-time dynamic recognition and shielding of excessive liver fluorescence.

Protocol

All surgical videos included in this study were obtained from patients who underwent elective ICG-guided laparoscopic cholecystectomy (ICGLC) at Peking Union Medical College Hospital. The study was approved by the relevant ethics committee, and informed consent was obtained from all participants before enrollment.

1. Data preparation and image annotation

1. Retrospectively collect surgical videos from 48 patients who underwent ICGLC at Peking Union Medical College Hospital between 2022 and 2024.
   NOTE: The patients included in this study were intravenously injected with 0.5 mg of ICG 20 min before surgery.
2. Instruct a general surgeon to review all surgical videos and extract segments encompassing the critical dissection steps of Calot's triangle. Select 76 surgical video clips manually, focusing on the dissection of the Calot's triangle. Extract fluorescence and white light imaging channels from each clip. Concatenate them along the RGB channel.
3. To minimize redundancy, sample frames at an 8:1 interval, downscale the resolution from 1920 × 1080 to 960 × 540, and import the images into a custom annotation platform.
4. Assign a general surgeon to manually delineate liver fluorescence contours and ensure annotation quality through supervision by a senior surgeon.

2. Model evaluation metrics

1. Evaluate the segmentation performance of the model using standard computer vision metrics, including precision, recall, and the Dice coefficient, based on the designated test dataset.
2. Assess real-time performance by measuring computational load during inference.
   1. Measure inference time using the time module from the Python/C standard library.
   2. Record the starting timestamp by calling time.perf_counter() prior to data loading.
   3. Invoke time.perf_counter() again after the model generates the prediction results to obtain the ending timestamp.
   4. Define the difference between the ending and starting timestamps as the inference time for a single sample.
   5. Repeat this procedure for each sample in the test dataset, and compute the average inference time across all samples to represent the overall model inference latency.
   6. Refer to the following pseudocode that summarizes the corresponding procedure：
      Initialize Inference_time = 0 and ind = 0
      Let X denote the test set storage path and model denote the inference model
      For each x_path in X :
      Synchronize CPU and GPU
      Start_timestamp = time.perf_counter() # starting timestamp
      x = load(x_path) # data loading
      y = model(x) # model inference
      Synchronize CPU and GPU
      End_timestamp = time.perf_counter() # ending timestamp
      Inference_time += (end_timestamp - start_timestamp)
      Ind += 1
      End for
      Inference_time = Inference_time / ind # mean inference time
3. Set the primary goal of hepatic and biliary fluorescence recognition to detect the liver region within the surgical field and replace the corresponding white-light information, thereby minimizing interference from hepatic fluorescence.
4. Use the Intersection over Union (IoU) to quantify the model's predictive accuracy.
   NOTE: Intersection over Union (IoU) is defined as the ratio of the number of pixels in the overlapping area between the predicted positive region and the ground-truth positive region to the total number of pixels in the union of the predicted and ground-truth positive regions.
5. Define a prediction as a True Positive (TP) if the IoU with the ground truth exceeds a specified threshold; otherwise, classify it as a False Negative (FN).
6. Mark any predicted segmentation without a corresponding annotation as a False Positive (FP).
7. After computing the Intersection over Union (IoU) between the predicted results and the ground-truth labels, set the IoU threshold to 0.1, considering the diffuse and low-contrast nature of fluorescence signals in surgical imaging.
8. Define and compute the evaluation metrics as follows:
   
   
   

3. Morphological post-processing

1. Downsample all hepatic and biliary fluorescence frames from 1920 × 1080 to 960 × 540 resolution using an interpolation-based method, specifically bilinear interpolation.
2. Apply zero-padding along the width to obtain a final input dimension of 960 × 544 by generating a 960 × 544 all-zero matrix and centering the current image within it.
3. Concatenate the white-light and fluorescence images along the channel dimension by storing their corresponding matrices in sequence within a tensor constructed using the PyTorch framework before feeding them into the model.
4. To reduce noisy predictions and refine boundary quality, perform morphological opening and closing operations on the initial segmentation outputs to filter spurious noise and improve edge delineation.
5. Begin with an opening operation to eliminate small spikes and edge artifacts in the predicted mask³².
6. Follow with a closing operation to remove isolated holes and fragmented regions.
7. Use a 25 × 25 structuring kernel for both operations to ensure effective refinement of organ boundaries and reduction of segmentation noise.

4. Training environment and hyperparameters

1. Train the model using a single NVIDIA A100 Tensor Core GPU.
2. Set the batch size to 16.
3. Initialize the encoder (backbone) using ImageNet³³ pre-trained weights to leverage transfer learning and accelerate convergence.
4. Use the Stochastic Gradient Descent (SGD)³⁴ optimizer for its robustness in large-scale image segmentation tasks and its capacity to mitigate noisy gradients.
5. Set the initial learning rate to 0.01 and apply a polynomial decay strategy to gradually reduce the learning rate, with a lower bound of 0.0001.
6. Continue training for a total of 50,000 iterations to obtain the final model.

5. Three-phase model training

1. In the Model Selection Phase, compare multiple segmentation models using dual-modality input, including DeeplabV3³⁵, DLinkNet34³⁶, and U-Net³⁷; select the optimal model based on segmentation accuracy and computational efficiency.
2. In the Algorithmic Structural Optimization Phase, investigate different feature fusion positions within the network to determine the most effective integration strategy for white-light and fluorescence data.
3. In the Generalization and Clinical Validation Phase, apply progressive data augmentation techniques to enhance model robustness; validate the final model in clinical scenarios to assess its real-world applicability.

6. Surgical video processing

1. Extract video segments depicting the intraoperative ICG fluorescence visualization of biliary anatomy, including the common bile duct (CBD) and cystic duct.
2. Process the selected videos in real-time using the optimal model; compute the average per-frame processing time to evaluate real-time computational performance.

7. Liver fluorescence contamination removal

1. Apply the model to detect and suppress liver fluorescence contamination in real-time during ICGLC procedures.
2. Ensure preservation of biliary structure visualization to facilitate efficient identification of bile ducts.
3. Maintain contamination suppression performance despite common intraoperative challenges such as instrument occlusion or camera movement.

8. Clinical evaluation of video quality

1. Recruit a panel of 10 senior general surgeons from external institutions to evaluate the effectiveness of the AI-processed videos.
2. Provide each surgeon with both original and AI-enhanced video clips; instruct them to complete a subjective assessment questionnaire based on five predefined evaluation criteria.

Results

In our study, a total of 48 patients who underwent elective indocyanine green laparoscopic cholecystectomy (ICGLC) between 2022 and 2024 in the Department of General Surgery at Peking Union Medical College Hospital were retrospectively enrolled. All patients received an intravenous injection of 0.5 mg of indocyanine green 20 min prior to surgery. Postoperative pathological examination confirmed that all cases were diagnosed with calculous cholecystitis.

After manual segmentation and selection,...

Discussion

This research leveraged AI deep learning to address a key challenge in ICGLC surgeries: identifying and masking hepatic fluorescence contamination. Surgeons using NIRF often report that liver fluorescence causes visual fatigue and complicates biliary imaging³⁸. By applying computer vision technology, this study aimed to mitigate these issues and potentially reduce the incidence of BDI, a common complication in ICGLC. The model's inference time for processing a single frame is approximately 0.0...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
4K Fluorescence Imaging Console	Hangzhou Kangji Medical instrument Co. Ltd.	KJ-EP4K-01	Provides three imgaging modes: white light, fluorescence, and fused fluorescence; Support expansion of AI modules
4K Fluorescence Camera	Hangzhou Kangji Medical instrument Co. Ltd.	KJ-EC-4K	Fully digitalized camera with multi-function button settings; Support video recording, photography, and fluorescence mode switching
Cold Light Source System	Hangzhou Kangji Medical instrument Co. Ltd.	KJ-CLS-1	LED cold light source and fluorescenct LED light source with a fluorescence wavelength of 808 nm
NVIDIA A100 Tensor Core	NVIDIA Corporation	900-21001-0020-100	GPU