Deep Learning-Based Spatiotemporal Analysis of Cataract Surgery Videos for Surgical Risk Assessment

Summary

This study presents a deep learning pipeline for predicting surgical risk from intraoperative microscopy videos using cGAN preprocessing, segmentation, GCN, ASFO, and transformer modeling. High performance is achieved on CaDIS and SICS-105 datasets, though clinical validity is limited by indirect labeling.

Abstract

Postoperative complications remain a significant challenge in vitrectomy surgery, often leading to impaired visual outcomes and repeated interventions. This study presents a deep learning framework for predicting surgical risk by extracting spatiotemporal features from intraoperative microscopy videos. During preprocessing, video enhancement techniques, including motion stabilization, illumination normalization, and cGAN-based artifact suppression, are applied to improve input quality. Contour-adaptive segmentation is employed to delineate relevant surgical regions, followed by Graph Convolutional Networks for structure-aware feature extraction. An Adaptive Sunflower Optimization approach is integrated to refine feature selection, and a transformer-based model captures temporal dependencies for final risk prediction. The framework is evaluated on two public datasets, CaDIS and SICS-105. The model achieved 98.9% accuracy, 97.5% precision, 98.2% recall, 97.9% F1-score, and 98.1% AUC on the CaDIS dataset. Furthermore, the segmentation module attained 98.9% pixel accuracy, 98.5% per-class accuracy, and 96.6% mIoU. The model achieved 99.1% accuracy, 98.5% F1-score, 98.7% sensitivity, 98.7% specificity, and an ROC AUC of 99.10% on the SICS-105 dataset. These results demonstrate consistent improvements over baseline models. Overall, integrating GAN-enhanced preprocessing, graph-based feature learning, optimization, and transformer modeling enhances predictive reliability. This approach highlights the potential of intraoperative video analysis for real-time clinical decision support and improved postoperative risk stratification.

Introduction

Pars plana vitrectomy (PPV) is a foundation for surgical procedures in the management of various vitreoretinal diseases, including retinal detachment, diabetic retinopathy, macular hole, and vitreous hemorrhage¹. Despite the advancement in surgical instrumentation and visualization, postoperative complications remain a significant morbidity source, with outcomes ranging from minor visual disturbances to vision-threatening events like recurrent retinal detachment, endophthalmitis, or uncontrolled intraocular pressure². Identification of patients at higher risk of these complications is a crucial unmet need in clinical pra....

Protocol

The CaDIS and SICS-105 datasets used in this study are publicly available and were originally collected with prior institutional review board approval and informed consent, as reported in their respective publications. This study involves only secondary analysis of fully anonymized data, and therefore, no additional ethical approval or informed consent was required.

The suggested model was designed to predict postoperative vitrectomy complications by exploiting spatiotemporal patterns from intraoperative microscopy videos. The pipeline comprises four major stages: preprocessing, segmentation, feature extraction, feature optimization, and cl....

Results

To evaluate the effectiveness of the proposed model, extensive experiments were conducted using real-world vitrectomy datasets. Performance was assessed across preprocessing, feature extraction efficacy, and prediction accuracy.

Performance analysis and comparative estimation

The proposed methodology was compared with baseline models using metrics including accuracy, precision, recall, F1-score, mIoU, pixel accuracy (PA), per-class accuracy (PAC), s.......

Discussion

An experimental evaluation on the CaDIS and SICS-105 datasets confirms the robustness and superiority of the proposed model over traditional approaches. Baseline models like ResNet-50, LSTM-CNN, and segmentation-driven networks (UNet, DeepLabV3+, UPerNet, HRNetV2) perform reasonably well but are hindered by inherent intraoperative video variability, including changes in illumination, occlusions, motion blur, and tool-tissue interactions^22,23. These limitations le.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
High-Performance Workstation	NVIDIA Corporation, USA	RTX-A6000	Used for model training; 48 GB VRAM GPU with Intel Core i9 CPU and 128 GB RAM
Python (v3.10)	Python Software Foundation	Open Source	Core programming language for model development and experimentation
TensorFlow (v2.15)	Google Research	Open Source	Framework used for implementing cGAN preprocessing and Transformer classifier
PyTorch (v2.2.0)	Meta AI	Open Source	Used for implementing GCN and Adaptive Sunflower Optimization modules
CUDA Toolkit (v12.1)	NVIDIA Corporation	CUDA-12.1	Provides GPU acceleration for all deep learning computations
Conditional Generative Adversarial Network (cGAN)	Adapted from Pix2Pix Framework	N/A	Used for video enhancement and artifact removal during preprocessing
Optical Flow Estimator (RAFT)	Teed & Deng (ECCV 2020)	N/A	Ensures temporal consistency across frames during preprocessing
Contour Adaptive Segmentation (CAS) Model	Custom Implementation	N/A	Modified active contour model for accurate boundary segmentation in vitrectomy videos
Dynamic Graph Convolutional Network (GCN)	Custom Implementation (based on Kipf & Welling, 2017)	N/A	Extracts spatiotemporal relationships among video feature nodes
Adaptive Sunflower Optimization Algorithm (ASFO)	Custom Implementation	N/A	Used for optimized feature selection and dimensionality reduction
Transformer Encoder Model	Custom Implementation (based on Vaswani et al., 2017)	N/A	Used for final postoperative complication classification with spatiotemporal attention pooling
Matplotlib (v3.8)	Python Software Foundation	Open Source	Used for visualization of performance metrics and plots
Seaborn (v0.13)	Python Software Foundation	Open Source	Used for advanced visualization and statistical graphics
Adam Optimizer	TensorFlow Built-in	N/A	Used for model training; learning rate = 1e-4, β1=0.9, β2=0.999
Loss Function Suite (L_adv, L_l1, L_perc, L_temp, L_G)	Custom Implementation	N/A	Defines adversarial, perceptual, pixel, and temporal losses for GAN training
Operating System	Ubuntu Linux 22.04 LTS	N/A	Base environment for deep learning experimentation
Evaluation Metrics Package	scikit-learn (v1.5.0)	Open Source	Used to compute accuracy, precision, recall, F1-score, AUC, and mIoU