Deep Learning-Based Spatiotemporal Analysis of Cataract Surgery Videos for Surgical Risk Assessment

Summary

This study presents a deep learning pipeline for predicting surgical risk from intraoperative microscopy videos using cGAN preprocessing, segmentation, GCN, ASFO, and transformer modeling. High performance is achieved on CaDIS and SICS-105 datasets, though clinical validity is limited by indirect labeling.

Abstract

Postoperative complications remain a significant challenge in vitrectomy surgery, often leading to impaired visual outcomes and repeated interventions. This study presents a deep learning framework for predicting surgical risk by extracting spatiotemporal features from intraoperative microscopy videos. During preprocessing, video enhancement techniques, including motion stabilization, illumination normalization, and cGAN-based artifact suppression, are applied to improve input quality. Contour-adaptive segmentation is employed to delineate relevant surgical regions, followed by Graph Convolutional Networks for structure-aware feature extraction. An Adaptive Sunflower Optimization approach is integrated to refine feature selection, and a transformer-based model captures temporal dependencies for final risk prediction. The framework is evaluated on two public datasets, CaDIS and SICS-105. The model achieved 98.9% accuracy, 97.5% precision, 98.2% recall, 97.9% F1-score, and 98.1% AUC on the CaDIS dataset. Furthermore, the segmentation module attained 98.9% pixel accuracy, 98.5% per-class accuracy, and 96.6% mIoU. The model achieved 99.1% accuracy, 98.5% F1-score, 98.7% sensitivity, 98.7% specificity, and an ROC AUC of 99.10% on the SICS-105 dataset. These results demonstrate consistent improvements over baseline models. Overall, integrating GAN-enhanced preprocessing, graph-based feature learning, optimization, and transformer modeling enhances predictive reliability. This approach highlights the potential of intraoperative video analysis for real-time clinical decision support and improved postoperative risk stratification.

Introduction

Pars plana vitrectomy (PPV) is a foundation for surgical procedures in the management of various vitreoretinal diseases, including retinal detachment, diabetic retinopathy, macular hole, and vitreous hemorrhage¹. Despite the advancement in surgical instrumentation and visualization, postoperative complications remain a significant morbidity source, with outcomes ranging from minor visual disturbances to vision-threatening events like recurrent retinal detachment, endophthalmitis, or uncontrolled intraocular pressure². Identification of patients at higher risk of these complications is a crucial unmet need in clinical practice, as it enables more informed surgical planning, personalized postoperative care, and timely interventions³. Traditionally, risk assessment in PPV has relied on static preoperative factors (such as comorbidities and preoperative ocular conditions) and subjective intraoperative judgment by the surgeon^4,5,6,7. Moreover, recent vitrectomy surgery is documented extensively over a high-resolution surgical microscopy video, offering a rich, but underutilized source of data. These recordings thus capture not only anatomical and pathological features but also surgeon-tissue interactions, intraoperative events, and surgical movements that may be predictive of further complications^8,9,10,11.

Deep Learning has shown remarkable performance across varied domains in ophthalmology, including fundus image classification, automated surgical skill assessment, and Optical Coherence Tomography (OCT) analysis¹². Moreover, some studies have systematically leveraged intraoperative video data to predict postoperative outcomes. The spatiotemporal nature of surgical videos poses unique challenges: high-dimensional data, variability across surgeons, temporal dependencies, patient anatomy, and devices^13,14,15,16. However, an eloquent integration of the clinical domain requires not only accurate or precise predictions but also robustness, interpretability, and generalizability across surgical settings in real time¹⁷.

In the fields of ophthalmology and surgical data science¹⁸, machine learning (ML) and deep learning (DL) have recently become quite popular. Their applications include outcome prediction, surgical skill evaluation, and diagnostic imaging¹⁹. DL-based frameworks for understanding intraoperative surgical situations have been investigated in several studies. For example, Nespolo²⁰ demonstrated generalizability across tasks by proposing a DL pipeline for the semantic interpretation of tissues and instruments in vitreoretinal surgery. To facilitate objective performance evaluation, this was further expanded to enable the extraction of comprehensive data on surgical techniques and instrument–tissue interactions. In a similar vein, Nespolo et al.²¹ demonstrated the viability of surgical guidance by developing a real-time detection and segmentation model utilizing intraoperative microscopy and an instance segmentation network (YOLACT++).

Additionally, AI-driven methods have been recognized as useful tools for improving workflow, training, and surgical evaluation^22,23. These techniques enable phase segmentation, real-time instrument tracking, and increased safety in ophthalmic microsurgery. The expanding use of machine learning (ML) in the analysis of intraoperative surgical recordings was further demonstrated by a systematic review by Mueller et al.²³, which also highlighted current issues with clinical translation and reliability. Several studies have used DL models with imaging and clinical data to predict postoperative outcomes. For instance, models based on OCT and clinical characteristics have been used to forecast visual outcomes in cases of idiopathic epiretinal membrane²⁴, and similar methods have shown efficacy in predicting postoperative outcomes in macular hole patients across multi-center datasets²⁵. Other studies have focused on determining the severity of proliferative vitreoretinopathy using multimodal DL frameworks26 and predicting retinal detachment recurrence using ultra-wide-field imaging²⁷. Furthermore, visual outcomes and underlying etiologies in vitreoretinal diseases have been predicted using machine learning models that incorporate demographic, clinical, and surgical variables^28,29,30.

Despite these developments, the majority of current methods place little emphasis on intraoperative dynamics and rely primarily on preoperative data or static postoperative imaging. Although some studies use surgical video analysis for segmentation and evaluation^20,21,22,23, they do not directly incorporate spatiotemporal intraoperative data to predict the likelihood of complications. As a result, there is still a significant gap in using intraoperative microscopic video data for predictive modeling. To create precise, clinically significant frameworks for predicting postoperative vitrectomy complications, this gap must be filled.

In this study, a novel multimodal DL framework is presented that integrates spatiotemporal features extracted from intraoperative vitrectomy videos with structured clinical metadata for predicting the risk of postoperative complications. By employing advanced video encoders (such as convolutional and transformer-based architectures) along with metadata fusion strategies¹⁸, this scheme aims to recognize high-risk intraoperative events, provide calibrated complication probabilities, and generate interpretable visualizations for physicians. By multi-center estimation, interpretability analyses, and comparative studies, the feasibility of using intraoperative video as a predictive biomarker for the surgical outcome is demonstrated¹⁸. Developing a strong DL framework that uses spatiotemporal data from intraoperative microscope recordings to forecast postoperative complications in vitrectomy surgery is the primary goal of this project. In terms of predictive accuracy and generalization, the proposed integrated approach, which combines cGAN-based preprocessing, GCN-driven feature extraction, ASFO-based optimization, and transformer-based modeling, is expected to surpass current methods.

This work efficiently leverages intraoperative microscope videos to present a comprehensive DL framework for postoperative vitrectomy problem prediction. In contrast to current methods, which mostly rely on static imaging modalities or preoperative clinical data, the proposed methodology combines advanced machine learning techniques with spatiotemporal video analytics in a novel way. In particular, the methodology uses Graph Convolutional Networks (GCNs) to capture structural and relational relationships within surgical scenes, Contour Adaptive Segmentation (CAS) for accurate region delineation, and a cGAN-based video enhancement module for artifact suppression. Furthermore, to improve model convergence and discriminative power, an Adaptive Sunflower Optimization (ASFO) algorithm is used to balance exploration and exploitation in feature selection. Lastly, temporal dynamics across surgical sequences are modeled using a transformer-based architecture with spatiotemporal attention pooling. A major improvement over current pipelines, this comprehensive integration of preprocessing, segmentation, graph-based feature learning, optimization, and attention-driven prediction enables more precise, reliable, and clinically meaningful risk prediction. The CaDIS and SICS-105 datasets are used in this study as proxy visual datasets to learn intraoperative spatiotemporal features, despite their primary purpose being cataract surgery analysis. Instead of using explicitly marked complication labels, these traits are then used to model and infer the risk of postoperative vitrectomy problems.

Protocol

The CaDIS and SICS-105 datasets used in this study are publicly available and were originally collected with prior institutional review board approval and informed consent, as reported in their respective publications. This study involves only secondary analysis of fully anonymized data, and therefore, no additional ethical approval or informed consent was required.

The suggested model was designed to predict postoperative vitrectomy complications by exploiting spatiotemporal patterns from intraoperative microscopy videos. The pipeline comprises four major stages: preprocessing, segmentation, feature extraction, feature optimization, and classification, as shown in Figure 1.

The intraoperative assessment of the risk of postoperative complications is the study's prediction target. The model cannot forecast clinically verified postoperative events within a specific time window because the CaDIS and SICS-105 datasets lack longitudinal follow-up data. Rather, it uses intraoperative surgical patterns to infer the probability of possible problems. Operational definitions of "complication" in this context include irregular phase transitions, anomalous instrument–tissue interactions, and variations in surgical workflow associated with greater surgical complexity. To ensure interpretability and reliable evaluation, the prediction job is designed as a binary classification problem that classifies cases into low- and high-risk groups. Instead of serving as a direct clinical diagnostic system, this formulation allows the suggested framework to serve as an intraoperative decision-support tool.

Dataset Description:

The proposed model was evaluated on two datasets: dataset A (CaDIS, Cataract Dataset for Image Segmentation) and the SICS-105 (Small-Incision Cataract Surgery) dataset.

(i) CaDIS³¹: This dataset was developed for the task of segmentation in cataract surgery videos. This is introduced to complement the publicly available CATARACTS challenge dataset and thus aims to advance the development of DL schemes for the analysis of surgical videos. The dataset consists of 25 video recordings totaling 4,670 annotated frames, depicting several stages of cataract surgery (Supplementary Figure 1). Each frame is meticulously annotated, providing pixel-level labels needed for training and estimating the segmentation scheme. It serves as a benchmark for evaluating the DL model's performance on surgical videos.

(ii) SICS-105 (small-incision cataract surgery) dataset³²: This dataset comprises 105 recordings performed by surgeons and annotated by four ophthalmologists over a 6-month period during 2023–2024, covering 20 phases (Supplementary Figure 2). The total video duration is 22 h and 39 min, with a resolution of 1920 × 1080 pixels (downsampled to 960 × 540) and 30 fps. An average video length is 12 minutes and 57 sec (σ=4:31 min) . A dataset is publicly accessible in the Zenodo repository at the URL: https://doi.org/10.5281/zenodo.13847781. Representative input samples are shown in Supplementary Figure 1 and Supplementary Figure 2. Although the CaDIS and SICS-105 datasets were originally designed for surgical phase recognition and segmentation in cataract surgery, there is limited information on postoperative vitrectomy complications. To address the lack of information on postoperative vitrectomy complications, we propose a set of proxy labels for risk information based on intraoperative visual patterns and events in the video sequence. The frames and video sequences in the datasets were annotated with three-level risk information (low, moderate, high) according to a set of heuristics inspired by clinical understanding, including instrument-tissue interaction complexity, surgical duration abnormalities, occlusion, instability, and abnormal motion. The proposed formulation does not use clinical outcomes as labels but rather treats them as proxy labels for risk information on potential postoperative complications.

Preprocessing using cGAN approach

Typically, raw frames are often degraded by illumination variation, blur, and specular artifacts. To address this, a conditional Generative Adversarial Network (cGAN) is used to enhance the video. Raw frames are the original, unprocessed frames extracted directly from intraoperative microscopy video recordings, prior to any enhancement or normalization. These frames typically contain artifacts such as illumination variability, motion blur, specular reflections, and noise arising from surgical instruments and camera movement. A generator restores clean frames while a discriminator enforces visual realism. The objective function that integrates pixel reconstruction, adversarial, and temporal consistency losses, ensuring artifact-free and stable sequences of video for further analysis.

A degraded frame I_t at time step t is the generator's input, while an improved frame Y_t = G(I_t) is its output. Conditioned on the input I_t, the discriminator is trained to distinguish between the generated output and the ground truth clean frame Y_t. The training objective incorporates several loss functions to guarantee high-quality reconstruction: (i) adversarial loss to enforce realism; (ii) pixel-wise reconstruction loss (L1 loss) to maintain intensity fidelity; (iii) perceptual loss to preserve structural consistency using deep feature representations; and (iv) temporal consistency loss to ensure smooth transitions across consecutive frames and prevent flickering artifacts.

To ensure high-quality input for feature extraction, intraoperative microscopy videos are initially enhanced using a cGAN. Each raw frame I_t* is considered a degraded observation of a clean frame . A generator G learns a mapping G(I_t) = I_t for restoring enhanced frames, whereas the discriminator D distinguishes between real and generated samples. A training objective integrates adversarial loss L_adv, , perceptual loss L_perc pixel-based reconstruction loss L_l1, and temporal consistency L_temp.. This model suppresses blur, specular artifacts, and noise while maintaining temporal and structural coherence, yielding stable, artifact-free video sequences for downstream analysis. The formulation of GAN is expressed in Equations [1-6] of the Supplementary File 1.

Segmentation using Contour Adaptive Segmentation (CAS)

In this, segmentation is performed using a contour-adaptive segmentation model. This defines preliminary outlines for segmenting affected vitrectomy images in microscopy images, thus reducing the energy function, as shown in equations [7-12] in Supplementary file 1. The low energy value ensures precise representations and localized evaluation of regions at risk. This, in turn, allows precise segmentation of affected regions, which subsequently facilitates feature extraction in the subsequent section.

Feature Extraction using Dynamic spatiotemporal GCN(dGCN)

Once the preprocessing stage is complete, the next step is to extract discriminative spatiotemporal features. For this reason, a Graph Convolutional Network (GCN) is used, as it is effective at capturing structured dependencies. The adopted model comprises stacked GCN layers with non-linear activations that mitigate overfitting and address the gradient vanishing problem through a normalization mechanism. The GCN formulation relies on a first-order spectral approximation of graph convolutions, making it suitable for large-scale semi-supervised learning tasks. Moreover, the linearized representation simplifies optimization, thereby ensuring effective parameter learning. The operation of simplified GCN is expressed in Equations [13–15] of Supplementary File 1. When applied, GCN efficiently extracts higher-level structural representations from intraoperative video data. Subsequently, only the most relevant or appropriate features are thus retained over an optimization-aided selection mechanism, which is described in the subsequent section.

Feature Optimization using Adaptive Sunflower-based feature optimization (ASFO)

The Adaptive Sunflower Optimization Algorithm (ASFO) is an enhanced version of the Classical Sunflower Optimization Approach (SFOA). In the suggested model, this biologically driven process is mathematically modeled to address refinement and feature selection in a high-dimensional dataset. Specifically, this approach is tailored to improve convergence speed, balance exploration and exploitation to preserve solution diversity, and mitigate premature convergence. The mathematical formulation of ASFO is stated in Equations [16–21] of Supplementary File 1. This scheme accelerates convergence once solutions are near optimal. The detailed algorithm is provided in Supplementary File 1. In the proposed pipeline, ASFO is used to refine the feature vector after transformer-based fusion. The selected features are texture-based, color and intensity, morphological and structural, motion and temporal, high-dimensional, and attention-weighted region descriptors. The objective is to minimize the feature-relevance loss function, thereby selecting and weighting features that enhance classification accuracy by reducing redundancy.

Transformer-based classification head with spatiotemporal attention pooling

A transformer-based model is used to capture temporal dependencies across surgical video sequences. The model is configured with multi-head self-attention, positional encoding, and fully connected layers for final risk prediction. Hyperparameter optimization is performed using ASFO. The integrated pipeline combines preprocessing, feature extraction, optimization, and transformer-based classification to enable accurate surgical risk prediction.

Results

To evaluate the effectiveness of the proposed model, extensive experiments were conducted using real-world vitrectomy datasets. Performance was assessed across preprocessing, feature extraction efficacy, and prediction accuracy.

Performance analysis and comparative estimation

The proposed methodology was compared with baseline models using metrics including accuracy, precision, recall, F1-score, mIoU, pixel accuracy (PA), per-class accuracy (PAC), s...

Discussion

An experimental evaluation on the CaDIS and SICS-105 datasets confirms the robustness and superiority of the proposed model over traditional approaches. Baseline models like ResNet-50, LSTM-CNN, and segmentation-driven networks (UNet, DeepLabV3+, UPerNet, HRNetV2) perform reasonably well but are hindered by inherent intraoperative video variability, including changes in illumination, occlusions, motion blur, and tool-tissue interactions^22,23. These limitations le...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
High-Performance Workstation	NVIDIA Corporation, USA	RTX-A6000	Used for model training; 48 GB VRAM GPU with Intel Core i9 CPU and 128 GB RAM
Python (v3.10)	Python Software Foundation	Open Source	Core programming language for model development and experimentation
TensorFlow (v2.15)	Google Research	Open Source	Framework used for implementing cGAN preprocessing and Transformer classifier
PyTorch (v2.2.0)	Meta AI	Open Source	Used for implementing GCN and Adaptive Sunflower Optimization modules
CUDA Toolkit (v12.1)	NVIDIA Corporation	CUDA-12.1	Provides GPU acceleration for all deep learning computations
Conditional Generative Adversarial Network (cGAN)	Adapted from Pix2Pix Framework	N/A	Used for video enhancement and artifact removal during preprocessing
Optical Flow Estimator (RAFT)	Teed & Deng (ECCV 2020)	N/A	Ensures temporal consistency across frames during preprocessing
Contour Adaptive Segmentation (CAS) Model	Custom Implementation	N/A	Modified active contour model for accurate boundary segmentation in vitrectomy videos
Dynamic Graph Convolutional Network (GCN)	Custom Implementation (based on Kipf & Welling, 2017)	N/A	Extracts spatiotemporal relationships among video feature nodes
Adaptive Sunflower Optimization Algorithm (ASFO)	Custom Implementation	N/A	Used for optimized feature selection and dimensionality reduction
Transformer Encoder Model	Custom Implementation (based on Vaswani et al., 2017)	N/A	Used for final postoperative complication classification with spatiotemporal attention pooling
Matplotlib (v3.8)	Python Software Foundation	Open Source	Used for visualization of performance metrics and plots
Seaborn (v0.13)	Python Software Foundation	Open Source	Used for advanced visualization and statistical graphics
Adam Optimizer	TensorFlow Built-in	N/A	Used for model training; learning rate = 1e-4, β1=0.9, β2=0.999
Loss Function Suite (L_adv, L_l1, L_perc, L_temp, L_G)	Custom Implementation	N/A	Defines adversarial, perceptual, pixel, and temporal losses for GAN training
Operating System	Ubuntu Linux 22.04 LTS	N/A	Base environment for deep learning experimentation
Evaluation Metrics Package	scikit-learn (v1.5.0)	Open Source	Used to compute accuracy, precision, recall, F1-score, AUC, and mIoU