Internal Validation of a Clinical Prediction Model for Delayed Cerebral Ischemia After Aneurysmal Subarachnoid Hemorrhage

Summary

This single-center retrospective study developed and internally validated a logistic regression model for delayed cerebral ischemia after aneurysmal subarachnoid hemorrhage. In 680 patients, six early clinical, laboratory, and imaging variables showed preliminary predictive value. Further external validation is required before clinical use.

Abstract

Delayed cerebral ischemia (DCI) is an important cause of secondary neurological injury after aneurysmal subarachnoid hemorrhage (aSAH). This retrospective single-center study developed and evaluated prediction models for DCI using data from 680 adult patients with aSAH treated between July 2022 and December 2024. Patients were divided using an outcome-stratified 8:2 split into a training cohort of 544 patients and a hold-out internal-validation cohort of 136 patients. DCI occurred in 175 patients in the training cohort and 42 patients in the internal-validation cohort. Predictor selection was performed in the training cohort using least absolute shrinkage and selection operator regression with 10-fold cross-validation. Six variables were retained: age, cerebral edema, hypoalbuminemia, modified Fisher grade, Hunt-Hess grade, and World Federation of Neurological Surgeons grade. Logistic regression, extreme gradient boosting, light gradient boosting machine, support vector machine, and k-nearest neighbor models were compared. Logistic regression showed an area under the receiver operating characteristic curve of 0.832 (95% confidence interval, 0.758–0.906) in the internal-validation cohort. Its calibration slope, calibration intercept, and Brier score were 0.98, 0.02, and 0.168, respectively, although confidence intervals for these calibration estimates were unavailable. The findings represent preliminary performance in a single hold-out internal-validation cohort. Incomplete reproducibility records, the absence of the model intercept, the lack of resampling-based optimism correction, and the absence of external validation currently prevent patient-level probability calculation and clinical implementation.

Introduction

Several prediction models for DCI after aSAH have been proposed, but many have been limited by small sample sizes, incomplete calibration assessment, insufficient reporting of model development procedures, and a lack of external validation. Many studies have focused mainly on discrimination, while calibration, reproducibility, and clinical-impact assessment have been reported less consistently. Therefore, model performance should be evaluated using transparent methods and interpreted cautiously when validation is limited to a single center.

The present study aimed to develop and internally evaluate prediction models for DCI using routinely available early clinical, laboratory, and imaging variables. Five modeling approaches were compared using the same set of selected predictors. The intended prediction time point was after the initial admission, laboratory, and imaging assessment but before the main DCI risk period. The models were developed for research-based risk estimation and were not intended to diagnose DCI, replace clinical judgment, or independently determine treatment. External validation, recalibration, and clinical-impact testing are required before clinical use.

Delayed cerebral ischemia (DCI) is one of these secondary insults that has been among the most clinically significant and potentially preventable conditions following aSAH. DCI is a complication that usually arises several days following the ictus, typically within 3–14 days, and is linked with new focal neurological impairment, worsening of the level of consciousness, and/or onset of cerebral infarction on subsequent neuroimaging. Although there are some differences in diagnostic practice found in different institutions, DCI is always associated with a long stay in the intensive care unit, high resource consumption, and poor neurological prognosis. The reported DCI incidence is typically 30–40 percent, underscoring its prevalence and significant prognostic implications^1,2,3,4,5,6. Notably, DCI is not a single-pathway process. Although the focus has always been on large-vessel cerebral vasospasm, accumulating evidence also shows that early brain injury, cortical spreading depolarization, neuroinflammation, microthrombosis, endothelial dysfunction, impaired autoregulation, and microcirculatory failure are all contributors to ischemic risk. This multifactorial pathophysiology helps explain why vasospasm-targeted treatments may not fully prevent DCI-associated infarction and why high-risk patients are difficult to identify in the usual clinical setting.

The risk stratification of DCI following aSAH is a priority because, with prompt identification of high-risk patients, it may be possible to monitor them more closely and escalate preventive or rescue measures promptly⁷. Monitoring plans in clinical practice can involve the use of more frequent neurological tests, transcranial Doppler ultrasonography, sophisticated imaging techniques, such as computed tomography perfusion, hemodynamic and intravascular volume optimization, strict compliance with nimodipine therapy, and early identification of neurological deterioration, such as hydrocephalus, rebleeding, seizures, infection, or metabolic imbalance. Nevertheless, global severity scales and clinicians' experience tend to influence clinical decisions rather than the personalized, data-driven DCI probability estimates^8,9,10. One reason is that the literature indicates that risk factors are heterogeneous, and the interplay among patient factors, hemorrhage burden, physiological derangements, and neurological grading scales may be nonlinear and complex. Previous researchers have investigated potential predictors of DCI, including age, hypertension, baseline neurological status, aneurysm characteristics, laboratory indices (such as serum sodium and albumin), imaging features indicative of hemorrhage load, and perioperative or treatment-related factors. Conventional regression techniques have been helpful, but may be constrained when the predictors are correlated, when the relationship between the predictors and the risk is nonlinear, or when there are interactions among the variables¹¹. Simultaneously, multiple instruments (most commonly nomograms) have been proposed to assess DCI risk. Even though nomograms are feasible and visually appealing, most of them are based on small samples, include few variables, and may be overfit or poorly validated. In addition, models developed and tested at one center might not be readily translatable to other facilities due to differences in patient mix, image interpretation, treatment, and DCI diagnostic thresholds.

Machine-learning methods may complement traditional clinical prediction models, as they allow exploration of more dimensions of clinical data and observation of more intricate trends without the need to make rigid linear assumptions. ML-based prediction models have found increasing applications in the field of cerebrovascular disease in recent years, including the prediction of outcomes and complications, as well as decision support^12,13,14. Commonly applied machine learning algorithms include ensemble tree methods such as Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM), kernel-based classifiers such as support vector machines (SVMs), and distance-based classifiers such as k-nearest neighbors (KNN). The models are, in principle, capable of modeling nonlinearities and interactions more than traditional approaches do¹⁵. However, there are also practical concerns associated with ML models, such as the risk of overfitting, reduced interpretability, and the need to handle preprocessing, tuning, calibration, evaluation, and validation. It is worth noting that in a wide range of clinical scenarios, well-specified logistic regression can perform as well as, or even more effectively than, more complex algorithms, especially when the number of predictors is small and the signal-to-noise ratio is moderate. Thus, a comparative analysis of various algorithms using similar predictors and evaluation metrics is needed to identify a model that correlates with performance and clinical applicability^16,17,18.

Several prediction models for delayed cerebral ischemia have been previously proposed; however, many suffer from important limitations, including small sample sizes, limited calibration assessment, lack of head-to-head algorithm comparison, and insufficient validation. Furthermore, prior studies frequently emphasize discrimination while underreporting calibration and clinical utility metrics, which are essential for real-world applicability. The present study addresses these gaps by developing a predictive model in a comparatively large cohort, implementing systematic predictor selection using the least absolute shrinkage and selection operator (LASSO) regression, comparing multiple machine learning algorithms within a unified modeling framework, and comprehensively evaluating discrimination, calibration, and decision-analytic performance.

Several prediction models for DCI after aSAH have been proposed, but many have important limitations, including small cohorts, limited calibration assessment, incomplete reporting of model development, and lack of external or temporal validation. In addition, many studies report only discrimination, while calibration and decision-curve analysis are presented less consistently. The present study does not solve the problem of external validation. Instead, it provides a single-center development and internal validation study using routinely available clinical, laboratory, and imaging variables. The intended use case is early risk stratification after admission and initial aneurysm treatment, before the main DCI risk window. The model may help clinicians identify patients who need closer neurological monitoring, vascular imaging surveillance, correction of physiological abnormalities, and early review by the neurocritical care team. It is not intended to replace clinical judgment, diagnose DCI, or guide treatment without external validation and clinical impact testing. Previous studies have proposed several models for predicting DCI after aSAH, but many have been limited by small sample sizes, incomplete reporting of calibration, limited comparisons between modeling methods, or a lack of external validation. The present study does not claim to provide a clinically validated decision tool. Instead, it aims to develop and internally validate a prediction model using routinely available early clinical, laboratory, and imaging variables from a single-center cohort. The intended use case is early risk stratification after admission and initial aneurysm treatment, before the main DCI risk period. In this setting, a high predicted risk would support closer neurological assessment, vascular monitoring, correction of physiological abnormalities, and early review by neurocritical care. The model is not intended to diagnose DCI or replace clinician judgment.

Protocol

The study was approved by the Institutional Ethics Committee of Yulin First Hospital, Shaanxi Province, China. Because this was a retrospective analysis of routinely collected hospital data, and because all data were anonymized before analysis, the requirement for written informed consent was waived. The study was conducted in accordance with institutional data-protection requirements and the principles of the Declaration of Helsinki.

Data and Methods
Study Population and Design

This was a retrospective single-center cohort study conducted at Yulin First Hospital, Shaanxi Province, China. The hospital records of patients admitted with aSAH between July 2022 and December 2024 were screened. Eligible patients were adults aged 18 years or older with a diagnosis of aSAH confirmed by cranial and intracranial vascular imaging. Patients were required to have been admitted within 72 h after symptom onset and to have undergone aneurysm treatment during the index hospitalization.

Patients were excluded if they had traumatic subarachnoid hemorrhage, non-aneurysmal subarachnoid hemorrhage, severe pre-existing hematological disease, other major intracranial disease affecting outcome assessment, insufficient records for DCI evaluation, or in-hospital death before an adequate DCI assessment period. Excluding early in-hospital deaths may introduce survivorship bias because the most severe cases may be removed from analysis. Therefore, the findings should be interpreted as applying mainly to patients who survived long enough for DCI evaluation.

Data Collection 

Clinical, laboratory, imaging, and treatment-related variables were extracted from the electronic medical record using a predefined data-extraction form. The extracted variables included demographic factors, medical history, aneurysm features, early neurological grade, early imaging findings, laboratory results, treatment approach, and hospital complications. The intended prediction time point was after completion of the initial admission and perioperative assessment, but before the main DCI risk window.

Age, sex, body mass index, smoking status, alcohol use, hypertension, diabetes, aneurysm size, aneurysm location, modified Fisher grade, Hunt-Hess grade, World Federation of Neurological Surgeons grade, intraventricular hemorrhage, and rebleeding were taken from admission records and initial imaging. Laboratory variables, including hemoglobin, serum albumin, and serum sodium, were obtained from the earliest available blood test prior to the prediction time point. Cerebral edema was assessed using baseline or immediate post-treatment imaging available before DCI diagnosis. Variables recorded only after DCI onset were not used for model development to reduce information leakage.

Anemia was defined as hemoglobin <110 g/L in female patients and <120 g/L in male patients. Hypoalbuminemia was defined as serum albumin <35 g/L. Hyponatremia was defined as serum sodium <130 mmol/L.

Patients were managed according to the institutional aSAH care pathway. Standard management included early aneurysm securing, neurological observation, blood pressure control, fluid and electrolyte management, and surveillance for complications including hydrocephalus, rebleeding, seizure, infection, vasospasm, and DCI. Nimodipine was used unless contraindicated. Vasospasm monitoring was performed using neurological examination, vascular imaging, and transcranial Doppler ultrasonography when clinically indicated. Hydrocephalus was managed with cerebrospinal fluid diversion when required. Hemodynamic optimization and rescue therapy were applied according to the treating team’s assessment. Because treatment intensity and rescue-therapy details were not completely captured in the retrospective dataset, these factors could not be fully adjusted in the model and were considered a limitation.

Diagnosis of DCI 

DCI was defined as a new focal neurological impairment, a decrease in level of consciousness, or a new cerebral infarction on follow-up computed tomography or magnetic resonance imaging occurring during the expected DCI risk window and not explained by another cause. The DCI risk window was defined as days 3–14 after the hemorrhage ictus. Alternative explanations, including rebleeding, hydrocephalus, seizure, infection, metabolic disturbance, sedative effect, and procedure-related infarction, were reviewed before classifying an event as DCI.

Outcome adjudication was performed by two clinicians with experience in the management of aSAH, including one neurosurgeon and one neurologist/neurocritical care physician. The adjudicators reviewed clinical notes, neurological examinations, imaging reports, and follow-up imaging. They were blinded to the final model output during outcome assessment. Disagreements were resolved through consensus discussion with review of the relevant imaging and clinical timeline. Formal inter-rater agreement was not measured, and this limitation was added because DCI classification may involve clinical judgment.

Sample Size Calculation 

The sample size was estimated using the 10-events-per-variable (EPV) rule of thumb¹¹. With 20 candidate predictor variables and an expected DCI incidence of approximately 35%, a minimum sample size of 20 × 10 / 0.35 ≈ 572 patients was required. Our final cohort of 680 patients exceeded this requirement. Although the events-per-variable (EPV) rule was applied as a general guideline, modern predictive modeling, particularly machine learning algorithms, may require more flexible sample-size considerations. The final cohort size was considered adequate to ensure model stability, minimize the risk of overfitting, and enable reliable estimation of model performance metrics.

Model development and validation 

Patients were allocated to a training cohort of 544 patients and a hold-out internal validation cohort of 136 patients using an outcome-stratified 8:2 split. Outcome stratification was used to maintain a similar proportion of DCI events in the two cohorts. DCI occurred in 175 of 544 patients in the training cohort and 42 of 136 patients in the internal-validation cohort. The 8:2 hold-out design was retained because it was the prespecified model-development strategy and provided a separate cohort for preliminary internal performance assessment.

Predictor selection was performed only in the training cohort using least absolute shrinkage and selection operator regression with 10-fold cross-validation. The selected penalty parameter was λ = 0.031. Six predictors were retained: age, cerebral edema, hypoalbuminemia, modified Fisher grade, Hunt-Hess grade, and World Federation of Neurological Surgeons grade.

Logistic regression, extreme gradient boosting, light gradient boosting machine, support vector machine, and k-nearest neighbor models were developed using the same set of selected predictors. All preprocessing, predictor selection, and model-development procedures were restricted to the training cohort. The internal-validation cohort was not used during predictor selection or model tuning and was evaluated only after model development.

The numerical random seed used for the original cohort allocation was not retained in the archived analysis records and could not be retrospectively verified. Bootstrap optimism correction and repeated k-fold internal validation were also not available. Therefore, the present analysis is described as a single hold-out internal validation rather than an optimism-corrected or repeated cross-validated performance assessment.

Machine-learning reproducibility

LASSO predictor selection used 10-fold cross-validation within the training cohort. Continuous predictors were standardized when required by the modeling algorithm, and categorical predictors were represented using predefined binary clinical categories. The validation cohort was not used for tuning or model selection.

The archived analysis materials did not retain the final tuning grids, selected hyperparameters, complete package versions, original random seed, or a confirmed class-imbalance procedure for XGBoost, LightGBM, SVM, and KNN. These settings were therefore not reconstructed or estimated. The machine-learning comparisons should consequently be interpreted as exploratory comparisons of algorithms rather than fully reproducible model-development experiments. Reproducibility information, both available and unavailable, should be summarized in Supplementary Table 1.

Bias Control & Sensitivity Analysis

Consecutive eligible patients were included to reduce selection bias. Standardized variable definitions and a structured data-extraction form were used to reduce information bias. Predictors were restricted to information available before the intended DCI prediction time point to reduce information leakage. Predictor selection, preprocessing, and model development were performed only in the training cohort.

Modified Fisher grade, Hunt-Hess grade, and WFNS grade represent related aspects of hemorrhage burden and neurological severity. Although all three variables were retained after LASSO selection, the numerical variance inflation factor values and the results from a sensitivity model excluding overlapping severity scales were not included in the archived analysis output. These analyses could not be reconstructed from the aggregate tables because patient-level correlations and fitted model outputs are required. Therefore, the individual coefficients of these severity measures were not interpreted as independent or causal effects. They were retained only as components of the prediction model selected in the training cohort.

Model Interpretability

Clinical applicability was believed to require model interpretability as a crucial element. In the most effective model, the effects of predictors were considered to assess the clinical plausibility and conformity with existing pathophysiological knowledge of delayed cerebral ischemia. This method enabled a clear assessment of model behavior and improved the potential for clinical translation.

Missing Data

Variable-level missingness was assessed during data preparation. However, the original pre-imputation missing counts and percentages for each candidate variable were not retained in the archived analysis records and could not be accurately reconstructed from the aggregate tables. Consequently, no assumption of complete data was made, and no unverified missingness percentages were reported.

A verified supplementary missing-data table should be generated directly from the original deidentified patient-level dataset. Supplementary Table 2 should report, for every candidate variable, the number and percentage of missing observations before data handling, the method used to address missingness, and the number of observations included in the final analysis. The inability to recover the original variable-specific missingness pattern was considered a limitation of the present report.

Statistical Analysis 

Predictor selection was performed in the training cohort using LASSO regression with 10-fold cross-validation. Discrimination was assessed using the area under the receiver operating characteristic curve with 95% confidence intervals. Calibration was described using calibration plots and point estimates of the calibration slope, calibration intercept, and Brier score. Bootstrap confidence intervals for calibration measures were not available.

Threshold-dependent classification measures were calculated from the available internal-validation confusion matrices. The exact numerical probability threshold used to generate the archived confusion matrices was not retained and could not be verified. Therefore, these measures were interpreted descriptively and were not used as the primary basis for model selection. Decision-curve analysis was also considered exploratory because the exact prespecified threshold range and patient-level net-benefit output were not retained. A two-sided P value below 0.05 was considered statistically significant.

Results

Comparison of baseline characteristics between training and internal-validation cohorts

A total of 680 patients with aSAH were included. The training cohort contained 544 patients, of whom 175 developed DCI, while the internal-validation cohort contained 136 patients, of whom 42 developed DCI. The DCI event rate was 32.2% in the training cohort and 30.9% in the internal-validation cohort. Baseline characteristics are presented in Table 1. Because some archived...

Discussion

This study developed and internally validated a prediction model for DCI after aSAH using early clinical, laboratory, and imaging variables. Six variables were retained after predictor selection: age, cerebral edema, hypoalbuminemia, modified Fisher grade, Hunt-Hess grade, and World Federation of Neurological Surgeons grade. Among the evaluated models, logistic regression showed stable internal discrimination and acceptable calibration. Because the validation cohort was drawn from the same center and time as the training...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Electronic medical-record system	Yulin First Hospital	Not applicable	Source of retrospective clinical data; exact commercial platform not retained in the archived records.
IBM SPSS Statistics	IBM Corporation	Version 25.0	Descriptive and inferential statistical analysis.
K-nearest neighbor implementation	Package/source not retained	Not available	Exploratory machine-learning model; exact package and version must be recovered from the original code environment.
LightGBM	Microsoft / open-source project	Version not available	Exploratory gradient-boosting model; exact package version and hyperparameters were not retained.
R statistical software	R Foundation for Statistical Computing	Version 4.3.2	Statistical modeling and internal performance assessment.
Support vector machine implementation	Package/source not retained	Not available	Exploratory machine-learning model; exact package, kernel settings, and version were not retained.
XGBoost	Open-source project	Version not available	Exploratory gradient-boosting model; exact package version and hyperparameters were not retained.