An R-Based Landscape Validation of a Competing Risk Model
September 16th, 2022
The present protocol describes codes in R for evaluating the discrimination and calibration abilities of a competing risk model, as well as codes for the internal and external validation of it.
Our protocol presents different methods of evaluation and validation of competing risk nomogram;which considers the presence of competing events in survival analysis. This protocol serves as a supplemental to the risk regression package in R.Such as the calculation of C-index and the internal validation, using bootstrap method. To begin for C-index discrimination, fit the matrix cob into the competing risk model mod CRR and get a predicted matrix SUV by executing the command.
Get the cumulative incidences in a certain month from SUV and calculate the C-index with the function R-core sends. For AUC discrimination, score the predictive performance of the competing risk model, using the function score, risk regression package. Then extract the AUC from the score by executing the command.
To obtain calibration curves with a 95%confidence interval of the competing risk model, get a data frame with the cumulative incidences of each individual in a certain failure time. Then divide the cohort according to the estimated cumulative incidences into five subgroups and calculate the average predicted cumulative incidences of each subgroup. Calculate the observed cumulative incidences that is the actual cumulative incidences using the function cuminc.
And then get the observed cumulative incidences with a 95%confidence interval in a certain failure time by executing the command. Plot the calibration curve with the predicted cumulative incidences as the X axis and the observed cumulative incidences as the Y axis. Using the function gg plot.
For calibration curve with risk scores of the competing risk model, valuate each level of all the variables and obtain the total RS by executing the command. Count the frequencies and calculate the observed cumulative incidences of the different total risk scores. Set the range of the X axis and calculate the predicted cumulative incidences of the total risk scores.
Then plot the calibration curve with risk scores by executing the command. To get the average predicted cumulative incidences using the bootstrap method, re-sample the original dataset, dataset with replace, to generate the bootstrap dataset. Dataset in.
Then establish a competing risk model:mod NCRR with the bootstrap dataset and use the function predict CRR to predict mod NCRR in loop B times to generate SUV all in. Next, get the average predicted cumulative incidences in a certain month. Calculate the C-index using interval cross validation with the function R-core sends.
For calibration, using external validation, get the predicted cumulative incidences using external data and cumulative incidences with the matrix of external data variables:Code-X by executing the command. Then calculate the C-index using external validation By executing the command. Two nomograms were obtained using the direct method and the weighted method, demonstrating that the points of each level of variables and the probabilities corresponding to the total points were almost the same.
While some slight differences were observed. The calibration curve for the competing risk model was close to the equivalence line and the 95%confidence interval of the observed frequency, fell into the equivalence line in each group. Indicating the accurate calibration ability of the model.
Calibration curves using internal and external validation were shown, indicating that the constructed model had a good calibration ability in the internal validation;but a poor one in the external validation. The results of the decision curve analysis of the competing risk nomogram were obtained. Demonstrating the changes in net benefit with increasing threshold probability.
Re-sampling the original dataset was replaced to generate the bootstrap dataset as important, in performing the internal validation of the competing risk nomogram. Besides the bootstrap method, the randomized display and K4 method can also be performed to generate data sets used for internal validation. Using our R based landscape validation of the competing risk model, clinicians can perform a prognosis analysis in real world considering the competing risk more easily.
View the full transcript and gain access to thousands of scientific videos
Overview
This protocol presents methods for evaluating and validating competing risk models in survival analysis. It includes R codes for assessing discrimination and calibration abilities, as well as internal and external validation techniques.
Key Study Components
Area of Science
- Survival Analysis
- Competing Risks
- Statistical Modeling
Background
- Competing risk models account for the presence of competing events in survival data.
- Evaluation metrics like C-index and AUC are crucial for model performance assessment.
- Bootstrap methods are commonly used for internal validation.
- This protocol supplements the risk regression package in R.
Purpose of Study
- To provide a comprehensive guide for evaluating competing risk nomograms.
- To enhance understanding of model calibration and discrimination.
- To facilitate the application of statistical methods in survival analysis.
Methods Used
- Calculation of C-index for model discrimination.
- Use of bootstrap methods for internal validation.
- Extraction of AUC for assessing predictive performance.
- Implementation of R commands for model fitting and evaluation.
Main Results
- Successful calculation of cumulative incidences from the predicted matrix.
- Demonstrated methods for scoring and validating competing risk models.
- Provided clear R code examples for reproducibility.
- Highlighted the importance of model evaluation in survival analysis.
Conclusions
- The protocol offers valuable tools for researchers in survival analysis.
- It emphasizes the need for rigorous evaluation of competing risk models.
- Future applications can build on these methods for improved model accuracy.
Frequently Asked Questions
Executive Industry Relevance
Competing risk models, validated through robust R-based workflows, address the limitations of traditional survival analysis by quantifying multiple event outcomes in clinical datasets. This approach enhances predictive confidence and supports risk-adjusted decision-making at critical discovery and translational inflection points. Reliable model calibration and discrimination metrics enable biopharma teams to prioritize targets and stratify patient populations with greater precision.
Strategic Applications in Biopharma R&D
Early Discovery & Target Validation
- Enables rigorous hypothesis testing by accounting for competing clinical outcomes in survival data.
- Supports functional target validation through quantitative discrimination and calibration metrics.
- Facilitates predictive confidence for early portfolio triage and mechanistic de-risking.
Screening & Assay Development
- Prepares validated statistical models for downstream translational workflows.
- Standardizes model evaluation using C-index, AUC, and calibration curves for reproducibility.
- Enables scalable, quantitative assessment of prognostic biomarkers and risk factors.
Translational & Preclinical Research
- Aligns model outputs with real-world clinical endpoints for translational continuity.
- Supports risk-adjusted advancement decisions by quantifying net benefit via decision curve analysis.
- Provides a framework for external validation, ensuring model generalizability across datasets.
Pipeline & Workflow Integration
This R-based validation protocol integrates from early discovery through translational research, supporting lead identification and preclinical model selection when competing risks are present.
- Discovery Biology: Quantifies the impact of competing events, clarifying biological pathways and reducing mechanistic ambiguity.
- Screening: Delivers reproducible, quantitative outputs (C-index, AUC) for model comparison and assay readiness.
- Analytics: Provides calibration curves and decision curve analysis to benchmark predictive performance.
- Translational Research: Enables external validation for continuity between discovery and clinical datasets.
- Enterprise Reuse: Establishes a reusable statistical validation framework for diverse survival analysis applications.
Operational & Enterprise Impact
- Scientific Value: Increases predictive confidence and target validation by integrating competing risk analysis.
- Operational Value: Standardizes model evaluation and validation for reproducibility and scalability.
- Strategic Value: Informs go/no-go decisions and reduces late-stage biological risk through robust statistical outputs.
- Portfolio Impact: Supports risk-adjusted prioritization and advancement of assets based on validated prognostic models.
Implementation Considerations
- Requires expertise in R and survival analysis statistical methods.
- Depends on access to robust computational infrastructure for bootstrap and external validation workflows.
- Necessitates cross-team standardization of model evaluation metrics and reporting.
- Adaptation may be needed for different disease models or clinical endpoints.
- External validation performance may vary based on dataset heterogeneity and event rates.
Biopharma FAQs
Chapters in this video
0:04
Introduction
0:35
Discrimination Ability of the Competing Risk Nomogram
1:17
Calibration Ability of Competing Risk Models
2:42
Internal Validation Using the Bootstrap Method and External Validation of the Competing Risk Model
3:45
Results: Codes for Evaluating the Discrimination and Calibration Abilities of a Competing Risk Model
4:44
Conclusion
