Constructing and Visualizing Models using Mime-based Machine-learning Framework

Summary

Mime is a flexible computational framework to construct a machine learning-based integration model with elegant performance. Here, we provide a detailed step-by-step procedure for developing predictive models with high accuracy, leveraging complex datasets to identify critical genes associated with disease progression, patient outcomes, and therapeutic response.

Abstract

The widespread high-throughput sequencing technology has significantly enhanced our understanding of biology and cancer heterogeneity. Machine learning algorithms on transcriptional data have become vital for predicting patient prognosis and clinical responses. Despite advancements in machine learning algorithms, an open-source platform that incorporates the most sophisticated machine learning algorithms on transcriptional data remains absent. To address this gap, we developed Mime, a versatile machine-learning framework to enhance the construction and visualization of predictive models for clinical characteristics and gene signatures. By integrating diverse datasets and employing the most advanced feature selection techniques, Mime addresses critical challenges in clinical predictions. It provides three main functions, including model construction, feature selection, and data visualization. Model construction encompasses a range of machine learning algorithms, including but not limited to decision trees, support vector machines, and ensemble methods, allowing researchers to select the best-fitted approach for their specific analysis. Feature selection utilizes advanced algorithms such as Recursive Feature Elimination and LASSO regression to streamline the dataset and focus on the most informative features. The framework supports customizable parameter tuning through cross-validation methods, optimizing model performance while mitigating overfitting risks. Visualization tools integrated within Mime enable researchers to interpret model outcomes effectively, providing graphical representations of feature importance and predictive performance metrics. In this manuscript, we provide a detailed tutorial on the stepwise procedures of this versatile machine-learning framework.

Introduction

The widespread adoption of high-throughput sequencing technologies has significantly influenced our understanding of biology and cancer heterogeneity¹. This groundbreaking advancement in biotechnology has not only deepened our scientific knowledge but has also revolutionized the field of medical research. By enabling scientists to sequence large amounts of genetic material quickly and accurately, high-throughput sequencing has accelerated the discovery of new genes, mutations, and biological pathways. A growing body of research has delineated specific molecular signatures associated with disease progression, patient prognosis, and therapeutic responsiveness from sequencing data^2,3,4. These specific signatures offer a comprehensive landscape of understanding the transcriptional regulatory network underlying tumor biology, including tumor origin, differentiation, migration, and treatment resistance⁵. These features are often diverse and varied, encompassing multiple facets rather than being confined to a single exhibit. This makes it difficult to screen and identify specific genes highly associated with the disease. Thus, there is an urgent need for sensible computational strategies to screen crucial genes implicated in disease.

Machine learning (ML) is a branch of artificial intelligence that focuses on building systems that can learn from complex datasets, identify patterns, and develop a predictive model with high accuracy to provide a reference for making decisions⁶. Recently, the development of machine learning-based models from transcriptome data to predict patient outcomes or diagnose diseases has advanced rapidly in a variety of diseases^7,8,9,10. The performance of these models often varies greatly due to the different data sets and algorithms used for training. Selecting the optimal model for subsequent experiments and clinical applications has emerged as a pressing challenge. A viable approach entails comparing the performance of various models and selecting the most promising one for further utilization^7,11. Furthermore, the diversity in parameters and result formats supported across various computing frameworks poses significant challenges for comparative analysis. However, there is currently no software that integrates cutting-edge machine learning algorithms to compare the strengths and weaknesses of different models and simplify the parameter selection process, making it easier for users to access. Thus, there is a need for the development of user-friendly software that can facilitate the integration of transcriptional data with diverse machine learning algorithms, while also addressing the current limitations of biomarker selection and model interpretation. Such advancements will greatly expand our ability to provide valuable insights into current research fields and ultimately improve patient outcomes.

In this study, we developed an open-source R package named Mime¹², which demonstrates robust performance across datasets and aims to streamline the integration of transcriptome datasets with various machine learning algorithms, thereby simplifying the associated processes. To identify potential candidates from large-scale transcriptome data and develop optimal models, Mime offers four application functions: an optimal prognosis model constructed by integrating 10 machine learning algorithms; a binary response model constructed using seven machine learning algorithms; core features related to prognosis identified using eight machine learning algorithms; and visualization of the performance of each model.

Protocol

NOTE: The tutorials for this study are all run on the Linux platform using R software. The version of the R package used in this protocol is listed in the Table of Materials. Each step necessary for analysis are shown below and detailed protocol can also be acquired on GitHub (https://github.com/l-magnificence/Mime). Users who encounter issues with Mime can visit GitHub Issue page (https://github.com/l-magnificence/Mime/issues) to provide feedback.

1. Preparation of Mime and example dataset

1. Install the development version of Mime from GitHub using the code below:
   devtools::install_github("l-magnificence/Mime")
2. Prepare multiple cohorts containing transcriptional sequencing data with information on survival or clinical response to therapy. Here, two example data (Example.cohort and Example.ici) was used for running Mime. Example.cohort contains two glioma datasets; each randomly selected 100 samples from the TCGA and CGGA database, respectively. While Example.ici contains randomly selected 100 pre-treatment samples with immune checkpoint inhibitors from previous study¹². All the example data can be acquired from GitHub (https://github.com/l-magnificence/Mime/tree/main/External%20data)​
   1. Include multiple datasets to construct predictive models for prognosis in Example.cohort. The first column ID in each dataset is the sample ID, the second and third columns OS.time and OS in each dataset are the survival time and status of patients; other columns in each dataset are the gene expression level scaled with log2(x+1). Dataset1 is the training dataset, while other datasets are for validation. Use the following format for Example.cohort:
      load ("./Example.cohort.Rdata" )
list_train_vali_Data [["Dataset1" ]][1:5,1:5]
#> ID OS.time OS MT-CO1 MT-CO3
#> TCGA.DH.A66B.01 1281.65322 0 13.77340 13.67931
#> TCGA.HT.7607.01 96.19915 1 14.96535 14.31857
#> TCGA.DB.A64Q.01 182.37755 0 13.90659 13.65321
#> TCGA.DU.8167.01 471.97707 0 14.90695 14.59776
#> TCGA.HT.7610.01 1709.53901 0 15.22784 14.62756
   2. Include multiple datasets to construct predictive models for the response in Example.ici. The first column ID in each dataset is the sample ID, the second column Var in each dataset is the therapeutic response of patients (N: No response; Y: response), and the other columns in each dataset are the gene expression level scaled with log2(x+1). Training is the training dataset, while other datasets are for validation. Use the following format for Example.ici:
      load("./Example.ici.Rdata")
list_train_vali_Data[["training"]][1:5,1:5]
#> ID Var FTH1 EEF1A1 ACTB
#> SAMf2ce197162ce N 10.114846 4.817746 11.230180
#> ERR2208915 Y 2.044180 5.038854 3.977902
#> G138701_RCCBMS Y 5.406008 5.341635 5.366668
#> SAMe41b1e773582 N 9.215794 4.707360 11.412721
#> SAM5ffd7e4cd794 N 9.003710 3.908884 10.440559
3. Prepare a gene set. Here, a gene set (genelist) associated with Wnt/β-catenin signaling from MSigDB was used for running Mime. Use the following format for genelist:
   load ("./genelist.Rdata" )
#> [1] "MYC" "CTNNB1" "JAG2" "NOTCH1" "DLL1" "AXIN2" "PSEN2" "FZD1" "NOTCH4" "LEF1" "AXIN1" "NKD1" "WNT5B"
#>[14] "CUL1" "JAG1" "MAML1" "KAT2A" "GNAI1" "WNT6" "PTCH1" "NCOR2" "DKK4" "HDAC2" "DKK1" "TCF7" "WNT1"
#>[27] "NUMB" "ADAM17" "DVL2" "PPARD" "NCSTN" "HDAC5" "CCND2" "FRAT1" "CSNK1E" "RBPJ" "FZD8" "TP53" "SKP2"
#>[40] "HEY2" "HEY1" "HDAC11"

2. Constructing predictive models for prognosis

1. Use the function ML.Dev.Prog.Sig() in Mime based on Example.cohort and genelist to construct predicting models for prognosis. Use the following codes:
   library (Mime1)
load ("./Example.cohort.Rdata" )
load ("./genelist.Rdata" )
res <- ML.Dev.Prog.Sig (train_data = list_train_vali_Data $Dataset1,
list_train_vali_Data = list_train_vali_Data,
unicox.filter.for.candi = T,
unicox_p_cutoff = 0.05,
candidate_genes = genelist,
mode = 'all',nodesize =5,seed = 5201314 )
2. Use function cindex_dis_all() in Mime to plot the C-index of each model and select the optimal model with the highest C-index. Use the following codes:
   cindex_dis_all (res,validate_set = names (list_train_vali_Data )[-1 ],
order =names (list_train_vali_Data ),width = 0.35 )
3. Calculate the survival curve of patients according to risk score using a specific model among different datasets and process that in Mime. Use the following codes:
   survplot <- vector ("list",2 )
for (i in c (1:2)) {
print (survplot [[i ]]<-rs_sur (res,
model_name = "StepCox[forward] + plsRcox",
dataset = names (list_train_vali_Data )[i ],
median.line = "hv",
cutoff = 0.5,
conf.int = T,
xlab ="Day",pval.coord =c (1000,0.9 )))
}
aplot ::plot_list (gglist =survplot,ncol =2 )
4. Next, use the function cal_AUC_ml_res() in Mime to calculate time-dependent AUC predicted by the constructed models. Use the following codes:
   all.auc.1y <- cal_AUC_ml_res (res.by.ML.Dev.Prog.Sig =res,
train_data = list_train_vali_Data [["Dataset1" ]],
inputmatrix.list =list_train_vali_Data,
mode = 'all',AUC_time = 1,
auc_cal_method ="KM" )
5. Use function auc_dis_all() in Mime to plot time-dependent AUC predicted by each model. Use the following codes:
   auc_dis_all(all.auc.1y,
dataset = names(list_train_vali_Data),
validate_set=names(list_train_vali_Data)[-1],
order= names(list_train_vali_Data),
width = 0.35,
year=1)
6. Process the time-dependent ROC curve of a specific model among different datasets in Mime. Use the following codes:
   roc_vis(all.auc.1y,
model_name = "StepCox[forward] + plsRcox",
dataset = names(list_train_vali_Data),
order= names(list_train_vali_Data),
anno_position=c(0.65,0.55),
year=1)

3. Constructing predictive models for response

1. Use another function ML.Dev.Pred.Category.Sig() in Mime based on Example.ici and genelist to construct predicting models for therapeutic response. Use the following codes:
   load("./Example.ici.Rdata")
load("./genelist.Rdata")
res.ici <- ML.Dev.Pred.Category.Sig(
train_data = list_train_vali_Data$training,
list_train_vali_Data = list_train_vali_Data,
candidate_genes = genelist,
methods = c('nb','svmRadialWeights','rf',
'kknn','adaboost','LogitBoost',
'cancerclass'),
seed = 5201314,
cores_for_parallel = 60
)
2. Use the function auc_vis_category_all() in Mime to plot the AUC predicted by each model. Use the following codes:
   auc_vis_category_all(res.ici,dataset = c("training","validation"),
order= c("training","validation"))
3. Process the ROC curve of specific model among different datasets in Mime. Use the following codes:
   plot_list<-list()
methods <- c('nb','svmRadialWeights','rf','kknn', 'adaboost','LogitBoost','cancerclass')
for (i in methods) {
plot_list[[i]]<-roc_vis_category(res.ici,model_name = i,
dataset = c("training","validation"),
order= c("training","validation"),
anno_position=c(0.4,0.25))
}
aplot::plot_list(gglist=plot_list,ncol=3)

4. Core feature selection

1. Use the function ML.Corefeature.Prog.Screen() in Mime based on Example.cohort and genelist to identify critical genes. Use the following codes:
   load("./Example.cohort.Rdata")
load("./genelist.Rdata")
res.feature.all <- ML.Corefeature.Prog.Screen(
InputMatrix = list_train_vali_Data$Dataset1,
candidate_genes = genelist,
mode = "all",nodesize =5,seed = 5201314 )
2. The output genes are closely associated with patient outcome and a higher frequency of variable screened means these are core feature (the most frequently filtered variables are defined as core features). Use the function core_feature_rank() in Mime to plot the rank of genes filtered by different methods. Use the following codes:
   core_feature_rank(res.feature.all, top=20)

Results

The genelist and Example.cohort including one training cohort and one validation cohort were used to construct prognositic models by integrating 10 machine learning algorithms in Mime. Among 117 prognosis models constructed by Mime, StepCox[forward] + plsRcox combined model (SPCOM) had the highest C-index among all cohorts, indicating its outstanding performance (Figure 1A). Patients were further separated into high-risk and low-risk groups according to the median risk score calculated by SP...

Discussion

In this study, we provide a detailed description of how to use the Mime package to develop robust and powerful machine-learning predictive models for transcriptomic data. In previous studies researchers often struggled with selecting the appropriate predictive model algorithm based on the specific characteristics of their sequencing data^13,14. Additionally, for researchers without a computer science background, there is a certain difficulty in stabilizing the mac...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Name of Package	Version	Software
aplot	0.1.10	R studio
BART	2.9.4	R studio
Boruta	8.0.0	R studio
cancerclass	1.38.0	R studio
caret	6.0-89	R studio
Ckmeans.1d.dp	4.3.5	R studio
compareC	1.3.2	R studio
ComplexHeatmap	2.15.1	R studio
compositions	2.0-4	R studio
data.table	1.14.0	R studio
doParallel	1.0.16	R studio
dplyr	1.1.3	R studio
e1071	1.7-7	R studio
forestploter	1.1.0	R studio
future	1.21.0	R studio
gbm	2.1.8.1	R studio
ggbreak	0.1.1	R studio
ggplot2	3.4.1	R studio
ggpubr	0.4.0	R studio
ggsci	2.9	R studio
glmnet	4.1-2	R studio
grid	4.1.3	R studio
gridExtra	2.3	R studio
GSEABase	1.54.0	R studio
GSVA	1.40.1	R studio
Hmisc	5.1-1	R studio
kknn	1.3.1	R studio
knitr	1.42	R studio
magrittr	2.7.2	R studio
Matrix	1.5-4	R studio
meta	5.2-0	R studio
miscTools	0.6-28	R studio
mixOmics	6.18.1	R studio
mixtools	1.2.0	R studio
pbapply	1.4-3	R studio
plsRcox	1.7.7	R studio
pROC	1.18.0	R studio
R	4.1.3	R studio
randomForestSRC	4.6-14	R studio
readr	1.4.0	R studio
recipes	0.1.17	R studio
reshape2	1.4.4	R studio
rmarkdown	2.8	R studio
ROCit	2.1.1	R studio
ROCR	1.0-11	R studio
scales	1.2.1	R studio
sparrow	1.0.3	R studio
stringr	1.5.0	R studio
superpc	1.12	R studio
survival	3.3-1	R studio
survivalROC	1.0.3	R studio
survivalsvm	0.0.5	R studio
sva	3.40.0	R studio
testthat	3.1.0	R studio
tibble	3.2.1	R studio
tidyr	1.3.0	R studio
tidyverse	1.3.1	R studio
UpSetR	1.4.0	R studio
viridis	0.6.1	R studio