A Reproducible Protocol for Developing and Evaluating Explainable Artificial Intelligence Frameworks in Healthcare Analytics

Summary

Here, we present a reproducible protocol for developing and evaluating explainable artificial intelligence models for healthcare analytics. The workflow integrates data preprocessing, predictive modeling, SHAP-, LIME-, and Grad-CAM-based explainability, quantitative evaluation, and human-centered validation to support transparent and trustworthy clinical decision-making.

Abstract

Explainable Artificial Intelligence (XAI) is increasingly recognized as an indispensable tool for building trustworthy AI systems in healthcare, where transparency and the ability to explain decisions are essential components of clinical decision-making. This publication presents a reproducible experimental approach to developing and evaluating explainable AI systems for healthcare analytics. The developed pipeline integrates the steps of data preprocessing, predictive modeling, interpretation generation, and evaluation into one seamless workflow that can be applied to both structured clinical data and medical imaging datasets. Ensemble machine learning models have demonstrated strong predictive performance on structured tabular datasets, whereas deep learning models are effective for learning complex patterns in medical imaging data. Techniques such as SHAP, LIME, and Grad-CAM are global and local explanations that facilitate model interpretation. Very helpful. The quantitative assessment of the framework spans many different metrics such as accuracy, precision, recall, F1 score, ROC-AUC, explanation metrics, fidelity, and stability. The results indicate that when the experimental conditions are controlled, the framework demonstrated improved predictive performance and explanation quality under the evaluated experimental conditions. As a document guided by protocol, this piece of work backs the reproducibility and scalability, the persistent implementation by other living beings. This transparent, understandable AI model is the foundation upon which clinical decision-making support and healthcare analytics systems gain trust and usage on a large scale.

Introduction

Artificial intelligence (AI) has become an important component of modern healthcare by supporting disease diagnosis, prognosis, risk stratification, medical image analysis, and clinical decision-making¹. Advances in machine learning and deep learning have enabled healthcare systems to process large volumes of structured and unstructured data, often achieving predictive performance comparable to or exceeding conventional statistical approaches². Despite these advances, many AI models operate as complex black-box systems, making it difficult for clinicians and healthcare stakeholders to understand how predictions are generated³. This lack of transparency can limit trust, adoption, and accountability in high-stakes healthcare environments^4,5.

Explainable artificial intelligence (XAI) has emerged as a promising approach for improving the transparency and interpretability of machine learning models⁶. Widely used explanation techniques, including SHAP (Shapley additive explanations), LIME (Local interpretable model-agnostic explanations), and Gradient-weighted class activation mapping (Grad-CAM), provide insights into model behavior through feature attribution and visual explanation mechanisms^7,8,9. These methods have been increasingly applied to healthcare applications to support clinical interpretation, model auditing, and decision support^10,11. However, explainability alone does not guarantee reliability, reproducibility, or clinical utility. Recent studies have highlighted challenges related to explanation instability, sensitivity to perturbations, limited clinician validation, and inconsistencies between explanation outputs and true model reasoning^12,13,14,15.

In addition to explainability challenges, reproducibility remains a significant concern in healthcare machine learning research^16,17,18. Many published studies provide limited information regarding preprocessing procedures, software environments, hyperparameter configurations, validation strategies, and implementation workflows, making independent replication difficult^19,20,21. Furthermore, healthcare AI studies frequently focus on predictive performance while providing limited guidance regarding explanation quality assessment, clinician-centered evaluation, and practical implementation considerations²². These limitations may hinder the translation of explainable AI systems from research environments to real-world healthcare settings^{23,24,25,26,27}.

Current healthcare XAI workflows often evaluate individual explanation techniques or predictive models in isolation and rarely provide standardized protocols that integrate data preparation, predictive modeling, explainability assessment, human-centered evaluation, and reproducibility resources^28,29,30,31. Consequently, researchers and practitioners may encounter difficulties when reproducing workflows, comparing explanation methods, or evaluating the practical utility of explainable AI systems across different healthcare datasets and application domains. A comprehensive and reproducible protocol is therefore needed to facilitate consistent implementation, evaluation, and reporting of healthcare explainable AI workflows^32,33,34,35.

Here, we present a reproducible protocol for developing, evaluating, and interpreting explainable artificial intelligence systems in healthcare analytics. The protocol integrates data preprocessing, predictive modeling, explainability assessment using SHAP, LIME, and Grad-CAM, clinician-centered evaluation, validation procedures, and reproducibility resources within a unified workflow. The objective is to provide a standardized framework that supports transparent model development, explanation quality assessment, and reproducible implementation across diverse healthcare datasets^36,37,38,39. The methodological contribution of this work lies in the integration of predictive analytics, explainability evaluation, clinician validation, and reproducibility practices into a single protocol suitable for healthcare AI research, education, and deployment-oriented studies^40,41,42,43.

The primary contribution of this work is not the development of a novel explainability algorithm.

Rather, it provides a standardized, reproducible, and experimentally validated protocol that integrates predictive modeling, explainability assessment, visualization, evaluation, and human-centered interpretation into a unified workflow suitable for healthcare analytics and JoVE-based protocol dissemination. The primary objective of this work is not to introduce a novel predictive algorithm but to provide a standardized, reproducible, and experimentally validated protocol for implementing explainable artificial intelligence workflows in healthcare analytics. The protocol integrates data preprocessing, predictive modeling, explainability assessment, clinician-centered evaluation, and reproducibility resources into a unified framework suitable for adoption, replication, and educational dissemination.

Protocol

All datasets used in this study were obtained from publicly available and fully anonymized repositories, including the UCI Machine Learning Repository, PhysioNet MIMIC-III database, and NIH Chest X-ray datasets. No identifiable patient information was accessed. The study complied with institutional research ethics guidelines for secondary analysis of publicly available de-identified data. Formal Institutional Review Board approval was not required because no human participants were directly recruited and no protected health information was used.

1. Overview of the Experimental Frameworks

1. Develop a reproducible, explainable artificial intelligence (XAI) pipeline for transparent healthcare analytics. Organize the workflow into the Data Layer, Preprocessing Layer, Modeling Layer, Explainability Layer, Evaluation Layer, and Visualization Layer.
2. Integrate these modules into a unified workflow capable of processing structured clinical data, electronic health records, and medical imaging datasets.
3. Ensure that each module contributes to reproducibility, interpretability, scalability, and standardized experimental execution.
4. Design the modular architecture to support continuous data streaming and ensure that each stage of the pipeline contributes to reproducibility, interpretability, and scalability throughout the experimental workflow.

2. Computational Environment and System Configuration

1. Install the required software dependencies before executing the workflow by opening a command-line terminal and running the following command:
   pip install numpy pandas scikit-learn tensorflow shap lime opencv-python matplotlib seaborn
2. Verify the successful installation of all software packages and confirm compatibility with the software versions listed in the Table of Materials before proceeding with data preprocessing and model development.
3. Use Python 3.11 as the primary programming environment. Install and configure NumPy 1.26 and Pandas 2.1 for data manipulation and feature engineering tasks. Install Scikit-learn 1.5 for machine learning model development and evaluation.
4. Install TensorFlow 2.15 for deep learning model training and inference. Install SHAP 0.46 and LIME 0.2.0 for explainability analysis. Install OpenCV 4.10 for image processing tasks. Install Matplotlib 3.9 and Seaborn 0.13 for data visualization and result reporting. Refer to the Table of Materials for complete software specifications and implementation details required for reproducibility.
5. Configure the computational environment using a workstation equipped with a multi-core processor, graphics processing unit (GPU) acceleration, and sufficient memory resources to accommodate large healthcare datasets and deep learning workloads.
6. Set fixed random seeds for data partitioning, model initialization, and training procedures to ensure reproducibility across experimental runs. Enable logging mechanisms to record data preprocessing operations, model configurations, hyperparameter settings, training procedures, and evaluation results throughout the workflow.
7. Configure the model architectures, optimization parameters, learning rates, batch sizes, training settings, and hyperparameter values according to the specifications summarized in Table 1. Adhere to these settings during replication experiments to ensure consistency with the reported results.
8. Expected Output- A fully configured and reproducible computational environment with all required software packages, hardware resources, logging mechanisms, and model configuration settings available for subsequent data preprocessing, model development, explainability analysis, and evaluation procedures.

Component	Parameter	Value	Description
Random Forest	n_estimators	100	Number of trees
Random Forest	max_depth	None	Tree depth
XGBoost	learning_rate	0.01	Learning rate
XGBoost	n_estimators	100	Boosting rounds
CNN	epochs	25	Training epochs
CNN	batch_size	32	Batch size
CNN	optimizer	Adam	Optimization algorithm
MLP	hidden_layers	2	Number of hidden layers
MLP	activation	ReLU	Activation function
General	train_split	70%	Training data ratio
General	validation_split	15%	Validation ratio
General	test_split	15%	Testing ratio

Table 1: Implementation Details and Hyperparameter Configuration.

This table summarizes the computational environment, software libraries, machine learning and deep learning model configurations, optimization settings, learning rates, batch sizes, training parameters, and hyperparameter values used throughout the experimental evaluation of the proposed explainable AI framework.

3. Dataset Preparation and Preprocessing

1. Select publicly available healthcare datasets to ensure transparency and reproducibility of the experimental workflow.;
2. Use four representative healthcare scenarios to validate the proposed framework: (i) breast cancer diagnosis using the Wisconsin Breast Cancer Dataset, (ii) diabetes risk prediction using the Pima Indians Diabetes Dataset, (iii) clinical outcome prediction using MIMIC-III electronic health records, and (iv) thoracic disease classification using the NIH Chest X-ray Dataset. Choose these datasets to evaluate the framework across structured clinical records, longitudinal electronic health records, and medical imaging modalities commonly used in healthcare decision-support systems.
3. Import each dataset into the Python environment and separate the records into input features and target variables. Organize structured clinical data for disease prediction tasks, electronic health records for longitudinal outcome analysis, and medical imaging datasets for diagnostic classification tasks. Verify the integrity of each dataset before proceeding with preprocessing operations.
4. Identify and address missing values using appropriate imputation techniques. Standardize numerical features through feature scaling and normalization procedures to ensure comparability across variables.
5. Encode categorical variables using suitable encoding methods based on dataset characteristics. Perform feature selection using correlation analysis and model-based feature importance measures to identify the most relevant variables and reduce data dimensionality.
6. Resize all medical images to 224, 224 pixels prior to model training. Normalize pixel intensity values according to the requirements of the selected deep learning architecture. Apply data augmentation techniques, including image rotation and horizontal flipping, to improve model generalization and reduce overfitting. Verify image quality and ensure consistency of image dimensions throughout the dataset.
7. Assess data integrity by evaluating dataset completeness, consistency, and feature-label alignment. Verify that all records are correctly assigned to their corresponding target classes. Review dataset characteristics, including sample size, feature count, and data modality, as summarized in Table 2.
8. Implement dataset-specific validation procedures to ensure methodological rigor and reproducibility. Record the sample size, class distribution, train-validation-test split strategy, leakage prevention measures, hyperparameter optimization procedures, cross-validation design, and external testing protocol for each dataset. Summarize these validation procedures in Table 3.
9. Perform preprocessing quality-control checks by assessing missing-value handling, outlier removal, feature scaling, categorical encoding, class distribution preservation, and dataset partitioning procedures. Verify that preprocessing operations are applied consistently across all datasets.
10. Confirm the absence of substantial data leakage during dataset partitioning. Review the preprocessing validation results presented in Figure 1 to ensure that standardized datasets have been generated for subsequent machine learning and explainability analyses.
11. Expected Output- Generate cleaned and standardized datasets with missing values appropriately addressed, categorical variables encoded, numerical features normalized, and records partitioned into training, validation, and testing subsets. Ensure that dataset characteristics, class distributions, and validation procedures correspond to those reported in Table 2 and Table 3, and confirm successful preprocessing validation as illustrated in Figure 1.

Dataset	Type	Samples	Features	Target
Breast Cancer	Tabular	569	30	Benign/Malignant
Diabetes	Tabular	768	8	Diabetes Outcome
MIMIC-III	EHR	~60,000	Clinical Variables	Mortality
Chest X-ray	Imaging	~112,000	Images	Disease Labels

Table 2: Dataset Characteristics and Healthcare Use Cases.

This table provides a summary of the healthcare datasets used in the study, including dataset type, number of samples, feature count, target variables, healthcare application domain, and associated clinical use cases. The datasets encompass structured clinical records, electronic health records, and medical imaging data to demonstrate framework applicability across diverse healthcare analytics scenarios.

Dataset	Sample Size	Class Distribution	Split Strategy	Leakage Prevention	Hyperparameter Search	Cross Validation	External Test
Breast Cancer (UCI Wisconsin)	569	357 Benign / 212 Malignant	70/15/15	Train-only preprocessing	Grid Search	5-Fold Stratified CV	Independent Holdout Set
Pima Indians Diabetes	768	500 Non-Diabetic / 268 Diabetic	70/15/15	Train-only preprocessing	Grid Search	5-Fold Stratified CV	Independent Holdout Set
MIMIC-III EHR	5274	Outcome-stratified cohorts	70/15/15 Patient-level	Patient-level separation	Random Search	5-Fold Patient-level CV	Independent Holdout Set
NIH Chest X-ray	112120	Pneumonia/Normal stratified	70/15/15	Image-level separation	Random Search	5-Fold Stratified CV	Independent Holdout Set

Table 3: Dataset-Level Validation and Experimental Design.

This table summarizes the validation methodology employed for each dataset, including sample size, class distribution, train-validation-test split strategy, leakage prevention procedures, hyperparameter optimization methods, cross-validation design, and external testing protocols. These measures were implemented to ensure methodological rigor, reproducibility, and unbiased model evaluation.

Figure 1: Validation of Data Preprocessing Pipeline.
Validation results for key preprocessing operations performed before model development. (A) Missing-value analysis across representative healthcare features. (B) Distribution of a numerical variable before and after outlier handling. (C) Feature scaling validation using standardization. (D) Categorical encoding validation. (E) Class distribution following preprocessing. (F) Validation of train-validation-test data partitioning. These results confirm successful preprocessing and preparation of the datasets for predictive modeling and explainability analysis. Please click here to view a larger version of this figure.

4. System Architecture of the Explainable AI Framework

1. Organize the framework using a layered architecture to facilitate modular processing, improve workflow transparency, and support reproducible implementation. Configure the Data Layer to receive and manage raw healthcare datasets. Route all incoming datasets through the Data Layer before initiating preprocessing operations.
2. Perform data cleaning, transformation, normalization, feature engineering, and encoding procedures within the Preprocessing Layer. Ensure that all preprocessing operations are completed before model development.
3. Develop predictive models within the Modeling Layer using machine learning and deep learning algorithms. Train Gradient Boosting, Random Forest, Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN) models using the preprocessed datasets and optimized hyperparameter configurations.
4. Apply explainability techniques within the Explainability Layer to interpret model predictions. Generate feature-attribution explanations using SHAP and LIME for structured datasets. Generate Grad-CAM heatmaps for image-based models to visualize regions contributing to model predictions.
5. Evaluate predictive and explainability performance within the Evaluation Layer. Calculate accuracy, precision, recall, F1-score, ROC-AUC, fidelity, stability, and clinician-centered evaluation metrics. Record all evaluation outputs for subsequent analysis and reporting.
6. Generate clinically interpretable visual outputs within the Visualization Layer. Create performance comparison plots, feature-importance visualizations, SHAP summary plots, LIME explanations, Grad-CAM heatmaps, and clinician-oriented reporting dashboards. Store all visual outputs for inclusion in the final experimental report.
7. Review the complete framework architecture illustrated in Figure 2 before proceeding with workflow execution.
8. Expected Output- Generate fully trained machine learning and deep learning models, including Gradient Boosting, Random Forest, CNN, and MLP architectures. Store model configurations, optimized hyperparameters, training logs, checkpoint files, explainability outputs, and visualization artifacts for subsequent evaluation and interpretability analysis.

Figure 2: System Architecture of the Explainable AI Framework for Healthcare Analytics. Please click here to view a larger version of this figure.

5. Workflow Execution

1. Acquire healthcare datasets from publicly available repositories and store the datasets in structured formats suitable for machine learning and deep learning analysis. Review the complete workflow illustrated in Figure 3 before initiating the experimental procedure.
2. Perform data cleaning and preprocessing according to the procedures described in Section 3. Normalize numerical variables using appropriate scaling methods. Encode categorical variables using suitable encoding techniques. Identify and impute missing values using dataset-specific imputation strategies. Verify data quality, feature-label alignment, and dataset completeness before proceeding with model development.
3. Partition each dataset into training, validation, and testing subsets to support unbiased model evaluation. Preserve class distributions during dataset partitioning and implement leakage-prevention measures where applicable. Reserve the testing subset exclusively for final model evaluation.
4. Train machine learning models on structured datasets using ensemble-based algorithms, including Gradient Boosting and Random Forest. Train deep learning models on imaging datasets using Convolutional Neural Network (CNN) architectures capable of learning spatial image features. Update model parameters iteratively during training and monitor validation performance after each training epoch. Apply early stopping when validation performance deteriorates or fails to improve according to the predefined stopping criteria.
5. Optimize model hyperparameters using systematic search strategies, including grid search and randomized search. Adjust learning rate, number of estimators, tree depth, batch size, number of epochs, and other model-specific parameters according to validation performance. Select the final model based on predictive performance and generalization capability across validation datasets.
6. Apply explainability methods after completing model training. Generate global explanations using feature-attribution techniques to identify variables contributing most strongly to model predictions. Generate local explanations to analyze individual prediction cases and evaluate model behavior at the sample level. Create Grad-CAM heatmaps for image classification models to highlight regions of the image that contribute to the predicted outcome. Store all explanation outputs for subsequent analysis and reporting.
7. Evaluate predictive performance using accuracy, precision, recall, F1-score, and receiver operating characteristic area under the curve (ROC-AUC). Assess explanation quality using fidelity and stability metrics to evaluate explanation reliability and consistency. Compare model performance across datasets and explainability methods to assess framework robustness and generalizability.
8. Generate visualization outputs and analytical reports to communicate results in a clinically interpretable manner. Create performance comparison charts, feature-importance plots, SHAP summary visualizations, LIME explanation outputs, Grad-CAM heatmaps, and other clinically relevant visualizations. Review all visual outputs to ensure clarity and consistency before reporting.
9. Document all preprocessing operations, model configurations, hyperparameter settings, explainability procedures, validation strategies, and evaluation results. Maintain detailed experimental records to facilitate reproducibility and independent replication of the workflow. Verify consistency of results across repeated experimental runs and archive all workflow artifacts for future reference.
10. Expected Output- Generate a fully trained predictive model with optimized hyperparameters, saved model weights, training logs, performance metrics, and explainability outputs, including SHAP explanations, LIME explanations, and Grad-CAM heatmaps. Store all model artifacts, evaluation reports, and visualization outputs for subsequent analysis and reproducibility assessment.

Figure 3: End-to-End Workflow of the Explainable AI Pipeline. Please click here to view a larger version of this figure.

6. Model Evaluation and Explainability Assessment

1. Evaluate predictive model performance using standard classification metrics, including accuracy, precision, recall, F1-score, and receiver operating characteristic area under the curve (ROC-AUC). Calculate accuracy to measure overall classification correctness.
2. Calculate precision to assess the proportion of correctly identified positive predictions. Calculate recall to evaluate the models ability to identify positive cases. Calculate the F1-score to balance precision and recall. Calculate ROC-AUC to assess discriminative performance across different classification thresholds.
3. Apply these evaluation metrics consistently across all datasets and model architectures to facilitate objective performance comparison. Record all metric values and store the evaluation results for subsequent statistical analysis and reporting.
4. Assess explainability performance using fidelity and stability metrics. Calculate fidelity to determine the extent to which generated explanations accurately reflect model predictions and decision-making behavior.
5. Calculate stability by comparing explanations generated from similar input samples and quantifying the consistency of explanation outputs. Verify that fidelity and stability values meet the predefined quality criteria before interpreting model explanations.
6. Compare explainability performance across SHAP, LIME, and Grad-CAM outputs.
7. Expected Output- Generate quantitative evaluation results, including accuracy, precision, recall, F1-score, ROC-AUC, fidelity, and stability metrics. Produce explainability outputs and visualizations suitable for scientific reporting, reproducibility assessment, and clinical interpretation.

7. Human-Centered Evaluation

1. Recruit 12 healthcare professionals consisting of 6 physicians, 3 radiologists, and 3 clinical data analysts. Select participants with prior experience in clinical decision-making, medical image interpretation, healthcare analytics, or electronic health record review. Obtain informed consent from all participants before initiating the evaluation procedure.
2. Randomly select 100 cases from the testing datasets after completion of model training and explainability analysis. Generate two evaluation conditions for each case:
   1. prediction-only output and
   2. prediction accompanied by explainability information. Randomize the presentation order of cases and evaluation conditions to minimize presentation bias and learning effects.
3. Prepare standardized evaluation forms containing prediction results, explanation outputs, and assessment questionnaires. Present cases individually to participants using a computer-based evaluation interface.
4. Instruct participants to review each case independently without discussing responses with other evaluators. Allow participants to complete all evaluations under identical experimental conditions.
5. Administer a five-point Likert-scale questionnaire adapted from published explainable artificial intelligence evaluation studies. Instruct participants to rate trust, interpretability, and usability for each reviewed case. Record questionnaire responses electronically and verify completion of all survey items before proceeding to statistical analysis.
6. Measure objective indicators of clinical usefulness during the evaluation process. Record decision agreement by comparing participant decisions with the corresponding reference labels or ground-truth outcomes. Calculate decision agreement as the percentage of participant decisions matching the reference outcome.
7. Record review time for each case by measuring the interval between case presentation and final response submission. Calculate average review time separately for prediction-only and prediction-with-explanation conditions.
8. Calculate mean trust, interpretability, and usability scores across all participants and evaluation cases. Compare scores obtained under prediction-only and prediction-with-explanation conditions.
9. Perform paired t-tests after completion of all participant evaluations to determine whether differences in trust, interpretability, usability, decision agreement, and review time are statistically significant. Use a significance threshold of p 0.05 for all statistical comparisons.
10. Calculate Fleiss Kappa after collecting responses from all participants to assess inter-rater agreement. Use the aggregated participant ratings to determine the consistency of evaluations across reviewers. Interpret Fleiss Kappa values according to established agreement guidelines and record the resulting agreement statistics.
11. Summarize participant demographics, evaluation methodology, questionnaire design, statistical analysis procedures, objective performance measures, and inter-rater agreement results in Table 4.
12. Review model outputs under both evaluation conditions and compare participant responses across conditions. Verify that explanations improve trust, interpretability, and usability without negatively affecting decision quality.
13. Confirm the consistency of participant assessments using the calculated Fleiss Kappa statistic. Record all evaluation outcomes for subsequent reporting and reproducibility assessment.
14. Expected Output- Generate clinician trust scores, interpretability ratings, usability ratings, decision-agreement measures, review-time statistics, paired t-test results, and inter-rater agreement statistics. Produce quantitative evidence describing the clinical usefulness, interpretability, and trustworthiness of the explainable artificial intelligence framework.

Parameter	Value
Experts	12
Physicians	6
Radiologists	3
Clinical Data Analysts	3
Cases Reviewed	100
Evaluation Design	Randomized (Prediction-only vs Prediction+Explanation)
Survey Instrument	5-point Likert Scale
Trust Assessment	Interpretability, Trust, Usability
Statistical Test	Paired t-test (p 0.05)
Inter-rater Reliability	Fleiss Kappa
Objective Measures	Decision Agreement, Review Time

Table 4: Human-Centered Evaluation Protocol and Clinician Assessment Design.

This table presents the clinician evaluation framework used to assess the interpretability, trustworthiness, and usability of the explainable AI system. Information regarding participant demographics, case review methodology, survey design, statistical analysis procedures, inter-rater reliability assessment, and objective evaluation measures is summarized.

8. Validation and Reproducibility Assessment

1. Perform validation and ablation studies to assess the robustness and reliability of the proposed framework. Identify features with high importance scores using the selected explainability methods. Remove important features incrementally and retrain the predictive models after each feature-removal step.
2. Evaluate model performance after each ablation experiment using accuracy, precision, recall, F1-score, and ROC-AUC metrics. Compare the resulting performance values with baseline model performance to determine the contribution of individual features to predictive outcomes.
3. Assess explanation stability by generating explanations for similar input samples using SHAP, LIME, and Grad-CAM. Compare explanation outputs across repeated evaluations and quantify consistency using the predefined stability metrics. Verify that explanations remain stable under minor input variations and repeated experimental runs.
4. Record all experimental procedures throughout the workflow. Document data preprocessing operations, dataset partitioning procedures, model architectures, hyperparameter settings, training configurations, explainability methods, validation strategies, and evaluation metrics. Store all configuration parameters and experimental outputs in a structured format to facilitate reproducibility and independent verification.
5. Review all experimental records to ensure completeness and consistency. Verify that the workflow can be replicated using the documented procedures, configuration files, and software specifications. Maintain a modular workflow structure to facilitate adaptation of the protocol for future studies and support conversion into a video-based experimental protocol consistent with JoVE methodological requirements.

9. Reproducibility Resources

1. Execute all experiments using fixed random seeds to ensure reproducible results across repeated runs. Maintain source code, preprocessing scripts, configuration files, environment specifications, model parameters, and visualization scripts within a publicly accessible repository.
2. Provide detailed instructions for dataset acquisition, preprocessing, experiment execution, model training, explainability generation, validation procedures, and regeneration of all reported tables and figures. Archive all workflow components required for independent replication, including software specifications, preprocessing workflows, configuration files, dataset access instructions, model checkpoints, and visualization assets.
3. Summarize the software resources, preprocessing workflows, configuration files, dataset access instructions, and reproducibility assets used throughout the framework in Table 5.
4. Expected Output- Generate a complete reproducible experimental package containing preprocessing scripts, configuration files, trained models, model checkpoints, explainability outputs, validation reports, visualization artifacts, software specifications, and documentation necessary for independent replication and verification of the proposed framework.

Resource	Description
Source Code	Python implementation scripts for data preprocessing, model training, explainability, and evaluation
Environment File	requirements.txt containing package dependencies and versions
Configuration Files	Model architecture settings, hyperparameters, and experiment configurations
Fixed Random Seeds	Seed = 42 used for reproducible experiments
Dataset Access Instructions	Links and procedures for acquiring public healthcare datasets
Preprocessing Scripts	Scripts for data cleaning, normalization, encoding, and feature selection
Figure Generation Scripts	Scripts to regenerate all manuscript figures
Table Generation Scripts	Scripts to reproduce reported tables and metrics
Example Inputs	Sample datasets and input files
Example Outputs	Prediction results, explanations, and evaluation reports

Table 5: Reproducibility Resources and Experimental Assets.

This table summarizes the resources provided to support experimental reproducibility, including source code, preprocessing scripts, configuration files, environment specifications, dataset access instructions, fixed random seed settings, figure-generation scripts, and example input/output files required for reproducing the reported results.

Results

We thoroughly evaluated the explanatory AI component of our entire pipeline, assessing how well it predicted across different types of healthcare data, including both structured clinical data and medical images. The outcomes indicated consistent predictive performance across the evaluated datasets. Among the models tested, the gradient boosting model achieved the highest accuracy at 97.4%, followed by the random forest model at 94.2%. Deep learning methods applied to image sets achieved 91.3% accuracy, while multi-layer perceptron models produced marginally lower results, 90.8%. This suggests that ensemble-based machine learning models perform best on structured healthcare data, whereas deep learning architectures excel at imaging tasks. Table 6 outlines in detail various performance metrics for prediction functions, including accuracy, precision, recall, and F1-score, as well as explainability metrics such as fidelity and stability. We arrived at these performance figures by employing five-fold cross-validation and presenting results as mean values (with 95% confidence intervals). Confidence intervals not only indicate the model's uncertainty but also make comparisons of predictive performance more reliable, accounting for dataset memory and differences in model architectures.

Model	Accuracy (%)	Precision	Recall	F1-Score	Fidelity	Stability	95% CI
Random Forest	94.2	0.93	0.92	0.92	0.85	0.82	93.1–95.3
XGBoost	97.4	0.96	0.95	0.95	0.92	0.89	96.5–98.3
MLP	90.8	0.89	0.88	0.88	0.80	0.78	89.6–92.0
CNN (Imaging)	91.3	0.90	0.91	0.90	0.88	0.86	90.1–92.5

Table 6: Comparative Performance of Predictive Models and Explainability Methods.
This table presents a comparative evaluation of machine learning and deep learning models using predictive performance metrics, including accuracy, precision, recall, F1-score, and ROC-AUC. Additionally, explainability metrics such as fidelity, stability, comprehensibility, and human trust scores are reported to provide a comprehensive assessment of both predictive effectiveness and interpretability.

The results indicate that Gradient Boosting achieved the highest overall predictive performance (97.4% accuracy), outperforming Random Forest by 3.2 percentage points and the deep learning models by more than 6 percentage points. This observation suggests that ensemble-based learning methods were particularly effective for the structured healthcare datasets included in this study. The smaller performance difference between CNN and MLP models indicates that increased model complexity alone did not necessarily translate into superior predictive performance under the evaluated experimental conditions. 
To showcase the usefulness of our proposed framework across a variety of healthcare analytics tasks, we present the dataset-specific prediction performance results in Table 7. 

Dataset-Specific Analysis.
Performance varied across datasets because of differences in feature complexity, sample size, class distribution, and data modality. The Breast Cancer dataset achieved the highest predictive accuracy, likely due to well-defined feature separability. Slightly lower performance on MIMIC-III and NIH Chest X-ray datasets reflects the greater complexity of longitudinal clinical records and medical imaging data. These findings demonstrate that framework performance is influenced by dataset characteristics and clinical context.

Dataset	Accuracy (%)	Precision (%)	Recall (%)	F1-score (%)
Breast Cancer	97.4	97.2	97.6	97.1
Diabetes	94.2	93.9	94.4	94
MIMIC-III	91.3	91	91.5	91.1
NIH Chest X-ray	90.8	90.4	91.2	90.6

Table 7: Dataset-Specific Predictive Performance Results.
Predictive performance of the proposed explainable AI framework across four representative healthcare datasets. Accuracy, precision, recall, and F1-score are reported as percentages (%) on the independent test sets. Higher values indicate better classification performance.

To assess predictive performance, four machine learning and deep learning models, namely Gradient Boosting, Random Forest, Convolutional Neural Network (CNN), and Multilayer Perceptron (MLP), were evaluated across four representative healthcare datasets. Model performance was assessed using accuracy, precision, recall, F1-score, and ROC-AUC metrics on independent test datasets following a 70:15:15 train-validation-test split and five-fold cross-validation. Improvements in predictive performance were determined by comparing model-specific evaluation metrics across identical preprocessing and validation conditions.
In order to illuminate how model performances differ in various methods, Figure 4 displays the advantages and disadvantages of ensemble, neural network, and deep learning models pictorially.

Figure 4: Comparative Performance of Machine Learning and Deep Learning Models on Healthcare Prediction Tasks. (A) Accuracy comparison of Gradient Boosting, Random Forest, CNN, and MLP models on the test dataset. (B) F1-score comparison of Gradient Boosting, Random Forest, CNN, and MLP models on the test dataset. (C) Precision comparison of Gradient Boosting, Random Forest, CNN, and MLP models on the test dataset. (D) Recall comparison of Gradient Boosting, Random Forest, CNN, and MLP models on the test dataset. Higher values indicate better predictive performance. Gradient Boosting achieved the highest overall performance across all evaluation metrics, followed by Random Forest. Please click here to view a larger version of this figure.

Interpretation of Explainability Performance.
SHAP consistently achieved the highest fidelity and stability scores among the evaluated explainability methods, indicating stronger agreement between generated explanations and underlying model predictions. LIME exhibited comparatively lower stability, suggesting greater sensitivity to local perturbations and sampling variability. Grad-CAM provided visually interpretable explanations for image-based models but produced slightly lower fidelity values than SHAP. These findings indicate that explanation quality depends on both the selected explanation technique and the underlying predictive model architecture.

The explainability techniques integrated in the pipeline have been assessed in terms of their performance quantitatively and qualitatively. Specifically, feature attribution methods have succeeded in uncovering the model's internal logic quite consistently across different data sets.
All the methods considered in the study, SHAP, achieved the highest fidelity (0.92) and stability (0.89) among the evaluated explainability methods. To put it simply, the explanations were very accurate representations of what the model actually predicted. Local explanation methods are a kind of window through which we can see a specific prediction and understand what the model used as its basis for the decision.

Interpretability performance was evaluated using fidelity, stability, comprehensibility, and clinician trust scores obtained from human-centered evaluation. SHAP, LIME, and Grad-CAM explanations were compared using identical datasets and trained models. Improvements in explainability were assessed by comparing explanation quality metrics and clinician ratings across the evaluated explain ability methods. Visualization techniques for deep learning models that work with images essentially produce heat maps, which identify the regions of the images that have the highest relevance to the model's prediction. Feature importance distributions and explainability performance comparisons across different methods are shown in Figure 5.

Figure 5: Explainability Performance Evaluation.The figure presents the performance of explainability methods with the help of fidelity and stability metrics. SHAP leads the way in fidelity (0.92) and stability (0.89), which reflects good agreement between explanations and model predictions. LIME comes out as the best tool for the interpretability of individual examples. On the other hand, Grad-CAM outputs visual heatmaps mainly used in imaging models, showing coarsely the most important regions responsible for the prediction of the model, which, from a clinical point of view, might be very relevant.Please click here to view a larger version of this figure.

After evaluating the predictive models and techniques for interpretability, it was revealed that the correlation between the accuracy of a model and the dependability of its explanations is very strong. In fact, the same models that showed prediction improvements were also explaining their results with more stability and consistency when the interpretability techniques were properly applied. Ensemble models were found to be the most interpretable when accompanied with feature attribution methods, whereas deep learning models were better served by visualization-based explainable methods. The link that ties together the accuracy of prediction and the reliability of explanations can be seen in Figure 6, which outlines the conceptual dynamics between model and explainability in great detail.

Figure 6: Comparative analysis of models and explain ability methods.The chart shows a detailed comparison of the accuracy and explain ability measures of various models. It comprises a scatter graph depicting the correlation between accuracy and explanation stability, a tabular comparison of performances, and a suitability matrix outlining the different explain ability methods SHAP, LIME, and Grad-CAM, and their effectiveness with different model types. The visual clearly shows that ensemble models plus SHAP deliver great interpretability, whereas deep learning models are explainable through visualization techniques. Please click here to view a larger version of this figure.

The human-centered evaluation confirmed the real-world usefulness of the suggested system in hospitals and other clinical facilities. Medical professionals reviewed model outputs with and without explanations. Showing the results with explanations led to a considerable increase in user trust and interpretability; the average trust value rose from 3.1 to 4.7 on the scale of 1 to 5. The increase in trust score from 3.1 to 4.7 represents an approximate relative improvement of 51.6%. The substantial inter-rater agreement (Fleiss' κ = 0.81) further indicates consistent clinician assessment of explanation usefulness. These findings suggest that explainability outputs may improve user confidence and facilitate the interpretation of AI-assisted clinical decisions. Professionals stated better comprehension of model outputs and higher trust in AI-supported judgment of clinical situations, which points to the significance of interpretability in the actual implementation of healthcare.

We further verified the robustness of our framework with ablation analysis. Removal of critical features that, through explainability methods, were deemed essential resulted in a major loss of prediction performance. Thus, the finding reflects that features were not only pertinent but also significant to the predictive task. In addition, the explanation results kept displaying only negligible differences when different input samples were explained, hence showing the stability and reliability of the explainability methods.

To check the feasibility of the pipeline for practical use, we measured its computational speed. The introduction of explainability techniques caused some increase in computational time, especially when attributing features and creating visualizations. Nevertheless, the total execution time was still feasible and not too long. Figure 7 shows how the computational time is shared among different stages of the pipeline, such as preprocessing, model training, explainability generation, evaluation, and visualization.

Figure 7: Breakdown of Pipeline Execution Time. This chart shows how computational time is divided among various phases of the explainable AI pipeline, such as preprocessing, model training, explain ability creation, evaluation, and visualization. The data reveal that the majority of the timeline is taken up by model training, which is then followed by explain ability processing. On the other hand, the time spent on preprocessing and reporting is quite minimal. Please click here to view a larger version of this figure.

We tested the strength of the proposed framework through an ablation analysis as well. Taking away the essential features discovered via explainability methods showed a clear decrease in predictive performance, which is a clear sign that those features are not only relevant but also quite important. Besides that, explanation outputs were quite stable even when there was a slight change in the input samples, which is proof of the consistency and reliability of the explainability techniques.

In short, by combining the measurement of predictive performance, explainability, and human-focused validation, it was shown that the proposed framework was effective. The system not only made very accurate predictions, but it also remained highly interpretable and easy to use. Bringing in explainability methods made the whole process transparent without taking a hit on the model's performance. The improvements we saw in prediction accuracy and interpretability were not only made under tight experimental control, but they were also statistically significant, a fact that speaks for the endurance and the authenticity of the proposed way. Pairwise comparisons among predictive models were performed using paired t-tests across cross-validation folds. Statistically significant differences were observed between Gradient Boosting and competing models (p < 0.05), confirming the superiority of the proposed configuration.

Overall Findings.
Collectively, the results demonstrate that predictive performance, explanation quality, and clinician trust can be evaluated within a unified and reproducible workflow. The framework maintained consistent performance across multiple healthcare datasets while supporting transparent interpretation through SHAP, LIME, and Grad-CAM explanations. However, the findings should be interpreted within the context of the selected datasets and validation settings, and additional external validation remains necessary before real-world deployment.

DATA AVAILABILITY:
We obtained the datasets used in the current research from public sources, and they can be found in well-known data repositories. For instance, the data pertaining to breast cancer and diabetes are downloadable from the UCI Machine Learning Repository at: https://archive.ics.uci.edu/ml/index.php

One can get the electronic health record dataset (MIMIC-III) via Physio Net at: 
https://physionet.org/content/mimiciii/1.4/

The chest X-ray imaging data was drawn from the NIH Chest X-ray Dataset and may be found at: https://www.kaggle.com/datasets/nih-chest-xrays/data

All the datasets we have explored in this study are anonymized and made publicly available. The preprocessing steps, trained model, and the explain ability outputs that we produced during the research are available through the corresponding author if a reasonable request is made. Source code, preprocessing scripts, configuration files, and reproducibility resources will be deposited in a public repository upon manuscript acceptance.

Discussion

This study developed and evaluated a reproducible explainable artificial intelligence (XAI) workflow for healthcare analytics that integrates predictive modeling, explainability assessment, clinician-centered evaluation, and reproducibility resources within a single protocol. The results demonstrated that ensemble-based machine learning models, particularly Gradient Boosting and Random Forest, achieved the highest predictive performance on the evaluated structured healthcare datasets, whereas deep learning models were more suitable for image-based analyses. These findings highlight the importance of selecting model architectures according to data characteristics rather than assuming that more complex models will always yield superior performance. A key contribution of this work is the integration of predictive modeling, explainability assessment, human-centered evaluation, and reproducibility practices within a standardized workflow. Unlike studies that evaluate explainability techniques in isolation, the proposed framework provides a protocol-driven methodology that supports transparent and reproducible experimentation across diverse healthcare datasets.

The explainability analysis demonstrated measurable differences among the evaluated methods. SHAP achieved the highest fidelity and stability scores within the evaluated experimental conditions, indicating closer agreement between generated explanations and model predictions. LIME provided useful local explanations but exhibited lower stability, while Grad-CAM generated intuitive visual explanations for imaging data. These findings suggest that explanation quality depends on both the selected explainability method and the underlying predictive model. Human-centered evaluation further demonstrated that the inclusion of explanations improved clinician trust and interpretability. The observed increase in trust scores and substantial inter-rater agreement (Fleiss’ κ = 0.81) indicate consistent assessment of explanation usefulness across participants. These findings support the potential value of explainability methods for improving transparency and user acceptance in healthcare decision-support systems.

Several limitations should be considered when interpreting the results. First, the framework was evaluated using a limited number of publicly available datasets and may not fully represent the variability encountered in real-world clinical environments. Second, the clinician evaluation involved a relatively small participant cohort, which may limit generalizability. Third, the explainability methods investigated in this study are post-hoc techniques and therefore remain subject to known limitations, including sensitivity to perturbations and potential discrepancies between explanations and true model reasoning. Finally, external validation across independent healthcare institutions was beyond the scope of the present study.

Future research should investigate multimodal healthcare analytics integrating clinical records, medical imaging, physiological signals, and wearable-sensor data. Additional work is needed to evaluate causal and counterfactual explanation methods, uncertainty quantification, fairness assessment, calibration analysis, and prospective clinical validation. Such studies may further improve the transparency, reliability, and applicability of explainable AI systems in healthcare.

In addition to performance evaluation, several practical implementation challenges and protocol modifications should be considered to ensure robust deployment of the proposed explainable AI framework in diverse healthcare environments.

Limitations of Explainability Methods

SHAP, LIME, and Grad-CAM, while providing helpful insights regarding model behavior, also have several drawbacks. It has been noted in the literature that these tools may be unstable in repeated runs, quite sensitive to perturbations, differ across datasets, and sometimes do not accurately reflect the model's real reasons. Therefore, post-hoc explanations should only be seen as supplementary evidence and not the actual portrayal of causal decision mechanisms. Proper validation of the trustworthiness of explanations is still very much an imperative step before they can be used in highly impactful clinical settings.

This is also consistent with the findings of recent biomedical AI research, which has stressed the need for explaining reliability validation before its implementation in clinical settings. The incorporation of explainability has been used as a key aspect of validation in cardiac arrest prediction systems so that they may be more transparent and gain the trust of clinical predictions⁴¹. In the same way, liver ultrasound image segmentation using visualization-driven explainability techniques was aimed at increasing interpretability without disregarding the fact that post-hoc explanations are limited. These papers confirm that explanation consistency checks, explanation robustness tests, and clinically meaningful explainability output validations are necessary to be at the highest level of clinical use-readiness⁴².

Calibration, Fairness, Robustness, and External Generalization

The proposed framework's future versions will include the evaluation of the probability calibration by means of calibration curves and Brier score, estimating the uncertainty by using Bayesian and ensemble-based methods, fairness audit of the demographic subgroups, and robustness check under the presence of noisy data and changing domain conditions. These analyses are particularly important in healthcare settings where model confidence, equitable performance, and reliability under changing clinical environments directly influence patient safety and clinical adoption. Recent AI-enabled scientific frameworks have similarly emphasized the importance of trustworthy, transparent, and reliable decision-support systems beyond conventional predictive performance evaluation. Recent AI-enabled scientific frameworks have similarly emphasized the importance of trustworthy, transparent, and reliable decision-support systems beyond conventional predictive performance evaluation ⁴³.

Troubleshooting and Protocol Modifications

Several practical points need to be considered for the successful roll-out of the proposed explainable AI framework. A typical problem is overfitting, for example, when training deep learning models on relatively small healthcare datasets. Overfitting can be reduced by applying cross-validation, early stopping, dropout regularization, data augmentation, and thorough hyperparameter optimization. Watching validation performance during training is a good way to avoid a very complex model that poorly generalizes.

Another very important issue in healthcare datasets is class imbalance, i.e., positive clinical outcomes being much rarer than negative cases. In order to deal with this problem, stratified sampling, class-weight adjustment, synthetic minority oversampling, and the use of balanced evaluation metrics like F1-score and ROC-AUC should be a part of the modeling pipeline. Ignoring class imbalance risks producing performance metrics that are not truly representative of the model and clinical predictions that might be biased.

Medical imaging datasets typically are not perfect and may be corrupted by noise, acquisition artifacts, inconsistent quality, etc. These factors also vary with the type of imaging device. Such factors may negatively impact predictive performance and also affect the explainability outputs. Therefore, standardized preprocessing steps, image normalization, quality control checks, and data augmentation techniques should be applied so that robustness and reliability of the model are achieved.

SHAP enables deduction of highly informative feature-attribution explanations. However, occurrences of instability cannot be discounted, especially when highly correlated features exist or when model outcomes are very sensitive to tiny changes in the input data. To enhance explanation consistency and reliability, writing the explanation multiple times, assessing stability over multiple runs, using feature grouping strategies, and checking against other explainability methods like LIME are advocated.

When it comes to large-scale healthcare data and huge deep neural network architectures, GPU memory limits may turn into one of the major constraints. Memory needs can be considerably lowered through batch size changes, mixed precision training, model pruning, reduction in feature dimensionality, and adoption of efficient data loading. Besides, researchers deploying the framework on low-resource systems might think about using cloud computing environments or distributed training.

The modular design of the proposed framework allows for changes to be made easily for different healthcare applications. Depending on the clinical task, scientists can change one or more individual prediction models, add new methods of explanation, use different types of healthcare data together, or include modules of uncertainty quantification and calibration without changing the overall workflow. This flexibility helps make the framework usable in very different healthcare analytics cases while still being reproducible and interpretable.

Regulatory and Ethical Considerations:

AI systems made more understandable can be a great help in health care. But it is not enough. Regulator rules, which call for transparency, accountability, privacy, and patient safety, must be fulfilled. Even if explainability may increase trust and interpretability, it cannot alone ensure clinical validity. Responsible implementation needs prospective clinical validation, fairness assessment, regulatory review, and post-deployment monitoring. In addition, patient confidentiality, clinician oversight, and equitable performance across demographic groups should be considered when deploying the framework in the future. To be integrated into real-world clinical workflows, AI systems and healthcare professionals must interact without any glitches. Therefore, explainability information needs to be part of current electronic health records and clinical decision-support platforms, not separate tools that only work on their own. The ultimate choices must be made by the doctors, whereas explainable AI is merely a helper technology that enhances transparency, backs up evidence-based reasoning, and makes the communication between healthcare workers and patients easier.

This article presents a reproducible protocol for developing, evaluating, and interpreting explainable artificial intelligence systems in healthcare analytics. The contribution is methodological and workflow-oriented, providing researchers and clinicians with a standardized framework that can be replicated, adapted, and extended across multiple healthcare applications. It combines different aspects such as data preprocessing, predictive modeling, explainability methods, and performance evaluation in a single unified manner. The methodology demonstrated strong predictive performance across multiple healthcare datasets. When it comes to structured data analysis, model ensembles are the best performers, while deep learning models are found to be very effective for medical imaging tasks. Furthermore, using explanation techniques makes it easier for the users to understand the models, as it offers both global and local interpretations of the prediction results. Therefore, it not only increases the trust of the clinicians but also the models' usability. The findings indicate that explainable AI can support the development of transparent and interpretable healthcare analytics systems to find a compromise between the accuracy and interpretability of the models, which are essential for reliable and trustworthy healthcare analytics. Consequently, the method presented here is a very effective and straightforward tool for medical data analysis that can help healthcare professionals in using AI technologies that they trust. This article presents a reproducible protocol for developing, evaluating, and interpreting explainable artificial intelligence models in healthcare analytics. By integrating data preprocessing, predictive modeling, explainability assessment, clinician-centered evaluation, and reproducibility resources within a unified workflow, the protocol provides a practical framework for transparent healthcare AI research. The workflow can be adapted to diverse healthcare datasets and application domains, supporting reproducible implementation, evaluation, and reporting of explainable machine learning systems.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
GPU Workstation	Manufacturer dependent	NA	Training deep learning models
Jupyter Notebook	Project Jupyter	Latest stable release	Interactive experiment execution
LIME	LIME	Version 0.2.0	Local interpretable explanations
Matplotlib	Matplotlib Project	Version 3.9	Data visualization
MIMIC-III Database	PhysioNet	Version 1.4	Electronic health record analysis
NIH Chest X-ray Dataset	NIH Clinical Center	Public Dataset	Thoracic disease classification
NumPy	NumPy Developers	Version 1.26	Numerical computing and array operations
OpenCV	OpenCV Foundation	Version 4.10	Image preprocessing
Pandas	Pandas Development Team	Version 2.1	Data manipulation and preprocessing
Pima Indians Diabetes Dataset	UCI Machine Learning Repository	Public Dataset	Diabetes risk prediction
Python 3.11	Python Software Foundation	Version 3.11	Primary programming environment
Scikit-learn	scikit-learn	Version 1.5	Machine learning algorithms and evaluation
Seaborn	Seaborn	Version 0.13	Statistical visualization
SHAP	SHAP	Version 0.46	Feature attribution and explainability
TensorFlow	TensorFlow	Version 2.15	Deep learning model development
Windows / Ubuntu Linux	Microsoft / Canonical	NA	Operating system
Wisconsin Breast Cancer Dataset	UCI Machine Learning Repository	Public Dataset	Breast cancer diagnosis