Enhancing Heart Disease Prediction Through Cluster-Visualized Distributed Machine Learning Paradigms for Secure Healthcare

Summary

This study develops a scalable framework for heart disease prediction using Hadoop MapReduce and K-Means clustering. Adjusting classification cut-offs affects detection sensitivity. CViHDKNN improves true positive detection while controlling false positives, whereas CViHDDT reduces false positives but may miss some heart disease cases.

Abstract

Heart disease remains one of the leading causes of mortality worldwide, creating an urgent need for accurate and scalable predictive systems that enable early diagnosis and timely clinical intervention. Traditional machine learning approaches often struggle to efficiently process large-scale medical datasets and lack interpretability, limiting their usefulness for supporting clinical decision-making. To address these challenges, this study proposes a Cluster Visualized Distributed Machine Learning framework for heart disease prediction. The framework incorporates two distributed algorithms: Cluster Visualized Hadoop Distributed Decision Tree (CViHDDT) and Cluster Visualized Hadoop Distributed K-Nearest Neighbor (CViHDKNN). The proposed models leverage Hadoop’s MapReduce framework for distributed computation across large datasets, while integrating K-Means clustering for improved data organization and visualization. This cluster-based visualization enhances interpretability by allowing clinicians to better understand relationships among patient risk factors and prediction outcomes. Experimental evaluation was conducted using the UCI Heart Disease dataset in a Hadoop-based distributed environment. The results show that CViHDKNN achieved superior predictive performance, achieving 85.25% accuracy and 88% recall, outperforming the CViHDDT model, which achieved 80.33% accuracy. Adjusting classification cut-off values also influenced sensitivity and detection rates: lower cut-offs improved true-positive detection while maintaining acceptable false-positive levels. These findings demonstrate that clustering-enhanced distributed learning improves scalability, predictive accuracy, and clinical interpretability for heart disease prediction.

Introduction

Heart disease is one of the leading causes of death worldwide and represents a major public health challenge. The rising prevalence of cardiovascular diseases highlights the urgent need for advanced diagnostic models that support early detection and treatment. Traditional diagnostic approaches rely heavily on manual evaluation and clinical judgment, which often struggle to manage large volumes of medical data efficiently. Recently, artificial intelligence (AI) and machine learning (ML) techniques have played an important role in predictive healthcare analytics, enabling data-driven disease prediction and improving risk assessment accuracy in medical systems^1,2.

Conventional diagnostic methods and rule-based decision systems often suffer from limited scalability and lower precision when applied to complex medical datasets. Although machine learning techniques have improved prediction performance, many models still face challenges in processing large-scale medical data and maintaining interpretability for clinical decision-making. Deep learning models may achieve high predictive accuracy but often function as “black-box” systems, making it difficult for healthcare professionals to understand the reasoning behind predictions^3,4,5. This lack of transparency limits their adoption in clinical environments where explainability is essential^6,7,8,9,10.

To overcome these challenges, distributed computing frameworks such as Hadoop have been increasingly adopted for large-scale healthcare analytics. Hadoop’s distributed architecture enables parallel processing of large datasets using the Hadoop Distributed File System (HDFS) and MapReduce, improving computational efficiency and scalability in medical data analysis. However, distributed machine learning models still require mechanisms that enhance interpretability and transparency. Integrating clustering and visualization techniques can help reveal hidden patterns within patient data and support clinicians in understanding predictive results more effectively^{11,12,13,14,15,16}.

Several researchers have explored distributed machine learning techniques for heart disease prediction using algorithms such as decision trees and K-Nearest Neighbor (KNN). Distributed Decision Tree (HDDT) models implemented on Hadoop frameworks have demonstrated improved scalability and classification accuracy compared with traditional decision tree approaches^{17,18,19,20,21,22,23}. Similarly, Hadoop Distributed KNN (HDKNN) methods leverage parallel processing to improve classification efficiency for large, high-dimensional medical datasets^24,25,26. However, these models often lack strong interpretability and visualization mechanisms, which are necessary for effective clinical decision-making and understanding patient risk profiles. Despite these advances, existing distributed models still lack integrated mechanisms for interpretability and visual understanding of patient risk patterns.

To address these limitations, this study proposes a Cluster Visualized Distributed Machine Learning (CVDML) framework for heart disease prediction. The framework introduces two distributed predictive models: Cluster Visualized Hadoop Distributed Decision Tree (CViHDDT) and Cluster Visualized Hadoop Distributed K-Nearest Neighbor (CViHDKNN). CViHDDT applies distributed decision tree construction to optimize the selection of medical features, such as symptoms and prior medical history, while CViHDKNN implements a distributed, parallelized KNN approach to classify patients based on similar medical characteristics. The integration of clustering and visualization techniques improves classification performance and enhances interpretability by grouping patients with similar disease patterns.

The main objective of this research is to develop a scalable and interpretable distributed machine learning framework for accurate heart disease prediction using large medical datasets. The key contributions of this study include: (1) Development of a Hadoop-based distributed machine learning framework capable of efficiently processing large healthcare datasets. (2) Integration of clustering visualization techniques with distributed decision tree and KNN models to improve interpretability and transparency in predictive healthcare analytics²⁷. (3) Demonstration of improved prediction performance through enhanced accuracy, recall, and anomaly detection capability compared with traditional machine learning approaches²⁸.

Protocol

Dataset acquisition

The UCI Heart Disease dataset is a widely used dataset in medical and machine learning research for predicting heart disease. It contains various clinical and diagnostic characteristics of patients, enabling healthcare professionals and researchers to develop data-driven prediction models. The dataset classifies individuals as either likely or unlikely to have heart disease based on several patient attributes, including age, gender, chest pain type, blood pressure, cholesterol levels, and electrocardiogram results (https://archive.ics.uci.edu/dataset/45/heart+disease)²⁹. The overall workflow of the proposed heart disease prediction framework, including data preprocessing, distributed model implementation, and evaluation stages, is illustrated in Figure 1.

Experimental environment setups

The experimental environment was deployed on Apache Hadoop 3.x as the core distributed computing framework for all implementations. The cluster used a master-worker architecture with one dedicated master node and multiple worker nodes. The master node managed job scheduling, resource allocation, and cluster coordination using YARN (Yet Another Resource Negotiator), while worker nodes executed distributed computation tasks in parallel to efficiently process large-scale medical datasets. Each node in the cluster was provisioned with Intel Core i7 processors (or equivalent), 16–32 GB of RAM, and approximately 1 TB of storage.

Data ingestion into HDFS

Dataset Storage

The experimental dataset was stored in HDFS in a block-distributed format, with the target variable indicating the presence or absence of heart disease, separated from the independent feature set, prior to storage across cluster nodes. Feature-specific preprocessing was applied to all stored data blocks using MapReduce workflows. Numerical features, including age, blood pressure, cholesterol levels, and heart rate, were normalized using a robust scaler based on the interquartile range, reducing the influence of outliers that are particularly prevalent in medical datasets where extreme values may represent rare or severe clinical conditions. Categorical variables with more than two categories, such as cp, restecg, and thal, were transformed using one-hot encoding, converting categorical attributes into binary numerical representations compatible with machine learning algorithm inputs^30,31,32. All preprocessing operations were executed as distributed MapReduce jobs across HDFS data blocks, ensuring uniform application of the full pipeline without centralizing raw data at any single point.

Partitioning across nodes

The dataset was partitioned into training and test sets using an 80:20 split, with 80% allocated to training and 20% reserved for evaluation on unseen data. This partitioning was applied consistently across all distributed worker nodes to ensure that each node processed a proportional and representative shard of the full dataset, preventing data skew and supporting balanced model generalization. Scaling ensured that all numerical variables contributed equally during distributed training by preventing features with larger magnitudes from dominating the learning process across nodes. This structured partitioning strategy enhanced predictive reliability and helped prevent overfitting by maintaining a clear separation between training and evaluation data across the distributed cluster.

Data preprocessing

Missing value handling

Medical datasets frequently contain incomplete records due to data entry errors, device malfunctions, or patient non-response during clinical data collection. Prior to model training, all dataset attributes were examined for missing or null values. Rows with missing values in critical clinical features, such as blood pressure, cholesterol, and heart rate, were identified and handled using mean imputation for numerical variables and mode imputation for categorical variables. This approach preserved the dataset's statistical distribution while ensuring that no training sample was discarded unnecessarily, maintaining maximum data availability for model learning across distributed HDFS nodes.

Feature scaling

Numerical features, including age, blood pressure, cholesterol levels, and maximum heart rate, exhibit significantly different value ranges, which can cause features with larger magnitudes to disproportionately influence model training. To address this, a robust scaler based on the interquartile range was applied to all continuous numerical attributes. This scaling strategy is particularly appropriate for medical datasets where extreme clinical values representing rare or severe conditions may otherwise distort the learning process. Scaling ensured that all numerical variables contributed equally during model training and was applied consistently across all distributed worker nodes using MapReduce workflows.

Encoding

Categorical variables with more than two distinct categories, including cp (chest pain type), restecg (resting electrocardiographic results), and thal (thalassemia type), were transformed using one-hot encoding. This process converted each categorical attribute into a set of binary numerical indicator columns, producing representations that machine learning algorithms can process effectively without imposing artificial ordinal relationships between category values. Binary categorical variables were retained in their original numerical form. All encoding operations were executed as distributed MapReduce jobs across HDFS data blocks, ensuring consistent transformation across all partitioned dataset shards.

Train/test split

The preprocessed dataset was partitioned into training and test subsets using an 80:20 split, with 80% allocated to model training and 20% reserved for performance evaluation on unseen data. The target variable, indicating the presence or absence of heart disease, was separated from the independent feature set prior to splitting. This partitioning was applied uniformly across all distributed HDFS nodes to ensure that each worker node processed a proportional and representative shard of the full dataset, preventing data skewing. The 80:20 split strategy enhanced predictive reliability, improved model generalization, and maintained a clear separation between training and evaluation data across the distributed cluster environment, thereby preventing overfitting.

Model implementation

The Cluster Visualized Hadoop Distributed Decision Tree (CViHDDT) model classifies patients into risk categories using a distributed decision tree. The decision tree algorithm recursively splits the dataset based on the most informative features, maximizing the separation between patients with and without heart disease. Within the Hadoop distributed framework, this process is executed across multiple computing nodes, allowing large datasets to be processed efficiently. The distributed architecture reduces computational time while improving scalability. The Cluster Visualized Hadoop Distributed K-Nearest Neighbor (CViHDKNN) algorithm uses the same dataset but applies a different classification strategy. Instead of constructing a decision tree, the model identifies the closest neighboring patients based on medical attributes such as blood pressure, cholesterol levels, and exercise-induced angina. Using distributed computation, the KNN algorithm efficiently clusters patients with similar medical characteristics while managing computational complexity.

The classification principle of the distributed K-nearest neighbor model is illustrated in Figure 2, where a new instance is assigned to a class based on the majority class among its nearest neighbors. Cluster visualization techniques enable healthcare professionals to identify groups of patients with similar clinical characteristics, thereby improving interpretability and supporting personalized treatment recommendations. The proposed heart disease prediction framework integrates data preprocessing, distributed machine learning algorithms, and cluster visualization techniques. By leveraging Hadoop’s distributed computing capabilities, the framework efficiently processes large healthcare datasets while maintaining high prediction accuracy and interpretability, enabling early heart disease detection and improved clinical decision-making.

Cluster visualized Hadoop Distributed decision tree (CViHDDT):

Distributed Decision Tree Training

The proposed Cluster Visualized Hadoop Distributed Decision Tree (CViHDDT) model differs fundamentally from traditional decision tree construction by distributing the tree-building process across multiple nodes in the Hadoop ecosystem rather than building the entire tree on a single machine. Individual worker nodes construct partial decision trees locally on their assigned subset of the dataset using either MapReduce or Apache Spark for parallel processing. These locally constructed partial trees are subsequently combined into a complete global decision tree that encompasses the full distributed dataset. This distributed training strategy significantly accelerates model training, enabling the framework to efficiently handle multi-terabyte medical datasets at scale. The parallel computing infrastructure provided by Hadoop ensures the CViHDDT model is inherently scalable and well-suited for big-data-driven healthcare solutions. Following distributed tree construction, cluster visualization techniques are applied to enhance model interpretability by grouping decision tree nodes into clusters of patients with similar medical conditions using algorithms such as k-means and hierarchical clustering. This clustering process produces clinically meaningful risk categories—such as mild, moderate, and severe heart disease—enabling healthcare professionals to identify patterns in patient data, understand disease progression, and formulate personalized treatment plans.

Feature Selection

Prior to distributed decision tree training, the CViHDDT model applies a structured preprocessing and feature selection pipeline to the raw medical data ingested from HDFS. Missing values are addressed through imputation algorithms to manage incomplete clinical records and prevent data loss without discarding patient samples. Robust Scaler normalization is applied to numerical features such as blood pressure and cholesterol levels to mitigate the disproportionate influence of outliers prevalent in medical datasets. Categorical variables such as gender and family history of heart disease are transformed using one-hot or label encoding to produce numerical representations compatible with machine learning algorithms. After preprocessing, feature extraction is performed to identify the key clinical attributes most predictive of heart disease. This stage eliminates irrelevant and redundant features from the dataset, reducing computational costs in subsequent distributed training stages and ensuring that only the most diagnostically informative attributes — such as chest pain type, resting blood pressure, serum cholesterol, maximum heart rate, and ST depression — are retained as inputs to the distributed decision tree construction process. This systematic feature reduction improves model efficiency, reduces training time across distributed nodes, and enhances the overall predictive reliability of the CViHDDT framework by focusing the learning process on attributes with the strongest clinical discriminative power.

MapReduce Workflow

The MapReduce programming model forms the computational backbone of the CViHDDT distributed training pipeline, enabling parallel processing of the heart disease dataset across all worker nodes in the Hadoop cluster. In the map phase, each worker node independently processes its assigned HDFS data shard, computing partial decision tree structures and local splitting statistics — including Information Gain and Gini Index values — for each candidate attribute, without requiring access to data stored on other nodes. In the reduce phase, the locally computed partial trees and sufficient statistics are aggregated across all nodes to construct the complete global decision tree, consolidating the distributed knowledge learned at each node into a single unified predictive model. This map-reduce decomposition of the tree-building process enables the CViHDDT model to scale linearly with the number of worker nodes, making real-time analysis of large-scale medical datasets computationally feasible. The MapReduce workflow also supports the distributed execution of cluster visualization procedures, in which clustering algorithms are applied in parallel across HDFS data blocks to group patient records into risk categories based on their decision-tree node assignments. Performance evaluation of the resulting model employs precision, recall, F1-score, and classification accuracy as primary metrics, with the distributed cluster visualization further reducing false negatives by enabling finer decision boundaries within the tree — directly improving sensitivity for identifying at-risk patients and enhancing the clinical reliability of the CViHDDT heart disease prediction framework.

Cluster Visualized Hadoop Distributed K-Nearest Neighbor (CViHDKNN)

Clustering

The CViHDKNN (Cluster Visualized Hadoop Distributed K-Nearest Neighbor) framework begins by applying clustering techniques to the heart disease dataset prior to classification, grouping patients with similar medical characteristics into coherent clusters before the KNN search is performed. The heart disease dataset, containing clinical features such as age, cholesterol level, blood pressure, ECG results, and heart rate, is preprocessed and distributed across nodes in the Hadoop cluster using HDFS. Clustering algorithms, including K-Means and Hierarchical Clustering, are then applied across these distributed data partitions to divide the dataset into groups of patients sharing related medical profiles. This pre-classification clustering step serves a critical computational purpose: by restricting the KNN search space to only the most relevant cluster rather than the entire dataset, the algorithm dramatically reduces the number of distance computations required per query instance. Visualizing these clusters provides additional clinical benefit by enabling the identification of patient subgroups with closely related medical characteristics and by supporting more meaningful categorization of risk profiles prior to the nearest-neighbor classification stage. The clustering-based optimization not only reduces computational overhead but also improves classification accuracy by ensuring that each query instance is compared only against the most contextually similar patient records, making the approach particularly well-suited for large-scale heart disease datasets where exhaustive distance computation across the full dataset would be computationally prohibitive.

Distributed KNN

The distributed KNN component of CViHDKNN addresses the fundamental scalability limitation of traditional KNN, which requires loading the entire dataset into memory before computing distances between the query instance and all stored data points. In the CViHDKNN framework, this distance computation is parallelized across multiple worker nodes in the Hadoop cluster using HDFS-distributed data partitions, ensuring that no single node is required to process the complete dataset. Each worker node independently computes the distance between the query instance and the patient records stored in its locally assigned HDFS data shard, identifying the locally nearest neighbors within its partition. By harnessing Hadoop's parallel processing capabilities, CViHDKNN drastically improves scalability and enables efficient management of vast quantities of health-related patient data. This distributed architecture also enhances data security, as sensitive patient records remain within the distributed cluster environment rather than being transferred to external cloud servers or centralized local machines. The combination of clustering-guided search space reduction and Hadoop-distributed distance computation yields a system that achieves both computational efficiency and predictive accuracy, enabling real-time heart disease prediction on large-scale medical datasets. Experimental results confirm that the distributed implementation achieves a classification accuracy of 85.25%, representing a significant performance improvement over the traditional non-distributed KNN baseline, attributable directly to the distributed, clustering-enhanced processing strategy.

Classification

The classification stage of CViHDKNN assigns each query patient instance to a heart disease class based on the majority vote among its K nearest neighbors identified through the distributed search process. The choice of the K value directly influences classification outcomes and predictive precision. When K = 1, the query instance is assigned to the class label of its single closest neighbor, resulting in a highly localized decision boundary that may be sensitive to noise in the training data. When K = 3, the classification is determined by the majority class among the three nearest neighbors — for example, if two neighbors belong to Class 1 (no heart disease) and one belongs to Class 2 (heart disease present), the query instance is classified as Class 1, providing a more robust and noise-tolerant decision. The MapReduce reduce phase aggregates the locally identified nearest neighbors from all worker nodes into a globally ranked list, from which the K nearest neighbors are selected, and then computes the majority vote to produce the final class prediction. Performance of the CViHDKNN classification framework is evaluated using precision, recall, F1-score, and overall classification accuracy as primary metrics. The integration of cluster-constrained search with distributed majority voting produces finer and more accurate decision boundaries than standard KNN, reducing false negatives in at-risk patient identification and improving sensitivity, both critical requirements for clinically reliable heart disease prediction in large-scale distributed healthcare analytics environments.

Results

The experimental evaluation demonstrates that the CViHDKNN model achieves higher predictive performance than the CViHDDT model in heart disease classification. The CViHDKNN model achieved a test accuracy of 85.25%, whereas the CViHDDT model achieved 80.33%, indicating improved predictive capability for the distributed K-nearest neighbor approach. These comparative performance results are summarized in Table 1.

The classification performance of...

Discussion

An effective heart disease prediction method based on the clustering technique CViHDDT was developed and evaluated using a Hadoop-based distributed decision tree framework. The model achieved a training accuracy of approximately 75.62% and a testing accuracy of 80.33%, demonstrating its ability to generalize to unseen data. The slightly higher test accuracy suggests that Hadoop’s parallel processing efficiently handles large datasets while minimizing overfitting. Previous studies have similarly reported that Hadoop...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Apache Spark	Apache Software Foundation	3.x	Distributed data processing
HDFS	Apache Software Foundation	Included in Hadoop 3.x	Distributed file storage
YARN	Apache Software Foundation	Included in Hadoop 3.x	Resource management and job scheduling
Matplotlib	Matplotlib Development Team		Data visualization
Seaborn	Seaborn Developers		Statistical plotting
Ubuntu OS	Canonical Ltd.	20.04 LTS	Operating system
RobustScaler	Scikit-learn		Feature normalization
UCI Heart Disease Dataset	UCI Machine Learning Repository	Dataset ID 45	Experimental dataset