Augmented Multimodal Fusion for Optimized Brain Tumor Detection: Evaluation and Comparative Analysis

Summary

This study proposes an optimized brain tumor detection method by integrating and evaluating five pretrained models. The approach achieves enhanced accuracy and generalization by utilizing advanced image augmentation, hyperparameter tuning, and various optimizers, offering significant contributions to medical imaging for brain tumor diagnosis.

Abstract

Brain tumors represent a significant medical challenge, necessitating accurate and efficient detection methods for timely intervention. This work integrates several pretrained base models, such as VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50, to propose a novel method for brain tumor diagnosis. A streamlined and standardized technique has been proposed to accommodate various base models, ensuring consistency and ease of maintenance and facilitating model comparison. To amplify the variety of the training dataset and enhance model generalization, notable image augmentation methods like adjusting brightness and contrast are utilized. Further, an effective training pipeline utilizing data generators is designed to process large datasets efficiently while conserving computing power.

The study conducted a thorough analysis using three different optimizers (Adam, Stochastic Gradient Descent, and Adamax) applied to each pretrained base model, with comprehensive adjustments of hyperparameters. Metrics like recall, accuracy, precision, F1-score, and confusion matrices are used to evaluate the model's performance, providing a comprehensive understanding of the model's behavior. A systematic comparison of each model's performance provided an in-depth examination of strengths and weaknesses, facilitating informed model selection and decision-making for brain tumor detection applications. MobileNetV2 achieved the highest overall performance with an accuracy of 96%, precision of 96%, recall of 94%, and an F1-score of 95% using the Adam optimizer. DenseNet121 and VGG16 also performed well, achieving accuracies of 95% and 94%, respectively. InceptionV3 demonstrated a slightly lower performance compared to the top-performing models, with an accuracy of 93%, precision of 93%, recall of 91%, and an F1-score of 92%. ResNet50 showed relatively lower performance with an accuracy of 77%, precision of 78%, recall of 76%, and an F1-score of 76%. These metrics demonstrate the robustness and efficacy of the proposed method for brain tumor detection.

Introduction

Brain tumors, whether benign or malignant, pose a significant threat to human health and well-being. The timely detection of these tumors plays a pivotal role in determining patient outcomes, making it a priority in the medical field^1,39(also see Supplemental Table S1). Detecting brain tumors at an early stage is not only a vital medical objective but also a crucial lifeline for patients. By enabling timely intervention and access to effective treatment options, it fosters improved prognosis, empowers patients, and reduces the overall burden of this challenging condition⁴⁰. Continuous monitoring of disease progression, facilitated by techniques like MRI, is essential for adjusting treatment strategies^1,3 and detecting potential complications. Early detection not only improves patient survival but also enhances quality of life by offering hope for successful outcomes and better long-term prognosis.

Deep learning is instrumental in the detection of brain tumors, offering unparalleled capabilities in analyzing intricate medical imaging data^4,5,6 with remarkable precision and speed. Deep learning models, particularly those using advanced neural network structures such as convolutional neural networks (CNNs), are highly effective in autonomously identifying intricate patterns and features from a variety of medical imaging modalities, such as computed tomography (CT), positron emission tomography (PET), and MRI scans^7,41.

One of the key benefits of deep learning in identifying brain tumors is its capacity to detect them in the very early stages^8,9,10,11. By discerning subtle abnormalities in medical images that may evade human observation, deep learning algorithms enable healthcare providers to initiate treatment promptly, thus averting tumor progression^{12,15,17,18,19,20} and enhancing patient outcomes^15,17. This capability is critical for guiding treatment decisions and predicting patient prognosis^24,25,26. Deep learning models, trained on extensive datasets^27,28,29 can grasp intricate relationships between imaging features and tumor pathology, providing precise and dependable diagnostic insights^30,31,32,33.

Recent advancements in deep learning have introduced transformer-based architectures, initially designed for natural language processing, as game-changing tools in medical imaging. Unlike traditional CNNs, transformers utilize self-attention mechanisms to capture global data dependencies, making them particularly effective for complex tasks like tumor segmentation and classification. Studies such as those conducted by Saeed et al.⁴¹ and Mehmood et al.³⁸ have highlighted the success of transformer-enhanced models like DeepLabV3+ and TransResUNet, which have significantly improved segmentation accuracy and explainability in brain tumor detection. These models provide a deeper understanding of medical images, as demonstrated by Xuanzhi et al.⁴⁰, where transformers were applied for predictive modeling.

Their ability to map intricate spatial relationships with less dependence on large, labeled datasets is transforming the field, paving the way for more adaptive and scalable diagnostic solutions. This marks a significant shift in medical imaging, emphasizing precision and efficiency in detection and analysis. Detecting brain tumors using deep learning has emerged as a key area of focus in medical imaging, driven by progress in transfer learning and optimization methods¹. Despite these advancements, challenges remain, such as variability in model performance, insufficient exploration of optimization techniques, and limited assessment of hyperparameter tuning and data augmentation strategies.

Transfer Learning in Medical Imaging

Optimization Techniques: Existing optimization algorithms often struggle with balancing computational efficiency and convergence speed, especially in high-dimensional spaces. There is a need for adaptive and hybrid optimization methods that can dynamically adjust to different problem landscapes.

Data Augmentation and Preprocessing: Current data augmentation techniques may introduce unrealistic transformations that do not generalize well to real-world scenarios. More research is needed on adaptive augmentation strategies that consider domain-specific constraints and maintain data integrity.

Model Comparisons: Most model comparison studies focus on benchmark datasets, which may not reflect real-world complexities. There is a research gap in evaluating models across diverse, noisy, and imbalanced datasets to understand their true generalization capabilities.

Feature Extraction via Deep Learning Models: While deep learning models can automatically extract features, they often lack interpretability. More work is needed on explainable feature extraction methods that provide insights into the learned representations.

Optimization Strategies: Many optimization strategies do not fully leverage domain-specific knowledge, leading to suboptimal solutions. Research is required on incorporating domain-aware heuristics into optimization frameworks for improved performance.

Challenges in Data Augmentation: Data augmentation can lead to redundant or misleading information, negatively impacting model performance. Addressing the trade-off between augmentation diversity and data realism remains an open challenge in deep learning applications.

A critical analysis of the experimental work done in the proposed research reveals notable strengths and weaknesses. On the positive side, transfer learning models have proven highly effective at extracting complex features, enabling them to excel in tasks like brain tumor detection. Additionally, advanced optimization techniques, including the widely used Adam optimizer and newer methods, have demonstrated their ability to improve convergence rates and overall model performance.

However, certain limitations persist. There has been limited exploration of the interactions between optimizers and models, which could provide deeper insights into achieving optimal performance. Furthermore, insufficient focus on hyperparameter tuning has been identified as a drawback, as it plays a crucial role in ensuring reproducibility and consistency across different applications.

This paper begins with foundational work on feature extraction and transfer learning, progresses to advanced areas like optimization and augmentation, and concludes by identifying key research gaps. Several key gaps are highlighted in the existing literature. One significant issue is the inconsistent benchmarking of transfer learning models, where they are not uniformly evaluated across different optimizers, as noted in studies^6,36. Another gap lies in the limited exploration of hyperparameter tuning and its critical role in influencing model performance, as identified by research^14,37. Additionally, there is a lack of sufficient justification for the augmentation strategies employed in various studies, particularly regarding their impact on model robustness and generalization, as observed in works^15,23. These gaps point to the need for more standardized and rigorous approaches in future research.

To address these gaps, this study stands out for its comprehensive evaluation of five pre-trained models-VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50-paired with three optimizers (Adam, SGD, Adamax). By addressing gaps in benchmarking and exploring optimizer-model interactions, it offers a thorough analysis of performance dynamics. Rigorous hyperparameter tuning using grid search refines key parameters such as learning rates, batch sizes, and dropout rates, leading to enhanced model performance.

The study also introduces customized augmentation strategies, including brightness and contrast adjustments, to improve model generalization. A standardized evaluation framework using key metrics ensures a fair and consistent comparison across models. Notably, MobileNetV2 emerges as the top-performing model for brain tumor detection, providing actionable insights and setting a new benchmark for reproducibility and efficiency in medical imaging research.

The pretrained models included in the study were VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50, as they have been highly tested and proved effective in tasks such as image classification while applying their unique architectural strengths for diversity representation. VGG16 provides an easy yet powerful framework for extracting hierarchical features, while mobile NetV2 is lightweight with architectures suitable for resource-constrained environments. DenseNet121 works to improve the efficiency through feature reuse with a dense connection. It utilizes the Inception modules which capture features at various scales using InceptionV3. Resnet50, along with the residual connections, avoids vanishing gradients perfectly. All three optimizers were opted for SGD to benchmark it against optimization, Adam, due to its hybridization of momentum together with adaptive learning rate, giving it fast convergence, and Adamax that has been quite robust, particularly with sparse gradients.

This study brings a unique contribution to the field by combining the evaluation of multiple architectures, optimizer analysis, and hyperparameter tuning within a single comprehensive framework, significantly enhancing the existing knowledge base. Its systematic methodologies and detailed reporting ensure reproducibility, allowing other researchers to replicate the findings or use them as a foundation for further advancements. Moreover, the practical significance of this work is evident, as its results can directly guide the development of future medical imaging systems, driving improvements in both accuracy and efficiency for clinical applications.

The work done in this research focuses on the development of a streamlined and standardized codebase that accommodates different base models, ensuring consistency in implementation, ease of maintenance, and facilitating model comparison Further, a novel approach by integrating pre-trained base models, including VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50, to leverage their unique architectures for brain tumor detection is introduced. The work involves image augmentation methods, including brightness and contrast modifications, to increase the training dataset's diversity and enhance model generalization. The aim of this work is to design and implement an effective training pipeline that makes use of data generators to process huge datasets smoothly and train models effectively while conserving computing power. Thorough analysis of the proposed work is done using three different optimizers: Adam, Stochastic Gradient Descent (SGD), and Adamax applied to five distinct pre-trained base models (VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50). Finally, an in-depth assessment of the model's performance is conducted by examining key metrics such as recall, accuracy, precision, F1-score, and confusion matrices to acquire a comprehensive understanding of the model's behavior.

Protocol

Figure 7 illustrates a schematic block diagram representing the overall methodology of the proposed approach. In the suggested method, MRI scan images of brain tumors are preprocessed using two popular techniques-normalization and scaling. Some popular pretrained transfer learning models like VGG16, MobileNetV2, DenseNet121, ResNet50, and InceptionV3 are utilized in this work. After training, the models are tested on hypothetical cases, and an extensive analysis is done to check the models' performance based on various performance metrics such as accuracy, recall, precision, and F1-score. The proposed technique uses deep learning to enhance the efficiency of brain tumor classification and its accuracy, which could improve the medical diagnosis process in the future.

1. Dataset acquisition

1. Acquire the dataset: Download the brain tumor MRI scan dataset from Kaggle, consisting of 7023 images categorized into four classes: pituitary, no-tumor, meningioma, and glioma.
2. Divide the dataset: Split the dataset into a training set (5,712 images) and a testing set (1,311 images).
   NOTE: Ensure that the class distribution is balanced across training and testing sets for unbiased model evaluation. Figure 8 illustrates the visual distribution of images among the various categories within the training dataset. Some sample datasets of MRI images from four different classes can be seen in Figure 9.

2. Data preprocessing

1. Image transformation:
   1. Convert each MRI image to a numerical representation using the Python Imaging Library (PIL).
      1. Ensure PIL (Pillow) and numpy are installed.
      2. Open the MRI scan using PIL.
      3. If needed, convert the image to grayscale to simplify processing.
      4. Convert to NumPy Array - Transform the image into a numerical format.
2. Data augmentation:
   1. Brightness adjustment: Use the ImageEnhance.Brightness function to randomly alter the image brightness by a factor ranging from 0.8 to 1.2.
   2. Contrast adjustment: Apply the ImageEnhance.Contrast function to randomly alter the image contrast within the range of 0.8 to 1.2.
      NOTE: These augmentation techniques are implemented to simulate variations in real-world imaging conditions, enhancing model robustness.
3. Normalization: Convert the augmented images to NumPy arrays by using PIL library to load the image and apply augmentation (e.g., flipping, rotation). Further, transform the augmented image into a numerical representation (img_array = np.array(img_aug). Depending on the ML model, reshape or flatten the array (img_vector = img_array.flatten()). Finally, normalize them by dividing pixel values by 255.0 to ensure they fall within the range [0, 1].
   NOTE: This step ensures consistent input for the model, facilitating stable training and performance.

3. Transfer learning for tumor classification

1. Model selection: Select pretrained models, including VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50 for transfer learning.
2. Feature extraction: Utilize pretrained models as feature extractors by setting the majority of their layers as non-trainable. Refer to the below line of code to set the layers as non-trainable. for layer in base_model.layers[:15]:
   layer.trainable = False
   NOTE: This step leverages the hierarchical representations learned from large, diverse datasets for better generalization on the target medical dataset.
3. Finetuning: Unfreeze the last three layers of each pretrained model to adapt to the specific task of brain tumor detection using the below code snippet:for layer in base_model.layers[-3:]: layer.trainable = True
   NOTE: Finetuning allows the model to focus on features pertinent to the detection of brain abnormalities.

4. Deep learning models for tumor classification

1. Standardized model structure:
   1. Add a Flatten layer to each model to transform hierarchical features into a one-dimensional array After flattening, add Dense layers for classification or regression tasks.
   2. Include Dropout layers (with rates of 0.3 and 0.2) to reduce overfitting and improve generalization.
2. Dense layers:
   1. Add a dense layer with 128 neurons using Rectified Linear Unit (ReLU) activation.
   2. Include a final dense layer with softmax activation for multi-class classification of brain tumor images.
      NOTE: This uniform structure across models ensures consistent evaluation and comparison.

5. Training and optimization

1. Input specifications: Standardize the input dimensions of images to (224, 224, 3) for all models. Set the include_top parameter to False, excluding densely connected classification layers and using the models solely as feature extractors.
2. Weight initialization: Use imagenet weights for initialization across all models to promote fair and meaningful comparisons.
3. Compilation: Compile all models using the sparse_categorical_crossentropy loss function and sparse_categorical_accuracy metric.
   NOTE: The selected loss function is appropriate for multi-class classification when labels are integers.
4. Optimization strategy:
   1. Batch sizes: Test batch sizes of 16, 32, and 64 to determine the optimal size for training speed and performance.
   2. Learning rates: Use a learning rate of 0.0001 for the Adam optimizer and 0.001 for SGD and Adamax optimizers.
   3. Dropout rates: Evaluate dropout rates of 0.2, 0.3, and 0.5 to prevent overfitting.
      NOTE: These parameters are tuned for each model to ensure optimal performance.
5. Training parameters: Set a uniform batch size of 20 and train for 50 epochs across all models.
   Implement early stopping based on epoch count to prevent overfitting and unnecessary computational load.
   NOTE: This standardized procedure enables fair comparison of model performance.

6. Model evaluation

1. Performance metrics: Evaluate each model using metrics such as accuracy, precision, recall, and F1-score.
2. Confusion matrices: Construct confusion matrices for a detailed analysis of misclassifications across classes.
   NOTE: These evaluations provide insights into the strengths and weaknesses of each model, guiding further improvements.

Results

The performance evaluation of each of the five models-VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50-includes a detailed analysis with accuracy curves, confusion matrices, and classification reports. These metrics and visualizations are important tools for evaluating the models' proficiency in brain tumor detection. The models' learning progress over epochs is dynamically represented by the accuracy curves, which also shed light on possible overfitting and convergence of the m...

Discussion

Model performance

The effective use of deep learning models (pretrained) is the primary reason for the strong performance in brain tumor identification. A combination of prominent architectures used to evaluate the performance are VGG16, MobileNetV2, DenseNet121, InceptionV3, and ResNet50. These models have impressive feature extraction abilities since they underwent pretraining using extensive collections of image data. The weights have been ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
GPU Infrastructure and Jupyter Notebook	Kaggle (Online Platform Provider)	Tesla-P100	GPU infrastructure provided by Kaggle’s Jupyter Notebook service, utilizing the latest NVIDIA Tesla P100 GPU
Intel Core i5-8350U	Intel	i5-8350U	Intel Core i5-8350U CPU @ 1.70GHz (1.90 GHz max) for local processing
RAM	Standard	8GB_RAM	8 GB of RAM for data processing
Operating System	Microsoft	Windows_64	64-bit Windows OS for local processing
Python Libraries	Python Community	N/A	NumPy, scikit-learn, PIL, TensorFlow 2.12, Keras 2.12