Convolutional Neural Network-based Framework for Brain Tumor Classification and Segmentation using Magnetic Resonance Images

Summary

Deep learning algorithms were utilized in MRI to perform brain tumor classification and segmentation with U-Net. InceptionV3, DenseNet201, and Inception-ResNet-v2 performed with excellent accuracy on tumor type and grade prediction. GPT-4.0 augmented hybrid models for automatic medical report generation and diagnostic assistance.

Abstract

Early diagnosis of brain tumors is critical for optimization of the prognosis and treatment selection of the patient. Accurate segmentation and categorization of brain tumors are essential to create specialist treatment techniques. As MRI utilization for brain diagnosis increases and computer vision technology also improves, having a good and effective model to identify and categorize tumors based on MRI scans remains challenging. To address this problem, the authors suggested a deep learning-based technique to segment and classify brain tumors from different datasets. Image preprocessing employed nine augmentation methods to enhance model performance. Segmentation of MRI was done by using a U-Net model.

The developed classification model based on InceptionV3 and DenseNet201 predicts the existence of the tumor and categorizes it into Glioma, Meningioma, and Pituitary. With 99.15% accuracy, InceptionV3 is higher than DenseNet201's 98.75% in tumor classification. Additional tumor classification was performed by Clustering as HGG and LGG on the basis of Inception-ResNet-v2. Tumor grades (1-4) are identified with 96.64% accuracy by Inception-ResNet-v2. An autonomous system integrates hybrid models with GPT-4.0 to generate reports. Hence, this novel framework could very well be suitable for clinics when used for automatically identifying and separating brain tumors utilizing input images captured from MRI scans.

Introduction

Brain tumors may significantly impair patients' and families' quality of life and account for one instance of every 100 cancers treated each year in the US^1,2,3. Glioma is the most prevalent primary brain tumor among people in the United States, which has an incidence of 6.5 per 100,000. They arise in astrocytes, oligodendrocytes, and ependymal cells, the glial cells that provide nutrition to the neurons of the brain. Glioma is categorized into different types on the basis of the affected glial cell in the tumor as well as its genetic profile, which may now be useful for anticipating the behavior the tumor may undergo over time and the treatments most likely to be effective^4,5,6. A glioma can disable the functioning of the brain and even be lethal based on its location and growth rate. Meningiomas form 15-20% of all primary brain tumors. Their prevalence in normal screening is one per every 100 individuals in an operative series. They are assumed to originate from the arachnoid cap cell. Meningiomas display a broad spectrum of behavior, ranging from benign to extremely aggressive, based on their location. They show up with many varieties of symptoms but most of the patients remain asymptomatic.

The associated symptoms and signs are mainly caused by compression of the adjacent structures, and thus they can also present with symptoms like motor seizures, sensory impairment, disturbance in speech, anosmia, and many more. Pituitary tumors are the third most frequent primary intracranial neoplasms, following gliomas and meningiomas, accounting for around 10% of all excised primary brain neoplasms. They may occur in many ways due to hyper- or hypo-secretion of pituitary hormones, constriction of the pituitary stalk, or effects on surrounding structures, particularly the optic chiasm.

When compared to standard brain CT scans, MRI images contain a large tissue contrast range^7,8,9. Development of a regular automatic brain tumor segmentation method, thus, would be of utmost significance in diagnosing and treating these patients properly. Therefore, the development of a reliable automatic technique for brain tumor segmentation would be of extreme importance in diagnosing and treating these patients properly. These intelligent methods would assist the neurosurgeons and radiologists to better assess volume, shape of the tumor, and define the boundary between the tumor and adjacent normal tissues more accurately.

Machine learning could be beneficial to determine the type and presence of brain tumors but requires more human intervention since its models are predictive. Deep learning models, in contrast, could learn and detect features because of neural networks, which would be the crux of a fully automated whole detection process. Many Machine Learning (ML) methods have been developed for radiologists so that they are given unusual visions during recognition and classification of MR images. The most effective method among a whole segment of modalities for cancer detection is through medical imaging. These methodologies are helpful in identifying and detecting malignant neoplasms. This methodology is crucial because it does not have the invasive nature associated with other medical practices. The procedures are not invasive^10,11,12. The novelty of this work is that it offers a unique DL-based framework for automatic, accurate and efficient multi-class categorization and grading of brain tumors using MRI images, which reduces the need for manual interpretation.

The authors proposed brain tumor classification models using nLBP and LBP feature extraction approaches. Models precisely characterized the most common types of brain cancers. The maximum accuracy of 95.56% was achieved using a feature extraction scheme nLBPD = 1 with the KNN model¹¹. The authors surveyed clinical deployment of Deep Learning (DL) in radiography and documented operations involved within this domain¹³. They emphasized the clinical implications of DL in several clinical fields. DL has demonstrated satisfactory results in certain radiological implementations, but technology is immature and cannot replace the diagnostic profession of a radiologist¹⁴. Combining DL algorithms with radiologists improves diagnostic effectiveness and efficiency. MRI has been evaluated in numerous investigations for its possible application in the classification of brain tumors via various research designs.

Afshar et al. suggested an improved CapsNet architecture for the classification of principal brain tumors with 3,064 images with auxiliary input regions of interest through application of tumor borders to invest greater effort and to improve on other methods to an accuracy of 90.89%¹⁵. Gumaei et al. have suggested a hybrid feature extraction-based method for the classification of brain tumors using RELM. The authors normalized the brain image using min-max normalization and used RELM for classification to attain an accuracy of 94.23%¹⁶. An integrated scheme using segmentation and brain tumor classification using MRI was proposed by Rezaei et al. Steps used were the removal of noise, segmentation through Support Vector Machine (SVM), extraction of features, and feature selection through using DE. The slices of the tumor were classified by using WSVM, KNN, and HIK-SVM classifiers. The classifiers, with ensemble methods based on MODE, achieved an accuracy of 92.46%¹⁷. Fouad et al. proposed a classification of a brain tumor through HDWT-HOG feature descriptors and WOA for reducing features. The method used ensemble methods with Bagging. With Bagging, an average accuracy of 96.4% was attained, and with Boosting, the value achieved is 95.8%¹⁸.

Ayadi et al. introduced the process of classifying brain tumors using techniques such as normalization and dense accelerated robust features with gradient histogram, boosting quality image searching, and resultant discriminative feature development. SVM classifier was employed, and the respective accuracy reached a high level of up to 90.27% with the evaluation dataset¹⁹. Srujan et al. presented a sixteen-layered Convolutional Neural Network (CNN) DL architecture that combines activation functions like ReLU and Adam optimizer to achieve 95.36% classification accuracy²⁰. Tejaswini et al. designed a CNN model for the diagnosis of meningioma, glioma, and pituitary brain cancers, which achieved a validation accuracy of 87.16% and training accuracy of 92.79%. The tumor area was also segmented by Otsu thresholding, fuzzy cmeans, and watershed techniques²¹. Huang et al. presented CNN-BCN for brain cancer classification. The network architecture was designed using the random graph method, and it achieved the target accuracy of up to 95.49%²². Ghassemi et al. designed a DL model for the classification of brain tumor images²³. Powerful features and learning patterns were obtained from MR images by utilizing pre-trained networks as GAN discriminators. Based on fivefold cross validation, the approach reached 95.6% accuracy by substituting techniques like data augmentation and dropout for fully connected layers. Deepak et al. combined SVM with CNN for classifying brain tumor images. Upon testing with a fivefold cross-validation protocol, the automated system achieved 95.82% accuracy, superior to other techniques²⁴. Noreen et al. employed the pre-trained and fine-tuned networks like Xception and InceptionV3 for brain cancer identification. These models employed a wide range of machine learning techniques like RF, SVM, and KNN to achieve 94.34% accuracy with an ensemble of InceptionV3²⁵.

Shaik et al. classified brain tumors in medical image processing and presented a manet approach that prioritizes tumors by combining space and cross-channel attention while maintaining temporal linkages between channels. On the primary brain tumor classification task, the approach achieved 96.51% accuracy²⁶. Ahmad et al. created a deep generative neural network for classifying brain tumors. The technology used VAEs and GANs to achieve 96.25% accuracy on MR images of the brain tumor^27,28. A DL model was presented by Alanazi et al. to identify brain tumor subtypes. The technique involved creating multiple CNN models and applying transfer learning to fine-tune weights of a 22 layer CNN model. The model achieved MRI image accuracy of 95.75 percent and 96.89 percent, respectively^29,30. Almalki et al. applied an ML technique to MRI to swiftly analyze the severity of four types of brain tumors. Breaking down the MRIs into 8 x 8-pixel images allowed them to extract Gaussian and nonlinear scale characteristics as well as tiny details. The important characteristics were identified, divided into 400 nonlinear scale features, and merged with each MR image. They used the SVM classifier and attained 95.33% accuracy^31,32. Three CNN models were compared by Kumar et al., namely, InceptionV3, AlexNet, and ResNet50, for the classification of primary tumor, and used data augmentation methods. AlexNet beat others in terms of specificity and accuracy, earning a score of 96.2%³³.

Ullah et al. delve into deeper learning models for brain tumor diagnosis. DeepEBTDNet improves MRI quality by applying DSIHE and ensuring interpretability through LIME^34,35. TumorResNet with 20 convolutional layers, which is used to achieve a 99.33% classification accuracy, presents a consistent, automated method of early brain tumor diagnosis and treatment planning³⁶. Multiple advanced DL architectures have been presented by Kumar et al. for brain tumor classification and detection. The PBTC model combines MRI pre-processing, ACLS segmentation, and OHBO-optimized HRNN-BiLSTM to achieve 97.8% accuracy³⁷. A two-channel CNN with the Berkeley Wavelet Transform and Enhanced Serval Optimization reaches 98.8% accuracy for four types of tumors³⁸. Disci et al. use MRI data to evaluate pre-trained DL models for classifying brain tumors and achieves 98.73% accuracy with Xception³⁹.

Protocol

Dataset description and exploratory analysis
The dataset consists of multiple sources to enhance the reliability and accuracy of the model. Merged_dataset contains 20,620 images from Dataset A (3,054), Dataset B (3,264), Dataset C (10,000), and Dataset D (4,292). Further, 1,425 images from the Brad data set were added for Glioma tumor grades (HGG, LGG). This diverse dataset will ensure better generalization, reduce biases, and improve the performance of the model. The large dataset enables comprehensive evaluations and thus, there is a higher probability of getting the right predictions in real-world applications of classification tasks.

Two distinct datasets were used in this investigation. Between 2005 and 2010, the first was acquired from Tianjing Medical University's Nanfang Hospital and General Hospital in China. T1-weighted contrast-enhanced images from 233 individuals with tumors, as well as Glioma Grades II and III, are included in the collection. The dataset offers a very comprehensive distribution of tumor categories and general data analysis. For High-Grade Glioma (HGG), there are 1,050 images available, and for Low-Grade Glioma (LGG), there are 375 images, meaning there is more attention given to more severe cases (HGG).

Figure 1 shows MRI scans that are categorized into three brain tumor types. Each row is one type of tumor, and there are three sample images for each category. The scans have different characteristics in various orientations and views, including axial, sagittal, and coronal. Glioma samples have irregular structures that infiltrate the brain tissues. Meningioma samples appear as localized masses near the surface of the brain, and pituitary tumor samples are located near the center of the brain. These examples present variability in tumor appearance, thus helping train ML models for accurate detection as well as classification of tumors in medical image analysis.

Figure 1: MRI scans of tumors. MRI scans of meningioma, glioma, and pituitary tumors in axial, sagittal, and coronal views, highlighting distinct imaging characteristics. Please click here to view a larger version of this figure.

Glioma has the highest number of images, which is just above 6,000. Meningioma has around 6,000 images, while Pituitary has almost as many images as Meningioma. There seems to be an almost equal split between the three categories, meaning that each of the tumor types is represented well for training or any analysis. Glioma has the biggest median file size, although meningioma has a more constrained range and more outliers. The pituitary's moderate variance contains a few apparent outliers. Each category contains some really huge file sizes.

Figure 2 shows the PCA of image features for the three tumor types. Principal component 1 captures the most variance and significantly separates the pituitary (right cluster) from the others. Gliomas and meningiomas overlap down the left, indicating that their feature representations are comparable and that their separability is diminished.

Figure 2: PCA of image features. Principal component analysis of extracted features differentiating glioma, meningioma, and pituitary tumor classes. Please click here to view a larger version of this figure.

The correlation heatmap shown in Figure 3 depicts the relationships between image metadata attributes: File_Size, height, and width. File_Size shows a very low correlation with both height (-0.01) and width (0.0039).

Figure 3: Correlation heatmap of metadata. Heatmap showing relationships among image metadata attributes such as file size, height, and width. Please click here to view a larger version of this figure.

Height and width also show a very low positive correlation of 0.0039. The diagonal values are 1, indicating perfect correlation of each variable with itself. In general, the attributes are mostly uncorrelated, which means independent variations among file size, image height, and width.

Methodology and proposed architecture
A workflow that systematically detects, classifies, and analyzes brain tumors based on MRI data is represented in Figure 4. The methodology includes advanced preprocessing, segmentation, and DL techniques to achieve robust tumor prediction and classification.

Figure 4: Workflow of proposed methodology. Stepwise workflow for tumor detection, classification, and analysis using MRI data. Please click here to view a larger version of this figure.

The process initiates with integrating a number of datasets (denoted as A, B, C, and D). Then, the datasets are combined into one comprehensive dataset to gain holistic data. It also involves the Brad Dataset for image segmentation, clustering, and classification purposes. The merged dataset involves appending all images in each dataset and combined directories, ensuring a comprehensive and unified dataset for further analysis and multi-classification model development.

Second, preprocessing is completed, which is an important step in improving the quality and variability of the data. Nine augmentation processes are applied to the original MRI images to improve model performance and dataset variety. Image resizing to a common size harmonizes them, and RGB to grayscale conversion simplifies processing. Horizontal/vertical flipping improves robustness, and rotation with zooming mimics different imaging conditions. Use of Sobel filtering improves feature acuteness, and noise addition improves variability. Unsharp masking improves image acuteness, and height and width shifting add position variations. Each of them, when combined, improves the model's generalizability and classification accuracy. All these guarantee that the model generalizes over a broad spectrum of imaging situations. There is a total of 20,620 MRI images, and post preprocessing total images used for model development yields 185,580 images.

Then, the merged dataset is segmented using the U-Net model, which is highly utilized in medical imaging. The architecture is fine-tuned and trained to identify tumor regions from MRI scans. This step produces masked images, which highlight the tumor and eliminate irrelevant details. Segmenting the images ensures better data inputs for classification and analysis.

In next step, these models are fine-tuned to categorize images into different classes, leveraging transfer learning to enhance accuracy while reducing training time. For classification, a number of pretrained deep learning models are used, each with special benefits. Because of its ease of use and efficiency in image categorization, VGG16 and VGG19 are frequently utilized. With optimal computational efficiency, EfficientNetB0 and EfficientNetB7 offer cutting-edge performance. ResNet101's deeper architecture improves classification accuracy by effectively capturing intricate patterns. Afterwards, accurate diagnosis is ensured by the trained algorithms' classification of MRI data into four tumor classifications. Glial cells are the source of glioma tumors, which impair brain function. Meningioma tumors develop in the meninges, which are protective layers of the brain and spinal cord. The pituitary gland is the site of pituitary tumors, which affect hormone balance and physiological processes.

Then, to help with accurate diagnosis, clustering techniques such as K-Nearest Neighbors (KNN) are used to forecast tumor grades. Grades three and four of High-Grade Glioma (HGG) denote severe tumor development. Grades 1 and 2 of Low-Grade Glioma (LGG) are slower-growing, less aggressive tumors, respectively. This grading is essential to establish the aggressiveness of the tumor and, therefore, direct clinical management.

Finally, to assess the categorization models' efficacy, they are contrasted according to important criteria. Their effects are evaluated to ascertain whether augmentation and segmentation strategies have improved model performance. To guarantee dependability and effectiveness, performance is also examined over a variety of datasets, and classification accuracy is contrasted with the most advanced models.

The workflow integrates data preparation, preprocessing, segmentation, classification, and grading in a coherent framework. It enhances the accuracy of brain tumor detection and classification through the application of cutting-edge DL methods and rigorous preprocessing. The inclusion of tumor grading further supports clinical decision-making, making this a comprehensive system for tumor analysis.

Figure 5: Deep learning framework. Proposed brain tumor classification framework using deep learning models. Please click here to view a larger version of this figure.

Figure 5 is an architecture in relation to a medical image classification system that identifies brain tumors with the aid of advanced computational techniques such as deep learning. It begins with a collection of MRI images that is being classified. Thus, this will be the backbone for the classifying process. Then, input data enter a data preprocessing module wherein the normalization, or resizing and image augmentation, as well as the removal of noise, are undertaken. Preprocessing standardizes input images to make the most of the model's training phases. The entire step is about getting ready to undergo subsequent computationally intensive stages. Following the preprocessing step, the data are fed to the feature extraction module. Here, the use of computational techniques or models, perhaps DL models such as CNNs, can be made to identify features in images. The features that are extracted could include patterns indicative of abnormalities or tumor characteristics. Then, this set of features is input into the classification model. Given that this would probably be aimed at distinguishing different classes, which include the existence or nonexistence of a tumor and the type of tumor, the model relies on a rather complex architecture, probably of Inception-ResNet-v2, among others, for accurate predictions. It determines whether or not a tumor exists in the scanned brain image. This classification step is into "Tumor" and "No Tumor." When a tumor is found, the system classifies the tumor type as Glioma, Meningioma, or Pituitary. Each of these types has characteristics that the model recognizes using the features extracted. For the tumors identified, the grade of the tumor is then determined. Grades range from 1 to 4, indicating the severity and progression of the tumor. Grade 1 is the least aggressive, while Grade 4 is the most severe. This classification helps in medical diagnosis and treatment planning. Final outputs include the absence or presence of tumor and its type and grade. Such output is very valuable for clinical applications for decision-making in the care and treatment of patients.

The process represents an integrated pipeline for medical image analysis and classification of tumor types. Starting with raw data, it goes on in succession through preparatory data, extraction of features, and classification. The system determines whether it indeed has a tumor, specifies its type, and gives the severity or grade. This pipeline, using more complex forms of computation, streamlines the diagnosis in medical imaging so that the assessments can be quicker and more accurate in brain tumors. A modular workflow ensures flexibility, where components can be optimized separately for enhanced performance.

Results

The training environment leverages Kaggle's NVIDIA Tesla T4 GPU, facilitating efficient model training. The libraries of importance are TensorFlow, PyTorch, Keras, NumPy, and Pinecone, which facilitate strong deep learning pipelines. DenseNet201, InceptionV3, and Inception-ResNet-v2 were chosen because of their demonstrated efficacy in medical imaging. These designs provide deep feature extraction, robust gradient flow, and hybrid strengths, which improve accuracy, minimize overfitting, and improve model generalization i...

Discussion

Early diagnosis of brain tumors may be essential to save the life of an individual, as brain tumors can be highly dangerous and fatal. Currently, tumor diagnostics rely on radiologists' manual interpretation, which can cause delays and human error in detecting malignancies in early stages. Hence, this paper introduces a multi-classification brain tumor diagnosis model that can precisely detect, localize, and classify tumors, especially when dealing with varying shapes, sizes, and textures. It combines hybrid methods and ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
fastText	Facebook AI	N/A	Word representation and classification
Google Colab	Google	N/A	Cloud-based Jupyter Notebook environment
Google Colab GPU/TPU	Google	N/A	Cloud-based hardware acceleration
Intel Core i5/i7 or AMD Ryzen 5/7	Intel / AMD	N/A	Processor for local execution (if required)
Matplotlib	Open-source	N/A	Data visualization library
NLTK	Open-source	N/A	Natural Language Toolkit for text processing
NumPy	Open-source	N/A	Numerical computing library
NVIDIA GTX 1650 or Higher (Optional)	NVIDIA	N/A	GPU for deep learning tasks
Pandas	Open-source	N/A	Data manipulation library
Python	Python Software Foundation	N/A	Programming language for ML and NLP
PyTorch	Meta AI	N/A	Deep Learning framework
RAM (8GB Minimum, 16GB Recommended)	Various	N/A	Memory requirement for ML tasks
Scikit-learn	Open-source	N/A	Machine Learning library
Seaborn	Open-source	N/A	Statistical data visualization
SpaCy	Explosion AI	N/A	Industrial-strength NLP library
SSD Storage (256GB Minimum, 512GB Recommended)	Various	N/A	Storage for dataset processing
TensorFlow	Google	N/A	Deep Learning framework