A Few-shot Multimodal Deep Learning Framework For Precision Diagnosis Of Parotid Gland

Summary

This article presents a few-shot multimodal deep learning framework for accurate classification of parotid gland tumors using magnetic resonance imaging sequences.

Abstract

Accurate differentiation between benign and malignant parotid tumors is imperative for patient prognosis. However, due to the high degree of heterogeneity of parotid tumors and the limited availability of imaging data, preoperative diagnosis remains a significant challenge. To address this issue, the present study proposes a multimodal deep learning framework based on small-sample learning. This framework integrates multi-sequence MRI information and incorporates a multi-scale spatial attention mechanism to improve feature extraction and classification performance in situations where samples are limited. The model was systematically evaluated using a retrospectively collected cohort of 198 patients with histopathologically confirmed primary parotid gland tumors (107 benign, 91 malignant) who underwent complete preoperative MRI—including T1-weighted, T2-weighted, and diffusion-weighted imaging—at our institution, with no prior treatment for the parotid lesion. The cohort was split at the patient level into training (70%), validation (15%), and test (15%) sets using stratified random sampling to preserve the benign-to-malignant ratio, ensuring mutually exclusive subsets and preventing data leakage. The model achieved an area under the curve (AUC) of 0.9822. This finding underscores the system’s capacity to differentiate between benign and malignant tumors. Moreover, the model exhibited substantial advantages in terms of diagnostic accuracy when compared to experienced radiologists, underscoring its potential application in preoperative benign-malignant differentiation and clinical decision support for parotid tumors.

Introduction

Epidemiological data indicate that the incidence of parotid gland tumors is increasing. The parotid gland, the largest salivary gland in the human body¹, is one of the high-incidence sites for head and neck tumors. While parotid gland tumors are relatively uncommon, accounting for approximately 5% of head and neck tumors, benign lesions far outnumber malignant ones, constituting about 80% of all parotid gland tumors^2,3. Benign and malignant parotid tumors exhibit significant differences in biological behavior, treatment strategies, and prognosis. Benign tumors typically grow slowly, exhibit expansile growth, and rarely metastasize. Favorable outcomes follow complete surgical resection. In contrast, malignant tumors demonstrate distinct invasive growth patterns and frequently invade surrounding tissues, such as the facial nerve, skin, and bone. They are also capable of distant metastasis, resulting in a poorer prognosis^4,5.

Nevertheless, preoperative differentiation between benign and malignant parotid tumors remains a clinical diagnostic challenge. Due to the absence of specific clinical manifestations, the initial symptom of most parotid tumors (regardless of malignancy) is a painless mass, which complicates differentiation based solely on signs or symptoms^6,7. Although symptoms such as pain or facial paralysis are often considered potential indicators of malignancy, they are uncommon in early malignant lesions. Conversely, some benign tumors with inflammatory responses may present similar manifestations. Preoperative evaluation of parotid tumors is primarily achieved through imaging studies, which represent the primary noninvasive method for such evaluation. However, it should be noted that each imaging modality has its limitations. Magnetic resonance imaging (MRI) is regarded as the optimal modality for parotid tumor evaluation due to its exceptional soft tissue resolution, with studies demonstrating a maximum accuracy of 93% in differentiating benign from malignant lesions⁸. However, conventional sequences exhibit significant overlap in signal characteristics⁹. CT is primarily used to assess bone involvement; however, its sensitivity is lower than that of MRI when it comes to distinguishing benign from malignant tumors^10,11. Ultrasound is a viable modality for preliminary evaluation of superficial lesions; however, its efficacy is contingent upon the operator’s expertise¹². PET-CT is characterized by its high metabolic sensitivity; however, its high cost and false-positive rate limit its use in initial diagnosis, making it more suitable for staging and follow-up of malignant tumors. Consequently, single imaging modalities are often unable to achieve precise diagnoses, and preoperative differentiation between benign and malignant lesions typically relies on a comprehensive analysis of multimodal information.

In recent years, the field of artificial intelligence (AI), particularly the subset of deep learning (DL) technologies represented by convolutional neural networks (CNNs), has emerged as a significant contributor to advancements in tumor imaging diagnosis. This is due to the powerful image feature extraction and pattern recognition capabilities afforded by these technologies. Deep learning models automatically extract complex multi-scale features through multi-layered neural networks, significantly reducing reliance on manual feature design while enhancing diagnostic performance. A substantial body of research has validated their potential in multimodal medical imaging. Gu Jionghui et al.¹³ developed a multimodal deep learning system integrating breast ultrasound and MRI that effectively predicted response to neoadjuvant chemotherapy, outperforming traditional methods. Lu G et al.¹⁴ achieved precise differentiation between benign and malignant breast tumors by integrating ultrasound, mammography, and MRI images with transfer learning strategies, yielding an AUC of 0.947. Horasan A et al.¹⁵ performed fusion analysis of prostate multiparametric MRI (mpMRI) using 3D CNNs with ResNet and Inception architectures, achieving approximately 91.3% accuracy. In parotid imaging, Zhang G et al.¹⁶ constructed a deep learning model integrating ultrasound and clinical data for benign-malignant differentiation; Hu X and Wang H.¹⁷ achieved 92.3% classification accuracy (AUC=0.96) using aResNet50 model based on CT images. The collective evidence from these studies indicates the substantial benefits of employing deep learning models for the analysis of parotid and other solid tumor images.

However, traditional deep learning methods frequently depend on substantial datasets to attain optimal performance. Parotid cancer, as a low-incidence tumor, possesses comparatively limited data¹⁸. Additionally, parotid tumors demonstrate high imaging heterogeneity, marked by intricate variations in signal intensity, morphological structure, and relationships with surrounding glandular tissue. These features span diverse spatial scales—for example, global tumor morphology and boundary definition, as well as local characteristics such as internal signal heterogeneity and infiltrative patterns—making it challenging for single-scale or simple feature extraction methods to fully capture key distinctions between benign and malignant lesions. Furthermore, previous studies have often relied on single MRI sequences (e.g., structural or functional imaging alone), which limits the ability to obtain comprehensive representations of tumor characteristics, as reported in prior work^19,20.

To address these challenges, we posit that a small-sample multimodal deep learning framework that integrates multi-sequence MRI and a multi-scale spatial attention mechanism can effectively differentiate benign and malignant parotid tumors under limited data conditions, and outperform conventional approaches and radiologist assessment. Specifically, this framework employs a few-shot learning²¹ strategy to learn discriminative features from a limited number of representative benign and malignant cases. The model utilizes a spatial attention mechanism to focus on key lesion regions, thereby improving the model’s accuracy in distinguishing benign from malignant parotid tumors. The model was evaluated using a retrospectively collected cohort of 198 patients with pathologically confirmed parotid gland tumors at our institution. This study integrates structural (T1WI, T2WI) and functional (DWI) MRI sequences to support morphological and functional information. The proposed framework is designed to facilitate preoperative differentiation between benign and malignant parotid tumors.

Protocol

All patient data used in this study were collected exclusively at Wuxi People’s Hospital and consisted of 198 pathologically confirmed parotid tumor cases. This study was approved by the Institutional Review Board of Wuxi People’s Hospital (IRB No. KY24068) and conducted in full accordance with institutional guidelines and international ethical standards. All data were fully anonymized prior to analysis.

MRI Image acquisition

All MRI sequences (T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging) were acquired on a 3.0 Tesla MRI scanner (Siemens MAGNETOM Verio or Prisma) using a 16-channel head and neck coil. The acquisition parameters were as follows: T1-weighted images were obtained with TR/TE = 500/15 ms, slice thickness = 3 mm, matrix = 256 × 256, field of view = 240 mm; T2-weighted images were acquired with TR/TE = 4000/90 ms, slice thickness = 3 mm, matrix = 320 × 320, field of view = 240 mm; diffusion-weighted images were obtained with TR/TE = 5000/80 ms, b-values = 0 and 1000 s/mm², slice thickness = 3 mm, matrix = 128 × 128, field of view = 240 mm, with apparent diffusion coefficient maps automatically generated. All sequences were acquired in the axial plane with no interslice gap.

Study Design

This retrospective study analyzed preoperative MRI data from 198 patients with histopathologically confirmed primary parotid gland tumors (107 benign and 91 malignant). All patients underwent complete parotid MRI examinations at our institution, including T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging sequences, with no prior treatment for the parotid lesion. Use surgical resection specimens or core biopsy serving as the reference standard. The inclusion criteria were a histopathologically confirmed primary parotid gland tumor, preoperative MRI with all required sequences available at our institution, and no prior surgery, radiotherapy, chemotherapy, or ablation for the parotid lesion. Exclusion criteria comprised metastatic tumors to the parotid gland, significant motion artifacts or incomplete sequences rendering images non-diagnostic, and recurrent tumors after previous treatment. The overall cohort consisted of 118 males (59.6%) and 80 females (40.4%) with ages ranging from 26 to 89 years (mean ± SD 53.0 ± 12.8 years), and demographic characteristics stratified by tumor type were as follows: benign group (n=107) with 59 males (55.1%) and 48 females (44.9%), age range 28–87 years (mean ± SD 54.2 ± 12.1 years), and malignant group (n=91) with 59 males (64.8%) and 32 females (35.2%), age range 26–89 years (mean ± SD 51.6 ± 13.4 years); a stratified random sampling approach was employed to partition the 198 patients into training (70%, n=138), validation (15%, n=30), and test (15%, n=30) sets while preserving the benign-to-malignant ratio across all subsets (validation and test sets each contained 16 benign and 14 malignant cases), with the splitting performed at the patient level rather than per-slice or per-image level, ensuring that all training, validation, and test sets were mutually exclusive with no patient overlap between them to prevent data leakage and enable unbiased evaluation of model generalizability.

Data preprocessing

The study implemented standardized preprocessing on all MRI images through four main steps: image registration and resampling, denoising, data augmentation, and normalization. To address variations in MRI protocols and patient positioning, all T1WI, T2WI, and DWI volumes were first rigidly registered to the T1WI sequence as the reference space and resampled to isotropic 1 mm³ resolution using trilinear interpolation. Denoising was then performed on all registered volumes using the Non-Local Means (NLM) algorithm²², with the smoothing parameter automatically set to 0.8 times the estimated noise standard deviation, a search window of 21×21×21 voxels, and a similarity window of 7×7×7 voxels. This approach effectively reduces random noise while preserving important structural details. Subsequently, to expand the effective training sample size, improve model generalization, and mitigate overfitting on the limited dataset, multiple data augmentation strategies, including random rotation (±15°) and random scaling (0.9–1.1)^23,24, were applied on-the-fly exclusively to the training set images during each training epoch, with all transformations generated using a fixed random seed of 42 for full reproducibility; the validation and test sets remained unaugmented to ensure objective model evaluation. Finally, to eliminate systematic variations in signal intensity across different scanning devices and sessions, Z-score normalization was applied to each volume by subtracting the mean and dividing by the standard deviation, after which the normalized pixel values were linearly scaled to the range [0, 1] to meet the input requirements of the PyTorch deep learning framework²⁵.

Model architecture

The present study proposes a multimodal deep learning framework based on few-shot learning that integrates complementary information from multiple MRI sequences (T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging) and incorporates a multi-scale spatial attention mechanism to enhance feature representation and classification performance under limited-sample conditions. All MRI volumes were acquired in the axial plane on the same scanner; prior to input, they underwent rigid registration to ensure spatial alignment across sequences, with alignment quality manually verified by a senior radiologist, and the complete 3D volumes centered on the parotid lesion were cropped to a standardized region of interest, resampled to isotropic resolution, and resized to consistent dimensions. By combining the few-shot learning paradigm, the multi-scale spatial attention module, and cross-modal feature fusion through channel-wise concatenation, the framework addresses challenges associated with model generalization in data-scarce settings and enables robust learning of discriminative features from a small number of annotated samples, with the complete architecture illustrated in Figure 1. Although the dataset includes 198 patients, the few-shot approach is adopted because parotid tumors exhibit high imaging heterogeneity and the annotated volumetric multimodal data remain relatively limited relative to task complexity, thereby better simulating real-world clinical scenarios with scarce labeled examples; a conventional supervised training approach was also evaluated for comparison, and the advantages of the few-shot configuration are demonstrated in the ablation studies in the Results section. The backbone network employs R3D-18²⁶ (a 3D ResNet-18 variant with full spatiotemporal convolutions), pretrained on Kinetics-400, as the shared-weight feature extractor for each MRI sequence; during adaptation, a prototypical network head is utilized for few-shot classification, with the multi-scale spatial attention module inserted following the convolutional stages of each modality-specific branch, followed by cross-modal fusion and a shared classifier. The novelty of the proposed multi-scale spatial attention mechanism lies in its use of parallel convolutional paths with kernel sizes of 1×1, 3×3, and 5×5, where each path independently generates a spatial attention map at a different receptive field that is subsequently normalized using softmax and adaptively fused with the input features through weighted summation; this design explicitly captures lesion characteristics across multiple spatial scales—from fine local details such as internal signal heterogeneity to broader contextual structures such as tumor margins and interfaces with surrounding tissues—and provides superior performance compared with conventional attention modules. The fine-tuning strategy proceeds in two phases: initially, the pretrained R3D-18 backbone layers are frozen to retain general spatiotemporal representations while only the attention modules, fusion layers, and prototypical head are trained; subsequently, the final two residual blocks (layer3 and layer4) of each backbone are unfrozen to enable end-to-end adaptation to MRI-specific volumetric features. Training was conducted using the Adam optimizer with an initial learning rate of 0.0001, reduced by a factor of 0.1 every 20 epochs via step decay scheduling; the model underwent 100 epochs of episodic training with a batch size of 8 episodes (each comprising support and query sets), following the standard few-shot protocol in a 2-way K-shot configuration (with K=5 during primary meta-training and evaluation); all experiments were performed on a high-performance GPU with 24 GB memory, and a fixed random seed of 42 was applied throughout data partitioning, weight initialization, and augmentation to ensure complete reproducibility.

Few-shot Learning Task Paradigm for Parotid Gland Tumor Classification

In view of the limited availability of high-quality annotated data for parotid gland tumors and the high imaging heterogeneity of these lesions, this study adopts a metric-based few-shot learning paradigm even though the overall dataset comprises 198 patients. The support set–query set task structure is employed to simulate the diagnostic reasoning process in clinical practice, where only a small number of representative examples are typically available for rare or heterogeneous tumor subtypes. Each training task is defined in an N-way K-shot format (N=2, corresponding to benign/malignant classification), enabling the model to learn discriminative feature representations from a small number of samples. Specifically, for each task, the support set includes K multimodal samples (T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging) per class to construct class prototypes, while the query set contains samples to be classified. The model performs classification decisions based on distance metrics by calculating the similarity between query sample features and class prototypes in the embedding space. This learning mechanism is notable for its ability to liberate the model from reliance on large-scale annotated data, thereby rendering it more suitable for practical applications in data-scarce clinical settings. For direct comparison, a conventional supervised training approach using the full training set without episodic few-shot sampling was also evaluated, and the advantages of the few-shot configuration in terms of generalization and stability under limited-sample conditions are quantitatively demonstrated in the ablation studies presented in the Results section.

In the classification phase, this study employs a prototypical network as the classifier. For each category c in the support set, the prototype vector p_c is computed as the mean of the embedded support samples:

     (1)

where f(·) denotes the embedding function (feature extractor), is the i-th support sample of category c, and K is the number of shots per class.

For a given query sample x, the probability of belonging to category c is calculated using the softmax function over negative Euclidean distances:

     (2)

where d(·, ·) represents the Euclidean distance function.

Architecture Based on Multi-modal Feature Fusion

To fully leverage the complementary value of multi-sequence MRI data, this study designed a three-branch feature fusion network that processes T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and diffusion-weighted imaging (DWI) sequences separately. Each sequence first undergoes feature extraction using a shared-weight R3D-18 model pretrained on the Kinetics-400 dataset with weights obtained from the official PyTorch TorchVision implementation; during adaptation, a two-stage fine-tuning strategy was applied in which all layers of the pretrained backbone were initially kept fixed to preserve general spatiotemporal representations while only the multi-scale spatial attention modules, cross-modal fusion layers, and prototypical network head were trained, after which the final two residual blocks (layer3 and layer4) of each backbone were unfrozen for end-to-end adaptation to MRI-specific volumetric features, with the earlier layers (layer1 and layer2) remaining frozen throughout training. This network utilizes 3D convolutional kernels sliding across spatial and temporal dimensions to effectively capture tumor volumetric morphology and spatial contextual information. During feature fusion, a concatenation-based cross-modal strategy is employed in which the attention-enhanced feature representations from the three modalities are concatenated along the channel dimension to form a comprehensive multimodal feature vector. This fusion approach preserves the unique information of each modality—T1WI’s anatomical clarity, T2WI’s tissue composition sensitivity, and DWI’s cellular density functionality—while achieving effective complementarity of diagnostic information, thereby significantly enhancing feature expression richness and discriminative power.

Multi-scale Spatial Attention Mechanism

This study incorporates a multi-scale spatial attention module to capture lesion characteristics at different spatial scales. As shown in Figure 2, the module employs parallel convolutional paths with kernel sizes of 1×1, 3×3, and 5×5. Each path independently generates a spatial attention map, which is normalized using softmax and then adaptively fused with the input features through weighted summation. This design enables the model to focus on regions more relevant to distinguishing benign from malignant lesions while reducing the influence of irrelevant background areas. The resulting attention maps may help highlight areas of interest in the images.

DATA AVAILABILITY:

A representative subset of fully anonymized multimodal MRI sequences (T1WI, T2WI, DWI) from the study cohort is publicly available in the Zenodo repository at: https://doi.org/10.5281/zenodo.17744149 (Version v1). These data are shared under a Creative Commons Attribution 4.0 International (CC-BY 4.0) license with no access restrictions. The complete anonymized dataset is not yet publicly available and will be released in a subsequent version of the repository following formal publication of this manuscript, subject to institutional and ethical approval.

Results

Model Performance Overview

In order to evaluate the diagnostic performance of the model, a set of standard metrics was employed, including accuracy, precision, recall, and F1-score. Under the 2-way 5-shot configuration, the multimodal few-shot learning model based on R3D-18 achieved the following results on the test set for the classification of parotid gland tumors: accuracy of 96.88%, precision of 97.50%, recall of 96.88%, and F1-score of 96.83%. The confusion matrix shown i...

Discussion

In this study, we hypothesized that a multimodal few-shot deep learning framework, which integrates multi-sequence MRI (T1-weighted, T2-weighted, and diffusion-weighted imaging) with a multi-scale spatial attention mechanism, can accurately differentiate benign and malignant parotid gland tumors under limited annotated data conditions and outperform both conventional deep learning approaches and experienced radiologists.

The experimental results strongly support this hypothesis. On the indepen...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Grad-CAM			https://github.com/jacobgil/pytorch-grad-cam
NumPy 1.25	NumPy Developers		https://numpy.org/
pandas 2.1	Pandas Developers		https://pandas.pydata.org/
Pycharm	JetBrains		https://www.jetbrains.com/zh-cn/pycharm/
Python 3.10	Python Software Foundation		https://www.python.org/
PyTorch 2.1	Meta Platforms, Inc.		https://pytorch.org/
scikit-learn 1.3	Scikit-learn Developers		https://scikit-learn.org/
SHAP			https://github.com/slundberg/shap
Siemens MAGNETOM Prisma 3.0T Magnetic Resonance Imaging System	Siemens Healthineers		https://www.siemens-healthineers.cn/magnetic-resonance-imaging/3t-mri-scanner/magnetom-prisma