Few-Shot and Zero-Shot Biomedical Named Entity Recognition: A Procedure for Enhancing BioBERT with Prompt-Based Learning and Large Language Models

Summary

This article presents a protocol for biomedical named entity recognition using BioBERT, prompt-based learning, and large language models for low-resource scenarios.

Abstract

Biomedical Named Entity Recognition (NER) plays a crucial role in extracting valuable information from clinical texts and biomedical literature. Traditional deep learning models, including BioBERT, ClinicalBERT, and RoBERTa, have demonstrated substantial improvements in NER tasks; however, they still struggle with low-resource biomedical terms, domain adaptation challenges, and unseen entity generalization. To address these limitations, this study introduces a hybrid framework combining BioBERT, prompt-based learning, and Large Language Models (LLMs) to enhance few-shot and zero-shot learning capabilities in biomedical NER. The proposed model effectively leverages contextual knowledge from pre-trained transformers, reducing dependency on extensive labelled datasets while maintaining high accuracy. After extensive experimental evaluation across multiple biomedical benchmark datasets, the proposed approach demonstrates consistently strong performance. Specifically, it achieves 89.8% accuracy, 88.7% precision, 90.5% recall, and an F1-score of 0.895, outperforming baseline methods and demonstrating its effectiveness for biomedical text mining tasks. Compared with other methods, prompt engineering helps the model adapt much better to the unique language of biomedical texts. Plus, boosting with a large language model (LLM) gives NER performance a serious lift by picking up on tricky semantic and grammatical details. These findings demonstrate the efficacy of few-shot and zero-shot learning in biomedical text mining, paving the way for better automated clinical data analysis and smarter decision-support systems.

Introduction

The explosion of biomedical literature, electronic health records (EHRs), and clinical trial data really needs smart, robust Biomedical Named Entity Recognition (BioNER) systems. BioNER is a specialized task in Natural Language Processing (NLP) that aims to identify entities such as diseases, genes, proteins, drugs, and chemicals in unstructured biomedical text¹. Being able to pick these entities out accurately is super important for a bunch of other uses, like mining biomedical knowledge, finding new drugs, supporting doctors with clinical decisions, or even automatically sorting through piles of research papers².

Although significant advances have been achieved in deep learning–based models—such as BioBERT, ClinicalBERT, and BlueBERT- most existing supervised biomedical named entity recognition (BioNER) approaches continue to rely heavily on large-scale annotated datasets. This strong dependence on extensive labeled data limits their generalization capability when applied to novel or previously unseen biomedical texts³. The classic BioNER models need tons of annotated data just to fine-tune them, which becomes a real headache in areas where there just isn't enough data, think rare diseases, pharmacogenomics, or biomedical research published in languages other than English⁴. On top of that, manually tagging all this biomedical text is slow, expensive, and usually needs real experts in the field⁵. That's why developing few-shot and zero-shot BioNER models is so important: it could significantly reduce the need for labeled data while ensuring models work well across a wide range of biomedical content⁶.

Few-shot learning (FSL) enables models to achieve competitive entity recognition performance using only a small fraction of labeled data, typically between 1% and 10% of the full training dataset, making it particularly valuable for low-resource biomedical domains⁷. Zero-shot learning (ZSL) goes even further, letting a model recognize entities it's never even seen before, just by tapping the power of big, pre-trained language models (LLMs), without needing any extra training data⁸. Lately, advances in LLMs like GPT-4, LLaMA, and Falcon have demonstrated strong zero-shot performance across general NER tasks, but their use in actual biomedical texts remains largely uncharted territory. That's because the language in biomedical stuff is just so specialized and complicated⁹.

This research aims to close that gap by bringing together BioBERT fine-tuning, few-shot learning, zero-shot LLM inference, and prompt-based learning, all with the aim of raising the bar for BioNER in low-resource biomedical settings.

To give an idea of what's already out there: (1) BioBERT is a version of BERT that's been retrained on PubMed abstracts and full-text articles, so it’s better suited for biomedical content¹⁰. (2) ClinicalBERT is a BERT variant trained on EHRs, so it's specially tuned for making sense of clinical notes and doctors' reports¹¹. (3) BlueBERT is another biomedical NLP model, trained on both PubMed and MIMIC-III datasets, and it's optimized for recognizing medical stuff in text¹². (4) PubMedBERT is a transformer model trained only on PubMed data, which helps it really pick up on those unique patterns and terminology found in biomedical writing¹³.

While these models achieve state-of-the-art performance on BioNER benchmarks, their reliance on large, labeled datasets and supervised fine-tuning limits their adaptability to new biomedical domains¹⁴. Moreover, domain shifts between scientific biomedical literature (e.g., PubMed abstracts) and clinical notes (e.g., i2b2 2010 dataset) further degrade the performance of supervised BioNER systems¹⁵.

The primary limitations in current BioNER research include: Dependence on Annotated Data: Existing transformer-based models require extensive labeled datasets, which are scarce in specialized biomedical domains¹⁶. Lack of Generalization to Unseen Entities: Current models struggle to recognize rare or novel biomedical terms not present in the training data¹⁷. Limited Exploration of Few-Shot and Zero-Shot Learning: Most research focuses on fully supervised learning, with minimal investigation into low-data paradigms for BioNER¹⁸. Inefficient Adaptation of LLMs: Although LLMs like GPT-4 and LLaMA show strong zero-shot performance in general NLP, their application to biomedical NER remains underexplored due to terminological complexity and lack of domain-specific prompts¹⁹.

To address these gaps, this study explores the following key research questions: How can few-shot learning improve BioBERT's performance in biomedical NER with minimal labelled data? Can zero-shot inference using LLMs accurately recognize biomedical entities without domain-specific fine-tuning? How can prompt-based learning enhance zero-shot performance in biomedical NER tasks? What is the optimal hybrid approach combining BioBERT and LLM-generated pseudo-labels for improving BioNER in low-resource settings? How does adversarial training impact the robustness of few-shot and zero-shot BioNER models?

The purpose of this research is to develop a novel, adaptive biomedical NER framework that leverages few-shot and zero-shot learning to address the challenges posed by the scarcity of labeled data. The study aims to enhance the generalization capabilities of BioBERT and LLMs by exploring prompt engineering techniques, contrastive learning, and adversarial training.

The research is structured around the following objectives: (1) Developing a few-shot learning pipeline for biomedical NER by fine-tuning BioBERT on limited annotated data. (2) Evaluating zero-shot LLMs (e.g., GPT-4, LLaMA, Falcon) for biomedical NER using domain-specific prompt engineering. (3) Creating a hybrid approach that integrates BioBERT fine-tuning with LLM-generated pseudo-labels for improved entity recognition in low-resource biomedical datasets. (4) Conducting a comparative analysis of few-shot vs. zero-shot learning using standard biomedical benchmarks (MedMentions, BC5CDR, i2b2 2010, PharmaCoNER, BioASQ). (5) Implementing adversarial training (FGSM, PGD) to enhance model robustness against noisy biomedical text variations.

This study contributes to the field of biomedical NLP and entity recognition by introducing few-shot and zero-shot BioNER strategies to address label scarcity in biomedical datasets and demonstrating how LLMs can be leveraged for zero-shot biomedical NER via optimized prompt engineering. Proposing a hybrid framework combining BioBERT fine-tuning and LLM-generated synthetic annotations for better generalization. Evaluating the impact of adversarial training on BioNER models to improve robustness against domain shifts. Providing a comparative benchmark of BioNER performance across supervised, few-shot, and zero-shot paradigms.

Literature survey
Sivarajkumar and Wang²⁰ developed HealthPrompt, a Zero-Shot Learning (ZSL) framework for clinical NLP that addresses the lack of annotated clinical datasets. Instead of fine-tuning a pre-trained language model (PLM), they employed prompt-based learning, where task definitions are tuned using structured templates. Their evaluation of six PLMs, including BioBERT and ClinicalBERT, on the MIMIC-III dataset showed that ClinicalBERT achieved the best accuracy (85%) and F1-score (86%) for clinical text classification. This study shows that prompt-based zero-shot learning (ZSL) can keep up with supervised models in clinical natural language processing (NLP), without requiring any training data.

Keming Lu and team²¹ developed BIODLM, a new approach to fine-tune discriminative language models (DLMs) for the biomedical domain via prompt-based continual pretraining. Their main goal was to improve performance on both few-shot and zero-shot tasks in biomedicine by leveraging clever prompt tuning and the Replaced Token Detection (RTD) method. To make it work, they play around with the vocabulary, deliberately mixing up domain-specific terms, and use PubMedBERT as a generator during pretraining. The results are solid: BIODLM actually beat the usual CLS-based fine-tuning, scoring a macro-average accuracy of 50.2% with full supervision and 43.4% in zero-shot mode.

Zonghai Yao and colleagues²² took a different approach, focusing on prompt generation that accounts for context variance. Their goal was to reduce the weird biases that can arise during prompt-based probing and to make the BioLAMA evaluation more transparent and interpretable. In other words, they’re trying to make sure these language models give us answers that make sense and aren’t just picking up on cleverly worded prompts.

They introduced a rank-change-based evaluation metric (UCM: Understand–Confuse–Misunderstand) instead of traditional Top-k accuracy to assess PLMs on biomedical entity recognition. Experiments on 12 PLMs, including BioBERT, ClinicalBERT, and RoBERTa-based models, showed improved BioLAMA performance for large-N-M relations and rare biomedical entities. BioBERT achieved an Accuracy@1 of 40.7% and UCM Understandability of 32.8%, outperforming traditional Top-k metrics.

Yeh et al.²³ investigated prompting for biomedical relation extraction using the ChemProt dataset to enhance domain-specific few-shot and full-data fine-tuning. They designed prompt-based templates that incorporate scoring metrics such as frequency, specificity, and similarity to optimize label word selection. Using BioMed-RoBERTa-base, their method achieved an F1-score of 90.09, outperforming SciFive-Large by 1.14 F1 and surpassing regular fine-tuning by 14.21 F1. This confirms that prompt-based learning improves biomedical NER performance with fewer training examples.

Susanti and Holsmoelle ²⁴ studied NLP-based verification of causal graphs, comparing fine-tuned language models and prompt-based LLMs. They trained BioBERT and GPT models for supervised causal relation classification and evaluated zero-shot and few-shot prompt-based LLMs. Results on biomedical and open-domain datasets revealed that fine-tuned models outperformed LLM prompts, achieving up to 20.5% higher F1 scores. The study highlights the need for advanced prompting and dataset-specific tuning to improve the accuracy of causal relation extraction.

Chen et al.²⁵ proposed a knowledge-guided instance-generation and prompt-contrastive-learning framework for few-shot BioNER. Their approach leverages domain knowledge graphs to generate diverse biomedical entities and employs question-prompt-based contrastive learning to enhance entity recognition by measuring mutual information between query–answer pairs. Evaluated on fbenchmark datasets (NCBI, BC5CDR-Disease, BioNLP11EPI, BioNLP13GE), their model achieved up to a 7.1% F1-score improvement over state-of-the-art models in 20-shot scenarios. Their code is publicly available at: https://github.com/cpmss521/KGPC.

Chen et al.²⁶also proposed a prompt-learning approach for biomedical claim detection by transforming text classification into a masked language modeling (MLM) task. Using BERT, RoBERTa, and T5 with hard and mixed templates, they improved claim prediction accuracy. On the BioClaim dataset, their T5 model with mixed templates achieved a 5.3% F1-score improvement over previous models, addressing the gap between pretraining and fine-tuning in PLMs using masked token prediction.

Ryan Shea Ying Cong Tan et al.²⁷ assessed LLMs for cancer disease response inference from radiology reports using transformer models, Bi-LSTM, CNN, and classical ML. Their method included data augmentation (sentence permutation and consistency loss) and prompt-based fine-tuning. The GatorTron transformer achieved the best accuracy: 0.8916 on the test set and 0.8976 after augmentation. Prompt-based fine-tuning enabled effective performance with as few as 500 labeled reports.

Hu et al.²⁸ evaluated GPT-3.5 and GPT-4 for clinical NER, enhancing performance via a four-part prompting framework: baseline prompts, guideline-based prompts, error-based instructions, and few-shot samples. On the MTSamples and VAERS datasets, GPT-3.5 and GPT-4 improved their relaxed F1 scores from 0.634/0.804 and 0.301/0.593 to 0.794/0.861 and 0.676/0.736, respectively. Though still behind BioClinicalBERT (F1: 0.901 on MTSamples, 0.802 on VAERS), these results show the growing effectiveness of LLMs in clinical NER with proper prompting strategies.

Protocol

1. Procedure

1. Data preprocessing and model fine-tuning
   The data was preprocessed from PubMed, BioASQ, MedQA, and MMLU using text normalization, tokenization (WordPiece/BPE), stopword removal, and IOB annotation. Conducted domain-specific fine-tuning of BioBERT using Masked Language Modeling (MLM) and a token classification head. Compared with adapter-based methods like LoRA.
2. Implementing PROMPT-BASED Few-Shot and Zero-Shot learning
   For Zero-Shot Learning: Used LLMs (GPT-4, GPT-3.5, T5) with natural language prompts (e.g., "Extract biomedical entities from the following text: [Input sentence]. The entities to identify are diseases, drugs, genes, and chemicals. List each entity and its type.") and instruction tuning. Used 5–20 labeled examples per entity type with Pattern-Exploiting Training (PET) and Prefix-Tuning.
3. Integrating LLMs and establishing a hybrid framework
   Contextual Embeddings Fusion and Multi-Stage NER Prediction (BioBERT detection + GPT-4 refinement) were performed. This framework used a two-stage, sequential fine-tuning approach (not end-to-end) to manage computational complexity. First, BioBERT was fine-tuned on the available few-shot data. Second, this frozen BioBERT model was integrated with the LLM component via the fusion layer, which was trained on the same data. Thus, it was incorporated using a two-stage fine-tuning strategy:
   Stage 1 – Supervised BioBERT fine-tuning for NER.
   Stage 2 – Prompt-based adaptation via GPT-4 outputs.
   
   where α is a learned coefficient, and Wp is the projection matrix.
   A weighted ensemble vote (W_BioBERT=0.45, W_GPT4=0.35, W_ClinicalBERT=0.20) was applied to merge predictions.
   The weights for each model (W_BioBERT=0.45, W_Hybrid=0.35, W_ClinicalBERT=0.20) were optimized on a held-out validation set using a grid search to maximize the micro F1-score. This weighting scheme reflects the relative contributions of each model: the fine-tuned BioBERT provides the strongest domain-specific signal, the hybrid model adds generalization, and ClinicalBERT offers complementary clinical context.
4. Applying refinement and ontology linking
   GPT-4 was used for ontology-guided refinement, accepting corrections when the cosine similarity with ontology embeddings was ≥ 0.8 (only corrections with > 0.8 are already mentioned). The entities were linked to UMLS, MeSH, and HPO databases for synonym resolution.
5. Deploying the system with optimized inference
   Adaptive confidence thresholding was implemented per entity class: T_C =μ_C + βσ_C (β=0.5).
   A key component of the inference pipeline is the adaptive confidence threshold, which dynamically balances precision and recall across entity types. Rather than using a static threshold, the adaptive mechanism computed a class-specific threshold TC based on the empirical confidence distribution of predictions:
   
   where μ_C and σ_C represent the mean and standard deviation of confidence scores for entity class C, and β is a tunable scaling factor (empirically set to 0.5). Infrequent biomedical entity types (e.g., rare diseases) tend to exhibit high variance in confidence scores. The adaptive threshold ensured that these are retained when σC is large, thereby preventing the over-filtering of rare entities. Conversely, for frequent entities, the threshold is tightened to reduce false positives.
   The system was deployed using a Flask-based interface with Multi-GPU processing (PyTorch DDP) and 8-bit quantization for real-time inference.

2. Environmental setup

1. Datasets
   The framework was tested on the standard biomedical NER datasets to make a fair comparison. The following datasets were employed, with their official splits preserved.
   BC5CDR: For Chemical and Disease entity recognition.
   NCBI Disease: Focused on Disease mentions.
   i2b2 2010: For clinical concept recognition in electronic health records (EHRs).
2. Evaluation
   To ensure statistical robustness, we performed 5 independent runs for each few-shot experiment (K=5,10,15) with different random seeds for data sampling and weight initialization. The accuracy, Precision, Recall, and F1-score were reported across these runs. Zero-shot results, being deterministic, were reported from a single run.
   To evaluate the feasibility of deploying the proposed Few-Shot and Zero-Shot Biomedical NER system in real-time biomedical environments, a detailed computational efficiency and resource profiling analysis was conducted. The profiling was carried out on an NVIDIA A100 GPU (40 GB VRAM) with CUDA 12.1 and PyTorch 2.3.1, using mixed-precision (FP16) inference to balance speed and memory efficiency.
3. Baseline models
   BioBERT (Fine-tuned): The BioBERT-v1.1 base model¹⁰was fine-tuned in a fully supervised fashion on the entire training data of each benchmark.
   ClinicalBERT & RoBERTa-base: Fine-tuned as domain-specific and general-domain baselines.
   GPT-3.5 & GPT-4 (Zero-Shot): Tested as described in step 3.3, without task-specific fine-tuning.
   HealthPrompt²⁰: The current state-of-the-art prompt-based zero-shot clinical NER model.
   For the purpose of complete reproducibility, the core software libraries, model checkpoints, and hardware details used in this study are listed in Table of Materials. This includes specific versions of key components, such as the BioBERT model variant, LLM, deep learning framework, and computational environment.

Results

The evaluation of the Few-Shot and Zero-Shot Biomedical Named Entity Recognition (NER) model was conducted on a diverse biomedical dataset derived from PubMed abstracts, clinical notes, and biomedical question-answering corpora. A comparative performance analysis against established models, including BioBERT, ClinicalBERT, RoBERTa, PhenoBERT²⁹, GPT-3.5, and GPT-4 was performed. The proposed model achieved a superior overall accuracy of 89.8%, with a precision of 88.7%, recall of 90.5%, and an F1-s...

Discussion

Conflicting results, unexpected findings, and discrepancies with other research
Despite the superior performance of the proposed model, certain conflicting results emerged when compared to existing studies. BioBERT-based models have traditionally demonstrated strong performance in biomedical Named Entity Recognition (NER), as reported in previous surveys and comparative studies of biomedical NER systems^4,10. While transformer-based architec...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Core Model	BioBERT-base (v1.1)	Hugging Face: monologg/biobert-v1.1
Deep Learning Framework	PyTorch 2.3.1	https://pytorch.org/
GPU Hardware	NVIDIA A100 (40GB VRAM)	NVIDIA
Large Language Model (LLM)	GPT-4	OpenAI API
Operating System	windows	Microsoft
Parameter-Efficient Tuning	LoRA (via PEFT library 0.6.2)	https://github.com/huggingface/peft
Programming Language	Python 3.9.18	Python Software Foundation
Zero-Shot Baseline LLM	GPT-3.5-Turbo	OpenAI API