Few-Shot and Zero-Shot Biomedical Named Entity Recognition: A Procedure for Enhancing BioBERT with Prompt-Based Learning and Large Language Models

Summary

This article presents a protocol for biomedical named entity recognition using BioBERT, prompt-based learning, and large language models for low-resource scenarios.

Abstract

Biomedical Named Entity Recognition (NER) plays a crucial role in extracting valuable information from clinical texts and biomedical literature. Traditional deep learning models, including BioBERT, ClinicalBERT, and RoBERTa, have demonstrated substantial improvements in NER tasks; however, they still struggle with low-resource biomedical terms, domain adaptation challenges, and unseen entity generalization. To address these limitations, this study introduces a hybrid framework combining BioBERT, prompt-based learning, and Large Language Models (LLMs) to enhance few-shot and zero-shot learning capabilities in biomedical NER. The proposed model effectively leverages contextual knowledge from pre-trained transformers, reducing dependency on extensive labelled datasets while maintaining high accuracy. After extensive experimental evaluation across multiple biomedical benchmark datasets, the proposed approach demonstrates consistently strong performance. Specifically, it achieves 89.8% accuracy, 88.7% precision, 90.5% recall, and an F1-score of 0.895, outperforming baseline methods and demonstrating its effectiveness for biomedical text mining tasks. Compared with other methods, prompt engineering helps the model adapt much better to the unique language of biomedical texts. Plus, boosting with a large language model (LLM) gives NER performance a serious lift by picking up on tricky semantic and grammatical details. These findings demonstrate the efficacy of few-shot and zero-shot learning in biomedical text mining, paving the way for better automated clinical data analysis and smarter decision-support systems.

Introduction

The explosion of biomedical literature, electronic health records (EHRs), and clinical trial data really needs smart, robust Biomedical Named Entity Recognition (BioNER) systems. BioNER is a specialized task in Natural Language Processing (NLP) that aims to identify entities such as diseases, genes, proteins, drugs, and chemicals in unstructured biomedical text¹. Being able to pick these entities out accurately is super important for a bunch of other uses, like mining biomedical knowledge, finding new drugs, supporting doctors with clinical decisions, or even automatically sorting through piles of research papers².

<....

Protocol

1. Procedure

1. Data preprocessing and model fine-tuning
   The data was preprocessed from PubMed, BioASQ, MedQA, and MMLU using text normalization, tokenization (WordPiece/BPE), stopword removal, and IOB annotation. Conducted domain-specific fine-tuning of BioBERT using Masked Language Modeling (MLM) and a token classification head. Compared with adapter-based methods like LoRA.
2. Implementing PROMPT-BASED Few-Shot and Zero-Shot learning
   For Zero-Shot Learning: Used LLMs (GPT-4, GPT-3.5, T5) with natural language prompts (e.g., "Extract biomedical entities from the following text: [Input sentence]. The entities to identify are....

Results

The evaluation of the Few-Shot and Zero-Shot Biomedical Named Entity Recognition (NER) model was conducted on a diverse biomedical dataset derived from PubMed abstracts, clinical notes, and biomedical question-answering corpora. A comparative performance analysis against established models, including BioBERT, ClinicalBERT, RoBERTa, PhenoBERT²⁹, GPT-3.5, and GPT-4 was performed. The proposed model achieved a superior overall accuracy of 89.8%, with a precision of 88.7%, recall of 90.5%, and an F1-s.......

Discussion

Conflicting results, unexpected findings, and discrepancies with other research
Despite the superior performance of the proposed model, certain conflicting results emerged when compared to existing studies. BioBERT-based models have traditionally demonstrated strong performance in biomedical Named Entity Recognition (NER), as reported in previous surveys and comparative studies of biomedical NER systems^4,10. While transformer-based architec.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Core Model	BioBERT-base (v1.1)	Hugging Face: monologg/biobert-v1.1
Deep Learning Framework	PyTorch 2.3.1	https://pytorch.org/
GPU Hardware	NVIDIA A100 (40GB VRAM)	NVIDIA
Large Language Model (LLM)	GPT-4	OpenAI API
Operating System	windows	Microsoft
Parameter-Efficient Tuning	LoRA (via PEFT library 0.6.2)	https://github.com/huggingface/peft
Programming Language	Python 3.9.18	Python Software Foundation
Zero-Shot Baseline LLM	GPT-3.5-Turbo	OpenAI API