Enhancing Fairness in Large Language Models for Clinical Artificial Intelligence Applications Through Fine-Tuning and Prompting

Summary

Healthcare-specific large language models face ethical and technical challenges, as they, like other large language models, reflect the biases inherent in their training data. The protocol presented here verifies the suppression of four types of bias (gender, race, occupation, religion) using prompt variants across three open-source models.

Abstract

Large Language Models (LLMs) are increasingly being applied to sensitive domains such as medical care, which pose multiple technical and ethical challenges. The most immediate issues include biases built into these models, which are trained on large datasets that may reflect societal prejudices. Such biases can yield outputs that are unfair, ungeneralizable, or even harmful, rendering them clinically inapplicable in the real world and noncompliant with ethical and regulatory constraints. We examine how well bias suppression works when fine-tuned models are prompted across four critical categories (gender, race, profession, and religion). We manually curate a diverse inference dataset and create twelve prompt variants -- six of which are debiased -- to test the outputs of three open-source LLMs: Llama2-7B, Mistral-7B, and Dolly-7B. The fairness and interpretability of the outputs are evaluated using a bias-scoring metric, where lower scores indicate better fairness and interpretability. We also note that debiased prompts reduce bias, and fine-tuning the model performs even better. The results emphasize the critical importance of timely action, model robustness, and ongoing ethical scrutiny for trustworthy and fair LLM deployment in real-world settings, such as medical imaging, and the maintenance of Electronic Health Records (EHR).

Introduction

Large Language Models have prodigious reverberation as tools to support diverse tasks, including decision-making^1,2, text generation³, text translation⁴, customer sentiment analysis⁵, question answering⁶, and generating clinical reports⁷. Several instances have occurred in recent times in which applications have used LLMs to generate content that harms socially disadvantaged individuals or groups^8,9,10. The main reason behind such platitudinous responses is that these models are trained on datasets that inherently contain societal biases associated with race, gender, socioeconomic class, and similar factors. But what you mean by "bias" and "fairness" in a healthcare AI system can be subjective and context-dependent. Researchers have exerted considerable effort to mitigate social biases in LLMs^11,12; however, further improvement remains necessary. The researchers have proposed multiple bias mitigation strategies, which can be classified as pre-processing, in-processing, post-processing, or combinations thereof, across a heterogeneous range of applications. This categorization is based on the stage at which the mitigation measure is taken. This protocol combines all three strategies to tackle and alleviate social bias in six variants of LLMs and investigates bias reduction across four social bias dimensions: Gender, Profession, Race, and Religion. First, an inference dataset is developed using a data augmentation technique to enable LLMs to generate inferences. Second, twelve types of prompts are designed to elicit stereotyped responses from LLMs. Third, Llama-2-7B¹³, Mistral-7B¹⁴, and Dolly-7B¹⁵ are fine-tuned on a dataset containing unbiased sentences across the four social categories: gender, race, profession, and religion, to get Llama-2-7B_Finetuned, Mistral-7B_Finetuned, and Dolly-7B_Finetuned, respectively. Lastly, inferences are drawn by prompting the fine-tuned LLMs to investigate their effectiveness in giving fair responses.

We formulated the following research questions and addressed them in this paper. For each formulated research question (RQ), a null hypothesis (H0) and an alternative hypothesis (H1) were framed.

RQ1: Are the inference dataset and basic prompting techniques effective in evaluating the societal biases in LLMs? Null hypothesis (H0₁): The bias scores derived from an inference dataset when combined with basic prompts are statistically indistinguishable from the baseline bias scores of the LLMs. Alternative hypothesis (H1₁): Bias scores from the inference dataset with basic prompts are statistically significantly different from baseline bias scores of the LLMs. To test the hypothesis, a dataset, NeutralSet, was curated by modifying StereoSet¹⁶ and evaluating each LLM using basic prompting techniques to assess its ability to produce non-stereotypical responses. In our settings, we include six basic prompts --  Standard (A zero-shot prompt to instruct LLM to make inferences), Chain-of-Thoughts (CoT is designed to ask the LLM to think step-by-step to make inferences), System1 (A prompt to let the LLM think quickly while answering), System2 (A prompt to make the LLM think slowly and thoughtfully), Human Persona 1 (HP1 is designed to make the LLM adopt identity of a human who think quickly while taking decisions), and Human Persona 2 (HP2 is designed to make the LLM to adopt the human identity who think slowly and thoughtfully while answering).

RQ2: Are the debiasing prompting techniques effective in revealing and mitigating societal biases in LLMs? Null hypothesis (H0₂): Debiasing prompting techniques are not effective in revealing and mitigating societal biases in LLMs. Alternative hypothesis (H1₂): Measured bias scores decrease significantly when debiasing prompting techniques are used in LLMs. We investigate the impact of debiased variants of the basic prompts on three open-source LLMs to achieve fair text generation free of any social discrimination.

RQ3: Can societal biases be further reduced by fine-tuning the LLMs? Null hypothesis (H0₃): Fine-tuning the LLMs does not further reduce societal biases beyond the reductions achieved through prompting techniques alone. Alternative hypothesis (H1₃): Fine-tuning LLMs further reduces societal biases beyond the reductions achieved through prompting techniques alone. To answer this question, we modify the dataset¹⁷ and fine-tune the LLMs on it to further assess the impact of fine-tuning in reducing stereotype responses.

Figure 1 illustrates the effect of prompt variation and LLM fine-tuning on the gender-neutral predicted outcome. The novelty of our work stems from integrating manual dataset curation, multiple debiasing prompt strategies, and fine-tuning under a single protocol to analyze an LLM for societal bias identification and alleviation, relevant to sensitive real-world applications such as medical imaging and EHR maintenance. We grouped the literature on bias in LLMs into four perspectives: datasets for bias and cognitive associations; bias detection and evaluation frameworks; bias mitigation approaches; and bias in specialized domains.

Researchers are continually generating datasets to enable bias evaluation and semantic association analysis in LLMs. For instance, in cognitive psychology and linguistics, free associations are always pivotal in organizing the conceptual knowledge. Simulating the same association in the latent distribution of LLMs, Abramski et al.¹⁸ have constructed a dataset, LLM World of Words (LWOW). The LWOW English free association norms dataset, comprising millions of responses from three LLMs: Mistral-7B¹⁴, Llama-3.1-8B¹⁹, and Claude-3-5-haiku-latest²⁰. It is inspired by Small World of Words (SWOW)²¹, the largest dataset of human English free association norms, which has been widely employed in a variety of psychological and linguistic investigations. Further, LWOW helps build a cognitive network comprising nodes, each representing a word, with nodes corresponding to semantically similar words that are closer together in the network. This helps evaluate bias in LLMs' output by estimating the distance between words in the artificial cognitive network as a key metric. One major stumbling block in using proprietary LLMs is that their internals are inaccessible. Even though significant efforts have been made in these models to address bias, alignment datasets are still scarce. To address this concern, a dataset named GenderAlign is proposed in Zhang et al.²² to mitigate gender bias in the LLMs. UnStereoEval²³, a benchmark dataset, is proposed to investigate stereotypical gender bias in occupations and emotions. Their findings reveal a low level of fairness across 28 different LLMs. Bartl and Leavy²⁴ developed a dataset, Tiny Heap, in which sentences featuring stereotypical mentions are replaced with their neutral equivalents and fine-tuned three language models (GPT-2²⁵, PHI-1.5²⁶, and RoBERTa²⁷). In their findings, they observe a significant reduction in the models' gender-stereotypical tendencies.

Several multilayer frameworks have been proposed to identify and alleviate biases in LLMs. In Raza et al.²⁸, a framework, NBIAS, is proposed that comprises four layers: dataset creation, model development, bias mitigation, and evaluation. The dataset within the framework is built from diverse domains, including healthcare, social media, and employment portals. They showcase the effectiveness of their model by outperforming the baseline (BERT²⁹) by 1% to 8% in fairness. In another such framework, GenderCARE³⁰, a strategy to mitigate gender bias in LLMS is proposed. In their extensive experimental setup, they effectively reduced gender bias by an average of 35% across twelve different LLMs. A framework, BiasAlet³¹, automatically detects social bias in the open text generated by the LLMs. It comes with some limitations: firstly, the study is conducted on a synthesized dataset due to the unavailability of a benchmark to evaluate bias in the LLM's open-text generation, and secondly, it is constructed on the antiquated dataset SBIC³², which makes it hard to distinguish between the relevance of implicit bias and the bias in the dataset itself. Bias in human-generated data may be introduced into models trained on it. The use of such models is controversial in high-stakes jobs, where their output may adversely impact disadvantaged groups. To tackle this, a framework, BiasBuster³³, is designed to detect, evaluate, and mitigate cognitive bias in the LLMs. Aligned with this, in another work³⁴, researchers propose an interactive framework to generate fair, logical, and critical text through System 2 prompts (designed to let the LLM think thoughtfully and slowly) that complement self-refined and implicative prompts. However, the framework is limited to the tasks within the LLMs' latent space. Dong et al.³⁵ develop a framework that utilizes indirect probing to mitigate and evaluate gender bias in ten open-source LLMs. In their probing method, without exploiting direct stereotypical mentions, they disclose indirect gender bias.

Lin et al.³⁶ investigate the political bias in text generation by the LLMs for both closed-ended (GPT-3.5-turbo, and GPT-4³⁷) and open-ended models (Llama-2-7B¹³, Mistral-7B¹⁴, Vicuna²⁹). They determined whether the model-generated text induced left- or right-leaning media bias. Furthermore, they devised a mitigation strategy by fine-tuning the model and by providing additional debiased prompts during text generation. After applying the bias mitigation technique, the model tends to generate neutral text; left- or right-leaning media do not influence that. In line with this, Raj et al.¹⁷ have proposed a debiasing solution, Social Contact Debiasing (SCD), based on the contact hypothesis in psychology, which includes prompt generation that comprehends the ideologies of different social groups to reduce prejudice in the text generation of three open-sourced LLMs. In parallel with this, Kamruzzaman and Kim³⁸ have proposed a social debiasing technique that exploits System 1 and System 2 chain-of-thought prompting to mitigate across twelve bias dimensions in five LLMs (GPT-3.5, GPT-4³⁷, Llama-2-7B¹³, Mistral-7B¹⁴, and Gemini-1.0³⁹). Here, the System 1 prompt is intended to make the LLM respond quickly, while the System 2 prompt is meant to guide it toward more thoughtful and deliberate answers. Although various studies were conducted that make use of prompt engineering to mitigate the bias in LLMs, hardly anyone has probed the impact of prompt variation on LLMs' output. To address this issue, Hida et al.⁴⁰ have investigated the sensitivity of the outputs of the twelve open-sourced LLMs to prompt variation and analyzed their impact on task performance and bias trade-offs. Their findings reveal that debiasing results are sensitive to the prompt; less bias in the models' output leads to lower task performance. Kamboj et al.⁴¹ have proposed a post-processing approach to mitigate gender bias in contextualized embeddings of a T5 model (Text-To-Text-Transfer-Transformer model⁴²). Oba et al.⁴³ provide manually created textual preambles as prompts to LLMs to suppress their biased generation.

Cognitive biases, which have been a source of diagnostic error in healthcare for decades, are at risk of being imprinted into and amplified by large language models (LLMs) in clinical decision-making. To combat this risk, Mahajan et al.⁴⁴ propose that mitigation strategies for implementing these technologies should centre on three themes: first, self-reflection (iterative re-evaluation of outputs), second, contextual reasoning (incorporating full patient history and evidence-based guidelines), and finally, transparent reasoning traces (explanations of reasoning processes made available for audit). LLMs, with their ubiquitous ability, are now also widely used to generate programming code. Hence, this is a prominent concern for researchers to investigate the adverse effects of any social bias in generated code. If such bias is detected, the corresponding mitigation techniques need to be devised. In the same direction, Huang et al.⁴⁵ proposed a social bias evaluation and mitigation technique and tested it on five widely used LLMs. In recent times, we have seen tremendous advancements in LLM architectures and their ability to generate responses that sound human-like. Whether LLMs can serve as proxies for humans in decision-making remains underexplored and warrants further research. In this direction, a framework and a dataset are curated by Tjuatja et al.⁴⁶ to investigate the ability of LLMs to give human-like responses in a survey questionnaire.

Despite these advances, three research gaps persist. First, previous research tends to focus on narrow types of bias (e.g., gender or political) or specific domains (e.g., a particular topic). Second, although prompt engineering is pervasive, few studies examine the effects of systematic prompt variation on LLM output. Third, limited research addresses debiasing in the challenging resource-restricted fine-tuning setting of open-source LLMs relevant to critical domains such as healthcare. Addressing these gaps, a single bias mitigation and evaluation protocol is presented that can help measure stereotypical bias across various model architectures, incorporating fine-tuning under computational constraints and assessing the robustness of model decisions against bias metrics.

Protocol

All the experiments are conducted on the Google Colab platform (A100 with 40 GB RAM). For performing experiments, API keys (Model cards) of Llama2-7B, Mistral-7B, and Dolly-7B were collected from Hugging Face.

The protocol is divided into three parts. First, build a dataset that serves as a basis for evaluating social bias in LLMs. Then design twelve distinct prompt variants, aligned with the work done by Kamruzzaman and Kim³⁸ to investigate the presence of social bias in the inferences generated by LLMs. Then, fine-tune the LLMs on a dataset of unbiased sentences related to race, religion, gender, and profession, and probe them again with the same prompts to investigate the effect of fine-tuning on the generated inferences. The proposed framework is shown in Figure 2.

Curating a dataset
The framework uses two datasets. One is used to derive inferences, and the other is applied to fine-tune LLMs. This study modifies StereoSet¹⁶ to build a new dataset, NeutralSet, which serves as the basis for inference generation. The LLMs used the StereoSet to make predictions across four bias types: race, religion, gender, and profession. StereoSet comprises two sub-datasets: intra-sentence and inter-sentence. An instance of NeutralSet is shown in Table 1. Give a sentence from the column context as a query to the LLM and ask to fill in the BLANK with any of the words from the columns anti-stereotype, stereotype, or neutral. The degree of bias in the LLM can be judged from the option it picks. The inclusion of a neutral column in NeutralSet distinguishes it from StereoSet. The purpose of adding it to the new dataset is to reduce the likelihood of selecting the stereotype option while making LLM inferences. Words are added in the neutral column according to the steps mentioned in Algorithm 1 (Supplementary File 1). Where V_s and V_as are the vectors of stereotype and anti-stereotype words, respectively. A word from the dictionary is then selected whose vector is closest to V_n, which is contextually positioned between the stereotype and anti-stereotype word vectors, and this word is added to the neutral column in the selected row. Table 2 summarizes the process of creating NeutralSet. Then, modify the dataset¹⁷to fine-tune the LLMs. The revised dataset includes 25,020 instances, each consisting of a query paired with a socially unbiased answer, used for training the models. Queries are designed to address all four types of social biases.

Prompting the LLMs
Examine the social biases in LLMs by providing a set of prompts as input instructions. First, create twelve types of prompts: six basic ones and six corresponding debiased variants. The contents of all prompting techniques are listed in Table 3. The prompts in this study underwent a cyclical, formalized process of crafting, implementation, reflection, and revision, resulting in a series of prompts that were honed over time through significant trial and error and refinement. This enabled coverage of a diverse set of prompt types, including baseline, CoT, and human-persona prompts. Prompt designs and scoring were not validated by external domain experts (e.g., clinicians, social or fairness specialists). Table 4 summarizes different prompt types, highlighting their reasoning styles, bias mitigation capabilities, and suitable use cases to guide appropriate selection. The LLMs' evaluation in this paper does not retain past information across runs or sessions. One thing to clarify in the final model is that each prompt is to be executed in isolation, meaning no conversation history is carried over from one prompt to another. This layout ensured past responses did not affect subsequent outputs. During the fine-tuning phase, modify only the model parameters based on the training dataset, as outlined in both Parameter-Efficient Transfer Learning (PEFT⁴⁷) and Quantized Low-Rank Adaptation (QLoRA⁴⁸). Construct the prompt as mentioned below:

Standard prompt: This is a zero-shot prompting technique in which the LLM is instructed with minimal information to make the inference. The prompt is" Insert the most appropriate word into the BLANK from the given choices.". This is used as a baseline to quantify and evaluate social bias in all six LLMs.

CoT prompt: The prompt includes the phrase "Think step by step and fill in the BLANK. Choose the best fit for the BLANK from the options provided…". In this prompting technique, the LLM is encouraged to engage its cognitive process and think step-by-step before providing an answer.

System1 and System2 prompts: By including the phrase "You respond to questions promptly…", the System1 prompts guide the LLM to provide quick, intuitive responses, similar to how human cognition operates in System1. On the other hand, System2 prompts are designed to elicit more thoughtful, detailed, analytical, and deliberate responses from the LLM by adding the phrase "You respond to questions thoroughly and deliberately. Your answers are thoughtful and dependable "in the instruction.

HP1 and HP2 prompts: The prompts begin with: "Adopt the identity of a person who answers the questions quickly and fill in the BLANK while staying in strict accordance with the nature of this identity… ". In the HP1 prompting technique, the LLM was instructed to adopt the identity of a human who responds quickly. The purpose of this setting is to make the LLM imitate the human in the System1 cognition process. In contrast, in the HP2 prompt, the LLM is driven to adopt a human-like identity that thinks thoughtfully and deliberately before answering questions, thereby achieving more accurate results. HP2 prompt starts with the phrase: "Adopt the identity of a person who answers the questions thoughtfully and deliberately and fill in the BLANK while staying in strict accordance with the nature of this identity…". The key idea behind including these prompts is to assess an LLM's ability to fully emulate human cognition.

Debias variants: A debias variant is constructed by adding a statement that instructs the LLM to make the decision neutrally, without any stereotypical prejudice.

Evaluating the LLMs
Evaluate six LLMs: 1) Llama-2-7B¹³, using the meta-llama/Llama-2-7b-chat-hf checkpoint on Huggingface; 2) Mistral-7B¹⁴, using the mistralai/Mistral-7B-Instruct-v0.3 checkpoint on Huggingface; 3) Dolly-7B¹⁵, using the databricks/dolly-v2-7b checkpoint on Huggingface; 4) Llama2-7B_finetuned, a fine-tuned variant of Llama-2-7B; 5) Mistral-7B_finetuned, a fine-tuned variant of Mistral-7B; 6) Dolly-7B_finetuned, a fine-tuned variant of Dolly-7B.

Perform all experiments on a high-speed GPU, such as A100 with 40 GB of memory or higher. For fine-tuning the LLMs, divide the dataset into training, validation, and test sets, using the standard 70%:10%:20% split. To avoid complete retraining of the model, this study uses PEFT⁴⁷ configuration for fine-tuning, which freezes a substantial amount of the model's parameters and adds only a few task-specific parameters. Use 4-bit precision in QLoRA⁴⁸ to load the LLM if computational resources are limited. This will enable faster fine-tuning without slackening the model's performance.

To evaluate stereotypical bias in LLMs, use Bias_score as a metric. As shown in Equation 1, Bias_score is computed as the ratio of stereotype responses to the total number of valid responses from the LLM for a specific prompt.

     (1)

Where N_s, N_as, and N_n represent the number of stereotypical, anti-stereotypical, and neutral responses, respectively. A lower bias score indicates the model's output is less stereotypical.

Results

Using the bias score defined in Equation 1, we systematically answer our research questions and evaluate societal biases in LLMs across four social dimensions. The statistically significant reductions of bias scores observed provide empirical validation of the proposed protocol for bias reduction in LLMs for high-stakes tasks. To assess the effectiveness of NeutralSet, i.e., the curated inference dataset, we query all three vanilla LLMs on both StereoSet and NeutralSet through the baseline standard prompt and calculate t...

Discussion

In this work, we present a scalable protocol for reducing social biases in LLMs that leverages Llama2-7B¹³, Mistral-7B¹⁴, and Dolly-7B¹⁵. This approach combines HP2 prompt engineering, debiasing prompt modifications, and targeted fine-tuning to develop a protocol for continued reduction of bias in LLM-generated outputs across the four major societal dimensions of gender, race, profession, and religion. While evaluating vanilla models with basic and d...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Google Colab	Google	https://colab.research.google.com/	used for running the experiments
Mendeley Desktop	Mendeley	https://www.mendeley.com/	used for managing citations and references.