Development of a Guideline-Based Retrieval-Augmented Generation Chatbot Referencing Web-Based Cancer Clinical Practice Guidelines

Summary

This protocol describes the development and evaluation of a guideline-based retrieval-augmented generation chatbot that retrieves and summarizes content from web-based cancer clinical practice guidelines to generate reference-grounded responses to user queries.

Abstract

The widespread availability of medical information on the Internet has improved access to health-related knowledge for patients; however, it has also increased exposure to inaccurate medical information. Cancer care involves complex information, making it difficult for patients to identify accurate and evidence-based sources. Large language models enable natural language interaction but frequently generate hallucinations, limiting their application in medical information delivery. Retrieval-augmented generation (RAG) has the potential to mitigate these risks by grounding responses in external reference sources. This study describes the development and evaluation of a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines. The system extracts URLs from the table-of-contents pages of a guideline, processes page text using morphological analysis and cosine similarity based on term frequency–inverse document frequency, and constructs reference information from highly relevant pages. A large language model uses these references to generate responses constrained to guideline content. The JLCS Guidebook for Lung Cancer Patients and Families was used as the reference guideline. Question sets included both in-scope cancer types covered by the guideline and out-of-scope cancer types to evaluate response control. Medical information was successfully extracted from the table-of-contents pages, with 98% of extracted URLs containing referenceable medical content. For in-scope questions, the chatbot generated responses by summarizing guideline content, and no hallucinations were identified during manual review under the defined test conditions. For out-of-scope questions, the chatbot consistently declined to answer and indicated that the available information was insufficient. The web structure of the guideline facilitated efficient scraping and organization of reference content at the URL level. This protocol provides practical guidance for constructing artificial intelligence systems that deliver medical information using web-based clinical practice guidelines.

Introduction

The widespread use of the internet and social media has made it easier for patients to access medical information^1,2. Cancer-care information is often complex and extensive, making it particularly difficult for patients to identify sources that are accurate, relevant, and based on scientific evidence³. While online resources can support patient understanding, they also contain substantial amounts of incorrect medical information³, which can negatively affect patients’ decisions regarding their care⁴. Because cancer-related misinformation can affect patient outcomes⁵, there is an increasing need for artificial intelligence (AI) systems that can help individual users search, summarize, and interpret medical information⁶.

Large language models (LLMs) enable users to access information through natural language conversation⁷; however, the generation of misinformation (i.e., hallucinations) remains a serious concern in the medical field^8,9,10,11. One key source of misinformation is the quality and accuracy of the training data used to develop the chatbot. In addition, the static nature of model training means that knowledge can become outdated over time, limiting the ability of LLMs to provide current and contextually appropriate information^12,13. These limitations highlight the importance of effective knowledge-management strategies to ensure that chatbot responses remain accurate, relevant, and up to date. In particular, systems that can readily access and reference current evidence-based resources, such as clinical practice guidelines, are desirable in medical settings, where recommendations and standards of care continually evolve. To address this challenge, retrieval-augmented generation (RAG), which generates responses by incorporating information from external knowledge sources, has received increasing attention^{10,13,14,15,16,17}.

Although RAG has been increasingly applied to medical chatbots, limited attention has been paid to how clinical practice guidelines should be systematically incorporated as reference information sources¹⁸. Many clinical practice guidelines are publicly available on the web; however, it remains unclear whether their existing web structure can be directly serve as a retrieval framework for RAG systems¹⁸. In addition, practical methods for constructing reference information without manual knowledge-base development have not been well established.

In this study, we aimed to establish design principles for guideline-referencing AI systems in the medical field by developing and evaluating a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines, thereby enabling simple and timely access to guideline-based information. As a proof of concept, we evaluated the technical feasibility of guideline retrieval, response generation, and response restriction using the existing web structure of publicly available clinical practice guidelines, with the goal of proposing a reproducible development protocol for guideline-referenced RAG systems.

Protocol

This study does not involve human subjects who require protection from risks involved in research. Therefore, it does not require review by an institutional review board according to the Japanese ethics guidelines for medical research involving human subjects¹⁹. This study is reported in accordance with the Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis (TRIPOD)-LLM guidelines²⁰.

Refer to Figure 1 for a schematic overview of the guideline-based RAG chatbot protocol.

Figure 1. Workflow of the Guideline-Based Retrieval-Augmented Generation (RAG) Chatbot Protocol. Schematic overview of the guideline-based RAG chatbot workflow. URLs are extracted from the table-of-contents page of a web-based clinical practice guideline and used to construct reference information. User queries are converted into keywords, and webpage content is processed through tokenization, term frequency–inverse document frequency vectorization, and cosine similarity analysis to identify relevant guideline pages. Selected reference information is integrated and provided to a large language model to generate responses based solely on the referenced guideline content. The right panel summarizes the evaluation items used for extracted URLs, reference information, and chatbot responses. Please click here to view a larger version of this figure.

1. Preparation of Reference Information from Web-Based Guidelines

1. Extract all URLs from the table-of-contents pages of the target web-based clinical practice guidelines using BeautifulSoup 4 (HTML parsing library) implemented in the Python (version 3.12) programming language.
   NOTE: Implement the URL extraction script using the programming language with the following packages: HTML parsing library (version 4.12.3) and requests (webpage retrieval library; version 2.32.3). Install packages using pip. The complete source code for URL extraction is provided in Supplementary File 1. See the Table of Materials for software details.
2. Manually review all extracted URLs to confirm successful access and relevance to the target clinical practice guideline.
3. Classify each URL as containing medical or nonmedical information. Perform URL classification by one author (S.N.).
   NOTE: Medical information was defined as referenceable guideline content, including clinical questions (CQs), background questions, and question sets. Nonmedical information included links, society or organizational pages, and other ancillary content that was not suitable for reference.
4. Compile the validated URLs into a Microsoft Excel workbook (.xlsx format), with one URL recorded per row. Store the resulting URL list as the input dataset for subsequent text retrieval, preprocessing, and similarity analyses.

2. Keyword Generation from User Queries

1. Input the natural language query into ChatGPT (Generative Pre-trained Transformer, version 4o; keyword-generation model).
2. Prompt the model to extract two keywords corresponding to the user query.
3. Use the following prompt, written in Japanese, for keyword generation:
   ""
   (English translation: “Extract two keywords from the following text.”)
4. Append the user’s natural language query to this instruction and submit it to the keyword-generation model for keyword generation.
5. Do not manually modify any model parameters. Perform keyword generation using the default settings of the keyword-generation model.
6. Generate two keywords for each user query. Retain both keywords without further modification, filtering, or manual selection, and use them for subsequent similarity calculations.
7. Store the generated keywords in a Microsoft Excel workbook (.xlsx format). For each record, store the original user query and its corresponding generated keywords for subsequent similarity-based retrieval.

3. Text Processing and Similarity Calculation

1. Retrieve the full text content from each extracted URL using the webpage retrieval library and HTML parsing library. Extract text content from the HTML source using the BeautifulSoup get_text() function.
2. Preprocess the extracted webpage text and generated keywords by removing HTML tags and non-Japanese characters.
3. Perform Japanese morphological analysis using MeCab (version 0.996; Japanese morphological analyzer) with the default dictionary. Retain only nouns and concatenate them into a space-delimited text string.
4. Combine the generated keywords into a single text string and apply the same preprocessing and tokenization procedures used for webpage text.
5. Generate term frequency–inverse document frequency (TF–IDF) vectors from the processed webpage text and keyword text using the TfidfVectorizer implementation in the scikit-learn library (version 1.6.1) with default parameter settings.
   NOTE: All TfidfVectorizer parameters were left at their default values, and no custom parameter settings were applied.
6. Calculate the cosine similarity between each webpage vector and the keyword vector using the cosine similarity function.
7. Rank URLs in descending order according to cosine similarity scores. Do not apply additional tie-breaking criteria.
8. Select pages with a cosine similarity score of ≥0.2 as candidate reference information.
   NOTE: The cosine similarity threshold (0.2) was selected empirically during system development to prioritize retrieval recall and avoid excluding potentially relevant guideline content.
9. Retain all candidate pages meeting the similarity threshold.

4. Construction of Reference Information

1. Sequentially extract the text content of the selected pages from the top of the ranked list.
2. Concatenate the extracted texts to form a single reference document.
3. Before submission to the LLM, replace newline characters and consecutive whitespace characters in the reference text with a single space.
4. Truncate the concatenated reference text to the first 4,096 characters before providing it to the LLM.
   NOTE: The truncation limit was based on characters rather than tokens. A limit of 4,096 characters was selected empirically to ensure that the combined prompt and reference information remained within the input capacity of the GPT-3.5-turbo-16k response-generation model.

5. AI-Based Response Generation

1. Provide the constructed reference information and the original user query to the response-generation model via the application programming interface (API).
2. Instruct the AI using the following prompt (written entirely in Japanese) to ensure that responses are generated solely based on the provided reference information:
   “You are a medical information assistant who provides guidance to cancer patients. Generate responses based solely on the following reference information, and do not use your own general knowledge or reasoning. Responses should be detailed and easy to understand, approximately 500 characters in length. If the reference information does not contain sufficient information to answer the question, respond with: ‘There is insufficient information in the text,’ without using any speculation or general knowledge.”
3. Provide the reference information and user query together within a single user message. Structure the input as: “Reference Information: [reference text]” followed by “Question: [user query]”.
4. Generate responses using the response-generation model with the following settings: temperature = 0.1, maximum output length = 1,024 tokens, n = 1, and stop = None.
5. Obtain the AI-generated response for subsequent evaluation.
   NOTE: Submit the reference information and user query as a single user message. Provide the response-generation instructions separately as a system message.

6. Reference Guideline and Question Formulation

1. Select the JLCS Guidebook for Lung Cancer Patients and Families (Lung Guidebook)²¹, which is freely available online, as the reference information for the guideline-based RAG chatbot system.
2. Create two categories of evaluation questions based on cancer type: thymic tumors, which are within the scope of the guideline, and pediatric glioma, which is outside the scope of the referenced information.
3. Formulate four questions covering diagnosis and treatment for each cancer type:
   “What diagnostic methods are available for XX (e.g., thymic or pediatric glioma) tumors?”
   “What are the pathological diagnostic approaches for XX tumors?”
   “What treatment options are available for XX tumors?”
   “What are the surgical treatment options for XX tumors?”
4. Submit questions related to cancer types covered by the referenced web-based guideline to the chatbot. For example, submit a question regarding thymic tumor diagnosis while using the Lung Guidebook as the reference information.
5. Submit questions outside the scope of the reference guideline to the chatbot to evaluate its ability to refrain from using external or unrelated information. For example, submit a question regarding pediatric glioma while using the Lung Guidebook as the reference information.
6. Evaluate each question in an independent chat session. Do not carry over conversation history between questions.
7. Record the AI-generated responses for subsequent evaluation.

7. Response Evaluation

1. Perform manual review of all generated responses.
2. Evaluate whether each response contains hallucinations.
3. Assess whether each response is supported by information contained within the reference guideline.
   NOTE: Hallucinations were defined as responses containing nonexistent events, names, or terms that could potentially cause medical harm¹⁴. All generated responses were reviewed for accuracy and absence of hallucinations by one author with 8 years of experience at a university hospital, including patient support and medical communication for patients with cancer. Any uncertainty regarding response classification was resolved through discussion with a physician coauthor with more than 20 years of experience in cancer information services at the National Cancer Center Japan.
4. Consider a response to be supported only when all clinically relevant statements can be directly confirmed from the retrieved reference guideline without requiring additional knowledge or unsupported inference.
5. Classify each response using predefined outcome labels. Classify responses containing only information supported by the reference guideline as “hallucination absent.” Classify responses containing unsupported or fabricated information as “hallucination present.”

Results

Retrievability of Medical Content via Table-of-Contents Pages

The web-scraping architecture collected URLs from the table-of-contents page of the guideline. From the Lung Guidebook, 106 URLs were extracted from the table-of-contents page, of which 104 (98%) contained medical information (Supplementary Table 1). The effectiveness of the chatbot responses based on the guideline content pages is summarized in Table 1. The chatbot generated respon...

Discussion

In this study, we examined the utility of a guideline-based RAG chatbot that used a web-based clinical practice guideline—the JLCS Guidebook for Lung Cancer Patients and Families—as its reference information source. The chatbot generated responses grounded in the guideline content by exploiting the web structure of the guideline and retrieving pages based on their similarity to user queries. This approach demonstrated that similarity-based retrieval combined with guideline-referenced response generation can s...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Beautiful Soup 4 (version 4.12.3)	Beautiful Soup Project	N/A	Python library used for URL extraction and HTML parsing
ChatGPT version 4o	OpenAI	N/A	Used for keyword generation
GPT-3.5-turbo	OpenAI	N/A	Used for response generation
JLCS Guidebook for Lung Cancer Patients and Families	Japan Lung Cancer Society	N/A	Web-based clinical practice guideline used as reference information
MeCab (version 0.996)	MeCab Project	N/A	Japanese morphological analyzer
OpenAI API	OpenAI	N/A	Used for AI-based response generation
Python (version 3.12)	Python Software Foundation	N/A	Programming environment
Requests (version 2.32.3)	Requests Project	N/A	Python library used for webpage retrieval
scikit-learn (version 1.6.1)	scikit-learn Developers	N/A	Used for TF–IDF vectorization and cosine similarity calculation.
urllib.parse	Python Software Foundation	N/A	Python library used for URL normalization and joining