Development of a Guideline-Based Retrieval-Augmented Generation Chatbot Referencing Web-Based Cancer Clinical Practice Guidelines

Summary

This protocol describes the development and evaluation of a guideline-based retrieval-augmented generation chatbot that retrieves and summarizes content from web-based cancer clinical practice guidelines to generate reference-grounded responses to user queries.

Abstract

The widespread availability of medical information on the Internet has improved access to health-related knowledge for patients; however, it has also increased exposure to inaccurate medical information. Cancer care involves complex information, making it difficult for patients to identify accurate and evidence-based sources. Large language models enable natural language interaction but frequently generate hallucinations, limiting their application in medical information delivery. Retrieval-augmented generation (RAG) has the potential to mitigate these risks by grounding responses in external reference sources. This study describes the development and evaluation of a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines. The system extracts URLs from the table-of-contents pages of a guideline, processes page text using morphological analysis and cosine similarity based on term frequency–inverse document frequency, and constructs reference information from highly relevant pages. A large language model uses these references to generate responses constrained to guideline content. The JLCS Guidebook for Lung Cancer Patients and Families was used as the reference guideline. Question sets included both in-scope cancer types covered by the guideline and out-of-scope cancer types to evaluate response control. Medical information was successfully extracted from the table-of-contents pages, with 98% of extracted URLs containing referenceable medical content. For in-scope questions, the chatbot generated responses by summarizing guideline content, and no hallucinations were identified during manual review under the defined test conditions. For out-of-scope questions, the chatbot consistently declined to answer and indicated that the available information was insufficient. The web structure of the guideline facilitated efficient scraping and organization of reference content at the URL level. This protocol provides practical guidance for constructing artificial intelligence systems that deliver medical information using web-based clinical practice guidelines.

Introduction

The widespread use of the internet and social media has made it easier for patients to access medical information^1,2. Cancer-care information is often complex and extensive, making it particularly difficult for patients to identify sources that are accurate, relevant, and based on scientific evidence³. While online resources can support patient understanding, they also contain substantial amounts of incorrect medical information³, which can negatively affect patients’ decisions regarding their care⁴. Because cancer-related misinforma....

Protocol

This study does not involve human subjects who require protection from risks involved in research. Therefore, it does not require review by an institutional review board according to the Japanese ethics guidelines for medical research involving human subjects¹⁹. This study is reported in accordance with the Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis (TRIPOD)-LLM guidelines²⁰.

Refer to Figure 1 for a schematic overview of the guideline-based RAG chatbot protocol.

....

Results

Retrievability of Medical Content via Table-of-Contents Pages

The web-scraping architecture collected URLs from the table-of-contents page of the guideline. From the Lung Guidebook, 106 URLs were extracted from the table-of-contents page, of which 104 (98%) contained medical information (Supplementary Table 1). The effectiveness of the chatbot responses based on the guideline content pages is summarized in Table 1. The chatbot generated respon.......

Discussion

In this study, we examined the utility of a guideline-based RAG chatbot that used a web-based clinical practice guideline—the JLCS Guidebook for Lung Cancer Patients and Families—as its reference information source. The chatbot generated responses grounded in the guideline content by exploiting the web structure of the guideline and retrieving pages based on their similarity to user queries. This approach demonstrated that similarity-based retrieval combined with guideline-referenced response generation can s.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Beautiful Soup 4 (version 4.12.3)	Beautiful Soup Project	N/A	Python library used for URL extraction and HTML parsing
ChatGPT version 4o	OpenAI	N/A	Used for keyword generation
GPT-3.5-turbo	OpenAI	N/A	Used for response generation
JLCS Guidebook for Lung Cancer Patients and Families	Japan Lung Cancer Society	N/A	Web-based clinical practice guideline used as reference information
MeCab (version 0.996)	MeCab Project	N/A	Japanese morphological analyzer
OpenAI API	OpenAI	N/A	Used for AI-based response generation
Python (version 3.12)	Python Software Foundation	N/A	Programming environment
Requests (version 2.32.3)	Requests Project	N/A	Python library used for webpage retrieval
scikit-learn (version 1.6.1)	scikit-learn Developers	N/A	Used for TF–IDF vectorization and cosine similarity calculation.
urllib.parse	Python Software Foundation	N/A	Python library used for URL normalization and joining