Development of a Guideline-Based Retrieval-Augmented Generation Chatbot Referencing Web-Based Cancer Clinical Practice Guidelines

Summary

This protocol describes the development and evaluation of a guideline-based retrieval-augmented generation chatbot that retrieves and summarizes content from web-based cancer clinical practice guidelines to generate reference-grounded responses to user queries.

Abstract

The widespread availability of medical information on the Internet has improved access to health-related knowledge for patients; however, it has also increased exposure to inaccurate medical information. Cancer care involves complex information, making it difficult for patients to identify accurate and evidence-based sources. Large language models enable natural language interaction but frequently generate hallucinations, limiting their application in medical information delivery. Retrieval-augmented generation (RAG) has the potential to mitigate these risks by grounding responses in external reference sources. This study describes the development and evaluation of a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines. The system extracts URLs from the table-of-contents pages of a guideline, processes page text using morphological analysis and cosine similarity based on term frequency–inverse document frequency, and constructs reference information from highly relevant pages. A large language model uses these references to generate responses constrained to guideline content. The JLCS Guidebook for Lung Cancer Patients and Families was used as the reference guideline. Question sets included both in-scope cancer types covered by the guideline and out-of-scope cancer types to evaluate response control. Medical information was successfully extracted from the table-of-contents pages, with 98% of extracted URLs containing referenceable medical content. For in-scope questions, the chatbot generated responses by summarizing guideline content, and no hallucinations were identified during manual review under the defined test conditions. For out-of-scope questions, the chatbot consistently declined to answer and indicated that the available information was insufficient. The web structure of the guideline facilitated efficient scraping and organization of reference content at the URL level. This protocol provides practical guidance for constructing artificial intelligence systems that deliver medical information using web-based clinical practice guidelines.

Introduction

The widespread use of the internet and social media has made it easier for patients to access medical information^1,2. Cancer-care information is often complex and extensive, making it particularly difficult for patients to identify sources that are accurate, relevant, and based on scientific evidence³. While online resources can support patient understanding, they also contain substantial amounts of incorrect medical information³, which can negatively affect patients’ decisions regarding their care⁴. Because cancer-related misinformation can affect patient outcomes⁵, there is an increasing need for artificial intelligence (AI) systems that can help individual users search, summarize, and interpret medical information⁶.

Large language models (LLMs) enable users to access information through natural language conversation⁷; however, the generation of misinformation (i.e., hallucinations) remains a serious concern in the medical field^8,9,10,11. One key source of misinformation is the quality and accuracy of the training data used to develop the chatbot. In addition, the static nature of model training means that knowledge can become outdated over time, limiting the ability of LLMs to provide current and contextually appropriate information^12,13. These limitations highlight the importance of effective knowledge-management strategies to ensure that chatbot responses remain accurate, relevant, and up to date. In particular, systems that can readily access and reference current evidence-based resources, such as clinical practice guidelines, are desirable in medical settings, where recommendations and standards of care continually evolve. To address this challenge, retrieval-augmented generation (RAG), which generates responses by incorporating information from external knowledge sources, has received increasing attention^{10,13,14,15,16,17}.

Although RAG has been increasingly applied to medical chatbots, limited attention has been paid to how clinical practice guidelines should be systematically incorporated as reference information sources¹⁸. Many clinical practice guidelines are publicly available on the web; however, it remains unclear whether their existing web structure can be directly serve as a retrieval framework for RAG systems¹⁸. In addition, practical methods for constructing reference information without manual knowledge-base development have not been well established.

In this study, we aimed to establish design principles for guideline-referencing AI systems in the medical field by developing and evaluating a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines, thereby enabling simple and timely access to guideline-based information. As a proof of concept, we evaluated the technical feasibility of guideline retrieval, response generation, and response restriction using the existing web structure of publicly available clinical practice guidelines, with the goal of proposing a reproducible development protocol for guideline-referenced RAG systems.

Protocol

This study does not involve human subjects who require protection from risks involved in research. Therefore, it does not require review by an institutional review board according to the Japanese ethics guidelines for medical research involving human subjects¹⁹. This study is reported in accordance with the Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis (TRIPOD)-LLM guidelines²⁰.

Refer to Figure 1 for a schematic overview of the guideline-based RAG chatbot protocol.

Figure 1. Workflow of the Guideline-Based Retrieval-Augmented Generation (RAG) Chatbot Protocol. Schematic overview of the guideline-based RAG chatbot workflow. URLs are extracted from the table-of-contents page of a web-based clinical practice guideline and used to construct reference information. User queries are converted into keywords, and webpage content is processed through tokenization, term frequency–inverse document frequency vectorization, and cosine similarity analysis to identify relevant guideline pages. Selected reference information is integrated and provided to a large language model to generate responses based solely on the referenced guideline content. The right panel summarizes the evaluation items used for extracted URLs, reference information, and chatbot responses. Please click here to view a larger version of this figure.

1. Preparation of Reference Information from Web-Based Guidelines

1. Extract all URLs from the table-of-contents pages of the target web-based clinical practice guidelines using BeautifulSoup 4 (HTML parsing library) implemented in the Python (version 3.12) programming language.
   NOTE: Implement the URL extraction script using the programming language with the following packages: HTML parsing library (version 4.12.3) and requests (webpage retrieval library; version 2.32.3). Install packages using pip. The complete source code for URL extraction is provided in Supplementary File 1. See the Table of Materials for software details.
2. Manually review all extracted URLs to confirm successful access and relevance to the target clinical practice guideline.
3. Classify each URL as containing medical or nonmedical information. Perform URL classification by one author (S.N.).
   NOTE: Medical information was defined as referenceable guideline content, including clinical questions (CQs), background questions, and question sets. Nonmedical information included links, society or organizational pages, and other ancillary content that was not suitable for reference.
4. Compile the validated URLs into a Microsoft Excel workbook (.xlsx format), with one URL recorded per row. Store the resulting URL list as the input dataset for subsequent text retrieval, preprocessing, and similarity analyses.

2. Keyword Generation from User Queries

1. Input the natural language query into ChatGPT (Generative Pre-trained Transformer, version 4o; keyword-generation model).
2. Prompt the model to extract two keywords corresponding to the user query.
3. Use the following prompt, written in Japanese, for keyword generation:
   ""
   (English translation: “Extract two keywords from the following text.”)
4. Append the user’s natural language query to this instruction and submit it to the keyword-generation model for keyword generation.
5. Do not manually modify any model parameters. Perform keyword generation using the default settings of the keyword-generation model.
6. Generate two keywords for each user query. Retain both keywords without further modification, filtering, or manual selection, and use them for subsequent similarity calculations.
7. Store the generated keywords in a Microsoft Excel workbook (.xlsx format). For each record, store the original user query and its corresponding generated keywords for subsequent similarity-based retrieval.

3. Text Processing and Similarity Calculation

1. Retrieve the full text content from each extracted URL using the webpage retrieval library and HTML parsing library. Extract text content from the HTML source using the BeautifulSoup get_text() function.
2. Preprocess the extracted webpage text and generated keywords by removing HTML tags and non-Japanese characters.
3. Perform Japanese morphological analysis using MeCab (version 0.996; Japanese morphological analyzer) with the default dictionary. Retain only nouns and concatenate them into a space-delimited text string.
4. Combine the generated keywords into a single text string and apply the same preprocessing and tokenization procedures used for webpage text.
5. Generate term frequency–inverse document frequency (TF–IDF) vectors from the processed webpage text and keyword text using the TfidfVectorizer implementation in the scikit-learn library (version 1.6.1) with default parameter settings.
   NOTE: All TfidfVectorizer parameters were left at their default values, and no custom parameter settings were applied.
6. Calculate the cosine similarity between each webpage vector and the keyword vector using the cosine similarity function.
7. Rank URLs in descending order according to cosine similarity scores. Do not apply additional tie-breaking criteria.
8. Select pages with a cosine similarity score of ≥0.2 as candidate reference information.
   NOTE: The cosine similarity threshold (0.2) was selected empirically during system development to prioritize retrieval recall and avoid excluding potentially relevant guideline content.
9. Retain all candidate pages meeting the similarity threshold.

4. Construction of Reference Information

1. Sequentially extract the text content of the selected pages from the top of the ranked list.
2. Concatenate the extracted texts to form a single reference document.
3. Before submission to the LLM, replace newline characters and consecutive whitespace characters in the reference text with a single space.
4. Truncate the concatenated reference text to the first 4,096 characters before providing it to the LLM.
   NOTE: The truncation limit was based on characters rather than tokens. A limit of 4,096 characters was selected empirically to ensure that the combined prompt and reference information remained within the input capacity of the GPT-3.5-turbo-16k response-generation model.

5. AI-Based Response Generation

1. Provide the constructed reference information and the original user query to the response-generation model via the application programming interface (API).
2. Instruct the AI using the following prompt (written entirely in Japanese) to ensure that responses are generated solely based on the provided reference information:
   “You are a medical information assistant who provides guidance to cancer patients. Generate responses based solely on the following reference information, and do not use your own general knowledge or reasoning. Responses should be detailed and easy to understand, approximately 500 characters in length. If the reference information does not contain sufficient information to answer the question, respond with: ‘There is insufficient information in the text,’ without using any speculation or general knowledge.”
3. Provide the reference information and user query together within a single user message. Structure the input as: “Reference Information: [reference text]” followed by “Question: [user query]”.
4. Generate responses using the response-generation model with the following settings: temperature = 0.1, maximum output length = 1,024 tokens, n = 1, and stop = None.
5. Obtain the AI-generated response for subsequent evaluation.
   NOTE: Submit the reference information and user query as a single user message. Provide the response-generation instructions separately as a system message.

6. Reference Guideline and Question Formulation

1. Select the JLCS Guidebook for Lung Cancer Patients and Families (Lung Guidebook)²¹, which is freely available online, as the reference information for the guideline-based RAG chatbot system.
2. Create two categories of evaluation questions based on cancer type: thymic tumors, which are within the scope of the guideline, and pediatric glioma, which is outside the scope of the referenced information.
3. Formulate four questions covering diagnosis and treatment for each cancer type:
   “What diagnostic methods are available for XX (e.g., thymic or pediatric glioma) tumors?”
   “What are the pathological diagnostic approaches for XX tumors?”
   “What treatment options are available for XX tumors?”
   “What are the surgical treatment options for XX tumors?”
4. Submit questions related to cancer types covered by the referenced web-based guideline to the chatbot. For example, submit a question regarding thymic tumor diagnosis while using the Lung Guidebook as the reference information.
5. Submit questions outside the scope of the reference guideline to the chatbot to evaluate its ability to refrain from using external or unrelated information. For example, submit a question regarding pediatric glioma while using the Lung Guidebook as the reference information.
6. Evaluate each question in an independent chat session. Do not carry over conversation history between questions.
7. Record the AI-generated responses for subsequent evaluation.

7. Response Evaluation

1. Perform manual review of all generated responses.
2. Evaluate whether each response contains hallucinations.
3. Assess whether each response is supported by information contained within the reference guideline.
   NOTE: Hallucinations were defined as responses containing nonexistent events, names, or terms that could potentially cause medical harm¹⁴. All generated responses were reviewed for accuracy and absence of hallucinations by one author with 8 years of experience at a university hospital, including patient support and medical communication for patients with cancer. Any uncertainty regarding response classification was resolved through discussion with a physician coauthor with more than 20 years of experience in cancer information services at the National Cancer Center Japan.
4. Consider a response to be supported only when all clinically relevant statements can be directly confirmed from the retrieved reference guideline without requiring additional knowledge or unsupported inference.
5. Classify each response using predefined outcome labels. Classify responses containing only information supported by the reference guideline as “hallucination absent.” Classify responses containing unsupported or fabricated information as “hallucination present.”

Results

Retrievability of Medical Content via Table-of-Contents Pages

The web-scraping architecture collected URLs from the table-of-contents page of the guideline. From the Lung Guidebook, 106 URLs were extracted from the table-of-contents page, of which 104 (98%) contained medical information (Supplementary Table 1). The effectiveness of the chatbot responses based on the guideline content pages is summarized in Table 1. The chatbot generated responses for all four clinical keywords when the questions fell within the scope of the guideline. For “thymic tumor diagnosis” and “pathological diagnosis,” three related URLs were extracted for each keyword, and all extracted URLs were deemed effective (100%). For “treatment methods” and “surgical treatment,” 15 URLs were extracted for each keyword, with 14 effective items (93%).

Query	Keywords	Referenced Content Items, n	Effective Reference Items, n (%)
What diagnostic methods are available for thymic tumors?	Thymic tumor diagnosis	3	3 (100)
What are the pathological diagnostic approaches for thymic tumors?	Thymic tumor pathological diagnosis	3	3 (100)
What treatment options are available for thymic tumors?	Thymic tumor treatment methods	15	14 (93)
What are the surgical treatment options for thymic tumors?	Thymic tumor surgical treatment	15	14 (93)

Table 1: Content of Reference Information and Chatbot Responses Based on the JLCS Guidebook for Lung Cancer Patients and Families. Summary of the user queries, generated keywords, number of referenced content items identified through similarity-based retrieval, and number of effective reference items containing medical information. Referenced content items represent the total number of related webpages identified at the URL level on the basis of each keyword. Effective reference items represent the number and percentage of unique webpages containing medical information after excluding duplicate URLs and nonmedical webpages.

Chatbot Responses to In-Scope Questions

Table 2 shows the chatbot responses to questions that fell within the scope of the provided guideline. When sufficient reference information was available, the chatbot generated responses by summarizing relevant portions of the guideline text. The generated responses consisted of approximately 350 Japanese characters. No hallucinations were identified during manual review under the defined test conditions. For these in-scope questions, multiple effective referenced items supported response generation (Table 1). The generated Japanese text was translated into English using ChatGPT version 5.2 and subsequently reviewed by the investigators for accuracy and consistency with the original Japanese responses.

Query	Responses					Character Count
What diagnostic methods are available for thymic tumors?	The diagnosis of thymic tumors requires both imaging diagnosis and pathological diagnosis. For imaging diagnosis, chest CT and chest MRI are performed. Thymoma and thymic carcinoma are solid tumors that do not contain air or fluid components within the tumor, and invasion into the surrounding normal tissues is known to be mild in thymoma and more pronounced in thymic carcinoma. When imaging studies determine that there is no invasion into surrounding tissues or no tumors distant from the primary lesion (metastases), the tumor is surgically resected, and pathological diagnosis is performed using the surgical specimen. When surgical resection is difficult, CT-guided biopsy is performed, and a diagnosis is made by obtaining a portion of the tumor tissue. In pathological diagnosis, thymomas are classified based on the proportion of normal lymphocytes within the tumor and the morphology of tumor cells, whereas thymic carcinomas are classified in detail according to histological subtype. For the determination of the clinical stage (stage), the extent of spread of the thymic tumor lesions is taken into consideration, and tumors are classified into stages I to IV based on three staging classification systems.					366
What are the pathological diagnostic approaches for thymic tumors?	In the pathological diagnosis of thymic tumors, pathological examination is performed by obtaining a part or the entirety of the tumor tissue. Thymomas are classified into subtypes such as type A and type AB based on the proportion of normal lymphocytes contained within the tumor and the morphology of the tumor cells. In contrast, thymic carcinoma is a malignant tumor characterized by cellular atypia, and squamous cell carcinoma is the most common histological subtype. Through pathological diagnosis, the biological characteristics and the degree of tumor progression are evaluated, which is useful for selecting appropriate treatment strategies and predicting prognosis. In addition, pathological diagnosis of thymic tumors allows determination of the disease stage (stage), providing important information for treatment planning. Clinical staging is classified using three systems—the Masaoka classification, the Masaoka–Koga classification, and the UICC–TNM classification—and tumors are categorized into four stages, from stage I to stage IV, according to the extent of tumor spread and invasion. Pathological diagnosis provides indispensable information for the appropriate treatment and prognostic evaluation of thymic tumors.					330
What treatment options are available for thymic tumors?	The treatment options for thymic tumors include surgical treatment (surgery), radiation therapy, and drug therapy (chemotherapy). Surgery is performed as a procedure in which the thymus is completely resected together with the tumor, and in cases of thymoma, additional treatment may not be required. In thymic carcinoma, postoperative radiation therapy may be added when the cancer has spread to surrounding tissues. Chemotherapy or chemoradiotherapy is performed in cases in which surgery is difficult or when there is recurrence or distant metastasis, and these are used as part of drug therapy. In addition, for advanced or recurrent thymic carcinoma, treatment centered on chemotherapy is performed, and multimodal therapy combining surgery and radiation therapy may also be administered as needed. Furthermore, for advanced or recurrent thymic carcinoma after chemotherapy, lenvatinib, a molecular targeted therapeutic agent, may be used. The appropriate combination of treatment modalities is selected according to the disease condition and the patient’s clinical status.					342
What are the surgical treatment options for thymic tumors?	Surgical treatment for thymic tumors is performed when the tumor is judged to be completely resectable. In surgery, it is generally standard practice to remove the entire thymus together with the tumor. In cases of thymoma, if the tumor is completely resected, no additional treatment is required. In contrast, in cases of thymic carcinoma, when cancer invasion into surrounding tissues is present, adjuvant postoperative radiation therapy may be added. When surgery is judged to be difficult, chemotherapy or chemoradiotherapy is performed first, and surgery is subsequently considered if it becomes feasible. If surgery cannot be performed, chemotherapy or chemoradiotherapy may be continued. In addition, even when surgery is possible, if complete resection cannot be achieved, postoperative radiation therapy or chemoradiotherapy may be administered to treat the residual tumor. Surgical treatment is selected appropriately according to the tumor type and the degree of disease progression.					342

Table 2: Responses Generated by the Guideline-Based Retrieval-Augmented Generation (RAG) Chatbot Based on the JLCS Guidebook for Lung Cancer Patients and Families. Representative responses generated by the guideline-based RAG chatbot for questions related to thymic tumors using reference information extracted from the JLCS Guidebook for Lung Cancer Patients and Families (https://www.haigan.gr.jp/public/guidebook/2024/). The table includes the user query, the chatbot-generated response, and the response length. Character counts were measured in the original Japanese-language responses before translation into English using ChatGPT version 5.2.

Chatbot Responses to Out-of-Scope Questions

The chatbot responses to questions that fell outside the scope of the Lung Guidebook are summarized in Table 3. Although a limited number of content items were referenced for these questions, the response-generation model determined that the available reference information was insufficient to generate an evidence-based response. Consequently, the chatbot returned responses indicating insufficient information for out-of-scope queries. The statements “There is insufficient information in the text” and “There are no references.” were predefined responses specified in the system prompt. The former was returned when the retrieved reference information was insufficient to answer the question, whereas the latter was returned when no reference information was retrieved.

Query	Keywords	Referenced Content Items, n	Chatbot Response
What diagnostic methods are available for pediatric glioma?	Pediatric glioma diagnosis	0	There are no references.
What are the pathological diagnostic approaches for pediatric glioma?	Pediatric glioma pathological diagnosis	13	There is insufficient information in the text.
What treatment options are available for pediatric glioma?	Pediatric glioma treatment methods	13	There is insufficient information in the text.
What are the surgical treatment options for pediatric glioma?	Pediatric glioma surgical treatment	12	There is insufficient information in the text.

Table 3: Chatbot Responses to Out-of-Scope Questions at the Clinical Question Level Based on the JLCS Guidebook for Lung Cancer Patients and Families. Responses generated by the guideline-based retrieval-augmented generation (RAG) chatbot for questions outside the scope of the reference guideline. The table summarizes user queries, generated keywords, the number of referenced content items identified through similarity-based retrieval, and the resulting chatbot responses. Referenced content items represent the total number of related webpages identified at the URL level on the basis of each keyword. Effective reference items were not calculated for out-of-scope questions because the retrieved content did not contain information relevant to the queried disease.

Supplementary File 1. Python Code for URL and Text Extraction from Web-Based Clinical Practice Guidelines. The code used to extract URLs and webpage text from the table-of-contents pages of web-based clinical practice guidelines. The extracted URLs and text content were used for subsequent text processing, similarity analysis, and reference information construction in the guideline-based retrieval-augmented generation (RAG) chatbot workflow.Please click here to download this file.

Supplementary Table 1. List of URLs Extracted from the Web-Based Guideline Table-of-Contents Page and Their Classification as Medical or Non-Medical Information. URLs extracted from the table-of-contents page of the JLCS Guidebook for Lung Cancer Patients and Families were manually reviewed and classified as medical or non-medical information. Medical information was defined as referenceable guideline content, including clinical questions (CQs), background questions, question sets, and supplementary educational materials relevant to patient guidance. Non-medical information included society or organizational webpages, navigation pages, hyperlinks, and other ancillary content not used as reference information for chatbot response generation. The table summarizes all extracted URLs and their classification outcomes. *URLs were extracted from the JLCS Guidebook for Lung Cancer Patients and Families table-of-contents page (https://www.haigan.gr.jp/public/guidebook/2024/).Please click here to download this file.

Discussion

In this study, we examined the utility of a guideline-based RAG chatbot that used a web-based clinical practice guideline—the JLCS Guidebook for Lung Cancer Patients and Families—as its reference information source. The chatbot generated responses grounded in the guideline content by exploiting the web structure of the guideline and retrieving pages based on their similarity to user queries. This approach demonstrated that similarity-based retrieval combined with guideline-referenced response generation can support reliable and efficient medical information delivery. Recent studies have highlighted both the potential and limitations of LLM-based medical chatbots, particularly regarding hallucinations and response reliability^12,13,14. Unlike previous studies that focused on general chatbot frameworks^10,12,13, the present protocol demonstrates a reproducible guideline-based RAG approach that retrieves patient-guideline content at the URL level and generates responses grounded in identifiable reference information. This study therefore provides a transparent protocol for cancer information delivery rather than a fully validated clinical chatbot system.

Rather than training the AI on the entire guideline corpus, our system selected only the CQs most relevant to each user query as the reference information (Figure 1). The relevance between the question text and each CQ was quantitatively evaluated using the programming language–based cosine similarity calculations with TF–IDF vector representations. Based on these similarity scores, CQs were ranked and selected in descending order of relevance, enabling systematic extraction of appropriate information sources from within the guideline without reliance on subjective judgment (Table 1). Furthermore, concatenating the retrieved CQ texts and presenting them collectively to the AI helped preserve a consistent evidentiary basis for response generation, thereby improving the accuracy and contextual coherence of the generated responses. In RAG systems, the generation of correct information critically depends on accurate and appropriate reference sources¹⁴. This design allowed the system to compensate for information that could not be sufficiently covered by a single CQ, contributing to more comprehensive and clinically plausible answers. In addition, RAG enables flexible retrieval of up-to-date CQ content and is well suited for evidence-based medical applications^10,11. Although guideline-based RAG approaches have been described previously^11,18, the present protocol differs in its use of the existing CQ-based web structure of a patient-oriented guideline as the primary retrieval framework. This design provides a transparent and reproducible workflow that can be implemented without manual knowledge-base construction or model retraining.

Importantly, the chatbot restricted its responses to the scope of the referenced guideline. When questions fell outside the guideline content, the system explicitly refrained from generating answers, thereby reducing the risk of unsupported responses. The use of cosine similarity to control reference selection further enhanced the safety and reliability of the generated responses (Table 2). Reference pages were also retrieved for questions outside the scope of the guideline (Table 3). The cosine similarity values were higher for in-scope questions (0.20–0.65) than for out-of-scope questions (0.20–0.31; data not shown), but low similarity scores were still assigned to some guideline pages even when the content was not clinically relevant. This occurred because cosine similarity was calculated solely on the basis of keyword matching between the query and the guideline text. Importantly, these low-relevance references did not result in inappropriate responses, as the chatbot refrained from generating answers when sufficient reference information was unavailable. No hallucinations were identified during manual review under the defined test conditions. The cosine similarity threshold of 0.2 was selected empirically during preliminary testing to balance retrieval sensitivity and specificity. Although this threshold was effective under the present test conditions, systematic evaluation of threshold sensitivity was beyond the scope of this protocol study. Future studies should investigate the effects of alternative threshold settings using larger benchmark datasets and quantitative retrieval metrics. The observed behavior suggests that RAG-based output control may be useful in medical AI applications. However, the current evaluation was based on a limited number of in-scope and out-of-scope questions and was intended primarily as a proof-of-concept assessment of the proposed workflow. Therefore, the findings should not be interpreted as a comprehensive validation of clinical accuracy, safety, or generalizability.

This study has several technical limitations. We employed the Japanese morphological analyzer and GPT-3.5-turbo-16k (LLM for response generation). The LLM was selected because it was a widely available and stable model at the time of system development and enabled reproducible implementation of the proposed workflow. Morphological analyzers and LLMs have distinct characteristics; thus, the choice of models or libraries influences both the accuracy of information extraction and the content of the generated responses^22,23. Although newer models (e.g., GPT-4- or GPT-5-class LLMs) may improve performance and external validity²², evaluation of alternative models was beyond the scope of the present protocol study. Newer LLMs may offer improved reasoning ability, instruction following, and response quality; however, the primary objective of this study was to evaluate the feasibility of a guideline-based RAG framework rather than to compare the performance of different LLMs. Further optimization of these components may enable more efficient and accurate response generation, and validation using different model configurations is necessary to assess the generalizability of these findings. In addition, the present protocol was developed using a Japanese-language guideline and Japanese morphological analysis. Although the retrieval framework based on TF–IDF vectorization, cosine similarity, and RAG is language-independent in principle, implementation in other languages would require language-specific text-processing tools and validation. Although this study was limited to a single guideline and therefore does not allow direct comparison of different guideline structures, the CQ-level organization facilitated systematic extraction of guideline content and supported the construction of reference information for the guideline-based RAG chatbot. In particular, the web structure of the guideline—wherein each CQ is associated with a unique URL—played an important practical role in enabling efficient scraping and organization of reference content at the URL level. Guidelines with different formats, including PDF-based documents or websites without CQ-level URLs, may require alternative segmentation and retrieval strategies. Therefore, the generalizability of the present workflow to other guideline structures and medical specialties remains to be established. Future work should evaluate the applicability of this approach across multiple diseases, guideline formats, and information sources.

The findings of this study provide practical guidance for designing guideline-referencing AI systems and demonstrate that a guideline-based RAG chatbot referencing web-based clinical guidelines has potential as a tool for delivering cancer-related medical information. However, this study represents a proof-of-concept evaluation intended to demonstrate the technical feasibility of guideline retrieval, response generation, and response-restriction mechanisms and should not be interpreted as formal clinical validation of a medical information system. Clinical effectiveness, safety, and user outcomes were not evaluated and remain to be established through future studies. From an ethical and user-safety perspective, restricting responses to guideline-supported information may reduce the risk of unsupported or hallucinated responses. However, excessive refusal behavior may also reduce user satisfaction and perceived usefulness. Future work should therefore evaluate the balance between safety and usability while maintaining appropriate safeguards against misinformation. By leveraging the information structure of web-based guidelines and providing focused responses to user questions, this system may serve as a useful model for future applications of AI in medical information delivery.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Beautiful Soup 4 (version 4.12.3)	Beautiful Soup Project	N/A	Python library used for URL extraction and HTML parsing
ChatGPT version 4o	OpenAI	N/A	Used for keyword generation
GPT-3.5-turbo	OpenAI	N/A	Used for response generation
JLCS Guidebook for Lung Cancer Patients and Families	Japan Lung Cancer Society	N/A	Web-based clinical practice guideline used as reference information
MeCab (version 0.996)	MeCab Project	N/A	Japanese morphological analyzer
OpenAI API	OpenAI	N/A	Used for AI-based response generation
Python (version 3.12)	Python Software Foundation	N/A	Programming environment
Requests (version 2.32.3)	Requests Project	N/A	Python library used for webpage retrieval
scikit-learn (version 1.6.1)	scikit-learn Developers	N/A	Used for TF–IDF vectorization and cosine similarity calculation.
urllib.parse	Python Software Foundation	N/A	Python library used for URL normalization and joining