Name: Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems (Video) | JoVE Journal
Uploaded: 2025-06-13T13:10:00.000+00:00
Duration: 5 min 47 s
Description: Learn how to synthesize biomedical knowledge using RUGGED, a platform combining LLMs with retrieval-augmented generation for hypothesis validation. This method enables researchers to explore literature and knowledge bases through explainable AI.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

The authors would like to thank Dr. Alex Bui for his guidance and thoughtful discussion. In addition, we thank Dr. Ding Wang for his helpful discussions. This work was supported in part by NIH 1U54HG012517-01 to P.P., K.W., and W.W.; NIH T32 HL13945 to A.R.P.; National Science Foundation Research Traineeship (NRT) 1829071 to A.R.P.; and the TC Laubisch Endowment to P.P. at UCLA.

The authors have nothing to disclose.

The RUGGED protocol leverages modern language models with up-to-date information to empower investigators to dynamically explore the evolving biomedical landscape and uncover new knowledge. This human-computer interaction propels an innovative process that exemplifies the efficiency of the machine (RUGGED) and the expertise and judgment of the investigator. This protocol is designed to be executed in the outlined sequence. Step 1 details the software installation. Step 2 and and step 3 are essential for preparing biomedical literature and resources, while&#160;step 4 indexes this information for retrieval-augmented generation and user interaction with the LLM system. Time-intensive steps may run concurrently, and/or sequentially. For example, creating the Neo4j graph (step 4.2.2) can begin during prediction analysis (step 3), and indexing can begin after constructing the knowledge graph (step 2.3) and text-mining (step 2.1). These steps must be repeated to gain the final outcome of these intermediate results. While designed for biomedical information retrieval, this protocol, with minor modifications, may also handle other text and graph data, such as in-house data, clinical notes, or electronic health records. Data formatting details are in step 4.2.
The operation of this platform relies on the proper installation and interconnection of several technologies, including language models, graph databases, and vector databases (see Table of Materials). To verify these services are properly installed and connected, test scripts are provided in the `test` folder within the GitHub repository. External services may incur costs, with prices subject to change by the vendor. These optional services also have locally hosted alternatives, requiring only sufficient computational resources. However, these alternatives may impact model performance and/or convenience, making them unsuitable for some use-case scenarios.
With the rapidly evolving LLM landscape, new landmark models and task-specific models are released regularly. At the time of this report, the most appropriate models were chosen for the task. Users can choose which LLM to use by updating the configuration file accordingly (see steps 1.3.2-1.3.4). Model selection depends on their relevance to a particular use case. For example, incorporating models focused on ensuring model responses are fair, censored, and free of hate speech73,74,75,76,77,78, into this workflow is essential for ethical considerations. Furthermore, prompt engineering is essential to guide reliable and responsible behavior from the LLM79,80,81,82. The prompts crafted for the RUGGED workflow are tailored to the employed models and presented use cases. To fine-tune the prompts for a different use case, users can edit prompts within the RUGGED workflow in the `configuration` folder within the `prompts.json` file.
While RAG systems aim to reduce hallucinations in LLMs by grounding responses in evidence, these models may still lead to inaccurate information or generally true, non-specific responses. A benchmark comparison of RUGGED against GPT-4o is provided in Supplementary File 1, Section B. Model hallucinations often occur when retrieved information exceeds the model&#39;s context window, analogous to dementia with memory loss and inability to locate the data content, resulting in inaccurate responses83,84,85. Choosing a suitable LLM model helps to mitigate this issue. For instance, GPT-4o has a context limit of 128k tokens, significantly more than GPT-3.5 Turbo&#39;s 16k token limit, albeit at a higher cost to the user. Furthermore, LLMs fine-tuned with specific domain knowledge can potentially enhance the accuracy and specificity of responses in biomedical applications86,87,88. Despite these measures, it is essential to cross-check the information before proceeding with costly wet lab experiments.
RUGGED leverages explainable AI within a RAG pipeline to scrutinize link predictions, identifying both reliable and previously undiscovered relationships. While traditional RAG systems rely on bulk similarity-based retrieval, this approach connects explainability with a targeted response augmentation. Table 4 highlights the model&#39;s strong performance, demonstrating high recall (validation: 0.975 test: 0.976) and balanced F1-scores (validation: 0.796, test: 0.797), indicating reliability in identifying true positives, albeit with a higher rate of false positives. The model&#39;s robustness is further supported by its AUROC (validation: 0.963, test: 0.964) and AUPRC (validation: 0.971, test: 0.972) values. Precision (validation: 0.673, test: 0.674), however, could benefit from threshold tuning, incorporating detailed node features, or improved handling of class imbalance. The model&#39;s effectiveness is highly dependent on the input knowledge graph; overfitting is a risk with smaller graphs, while larger graphs demand greater computational resources. However, any RAG-based approach depends heavily on the quality of the data underlying the retrieval. For example, the construction of a knowledge graph is often time- and labor-intensive due to intrinsic noise on the original graph. This requires manual effort to denoise and label as well as ongoing costs for maintenance and update for the databases.
RUGGED&#39;s primary use is in knowledge synthesis and hypothesis exploration. By investigating various hidden relationships, such as disease mechanisms and drug treatments, RUGGED efficiently conducts literature triage. To reduce computational burden, most applications can be hosted on a server (e.g., AWS or computational server) and configured to update periodically with the latest information. Furthermore, this workflow can be adapted to accomplish domain-specific applications, such as serving as a platform to include patient data with local models to uphold security, privacy, and confidentiality. Beyond biomedical research, RUGGED&#39;s modular design allows it to support tasks across information retrieval, inference, and summarization by customizing the RAG pipeline and prompt engineering strategies tailored to the target domain. Successful adaptation necessitates careful consideration of domain-specific challenges, such as pre-processing of diverse data formats and evaluating the appropriate models for task- and domain-specific needs.

The hypothesis exploration process in the biomedical enterprise is essential to uncover novel molecule-drug-disease interdependencies underlying pathogenesis and to unlock therapeutic potential1,2. This process draws evidence from existing biomedical knowledge, synthesizing new findings based on logical leads embedded within peer-reviewed literature (e.g., &#62;36 reports from PubMed), and integrating high-confidence curated evidence rooted among biomedical knowledge bases. Recent advancements reduce laborious manual effort by applying text mining on literature corpra3,4,5 as well as employing graph-based analyses6,7,8,9 to synthesize relevant information and uncover new avenues for investigation. Despite these efforts, current approaches often do not support deep contextual understanding due to fragmented data. Furthermore, they lack the ability to draw evidence-based inferences and to interactively explore new hypotheses.
Recent developments in Large Language Models (LLMs) shed new light on these challenges, demonstrating high-level contextual understanding by training on vast amounts of information across multiple disciplines10,11,12. In the biomedical domain, LLMs have shown a promising role in extracting patient information13 and general clinical question answering14,15, whereas applications in domain-specific question answering16 and utilities in primary clinical care17 remain to be explored. These models exhibit the ability to reason and draw inferences from complex datasets, rendering them potentially suited for conducting hypotheses exploration and knowledge synthesis. Furthermore, some models feature chat-like interaction to engage users and enable dynamic exploration of topics, surpassing the conventional boundaries of query-based search engines and knowledge bases18,19.
In addition to these potentials, LLMs face significant challenges, such as possible hallucination of information, displaying unwarranted confidence in potentially inaccurate explanations, lacking interpretability, and being susceptible to biased or inappropriate content20,21,22,23,24. Applied directly to guiding clinical decision-making, the LLM-derived responses and predictions have high stakes; any errors may potentially result in costly laboratory experiments or negatively affect patient health trajectories25,26. Thus, reliable and trustworthy LLM responses are paramount, as their advice must be firmly rooted in evidence. In these scenarios, interpretability is not a luxury but a necessity for understanding why these models make the predictions they do.
To this end, Retrieval-Augmented Generation (RAG) is a system designed to minimize LLM hallucinations, grounding LLM responses in evidence to enhance their accuracy and reliability27,28. This approach typically involves retrieval of relevant text passages, such as integrating an LLM (e.g., ChatGPT) with PubMed, allowing for the identification of relevant citations to user queries29,30. Not limited to text, retrieval on Knowledge Graphs (KGs) show promise in application to LLMs for tasks such as fact-checking31,32,33, transparent reasoning34,35,36, encoding knowledge37, improving question answering38, and completing knowledge graphs39. By encoding factual information from verified sources, KGs enhance the accuracy, transparency, and reliability of LLM responses. Link prediction techniques within these graphs leverage deep learning to identify previously hidden relationships among molecules, drugs, and diseases5,40,41. Recent advancements in explainable AI predictions further enhance the transparency and interpretability of these link prediction tasks, lending potential support to interpret biomedical hypotheses as a viable avenue for investigation42,43,44. These advancements ensure that LLM-generated responses are balanced and drawn from the evidence, significantly boosting their applicability in biomedical enterprise.
This protocol presents RUGGED (Retrieval Under Graph-Guided Explainable disease Distinction) as an accessible and efficient workflow for the exploration and validation of clinical therapeutic insights (Figure 1). This workflow protocol leverages the vast resources of biomedical literature and knowledge bases for the extraction and validation of relevant information, enabling query-tailored retrieval processes (Figure 2). An explainable artificial intelligence prediction model is employed to uncover interpretable and actionable insights from the existing biomedical knowledge, thereby enhancing the transparency and utility of predictive models. The completed workflow streamlines the exploration of knowledge graphs and model predictions via RAG-enabled LLMs, facilitating intuitive and informed interactions for investigators, clinicians, and clinical professionals.
This section lays the groundwork for the protocol, with steps to implement this approach described in the following section. Next, a translational clinical use case is showcased to demonstrate this approach, applied to the evaluation of drugs for molecular interactions as well as therapeutic strategies for cardiovascular medicine. Finally, the implications and discussion of this protocol are discussed.

<table><tbody><tr><td>Hardware/Software - Graphics Card and software driver</td><td>Nvidia</td><td>https://www.nvidia.com</td><td>A graphics card and its associated driver software are highly recommended to significantly reduce runtime for computationally intensive tasks, such as local LLM and predictive analyses. For devices equipped with an NVIDIA RTX GPU, download and install the necessary drivers and CUDA Toolkit from the NVIDIA website (https://developer.nvidia.com/cuda-downloads).</td></tr><tr><td>Software - Commercial Large Language Model Service</td><td>OpenAI</td><td>https://openai.com</td><td>RUGGED supports the OpenAI API for models such as GPT-3.5 and GPT-4o. To setup using OpenAI models, first obtain an OpenAI API Key. Proceed to OpenAI&#039;s website (https://openai.com/blog/openai-api) to create an account, load funds, and obtain an API key. This API key is required to enable RUGGED to use OpenAI models. Determine which LLM agents within RUGGED&#039;s system will use OpenAI models from their documentation (https://platform.openai.com/docs/models).&lt;br/&gt; NOTE: OpenAI API is a paid service. At time of publication, cost for GPT-4o is $5.00 per 1 million input tokens and $2.50 per 1 million output tokens (For more, visit https://openai.com/pricing).</td></tr><tr><td>Software - Containerization</td><td>Docker</td><td>https://www.docker.com</td><td>Docker aids in maintaining a consistent computational runtime environment, streamlining software installation and execution across different machines. To install Docker, visit the Docker website (https://www.docker.com/), click on &#039;Get started&#039;, download and install the appropriate version for the OS. Verify installation by typing `docker --version` in the terminal; successful installation reports the Docker version installed.</td></tr><tr><td>Software - Graph Database</td><td>Neo4j</td><td>https://neo4j.com</td><td>Neo4j is a graph database software that efficiently manages and queries graph-based nodes and relationships. RUGGED supports Neo4j in multiple forms: Docker container, Neo4j Desktop, or Neo4j AuraDB online server. Choose the option best suited to the use case.&lt;br/&gt; Setting up Neo4j as a Docker container. Run these commands to set up Neo4j in Docker, with the file-path for the folder (e.g., /Users/username/RUGGED) as &#039;PATH_TO_FOLDER&#039;. For more details on troubleshooting, refer to the Neo4j Docker website (https://hub.docker.com/_/neo4j).&lt;br/&gt; docker pull neo4j&lt;br/&gt; docker run &amp;ndash;name neo4j --net rugged_network --publish=7474:7474 --publish=7687:7687 -d -v &#039;PATH_TO_FOLDER&#039;\neo4j\data:/data neo4j&lt;br/&gt; NOTE: Initialize Neo4j in Docker for the first time by setting a username and password. Run the neo4j_setup.py script (e.g., python neo4j_setup.py) or via web interface at http://localhost:7474.&lt;br/&gt; Setting up Neo4j Desktop. If using Neo4j Desktop, download and install from Neo4j website (https://neo4j.com/). Create a new project by clicking &quot;New&quot;, then click &quot;Add&quot; to create a new Database Management System (DBMS). Select &quot;Local DBMS&quot;, set a password, click &quot;Create&quot;, then click &quot;Start&quot;. A green &quot;ACTIVE&quot; text indicates it is running.&lt;br/&gt; Setting up Neo4j AuraDB. Visit the Neo4j website at (https://neo4j.com/cloud/aura-free/) to create an account and log in. Select &quot;New Instance&quot; to create an empty instance and save the URI and initial password to access the bolt interface (e.g., bolt://myurl.neo4j.com). Click the play button to start the instance, which will display the connection URI in the information box.&lt;br/&gt; NOTE: Neo4j AuraDB offers a free tier up to 200,000 nodes and 400,000 relationships. For larger graphs, visit Neo4j pricing (https://neo4j.com/pricing).</td></tr><tr><td>Software - Local Large Language Model Service</td><td>Ollama</td><td>https://ollama.com</td><td>RUGGED supports the use of local models using Ollama (e.g., Llama3). To enable, first install Ollama on the device or download the Docker container. To install Ollama, visit the Ollama website (https://ollama.com/download) and follow installation instructions. To install Ollama on Docker, run the following command:&lt;br/&gt; docker pull ollama/ollama&lt;br/&gt; NOTE: At time of publication, there is no stable release for Ollama on Windows OS.</td></tr><tr><td>Software - Version control</td><td>Git</td><td>https://www.git-scm.com</td><td>Version control software enables efficient installation and updating of software. To install Git, visit the Git website (https://www.git-scm.com/), click on &#039;Downloads&#039;, download and install the appropriate version for the OS. Verify installation by typing `git --version` in the terminal; successful installation will report the version of Git installed.</td></tr></tbody></table>

evidence-based knowledge synthesis and hypothesis validation: navigating biomedical knowledge bases via explainable ai and agentic systems

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

The scale of biomedical knowledge, spanning scientific literature and curated knowledge bases, poses a significant challenge for investigators in processing, evaluating, and interpreting findings effectively. Large Language Models (LLMs) have emerged as powerful tools for navigating this complex knowledge landscape but may produce hallucinatory responses. Retrieval-Augmented Generation (RAG) is essential for identifying relevant information to enhance accuracy and reliability. This protocol introduces RUGGED (Retrieval Under Graph-Guided Explainable disease Distinction), a comprehensive workflow designed to support knowledge integration, to mitigate bias, and to explore and validate new research directions. Biomedical information from publications and knowledge bases are synthesized and analyzed through text-mining association analysis and explainable graph prediction models to uncover potential drug-disease relationships. These findings, along with the source text corpus and knowledge bases, are incorporated into a framework that employs RAG-enhanced LLMs to enables users to explore hypotheses and investigate underlying mechanisms. A clinical use case demonstrates RUGGED&#39;s capability in evaluating and recommending therapeutics for Arrhythmogenic Cardiomyopathy (ACM) and Dilated Cardiomyopathy (DCM), analyzing prescribed drugs for molecular interactions and potential new applications. The platform reduces LLM hallucinations, highlights actionable insights, and streamlines the investigation of novel therapeutics.

These representative results were obtained by following the procedure outlined in this protocol. A text mining association analysis was performed following the CaseOLAP LIFT protocol5 with default parameters, studying eight broad categories of cardiovascular diseases72 and their association with mitochondrial proteins (GO:0005739). In total, 635,696 reports through May 2024 were determined as relevant to these diseases; among them, 4,655 high-confidence protein-disease associations were identified to inform downstream analyses. A biomedical knowledge graph was constructed using the software code from Know2BIO using default settings in May 20249. The resulting knowledge graph consists of 219,450 nodes, 6,323,257 edges, as well as node features for 189,493 nodes with node descriptions, protein/gene sequences, chemical structure, etc. where available. An estimate of computational time for all steps in the protocol is presented in Table 1.
The RUGGED system was initialized by constructing the vector databases for both knowledge graph nodes and features as well as the CVD-relevant publications. All knowledge graph nodes, edges, and node features were processed with a chunk size of 20 tokens with the BART71 embedding model to prepare for RAG vector search. Similarly, original contributions and review articles were processed using a chunk size of 500 tokens and the BART embedding model to prepare for the RAG vector search. For literature retrieval, full-text publications greater than 500 tokens were hierarchically summarized based on the individual sections of a publication by the BART embedding model. The GPT-4o model was used for the remaining LLM agents in the system.
These representative results showcase an example use case to investigate potential drug therapeutics for Arrhythmogenic Cardiomyopathy (ACM) and Dilated Cardiomyopathy (DCM), identified as MeSH_Disease: D019571 and MeSH_Disease: D002311, respectively. A series of inquiries is outlined in Figure 3, with highlighted examples of model responses shown in Figure 4, and full response reported in Supplementary File 1, Section A. The direction of inquiry was adapted to the investigator-validated responses, crafting subsequent queries based on the results of the previous responses. The analysis revealed 11 drug candidates classified under beta blockers and antiarrhythmics. Novel avenues for therapeutic treatment were assessed using a Graph Convolutional Neural Network link prediction model on a subset of the complete knowledge graph, including nodes within 1-hop from study disease and drug nodes and their interconnections, with evaluation metrics reported in Table 4. The top 10 relevant edges for each prediction by the model were further examined by a graph explainability module, GNNExplainer44, to identify the top nodes and edges contributing to each prediction, respectively. The total cost of using commercial LLM for all steps of the RUGGED protocol for this use case is estimated at $1.50 at the time of writing.
<img alt="Biomedical knowledge extraction diagram; text mining, protein-disease links, graph database creation." class="xfigimg" src="/files/ftp_upload/67525/67525fig1v2.jpg" data-altupdatedat="1788304762000" data-alt="Figure 1"> 
Figure 1: Retrieval Under Graph-Guided Explainable disease Distinction (RUGGED) workflow. RUGGED consists of four primary components: (1) assembling and processing data from ethically sourced and professionally managed resources (e.g., PubMed and curated biomedical knowledge bases), (2) integrating peer-reviewed research findings into a unified knowledge graph, (3) structuring the text and graph data within database services, (4) modeling and predicting explainable relationships among biomedical entities within the knowledge graph, and (5) retrieving and synthesizing knowledge through a Retrieval Augmented Generation (RAG) workflow (Figure 2) to validate complex molecular relationships and explore AI-driven disease predictions. A human-in-the-loop review step can be conducted by the user to enhance the accuracy of the output. <a href="https://www.jove.com/files/ftp_upload/67525/67525fig1v2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Biomedical text and knowledge graph retrieval diagram; document validation; AI prediction integration." class="xfigimg" src="/files/ftp_upload/67525/67525fig2v2.jpg" data-altupdatedat="1788304762000" data-alt="Figure 2"> 
Figure 2: Retrieval architecture and bias mitigation workflow. The Retrieval Augmented Generation (RAG) framework employs multiple LLM agents, each executing specific tasks to support access to relevant information based on the user query. This system provides documented evidence for the user-facing GPT-based Reasoning Agent, facilitating user-agent interaction and synthesis of knowledge. (1) Biomedical Text Retrieval: Peer-reviewed original contributions and review articles are filtered based on their relevance to understanding disease associations. A vector database is constructed for author and editor-validated text evidence weighted based on the corresponding section of the publication, respectively: 70% Abstract, 10% Results, 10% Metadata, and 10% for all other subsections. A keyword search and similarity search against the text embedding of the user query together identify relevant documents. Summaries of each document are generated using a BERT-based summarizer, with the GPT-based Text Evaluator Agent refining the search to validate query-document relevance. (2) Knowledge Graph Retrieval: A BERT-based named entity recognition and GPT-based relation extraction module connects the user query to relevant entities in the knowledge graph. A similarity search in a vector database identifies pertinent nodes and edges. Data is retrieved from the Neo4j database via Cypher queries generated by the GPT-based Cypher Query Agent and refined by the Query Verification Agent. (3) The individual responses from the Biomedical Text Retrieval or Knowledge Graph Retrieval pipelines are presented to the Reasoning Agent, which synthesizes a concise response with minimal bias to the user's query. This system is guided to maintain accuracy and impartiality in presenting factual information. <a href="https://www.jove.com/files/ftp_upload/67525/67525fig2v2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Knowledge graph of prescribed drugs; diagram; beta blockers, anti-arrhythmic, anti-fibrotic drugs." class="xfigimg" src="/files/ftp_upload/67525/67525fig3.jpg" data-altupdatedat="1788304762000" data-alt="Figure 3"> 
Figure 3: Use case on knowledge synthesis and hypothesis exploration via tiered query cascade. This figure showcases a highlighted use case focusing on a chain of related questions and concepts an investigator and/or healthcare professional might pose to the RUGGED system. Queries from the user are presented to the system in numerical order, with arrows representing inferred logical and domain-specific reasoning among each question. The system retrieves from the implicit and relevant information (source shown in blue), responding to the query. Examples of system responses are presented in Figure 4. <a href="https://www.jove.com/files/ftp_upload/67525/67525fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Beta-blocker analysis graph; drug interaction diagram; cardiomyopathy study; prediction model." class="xfigimg" src="/files/ftp_upload/67525/67525fig4.jpg" data-altupdatedat="1788304762000" data-alt="Figure 4"> 
Figure 4: Use case cardiovascular pathology: elucidating CVD pathogenesis. Query-response pairs between the user and the RUGGED system are shown. In the upper left panel, questions 1-6 retrieve information by extracting information from the knowledge graph database to formulate evidence-rooted responses. Question 7 employs an explainable graph link prediction to identify top-scoring therapeutics. The query prompts a prediction analysis, which is executed and processed automatically by the system, and key findings are succinctly summarized. Question 8 evaluates literature evidence from the defined text data corpus that is retrieved as relevant evidence to verify, validate, and corroborate the predicted finding. System responses have been reviewed by a human-in-the-loop inspection process and modified for readability and brevity. A full transcript of these findings is detailed in the Supplementary File 1. <a href="https://www.jove.com/files/ftp_upload/67525/67525fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Steps</td>
			<td>Description</td>
			<td>Time</td>
		</tr>
		<tr>
			<td colspan="2">Accessing Biomedical Knowledge</td>
			<td>30% total</td>
		</tr>
		<tr>
			<td>Prepare biomedical literature corpus</td>
			<td>Connect to PubMed and PubMed Central, download and parse publication data for downstream tasks. </td>
			<td>20%</td>
		</tr>
		<tr>
			<td>Prepare knowledge base data</td>
			<td>Connect to biomedical knowledge bases, download and parse necessary information for downstream tasks.   </td>
			<td>5%</td>
		</tr>
		<tr>
			<td colspan="2">Information Extraction</td>
			<td>30% total</td>
		</tr>
		<tr>
			<td>CaseOLAP LIFT Text Mining Analysis</td>
			<td>Identify high level disease-protein relationships within the biomedical text corpus.</td>
			<td>25%</td>
		</tr>
		<tr>
			<td>Knowledge Graph Construction</td>
			<td>Connect and integrate disparate information from biomedical knowledge bases into a unified knowledge graph.</td>
			<td>5%</td>
		</tr>
		<tr>
			<td colspan="2">Prediction Analysis</td>
			<td>10% total</td>
		</tr>
		<tr>
			<td>Train Graph Neural Network</td>
			<td>Train the model on the biomedical knowledge graph data to learn hidden patterns within the graph.</td>
			<td>5%</td>
		</tr>
		<tr>
			<td>Relevance Ranking Analysis</td>
			<td>Apply explainability module to highlight the most pertinent nodes and edges relevant to study disease.</td>
			<td>2.5%</td>
		</tr>
		<tr>
			<td>Link Prediction</td>
			<td>Utilize explainability module to identify key nodes and edges contributing to new predicted edges.</td>
			<td>2.5%</td>
		</tr>
		<tr>
			<td colspan="2">Hypothesis Generation and/or Validation</td>
			<td>30% total</td>
		</tr>
		<tr>
			<td>Database Setup for Retrieval Augmented Generation </td>
			<td>Initialize the graph database for querying the knowledge graph and the vector database for text retrieval.</td>
			<td>25%</td>
		</tr>
		<tr>
			<td>Hypothesis Exploration</td>
			<td>Enable user interaction with RUGGED to access and scrutinize relevant information for hypothesis exploration.</td>
			<td>5%</td>
		</tr>
	</tbody>
</table>
Table 1: Workflow and rate-limiting steps. This table provides rough estimates of the computational time required for each stage of the workflow. Rate-limiting steps include accessing, extracting, and indexing biomedical knowledge necessary for retrieval-augmented generation. Hypothesis exploration may be repeated continuously without the need to re-execute rate-limiting steps.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Disease Category</td>
			<td>MeSH Tree Numbers</td>
			<td># PMIDs</td>
			<td># Original Contributions</td>
			<td># Review Articles</td>
		</tr>
		<tr>
			<td rowspan="2">Cardiomyopathies (CM)</td>
			<td>C14.280.238</td>
			<td rowspan="2">132,531</td>
			<td rowspan="2">102,337</td>
			<td rowspan="2">19,942</td>
		</tr>
		<tr>
			<td>C14.280.434</td>
		</tr>
		<tr>
			<td rowspan="2">Cardiac Arrhythmias (ARR)</td>
			<td>C14.280.067</td>
			<td rowspan="2">125,286</td>
			<td rowspan="2">92,374</td>
			<td rowspan="2">13,854</td>
		</tr>
		<tr>
			<td>C23.550.073</td>
		</tr>
		<tr>
			<td>Congenital Heart Defects (CHD)</td>
			<td>C14.280.400</td>
			<td>82,006</td>
			<td>54,023</td>
			<td>6,379</td>
		</tr>
		<tr>
			<td>Heart Valve Diseases (VD)</td>
			<td>C14.280.484</td>
			<td>72,016</td>
			<td>50,119</td>
			<td>5,743</td>
		</tr>
		<tr>
			<td>Myocardial Ischemia (IHD)</td>
			<td>C14.280.647</td>
			<td>256,986</td>
			<td>210,042</td>
			<td>30,223</td>
		</tr>
		<tr>
			<td>Cardiac Conduction System Disease (CCD)</td>
			<td>C14.280.123</td>
			<td>53,050</td>
			<td>35,399</td>
			<td>4,363</td>
		</tr>
		<tr>
			<td>Ventricular Outflow Obstruction (VOO)</td>
			<td>C14.280.955</td>
			<td>22,244</td>
			<td>15,504</td>
			<td>1,686</td>
		</tr>
		<tr>
			<td>Other Heart Diseases (OTH)</td>
			<td>C14.280.195 C14.280.282 C14.280.383 C14.280.470 C14.280.945 C14.280.459 C14.280.720</td>
			<td>114,085</td>
			<td>77,302</td>
			<td>11,799</td>
		</tr>
		<tr>
			<td></td>
			<td>Total</td>
			<td>635,696</td>
			<td>478,404</td>
			<td>69,690</td>
		</tr>
	</tbody>
</table>
Table 2: Biomedical literature statistics. This table details the study disease categories with their corresponding MeSH tree numbers and the number of PubMed documents retrieved from through May 2024, used as the corpus for text mining. A subset of these publications, consisting of original contribution research articles and review articles, is indexed into a vector database for retrieval by RUGGED during hypothesis generation.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Category</td>
			<td>Number of Nodes</td>
			<td>Number of Edges</td>
			<td>Data Source(s)</td>
		</tr>
		<tr>
			<td>Anatomy</td>
			<td>5,049</td>
			<td>122,533</td>
			<td>Bgee, PubMed, MeSH, Uberon, </td>
		</tr>
		<tr>
			<td>Biological Process</td>
			<td>27,047</td>
			<td>108,106</td>
			<td>Gene Ontology</td>
		</tr>
		<tr>
			<td>Cellular Component</td>
			<td>4,057</td>
			<td>52,238</td>
			<td>Gene Ontology</td>
		</tr>
		<tr>
			<td>Compound</td>
			<td>27,278</td>
			<td>3,292,028</td>
			<td>DrugBank, MeSH, CTD, UMLS, KEGG, TTD, SIDER, Inxight Drugs, Hetionet, PathFX, MyChem.info</td>
		</tr>
		<tr>
			<td>Disease</td>
			<td>21,938</td>
			<td>311,773   </td>
			<td>PubMed, MeSH, DisGeNET, SIDER, ClinVar, ClinGen, PharmGKB, MyDisease.info, PathFX, UMLS, OMIM, Mondo, DOID, KEGG</td>
		</tr>
		<tr>
			<td>Drug Class</td>
			<td>5,721</td>
			<td>8,283</td>
			<td>ATC</td>
		</tr>
		<tr>
			<td>Gene</td>
			<td>29,810</td>
			<td>943,419</td>
			<td>HGNC, GRNdb, KEGG, ClinVar, ClinGen,</td>
		</tr>
		<tr>
			<td>Molecular Function</td>
			<td>11,151</td>
			<td>47,086</td>
			<td>SMPDB, DisGENET, PharmGKB, MyGene.info</td>
		</tr>
		<tr>
			<td>Pathway</td>
			<td>52,012</td>
			<td>234,944</td>
			<td>Gene Ontology</td>
		</tr>
		<tr>
			<td>Protein</td>
			<td>20,740</td>
			<td>1,074,809</td>
			<td>Reactome, KEGG, SMPDB</td>
		</tr>
		<tr>
			<td>Reaction</td>
			<td>14,647</td>
			<td>128,038</td>
			<td>UniProt, Reactome, TTD, SMPDB, STRING, HGNC</td>
		</tr>
		<tr>
			<td>Subtotal</td>
			<td>219,450</td>
			<td>6,323,257</td>
			<td>Reactome</td>
		</tr>
		<tr>
			<td>Text-mining Associations</td>
			<td>8</td>
			<td>4,670</td>
			<td></td>
		</tr>
		<tr>
			<td>Total</td>
			<td>219,458</td>
			<td>6,327,927</td>
			<td></td>
		</tr>
	</tbody>
</table>
Table 3: Knowledge graph statistics. This table details 11 broad biomedical categories comprising the constructed Know2BIO knowledge graph, enriched with additional edges derived from text mining analysis and predictive analysis. The resulting knowledge graph and predictions are managed by the Neo4j graph database for retrieval by RUGGED during hypothesis generation.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td></td>
			<td>Accuracy</td>
			<td>Precision</td>
			<td>Recall</td>
			<td>F1-score</td>
			<td>AUROC</td>
			<td>AUPRC</td>
		</tr>
		<tr>
			<td>Validation</td>
			<td>0.7158</td>
			<td>0.6639</td>
			<td>0.8743</td>
			<td>0.7547</td>
			<td>0.8437</td>
			<td>0.8637</td>
		</tr>
		<tr>
			<td>Test</td>
			<td>0.703</td>
			<td>0.6367</td>
			<td>0.9455</td>
			<td>0.761</td>
			<td>0.8961</td>
			<td>0.9094</td>
		</tr>
	</tbody>
</table>
Table 4: Explainable AI model evaluation. This table reports the evaluation metrics for the knowledge graph link prediction using a two-layer graph convolutional neural network. Metrics were assessed by partitioning graph edges into 85% training, 5% validation, and 10% test datasets. Accuracy indicates the proportion of correctly classified predictions. Precision reports the proportion of correct positive predictions among all positive predictions. Recall measures the proportion of correct positive predictions among actual positive edges. The F1-score is the harmonic mean of precision and recall, balancing the two metrics. AUROC evaluates the model's ability to differentiate between positive and negative predictions. AUPRC quantifies the trade-off between precision and recall across different thresholds. With all metrics, higher values indicate better model performance.
Supplementary File 1: This file details the full model response from RUGGED and a comparison against GPT-4o. Section A presents the complete human-computer interaction with RUGGED, expanding on the chain-of-query approach outlined in Figure 3 and providing the full response beyond the summary highlighted in Figure 4. Section B evaluates GPT-4o's responses without retrieval against RUGGED's, assessing attributes such as precision, depth, confidence scoring, evidence reliability, and cost. <a href="https://www.jove.com/files/ftp_upload/67525/67525_Supplementary%20File%201.docx" target="_blank">Please click here to download this file.</a>

This article describes RUGGED (Retrieval Under Graph-Guided Explainable disease Distinction), which integrates Large Language Model (LLM) inference with Retrieval-Augmented Generation (RAG). It draws evidence from expert-curated biomedical knowledge bases and peer-reviewed biomedical publications to synthesize new knowledge from up-to-date information, identify explainable and actionable predictions, and pinpoint promising directions for hypothesis-driven investigations.

The scale of biomedical knowledge, spanning scientific literature and curated knowledge bases, poses a significant challenge for investigators in ...

evidence-based-knowledge-synthesis-hypothesis-validation-navigating

Research

JoVE Journal

Medicine

A computational protocol, CaseOLAP LIFT, and a use case are presented for investigating mitochondrial proteins and their associations with cardiovascular disease as described in biomedical reports. This protocol can be easily adapted to study user-selected cellular components and diseases.

A Knowledge Graph Approach to Elucidate the Role of Organellar Pathways in Disease via Biomedical Reports

In the following sections, we outline procedures for the preparation of alginate microspheres for use in biomedical applications. We specifically illustrate a technique for creating multilayered alginate microspheres for the dual purpose of cell and protein encapsulation as a potential treatment for type 1 diabetes.

Generation of Alginate Microspheres for Biomedical Applications

We describe a highly reproducible method for the permanent occlusion of a rodent major cerebral blood vessel.  This technique can be accomplished with very little peripheral damage, minimal blood loss, a high rate of long-term survival, and consistent infarct volume commensurate with the human clinical population.

Permanent Cerebral Vessel Occlusion via Double Ligature and Transection

The tibial nerve transection model is a well-tolerated, validated, and reproducible model of skeletal muscle atrophy. The model surgical protocol is described and demonstrated in C57Black6 mice.

Tibial Nerve Transection - A Standardized Model for Denervation-induced Skeletal Muscle Atrophy in Mice

Human endogenous retroviruses (HERV), which occupy 8% of the human genome, retain scarce coding capacities but a hundred thousand long terminal repeats (LTRs). A custom Affymetrix microarray was designed to identify individual HERV locus expression and was used on prostate cancer tissues as a proof of concept for future clinical studies.

Microarray-based Identification of Individual HERV Loci Expression: Application to Biomarker Discovery in Prostate Cancer

Here we describe a protocol using the web-based drug repurposing hypothesis generation tool: "RE:fine Drugs." This protocol can be modified to a user's preferences at the level of the query type (gene, drug or disease) and/or the range of available advanced options.

Drug Repurposing Hypothesis Generation Using the "RE:fine Drugs" System

The practical application and performance of methods for generation of representative tissue samples of porcine animal models for a broad spectrum of downstream analyses in biobank projects are demonstrated, including volumetry, systematic random sampling, and differential processing of tissue samples for qualitative and quantitative morphologic and molecular analyses types.

Sampling Strategies and Processing of Biobank Tissue Samples from Porcine Biomedical Models

The purpose of this work is to describe a protocol for creating a practical fat-water phantom that can be customized to produce phantoms with varying fat percentages and volumes.

Fat-Water Phantoms for Magnetic Resonance Imaging Validation: A Flexible and Scalable Protocol

We present a protocol and associated programming code as well as metadata samples to support a cloud-based automated identification of phrases-category association representing unique concepts in user selected knowledge domain in biomedical literature. The phrase-category association quantified by this protocol can facilitate in depth analysis in the selected knowledge domain.

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications

Presented here is a protocol for the standardized methodology of rodent tissue preparation after the ischemia-reperfusion experiment and guidelines for establishing lighting and camera setups for high-resolution image acquisition. This method is applicable to all experimental small-animal organ photography.

Rodent Heart and Brain Tissue Preparation for Digital Macro Photography after Ischemia-reperfusion

Learn how to synthesize biomedical knowledge using RUGGED, a platform combining LLMs with retrieval-augmented generation for hypothesis validation. This method enables researchers to explore literature and knowledge bases through explainable AI.

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems (Video) | JoVE Journal

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Explore More Videos