Readability and Quality of Large Language Model Patient Education for Trigeminal Neuralgia: A Cross-Sectional Study

Summary

Here, we present a protocol to systematically evaluate the readability, educational suitability, and quality of large language model–generated patient information on trigeminal neuralgia to benchmark models and guide patient-facing communication.

Abstract

Large language models (LLMs) are increasingly used for patient health education, yet the readability and educational quality of LLM-generated information on trigeminal neuralgia (TN) have been insufficiently evaluated. This cross-sectional benchmarking study compared TN educational content generated by five publicly available LLMs (Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5). Twenty frequently asked TN questions covering basic disease knowledge, etiology/risk factors, diagnosis, treatment, and prevention/rehabilitation were presented to each model using standardized prompts. Readability was assessed using seven established indices, educational suitability using the Patient Education Materials Assessment Tool for Understandability and Actionability (PEMAT), and overall information quality using the Global Quality Score (GQS). Two clinical experts independently evaluated all responses, with disagreements resolved by a senior adjudicator. Statistical analyses compared model performance, thematic differences, and correlations among the evaluation metrics. Significant differences were observed among the models for readability, PEMAT, and GQS scores. GPT-5 generated the most linguistically complex responses but achieved the highest ratings for educational suitability and information quality. In contrast, Wenxin Yiyan produced the most readable text but generally scored lower on PEMAT and GQS. Content category influenced readability, with prevention/rehabilitation and etiology/risk-factor topics being more difficult to read, whereas PEMAT and GQS remained relatively consistent across themes. Readability indices showed strong internal consistency and weak-to-moderate positive correlations with PEMAT and GQS, while PEMAT and GQS demonstrated a moderate positive correlation. These findings suggest that model selection influences expert-rated educational suitability and overall information quality, whereas topic complexity primarily affects readability. Because patient comprehension, satisfaction, trust, health outcomes, factual accuracy, and clinical safety were not evaluated, these results should be interpreted as an expert-rated benchmarking analysis rather than evidence of clinical readiness.

Introduction

Trigeminal neuralgia (TN) is a neuropathic pain syndrome characterized by recurrent episodes of unilateral, brief, and extremely severe facial pain commonly described as electric shock-like or stabbing. According to the International Classification of Headache Disorders, 3rd edition (ICHD-3), pain is typically localized to the distribution of one or more branches of the trigeminal nerve, is often intolerable in intensity, and can be triggered by innocuous stimuli such as light touch, facial washing, tooth brushing, speaking, or chewing. Attacks are marked by sudden onset and abrupt cessation, with durations ranging from fractions of a second to several minutes

Protocol

This study analyzed only AI-generated text and did not involve human participants, animals, biological specimens, medical records, identifiable personal information, or intervention in clinical care. Therefore, ethics review and formal informed consent were not applicable.

Study design

This cross-sectional study evaluated the readability, quality, and educational suitability of responses generated by five LLMs in response to frequently asked questions about TN.

Identification of commonly asked TN-related questions

On November 25....

Results

This study systematically evaluated the effects of large language models (LLMs) and different categories of health education content on the readability and quality of generated texts. First, we compared the performance of five LLMs—Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5—with respect to patient education suitability, overall information quality, and multiple readability metrics, including the PEMAT score, GQS, and established readability indices (ARI, FRES, GFOG, FKGL, CL, SMOG, and LW). Second, we exam.......

Discussion

This cross-sectional benchmarking study systematically compared the readability, educational suitability, and overall information quality of trigeminal neuralgia (TN) patient education materials generated by five publicly available large language models (LLMs). Four principal findings emerged. First, readability differed substantially across the models, with GPT-5 generating the most linguistically complex responses and Wenxin Yiyan producing the most readable content. Second, readability and expert-rated information qua.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
DeepSeek chat platform	DeepSeek	https://chat.deepseek.com/	Publicly accessible large language model interface used to generate responses
Doubao chat platform	Doubao	https://www.doubao.com/chat/	Publicly accessible large language model interface used to generate responses
GPT 5	OpenAI	https://chat.openai.com/	Publicly accessible large language model interface used to generate responses
R software, version 4.4.3	R Foundation for Statistical Computing	Not applicable	Statistical analysis and visualization
Readability Formulas online readability calculator	Readability Formulas	Not applicable	Online tool used to compute readability indices
Tongyi Qianwen chat platform	Tongyi Qianwen	https://www.tongyi.com/	Publicly accessible large language model interface used to generate responses
Wenxin Yiyan chat platform	Wenxin Yiyan	https://yiyan.baidu.com/	Publicly accessible large language model interface used to generate responses