Explainable AI Framework for Accuracy, Fairness, and Learner Perception in English Writing Assessment

摘要

This study develops a three-tier evaluation framework and fairness mediation model to assess AI-assisted English writing systems. Using 764 cross-linguistic samples, results show accuracy disparities, fairness bias against non-native learners (especially Chinese A2 proficiency level), and fairness perception as the key mediator of user satisfaction, offering theoretical and practical implications.

摘要

In the context of global educational digital transformation, automated writing evaluation (AWE) has been widely adopted due to its real-time and standardized advantages; however, traditional accuracy-oriented frameworks often neglect equity concerns and learners' perceptions, thereby limiting transparency and educational value. To address this limitation, this research proposes an explainable AI (XAI) framework designed to provide transparent and interpretable feedback, allowing learners to understand and trust automated evaluation, and integrates a multi-level validation model, the Three-Level Evaluation Framework (TLEF), spanning technical accuracy, group and individual equity, and learner perception, together with the AI Fairness Mediation Model (AFMM). Using stratified random sampling, data were collected from 764 multilingual learners (native speakers of English, Chinese, and Spanish) across Common European Framework of Reference for Languages (CEFR) levels A2 to C1 through writing tasks, dual scoring by AI and human experts, and structured questionnaires. Instead of listing individual tests, multiple statistical analysis was employed to examine validity, fairness, and the learner-perception relationship. Statistical analyses combined correlation, root mean square error (RMSE), Equalized Odds testing, and Structural Equation Modeling (SEM). The findings reveal that while the AI-assisted writing evaluation (AWE) system (ETS Criterion) achieves overall validity (r = 0.82), significant disparities remain: Chinese native speakers demonstrate the lowest agreement with human raters (0.72) and the highest RMSE (median 2.15), fairness biases are most pronounced at lower proficiency levels (ΔEO = 0.15 for A2 learners), and perceived fairness fully mediates the link between perceived accuracy and learner satisfaction, with proficiency moderating fairness sensitivity. By reframing fairness and perception as essential dimensions of explainability, the research enhances the theoretical grounding of AWE and provides a practical pathway for increasing transparency, equity, and social acceptance in educational technologies.

引言

The intensive globalization of education and digital technologies has increased the need to evaluate the level of writing in English scientifically and credibly for language teaching, academic development, and career advancement¹. Conventional writing evaluations, as practiced by human rating, can measure subjective aspects of writing like the thoroughness of argumentation and cultural suitability², but are susceptible to long turnaround times, high labor expenses, and bias due to evaluator experience and leanings^3,4. These constraints are especially acute in lar....

方案

The ethical approval and participant recruitment process, including essay administration, dual scoring by ETS Criterion and experts, learner-perception evaluation, and statistical analysis, are summarized in this section. It highlights how accuracy, fairness, and SEM-based perception modeling are integrated into a unified XAI validation pipeline. The XAI-driven AWE evaluation framework is illustrated in Figure 1.

Procedure:

The procedure involved several steps. First, IRB approval was obtained, and informed consent was collected from all participants. Independent, de....

结果

The section presents the research results based on five analytical dimensions: experimental design, participant characteristics, scoring accuracy, fairness assessment, and modeling of learning and perception. The outcomes include statistical performance, group differences, fairness disparities, and SEM-based mediation and moderation.

Experimental setup

The key software steps involved setting up ETS Criterion through its API to automatically score th.......

讨论

The research explored an AWE system under a three-level approach, encompassing technical accuracy, group and individual fairness, and learner perception, and identified that overall validity and systematic group differences are simultaneously present. There were strong correlations between AI and expert ratings (aggregate r = 0.82), but differences were observed by subgroup (native r = 0.89 vs. non-native r = 0.76; Chinese r = 0.72; Table 6). The distributions of RMSEs also indicated higher errors and variability in Chin.......

材料

本文使用的材料清单

姓名	公司	目录编号	评论
Data Storage System	加密、访问控制的服务器用于存储匿名数据。	机构服务器	存储-002
ETS标准系统	用于评分写作任务的AI辅助写作评估系统。	教育测试服务（ETS）	ETS-001
公平与准确分析工具	RMSE、均衡赔率和统计分析工具。	自定义脚本/统计包	工具-FA-001
<强>人类专家评级	由三位拥有超过10年经验的语言学家提供独立评级。	内部评级员	HR-EXP-003
<强>学习者感知问卷	一份包含8项公平与满意度的问卷，评分为7分李克特量表。	内部开发	QUES-008
统计软件（R 4.3.1）	用于数据分析，包括结构方程建模（SEM）。	R基金会	R-SW-431
<强>分层随机抽样数据	从CEFR A2至C1级别的764名多语学习者收集的数据。	研究参与者	数据-764
<强>写作任务提示	三个标准化论文题目，分别涉及全球化、在线教育和人工智能伦理。	基于 Moodle 的平台	提示-003