Investigating The Effect Of Automated Writing Evaluation Through Deep Neural Network And Teachers' Written Evaluation On English Writing Performance

Summary

Provision of feedback in the form of two NLP semantic-based features through the computer system and the teacher’s written feedback was employed to improve students' English writing fluency and correctness. Results showed positive results concerning the former.

Abstract

Computer-assisted language learning technologies have made the greatest gains in language learning, especially in the context of artificial intelligence (AI), over the past few years. There are various technologies, such as automated writing evaluation (AWE hereafter) and automated essay scoring (AES), that are considered pillars of language learning, not only in computer disciplines but also in education. The evaluation can be done only in the form of holistic scores using AWE technology, which has been quite effective in enhancing language learning and, therefore, cannot provide detailed or in-depth feedback. To provide detailed writing feedback on two major elements of a language (i.e., Grammar and Fluency), a computer-aided implementation system entailing neural network models, and two semantic-based natural language processing (NLP hereafter) methods have been considered. To that end, 90 Pakistani university students who were English second-language learners (ESL) were randomly assigned to the control, instructor feedback, and experimental groups. The computer-assisted evaluation encompassed a neural network. The results of the comparison test between the AWE baseline model and the instructors who graded showed a correlation between the computer-aided feedback that used these neural networks. The implications of such results lie in ESL writing pedagogy, especially in situations where linguistic accuracy and fluency pose a challenge.

Introduction

Feedback in writing addresses various attributes of language, such as organization, grammar, fluency, content, and mechanics, enabling learners to either rewrite or create a new piece of written text^1,2,3. Presently, teachers of English writing have been a major beneficiary because of the fast changes and advancements of computer-assisted language learning (CALL hereafter) and computer-aided writing assessment in the shape of AWE or AES and the grading of written essays⁴. Additionally, computer-assisted assessment of English writing gives diagnostic fe....

Protocol

All the software and tools used in the study are listed in the Table of Materials.

Grading criteria

The classification of evaluation adopted in the present study addresses two major domains, namely, fluency and correctness, which incorporate all the requirements for comprehensive evaluation of writing in English as a Foreign Language (EFL). These dimensions are intended to measure the key points of the writing quality that help in effective communication and demonstration of skill in language proficiency. The Fluency module measures naturalness, expressiveness, and coherence in w....

Results

Validity and reliability of the statistical approach

The system was tested on various complementary statistical methods that all give comprehensive evidence of the effects of the EG on the development of writing. This is a multifaceted analytical method that acknowledges that educational interventions are subject to complex statistical analysis that cannot be reduced to mere comparisons of groups but rather to the analysis of change patterns, the magnitude of effects, and the .......

Discussion

The experimental results demonstrate several significant achievements. First, the dual-model system achieved a strong correlation with human expert judgments (r = 0.923), substantially outperforming single-model approaches and contemporary AWE systems. Second, the controlled intervention study revealed significant improvements in ESL students' writing performance, with large effect sizes (d = 1.28) exceeding those reported in recent literature. Third, the system demonstrated superior performance across multiple evalu.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
SOFTWARE / LIBRARIES USED IN THE STUDY
Deep Learning Framework	PyTorch	1.13.1	Deep learning framework for neural network implementation
Pre-trained Models	Transformers (Hugging Face)	4.21.3	Pre-trained model loading, BERT and T5 model implementations
Language Model	BERT (Bidirectional Encoder Representations from Transformers)	bert-base-uncased	Fluency assessment, semantic coherence evaluation, contextual understanding
Language Model	FLAN-T5 (Fine-tuned Language Net T5)	pszemraj/flan-t5-large-grammar-synthesis	Grammatical error correction, text-to-text transformation, error detection
Numerical Computing	NumPy	1.23.5	Numerical computations, array operations for mathematical functions
Data Analysis	Pandas	1.5.2	Data manipulation, statistical analysis and preprocessing
Machine Learning	Scikit-learn	1.1.3	Statistical metrics calculation, correlation analysis, evaluation metrics
Visualization	Matplotlib	3.6.2	Data visualization, training progress plots, performance charts
Visualization	Seaborn	0.12.1	Statistical data visualization, correlation heatmaps
NLP Toolkit	NLTK (Natural Language Toolkit)	3.7	Text preprocessing, tokenization, linguistic analysis
NLP Processing	SpaCy	3.4.4	Advanced NLP preprocessing, POS tagging, syntactic analysis
Tokenization	Tokenizers	0.13.2	Fast tokenization for BERT WordPiece and T5 tokenization
STATISTICAL AND ANALYSIS SOFTWARE
Statistical Software	SPSS Statistical Software	Version 26.0	Statistical analysis, independent samples t-tests, ANOVA, correlation analysis
Training Dataset	Kaggle ASAP Dataset	2012 version	Training corpus reference (90% of AWE models use this dataset)
PYTHON OPTIMIZATION AND TRAINING LIBRARIES
Optimizer	torch.optim.AdamW	PyTorch 1.13.1	Model optimization with weight decay regularization (λ=0.01), β1=0.9, β2=0.999, ε=1×10-8
Loss / Activation	torch.nn.functional	PyTorch 1.13.1	Sigmoid activation, loss functions, dropout regularization
Data Loading	torch.utils.data.DataLoader	PyTorch 1.13.1	Progressive batch sizing (4→16), data loading and batching
Tokenization	transformers.AutoTokenizer	Transformers 4.21.3	BERT WordPiece tokenization, text preprocessing
Model Loading	transformers.AutoModel	Transformers 4.21.3	Pre-trained model loading for BERT and T5 architectures
Gradient Control	torch.nn.utils.clip_grad_norm_	PyTorch 1.13.1	Gradient clipping (threshold: 1.0) for training stability
LR Scheduling	torch.optim.lr_scheduler	PyTorch 1.13.1	Learning rate scheduling with linear decay and warmup
Metrics	sklearn.metrics	Scikit-learn 1.1.3	Pearson correlation, MAE, RMSE calculation for evaluation
PYTHON NEURAL NETWORK IMPLEMENTATION
Base Class	torch.nn.Module	PyTorch 1.13.1	Base class for custom neural network architectures (Fluency & Correctness modules)
Linear Layers	torch.nn.Linear	PyTorch 1.13.1	Linear regression head implementation (768→512→256→128→1 neurons)
Regularization	torch.nn.Dropout	PyTorch 1.13.1	Dropout regularization (0.1 for BERT, 0.3 for regression head)
Activation	torch.nn.functional.sigmoid	PyTorch 1.13.1	Sigmoid activation for fluency score normalization [0,1]
Activation	torch.nn.functional.relu	PyTorch 1.13.1	ReLU activation in regression layers for non-linear transformations
Transformer	transformers.BertModel	Transformers 4.21.3	12 transformer layers with multi-head self-attention for fluency assessment
Seq2Seq	transformers.T5For ConditionalGeneration	Transformers 4.21.3	Encoder-decoder architecture for grammatical error correction
Loss Function	torch.nn.MSELoss	PyTorch 1.13.1	Mean Squared Error loss function for fluency regression training
PYTHON EVALUATION AND METRICS LIBRARIES
Correlation	scipy.stats.pearsonr	SciPy 1.9.3	Pearson correlation coefficient for model-human agreement
Error Metric	sklearn.metrics.mean_absolute_error	Scikit-learn 1.1.3	Mean Absolute Error (MAE) calculation for prediction accuracy
Error Metric	sklearn.metrics.mean_squared_error	Scikit-learn 1.1.3	Root Mean Square Error (RMSE) for error magnitude assessment
Statistical Test	scipy.stats.ttest_ind	SciPy 1.9.3	Independent samples t-test for statistical significance testing
Correlation Matrix	numpy.corrcoef	NumPy 1.23.5	Correlation matrix calculation for inter-rater reliability
Data Correlation	pandas.DataFrame.corr	Pandas 1.5.2	Data correlation analysis and statistical reporting
Visualization	matplotlib.pyplot	Matplotlib 3.6.2	Performance visualization, correlation plots, training progress charts
Heatmap	seaborn.heatmap	Seaborn 0.12.1	Correlation matrix visualization and statistical data presentation
PYTHON TEXT PROCESSING AND NLP LIBRARIES
Text Processing	re (Regular Expressions)	Python 3.9+ built-in	Text pattern matching, data cleaning and preprocessing
Config Handling	json	Python 3.9+ built-in	Configuration file handling, model parameter storage
Serialization	pickle	Python 3.9+ built-in	Model serialization and checkpoint saving/loading
Progress Tracking	tqdm	4.64.1	Progress bars for training loops and data processing
Logging	logging	Python 3.9+ built-in	Training progress logging, error tracking and debugging
Reproducibility	random	Python 3.9+ built-in	Random seed setting for reproducible experiments
File System	os	Python 3.9+ built-in	File system operations, model path management
CLI Parsing	argparse	Python 3.9+ built-in	Command-line argument parsing for training configurations
COMMERCIAL AWE / AES PLATFORMS (Referenced)
Commercial Platform	Pigai.org	Commercial AWE platform	Chinese EFL writing assessment, automated feedback systems
Commercial Platform	Bingo English	Commercial AWE system	English language learning support, writing skill development
Educational Platform	My Access	Educational writing platform	Student writing assessment and feedback delivery
Scoring Engine	E-rater	ETS automated scoring engine	Standardized test essay scoring, holistic evaluation
Assessment Tool	I-write	Writing assessment tool	Academic writing evaluation and diagnostic feedback
Practice Service	Criterion	ETS writing practice service	Writing skill development and automated feedback
AI Language Model	ChatGPT	OpenAI language model	AI-assisted writing feedback and evaluation (referenced in literature)
DEEP LEARNING ARCHITECTURES (Referenced in Literature)
Architecture	Recurrent Neural Networks (RNN)	Cai, 2019	Sequential text processing — writing feedback systems and automated evaluation
Architecture	Long Short-term Memory (LSTM)	Jin et al., 2018	Long-range dependency modeling — automated essay scoring and sequence analysis
Architecture	Convolutional Neural Networks (CNN)	Dong et al., 2017	Local pattern recognition — text classification and feature extraction for scoring
Architecture	Transformer-LSTM Hybrid	Xuan, 2025	Combined contextual and sequential processing — automatic scoring and feedback generation
Architecture	Variational Autoencoders (VAEs)	Kumar et al., 2024	Generative modeling — enhanced writing skill development and personalized feedback
Architecture	Multi-Agent Systems	Thompson et al., 2024	Collaborative AI processing — advanced writing assistance (AcademiCraft platform)
Mechanism	Attention Mechanisms	Dong et al., 2017	Self-attention and multi-head attention — improved AES performance and contextual understanding
TRADITIONAL MACHINE LEARNING TECHNIQUES (Referenced)
Statistical Method	Bayes' Theorem	Rudner & Liang, 2002	Traditional ML approach — early AES/AWE development and probabilistic scoring
Statistical Method	Linear Regression	Phandi et al., 2015	Statistical modeling — feature-based writing assessment and score prediction
Learning Method	Rank Preference Learning	Chen & He, 2013	Ranking-based methodology — comparative essay scoring and preference modeling
Language Model	N-gram Models	Xie et al., 2015	Statistical language modeling — grammatical error correction and pattern recognition
Architecture	Encoder-Decoder Architecture	Ge et al., 2018	Sequence-to-sequence modeling — grammatical error correction and text transformation
EVALUATION STANDARDS AND FRAMEWORKS
Assessment Standard	IELTS Writing Assessment	Scoring rubric adaptation	Standardized evaluation criteria for fluency and correctness
Assessment Standard	TOEFL Writing Assessment	Academic writing standards	Proficiency evaluation and standardized scoring
Statistical Measure	Cohen's d Effect Size	0.2=small, 0.5=medium, 0.8=large	Practical significance assessment for intervention effectiveness
Reliability Measure	Inter-rater Reliability (κ)	Fluency: 0.847 / Correctness: 0.823	Kappa coefficient — human evaluator consistency and agreement validation