A Behavioral Protocol for Assessing Cognitive Effort and Strategic Revision in English-Chinese Translation and Sight Interpreting Tasks

Summary

This protocol integrates multimodal behavioral records to assess observable correlates of cognitive effort and the functional effectiveness of revision in English-Chinese written translation and sight interpreting. It combines keystroke logging, screen capture, audio analysis, transcript-based coding, and standardized quality assessment.

Abstract

This article presents a multimodal behavioral protocol for examining observable correlates of cognitive effort and strategic revision during English-Chinese written translation and sight interpreting. Keystroke logs and screen recordings are synchronized for written translation, whereas audio recordings and verbatim transcripts are used for sight interpreting. Revision and self-repair episodes are coded by type and by functional outcome (effective, neutral, or detrimental) independently of holistic output-quality ratings. In a demonstration with 62 participants assigned to novice and trained groups, both groups completed the two task conditions in counterbalanced order. Trained participants had higher mean output-quality scores in written translation (20.1 ± 2.1 vs. 16.8 ± 2.3) and sight interpreting (18.9 ± 2.4 vs. 15.4 ± 2.6), and a larger proportion of effective revisions or self-repairs. Because pauses, deletions, onset latency, and revision density are indirect and task-sensitive, they are interpreted as behavioral correlates that require triangulation rather than as direct measures of a unitary construct. The protocol is intended for controlled expository texts and can be adapted to other language pairs after recalibrating input-method handling, pause thresholds, text comparability, and scoring procedures.

Introduction

English-Chinese written translation and sight interpreting require source-text comprehension, cross-linguistic transfer, and target-language formulation, but they impose different task conditions. Written translation permits recursive reading and delayed editing, whereas sight interpreting requires oral production under tighter temporal constraints. Cognitive load, processing difficulty, and cognitive effort are related but non-interchangeable constructs: load concerns task demands, while effort concerns the resources a participant invests under those demands. Cognitive effort is latent and multidimensional; therefore, latency, pauses, deletions, and revision density are treated here as behavioral correlates that may reflect planning, monitoring, motor processes, or difficulty and must be interpreted jointly rather than as direct measurements of effort^{1,2,3,4,5,6,7,8}.

Process-oriented research combines product scores with time-aligned behavioral records to identify when and how translation decisions occur. Keystroke logging is informative for production and revision sequences, but Chinese input-method composition can separate physical key events from committed characters; synchronized screen recording is therefore needed to reconstruct the visible text state^9,10,11. Pause thresholds are analytical choices rather than universal cognitive boundaries, and shorter thresholds can materially change pause-based conclusions^8,12. The protocol consequently requires explicit synchronization checks, threshold sensitivity analyses, and separation of task-specific from shared measures.

Revision frequency alone does not indicate whether monitoring improves an output. Written revisions and oral self-repairs can target lexical choice, syntax, meaning, or style, but their functional outcome must be judged against the source meaning and the immediately preceding target segment. Sight-interpreting research on English-Chinese and Chinese-English processing shows that training, directionality, eye-voice coordination, and language-specific problem triggers can alter performance patterns^13,14,15,16. For this reason, episode type, episode outcome, final product quality, and delivery-related behavior are coded as separate variables.

The protocol is designed for adult learners or trained translators/interpreters working with short, non-specialist expository texts in controlled computer-based sessions. It is most suitable when keystroke and screen records can be captured for written production, and high-quality audio can be recorded for oral production. The framework can be adapted to other language pairs, but text matching, segmentation into information units, keyboard or input-method behavior, pause thresholds, and rating anchors must be revalidated for each language pair and writing system¹⁷. The demonstration compares task conditions rather than attributing all differences to modality alone, because text length, time limits, output mode, and revision opportunities also differ.

Protocol

All methods that involve the use of human subjects were performed in compliance with institutional guidelines and received explicit approval from the Human Research Ethics Committee of Guilin Institute of Information Technology (Approval No. UYHAJ2025899). Written informed consent was obtained from all participants prior to their participation. Please refer to the Table of Materials for a comprehensive list of the equipment and software used in this study.

1. Participant recruitment and group assignment

1. Recruit 62 participants, comprising English or translation majors and professional interpreter trainees. Verify that all participants are native Chinese speakers possessing normal or corrected-to-normal vision and hearing.
2. Assign participants with less than 12 months of structured translation training to the novice group, and assign those with at least 12 months to the trained group. Report prior written translation and sight-interpreting practice separately for all participants.
3. Screen participants before task assignment to exclude those with prior exposure to the source texts, uncorrected sensory impairment, or an inability to use the standardized Chinese input method.
4. Exclude tasks post-recording only for prespecified reasons, such as prohibited external-tool use, incomplete execution, corrupted files, or unusable audio.
5. Maintain a task exclusion log without personally identifying information. Report the final analyzed sample consistently in the text and Figure 1A–D.

2. Material selection and pilot evaluation

1. Select short English expository texts from general academic, educational, social, or public-communication domains. Use written-translation texts of approximately 180–220 words and sight-interpreting texts of approximately 120–160 words.
2. Match or statistically adjust text length, lexical difficulty, syntactic complexity, information density, and topic familiarity across task conditions.
3. Instruct three bilingual experts to independently rate lexical difficulty, syntactic complexity, information density, topic familiarity, and transfer suitability on a 1-to-5 point scale (1 = very low, 5 = very high).
4. Calculate a two-way random-effects intraclass correlation coefficient (ICC) with a 95% confidence interval for the averaged ratings. Revise items demonstrating poor agreement (ICC < 0.75).
5. Retain text pairs only when the standardized matching variables fall within a ±10% tolerance margin.
6. Define an information unit as the smallest source-text proposition that expresses one independently scoreable idea, relation, or event. Resolve boundary disagreements before task administration.
7. Pilot the instructions, input method, synchronization, sound quality, and time limits with participants who are not included in the main analysis. Record completion-time distributions and the proportion reaching each time limit.
8. Revise a text or time limit when pilot participants show ceiling/floor performance, persistent ambiguity, or recording instability. Document every change prior to launching the main study.

3. Task administration and recording setup

1. Test participants individually in a quiet, sound-attenuated room using standardized hardware and software configurations.
2. Read a standardized instruction script and allow one non-experimental practice item. Prohibit dictionaries, search engines, generative artificial intelligence, machine translation, and external corpora.
3. Counterbalance the two task orders (written translation first vs. sight interpreting first) within each training group and maintain a 5-minute rest interval.
4. Prepare the allocation list before recruitment completion and conceal the assignment until eligibility is confirmed. If multiple source texts are used, counterbalance text identity independently of task order and retain a text-ID variable for analysis.
5. Present the English source text continuously and instruct the participant to produce an accurate, natural Chinese translation in the designated input field. Permit unrestricted editing but strictly prohibit pasted external text.
6. Apply the 20-minute ceiling established in the pilot without displaying a countdown clock, and record whether the ceiling is reached.
7. Capture the entirety of the written translation process continuously, utilizing keystroke logging software synchronized with continuous screen recording. Retain the final translation outputs alongside the time-stamped log files for subsequent behavioral extraction.
8. Present the sight-interpreting source text on screen and instruct the participant to begin an oral Chinese rendition without written notes.
9. Apply the 6-minute ceiling consistently and record whether this ceiling is reached. Distinguish a time-limited observation from an ordinarily completed observation.
10. Record the oral output using digital audio capture software. Generate rigorous verbatim transcripts from these recordings to facilitate subsequent coding.

4. Extraction of behavioral process indicators

1. Extract written-task behavioral correlates from the log files, defining initial latency as the interval from stimulus onset to the first committed Chinese character.
2. Extract total task time, pause count and duration, deletion rate, revision density, and production rate. Retain the raw timestamps and the exact denominator used for each normalized measure.
3. Use 2 s as the operational written-pause threshold. Report whether pauses during cursor navigation, text selection, or input-method candidate selection are included.
4. Extract oral-task behavioral correlates from the waveform and verified transcript, defining onset latency as the interval from stimulus presentation to the first meaningful target-language utterance.
5. Exclude inhalation, throat clearing, and abandoned nonlexical sounds from onset latency. Extract total interpreting time, silent pauses, filled pauses, repetitions, false starts, self-repairs, and speech rate.
6. Define an oral silent pause as a within-utterance silence of at least 1.0 s. Inspect pause boundaries against the waveform and verified transcript, and exclude leading and trailing silence from within-utterance pause counts.
7. Complete a quality-control record for every administered task. Document whether the applicable process file is readable and complete, whether the transcript has been verified, and whether coding has been completed or adjudicated. Preserve the original record and track any corrections in the audit trail.

5. Coding of strategic revision and oral self-repair

1. Reconstruct each written revision by aligning the keystroke log, screen video, and final text.
2. Treat an episode as a continuous sequence of editing actions directed at the same target segment, and close the episode when production resumes elsewhere for at least 2 s or a different segment is edited.
   NOTE: For Chinese input, distinguish pre-commitment composition changes from post-commitment target-text deletions and code only the latter as textual revisions.
3. Assign one primary revision type based on semantic, syntactic, lexical, or stylistic changes. Allow a secondary tag for overlapping cases, but use the primary tag for mutually exclusive counts. Apply the worked examples provided in Supplementary File 1.
4. Produce a timestamped verbatim transcript before coding oral self-repair. Instruct the first transcriber to mark lexical content, filled pauses, repetitions, false starts, cut-offs, and unintelligible spans.
5. Have a second trained reviewer check the transcript against the audio and resolve discrepancies while preserving the original audio timestamps.
6. Define an oral self-repair as an overt reformulation that replaces, completes, or corrects an immediately preceding target-language segment. Do not infer covert monitoring from silence alone. Apply the same lexical, syntactic, semantic, and stylistic decision rules to oral self-repair.
7. Code an exact repetition without corrective change as disfluency, an abandoned start followed by a new construction as a false start plus repair, and an omission or mistranslation without overt reformulation as an output error.
8. When the audio is unclear or when two categories remain equally plausible, mark the episode as ambiguous. Retain provisional tags for ambiguous episodes and adjudicate them before reliability analysis.
9. Judge functional outcome by comparing the pre-episode and post-episode target segments against the source information unit and local target context. Make this judgment before revealing the final holistic score. Do not use a change in the holistic score as the definition of effectiveness.
10. Double-code at least 20% of outputs selected across both groups and task orders, and calculate reliability on the independent codes before adjudication.
11. Retrain and recode when inter-rater agreement falls below a Cohen’s kappa of 0.80. Document this exact threshold and any corrective actions taken.

6. Quality evaluation and statistical modeling

1. Score each final written translation and sight-interpreting rendition with the 25-point analytic rubric in Supplementary File 1.
2. Rate accuracy, completeness, lexical appropriateness, coherence, and target-language naturalness from 0 to 5 using the stated anchors. Do not interpolate scoring criteria after inspecting group membership.
3. Instruct raters to use both the audio and the verified transcript for sight interpreting dimensions. Analyze pause and disfluency measures separately from the rubric scores.
4. Train two independent raters using the codebook and at least 10 practice outputs, blinding raters to participant group and task order.
5. After independent scoring and episode coding, calculate reliability before discussion. Resolve disagreements by consensus and consult a third rater only when consensus cannot be reached.
6. Report a two-way random-effects, absolute-agreement intraclass correlation coefficient for averaged continuous quality scores. Report Cohen's kappa for the primary categorical revision code and a separate kappa for the effective/neutral/detrimental outcome code. State the exact number and percentage of double-coded episodes.
7. Verify that each process file is readable and complete prior to analysis, align written-task logs with corresponding screen records, and verify transcripts against the audio.
8. Record alignment failures, intelligibility issues, and corrective actions in a quality-control log. Normalize frequency measures using 100 source-text information units for pause frequency and 100 produced Chinese characters for written revision density.
9. Code functional outcome independently of the final holistic quality score. Code an episode as effective when the post-revision segment improves the text without introducing new problems, as neutral when it produces no material change, and as detrimental when it introduces an error.
10. Calculate the effective rate as effective episodes divided by all classifiable episodes, excluding ambiguous episodes from the denominator.
11. Analyze the demonstration as an exploratory repeated-measures dataset. Specify the fixed-effects model to predict total quality score using task condition, training group, pause metrics, revision density, effective rate, disfluency burden, and speech rate.
12. Include a participant random intercept, and include source-text random effects only if the design contains enough independently sampled texts.
13. Center continuous predictors, inspect residuals, and check collinearity. Report the estimation method, software version, β, SE, 95% CI, exact p values, variance components, AIC, BIC, and marginal/conditional R². Describe the analysis as exploratory and avoid confirmatory causal claims.
14. Export a reproducible analysis package containing de-identified data, dictionary, codebook, rubric, software details, scripts, and output. Confirm that every figure and table value can be traced to a named variable and analysis step before release.

Results

Participant baseline characteristics and data availability

The questionnaire contained 62 participants (31 novice and 31 trained), and the task sheet contained 124 records, indicating two completed task records per participant (Table 1). The novice group had less than 12 months of structured translation training (observed maximum, 8.6 months), whereas the trained group had at least 12 months (observed minimum, 13.3 months). Age, sex, English-learning duration...

Discussion

This study demonstrates a multimodal protocol for measuring observable process indicators and coding strategic revision in English-Chinese written translation and sight interpreting. The protocol does not directly measure a unitary cognitive-effort variable. Instead, latency, pauses, deletions, and revision patterns are interpreted as task-sensitive behavioral correlates whose meaning depends on triangulation with synchronized records and output-quality evidence^1,2

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Acoustic analysis software (Praat)	Paul Boersma and David Weenink	https://www.fon.hum.uva.nl/praat/
Desktop computer	Dell	OptiPlex
Digital audio recording software (Audacity)	Audacity Team	https://www.audacityteam.org/
Keystroke logging software (Inputlog)	University of Antwerp	https://www.inputlog.net/
Screen recording software (OBS Studio)	OBS Project	https://democreator.wondershare.com/
Statistical analysis software (SPSS)	IBM	Version 27