Evaluating University Student Classroom Engagement, Reflective Expression, and Prosocial Intention Using Behavioral Observation and Learning Analytics

Summary

This protocol integrates classroom observation, reflective writing, prosocial-intention measurement, and learning analytics to evaluate university students' behavioral engagement, reflective expression, and socially oriented learning outcomes over a 4-week instructional window. It specifies ethics approval, instrument contents, software workflow, missing-data handling, troubleshooting procedures, and interpretation of representative outcomes, including observed implementation constraints.

Abstract

To address the need for multidimensional assessments of student involvement beyond traditional metrics, this protocol describes a structured approach for evaluating classroom engagement, reflective expression, learning analytics activity, and prosocial intention in undergraduate courses. The protocol combines interval-based behavioral observation, weekly structured reflective writing, de-identified learning-management-system records, and a pre/post prosocial-intention questionnaire. The revised procedures provide a reproducible behavioral coding model, reflective-expression rubric with score examples, software and data-management steps, and explicit troubleshooting guidance for observer disagreement, missing platform records, low reflection submission rates, poor camera visibility, participant attrition, and inconsistent prerequisite knowledge. This protocol should be used when researchers need a low-intrusion, reproducible way to combine live classroom behavior, student self-interpretation, and routine learning management system (LMS) activity during undergraduate instruction. It is less suitable for fully asynchronous courses without observable interaction, settings that prohibit independent observation or record linkage, or studies requiring biometric or emotion-recognition inference. The representative results are reported from the implemented empirical application after verified ethics approval, de-identification of course records, and repository preparation.

Introduction

Evaluating student participation in higher education requires a comprehensive, multi-source approach. Although student engagement is widely conceptualized as a multidimensional construct encompassing observable behavior, cognitive effort, and environmental interaction, current analytical approaches predominantly rely on superficial log data, such as login frequencies and system-recorded actions¹. This overreliance presents a significant challenge for practical classroom-based research. The most critical forms of instructional participation—direct peer interactions, cognitive reflection, and social contributions—often occur entirely beyond the limited scope of platform visibility².

The complexity of the modern university classroom has intensified with the adoption of hybrid pedagogical models, blending digital platforms with traditional face-to-face instruction. While systematic reviews of learning analytics confirm that digital traces offer valuable insights into time allocation, access patterns, and long-term retention rates, these metrics alone cannot determine whether a learner's attention is genuinely efficient or meaningful³. Consequently, recent scholarship advocates for grounding learning analytics in stronger theoretical frameworks by aligning digital traces with richer, context-dependent measures of participation, such as classroom behavior and reflective activities⁴.

Reflective writing provides a vital qualitative dimension to this assessment. Structured post-class reflections and journals capture students' interpretations of learning activities, clarify instructional challenges, and bridge theoretical content with subsequent practical actions. Recent empirical evidence conceptualizes reflective expression as an integrated, multi-component construct comprising descriptive recall, analytical inference, self-referential representation, and meaning generalization⁵. Furthermore, research indicates that providing students with repeated reflective guidance gradually strengthens their capacity for introspection; as the frequency of structured reflection increases during coursework, students' independent exploratory skills concurrently improve⁶.

A second major gap in current engagement research is the frequent marginalization of classroom socialization. While existing studies heavily emphasize individual behavior or cognition, university classrooms function as inherently social learning environments where students influence one another through collaboration and responsive interactions. Emerging evidence demonstrates that classroom climate and participation structures correlate strongly with prosocial behavior and a contribution-oriented mindset—outcomes that remain under-investigated compared to traditional metrics of academic performance or student satisfaction. Prosocial behavior in academic settings is not merely a byproduct of emotional states or stress; rather, it relates fundamentally to how students position themselves within collective learning activities⁷. Therefore, measuring prosocial intention serves as an appropriate lens for determining whether classroom engagement transcends isolated task completion to foster cooperative, encouraging behaviors beyond the immediate instructional environment⁸.

These conceptual developments underscore the necessity of an integrated evidence-gathering framework deployed within a single instructional window. A multi-source design mitigates the limitations of relying on any single metric, enabling cross-validation among different modes of participation and revealing how students interpret their post-class activities to foster independent learning⁹. While reflective assessments capture the learner' s internal cognitive state, anchoring these self-reports with objective behavioral traces significantly enhances the interpretability of the learning data. Currently, there is a notable absence of practical, classroom-ready protocols that seamlessly synthesize systematic behavioral observation, structured reflective reporting, and learning analytics for routine university implementation.

To address these methodological shortcomings, this paper introduces an evaluation protocol designed to measure university students' classroom engagement, reflective expression, and prosocial intention simultaneously. Developed specifically for undergraduate settings, the protocol offers a streamlined methodology that requires minimal administrative burden without sacrificing scientific reproducibility. By bridging the analytical gap between observable behaviors, articulated self-experience, and socially oriented learning outcomes, this framework answers the growing call for greater conceptual clarity in human-technology interaction research within educational environments¹⁰.

Protocol

All methods involving human participants were performed in compliance with the Declaration of Helsinki and the relevant institutional guidelines. For the implemented study, the protocol was approved by the Ethics Committee of Nanchang Institute of Technology before recruitment began (approval no.: NITE778291). Written informed consent was obtained from every participating student. Only de-identified aggregate classroom observation, reflection-score, learning analytics, and questionnaire data were used for analysis; identifiable classroom video, names, institutional IDs, and unredacted reflection text were excluded from public release.

1. Study design and participant selection

1. Implement the protocol across four consecutive teaching periods for each selected class section. Record de-identified course labels, instructional format, class size, instructor experience with the course, and whether each class includes whole-class instruction, group work, or individual practice. Target an initial enrollment of approximately 180 to 210 undergraduate students aged 18 years or older, or adapt the sample-size target to the course context.
2. Ensure all participants have an active institutional account for the course platform, will attend the observed classes, and have received the prerequisite course guidance normally provided to the class. When prerequisite knowledge varies substantially, provide a short bridging handout, pre-class resource list, or instructor-led review before the observation window and record the support provided as contextual metadata.
3. Obtain written informed consent from all students after providing a comprehensive written information sheet. State clearly in the consent form that the study links classroom behavioral observation, weekly reflective writing, and de-identified learning-platform records.
4. Enter only consenting students into the research dataset. Guarantee that refusal to participate does not affect attendance, grades, course access, or instructor communication.
   1. Explain the permitted and prohibited use of artificial-intelligence (AI) tools before the first reflection task.
      NOTE: AI tools may be used only when allowed by the course policy and only for general study support; reflective submissions and prosocial-intention responses must describe the student's own experience and must disclose any AI assistance.
   2. If the instructor permits AI-supported bridging of prerequisite knowledge, provide the same approved prompts or resources to all students and record this support as a protocol adaptation. Do not use AI-generated summaries as primary evidence of student reflection unless the validation study explicitly evaluates AI-assisted responses.
5. Exclude students prior to analysis if they decline consent, withdraw from the course during the protocol window, or lack matching data across observation, reflection, and analytics records. Employ a repeated-measures design to collect consistent data from the three distinct sources over a 4-week period².

2. Baseline data acquisition and identity management

1. Designate a single data administrator to assign an eight-character alphanumeric study ID to each consenting participant through a pseudo-random process.
2. Create an identity-study ID linkage file containing student names, institutional IDs, class sections, and study numbers. Store this file securely and separately from the analytic files, ensuring it remains completely inaccessible to observers, reflection coders, and researchers.
3. Destroy or archive this linkage file after final data verification according to the approved institutional data-retention policy, while maintaining compliance with current higher education learning-analytics practices regarding restricted linkage and de-identification¹.
4. Gather baseline demographic data on participant age, gender, academic year, and major during the initial week prior to the first observed class.
5. Use the six-item prosocial-intention questionnaire in Supplementary File 1 as the structured baseline instrument before the first observed class. Administer it electronically through the LMS or on paper using the assigned Study ID only. Students rate each item on a 5-point scale from 1 = strongly disagree to 5 = strongly agree; calculate the baseline score as the mean of completed items when at least five of six items are answered.
6. Export all subsequently collected reflective texts without names, email addresses, or other personal identifiers to ensure complete anonymity before the qualitative coding process begins.

3. Classroom observation preparation

1. Schedule observations for each class section once per week for 4 consecutive weeks. For a standard 90-min class, define a 70-min observed window by excluding the first 10 min and final 10 min from coding⁵.
2. Begin coding at 10^th min and stop at 80^th min of a standard 90-min class. These buffer periods are not part of the 70-min observation window; therefore, a standard class yields 70 min of coded observation rather than a 30-min session.
3. Exclude any interrupted instructional block from the final evaluation if the class experiences a disruption lasting more than 5 min due to alarms, room changes, or platform outages.
4. Treat classroom engagement as an observable composition of multiple behaviors by recording several behavioral indicators simultaneously at each measurement point¹¹. Indicators include on-task attention, verbal participation, peer collaboration, task-focused note-taking or device use, off-task distraction, and help-seeking behavior.
5. Arrange the classroom seating by group before the first observation and strictly maintain this seating arrangement throughout the entire 4-week period.
6. Position two trained observers in fixed locations with full-room visibility to code independently. If institutional filming is permitted and participants consent, install two stationary cameras in the rear-center and rear-side areas and synchronize them before the session.
7. Use camera recordings only for observer training, reliability verification, and adjudication of visibility problems; do not use video recordings as the primary coding source and do not perform facial-emotion, voice-emotion, biometric, or identity-recognition analysis.

4. Interval-based behavioral coding

1. Divide the 70-min observation window into consecutive intervals consisting of a 20-s visual scan followed by a 10-s recording period. If a large lecture, laboratory, or online class requires a different interval length, report the adapted interval structure and justify it before data collection begins.
   NOTE: This 20-s/10-s structure balances ecological classroom observation with coder workload: the scan period is long enough to detect sustained attention, collaboration, and off-task behavior, while the recording period reduces memory burden and supports consistent binary decisions.
2. Scan a clearly defined group of up to ten students per interval in a continuous loop based on the seating arrangement.
3. Utilize specific scanning criteria to prevent coder inconsistency caused by loose operational definitions⁴. Code six binary student-level indicators during each scan. These include on-task attention, verbal participation, peer collaboration, task-focused note-taking or device use, off-task distraction, and help-seeking behavior.
4. Mark an indicator as "1" if the designated behavior occurs at any point during the 20-s scan. Otherwise, mark it as "0". Code the interval as missing if a student is not visible for at least 10 s of the scan.
5. Record on-task attention when a student visually orients toward the instructor, assigned tasks, presented information, or peers engaged in task discussion for at least 5 consecutive s. Code verbal participation for any audible and task-related contribution.
6. Code peer collaboration when students engage in visible task-based exchanges, joint problem-solving, or peer explanations. Record task-focused note-taking or device use for active writing, typing, or digital interaction with course materials.
7. Mark off-task distraction if a student engages in unrelated conversations, non-academic device use, or sustained disengagement for at least 5 consecutive s. Record help-seeking when a student explicitly requests clarification or assistance from the instructor or peers.
8. Calculate the weekly behavioral score for each student as the total count of positive judgments divided by all valid ratings. Derive 4-week means for all six indicators subsequently.
9. Compute a combined Observed Classroom Engagement Index (OCEI) for student i in week w using the prespecified mathematical model: OCEI_iw = [On-task attention_iw + Verbal participation_iw + Peer collaboration_iw + Task-focused note-taking/device use_iw + (1 - Off-task distraction_iw)]/5.
10. Maintain help-seeking as an independent analytical variable because it can represent either constructive clarification or difficulty with instructional content, depending on context¹². Refer to Figure 1 for the study workflow and Table 1 for the observation indices and coding rules.

5. Observer training and reliability assessment

1. Conduct observer training using a 60-min manual-training session and execute three calibration rounds prior to the formal program.
2. Perform the first calibration round using guided coding of a non-study classroom video. Complete the subsequent two rounds as independent codings of different non-study videos, followed by direct statistical comparisons with the senior coder.
3. Commence formal data collection only when Cohen's Kappa reaches or exceeds 0.75, and total inter-rater agreement surpasses 85%.
4. Implement double-blind drift verification on 10% of the full sessions and one randomized 10 min segment from the remaining sessions throughout the 4-week protocol.
5. Recalibrate the coding team within 72 h and reverify the affected sessions before data locking if the Kappa value falls below 0.70 for any indicator. Continuously provide feedback and organize calibration exercises to maintain evaluation stability over time¹³.

6. Reflective expression collection and evaluation

1. Instruct students to submit one structured reflective text (100-350 words) for every observed class, generating 4 entries per participant across the 4-week protocol.
2. Release the reflection prompts through the LMS or a paper form within 2 h after the observed class, and close the submission window 72 h later. The form should display the Study ID field and the following three prompts in the same order for every student:
   1. Identify where they felt most and least involved during the class.
   2. Explain the contextual causes, including task difficulty, prerequisite knowledge, group interaction, instructor explanation, or platform use.
   3. Describe how their participation influenced the classroom atmosphere and what future action they intend to take.
      NOTE: Retain entries under 100 words, but flag them in the database as low-information. Also flag entries that omit the involvement, contextual-cause, or future-action/social-impact prompt, because these omissions affect reflection-quality scoring and feasibility reporting.
3. Assign two blinded coders to evaluate the submissions across descriptive specificity, analytic depth, self-regulatory orientation, and social-prosocial meaning.
   1. Rate each dimension using a 4-point scale where 0 indicates absence, 1 indicates weak or vague evidence, 2 indicates clear but limited evidence, and 3 indicates strong and elaborated evidence. For example, analytic depth is scored as 0 when no reason is provided, 1 when the student states only that the activity was difficult, 2 when the student links difficulty to a specific concept or classroom condition, and 3 when the student explains the interaction among prior knowledge, peer discussion, instructor feedback, and a planned learning strategy. Table 2 and Supplementary File 1 provide score-level examples for all four reflective dimensions.
4. Initiate formal assessment only after achieving an intra-class correlation coefficient of 0.80 or higher for each dimension on the pilot texts¹⁴.
5. Randomly double-score 15% of the study reflections during the protocol to control for rater drift. Evaluate the reflective writings based on these multidimensional qualitative metrics rather than relying solely on text length or word frequency.
6. Develop or adapt the reflective-expression rubric through expert review by at least two education researchers and one course instructor. Pilot the rubric on 20 to 30 non-study reflections, revise ambiguous descriptors, and document the final rubric version before formal coding begins.
7. Administer the prosocial-intention questionnaire shown in Supplementary File 1. Ask students to rate their willingness to help classmates who encounter difficulties, share learning resources, contribute constructively to discussions, encourage peers, act fairly during group tasks, and improve the classroom learning climate. Use a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Calculate the mean score when at least five of six items are completed, and report Cronbach's alpha for the baseline and post-protocol administrations.

7. Extraction of learning analytics

1. Assign an external administrator to extract the learning analytics data after Week 4 to prevent any contamination of the observation and reflection processes.
2. Set the extraction window from 00:00 on the day of the first observed class to 23:59 on the closing day of the final reflection window.
3. Extract only the prespecified variables necessary for the protocol. Capture the number of active days on the platform, distinct resource views, the proportion of assigned resources opened, the quantity of discussion replies posted, the mean video completion rate, and assignment punctuality.
4. Clean the raw exports by removing accounts belonging to instructors and teaching assistants, duplicate records, testing events, and data points outside the designated timeframe.
5. Calculate an active day as any calendar date containing at least one authenticated course access followed by a meaningful interaction, such as opening files, watching videos, or posting replies.
6. Count unique resource openings while ignoring duplicated refresh events occurring within a 60-s span. Determine the video completion rate by truncating the viewed fraction relative to the scheduled time at 01:00, then averaging across all assigned media items.
7. Calculate assignment punctuality based on the proportion of tasks submitted before the scheduled deadlines. Compute the late-night access proportion as the ratio of meaningful events occurring between 00:00 and 06:00 to the total meaningful activities within the observation period. Rely on these time-bound and contribution-oriented metrics to provide richer information than simple log accumulation¹⁵.
8. Use a prespecified data-management workflow for the raw learning-management-system export. Save the original export as a read-only file, create a de-identified working copy, remove instructor and teaching-assistant accounts, remove duplicate or test events, and retain a processing log with row counts after each cleaning step.
9. Conduct data cleaning in a spreadsheet or later and statistical analysis in R version 4.4.0 or later, or equivalent software.
   1. In a spreadsheet, follow the steps described in 7.9.1.1–7.9.1.2.
      1. Import the de-identified CSV files using Data > Get Data > From Text/CSV, and use Data > Remove Duplicates to remove duplicate Study ID/time-stamp/event rows.
      2. Apply Filter to exclude instructor or teaching-assistant accounts and out-of-window records, and save a processing log with row counts after each step.
   2. In R, follow the steps described in 7.9.2.1–7.9.2.3.
      1. Import files with readr::read_csv(), reshape weekly records with dplyr::mutate(), dplyr::group_by(), dplyr::summarise(), tidyr::pivot_longer(), and tidyr::pivot_wider().
      2. Compute Cohen's kappa with irr::kappa2(), ICCs with psych::ICC(), and Cronbach's alpha with psych::alpha().
      3. Fit section-adjusted linear models with stats::lm(); use lme4::lmer() when a prespecified class-section random intercept is required, and use lme4::glmer(family = binomial) only for binary outcomes. Archive the exact script and package versions with the submission materials.
10. Use the following reproducible analysis sequence.
    1. Calculate student-week observation proportions, compute OCEI using the formula in step 4.9, and merge weekly reflection scores by Study ID and week.
    2. Aggregate learning analytics variables across the identical 4-week window, inspect missingness by data source, compute descriptive statistics and reliability estimates, and fit the prespecified association models.
    3. Do not add unplanned predictors after inspecting results unless they are labeled exploratory.
11. (Optional) For transparency, deposit a de-identified analysis script, codebook, blank coding forms, reflection prompts, reflection rubric, and prosocial questionnaire in the public repository or supplementary materials. Do not deposit identifiable raw classroom video, names, institutional IDs, or unredacted reflection text.

8. Post-protocol measurement and dataset integration

1. Administer the prosocial intention instrument a second time immediately after the conclusion of the final reflection window.
2. Ask students to rate their willingness to help classmates, share resources, contribute constructively to discussions, encourage peers, act fairly, and improve the learning climate on a scale ranging from 1 (strongly disagree) to 5 (strongly agree). Compute each individual's total post-protocol score by averaging responses, provided they answered at least five of the six items.
3. Merge the observation files, reflection scores, and learning analytics records using the anonymized Study ID. Construct two final datasets from this merge to create a participant-level master file and a week-level long format file.
4. Do not impute missing observation intervals caused by student non-visibility, nor missing weekly reflections caused by non-submission. Flag partially completed reflections, incomplete learning analytics variables, low-information reflections, and unmatched Study IDs before analysis.
5. Use complete-case analysis for prespecified association models, report the number of excluded cases for each model, and conduct a sensitivity check comparing included and excluded students on available baseline variables.
6. Retain prosocial intention as a primary outcome measure to reflect the supportive and collaborative attitudes that underpin structural changes in university students' participation.

9. Troubleshooting and protocol adaptations

1. If two observers disagree by more than one binary category pattern across the same student cluster or if Cohen's kappa falls below 0.70 for any indicator, pause formal scoring for the affected session, review the operational definitions, complete a 10-min recalibration segment, and re-code the affected interval before data locking.
2. If platform exports are incomplete, document the missing field, request a second export from the external administrator, and retain the student in observation/reflection summaries while excluding the unavailable analytics variable from models requiring that field.
3. If fewer than 80% of expected reflections are submitted within 48 h after task opening, send a neutral reminder to the whole class, extend technical support for access problems, and report the final on-time, late, and missing counts separately.
4. If more than 10% of students in a cluster are not visible for at least half of the scan window, code the affected intervals as missing and rely on live observer records rather than attempting to infer behavior from obstructed video.
5. If a student withdraws, declines further participation, or lacks linkable data across sources, remove that student from linked association models, but retain aggregate protocol-feasibility counts when permitted by the ethics approval.
6. If multiple students signal that a section is difficult to understand, the instructor should provide the normal pedagogical response, such as a short re-explanation, worked example, peer discussion, or follow-up resource. Record this as a contextual event because it can influence help-seeking, reflection depth, and platform activity.
7. Adapt the student-cluster size, interval length, reflection prompts, and analytics variables to the instructional format, but preserve the same logic of predefined behavioral indicators, de-identified records, independent coding, and transparent missing-data rules.
8. Treat low reliability, high missingness, limited reflection depth, weak prosocial change, and weak agreement among data streams as informative implementation outcomes rather than protocol failures. Report these outcomes and explain whether they reflect classroom context, instrument limitations, or data-capture problems.

Results

Participant inclusion, data completeness, and score reliability
An initial cohort of 204 students across all class sections was invited to participate, yielding 198 consenting individuals. Fourteen cases were subsequently excluded due to course withdrawal during the protocol window or incomplete data linkage across the observation, reflection, and platform records. Consequently, the final analytic sample comprised 184 students. Participant characteristics, weekly completion rates, and overall data co...

Discussion

This study proposes a practical protocol that integrates behavioral observation, reflective writing, and learning analytics to comprehensively investigate university students' participation, self-expression, and prosocial awareness within a single instructional window. By capturing different dimensions of engagement, this tri-modal approach addresses recent critiques in higher education research, which argue that time-in-use metrics and sensor-based monitoring are insufficient when evaluated in isolation

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Classroom observation coding sheet	In-house	Six binary behavioral indicators	Interval-based coding of on-task attention, verbal participation, peer collaboration, task-focused note-taking/device use, off-task distraction, and help-seeking.
GraphPad Prism	Graphpad	Version 10.2.3	Generation of workflow diagrams, weekly trend plots, distribution plots, and association figures.
Institutional learning management system / course platform	Host university	Existing institutional LMS account system	Collection of de-identified learning analytics.
Microsoft PowerPoint	Microsoft	Microsoft 365 v2402	Generation of workflow diagrams, weekly trend plots, distribution plots, and association figures.
R	R Foundation		ggplot2 3.5.1
Spreadsheet software	Microsoft Corporation	Microsoft Excel 365 v2402	Data entry, cleaning, de-identification checks, and merging of observation/reflection/analytics datasets.
SPSS Statistics	IBM Corp	Version 26.0	Statistical analysis software; Descriptive statistics, repeated-measures comparisons, Cohen's kappa, ICC, Cronbach's alpha, correlations, and regression models.