Evaluating University Student Classroom Engagement, Reflective Expression, and Prosocial Intention Using Behavioral Observation and Learning Analytics

Summary

This protocol integrates classroom observation, reflective writing, prosocial-intention measurement, and learning analytics to evaluate university students' behavioral engagement, reflective expression, and socially oriented learning outcomes over a 4-week instructional window. It specifies ethics approval, instrument contents, software workflow, missing-data handling, troubleshooting procedures, and interpretation of representative outcomes, including observed implementation constraints.

Abstract

To address the need for multidimensional assessments of student involvement beyond traditional metrics, this protocol describes a structured approach for evaluating classroom engagement, reflective expression, learning analytics activity, and prosocial intention in undergraduate courses. The protocol combines interval-based behavioral observation, weekly structured reflective writing, de-identified learning-management-system records, and a pre/post prosocial-intention questionnaire. The revised procedures provide a reproducible behavioral coding model, reflective-expression rubric with score examples, software and data-management steps, and explicit troubleshooting guidance for observer disagreement, missing platform records, low reflection submission rates, poor camera visibility, participant attrition, and inconsistent prerequisite knowledge. This protocol should be used when researchers need a low-intrusion, reproducible way to combine live classroom behavior, student self-interpretation, and routine learning management system (LMS) activity during undergraduate instruction. It is less suitable for fully asynchronous courses without observable interaction, settings that prohibit independent observation or record linkage, or studies requiring biometric or emotion-recognition inference. The representative results are reported from the implemented empirical application after verified ethics approval, de-identification of course records, and repository preparation.

Introduction

Evaluating student participation in higher education requires a comprehensive, multi-source approach. Although student engagement is widely conceptualized as a multidimensional construct encompassing observable behavior, cognitive effort, and environmental interaction, current analytical approaches predominantly rely on superficial log data, such as login frequencies and system-recorded actions¹. This overreliance presents a significant challenge for practical classroom-based research. The most critical forms of instructional participation—direct peer interactions, cognitive reflection, and social contributions—often occur entirely beyond the limited scope of platform visibility².

The complexity of the modern university classroom has intensified with the adoption of hybrid pedagogical models, blending digital platforms with traditional face-to-face instruction. While systematic reviews of learning analytics confirm that digital traces offer valuable insights into time allocation, access patterns, and long-term retention rates, these metrics alone cannot determine whether a learner's attention is genuinely efficient or meaningful³. Consequently, recent scholarship advocates for grounding learning analytics in stronger theoretical frameworks by aligning digital traces with richer, context-dependent measures of participation, such as classroom behavior and reflective activities⁴.

Reflective writing provides a vital qualitative dimension to this assessment. Structured post-class reflections and journals capture students' interpretations of learning activities, clarify instructional challenges, and bridge theoretical content with subsequent practical actions. Recent empirical evidence conceptualizes reflective expression as an integrated, multi-component construct comprising descriptive recall, analytical inference, self-referential representation, and meaning generalization⁵. Furthermore, research indicates that providing students with repeated reflective guidance gradually strengthens their capacity for introspection; as the frequency of structured reflection increases during coursework, students' independent exploratory skills concurrently improve⁶.

A second major gap in current engagement research is the frequent marginalization of classroom socialization. While existing studies heavily emphasize individual behavior or cognition, university classrooms function as inherently social learning environments where students influence one another through collaboration and responsive interactions. Emerging evidence demonstrates that classroom climate and participation structures correlate strongly with prosocial behavior and a contribution-oriented mindset—outcomes that remain under-investigated compared to traditional metrics of academic performance or student satisfaction. Prosocial behavior in academic settings is not merely a byproduct of emotional states or stress; rather, it relates fundamentally to how students position themselves within collective learning activities⁷. Therefore, measuring prosocial intention serves as an appropriate lens for determining whether classroom engagement transcends isolated task completion to foster cooperative, encouraging behaviors beyond the immediate instructional environment⁸.

These conceptual developments underscore the necessity of an integrated evidence-gathering framework deployed within a single instructional window. A multi-source design mitigates the limitations of relying on any single metric, enabling cross-validation among different modes of participation and revealing how students interpret their post-class activities to foster independent learning⁹. While reflective assessments capture the learner' s internal cognitive state, anchoring these self-reports with objective behavioral traces significantly enhances the interpretability of the learning data. Currently, there is a notable absence of practical, classroom-ready protocols that seamlessly synthesize systematic behavioral observation, structured reflective reporting, and learning analytics for routine university implementation.

To address these methodological shortcomings, this paper introduces an evaluation protocol designed to measure university students' classroom engagement, reflective expression, and prosocial intention simultaneously. Developed specifically for undergraduate settings, the protocol offers a streamlined methodology that requires minimal administrative burden without sacrificing scientific reproducibility. By bridging the analytical gap between observable behaviors, articulated self-experience, and socially oriented learning outcomes, this framework answers the growing call for greater conceptual clarity in human-technology interaction research within educational environments¹⁰.

Protocol

All methods involving human participants were performed in compliance with the Declaration of Helsinki and the relevant institutional guidelines. For the implemented study, the protocol was approved by the Ethics Committee of Nanchang Institute of Technology before recruitment began (approval no.: NITE778291). Written informed consent was obtained from every participating student. Only de-identified aggregate classroom observation, reflection-score, learning analytics, and questionnaire data were used for analysis; identifiable classroom video, names, institutional IDs, and unredacted reflection text were excluded from public release.

1. Study design and participant selection

1. Implement the protocol across four consecutive teaching periods for each selected class section. Record de-identified course labels, instructional format, class size, instructor experience with the course, and whether each class includes whole-class instruction, group work, or individual practice. Target an initial enrollment of approximately 180 to 210 undergraduate students aged 18 years or older, or adapt the sample-size target to the course context.
2. Ensure all participants have an active institutional account for the course platform, will attend the observed classes, and have received the prerequisite course guidance normally provided to the class. When prerequisite knowledge varies substantially, provide a short bridging handout, pre-class resource list, or instructor-led review before the observation window and record the support provided as contextual metadata.
3. Obtain written informed consent from all students after providing a comprehensive written information sheet. State clearly in the consent form that the study links classroom behavioral observation, weekly reflective writing, and de-identified learning-platform records.
4. Enter only consenting students into the research dataset. Guarantee that refusal to participate does not affect attendance, grades, course access, or instructor communication.
   1. Explain the permitted and prohibited use of artificial-intelligence (AI) tools before the first reflection task.
      NOTE: AI tools may be used only when allowed by the course policy and only for general study support; reflective submissions and prosocial-intention responses must describe the student's own experience and must disclose any AI assistance.
   2. If the instructor permits AI-supported bridging of prerequisite knowledge, provide the same approved prompts or resources to all students and record this support as a protocol adaptation. Do not use AI-generated summaries as primary evidence of student reflection unless the validation study explicitly evaluates AI-assisted responses.
5. Exclude students prior to analysis if they decline consent, withdraw from the course during the protocol window, or lack matching data across observation, reflection, and analytics records. Employ a repeated-measures design to collect consistent data from the three distinct sources over a 4-week period².

2. Baseline data acquisition and identity management

1. Designate a single data administrator to assign an eight-character alphanumeric study ID to each consenting participant through a pseudo-random process.
2. Create an identity-study ID linkage file containing student names, institutional IDs, class sections, and study numbers. Store this file securely and separately from the analytic files, ensuring it remains completely inaccessible to observers, reflection coders, and researchers.
3. Destroy or archive this linkage file after final data verification according to the approved institutional data-retention policy, while maintaining compliance with current higher education learning-analytics practices regarding restricted linkage and de-identification¹.
4. Gather baseline demographic data on participant age, gender, academic year, and major during the initial week prior to the first observed class.
5. Use the six-item prosocial-intention questionnaire in Supplementary File 1 as the structured baseline instrument before the first observed class. Administer it electronically through the LMS or on paper using the assigned Study ID only. Students rate each item on a 5-point scale from 1 = strongly disagree to 5 = strongly agree; calculate the baseline score as the mean of completed items when at least five of six items are answered.
6. Export all subsequently collected reflective texts without names, email addresses, or other personal identifiers to ensure complete anonymity before the qualitative coding process begins.

3. Classroom observation preparation

1. Schedule observations for each class section once per week for 4 consecutive weeks. For a standard 90-min class, define a 70-min observed window by excluding the first 10 min and final 10 min from coding⁵.
2. Begin coding at 10^th min and stop at 80^th min of a standard 90-min class. These buffer periods are not part of the 70-min observation window; therefore, a standard class yields 70 min of coded observation rather than a 30-min session.
3. Exclude any interrupted instructional block from the final evaluation if the class experiences a disruption lasting more than 5 min due to alarms, room changes, or platform outages.
4. Treat classroom engagement as an observable composition of multiple behaviors by recording several behavioral indicators simultaneously at each measurement point¹¹. Indicators include on-task attention, verbal participation, peer collaboration, task-focused note-taking or device use, off-task distraction, and help-seeking behavior.
5. Arrange the classroom seating by group before the first observation and strictly maintain this seating arrangement throughout the entire 4-week period.
6. Position two trained observers in fixed locations with full-room visibility to code independently. If institutional filming is permitted and participants consent, install two stationary cameras in the rear-center and rear-side areas and synchronize them before the session.
7. Use camera recordings only for observer training, reliability verification, and adjudication of visibility problems; do not use video recordings as the primary coding source and do not perform facial-emotion, voice-emotion, biometric, or identity-recognition analysis.

4. Interval-based behavioral coding

1. Divide the 70-min observation window into consecutive intervals consisting of a 20-s visual scan followed by a 10-s recording period. If a large lecture, laboratory, or online class requires a different interval length, report the adapted interval structure and justify it before data collection begins.
   NOTE: This 20-s/10-s structure balances ecological classroom observation with coder workload: the scan period is long enough to detect sustained attention, collaboration, and off-task behavior, while the recording period reduces memory burden and supports consistent binary decisions.
2. Scan a clearly defined group of up to ten students per interval in a continuous loop based on the seating arrangement.
3. Utilize specific scanning criteria to prevent coder inconsistency caused by loose operational definitions⁴. Code six binary student-level indicators during each scan. These include on-task attention, verbal participation, peer collaboration, task-focused note-taking or device use, off-task distraction, and help-seeking behavior.
4. Mark an indicator as "1" if the designated behavior occurs at any point during the 20-s scan. Otherwise, mark it as "0". Code the interval as missing if a student is not visible for at least 10 s of the scan.
5. Record on-task attention when a student visually orients toward the instructor, assigned tasks, presented information, or peers engaged in task discussion for at least 5 consecutive s. Code verbal participation for any audible and task-related contribution.
6. Code peer collaboration when students engage in visible task-based exchanges, joint problem-solving, or peer explanations. Record task-focused note-taking or device use for active writing, typing, or digital interaction with course materials.
7. Mark off-task distraction if a student engages in unrelated conversations, non-academic device use, or sustained disengagement for at least 5 consecutive s. Record help-seeking when a student explicitly requests clarification or assistance from the instructor or peers.
8. Calculate the weekly behavioral score for each student as the total count of positive judgments divided by all valid ratings. Derive 4-week means for all six indicators subsequently.
9. Compute a combined Observed Classroom Engagement Index (OCEI) for student i in week w using the prespecified mathematical model: OCEI_iw = [On-task attention_iw + Verbal participation_iw + Peer collaboration_iw + Task-focused note-taking/device use_iw + (1 - Off-task distraction_iw)]/5.
10. Maintain help-seeking as an independent analytical variable because it can represent either constructive clarification or difficulty with instructional content, depending on context¹². Refer to Figure 1 for the study workflow and Table 1 for the observation indices and coding rules.

5. Observer training and reliability assessment

1. Conduct observer training using a 60-min manual-training session and execute three calibration rounds prior to the formal program.
2. Perform the first calibration round using guided coding of a non-study classroom video. Complete the subsequent two rounds as independent codings of different non-study videos, followed by direct statistical comparisons with the senior coder.
3. Commence formal data collection only when Cohen's Kappa reaches or exceeds 0.75, and total inter-rater agreement surpasses 85%.
4. Implement double-blind drift verification on 10% of the full sessions and one randomized 10 min segment from the remaining sessions throughout the 4-week protocol.
5. Recalibrate the coding team within 72 h and reverify the affected sessions before data locking if the Kappa value falls below 0.70 for any indicator. Continuously provide feedback and organize calibration exercises to maintain evaluation stability over time¹³.

6. Reflective expression collection and evaluation

1. Instruct students to submit one structured reflective text (100-350 words) for every observed class, generating 4 entries per participant across the 4-week protocol.
2. Release the reflection prompts through the LMS or a paper form within 2 h after the observed class, and close the submission window 72 h later. The form should display the Study ID field and the following three prompts in the same order for every student:
   1. Identify where they felt most and least involved during the class.
   2. Explain the contextual causes, including task difficulty, prerequisite knowledge, group interaction, instructor explanation, or platform use.
   3. Describe how their participation influenced the classroom atmosphere and what future action they intend to take.
      NOTE: Retain entries under 100 words, but flag them in the database as low-information. Also flag entries that omit the involvement, contextual-cause, or future-action/social-impact prompt, because these omissions affect reflection-quality scoring and feasibility reporting.
3. Assign two blinded coders to evaluate the submissions across descriptive specificity, analytic depth, self-regulatory orientation, and social-prosocial meaning.
   1. Rate each dimension using a 4-point scale where 0 indicates absence, 1 indicates weak or vague evidence, 2 indicates clear but limited evidence, and 3 indicates strong and elaborated evidence. For example, analytic depth is scored as 0 when no reason is provided, 1 when the student states only that the activity was difficult, 2 when the student links difficulty to a specific concept or classroom condition, and 3 when the student explains the interaction among prior knowledge, peer discussion, instructor feedback, and a planned learning strategy. Table 2 and Supplementary File 1 provide score-level examples for all four reflective dimensions.
4. Initiate formal assessment only after achieving an intra-class correlation coefficient of 0.80 or higher for each dimension on the pilot texts¹⁴.
5. Randomly double-score 15% of the study reflections during the protocol to control for rater drift. Evaluate the reflective writings based on these multidimensional qualitative metrics rather than relying solely on text length or word frequency.
6. Develop or adapt the reflective-expression rubric through expert review by at least two education researchers and one course instructor. Pilot the rubric on 20 to 30 non-study reflections, revise ambiguous descriptors, and document the final rubric version before formal coding begins.
7. Administer the prosocial-intention questionnaire shown in Supplementary File 1. Ask students to rate their willingness to help classmates who encounter difficulties, share learning resources, contribute constructively to discussions, encourage peers, act fairly during group tasks, and improve the classroom learning climate. Use a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Calculate the mean score when at least five of six items are completed, and report Cronbach's alpha for the baseline and post-protocol administrations.

7. Extraction of learning analytics

1. Assign an external administrator to extract the learning analytics data after Week 4 to prevent any contamination of the observation and reflection processes.
2. Set the extraction window from 00:00 on the day of the first observed class to 23:59 on the closing day of the final reflection window.
3. Extract only the prespecified variables necessary for the protocol. Capture the number of active days on the platform, distinct resource views, the proportion of assigned resources opened, the quantity of discussion replies posted, the mean video completion rate, and assignment punctuality.
4. Clean the raw exports by removing accounts belonging to instructors and teaching assistants, duplicate records, testing events, and data points outside the designated timeframe.
5. Calculate an active day as any calendar date containing at least one authenticated course access followed by a meaningful interaction, such as opening files, watching videos, or posting replies.
6. Count unique resource openings while ignoring duplicated refresh events occurring within a 60-s span. Determine the video completion rate by truncating the viewed fraction relative to the scheduled time at 01:00, then averaging across all assigned media items.
7. Calculate assignment punctuality based on the proportion of tasks submitted before the scheduled deadlines. Compute the late-night access proportion as the ratio of meaningful events occurring between 00:00 and 06:00 to the total meaningful activities within the observation period. Rely on these time-bound and contribution-oriented metrics to provide richer information than simple log accumulation¹⁵.
8. Use a prespecified data-management workflow for the raw learning-management-system export. Save the original export as a read-only file, create a de-identified working copy, remove instructor and teaching-assistant accounts, remove duplicate or test events, and retain a processing log with row counts after each cleaning step.
9. Conduct data cleaning in a spreadsheet or later and statistical analysis in R version 4.4.0 or later, or equivalent software.
   1. In a spreadsheet, follow the steps described in 7.9.1.1–7.9.1.2.
      1. Import the de-identified CSV files using Data > Get Data > From Text/CSV, and use Data > Remove Duplicates to remove duplicate Study ID/time-stamp/event rows.
      2. Apply Filter to exclude instructor or teaching-assistant accounts and out-of-window records, and save a processing log with row counts after each step.
   2. In R, follow the steps described in 7.9.2.1–7.9.2.3.
      1. Import files with readr::read_csv(), reshape weekly records with dplyr::mutate(), dplyr::group_by(), dplyr::summarise(), tidyr::pivot_longer(), and tidyr::pivot_wider().
      2. Compute Cohen's kappa with irr::kappa2(), ICCs with psych::ICC(), and Cronbach's alpha with psych::alpha().
      3. Fit section-adjusted linear models with stats::lm(); use lme4::lmer() when a prespecified class-section random intercept is required, and use lme4::glmer(family = binomial) only for binary outcomes. Archive the exact script and package versions with the submission materials.
10. Use the following reproducible analysis sequence.
    1. Calculate student-week observation proportions, compute OCEI using the formula in step 4.9, and merge weekly reflection scores by Study ID and week.
    2. Aggregate learning analytics variables across the identical 4-week window, inspect missingness by data source, compute descriptive statistics and reliability estimates, and fit the prespecified association models.
    3. Do not add unplanned predictors after inspecting results unless they are labeled exploratory.
11. (Optional) For transparency, deposit a de-identified analysis script, codebook, blank coding forms, reflection prompts, reflection rubric, and prosocial questionnaire in the public repository or supplementary materials. Do not deposit identifiable raw classroom video, names, institutional IDs, or unredacted reflection text.

8. Post-protocol measurement and dataset integration

1. Administer the prosocial intention instrument a second time immediately after the conclusion of the final reflection window.
2. Ask students to rate their willingness to help classmates, share resources, contribute constructively to discussions, encourage peers, act fairly, and improve the learning climate on a scale ranging from 1 (strongly disagree) to 5 (strongly agree). Compute each individual's total post-protocol score by averaging responses, provided they answered at least five of the six items.
3. Merge the observation files, reflection scores, and learning analytics records using the anonymized Study ID. Construct two final datasets from this merge to create a participant-level master file and a week-level long format file.
4. Do not impute missing observation intervals caused by student non-visibility, nor missing weekly reflections caused by non-submission. Flag partially completed reflections, incomplete learning analytics variables, low-information reflections, and unmatched Study IDs before analysis.
5. Use complete-case analysis for prespecified association models, report the number of excluded cases for each model, and conduct a sensitivity check comparing included and excluded students on available baseline variables.
6. Retain prosocial intention as a primary outcome measure to reflect the supportive and collaborative attitudes that underpin structural changes in university students' participation.

9. Troubleshooting and protocol adaptations

1. If two observers disagree by more than one binary category pattern across the same student cluster or if Cohen's kappa falls below 0.70 for any indicator, pause formal scoring for the affected session, review the operational definitions, complete a 10-min recalibration segment, and re-code the affected interval before data locking.
2. If platform exports are incomplete, document the missing field, request a second export from the external administrator, and retain the student in observation/reflection summaries while excluding the unavailable analytics variable from models requiring that field.
3. If fewer than 80% of expected reflections are submitted within 48 h after task opening, send a neutral reminder to the whole class, extend technical support for access problems, and report the final on-time, late, and missing counts separately.
4. If more than 10% of students in a cluster are not visible for at least half of the scan window, code the affected intervals as missing and rely on live observer records rather than attempting to infer behavior from obstructed video.
5. If a student withdraws, declines further participation, or lacks linkable data across sources, remove that student from linked association models, but retain aggregate protocol-feasibility counts when permitted by the ethics approval.
6. If multiple students signal that a section is difficult to understand, the instructor should provide the normal pedagogical response, such as a short re-explanation, worked example, peer discussion, or follow-up resource. Record this as a contextual event because it can influence help-seeking, reflection depth, and platform activity.
7. Adapt the student-cluster size, interval length, reflection prompts, and analytics variables to the instructional format, but preserve the same logic of predefined behavioral indicators, de-identified records, independent coding, and transparent missing-data rules.
8. Treat low reliability, high missingness, limited reflection depth, weak prosocial change, and weak agreement among data streams as informative implementation outcomes rather than protocol failures. Report these outcomes and explain whether they reflect classroom context, instrument limitations, or data-capture problems.

Results

Participant inclusion, data completeness, and score reliability
An initial cohort of 204 students across all class sections was invited to participate, yielding 198 consenting individuals. Fourteen cases were subsequently excluded due to course withdrawal during the protocol window or incomplete data linkage across the observation, reflection, and platform records. Consequently, the final analytic sample comprised 184 students. Participant characteristics, weekly completion rates, and overall data completeness are detailed in Table 3.

Over the 4-week program, 24 class sessions were observed, generating a synchronized learning-analytics record for each participant. Of the 736 expected reflective reports, 701 were submitted within the 72-h window, 18 were submitted late, and 17 were missing. Ultimately, 171 students successfully completed all weekly reflections on time. Learning analytics were completely extracted for the final 184 participants after filtering out instructor accounts, duplicate entries, and out-of-window activities.

Observational coding remained highly stable throughout the protocol period. Inter-rater reliability for the six behavioral indices of classroom observation demonstrated Cohen's kappa values ranging from 0.76 to 0.84, with overall agreement during drift tests exceeding 86%. For the reflective expression scores, intra-class correlation coefficients (ICCs) across the four dimensions varied between 0.82 and 0.89. Table 4 presents the descriptive statistics and reliability indices for the primary study variables.

Classroom behavioral observation
Observed classroom behaviors exhibited gradual improvement over the 4-week protocol. As illustrated in Figure 2, on-task attention, verbal participation, and peer collaboration were significantly higher in Week 4 compared to the initial weeks. Specifically, verbal participation increased from 0.18 ± 0.09 in the first week to 0.24 ± 0.10 by the fourth week (p < 0.05). Peer collaboration similarly rose from 0.29 ± 0.11 to 0.35 ± 0.12 (p = 0.006). On-task attention demonstrated a highly significant improvement, shifting from 0.71 ± 0.12 to 0.76 ± 0.11 (p < 0.001).

Conversely, task-focused note-taking or device use showed only marginal weekly increases that did not reach statistical significance (p = 0.08). Help-seeking behavior remained consistently low throughout the observation period, displaying no significant time effect (p = 0.21). Off-task distraction significantly decreased from 0.14 ± 0.08 in Week 1 to 0.11 ± 0.07 in Week 4 (p = 0.03). Overall, the composite Observed Classroom Engagement Index rose significantly from 0.52 ± 0.10 during the first week to 0.58 ± 0.11 by the final week (p < 0.001).

Reflective expressions over the 4-week protocol
Reflective expression evolved over the study period, though the magnitude of change varied across its sub-dimensions. As shown in Figure 3, the total reflection score increased from Week 1 to 7.2 ± 1.9 by Week 4 (p < 0.001).

Analytic depth was notably greater at the end of Week 4 compared to the first two weeks (p < 0.001). Similarly, social-prosocial meaning grew from 1.08 ± 0.55 to 1.54 ± 0.60 (p < 0.001), and self-regulatory orientation increased from 1.42 ± 0.63 to 1.69 ± 0.65 (p < 0.05). In contrast, descriptive specificity exhibited a slight upward trend, from 2.14 ± 0.57 to 2.24 ± 0.55, which was not statistically significant (p = 0.09).

The submissions averaged between 223 and 231 words per entry throughout the protocol. Of the 736 expected reflective reports, 18 were submitted late, and 17 were missing; low-information entries and responses that omitted the contextual-cause or future-action/social-impact prompt were flagged in the database. These cases were retained as feasibility indicators and were excluded only from week-specific reflection-quality models when the relevant score could not be assigned.

Learning analytics patterns during the study window
Platform activity metrics exhibited considerable inter-individual variance over the 4-week window, as depicted in Figure 4.

Over the total period, students averaged 9.7 ± 2.4 active days on the platform. The mean number of distinct resource views was 27.3 ± 8.6, with students opening an average proportion of 0.81 ± 0.14 of the assigned resources. Furthermore, the mean discussion reply count was 4.8 ± 2.7. The average assignment punctuality and mean video completion rate were 0.86 ± 0.18 and 0.79 ± 0.17, respectively. The proportion of late-night platform access averaged 0.12 ± 0.09.

Notably, discussion replies and assignment punctuality demonstrated more pronounced positive correlations with other participation indicators, despite the wide distribution of punctuality scores.

Cross-association of observations, reflections, and learning data
The three data sources demonstrated meaningful, though non-uniform, associations. As shown in Figure 5, students with higher Observed Classroom Engagement Indices tended to achieve higher total reflection scores. In the section-adjusted model, observed classroom engagement positively predicted the reflection total score (β = 0.34, p < 0.001).

Among the learning analytics variables, discussion replies and assignment punctuality were significantly and positively correlated with reflection quality. As detailed in Table 5, discussion replies positively predicted the reflection total score (β = 0.19, p = 0.013), as did assignment punctuality (β = 0.17, p = 0.028). While raw resource views and the proportion of assigned resources opened showed positive associations initially, these correlations weakened significantly after adjusting for confounding variables. Late-night access proportion showed no significant association with the reflection total score (p = 0.18).

The data also revealed mild cross-correlations between observed classroom behavior and specific platform activities; for example, raw resource views correlated only marginally with the Observed Classroom Engagement Index. Furthermore, students who engaged more frequently in group discussions exhibited higher verbal communication, while those with higher behavioral regulation displayed less procrastination in task submissions.

Changes in prosocial intention
Prosocial intention demonstrated significant growth from baseline to post-test. The mean prosocial intention score increased from 3.61 ± 0.53 at baseline to 3.84 ± 0.50 following the 4-week protocol (p < 0.001). Figure 6 illustrates the paired distribution and adjusted associations.

In the final adjusted model, a higher Observed Classroom Engagement Index predicted a greater increase in post-protocol prosocial intention relative to the baseline score (β = 0.21, p < 0.05). Among the reflective dimensions, social-prosocial meaning exerted the most substantial independent effect on post-protocol prosocial intention (β = 0.29, p < 0.001). Although verbal participation exhibited a positive unadjusted correlation with post-test prosocial outcomes, this association lost statistical significance after controlling for baseline values (p = 0.09). Finally, the late-night access rate remained unrelated to post-protocol prosocial intention (p = 0.27).

Observed implementation constraints and null findings
The implemented study produced several suboptimal but informative outcomes rather than protocol-stopping failures. No observed session required exclusion because of an observer kappa below 0.70, no learning-analytics field remained unavailable after the final export, and no visibility problem exceeded the prespecified missingness threshold. The observed constraints were concentrated in reflection completion and in null or marginal behavioral findings: 18 reflections were submitted late, 17 expected reflections were missing, task-focused note-taking/device use showed only a marginal increase (p = 0.08), help-seeking showed no significant time effect (p = 0.21), and late-night access was not associated with reflection total score or post-protocol prosocial intention.

These findings were retained because they describe implementation feasibility and boundary conditions of the protocol. Late or missing reflections were reported in the participant-flow and data-completeness summaries; marginal or null associations were reported without selective omission; and the troubleshooting procedures in the Protocol section specify how hypothetical failures such as observer drift, obstructed visibility, or missing LMS fields should be handled in future implementations.

De-identified analysis files, the data dictionary, analysis scripts, blank coding forms, reflection prompts, reflection rubric, and the prosocial-intention questionnaire have been deposited in Zenodo: https://zenodo.org/records/20607517. Identifiable raw classroom video, names, institutional IDs, and unredacted reflection text were not publicly released.

Figure 1: Study workflow and multi-source data integration protocol. (A) Participant recruitment, consent collection, and study-ID assignment across two undergraduate courses and six class sections. (B) The 4-week data collection cycle: weekly classroom observation, reflective writing task, and synchronized learning-analytics capture window. (C) Three parallel data streams: classroom behavioral observation, reflective expression scoring, and de-identified learning-platform records. (D) Construction of the participant-level master dataset and the week-level long-format database, yielding the primary research outputs: observed classroom engagement index, reflection score, and prosocial intention. Please click here to view a larger version of this figure.

Figure 2: Weekly changes in observed classroom behavior during the 4-week protocol. (A) On-task attention. (B) Verbal participation. (C) Peer collaboration. (D) Task-focused note-taking or task-focused device use. (E) Off-task distraction. (F) Help-seeking. Values are weekly student-level proportion scores derived from interval-based classroom observations. Points indicate weekly means, and error bars indicate standard deviations. Please click here to view a larger version of this figure.

Figure 3: Weekly changes in reflective expression across the four reflective submissions. (A) Descriptive specificity. (B) Analytic depth. (C) Self-regulatory orientation. (D) Social-prosocial meaning. (E) Total reflection score. Reflection scores were assigned by two blinded coders using the prespecified rubric. Points indicate weekly means, and error bars indicate standard deviations. Please click here to view a larger version of this figure.

Figure 4: Distribution of learning-analytics indicators during the 4-week study window. (A) Active days on the learning platform. (B) Resource views. (C) Proportion of assigned resources opened. (D) Discussion replies posted. (E) Mean video completion rate. (F) Assignment punctuality. (G) Late-night access proportion. All values were calculated over the same 4-week window used for classroom observation and reflective writing. Violin plots show student-level distributions, and box plots indicate medians and interquartile ranges. Please click here to view a larger version of this figure.

Figure 5: Associations among observed classroom engagement, reflective expression, and selected learning-analytics indicators. (A) Association between the Observed Classroom Engagement Index and reflection total score. (B) Association between discussion replies posted and reflection total score. (C) Association between assignment punctuality and reflection total score. (D) Association between verbal participation and reflection total score. Trend lines represent fitted linear associations; shaded bands indicate 95% confidence intervals. Please click here to view a larger version of this figure.

Figure 6: Change in prosocial intention and its associations with classroom engagement and reflective meaning. (A) Paired distribution of baseline and post-protocol prosocial intention scores. Lines connect paired pre/post scores from the same student. (B) Association between the Observed Classroom Engagement Index and post-protocol prosocial intention after controlling for the baseline score. (C) Association between social-prosocial meaning and post-protocol prosocial intention. (D) Association between verbal participation and post-protocol prosocial intention. Shaded bands in panels B-D indicate 95% confidence intervals. Please click here to view a larger version of this figure.

Table 1: Student-level classroom observation indicators and coding rules. The table details the operational definitions and coding criteria for on-task attention, verbal participation, peer collaboration, task-focused note-taking or device use, off-task distraction, and help-seeking. The 70 min coding window, 20-s scan/10-s recording cycle, cluster-scanning procedure, missing-visibility rule, and OCEI formula are described in the Protocol section. Please click here to download this Table.

Table 2: Reflective expression scoring framework, score-level examples, and learning-analytics variable definitions. The table outlines the scoring rubric for reflective texts, including examples of scores 0, 1, 2, and 3 for each reflective dimension, and the operational definitions for extracted learning-analytics variables. Reflective components include descriptive specificity, analytic depth, self-regulatory orientation, social-prosocial meaning, overall reflection score, word count, and submission status. Learning analytics comprise active days, resource views, proportion of assigned resources opened, discussion replies posted, mean video completion rate, assignment punctuality, and late-night access proportion. Please click here to download this Table.

Table 3: Participant characteristics, inclusion flow, and data completeness. The table summarizes participant eligibility, consent rates, exclusion criteria, final sample size, and data completeness across the three data sources. It reports baseline demographic characteristics alongside expected reflective entries, on-time submissions, late submissions, missing reflections, and successful learning-analytics extractions. Values are presented as n (%) unless otherwise specified. Rounding may result in percentages not summing exactly to 100%. Following the protocol, late reflections are archived but excluded from week-specific reflection analyses. Complete learning-analytics extraction denotes the successful retrieval of de-identified study-window records following the removal of instructor accounts, duplicate entries, and out-of-window events. Please click here to download this Table.

Table 4: Descriptive statistics and test-reliability scores of the primary study variables. The table presents week-specific mean values, standard deviations, and reliability indices for observed classroom behavior, reflective expression, learning analytics, and prosocial intention. The reliability of classroom observation data is evaluated using Cohen's kappa. Reflection scoring reliability is presented as intra-class correlation coefficients (ICCs) for double-scored texts. The internal consistency of the prosocial intention measure is reported using Cronbach's α. Learning-analytics variables are calculated over the identical 4-week period utilized for observations and reflections. Please click here to download this Table.

Table 5: Multivariable associations among observed classroom engagement, reflective expression, learning analytics, and prosocial intention. The table details the inferential models utilized in the main analyses. Presents multivariable predictors of the reflection total score, adjusted predictors of post-protocol prosocial intention (controlling for baseline levels), and selected unadjusted associations. β values represent unstandardized regression coefficients. All multivariable models are adjusted for class section. Standard errors (SE), 95% confidence intervals (CI), and p-values are provided for all estimates. Please click here to download this Table.

Discussion

This study proposes a practical protocol that integrates behavioral observation, reflective writing, and learning analytics to comprehensively investigate university students' participation, self-expression, and prosocial awareness within a single instructional window. By capturing different dimensions of engagement, this tri-modal approach addresses recent critiques in higher education research, which argue that time-in-use metrics and sensor-based monitoring are insufficient when evaluated in isolation¹. Over the 4-week protocol, visible classroom engagement significantly improved. Specifically, verbal participation and peer collaboration increased alongside on-task attention, while help-seeking and task-focused note-taking exhibited minimal change. These behavioral shifts indicate that as students acclimate to the instructional environment, their engagement evolves into more strategic, socially interactive, and contribution-oriented participation¹⁶.

From a method-focused perspective, the protocol's value depends on disciplined implementation rather than on observing uniformly positive results. The revised workflow therefore emphasizes a prespecified OCEI formula, independent observation, documented software processing, rubric examples, reliability thresholds, and explicit missing-data rules. These additions are intended to make the protocol reproducible in ordinary classroom settings where platform records, student attendance, camera visibility, and reflection quality are imperfect.

The protocol should also be interpreted as format-sensitive. In large lecture courses, observers may need smaller rotating clusters and a longer calibration period; in online classes, behavioral observation may rely on interaction logs and visible participation rather than room scanning; in laboratories, safety-related behavior may require an additional indicator; and in culturally diverse classrooms, reflection prompts and prosocial-intention items should be reviewed for linguistic clarity and cultural appropriateness before use.

Reflective writing provides a secondary layer of insight into these behavioral patterns. Total reflection scores demonstrated a steady upward trend throughout the protocol, with the most pronounced improvements occurring in analytic depth and social-prosocial meaning. Rather than merely recounting classroom events, students demonstrated a growing capacity to analyze instructional situations from a broader, more socially conscious perspective. This aligns with recent findings that repeated, structured reflection tasks foster higher-order meaning-making and knowledge-based discourse, even when surface-level text volume remains relatively constant⁹. Consequently, reflective writing in higher education is best understood as a multidimensional construct, wherein skills related to interpretation, transfer, and self-regulatory projection develop at different rates over time⁵.

The integration of learning analytics further justifies the necessity of a multi-source protocol. Substantive actions, such as discussion replies and assignment punctuality, proved to be far more informative indicators of engagement than raw activity volume. Conversely, late-night platform access showed no meaningful association with either reflection quality or prosocial intention. These findings support previous research cautioning against the overreliance on simple frequency or login metrics to evaluate student involvement⁴. Our results suggest that students who interact constructively with course materials develop a stronger capacity for reflection; however, basic platform activity does not perfectly mirror observable classroom engagement. The moderate overlap—rather than excessive redundancy—among observation, reflection, and analytics thus validates the use of composite measurement strategies over independent, single-source inferences¹⁷.

A secondary contribution of this research is the expansion of evaluated outcomes beyond standard participation metrics. Prosocial intention exhibited a measurable increase from the pre-test to the post-test, showing the strongest associations among students who articulated social-prosocial themes in their reflections and maintained an extensive interaction profile. This is consistent with current higher education research, which posits that cooperative, contribution-oriented characteristics are integral to classroom interactions rather than mere side effects⁸. Furthermore, these results support the conceptualization of classrooms as complex social systems. When students actively evaluate their relative contributions to the learning community, their behavior tends to shift from basic compliance toward constructive, prosocial engagement¹⁸.

These conclusions offer actionable insights for pedagogical design. The proposed protocol requires no specialized sensors or intrusive monitoring systems. By relying exclusively on interval-based observation, weekly structured reflections, and standard course-platform data, this combination provides a highly feasible solution for undergraduate teaching environments that require transparent, formative evidence of engagement but lack advanced technological infrastructure. Recent pedagogical frameworks emphasizing learning communities and value-creation teaching suggest that repositioning students as active co-creators, rather than passive consumers, infuses higher education participation with greater meaning¹⁹. Our empirical finding that contribution-related indicators cluster tightly with reflection quality and prosocial intention—far more so than simple platform traffic—strongly supports this pedagogical shift.

Several limitations must be acknowledged. First, because the protocol is designed for a limited number of class sections, its transferability to laboratory-based, lecture-dominant, fully asynchronous online, or culturally diverse environments requires planned adaptation and renewed reliability testing. Second, prosocial intention is measured by self-report and should be interpreted as a reported intention rather than a direct behavioral ground truth. Third, learning analytics capture platform-mediated activity and may miss offline preparation, informal peer support, and AI-assisted study that is not disclosed. Fourth, the 4-week observation window can identify short-term implementation patterns but may not capture longer developmental trajectories. Finally, the 4-week representative values should be interpreted as context-specific implementation evidence rather than generalizable causal estimates²⁰.

Future research should extend this framework in several directions. Replicating the protocol across diverse academic disciplines and instructional modalities is a critical next step. Additionally, longitudinal follow-ups are necessary to determine whether these short-term reflective practices generate lasting prosocial attitudes or sustained behavioral benefits. Subsequent studies could also compare this observational framework with alternative methodologies, such as peer assessment, group-level artifact analysis, or event-sequence analysis. Nevertheless, in its current form, this protocol provides a robust, operational mechanism for triangulating what students do in the classroom, how they interpret those actions afterward, and how they position themselves socially within the broader learning environment.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Classroom observation coding sheet	In-house	Six binary behavioral indicators	Interval-based coding of on-task attention, verbal participation, peer collaboration, task-focused note-taking/device use, off-task distraction, and help-seeking.
GraphPad Prism	Graphpad	Version 10.2.3	Generation of workflow diagrams, weekly trend plots, distribution plots, and association figures.
Institutional learning management system / course platform	Host university	Existing institutional LMS account system	Collection of de-identified learning analytics.
Microsoft PowerPoint	Microsoft	Microsoft 365 v2402	Generation of workflow diagrams, weekly trend plots, distribution plots, and association figures.
R	R Foundation		ggplot2 3.5.1
Spreadsheet software	Microsoft Corporation	Microsoft Excel 365 v2402	Data entry, cleaning, de-identification checks, and merging of observation/reflection/analytics datasets.
SPSS Statistics	IBM Corp	Version 26.0	Statistical analysis software; Descriptive statistics, repeated-measures comparisons, Cohen's kappa, ICC, Cronbach's alpha, correlations, and regression models.