Classroom Behavior Recognition via Transformer and Attention Mechanism: Application in Teaching Quality Evaluation for AI Medical Education

Summary

This study presents a Transformer-based AI system for recognizing classroom behavior in medical and nursing education. Using multimodal data, the system assesses learner engagement and emotional states. The results demonstrate higher accuracy compared to traditional models, enabling real-time feedback and enhanced support for teaching quality and student mental health.

Abstract

This study introduces a Transformer-based artificial intelligence system designed for classroom behavior recognition, aimed at enhancing teaching quality assessment and mental health monitoring in medical and nursing education. The model utilizes multimodal data, specifically video, audio, and skeletal movement, to evaluate key indicators of student engagement, such as attention span, interaction frequency, and emotional fluctuations. A novel multimodal fusion strategy, combined with spatiotemporal attention mechanisms, enables the system to capture complex classroom behaviors with improved precision and robustness. Experimental results on the EduSense and SBU Kinect datasets demonstrate that the proposed method significantly outperforms traditional models in both accuracy and false alarm rate. Additionally, ablation studies confirm the contribution of spatial and temporal attention modules to the system's recognition capability. The framework supports real-time feedback and visual behavior mapping, offering practical value in experiential learning environments. This approach provides a scalable and adaptive solution for classroom behavior monitoring and supports early intervention strategies in medical education.

Introduction

With the increasing mental health challenges among nursing and medical students due to academic and clinical stress, integrating artificial intelligence into classroom environments can provide not only teaching quality monitoring but also real-time psychological support. This study positions behavior recognition within medical education to support experiential learning and promote student well-being¹. Intelligent classroom refers to a new and high-quality educational model that combines intelligent and personalized teaching processes through the application of cutting-edge technologies, such as information technology, artificial intelligence, b....

Protocol

This study uses publicly available datasets that do not contain any personally identifiable information. All data used in the research have been anonymized to ensure the privacy of the participants. Informed consent was obtained from all participants at the time of data collection, and the study adheres to ethical guidelines. The protocol explains the intelligent classroom behavior recognition method, which consists of feature extraction based on multimodal data fusion and spatiotemporal behavior recognition based on Transformer and attention. The performance of the entire method is optimized through joint training.

1. Multimodel data....

Results

To assess the efficacy and superiority of the raised intelligent classroom behavior recognition method with Transformer and AM, experimental performance verification and analysis were conducted. Firstly, the experimental environment and parameters were configured, as denoted in Table 1.

Based on Table 1, the study selected the EduSense Dataset and SBU Kinect Interaction Dataset as experimental datasets, named D1 and D2, respectively. D1 collected the behaviors.......

Discussion

A behavior recognition method for intelligent classrooms based on Transformer and AM was proposed to address the challenges of complex student behavior and multimodal data recognition. A ViT network with global spatiotemporal modeling capability was constructed by integrating data from video and skeletal modalities, and self-attention and cross AMs were utilized to capture spatiotemporal features of behavior. By integrating video and audio data into a new Intelligent Sensing Framework (ISF) framework using a Multimodal F.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
128 GB DDR4 RAM	Multiple vendors	https://www.crucial.com/memory/ddr4	Sufficient memory for multimodal data processing
2 TB SSD Storage	Multiple vendors	https://www.samsung.com/ssd	For storing datasets and models
BERT (Base configuration)	Hugging Face	https://huggingface.co/bert-base-uncased	Language model for skeletal sequence embedding
CUDA 11.1 Toolkit	NVIDIA	https://developer.nvidia.com/cuda-toolkit	Enables GPU acceleration for PyTorch
cuDNN 8.0 Library	NVIDIA	https://developer.nvidia.com/cudnn	GPU-accelerated library for deep neural networks
EduSense Dataset	Carnegie Mellon University	https://www.cs.cmu.edu/~./edusense	Real-life university classroom dataset
Faster R-CNN (ResNet-50)	Open Source (PyTorch)	https://github.com/facebookresearch/detectron2	Object detector used for video feature extraction
Intel Xeon E5-2680 v4 CPU	Intel	https://www.intel.com/products/xeon-e5-2680-v4	High-performance processor for model training
Matplotlib	Open Source	https://matplotlib.org	Used for plotting performance metrics
NVIDIA Tesla V100 GPU	NVIDIA	https://www.nvidia.com/en-us/data-center/tesla-v100	Used for training and inference; supports real-time behavior recognition
OpenPose	CMU	https://github.com/CMU-Perceptual-Computing-Lab/openpose	Extracts 2D skeletal keypoints from video frames
PyTorch 1.8 Framework	Open Source	https://www.pytorch.org	Deep learning library used for model development
SBU Kinect Interaction Dataset	Stony Brook University	https://www3.cs.stonybrook.edu/~kyunghan/sbu_kinect	Interaction-based skeletal dataset
TensorBoard	Open Source (Google)	https://www.tensorflow.org/tensorboard	Used for training visualization
Ubuntu 20.04 LTS Operating System	Canonical	https://www.ubuntu.com/download	Linux OS used as the development environment
Vision Transformer (ViT)	Google / Hugging Face	https://huggingface.co/google/vit-base-patch16-224	Used for global spatiotemporal behavior modeling