This work presents a system that measures the attention level of a student in a classroom using cameras, smartwatches, and artificial intelligence algorithms. This information is subsequently presented to the teacher for them to have an idea of the general state of the class.
One of the main critical steps of the protocol is the synchronization of the smartwatch information with the color camera image, as these have different frequencies. This was solved by deploying raspberries as servers that receive information from the smartwatch and cameras with their respective timestamps and perform a rough matching of this information. Finally, this information is sent to a centralized server for further processing. The other critical step of the protocol is the definition of the final classifier to generate the final inference with the data obtained from different sources. To resolve this point, the raw data must be preprocessed through different subsystems to generate valuable information such as the pose, emotion, or action of the user. This information, together with biometric data, is normalized and combined to estimate the attention level of each student.
The experimental results suggest that head direction and pose estimation can be accurately estimated using zenith and single cameras, while emotion recognition performs better with the most common emotions, such as happiness or neutrality. Regarding the action classifier with the smartwatch, highly distinguishable actions, such as writing, typing, or texting, present good detection accuracy and are suitable for the system.
This system offers several advantages and distinct features when compared to existing or alternative methods, as described below.
Objective and continuous measurements
Smartwatches and cameras provide objective and continuous measurements of students' attention throughout the class session. Traditional methods, such as self-reporting with questionnaires, can be subjective and prone to bias. The use of handheld devices and cameras eliminates reliance on self-reporting or external judgments, allowing educators to obtain more reliable and detailed data on attention levels.
Real-time information
The method captures attention data in real-time within the natural context of the classroom. It allows teachers to understand attention patterns during the course of a lesson. This is crucial as attention fluctuates during a class, and capturing real-time data allows for a more accurate assessment of attention dynamics.
Multimodal data integration
By combining data from smartwatches, which provide physiological measures, and cameras, which provide visual information, educators can gain a better understanding of attention. Physiological measures, such as movement patterns, can complement visual observations from cameras, providing a richer and more nuanced representation of attentional states. This multimodal approach increases the reliability and validity of attention assessment.
However, although this system has several advantages, there are some limitations to be considered, as described below.
Ethical and privacy concerns
Collecting physiological and visual data from students raises privacy concerns. It is essential to ensure that proper informed consent is obtained from participants and that data are anonymized and securely stored to protect the privacy and rights of the individuals involved. Unauthorized access or misuse of sensitive data must be prevented.
Reliability and validity of data
Although smartwatches and cameras can provide objective measurements, ensuring the reliability and validity of the data collected can pose challenges. Technical limitations of the devices, such as sensor accuracy or signal interference, can affect data quality. Calibration and validation procedures are necessary to establish the accuracy and consistency of the measurements.
Interpretation of attention signals
Interpreting attention-related signals obtained from smartwatches and cameras requires careful analysis. Attention is a complex cognitive process influenced by various factors, and physiological signals or visual information may not always directly correlate with attention levels, as they might not capture other important aspects of attention, such as cognitive engagement, selective focus, or mental effort.
Invasive or disruptive nature
Wearing devices or being observed by cameras may alter students' behavior or attention levels. Some students may feel self-conscious or uncomfortable, which may affect their natural attention patterns. It is important to consider the potential impact of the method itself on the attention being measured and to minimize any disruption to the learning environment. To overcome some potential distractions, we simplified the use of the smartwatch, which works by simply wearing it like a normal watch, and any other functions were disabled, such that it is no longer a distraction that can alter the learner's level of attention. The students receive a brief explanation of the experiment and their goals to give them time to get accustomed to the setting.
Regarding head pose17, pose estimation18, and emotion prediction19, comprehensive and exhaustive experimentation is carried out in the corresponding articles. In summary, it should be noted that these three systems work properly, achieving high accuracy and resolving the corresponding tasks. However, despite the good performance, the approaches have some limitations. Regarding head direction estimation, it tends to provide erroneous predictions when the head is heavily tilted towards any direction. When this happens, the landmark estimation system performs poorly as many of the interesting points are not present in the image as a consequence of the abovementioned self-occlusion. Thus, this event inevitably leads to poor performance. This case is also present in the pose estimation method, as the approach is similar. If there are important key points that are not visible within the input image, the pose estimation methods tend to make them up, leading to erroneous and impossible poses. This effect can be mitigated by correctly placing the zenithal and individual cameras. As for emotion prediction, it tends to accurately detect some facial expressions whilst struggling with others. For instance, neutral and happy emotions are consistently detected, while the system is more prone to fail on fear, disgust, and surprise emotions if they are only slightly or fleetingly shown.
Regarding network interferences, the specifications of the D-Link DSR-1000AC Router indicate a maximum WLAN data transfer rate of 1300 Mbps. Consequently, an average participant in the experiment would transmit 4.344 Mb. This calculation takes into account the transmission of 3 images per second, each with an average size of 180 kB, and a sensor data package from the smartwatch of 3 kB per second. Therefore, considering the zenithal images, the theoretical maximum number of students that can be connected simultaneously is 297. However, this number will be influenced by the final number of devices connected to the router and the congestion level of the WiFi channel at the moment of the experiment.
Finally, the proposed method has several important implications and potential applications in various research areas, as described below.
Education and research
This method can provide valuable insights into the factors that affect students' attention and engagement in the classroom. Researchers can analyze the data collected from smartwatches and cameras to understand how different teaching methods, classroom environments, or even individual student characteristics influence attention levels.
Special education
This can be especially beneficial for studying attention-related challenges in students with special needs. Researchers can use data collected from smartwatches and cameras to identify patterns that lead to attention difficulties. This information can help develop targeted interventions and personalized strategies to support students with attention deficit disorders or other attention-related conditions.