Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students

Summary

A protocol to evaluate solutions based on commercial-off-the-shelf (COTS) wrist wearables to estimate stress in students is proposed. The protocol is carried out in two phases, an initial laboratory-based stress induction test, and a monitoring stage taking place in the classroom while the student is performing academic activities.

Abstract

Wearable commercial-off-the-shelf (COTS) devices have become popular during the last years to monitor sports activities, primarily among young people. These devices include sensors to gather data on physiological signals such as heart rate, skin temperature or galvanic skin response. By applying data analytics techniques to these kinds of signals, it is possible to obtain estimations of higher-level aspects of human behavior. In the literature, there are several works describing the use of physiological data collected using clinical devices to obtain information on sleep patterns or stress. However, it is still an open question whether data captured using COTS wrist wearables is sufficient to characterize the learners' psychological state in educational settings. This paper discusses a protocol to evaluate stress estimation from data obtained using COTS wrist wearables. The protocol is carried out in two phases. The first stage consists of a controlled laboratory experiment, where a mobile app is used to induce different stress levels in a student by means of a relaxing video, a Stroop Color and Word test, a Paced Auditory Serial Addition test, and a hyperventilation test. The second phase is carried out in the classroom, where stress is analyzed while performing several academic activities, namely attending to theoretical lectures, doing exercises and other individual activities, and taking short tests and exams. In both cases, both quantitative data obtained from COTS wrist wearables and qualitative data gathered by means of questionnaires are considered. This protocol involves a simple and consistent method with a stress induction app and questionnaires, requiring a limited participation of support staff.

Introduction

State-of-the-art wearable technologies are widely available, and their application environments are continuously expanding. We can find in the market many different devices, among which COTS wrist wearables¹, such as smart watches and smart bands, are popular among athletes as a personal physical fitness monitoring tool². By applying data analytic techniques, the data obtained using these devices can be processed to provide indicators such as general physical state, sleep quality or recovery factor. The demonstrated applicability in this area raised interest in the academic community about their possible application to other fields, especially in the health domain^3,4, although the strict requirements of clinical trials limit their introduction. However, in a less demanding context such as education, we can find in the literature recent investigations involving the use of different types of wearable devices, both related to teaching activities^5,6 and to the estimation of certain characteristics of the student such as sleep patterns⁷, or the analysis of students' engagement in different educational activities⁸.

In our case, we focus on analyzing COTS wrist wearable devices as means to collect physiological signals that would eventually facilitate stress estimation, which in turn is a key aspect in educational contexts. Stress has a relevant influence in the development of academic activities and overall students' performance. For example, stress levels are directly related to the onset of the burnout syndrome in students^9,10,11, and high stress levels are especially relevant during the freshman year, where drop-out rates between 20% and 30%^12,13 are common. Detecting and controlling stress indicators could dramatically improve academic performance.

The use of COTS wrist wearable devices is justified because they have sensors that provide information on physiological signals that have been widely used by the scientific community in stress assessment and detection. Some of the signals referred to in the literature used for this purpose include heart rate (HR)¹⁴, heart rate varaibility¹⁵, skin temperature (ST)¹⁶, respiration¹⁴, and galvanic skin response (GSR)¹⁷. These signals can be collected by COTS wrist wearables. However, they do not offer the same performance as clinical devices. There are differences related to the accuracy of sensors among devices^18,19,20,21. Nevertheless, previous works^18,19,20,21 have shown that, in a slow movement scenario, COTS wrist wearable sensors have error patterns similar to specialized devices.

The aim of this paper is to introduce a protocol to evaluate different solutions for stress estimation in students using COTS wrist wearables. There are many arrangements that can be proposed to estimate stress levels, involving the use of different wrist wearable devices and data analytics techniques, and more specifically machine learning algorithms. COTS wrist wearables are characterized by their high fragmentation, heterogeneity and interoperability problems²². Three companies have an aggregated market share of almost 50%²³, but many other companies account for much smaller individual market shares, with an aggregated share above 50%. On the other hand, in terms of heterogeneity, not all wearables have the same number and type of sensors, with accelerometers and h sensors being the most common, and ST's and GSR's being only present in 5% of the devices studied. As for interoperability, there are different operating systems and data collection approaches that are not compatible with each other. As for the machine learning techniques that can be applied to estimate stress from the data collected by means of a wrist device, there are many options available²⁴, including decision trees, neural networks, nearest neighbor approaches, Naïve Bayes classifiers, etc. To sum up, there is a great variety of solutions that may be developed for stress estimation, so it is instrumental to design an evaluation protocol to facilitate the comparison among different tentative options to eventually select the most suitable in a given context.

For the implementation of the protocol, several tools are needed (Figure 1). First, a COTS wrist wearable device is needed to fetch physiological data. This wearable device should have at least HR monitoring capabilities, but additional sensors are desirable (e.g., accelerometer, ST, GSR sensors). Second, a smartphone running the PhysiologicalSignal app is required to collect the data captured by the wearable device. Third, a tablet running the StressTest app is needed to run stress induction exercises (the smartphone could be used instead the tablet for this purpose). Fourth, some questionnaires to collect qualitative data on students' perception on stress. Fifth, a server with a Web service²⁵ to perform data collection and pre-processing, and a Web dashboard to show the evolution of the signals. And finally, a data analytics package²⁶ to process the data collected about students using machine learning techniques.

The evaluation protocol is organized into two phases. The first one, the laboratory phase, is carried out in a comfortable room, where different stress levels (i.e., "relax", "concentrated stress" and "stress") are induced to a target subject (a student) through several common stress-inducing tasks. The second part takes place in the classroom, and it involves monitoring the student during the accomplishment of several academic activities: theoretical explanations, individual activities, short tests, exams, etc. During the implementation of this protocol, the subject's physiological signals are captured by means of a wrist device. Finally, these signals are processed by machine learning algorithms to provide estimations on the level of stress.

During the laboratory phase, the StressTest app is used to induce different stress levels. This app guides the subject to the completion of four different tasks. The first task is to create a baseline for stress analysis. In this task, the student visualizes a 4-minute relaxing video in which different shots of a sunset on a bridge are shown. The second task is an adaptation of the Stroop Color and Word Test²⁷ (SCWT). Every two seconds, the subject must choose the color in which the name of a color is painted (red, green, orange, blue and purple). Several buttons located at the bottom of the screen containing the initial letter of each color are available for the subject to choose the painted color at each time. For example, the button that refers to blue depicts the letter B. In our case, this test is divided into three different levels of difficulty. For the first level (SCWT1), the colored "words of colors" will appear in the same order as the buttons, so color and name match directly. This level is taken as baseline, as it does not involve any difficulty and the subject should only press the buttons properly, always in the same order. For the second level (SCWT2), the colored "words of colors" appear randomly, but the correspondence between name and color is maintained. Every time the subject fails a beep is emitted, and if two errors are made, the correct color score will be reset. For the last, most difficult level (SCWT3), name and color do not match. In this way this level is intended to be more complex and stressful for the subject. The third task consists on the Paced Auditory Serial Addition test (PASAT)²⁸, which measures how the student experiences a concentration test. During this task, a sequence of consecutive numbers is played aloud, and the student must add the last two numbers and write the result in the provided on-screen box before listening to the next number. In this task, if the subject makes a mistake, a disturbing event occurs to generate stress (two numbers sound at the same time or a long period of silence in maintained). In this case, if three errors are committed, the sum account will be reset. The fourth task consists on a hyperventilation activity to induce the same variation in the physiological signals that would provoke a stressful situation¹⁷. At the end of each task and level, the subject has to indicate the level of perceived stress, using the application itself, according to a 5-value Likert scale.

During the classroom phase, students carry out their ordinary academic activities together with the rest of their classmates. The protocol focuses on the stress levels that occur during classroom-specific activities. At the end of the lecture, a brief questionnaire (Annex 1) is completed by the student to indicate the perceived level of stress in the several activities according to a 5-value scale.

Protocol

All methods described below have been approved by the regional government of Galicia's committee for research ethics of Pontevedra-Vigo-Ourense (reg. code 2017/336). The protocol was implemented for first year students at the School of Telecommunication Engineering - University of Vigo, both in a comfortable laboratory room and in several lectures and practice sessions of a bachelor's degree course on Computer Architectures.

1. Prepare the Devices

1. Connect the smartphone and tablet device to a stable internet connection.
2. Turn on Bluetooth communications in the smartphone.
3. In the smartphone, search in the corresponding official app store the wrist wearable application. Download and install it.
4. In the smartphone, search for the PhysiologicalSignals app to capture physiological signals. Download and install it.
   NOTE: Currently, the app is a beta version and access can be provided by request.
5. In the tablet, search for the StressTest app to be used in the research laboratory experiments. Download and install it.
   NOTE: Currently, the app is a beta version and access can be provided by request.
6. Turn on the COTS wrist wearable device and place the wearable.
7. In the smartphone, open the official COTS wrist wearable application.
   NOTE: The app will proceed to synchronize the wearable device with the smartphone. In some devices, an e-mail address is required.
8. In the smartphone, open the PhysiologicalSignals app.
   1. In case of being notified of a sensor access request, accept it.
   2. Check the device. Wait for the PhysiologicalSignals app to display the word Weared in green.
      NOTE: This indicate that the wearable device has been detected and therefore the transmission of information from the sensors to the smartphone is enabled. If this message does not appear repeat from step 1.6.

2. The Laboratory Phase

1. Prepare the laboratory setting. Choose a comfortable and non-disturbing room without distracting noise and with a comfortable temperature (between 22 °C and 26 °C).
2. Turn on the wrist wearable device, place it around the subject´s non-dominant wrist and place the headphones on the head of the student. Fit the wearable tightly but comfortably around the wrist.
3. Connect the smartphone and tablet to a stable internet connection and verify that the Bluetooth connection is active.
4. In the smartphone, launch the PhysiologicalSignals app.
   1. Wait for the app to display the word Weared in green.
   2. Select the Change User option in the left configuration menu and provide the ID of the subject who will complete the tests and click Save.
5. In a laptop, access the dashboard and enter the test administrator's ID and password.
   NOTE: Currently, for private and security concerns, access to the dashboard is only available under request.
   1. Select the subject ID and the subject's stress tab.
   2. Check the physiological signals evolution and wait for the wearable device to reach thermal stability before starting the experiment.
      NOTE: The thermal stability is identified as a plateau in the graph.
6. In the tablet, launch the StressTest application.
   1. Explain to the subject the four laboratory tasks. Show some of the screens and actions to perform during each one of the tasks.
      NOTE: This is very important, because the subject should feel stressed or relaxed in accordance to the performed activities, and not fear or concern about what is going to happen.
   2. Tell the student not to rest their arms on the table and to use the hand where the wearable device is placed to perform the activities.
   3. Enter the same user ID as in step 2.4.2 and click the arrow.
7. Launch the video task and give full control to the student.
   1. Observe that the task is carried out without incident.
   2. When the task is finished, check that the subject provides the perceived stress.
8. Launch the Stroop Color task (SCWT) consecutively for levels 1, 2 and 3.
   1. For each level, observe that the subtask is carried out without incident.
   2. When each subtask is finished, check that the subject provides the perceived stress.
   3. Only for level 3 and only in case the subject does not solve it after 4 minutes, terminate the task by pressing the arrow located at the top of the screen.
9. Launch the Paced Auditory Serial Addition test (PASAT).
   1. Observe that the task is carried out without incident.
   2. In case the subject does not solve the PASAT test after 4 minutes, terminate the task by pressing the arrow located at the top of the screen.
   3. When the task is finished, check that the subject provides the perceived stress.
10. Launch the Hyperventilation test.
    1. Observe the evolution of HR using the dashboard. If physiological signals do not change significantly, ask the subject to increase inspiration and expiration rates gradually.
    2. In case the subject feels dizziness or uncomfortable halt this task. In any case, complete the task after four minutes.
    3. When the task is finished, check that the subject provides the perceived stress.

3. The Classroom Phase

1. Turn on the wrist wearable device and place the wearable around the subject´s non-dominant wrist. Fit the wearable tightly but comfortably around the wrist.
2. Connect the smartphone to a stable internet connection and verify the Bluetooth connection is active.
3. In the smartphone, launch the PhysiologicalSignals app.
   1. Wait for the app to display the word Weared in green.
   2. Select in the configuration menu the Change User option, provide the ID of the subject who will complete the tests and click Save.
4. In a laptop, access the dashboard and enter the test administrator's ID and password.
   1. Select the subject ID and the subject's stress tab.
   2. Check the evolution of physiological signals.
5. Take annotations about any relevant event occurring in the classroom in relation to the student-teacher interaction.
   NOTE: Relevant information and basic events will be used to label physiological samples afterwards. Example events are a question from the teacher to the student, or a theoretical explanation is initiated.
6. At the end of the lecture, ask the subject to complete the questionnaire about their level of stress at specific times during the session, according to a 5-level scale.

4. Data Analysis

1. In a laptop, access the dashboard and enter the test administrator's ID and password.
   1. Select the subject ID and the subject's stress tab.
   2. Select the day of a classroom experiment.
2. Label the samples of the subject by identifying activities and perceived stress levels.
   1. Identify lecture-room activities and their duration according to the starting and finishing times and their types.
   2. For each activity, select a perceived stress level.
3. For each subject and each session, download the file with the tagged samples.
   NOTE: A comma-separated-values (CSV) file is created for each student, each row reflecting the values of the physiological signals with their standard deviation, slope and diff, the activity type, the activity-based stress (i.e., the stress associated by default to the activity) and the subject perceived stress.
4. Launch the data analytics package.
   1. Choose a set of classifiers (e.g., SVM, C4.5, k-NN, Random Forest, Naïve Bayes and Zero R) and import the CSV file for all students for each session.
   2. Train and evaluate classifiers using the 10-fold cross-validation technique.
      NOTE: Depending on the analyses, activity type, activity-based stress or stress perceived, shall be selected as dependent variable for the analysis.
   3. Finally, check the results for accuracy and error rates.

Results

The protocol discussed was put into practice in a Computer Architectures course in the first year of the Telecommunication Engineering degree at the University of Vigo. This course has more than 200 students enrolled who are organized into 10 working groups. To carry out this experiment, students from four of the groups were invited to enroll at the beginning of the academic year. The project attracted considerable interest among the students, and around 30 students volunteered to partici...

Discussion

COTS wearable devices are among the most popular consumer electronics products available today. These devices are typically used to monitor physical activities, but their capabilities and performance could be of great interest in other areas. In this paper, a protocol to evaluate the use of COTS wearable devices for estimating stress in learning environments is discussed. The definition of such a protocol is especially relevant in order to analyze different solutions involving wearables and machine learning algorithms. T...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Microsoft Band 2	Microsoft		Wearable
Nexus 5	LG/Google		Smartphone
PhysiologicalSignals	Developed by the paper authors		App to collect data from wearables
StressTest	Developed by the paper authors		App to develop laboraty experiment: Video visualization, Stroop color and word test, paced auditory serial addition test (PASAT) , hyperventilation activity
Quizs	Developed by the paper authors		Questionnaires to collect qualitative data on students’ perception on stress
Server	Developed by the paper authors		Server to store, analyze and display data
Weka application	University of Waikato		Application to process the data using machine learning techniques