Eye-tracking Technology and Data-mining Techniques used for a Behavioral Analysis of Adults engaged in Learning Processes

Summary

We present a protocol for a behavioral analysis of adults (ages 18 to 70-year-old) engaged in learning processes, undertaking tasks designed for Self-Regulated Learning (SRL). The participants, university teachers and students, and adults from the University of Experience, were monitored with eye-tracking devices and the data were analyzed with data-mining techniques.

Abstract

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous technological changes and scientific advances, there is a need for life-long learning and education within both formal and non-formal educational environments. In response to this challenge, the use of eye-tracking technology and data-mining techniques, respectively, for supervised (mainly prediction) and unsupervised (specifically cluster analysis) learning, provide methods for the detection of forms of learning among users and/or the classification of their learning styles. In this study, a protocol is proposed for the study of learning styles among adults with and without previous knowledge at different ages (18 to 69-year-old) and at different points throughout the learning process (start and end). Statistical analysis-of-variance techniques mean that differences may be detected between the participants by type of learner and previous knowledge of the task. Likewise, the use of unsupervised learning clustering techniques throws light on similar forms of learning among the participants across different groups. All these data will facilitate personalized proposals from the teacher for the presentation of each task at different points in the chain of information processing. It will likewise be easier for the teacher to adapt teaching materials to the learning needs of each student or group of students with similar characteristics.

Introduction

Eye-tracking methodology applied to behavioral analysis in learning
Eye-tracking methodology, among other functional uses, is applied to the study of human behavior, specifically during task resolution. This technique facilitates monitoring and analysis during the completion of learning tasks¹. Specifically, the attention levels of students at different points of the learning process (start, development, and end) in different subjects (History, Mathematics, Science, etc.) can be studied with the use of eye-tracking technology. In addition, if the task includes the use of videos with a voice that guides the learning process, Self-Regulated Learning (SRL) is facilitated. Therefore, the implementation of eye-tracking technology in the analysis of tasks to which SRL (that include the use of videos) is proposed as a significant resource to understand how learning is developed^2,3,4. This combination will also mean that the differences between instructional methods (with or without SRL, etc.) may be checked with different types of students (with or without prior knowledge, etc.)⁵. In contrast, the presentation of multi-channel information (simultaneous presentation of both auditory and visual information, whether verbal, written, or pictorial) can facilitate both the recording and the analysis of relevant versus non-relevant information from the above-mentioned variables⁶. Students with prior knowledge exposed to multimedia learning channels appear to learn more effectively than those with little or no prior knowledge. Students with high levels of prior knowledge of the subject matter will integrate textual and graphical information more effectively⁷. This functionality has been observed in the learning of texts⁸ that include images⁹. Eye-tracking technology offers information on where attention is focused and for how long. These data give insight into the development of a learning process in a more precise way than through the simple observation of the resolution process during the completion of a task. Also, the analysis of these indicators facilitates the study of whether the student develops deep or superficial learning. Furthermore, the relationship between these data and the learning results facilitates the validation of the information obtained with eye-tracking technology^4,10. In fact, this technique together with SRL are increasingly used in Higher Education and in Adult Education¹¹ learning environments, both on regulated and on non-regulated courses¹².

Eye-tracking technology offers different metrics: distance, speed, acceleration, density, dispersion, angular velocity, transitions between Areas of Interest (AOI), sequential order of AOI, visits in the fixations, saccades, scan path and heat map parameters. However, the interpretation of these data is complex and requires the use of supervised (regression, decision trees, etc.) and unsupervised (k-means cluster techniques, etc.)^13,14 data-mining techniques. These metrics can be applied for monitoring the behavior of the same subject over time or for a comparison between several subjects and their performance with the same task¹⁵, by analyzing the difference between participants with previous knowledge versus no previous knowledge¹⁶. Recent research^11,17 has revealed that novice apprentices fixate longer on the stimuli (i.e., there is a greater fixation frequency while similar scan-path patterns are recorded). The average duration of fixation was longer for experts than for novices. The experts presented their focus of attention on the middle points of the information (proximal and central), differences that may also be seen in the visualization points within the AOI on the heat maps.

Interpretation of metrics in eye tracking
Recent studies¹⁸ have indicated that information acquisition is related to the number of ocular fixations on the stimuli. Another important metric is the saccade, which is defined as the rapid and sudden movement of a fixation with an interval of [10 ms, 100 ms]. Sharafi et al. (2015)¹⁸ found differences in the number of saccades, depending on the information coding phase of the student. Another relevant parameter is the scan-path, a metric that captures the chronological order of the steps that the participant performs for the resolution of the learning task within the AOI defined by the researcher¹⁸. Similarly, eye-tracking technology can be used to predict the participant's level of understanding, which appears to be related to the number of fixations. Recent studies have indicated that variability in gaze behavior is determined by the properties of the image (position, intensity, color, and orientation), the instructions for performing the task, and the type of information processing (learning style) of the participant. These differences are detected by analyzing the student's interaction with the different AOI¹⁹. Quantitative²⁰ (frequency analysis) and/or qualitative or dynamic²¹ (scan path) techniques can be used to analyze the data collected from the different metrics. The former techniques are analyzed with traditional statistical techniques (frequency analysis, mean difference, variance difference, etc.) and the latter are analyzed with Machine Learning techniques (Euclidean distances with string-edit methods^21,22, and clustering¹⁷). The application of these techniques facilitates clustering, by considering different characteristics of the subjects. One study¹⁷ found that the more expert the student, the more effective the spatial and temporal information processing strategy that is implemented. A descriptive table of the measurement parameters that were used in this study can be consulted below in Table 1.

Table 1: Most representative parameters that can be obtained with the eye-tracking technique, adapted from Sáiz, Zaparaín, Marticorena, and Velasco (2019).²⁰ Please click here to download this Table.

Application of the eye-tracking methodology to the study of the learning process
The use of the technological advances and the data-analysis techniques described above⁵ will add greater precision to behavioral analysis of learners during problem solving in the different phases of information processing (task initiation, information processing, and task resolution). It will all facilitate individual behavioral analysis, which will in turn permit the grouping of students with similar characteristics²⁴. Likewise, predictive techniques (decision trees, regression techniques, etc.)²⁵ can be applied to learning, related both to the number of fixations and to the task-resolution results of each student. This functionality is a very important advance in the knowledge of how each student learns and for the proposal of personalized learning programs within different groups (people with or without learning difficulties²⁶). Therefore, the use of this technique will contribute towards the achievement of personalization and optimization of learning²⁷. Life-long learning must be understood as a cycle of continuous improvement since the knowledge of society is constantly advancing and progressing. Evolutionary psychology indicates that resolution skills and effectiveness in information processing decrease with age. Specifically, saccade frequency, amplitude, and speed of eye movements among adults have been found to decrease with age. In addition, at older ages, attention is focused on the lower areas of visual scenes, which is related to deficits in working memory¹⁴. Nevertheless, activation increases in the frontal and prefrontal areas at an older age, which appears to compensate for these deficits in task resolution. This aspect includes the level of previous knowledge and the cognitive compensation strategies that the subject can apply. Experienced participants learn more efficiently, since they manage attention more effectively, due to the application of automated supervision processes²⁸. In addition, if the information to be learned is imparted through SRL techniques, the aforementioned deficiencies are mitigated¹⁷. The use of such techniques means that visual tracking patterns are very similar, both in subjects without prior knowledge and in subjects with prior knowledge⁷.

In summary, the analysis of multimodal-multichannel data on SRL obtained with the use of advanced learning (eye-tracking) technologies is key to understanding the interaction between cognitive, metacognitive, and motivational processes, and their impact on learning²⁹. The results and the study of differences in learning have implications for the design of learning materials and intelligent tutoring systems, both of which will enable personalized learning that is likely to be more effective and satisfactory for the student³⁰.

In this research, there were two investigation questions asked: (1) Will there be significant differences in the learning results and in the ocular fixation parameters between students and expert versus non-expert teachers in Art History differentiating students with official degrees versus students with non-official degrees (University of Experience - Adult education)? and (2) Will clusters of each participant with learning results and ocular fixation parameters coincide with the type of participants (students with official degrees, students with non-official degrees (University of Experience - Adult education) and teachers)?

Protocol

This protocol was performed in compliance with the procedural regulations of the Bioethical Committee of the University of Burgos (Spain) nº Nº IR27/2019. Prior to their participation, the participants had been made fully aware of the research objectives and had all provided their informed consent. They received no financial compensation for their participation.

1. Participant recruitment

1. Recruit participants from among a group of adults within two environments (students and teachers), with an age rank of 18 to 69 year-old in the environment of Higher Education (formal and non-formal education).
2. Include participants with normal or corrected-to-normal vision and hearing.
3. Exclude participants with neurological, psychiatric, and sleep disorders, disabilities related to educational special needs, perceptual difficulties (impaired sight and hearing), and cognitive disabilities.
   ​NOTE: In this study we worked with a sample of 40 participants, 6 students from the University of Experience (one participant was excluded in the category of students from the university of experience because of visual difficulties), 25 university professors in the disciplines of health sciences, engineering, and history and heritage, and 9 undergraduate and master's students following courses in health sciences, engineering, and history and heritage. The participants had no cognitive, hearing, nor visual problems, and they all had normal or corrected-to-normal vision (Table 2). That is why, one of the participants was eliminated before starting the experiment because nystagmus had been detected on him and therefore the task was applied to a sample of 39 participants. The participants received no financial nor professional compensation; that is why participants' motivation was high as it was only based on their interest to know how this eye-tracking method works during a learning process related with cultural heritage, specifically the origin of European monasteries.

Table 2. Characteristics of the sample. Please click here to download this Table.

2. Experimental Procedure

1. Session 1: Collection of informed consent, personal data, and background knowledge
   1. Obtain informed consent. Before the test, inform each participant of the aims of the study and the collection, treatment, and storage of their data. The agreement of each participant is given by signing the informed consent form.
      NOTE: Participation in this study was voluntary and there was no financial reward. This aspect ensured that the completion of the tasks had no economic motivation. Before starting the task, the interviewer, an expert in the field, fills up a questionnaire with questions on age, gender, occupation, and prior knowledge of the subject matter, in this case, the origin and historical development of monasteries in Europe (see Table 3). This study is part of a European Project (2019-1-ES01-KA204-095615-Coordinator 6) on adult learning about the Cultural Heritage of Humanity throughout life; that is why this type of task was chosen. Each investigator will choose the topic depending on his or her work field.

Table 3. Interview questionnaire. Please click here to download this Table.

2. Session 2: Calibration
   1. Inform the participant about how eye-tracking technology works and how the information will be collected and recorded and calibrate: "We will use eye-tracking technology to observe the completion of the learning task on the origin and the development of European monasteries. Eye tracking is a technology that allows you to follow your gaze while you perform the activity and it has no side effects, nor it is invasive, since in this study only eye tracking is recorded".
   2. Explain to the participant that a valid test requires proper positioning. Have the participant must sit at a certain distance [45 to 60 cm] from the monitor. The distance will depend on the height of the participant, the lower the height, the shorter the distance.
   3. Inform the participant that a series of points will appear on the cardinal points of the screen and that as each point appears the participant must observe it with the eyes. The participant can move from one point to another by using the "enter" cursor. The calibration phase has a duration of 10-15 minutes.
      NOTE: An Eye-tracking iViewer XTM, SMI Experimenter Center 3.0, and SMI Be Gaze and a monitor with a resolution of 1680×1050 were used for the task-resolution exercise. This equipment registers ocular movements, their coordinates, and pupillary diameters of each eye. In this study, 60 Hz were applied, scan-path metrics and dynamic scan-path metrics were used, and AOI statistics were determined.
   4. Check the calibration setting. The professional supervising the test analyzes the calibration setting on the control screen.
      1. Perform calibration through the calibrating system that is included in the Eye-tracking iViewer XTM. Before starting this task, each participant realizes a visual follow-up of four points on a screen to the four corners (up-right, up-left, down-right, down-left). Afterward, the software has an execution verification process of the right position of these stimuli and gives information on the parameter adjustment in degrees. If this adjustment is situated between 0.6º ± 1 in the right and left eye, it is considered that the calibration is correct, and the task execution starts. An example of the process can be verified in Figure 1.
         ​NOTE: Correct task completion is considered when the degrees in the right and left eye are set at 0.6º ± 1 standard deviation. In this study, two calibrations were detected among the group of university professors that exceeded the adjustment criterion of 0.6º ± 1 and two participants were therefore removed. The 25 participants in the first sample were therefore reduced to 23 participants.

Figure 1. Process of eye-tracking calibration Please click here to view a larger version of this figure.

3. Session 3: Performing the learning task
   1. Explain the contents of the task to the participant. An expert in instructional psychology explains to the participant what the task will consist of and how to perform it: "The video is 1:14 seconds long and consists of 5 voice-over images. At the end, the participant is invited to complete a small crossword puzzle to check that the information presented in the video has been understood".
   2. Watch the video clip. The video used in the task can be viewed at the following link: https://youtu.be/HlGGgrYDTFs.
      NOTE: The task consists in watching a video that offers information on the origins of European monasteries. The information has been elaborated by a specialist, an Art History teacher. The information is organized in two channels, one visual which includes images and written information presented as outlines and another audio one because a SRL specialist teacher is speaking throughout the video insisting on the most significant contents using verbal emphasis.
   3. Performing the crossword puzzle on a Moodle-based virtual platform. Clicking on the crossword icon takes the participant to a virtual platform where the crossword may be completed, to check whether the knowledge has been acquired. The crossword puzzle is presented in Figure 2.

Figure 2. Crossword puzzle to check the acquired knowledge. Please click here to view a larger version of this figure.

4. Session 4: Data analysis
   1. Choose the Areas Of Interest (AOI). AOIs are defined in the video and are divided into AOIs that contain relevant information versus AOIs that include non-relevant information.
      NOTE: The AOI assignation is realized by the experimenter who decides which are the relevant or irrelevant AOIs in relation to the presented information.
   2. Extract the database relating to the parameters for AOI Fixations ("Event Start Trial Time", "Event End Trial Time" and "Event Duration"; "Fixation Position X", "Fixation Position Y", "Fixation Average Pupil Size", "Fixation Average Pupil Size Y px", "Fixation Average Pupil Diameter", "Fixation Dispersion X" and "Fixation Dispersion Y").
   3. Import the database into a statistical processing software package and select the option analyze and then classify, followed by the option k-means cluster. Then select cross-table in the statistical software package, for example SPSS, followed by the 'ANOVA' option, to analyze the differences between the participants (type of adult groups and degree of prior knowledge) with regard to their AOI Fixation parameters³¹.
      NOTE: Clustering or cluster analysis is an 'unsupervised' machine-learning technique, and, within k-means, it is a grouping method, the aim of which is to partition a set of n observations into k groups, in which each observation belongs to the group with the closest mean value. In this experiment, k-means clustering was used to check the clusters of participants in the learning task. This correspondence is important, because it offers the teacher or therapist information on the homogeneous functional development of users that goes beyond the diagnosis, providing information to propose similar intervention programs in some areas of functional development. This option is expected to facilitate full use of the educational or therapeutic service and its personal and material resources.
   4. Perform a visualization analysis of the data (descriptive and cluster analysis) that are processed, using a visualization software such as Orange³².
   5. Extract the data on the parameters of Detailed Statistics: Dwell Time, Glance Duration, Diversion Duration, Glance Count, Fixation Count, Average Fixation, and Duration then import that database into a statistical software package. Select the option 'ANOVA' in the statistical package and then conduct a visualization analysis of the data that were processed (means). Use the spreadsheet to generate a spider chart and specific bar graphs for the groups of participants.
5. Session 5: Personalized learning proposals
   1. Perform an intervention program to improve learning outcomes among the participants detected in the cluster analysis, due to their lower scores.
      NOTE: A summary of the phases followed in the experimental Procedure is shown in Figure 3.

Figure 3. Phases of the experimental procedure. Please click here to view a larger version of this figure.

Representative Results

The 36 participants recruited for the present study were from three groups of adults (students from the university of experience, university professors, and undergraduate and master's degree students) with ages ranging between [18 and 69] years (Table 2). The protocol was tested over 20 months at the University of Burgos. An outline of the development can be seen in Table 4.

Table 4. Outline of the development of the learning behavior analysis protocol. Please click here to download this Table.

First, the gaze position parameters of the fixations were analyzed (Table 5). In this study, using a video, the start and end time was the same for all participants: start 0 ms and end 1:14 s, duration 1:14 s.

Table 5. Fixations parameter results. Please click here to download this Table.

Figure 4 shows a graph of the tasks developed by the three groups with respect to the fixation parameters. The group of men and women in each of the participating groups (University of Experience Students, University Teachers and Graduate & Master's Students) each completed the tasks in different ways.

Figure 4. Graph of the three groups and their development of the task related to the fixation parameters. Please click here to view a larger version of this figure.

A two-factor fixed-effects ANOVA (type of participant and previous knowledge) was then applied to check whether there were significant differences in the parameters of fixation positions between the three groups (University of Experience students, University professors and University students). No significant differences were found in any of the fixation parameters, but a trend towards differences for Fixation Average Pupil Size Y, Fixation Average Pupil Diameter, and Fixation Dispersion X was noted, although with low effect values (see Table 6).

Table 6. Two-factor fixed-effects ANOVA (type of participant and prior knowledge) and effect value. Please click here to download this Table.

Subsequently, the k-means cluster was applied to study whether there were different groupings in the initial research group (University of Experience students, University professors and University students) with respect to the results in the parameters of fixation positions, previous knowledge and the crossword puzzle results. Three clusters were found (Table 7). A visualization of the clusters can be seen in Figure 5.

Table 7. Final Cluster Centers. Please click here to download this Table.

Then, a cross table was prepared between the values of the cluster of group membership assigned to each participant with respect to the category type of participant (University of Experience students, University professors and University students) (Table 8). Figure 5 shows the position of the participants within the clusters with respect to the three groups for the fixation position parameters.

Table 8. Participant * Cluster Number of Case Crosstabulation. Please click here to download this Table.

A two-factor fixed-effect ANOVA, "participant group" and "background", was performed for the following eye-tracking measurement parameters: Dwell Time, Glance Duration, Diversion Duration, Glance Count, Fixation Count, Average, and Duration obtained in the task start phase (Slide 1) and in the task end phase (Slide 5) (Table 6). Significant differences were found depending on the background knowledge variable in Diversion Duration 1 (analyzing the input, dwell and output time for each stimulus inserted in each AOI). It can therefore be concluded that the way of entering, remaining, and exiting in the different AOIs was different, depending on the variable "participant group" during the initial phase of information access (F₂,₃₂ = 4.07, p = 0.03, η² = 0.23). Differences were also found in the Average Fixation Duration parameter (longer fixations refer to the participant spending more time analyzing and interpreting the information content within the different AOIs (F₂,₃₂ = 3.53, p = 0.04, η² = 0.21). Bonferroni's mean difference test was applied to establish group membership, with which it was established that they were between the group (University of Experience students) and the group of University Professors [mean difference = 0.04, p = 0.04 CI 95% (0.03-2.75)]. The means were higher for group 1 (University of Experience students) where participants spent more time analyzing and interpreting the AOIs during the data entry phase (see Table 9 and Figure 6).

Table 9. Two-factor fixed-effects ANOVA (type of participant and previous knowledge) and effect value for the eye-tracking measurement parameters: Dwell Time, Glance Duration, Diversion Duration, Glances Count, Fixation Count, Average Fixation and Duration. Please click here to download this Table.

Figure 6. Graph of the three groups and their development of the task involving fixation parameters with respect to the prior knowledge variables at the start and at the end of task processing. Please click here to view a larger version of this figure.

Based on the results found in this study, the development of a personalized learning program was proposed to improve task-resolution learning outcomes. This program was focused on the work with the participants grouped in cluster 3, as they obtained scores of 3 points out of 5 in the test to check the learning results, which represented 85.43% of the total participants. Participants from the three study groups (University of Experience Students; University Teachers and Graduate & Master's Students) were found in this cluster. The program will focus on the reinforcement of the concepts worked on in the video for which an extension and specification of the concepts will be made.

Figure 5. Cluster analysis of the fixation parameters in the three groups (Experience university students, University professors, and University students). Please click here to view a larger version of this figure.

SUPPLEMENTARY FILES

The authorizations for recording the video from which images have been used to reference the observation of functional abilities in children with learning difficulties are attached.

Discussion

The research results indicated that the average fixation duration on the relevant stimuli was longer among participants with previous knowledge. Likewise, the focus of attention on this group is on the middle points of information (proximal and distal)⁷. The results of this study have revealed differences in the way participants processed the information. Furthermore, their processing was not always linked to the initial grouping (University of Experience Students, University Teachers and Graduate & Master's Students). These differences were found with respect to the analysis of the participants' visual position on the X-axis in the fixations on the images presented in the learning video. In cluster 2, the highest frequencies of Position X fixation, Position Y fixation, Average Pupil Diameter fixation, X- and Y-axis fixation dispersion were found. This fact coincides with the fact that the members of cluster 2 were themselves members of the University Teachers group, had previous knowledge of the subject, and obtained the highest score in the results verification test. Likewise, it is relevant to point out that most of the participants in the study (85.43%), regardless of their group of origin, were in cluster 3 where they obtained the lowest results in the verification test.

Another relevant aspect of this protocol is the presentation of the information in a video with SRL. This form of presentation, on the one hand, guides and focuses the attention and on the other hand minimizes the individual actions of each participant, which is to say that it unifies the way of accessing the information. Where necessary, it is an aspect that can compensate possible deficits or difficulties in processing. Evidence of this statement is that no significant differences were detected in learning outcomes between the three groups, with the average performance interval [3.60, 4.86] established at 5. Significant differences were only found in the parameters of Diversion Duration of the processing (a parameter used to analyze the input, the permanence and the output time for each stimulus inserted in each AOI) and of Average Fixation Duration (this parameter refers to longer fixations, which indicates that the participant spends more time analyzing and interpreting the content of the information within the different AOIs) at the initial moment of the processing but not at the end of it. The longest times in these parameters were detected among the group of the Graduate & Master's Students and the University of Experience Students in this study. These results are supported by the conclusions of other studies^4,12,15. Finally, this protocol can be applied to study differences during information processing, depending on the coding phase of the student.

Based on the above, the first conclusion is that the use of the eye-tracking methodology during the completion of the tasks provided useful data for the study of information processing^1,2. Likewise, the data extracted from eye-tracking technology and analyzed with unsupervised learning techniques (clustering) facilitated the knowledge of clusters according to the pre-determined parameters^11,15. This aspect is very relevant for the study of information processing in each participant and for the proposal concerning personalized educational responses^{3,4,5,6,22,23} which is expected to lead to more effective learning²³.

It is also important to point out that the experimental results reaffirmed the results of other research with respect to: differences in processing according to novice-expert variables and participant age, and the use of SRL materials that minimize the effects of these variables and increases the expected performance of the participants^7,8,9,10.

However, it is necessary to treat any generalization of these results with caution, since convenience sampling was applied, and the work was focused on specific content relating to Art History. Therefore, the sample will be extended in future research and the findings will be checked in other disciplines.

As indicated in the introduction, the presentation of information in a multimodal way through various information channels (auditory, visual or both) with SRL methodology, together with the use of eye-tracking technology and machine-learning techniques, is key to understanding the way in which the participants process the information so as to offer personalized learning design according to the educational needs of each user and, in consequence, promote successful learning and its development among all students^29,30.

In sum, the use of the eye-tracking technique is not a methodology of usual application in education frameworks due fundamentally to cost factors from material to personal resources. However, its use is starting to increase little by little and it is important to see its usual use in the observation of tasks in autoregulated learning. The advantage of this eye-tracking technology is that it permits the recording of the learner's interaction with the task which has got advantages of reliability and validity, in this case, on the simple observation of the learning process. In addition, the eye-tracking methodology offers different techniques of registered information visualization and this information in data bases can also be analyzed with computer programs or more powerful tools. Therefore, it opens a wide variety of possibilities for investigation in natural contexts.

Materials

Name	Company	Catalog Number	Comments
iViewer XTM	iViewer
SMI Experimenter Center 3.0	SMI
SMI Be Gaze	SMI