Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties

Summary

The current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs.

Abstract

Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics. Each of the disorders making up the LDs involve different deficits; however, some commonalities can be found within that heterogeneity, such in terms of learning self-regulation and metacognition. Unlike in early ages and later educational levels, there are hardly any evidence-based evaluation protocols for adults with LDs. LDs influence academic performance but also have serious consequences in professional, social, and family contexts. In response to this, the current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs. The assessment is carried out through analysis of the on-line learning process using a variety methods, techniques, and sensors (e.g., eye tracking, facial expressions of emotion, physiological responses, concurrent verbalizations, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This theoretically-driven and empirically-based guideline aims to provide an accurate assessment of LDs in adulthood in order to design effective prevention and intervention proposals.

Introduction

Specific learning disorders (SLDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics^1,2. There are different estimations of prevalence rates depending on the age, language and culture analyzed but they are between 5% and 15%^1,3. Within the global category of neurodevelopmental disorders in the Diagnostic and Statistical Manual of Mental Disorders (5^th Ed.)¹, it is also necessary to focus on the incidence of Attention-Deficit/Hyperactivity Disorder (hereinafter ADHD) as it is a common disorder that has given rise to various controversies about how to approach it in recent years. Based on the DSM-5¹, it can be defined as a pattern of persistent behaviors of inattention and/or hyperactivity-impulsivity. Likewise, autism spectrum disorder (hereinafter ASD) is a category in the same manual that includes students who present neurodevelopmental disorders as a result of multifactorial dysfunctions of the central nervous system, which result in qualitative dysfunctions in three fundamental areas of the development of the person: social interaction, communication and interests and behaviors^1,2.

On these lines, a new concept has emerged moving away from the sense of deficit and offering a more positive approach to these disorders to be consistent with current ideas of neurodevelopmental difficulties as highly coexistent and overlapping⁴. From these new models, it is understood that the skills involved in high-level cognitive processes, which allow managing and regulating one's behavior in order to achieve a desired goal, are crucial for self-regulation and, therefore, for activities of daily living, including the academic ones⁵. In the context of adulthood, neurodiversity has evolved to include various types of difficulties, including ADHD and ASD, as well as dyslexia, dyspraxia, and/or dyscalculia. Accordingly, we are approaching this neurodiversity from a broad conception of learning difficulties (LDs). The increase in students with this diversity enrolled in postsecondary education is well documented and is due, in part, to the increase in high school graduation rates for students with disabilities⁶, but at the same time, there is less research about the learning process of these students than necessary⁷.

Each of the disorders approached in isolation involve different deficits and manifestations; however, some commonality can be found within that heterogeneity in terms of LD, such as metacognitive, self-regulatory, and emotional malfunctioning^8,9,10,11. Three fundamental foundations in the literature of learning in general, and LDs in particular, that represent the basis of successful learning and play an essential role in these well-known difficulties at the academic level¹². As well as this, other approaches understand that there could be a certain commonality between deficits in executive functions, such as problems in automatic processing or working memory, that occur in different disorders such as ADHD and reading disorders¹³ or ADHD and ASD⁵. However, there is still work to be done in this field, since not all studies reach the same conclusions about these points in common in relation to executive functions. It could be due to the variations presented by the samples from which the studies are based and the evaluation procedures of the executive functions used in the investigations^5,14.

In educational terms, this diverse mix affects not only the quality of learning, due to the fundamental nature of the affected functions, but also phenomena such as school dropout, change of degree, etc., with economic implications for governments and universities¹⁵. The dropout rate for students with LDs is higher than for students in the general population¹⁶ but also higher than the dropout rates for any other category of psychological disabilities except for those students with emotional disturbances¹⁷. In contrast, the number of students with LDs who are accessing post-compulsory education (vocational training, college, etc.) is increasing¹⁵, specifically in higher education^19,20,21,22. Moreover, one might well assume that there are many more students with LD than those who officially pass through student services and typically make up the prevalence statistics²³.

These difficulties are not always detected during childhood, especially in adults born before these disorders were considered in the regular academic system, and the symptoms of these disorders persist throughout people’s lives and cause difficulties in work, education and personal lives²⁴. Research has shown that although people might overcome some of their difficulties, most continue to exhibit struggles with learning during adulthood and their persistence is still problematic at those higher educational levels²⁵.

Paradoxically, unlike in previous educational levels and earlier ages, there are hardly any evidence-based instruments or evaluation protocols for adults with LDs. Despite the proliferation of diagnostic tools to evaluate LDs during childhood, the availability of valid, reliable instruments and methodologies for the adult population is significantly limited²⁴. A recent literature review about learning disabilities in higher education found that most of the information collected in this regard is done through interviews, and only occasionally are self-report questionnaires used²⁶. Self-report methodology and interviews, although valuable, are not enough to accurately assess metacognitive, self-regulation, and emotional skills processes, in fact, among others, because of the process nature. The importance of scales and interview methodology for measuring those processes is undeniable^27,28, but so too are the associated problems of validity²⁹ and incongruence with other innovative methods of assessment³⁰. An additional problem in the detection of LDs is the bias in the diagnosis of the disorder due to the absence of comprehensive assessment protocols. The fact that professionals do not have a reference protocol based on objective variables is frequently causing many false positive and false negative cases of LDs³¹.

In response to both scarcity of instruments for adults and the need to improve existing methodology, the current study proposes a multimodal evaluation protocol focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs. In line with the current literature, we propose a move toward integrative and multichannel measurement^32,33. The assessment is carried out through an analysis of the on-line learning process using several methods, techniques, and sensors (e.g., hypermedia learning environment, virtual reality, eye tracking, facial expressions of emotion, physiological responses, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This mixed methodology provides evidence of the deployment of target processes before, during, and after learning that can be triangulated to enhance the understanding of how students learn and where the problem lies, if there is one³⁴.

The evaluation protocol is carried out over two sessions. The sessions can be done in one sitting or may need partial applications depending on the person. The first is focused on the detection or confirmation of LDs and what specific kind of disorder we are facing, and the second is designed to go into the metacognitive, self-regulation, and emotional processes of each individual case in depth.

Session 1 is intended to be a diagnostic or confirmation assessment of the participant’s learning disabilities: SLD, ADHD and/or ASD (high functioning) to determine what type of specific problems the participants have. This assessment is essential for two reasons. 1) Adults with Learning Disabilities rarely have accurate information about their dysfunctional behavior. Some of them suspect that they have a LD but have never been evaluated. Others may have been assessed when they were children but do not have any reports or further information. 2) There may be discrepancies with previous diagnoses (e.g., a previous dyslexia diagnosis as opposed to a current diagnosis of attention deficit and slow processing speed; previous ASD diagnosis in contrast to current limited intellectual ability, etc.). The participant is interviewed, and questionnaires and standardized tests are applied. This session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context in different offices of a Spanish Psychology Faculty. The session begins with a structured interview that collects biographical information along with the presence of symptoms related to SLDs that are referred to in the DSM-5¹. Following that, the reference intellectual ability test WAIS-IV³⁵ is used in case of exclusion criterion implementation and because it provides very valuable information for learning difficulties from the scales “work memory” and “processing speed”³⁶. Additionally, the PROLEC SE-Revised Test³⁷ is extensively used to evaluate reading disabilities (lexical, semantic and/or syntactic processes of reading), one of the most prevalent and disabling difficulties for learning in current academic contexts, which overlaps with other disorders such as ADHD³⁸. This evaluation collects reading accuracy, speed and fluency along with reading disabilities, and more importantly, in which reading process the failure occurs³⁷ (this test has been evaluated with pre-university students. Currently, there are no tests in Spain that are adapted to the general adult population, so this test was selected because it is the closest to the target population). Then, we screen symptoms of ADHD through the World Health Organization Adult ADHD Self-Report Scale (ASRS)³⁹ and refine the evaluation of this disorder, introducing multimodality with a cutting-edge virtual reality continuous performance test for the evaluation of attentional processes and working memory in adults, the Nesplora Aquarium^31,40. This test is a very useful tool when diagnosing ADHD in adults and adolescents over 16 years old in an ecological scenario, providing objective, reliable data. It evaluates selective and sustained attention, impulsivity, reaction time, auditory and visual attention, perseverance, quality of attentional focus, motor activity, work memory and cost of change of task. Additionally, along with the WAIS-IV³⁵ as a whole for collecting information about the participant’s intellectual ability, we pay special attention to the scales “work memory” and “processing speed” because they are related to learning difficulties and the results of these scales are used in the final decision. Finally, we include the Autism Spectrum Quotient (AQ-Short)⁴¹ in the protocol, the short version of the reliable AQ-Adult from Baron-Cohen, Wheelwright, Skinner, Martin and Clubley⁴².

Session 2 focuses on a multimodal assessment of the participant’s learning process. The key to understanding complex learning lies in understanding the deployment of students’ cognitive, metacognitive, motivational, and affective processes⁴³. To that end, participants work with MetaTutor, where the use of metacognitive and cognitive strategies deployed are observed while they are learning. MetaTutor is a hypermedia learning environment that is designed to detect, model, trace, and foster students’ self-regulated learning while learning different science topic⁴⁴. The design of MetaTutor is based on extensive research by Azevedo and colleagues^43,45,46,47 and belongs to a new trend in the measurement of SRL, the so called third wave, which is characterized by combined use of measurement and advanced learning technologies³³. The use of MetaTutor also provides multimodal trace data, incorporating measures such as, eye tracking, emotional physiological responses (galvanic skin response (GSR) and facial expressions of emotions)⁴⁸, log-data and questionnaires. All these measures are combined to reach a deeper understanding of the participants SRL and metacognition.

Eye tracking provides an understanding of what attracts immediate attention, which target elements are ignored, in which order elements are noticed, or how elements compare to others; electrodermal activity lets us know how emotional arousal changes in response to the environment; facial-emotion-recognition allows the automatic recognition and analysis of facial expressions; and data logging collects and stores the student´s interaction with the learning environment for further analysis. Concerning the questionnaires, the Mini International Personality Item Pool⁴⁹ informs about a range of activities and thoughts that people experience in everyday life assessing each of the five major personality traits (extraversion, agreeableness, conscientiousness, neuroticism and openness). The Connotative Aspects of Epistemological Beliefs⁵⁰ provides information about participants’ beliefs about knowledge. The Rosenberg Self-esteem scale shows how the participants feel about themselves overall⁵¹. The Emotion Regulation Questionnaire⁵² provides information about participants’ emotion regulation. The Achievement Emotions Questionnaire (AEQ)⁵³ informs about emotions typically experienced at university.

In short, assessing LDs during adulthood is particularly difficult. Education and experience allow many adults to compensate for their deficits and later show undifferentiated or masked symptoms, on which scientific knowledge is still scarce. Taking into account the critical research gap that arises, this current work aims to ensure theoretically-driven, empirically-based guidelines for accurate assessment of LDs during adulthood in order to design effective prevention and intervention actions.

To help readers decide whether the method described is appropriate or not, it is necessary to specify that the protocol is not suitable for people with intellectual disabilities because their diagnosis invalidates the diagnosis of learning difficulties. In addition, due to the singularities of the equipment used and the format of showing the learning content, it is still not possible to evaluate people with motor disabilities (upper limbs, neck and/or face), hearing or visual impairment. Nor would it be suitable for participants with severe psychiatric disorders. It would require the use of drugs that could alter information processing or the physiological expression of emotions.

Protocol

The research ethics committee of the Principality of Asturias and the University of Oviedo approved this protocol.

1. Session 1: diagnosis assessment

NOTE: In this session of the protocol, evaluation tests from different publishers are used, which have their own specific application and interpretation manuals. Since these tests, or other similar ones, are widely known by the scientific community in the field of psychology and education, the procedure to apply them is not detailed step by step (for example, given the aim of this paper, it does not make sense to detail each step of the WAIS-IV³⁵ application).

1. Informed consent
   1. Explain to the participants the ethical and confidentiality aspects of the research and ask them to acknowledge and sign the individual informed consent.
2. Structured interview
   1. Explain the following instructions to the participant: "Now, I´m going to interview you in order to get important information about your life and academic issues. There are open and closed questions but you can interrupt me whenever you want. Please, let me know if you need me to clarify any point. After this initial interview, I may ask you to do some evaluation tests and questionnaires. I will tell you the specific instructions for each one. Are you ready?"
   2. Collect the biographical information along with the presence of symptoms related to SLD and exclusion criteria that are referred in the DSM-5¹ following the interview script (see Supplemental File A ).
3. First decision point in relation to the structured interview (exclusion criteria)
   1. Finish the assessment if the participant meets the initial exclusion criteria, that is, they explain that they have a motor disability (upper segments), sensory disability (visual or auditory), a diagnosis of intellectual disability or a serious mental disorder.
   2. Continue the assessment if it seems that the participant has or thinks he/she has an SLD and does not meet exclusion criteria.
4. Intellectual ability
   1. Apply the WAIS-IV³⁵ test to collect information about participant’s intellectual ability following the instructions in the manual.
5. Second decision point in relation to intellectual ability (exclusion criteria)
   1. Finish the assessment if the participant does not understand the instructions of the test, if cannot be evaluated, or they have an IQ of less than 70.
   2. Continue the assessment if the person has normal or limited intellectual ability.
      NOTE: The limit of the IQ accepted in the present study has been set as a score of over 70.
6. ADHD
   1. Ask the participant to complete the six items of the Self-reported Screening Questionnaire of the Adult-v1.1. (ASRS³⁹) of the World Health Organization (WHO) International Composed Diagnostic Interview.
      NOTE: This questionnaire provides information on the presence of symptoms related to ADHD that are referred to in the DSM-IV⁵⁴.
   2. Apply the Nesplora Aquarium test⁴⁰ if the participant scores 12 or more in the previous ASRS³⁶ questionnaire.
7. Reading difficulties
   1. Apply the PROLEC SE-R Screening Test of reading difficulties³⁷ follow the instructions in the manual.
8. Autism spectrum disorder (level 1)
   1. Ask the participant to complete the 28 items of the Autism Spectrum Quotient (AQ-Short) questionnaire from Hoekstra et al.⁴¹
      NOTE: This questionnaire provides information on the presence of symptoms related to social behavior, social skills, routine, switching, imagination and numbers/patterns.
9. Analyze the results.
   1. Analyze each participant’s interview, questionnaires and test results and decide if they have significant learning difficulties or not or are at risk of having them.
      NOTE: Two members of the expert committee (the evaluator and another member of the research team) analyze each participant’s learning profile and decide if they is a student with SLD, ADHD and/or ASD or not or are at risk of having them. No test can substitute the expert´s judgment.
10. Final decision point
    1. Finish the assessment if the participant is clearly not a student with learning difficulties.
    2. Continue the assessment if the participant is a person with LDs (or at risk) and go to Session 2.

2. Session 2: multimodal assessment

NOTE: Session 2 must be done between 1 and 7 days after Session 1.

1. Prepare the participant.
   1. Remind the participants that the session lasts approximately 2 hours, and that they are going to complete some questionnaires and tasks in the MetaTutor learning environment while some devices are recording their performance throughout the session.
   2. Ask the participants tie back their hair, clear their neck, remove their glasses and remove chewing gum if applicable.
      NOTE: If the participant is wearing glasses, has long hair or bangs that cover part of their face, the eye tracker will not be able to read their eyes movements.
   3. Introduce MetaTutor to the participants. Explain that the objective of the session is to autonomously learn about the circulatory system using the tool.
   4. Make sure the speakers are connected and working.
      NOTE: The participant can also use headphones if preferred.
2. Galvanic skin response preparation and calibration
   NOTE: Remember that there are many types of GSRs manufactured by different companies. Use it according to the supplier's specifications.
   1. Clean the GSR and the participant´s fingers with alcohol.
   2. Put the finger/wristband GSR sensors on the index and ring fingers with the connectors on the fingertip side or according to the manufacturer's instructions.
   3. Ask the participant to rest their hand on the table quietly and try to relax for 5 min.
   4. Open the software in the computer.
   5. Make sure the registration graph is working. Check the registration graph is registering.
   6. Click Run experiment > Rate 10 per second > Duration > 10 > Minute. Record the information for ten minutes to establish the baseline.
      NOTE: Rate 10 per second means the frequency with which measures are taken.
   7. Minimize the screen.
   8. Continue with the calibration of other devices, and after 10 minutes save the information in a .csv file.
3. Eye tracking and webcam preparation and calibration
   NOTE: Remember that there are many types of eye tracking and webcam manufactured by different companies. Use them according to the supplier's specifications.
   1. Open the software in the side laptop and in the computer.
      NOTE: The eye movements are captured on the PC the participant is working on, but the data is recorded on the side laptop. In addition, in the side laptop, the experimenter can see the movements that the participant is making and correct the participant’s position if necessary.
   2. Indicate which session will be recorded (Metatutor in this case) and the participant’s registration data: File > Recent Experiment > Metatutor > Include Registration data of the participant > OK.
   3. Check that the two computers are connected to each other and that the eye tracking infrared lights are on and ready to capture the movement of the eyes.
   4. Adjust the webcam on the computer to the participant’s position.
   5. Ask the participant to sit facing forward and be as neutral as possible, although it is expected that their facial expressions will vary during the learning session.
      NOTE: During the learning session a video of the participant´s face is recorded with the webcam which is later analyzed using a desktop app⁵⁵.
   6. Ask the participant to be still and to stare at the different points of the screen with their nose put in line with/slightly over the edge of the desk (at 90°).
   7. Click Record > Write the registration data of the participant > Ok to start the calibration process.
   8. Ask the participant to press the space bar and follow the points on the screen with their eyes.
   9. Make sure that the participant’s eyes, when looking at the screen, are centered before moving on to the next step, using the side laptop to check this information.
      NOTE: The participant's gaze is centered when the movements of their eyes are registered on the side laptop screen with two white circles. When the gaze leaves the registration area, the software warns with yellow arrows (if slightly deviated), with red arrows (if deviated a lot) or without white circles (if not registering). The path of the movement of the eyes is reflected with a yellow light (attentional focus) and the track through the screen with a green line.
   10. Ask the participants to avoid touching their face or resting their head in their hands as much as possible.
   11. Minimize the screen.
4. Multimodal tracking of the learning session
   1. Maximize the GSR screen and click Run experiment > Rate 10 per second > Duration > 5 > hours > Record and minimize the screen again.
   2. Maximize the eye tracking and webcam screen, make sure the software is working correctly, click Record on the computer and on the side laptop to register and record the session and minimize the screen again.
      NOTE: Once the devices have been calibrated, do not forget to start recording the evaluation session in each of them. From this point, the entire participant interaction with the learning tool will be recorded until the end of the session.
5. Questionnaires and learning session in MetaTutor
   1. Open the software in the PC and complete the participant’s registration data. Complete ID > Experimenter > Day > Questionnaires yes > Continue.
      NOTE: All the logs will be registered during the session in a file-data log.
   2. Explain to the participant that they must follow the instructions in the tool and that they will only be interacting with the computer during the learning session. Explain that the researcher will be in the next room in case anything happens.
      1. Ask the participant for sociodemographic and academic information. Complete Name > Gender > Age > Ethnic group > Educational level > University > Degree > GPA > Information about biology courses taken if applicable > Continue. Before clicking Continue, explain to the participants that they must follow all the instructions that the tool will give them. Also, that they will only interact with the computer during the learning session.
      2. Ask the participant to complete some questionnaires.
         NOTE: The participant has to complete five metacognitive and self-regulated learning questionnaires: a) The Mini International Personality Item Pool⁴⁹; b) The Connotative Aspects of Epistemological Beliefs⁵⁰; c) The Rosenberg Self-esteem Scale⁵¹; d) The Emotion Regulation Questionnaire⁵²; e) The Achievement Emotions Questionnaire (AEQ)⁵³ and one questionnaire about general knowledge about the circulatory system.
      3. Show the participant the interface of MetaTutor and its different parts.
         1. Explain the participant that the content area is where the learning content is displayed throughout the session in text form.
         2. Show the participant that they can navigate through a table of contents at the side of the screen to go to different pages.
         3. Show the participant that the overall learning goal is displayed at the top of the screen during the session.
         4. Show the participant that the sub-goals learners set are displayed at the top in the middle of the screen, and they can manage sub-goals or prioritize them here.
         5. Show the participant that there is a timer located at the top left corner of the screen displays the amount of time remaining in the session.
         6. Show the participant the list of self-regulating processes, which are displayed in a palette on the right hand side of the screen, and the participant can click on them throughout the session to deploy planning, monitoring and learning strategies.
         7. Show the participant the static images relevant to content pages are displayed beside the text to help learners coordinate information from different sources.
         8. Show the participant the text entered on the keyboard and how students´ interactions with agents are displayed and recorded in this part of the interface.
         9. Show the participant the four artificial agents who help students in their learning throughout the session.
            NOTE: These agents are Gavin the Guide, Pam the Planner, Mary the Monitor, and Sam the Strategizer.
      4. Ask the participant to click Start to begin the learning session whenever they are ready.
         NOTE: The participant interacts with the tool.
      5. Once the session is finished, ask the participant to complete the knowledge questionnaire again.

3. Logoff

1. At the end of the session save the recorded data from GSR, eye tracking/webcam and Metatutor along with the registration data of the participant. Extract the data in a .csv file for easier use.
2. Remove the GSR sensors from the participant's hand and clean the galvanic sensors with alcohol again.
3. Thank the participants for their collaboration and say goodbye.

4. Analysis of learning difficulties

1. Analyze each participant’s learning performance based on the different reports produced (see Results section) to obtain a multimodal profile.
   NOTE: At least two members of the expert committee analyze each participant’s learning process. Although the evaluation can be done exhaustively using new instruments and tools, no report can replace the expert's judgment.

Representative Results

This section illustrates the representative results obtained from the protocol, including an example of conjoint results of Session 1 and an example of each source of information from Session 2.

The results about disorders are collected in Session 1 through diagnostic tests taking into account the procedures and cut-off points specified for the diagnostic assessment of participants’ learning difficulties (SLD, ADHD, and ASD). The expert committee decides whether the participant has learning disabilities or is at risk of having them or not (see an example of decision making in Figure 1). If the participant exhibits learning disabilities and takes part in Session 2, data from alternative sources are collected.

During Session 2 the protocol collects results from five different sources: participants´ GSR, face emotions, eye-movements, questionnaires and log-data.

Firstly, we obtain a measure of the GSR as an indication of emotional arousal during learning session (calm/excited)⁵⁶. Learning disabilities are linked to anxiety in adults, and several studies have found that students with learning disabilities from first grade to university report higher anxiety symptoms, acting as a factor in decreased performance^57,58,59. However, there is no one-to-one relationship between understanding and remediation; every case needs to be analyzed individually by the expert committee taking into account each participant’s specific baseline. Figure 2 shows two paradigmatic cases that can show us whether anxiety regulation is a key point for intervention.

Secondly, we obtain a recording of the participant’s face throughout the session that show us the different emotions they were feeling during the learning process to consider the theoretical relationship with metacognition and self-regulation. There is a variety of facial-emotion-recognition software to gather that information. In the current protocol, we use a tool⁵⁵, which includes emotion recognition, returning the confidence across a set of emotions for each face in the video (disgust, fear, anger, happiness, contempt, neutral, sadness, and surprise). These emotions are understood to be cross-culturally and universally communicated with specific facial expressions⁶⁰. Participants tended to experience all the detected emotions during the session, but we can obtain a general index for each giving information about the general trend. Positive activating emotions such as happiness, surprise and enjoyment, are thought to promote both intrinsic and extrinsic motivation, facilitating use of flexible learning strategies, and fostering self-regulation. Conversely, negative deactivating emotions, such as boredom and sadness, are posited to uniformly reduce motivation and the effortful processing of information, producing negative effects on learning outcomes. For neutral deactivating and negative activating emotions, such as anger, fear, contempt, and disgust, the relationships are presumed to be more complex. Specifically, anger and fear can undermine intrinsic motivation, but can induce strong extrinsic motivation to invest effort to avoid failure, meaning that the effects on students’ learning need not be negative⁵³ (see Figure 3). The results indicate the degree of coincidence with one of the analyzed emotions, assigning values between 0 and 1 to each of them.

Thirdly, we use data from eye-tracking. Eye-trackers capture gaze information in terms of fixations, and saccades (Figure 4). In the current protocol, we are interested in analyzing fixations, particularly the proportion of fixation time and pattern of fixations. For that purpose, we defined seven Areas of interest (AOIs) in the MetaTutor interface for self-regulation assessment (labeled with rectangles in Figure 5): AOI1 Timer, AOI2 Goal and Sub goals, AOI3 Agent/avatar for scaffolding, AOI4 Table of Contents, AOI5 Text Content, AOI6 Image Content, AOI7 Learning Strategies Palette.

In terms of assessment for concise intervention guidance, we can infer the following.

Fixations in AOI1 denote time management and/or resource management strategies. Reduced or massive fixations in AOI1 denote incorrect time management skills. It should be checked promptly.

Fixations in AOI2 denote planning, setting and prioritizing goals and sub-goals. Previous studies show that this particular AOI, along with the AOI7, is especially important for assessing learning with MetaTutor⁶¹. As this information is concise, short and visual, the proportion of fixations should not be very high (Figure 6).

Fixations in AOI3 Agent show that the participant is taking advantage of the prompts and feedback which the agents provide during the interaction in response to participants’ goals, behaviors, self-evaluations, and progress. It is worth noting that a lack of fixations on the Agent AOI must be considered carefully, because learners may not always need to look at an agent to process its audio prompts and feedback⁶¹. This AOI should be checked occasionally. Avatars do not speak frequently, so there should be a small percentage of fixations compared to other areas, but it would reflect that they have established an interaction with the agent (Figure 6).

Fixations in AOI4 and/or transitions between text and image/graph (AOI5 and AOI6) point to participants’ strategy-use for coordinating informational sources (COIS), associated with conceptual gains⁴⁵. The length of fixations on texts and images indicate integration processes contributing to accurate mental representations of the information presented⁶². COIS are operationalized as a sequence of two transitions between eye fixations on text and image/graph areas (e.g., text/graph/text). AOI4 should be checked with some frequency. As the information is clear, short and visual, the proportion of fixations should not be very high. The highest proportion of fixations should be in AOI5 and AOI6. The subject should spend most of their time reviewing the content (i.e. the written texts) and spend a notable amount of time on the images and graphs to coordinate and integrate both sources of knowledge (Figure 6).

Fixations on AOI7 indicate the use of cognitive strategies (taking notes, writing a summary, making an inference) and metacognitive strategies (activating prior knowledge, evaluating content relevance, assessing understanding and knowledge)⁶³. It is reasonable for the participant to review the available resources or learning strategies with some frequency (Figure 6).

For the subsequent analysis, it is necessary to focus on data related to students interacting with MetaTutor, excluding the parts of the interaction during which participants watch system tutorials. The collected data can be noisy and needs expert validation. The main source of noise is due to participants looking away from the screen, which the eye-tracker interprets as invalid data; in this case, it is advisable to remove the corresponding segments from gaze data. Figure 6 shows a participant with metacognitive malfunctioning and a participant with an adaptive use of strategies at this level.

Fourthly, questionnaires are analyzed together with the rest of the information and are scored according to the authors’ instructions. They provide data at the participant level of self-esteem and emotional regulation. A favorable level of self-esteem or correct emotional regulation strategies facilitates learning processes⁶⁴. To see examples of interpretation (Figure 7).

Finally, all interactions of learners with content, agents, and the learning environment are recorded in logs for further detailed analysis following the scheme in Figure 8. The MetaTutor log data provides us with a wide range of possibilities for determining, among other things, the number of times that learners deployed self-regulatory learning strategies (e.g., note-taking, summaries, monitoring progress toward goal, content evaluation, judgments of learning, feelings of knowing, planning, prior knowledge activation, etc.), whether these strategies were self or externally generated by the external scaffolding, and the time each participant spent viewing material in MetaTutor that was relevant/irrelevant to their current active sub-goal^65,66. Pattern Mining, Process Mining, Association Rules, and other potential approaches^67,68 would provide a measure of students’ use of cognitive and metacognitive monitoring and regulation throughout the learning session.

Figure 1. Example of making decision points of Session 1. This case shows a participant that has had learning problems since childhood, mostly in reading processes. The expert can see that these reading disabilities are more significant in lexical and syntactic processes (b). In addition, it is observed that the participant does not have any motor, sensory or mental disability. It is observed that the participant has a normal intellectual ability and is not at risk in relation to autism spectrum disorder or ADHD (a) omissions, commissions and reaction time, in visual and auditory channels, are less than 60, so are in the normal range). In this case, reading problems are detected and exclusion criteria are not observed, so it is considered that the participant has SLD due to reading disabilities. Please click here to view a larger version of this figure.

Figure 2. Results of a stable activation level and unstable activation level during learning session. This image represents the results of two participants. Participant A with stable activation levels and participant B with unstable activation levels during the learning session since the participant B line is more irregular and with many peaks. Please click here to view a larger version of this figure.

Figure 3. Image of emotion recognition. a) Example of neutral emotion; b) Example of sadness emotion; and c) Example of happiness emotion trend. In the yellow circle it is possible to see the emotion trend. Please click here to view a larger version of this figure.

Figure 4. Example showing transition data between text and graph (AOI5 and AOI6) during a MetaTutor learning session. Circles and lines represent areas of fixation and transitions between areas. Please click here to view a larger version of this figure.

Figure 5. Areas of interest (AOIs) of the MetaTutor interface for the self-regulation assessment: AOI1 to AOI7. AOI1 Timer, AOI2 Goal and Sub-goals, AOI3 Agent, AOI4 Table of Contents, AOI5 Text Content, AOI 6 Image Content, AOI7 Learning Strategies Palette. Please click here to view a larger version of this figure.

Figure 6. Proportion of fixations in the MetaTutor interface AOIs expressed as a percentage. a) Example of a participant deploying self-regulation malfunctioning; b) Example of a participant deploying self-regulatory behaviors. Proportion of fixations in each area (values between 0 and 1). a) Real data from a participant that spends more than 80% of the time reading the written text (AOI5) he underuses the resources designed to help him understand that content (AOI6); he hardly reviews the content scheme to check what he has already learned and what is left to learn (AOI4); neglects learning objectives and sub-goals (AOI2) and he rarely reviews the palette of learning strategies (AOI7). In addition, he does not monitor the time assigned to the task (AOI1) and ignores the avatars that try to help him (AOI3); b) Real data from a participant that spends half the time (50% approximately) reading the written text (AOI5) and frequently reviews the graph designed to help him to understand the content (AOI6). Although he spends most of his time on content, he reviews the content scheme frequently to check what he has learned and what he has left to learn (AOI4); he pays attention to learning objectives and sub-objectives (AOI2) to ensure that he is reaching them and he goes to the learning strategies palette (AOI7) when needed. In addition, he monitors the time without worrying too much about it (AOI1) and establishes some interaction with agents (AOI3). Please click here to view a larger version of this figure.

Figure 7. Example of interpretation of the questionnaires results. In graphic left) Rosenberg self-esteem scale⁵¹, higher scores indicate higher self-esteem (minimum = 10; maximum = 40). In graphic right), Emotion Regulation Questionnaire⁵², cognitive Reappraisal (minimum = 7; maximum = 42); Expressive Suppression (minimum = 4; maximum = 28). Higher scores indicate higher use of reappraisal or suppression strategies. Cognitive reappraisal is a form of change at the cognitive level that helps one to interpret a situation that provokes emotions in another way, thereby changing their emotional impact (using reappraisal strategies help one to think about negative situations and about some alternative construal to resolve them). Expressive suppression is a form of response modulation that involves inhibiting ongoing emotion-expressive behavior (recurrent users of suppression strategies should have less understanding of their moods, view them less favorably, and manage them less successfully). Please click here to view a larger version of this figure.

Figure 8. Log data processing. This image represents the management of log data. The system collects the raw interaction data between the student and MetaTutor, then performs data preprocessing to subsequently apply Learning Analytics and/or Data Mining technics for discovering, analyzing or visualizing the complete learning process. Please click here to view a larger version of this figure.

Discussion

The current protocol proposes a multimodal evaluation focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs.

Session 1 is essential because it is intended to be a diagnostic assessment of the participant’s learning disabilities. Note that this session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context. We use these tools in Spain, so researchers from other countries should select tests adapted to their population. The significance of the method with respect to existing methods is that many of the scales for ADHD, SLDs and ASD were designed for use in children, with neuropsychological testing and neuroimaging being the better, but less realistic, alternative to this paucity of instruments²⁴. Additionally, all the aforementioned disabilities are usually evaluated through their specific symptoms in isolation, without taking into account well-known commonalities found in LDs, such as metacognitive, self-regulatory, and emotional malfunctioning. In any case, most of the knowledge about metacognition, self-regulation and emotions is based on self-reported data at early or adult ages. However, self-reports of any kind are vulnerable to various types of biases⁶⁹ and several times no correlations between physiological and self-reported data have been found in LD samples⁷⁰.

For this reason, Session 2 of the protocol is critical. It focuses on the core processes of learning (metacognitive, self-regulation, and emotional behavior), the significance of the method compared to alternative methods is that it is a multimodal assessment of the participant’s learning process providing multichannel trace data. The tool that makes the integration of all those sources of information possible is MetaTutor⁴³, a metacognitive tool based on advanced learning technology and one of the best representatives and most well-known lines of research of the so-called third wave of self-regulation measurement³³.

Regarding galvanic skin responses, the majority of psychophysiological studies of LD subjects have focused on one of three related topics: arousal, orienting, and attention. In this protocol, arousal provides a unique framework for understanding emotion and cognition that cannot be provided by static measures like self-reports⁷¹. With facial expressions, previous research has indicated that academic emotions are significantly related to students' motivation, learning strategies, cognitive resources, self-regulation, and academic achievement⁷². When it comes to eye movements, we know the value of gaze data in predicting student learning during interaction with MetaTutor⁶¹ and multiple researchers have suggested that the duration of fixations indicate deeper cognitive processing during learning⁷³. The questionnaires provide complementary information about participants’ performance during the learning session in MetaTutor, their perceptions of themselves as learners and their behavior when they learn. Finally, the log data is an additional source of information about participants’ self-regulatory processes. After the collection of raw data and data preprocessing, emerging Learning Analytics and Educational Data Mining techniques let us discover, analyze and visualize, or to put it another way, dive into the learning process^74,75,76.

This mixed methodology provides evidence of the deployment of target processes before, during, and after learning that can be triangulated to enhance our understanding of how adults with LDs learn and where problems lie.

This proposal is a protocol, which means a procedure and system of instruments, so it is advisable to remember that the proposed measures do not have the same value in isolation as they do when they form part of the whole, and therein lies the interest in this proposal. The objective is to converge those data streams, to understand how adults with LDs monitor and control their cognitive, metacognitive, and affective processes during learning.

Although this protocol is an effective toolbox for screening and diagnosis by the practicing psychologist, it is not without limitations. Diagnosis of adult LDs is particularly difficult. Education and experience allow many adults to compensate for their deficits and these adults subsequently show individual characteristics on testing²⁴. As the results indicate, it is difficult to provide accurate cut-off points from some of the data sources (e.g., GSR, log data, etc.) as a general rule in the target population.

Another challenge, rather than limitation, is about the complexity in dealing with the resulting complex, noisy, messy data, which needs the involvement of experts from different domains such as psychologists, physiologists, computer and educational scientists, etc. As recently noted by Azevedo and Gašević⁷⁷ we need to integrate a complex mosaic of theoretical models and frameworks from the psychological, educational, instructional, and computational sciences. In addition to this, instrumentation errors, internal and external validity, ecological validity versus experimental rigor, converging data channels, and inferences about process data are only some of the methodological issues that result from collecting multimodal multichannel data that researchers must address^77,78.

Nonetheless, the future direction of this methodology surpasses the goal of assessment, currently the possibility is open to use real-time multimodal multichannel data to design preventive interventions based on Adaptive Hypermedia Learning Environments⁷⁹ or provide learners with real-time, intelligent, adaptive scaffolding (modeling cognitive strategies, regulating metacognition via an artificial agent, prompting emotion regulation, introducing visualization tools to discover hidden processes, etc.)^77,80.

Finally, LDs should be tracked over their lifetimes; the longitudinal course of SLDs, ADHD and ASD and their long-term sequelae are only beginning to be explored²¹. We hope that widespread use of this theoretically-driven, empirically-based guideline will help to identify the population of adults with LDs and spur deeper understanding of these disorders in order to design effective prevention and intervention actions.

Materials

Name	Company	Catalog Number	Comments
AQUARIUM	Nesplora
Eye-tracker RED500 Systems	SensoMotoric Instruments GmbH
Face API	Microsoft
GSR NUL-217	NeuLog