Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques

María Consuelo Sáiz-Manzanares; Laura Alonso-Martínez; Alberto Calvo Rodríguez; Caroline Martin

doi:10.3791/63601

Behavior

Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques

Published: December 9, 2022 doi: 10.3791/63601

María Consuelo Sáiz-Manzanares¹, Laura Alonso-Martínez², Alberto Calvo Rodríguez³, Caroline Martin⁴

¹Department of Health Sciences, University of Burgos, ²Department of Education Sciences, University of Burgos, ³Department of Geography, History and Communication, University of Burgos, ⁴Department of Philology, University of Burgos

Summary

Here, we present a protocol for the behavioral analysis of a project-based learning methodology for health sciences students (20-56 years old). The protocol facilitates the comparison of the participants' performance in e-Learning versus blended-Learning (b-Learning) through a monitoring tool. The results are analyzed using Educational Data Mining and qualitative techniques.

Abstract

Academic leaders all over the world are encouraging the use of active methodologies in teaching, especially in higher education. The reason for this is that social changes are happening at an ever-increasing rate, and they require students and teachers to develop digital skills. This is especially significant for health sciences degrees, in which future graduates must have effective problem-solving skills. To respond to this challenge, the use of a project-based learning (PBL) methodology, together with various monitoring techniques based on the use of Educational Data Mining (EDM) and mixed methods, will provide teachers with information about the effectiveness of the methodology and guide the implementation of personalized educational responses.

This study provides a protocol for the application of the PBL methodology in e-Learning and blended-Learning (b-Learning) teaching modalities for health sciences students studying occupational therapy in higher education. In addition, statistical techniques for the analysis of covariance and unsupervised learning allow differences to be detected between the two teaching modalities, thus specifying their effectiveness in terms of a range of variables related to behavioral patterns, performance, and satisfaction. Data visualization also helps in understanding the qualitative aspects of the learning process. These data will help teachers to produce more effective proposals for the implementation of the PBL methodology based on the based on the context of the teaching-learning process. Therefore, this protocol offers many resources and materials to help teachers implement the PBL methodology in e-Learning and b-Learning teaching methods.

Introduction

Characteristics of the project-based learning methodology
Nowadays, professionals in various sectors face numerous (technical, sociopolitical, and economic) challenges arising from globalization in a variety of spheres, such as the environment¹. These issues influence the global spread of diseases, thus diminishing resources, increasing poverty, and necessitating the responsibility to create sustainable societies¹. Health experts have the potential and the skills to ensure the viability of communities by advancing care strategies and techniques on a large scale while contributing to the improvement of people's quality of life². The United Nations has highlighted the need to address these challenges with the active participation of all countries in its Sustainable Development Goals (SDGs) ¹. Health professionals' activities can be linked to many of the proposed goals. Therefore, those who train healthcare students must acquire pedagogical approaches that help students to gain the competencies that will allow them to face these new challenges¹^,².

Among the various existing pedagogical methodologies, project-based learning (PBL)³ stands out as a methodology that helps students develop the competencies they will need to safely and effectively deal with the future challenges of their professions³. PBL is an innovative pedagogical approach that overcomes the limitations of traditional methodologies by transforming the teaching-learning process to make students the protagonists of their own learning. This change helps students acquire problem-solving skills in real-life contexts⁴. PBL is based on the teacher producing research questions⁵ that are included in practical scenarios related to the discipline. The students have to solve the problems in collaborative groups. The final objective is the acquisition of knowledge (conceptual, procedural, and attitudinal) through group work in the resolution of the task or problem⁶.

This approach originated in the fields of education and constructivist psychology, and from those beginnings, it has been adapted to different disciplines⁶. However, implementing this methodology can be affected by a lack of educational resources and infrastructure, and because of this, a hybrid model is sometimes used that combines traditional teaching with PBL⁷^,⁸. However, these models have exhibited limitations in the learning experience¹. For this reason, it is important to develop pedagogical protocols that guide and facilitate the use and interpretation of these methodologies. In addition, in recent years, the PBL methodology has been implemented through virtual learning platforms - Learning Management Systems (LMS)⁹ - in what is called Online Project-Based Learning (OPBL)¹⁰.

Project-based learning methodology for university students in the health sciences field
The protocol is important for helping teachers to apply these methodologies and interpret the results arising from the teaching and learning process⁹. The protocol can be used from the beginning of schooling to higher education and can be tailored to different mental, physical, and socioeconomic conditions¹¹. The presented protocol focuses on working with university students in the health sciences. The teaching model is based on organizing learning around projects prepared by the teacher, who serves as an initial stimulus, and from this framework, the student becomes the main focus of their own learning¹¹. The approach is an active, comprehensive, and constructive way for students to acquire basic competencies (conceptual, procedural, and attitudinal) that are closely related to their future professional practice⁴^,¹².

Learning methods are combined to stimulate the acquisition of cognitive and metacognitive competencies related to conceptual and procedural knowledge⁴^,¹³^,¹⁴, all of which contribute to students developing critical thinking, increased motivation toward learning, and autonomy in decision-making and problem-solving¹³. However, one of the keys to the successful application of this methodology is that the projects the teacher prepares must be directly related to real practice, and the students must be given autonomy in how they approach the problems based on minimum initial guidelines⁴^,⁹. This means that the teachers need to clearly define the tools for evaluating the students' competencies and to give them guidance about the evaluation criteria that will be used to shape the PBL, as well as to assess how well the evaluation is done¹⁵^,¹⁶. Furthermore, students benefit from collaborating with their peers to learn to argue and be assertive through debate, thus developing communication and teamwork skills⁴^,¹⁶.

The main aim for university students is to "learn to learn" by dealing with the challenges the teacher sets to facilitate the acquisition of these competencies¹⁴ (which will later be implemented in the future working lives of the students). The results of using this methodology indicate that it encourages autonomy in learning and solving practical problems⁴. An added advantage of this methodology is that it is very effective in helping students acquire competencies, specifically in health sciences, in online teaching and blended-Learning (b-Learning)¹⁷^,¹⁸^,¹⁹. This has become especially important due to the COVID-19 pandemic²⁰^,²¹.

The common elements of PBL can be summed up as follows: (1) first, the concepts related to the project are explained to the students (in online teaching environments, flipped learning experiences can be used); (2) the project plan is defined and analyzed; (3) a review of the supporting theory is carried out, and a plan is drawn up to help understand the object of study; (4) objectives and questions are proposed for addressing the problem; (5) possible solutions are formulated, and the most viable options are evaluated; (6) these solutions are implemented; (7) the results are reported; (8) critical reflection is conducted, feedback is provided, and the process is evaluated, including suggesting new questions; and (9) the process is finished by presenting the work⁴^,⁶^,¹³.

It should also be kept in mind that not all students respond in the same way to this type of methodology. Students who are more actively involved in their learning, those who are more critical thinkers, and those who have better interpersonal relationship skills tend to achieve better results²². It is also possible that students who are used to traditional learning methodologies may feel frustrated. This is why students should be given clear working rules and a timetable for the implementation of the project phases from the beginning⁸. As noted above, PBL has been widely used within health sciences degrees, and there is extensive evidence supporting it in the literature¹⁸^,²³^,²⁴. However, some aspects that could be improved have been identified, which are related to clinical skills seminars and the dissemination of experience in the scientific community²⁵^,²⁶.

This methodology seems to have better results for degrees with a higher proportion of internships (for example, nursing vs. medicine, occupational therapy versus psychology or nutrition)³. In addition, recent studies have suggested implementing PBL methodology through joint training projects between different degrees to work on interaction in real contexts. The objective would be the acquisition of multidisciplinary and interdisciplinary competencies²⁷^,²⁸. However, to implement this type of methodology well, the teaching staff must be qualified to implement it, and the students must develop critical and divergent thinking with respect to clinical practice, especially in the case of health sciences³^,¹⁵^,¹⁸^,²⁹.

Another important aspect of the implementation is the need to evaluate student satisfaction with the PBL methodology training and analyze their ideas for improvement¹⁹^,³⁰^,³¹. More specifically, it has been reported that occupational therapy students have found this methodology essential for their learning as it allows them to learn how to solve similar problems to those they will face in their professional lives, which enhances their motivation³¹. Similarly, a longitudinal study that began in 1993-in which a hybrid model of PBL was evaluated in occupational therapy students-demonstrated that the students had very positive opinions of group learning for their future professional practice⁷. Another study on occupational therapy students found a high degree of satisfaction with the methodology. This is because students think that this way of learning helps them acquire knowledge through practice and enables them to respond appropriately in their future work¹³.

In summary, using the PBL methodology in health sciences courses is key to students' acquisition of professional competencies. Various studies have indicated the steps to follow to prioritize the interventions and the design of the materials to achieve effective, satisfactory learning. Therefore, it is very important to monitor students' learning processes to detect early or potential problems with learning and address them. Educational Data Mining (EDM) and mixed-methods analysis techniques should be used for monitoring, the essential fundamentals of which are discussed below.

Educational Data Mining techniques
When LMSs are used in either e-Learning or b-Learning teaching models, a series of logs or log files are generated that capture each of the student and teacher interactions. These logs are evidence of the learning behaviors and can be analyzed using EDM or Learning Analytics (LA)³² techniques. These techniques allow the teacher to learn what student interaction exists, how much interaction exists, and what is the quality of the interaction³³. In particular, EDM techniques make it easier to discover students' learning behavior patterns and their interactions⁴. EDM techniques can be used to study different profiles³⁵^,³⁶ (oriented toward students or oriented toward educators).

The final objective is to provide feedback for the instruction, evaluate the course content structure, analyze elements that have been effective in the learning processes, classify the type of students, and identify needs for the guidance and monitoring of learning. This helps in determining each student's most common learning patterns and the frequency of errors, which can then be used to tailor the most appropriate educational responses. One of the things that EDM techniques can be used for is monitoring students' learning processes to offer appropriate help via mentoring. EDM techniques include supervised learning techniques (prediction or classification techniques)³⁷ and unsupervised learning techniques³⁶ (clustering techniques)³⁸. Using EDM techniques in teaching processes that include active teaching methodologies, such as PBL, has been shown to be very effective for studying individual student behavior and student behavior in collaborative groups¹⁰.

The application of computer-assisted qualitative analysis techniques
In the last two decades, quantitative and qualitative techniques, or a combination of the two, have been applied in research in what has been called mixed methods³⁹. The use of mixed methods for data analysis is especially useful when dealing with complex problems of analysis, such as monitoring students' learning behaviors. The use of these methods allows for the transformation of qualitative data, such as student feedback from open-ended questions in a survey, into qualitative data, and the analysis of the data with different data visualization techniques⁴⁰. A variety of computer-assisted (or -aided) qualitative data analysis software (CAQDAS) can be used to facilitate data transformation and analysis⁴¹.

A summary of the procedure for teaching through the PBL methodology, the analysis of learning behaviors, the use of EDM, and computer-based qualitative analysis techniques is shown in Figure 1.

Figure 1: PBL work and the use of EDM and computer-based qualitative analysis techniques. Data collection and processing applying EDM and text mining techniques in e-Learning and b-Learning teaching environments. Abbreviations: PBL = project-based learning; EDM = Educational Data Mining; DB = database; e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.

Three research questions were posed in this study: RQ1: Are there significant differences in the learning outcomes and satisfaction of health sciences students studying occupational therapy depending on whether the PBL methodology is implemented via e-Learning versus b-Learning, considering the effects of students' prior knowledge? RQ2: Do the clusters of participants found match the learning outcomes, learning behaviors, and perceived satisfaction as a function of teaching modality (e-Learning vs. b-Learning)? RQ3: Are the students' suggestions for the improvement of the PBL methodology different depending on the teaching modality, e-Learning versus b-Learning?

The following protocol can be used by teachers in healthcare and can also be modified to work with students in other knowledge areas.

Subscription Required. Please recommend JoVE to your librarian.

Protocol

This protocol was executed in compliance with the procedural regulations of the Bioethics Committee of the University of Burgos (Spain) number IO 03/2022. Before participating, the respondents were made fully aware of the research objectives and provided their informed consent. They received no financial compensation for their participation.

1. Participant recruitment

Recruit adult participants aged between 20 years old and 56 years old from two groups (students and teachers) in higher education, specifically undergraduate students in occupational therapy.
NOTE: Table 1 shows the list of students disaggregated by teaching modality and by age and gender. The PBL methodology was used at two different periods during the COVID-19 pandemic: the state of emergency with full confinement (teaching had to be via e-Learning due to total confinement) and the state of emergency with partial confinement (only students who tested positive for COVID-19 or who had been in direct contact with someone testing positive were confined; in this case, b-Learning was applied).

Table 1: Sample characteristics. Sample characteristics (age, gender) in both groups (e-Learning teaching method vs. b-Learning teaching method). Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes); M age =Mean age; SD age=Standard deviation age. Please click here to download this Table.

2. Experimental procedure

Session 1: Collecting consents and briefing the participants
1. Collect informed consent, personal data, and background knowledge.
2. During the first week of teaching, inform the trainees about the objectives of the study and about data collection, processing, and storage. If the students agree to participate, ask them to sign an informed consent form.
  NOTE: Participation in this study was voluntary, and there was no financial compensation. This aspect ensures motivation toward the execution of the task free of economic conditioning factors.
Session 2: Assessing the students' prior knowledge
1. During the first week of teaching, ask the students to complete a questionnaire about their prior knowledge of the core subject concepts. Record the answers to the closed questions in a dichotomous manner or on a Likert-type scale from 1 to 5 (Table 2).
  NOTE: The prior knowledge questionnaire contains closed questions related to the methodology to be applied in the subject, which, in this case, includes personalized voice assistants to report course events, process-oriented automatic feedback systems, virtual laboratories, flipped classroom experiences, and PBL. The questionnaire was applied anonymously through the UBUVirtual learning platform, which is a Moodle environment (Modular Object-Oriented Dynamic Learning Environment) and was filled in by the students online. The questionnaire in this study had a Cronbach's alpha reliability index of α = 0.88 for the overall scale and α = 0.87-0.89 for each of the items if the item was removed. The questionnaire also includes an open-ended question about the student's motivation toward the subject.
Session 3: Informing the students about the project-based teaching methodology and the different resources to be used
1. Provide the students with a guide for how PBL will be delivered as well as rubrics for the evaluation of the project and the presentation of the project.
2. Inform the students about the advanced learning technology resources available (intelligent voice assistants, process-oriented personalized feedback, virtual laboratories, questionnaires with feedbackon the qualification, flipped classroom experiences,and PBL experiences).
  NOTE: Students have access to a document detailing the PBL methodology (see Supplemental Material Appendix 1) and two rubrics for carrying out the project and presenting it (see Supplemental Material Appendix 2).
  1. Ensure that, in the e-Learning group, students are assisted by an intelligent voice assistant that informs them about course events.
  2. In the b-Learning group, do not use this resource, but ask the participants to carry out the project in a physical simulation laboratory and not only virtually, as in the case of the e-learning group.
Session 4: Choice of case study
1. Organize the students into groups of two to five participants and ask each group to choose from a series of practical cases related to different developmental disorders (physical, psychological, or sensory) in children aged 0-6 years.
  NOTE: The case studies include data from the clinical history (an example can be found in the Supplemental Material Appendix 3).
Session 5: Description of the scope of the project intervention
1. Ask each group to prepare an introduction describing the type of service in which they will implement their project (they can choose between an intervention in the field of health, education, patient groups, or private services within the framework of early care).
  NOTE: The students elaborate the works in a creative way; some examples can be consulted in the Supplemental Material Appendix 4.
Session 6: Description of the professionals working in the intervention area
1. Ask each group to describe the role of the professionals who will intervene, as well as the relationship structure that will be applied, encouraging interdisciplinary working.
Session 7: Description of the intervention case study
1. Ask each group to describe the characteristics of the pathology or developmental disorder.
Session 8: Creation of the intervention program
1. Ask each group to prepare various phases within the intervention program that address the following elements: an initial user assessment, the proposed evaluation indicators based on the results from the initial assessment, the proposed intervention procedure to achieve the development of skills or behaviors in children, the materials required for the intervention, a proposal of the generalization activities, and a plan for the follow-up for the intervention.
  1. In virtual laboratories, have the teacher help the students in the e-Learning group.
    NOTE. For an example of the help provided to the e-Learning students in virtual laboratories, see Supplemental Material Appendix 5.
  2. Carry out this part of the project via practical classes in the simulation center for the b-Learning group.
Session 9: Production of a project document
1. Ask each group to provide a document explaining the project created for the practical case chosen.
  NOTE: The project is evaluated using the project development evaluation rubric (see Supplemental Material Appendix 2).
Session 10: Presentation of the project
1. Ask each group to present the project about the practical case they chose.
  NOTE: The presentation is given to their classmates and the professor.
2. Evaluate the presentation. Perform the evaluation of the presentation according to the presentation evaluation rubric (see Supplemental Material Appendix 2).
Session 11: Follow-up of the students' work
1. Assess the interaction of the student groups on the UBUVirtual platform via the students' behavior monitoring software⁴¹ tool, which allows the interaction of students to be analyzed individually (Figure 2) and in groups (Figure 3).
2. To perform group analysis: Click the Group Icon > Group 01 > Select All > Logs. Select Components > Select All pointing from Visual Analytics and then select Heatmap. This will generate a heatmap that allows the interaction to be analyzed individually, as shown in Figure 2.
3. To perform the individual analysis: Click Select All > Comparisons. Select Logs > Components. Then, click Select All pointing from Visual Analytics and select Heatmap. This will generate a heatmap that allows the interaction to be analyzed as a group, as shown in Figure 3.
4. Use the tool to determine what interactions each student has individually (Figure 4) and within a group (follow step 2.11.2 and step 2.11.3).
Session 12: Evaluation of student satisfaction with project-based learning
1. Ask each student to complete an opinion survey about their satisfaction with PBL work at the end of the course (Table 3).
  NOTE: The PBL satisfaction survey consists of 17 closed response questions measured on a Likert-type scale from 1 to 5 and two open response questions. The Cronbach's alpha reliability index in this study was α = 0.89 for the overall scale and α = 0.88-0.91 for each item if the item was removed.
Session 13: Data analysis
1. For the first question (RQ1: Are there significant differences in the learning outcomes and satisfaction of health sciences students studying occupational therapy depending on whether the PBL methodology is implemented via e-Learning versus b-Learning, considering the effects of students' prior knowledge?), import all the data from the spreadsheet into the statistical program, SPSS⁴²: elaboration of PBL (LOEPBL), presentation of PBL (LOEXPBL), learning outcomes total (LOT), obtaining LMS access (LMSA), satisfaction of the students with teaching (SPBL), the independent variable (the type of e-Learning modality, i.e., implementation year of the teaching, and the covariate (prior knowledge).
  1. Do this for both the groups of e-Learning versus b-Learning teaching modalities (during the first year of the COVID-19 pandemic with online teaching) and for b-Learning (during the second year of the COVID-19 pandemic with mixed teaching, partly in person and partly online).
  2. Perform an ANCOVA with fixed effects (teaching modality: e-Learning vs. b-Learning) and the covariate (prior knowledge).
  3. Select Multivariate Analysis and include the dependent variables (LOEPBL, LOEXPBL, LOT, LMSA, and SPBL), the independent variable (the type of teaching modality for b-Learning), and the covariate (prior knowledge).
  4. Estimate the marginal means and perform the ANCOVA analysis by pressing the OK button.
2. For the second question (RQ2: Do the clusters of participants found match the learning outcomes, learning behaviors, and perceived satisfaction as a function of teaching modality [e-Learning vs. b-Learning]?), perform the following steps:
  1. Select Means cluster analysis and then the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL. Then, select Cluster Membership then pause very slightly, click Continuous and pause very slightly again. Select Initial cluster centers > ANOVA table > cluster information for each case. Then, select Continue.
  2. Select the cross table between the data found for cluster membership and the e-Learning versus b-Learning teaching modality groups. For Row, select the Variable Year (the type of e-Learning modality, i.e., teaching implementation year). For Column, select the Cluster Number of the case.
  3. Perform a cluster analysis using visualizing software. To do so, import the data into the visualizing software and select Heat Map > Radviz. Implement the visualization of the clusters with the different variables studied with respect to the relationship with the teaching modality used.
    NOTE: The free data mining software⁴³ (see the Table of Materials) was used for this analysis.
3. For the third question (RQ3: Are the students' suggestions for the improvement of the PBL methodology different depending on the teaching modality, e-Learning versus b-Learning?), perform the following steps.
  1. Perform a qualitative analysis of the open answers found in the PBL¹³ satisfaction scale for the two groups of e-Learning versus b-Learning using qualitative data analysis software⁴⁴. Import the answers to the open-ended questions obtained on the satisfaction of the students with the teaching (SPBL) into the software.
  2. Select Categorizing Student Responses in the two teaching modalities, e-Learning versus b-Learning.
  3. Select the Document group (teaching modality: e-Learning vs. b-Learning) analysis.
  4. Select Sankey Diagram.
  5. Export the results into a spreadsheet software.
    NOTE: First, the responses to the open-ended questions were categorized in each teaching modality group (e-Learning vs. b-Learning), and a frequency analysis was performed on the responses by group and categorization. The results were visualized using a Sankey diagram. This was done using qualitative data analysis software⁴⁴ (see the Table of Materials)

Table 2: Prior knowledge questionnaire¹³. Open and closed questions from the prior knowledge questionnaire. Please click here to download this Table.

Table 3: Scale of satisfaction with PBL¹³. Open and closed questions from the PBL satisfaction scale. Abbreviation: PBL = project-based learning. Please click here to download this Table.

Figure 2: Analysis of a single student's behavior on the UBUVirtual platform. Heat map of the behaviors carried out by the student on the virtual learning platform. Please click here to view a larger version of this figure.

Figure 3: Analysis of all student behaviors on the UBUVirtual platform. Analysis of the learning behaviors performed by the students on the virtual learning platform with respect to task completion. Please click here to view a larger version of this figure.

Figure 4: Analysis of student behavior in a group on the UBUVirtual platform. Analysis of the learning behaviors performed by the students in work groups on the virtual learning platform with respect to the completion of tasks. Please click here to view a larger version of this figure.

Subscription Required. Please recommend JoVE to your librarian.

Representative Results

The 98 participants recruited for the present study were undergraduate occupational therapy students aged between 21 years old and 56 years old (Table 1). The protocol was tested over 2 academic years at the University of Burgos. The delivery of the protocol is summarized in Table 4.

Table 4: Summary of the application of the protocol for project-based learning for university students. Abbreviation: LMS = Learning Management Systems. Please click here to download this Table.

The first research question (RQ1) was as follows: are there significant differences in the learning outcomes and satisfaction of health sciences students studying occupational therapy depending on whether the PBL methodology is implemented via e-Learning versus b-Learning, considering the effects of students' prior knowledge?

As Table 5 shows, there were significant differences in the elaboration (LOEPBL) scores, with the e-Learning group scoring higher. The b-Learning group scored higher in presentation (LOEXPBL) and LMS access (LMSA). No significant differences were found in the total learning outcomes (LOT), and no effect was found for prior knowledge as the covariate.

Table 5: ANCOVA of a fixed effects factor (teaching modality, e-Learning vs. b-Learning), the covariate (level of prior knowledge), and the effect value, η². ANCOVA of a fixed effects factor (teaching modality, e-Learning vs. b-Learning), the covariate (level of prior knowledge), and the effect value, η². Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; e-Learning = online classes; b-Learning = blended classes (online and physical classes); M = mean; SD = standard deviation; ANCOVA = analysis of covariance. Please click here to download this Table.

The second research question (RQ2) was as follows: do the clusters of participants found match the learning outcomes, learning behaviors, and perceived satisfaction as a function of teaching modality (e-Learning vs. b-Learning)?

Two clusters were found in which differences were detected in the different variables, but it could not be established that one was superior to the other in all the variables (Table 6).

Table 6: Final cluster centers. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL. Please click here to download this Table.

Subsequently, a cross table was prepared between the values of the clusters of belonging assigned to each participant with respect to the teaching modality variable (e-Learning vs. b-Learning) (Table 6) and the percentage of belonging to each of the groups. A contingency coefficient of C = 0.40, p < 0.001, was obtained (Table 7).

Table 7: Cross-tabulation of participant by cluster number Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to download this Table.

In addition, an ANOVA was performed between clusters for the variables of elaboration (LOEPBL), presentation (LOEXPBL), total learning outcomes (LOT), LMS access (LMSA), and students' satisfaction with teaching (SLPBL). Significant differences were found in presentation (LOEXPBL, p = 0.03) and LMS access (LMSA, p < 0.001) (see Table 8).

Table 8: ANOVA clusters. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; df = degrees of freedom; ANOVA = analysis of variance. Please click here to download this Table.

A visual cluster analysis was performed by means of the k-means technique using visualization software⁴³, taking teaching modality, e-Learning versus b-Learning, as a variable with respect to the different variables studied: elaboration (LOEPBL), presentation (LOEXPBL), total learning outcomes (LOT), access to the LMS (LMSA), and student satisfaction with teaching (SPBL) (Figure 5).

Figure 5: Cluster analysis of the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL with respect to the variable "Teaching modality, e-Learning versus b-Learning". Visualization of the clusters found with respect to the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.

Next, a heat map was constructed for the behavior of the variables in the clusters. Greater discrimination was found in cluster 1 (the students included in this cluster had less access to the virtual learning platform) and cluster 2 (the students included in this cluster had more access to the virtual learning platform) in the behaviors of students on the virtual learning platform (UBUVirtual is a virtual learning platform based on Moodle) and in the type of teaching (e-Learning vs. b-Learning) (Figure 6).

Figure 6: Heat map visualization of the clusters in the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL. Please click here to view a larger version of this figure.

The third research question (RQ3) was as follows: are the students' suggestions for the improvement of the PBL methodology different depending on the teaching modality (e-Learning vs. b-Learning)?

The answers were categorized into two groups, the e-Learning versus b-Learning groups, using the qualitative data analysis software⁴⁴. The frequencies by categorization code are shown in Table 9, and Figure 7 shows an analysis of the students' responses in the two groups by applying a Sankey diagram.

Table 9: Frequency analysis of the categorized responses of the students in the two intervention groups (Group 1, e-Learning teaching, and Group 2, b-Learning teaching). Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to download this Table.

Figure 7: Sankey plot of the categorized responses in the two intervention groups (Group 1, e-Learning teaching, and Group 2, b-Learning teaching). Sankey plot of the categorized responses in the two intervention groups. Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.

How do these results indicate that the suggestions for the improvement of the teaching were different depending on whether the teaching was via e-Learning or b-Learning? The students in both groups agreed that the teaching had been satisfactory. Nonetheless, the b-Learning group noted the importance of using the simulation center and their desire for real practice with children, thereby increasing the teaching in classroom practice. The teaching modality used (e-Learning vs. b-Learning) led to differences in the results of applying the PBL methodology.

More specifically, the students in the e-Learning group scored higher in the execution of the project, while the b-Learning group scored higher in presenting the project. There was also increased access to the virtual platform in the b-Learning group. Two clusters were found, with cluster 2 showing better results in all the variables studied except for two (total learning outcomes, for which the results were the same between the two groups, and satisfaction with the teaching methodology, for which the results were very similar in both groups).

Similarly, in the e-Learning group, 82.6% of the students were in cluster 1, and 17.39% were in cluster 2. In the b-Learning group, 59.62% were in cluster 2, and 40.38% were in cluster 1. Therefore, it can be concluded that the teaching modality used seems to have an effect on the student learning outcomes and student activity in the LMS, with b-Learning being more effective than e-Learning in project presentation and student activity in the LMS. In contrast, e-Learning seems to be more effective with regard to creating the project. Nonetheless, the PBL methodology is very effective in both cases since the means of the final learning outcomes were high in both groups with very low variability (e-Learning mean = 8.89, standard deviation = 0.48; b-Learning mean = 8.65, standard deviation = 0.92). In addition, satisfaction was very high in both modalities, with no significant differences between the groups.

These results support the first question from RQ1 and indicate the directions for future research to examine these aspects more deeply. The differences noted above can also be seen by analyzing the qualitative data regarding student satisfaction within the two teaching modalities. It is important to remember that the e-Learning was delivered at the most critical point in the state of emergency due to COVID-19, during which neither in-person nor blended teaching could be carried out. The students' evaluations of how the PBL methodology was implemented in the online mode showed that it was very satisfactory, as it allowed them to continue their training in a way that was close to normal and to relate it to their professional practice. This may be why the e-Learning group did not make many suggestions for improvements. In contrast, the b-Learning group, who were taught during the second year of the pandemic, felt that the PBL methodology imparted via the LMS and in the face-to-face simulation center was beneficial for their training. However, this group suggested many more ideas for improvements to the teaching, which were mainly focused on increasing the in-person classes, specifically in terms of practical activities with real users.

Supplemental Material Appendix 1. This includes Table 1 and Table 2. (Table 1) Document including the relationship between metacognitive strategies, sub-strategies, and the activities to develop them. Project-based learning is described from the current state of the field. In addition, information is presented on the relationship between the steps of PBL and the development of different metacognitive strategies. This is a very useful document for teachers who want to apply this educational methodology. (Table 2) Rubric for the evaluation of the project resolution process. A rubric is presented for monitoring the project development process with respect to the use of different metacognitive strategies. Please click here to download this File.

Supplemental Material Appendix 2: This includes Table 1 and Table 2. (Table 1) Rubric for the evaluation of the presentation of the PBL. A rubric is presented for the evaluation of the project development process with respect to the presentation and defense of the project. (Table 2) Rubric for the evaluation of the elaboration of the PBL. A rubric is presented for the assessment of the project development process with respect to project evaluation. Please click here to download this File.

Supplementary Material Appendix 3: An example of a case study. Please click here to download this File.

Supplementary Material Appendix 4: Examples of work produced by the students. Please click here to download this File.

Supplementary Material Appendix 5: Examples of the virtual laboratories developed. Please click here to download this File.

Subscription Required. Please recommend JoVE to your librarian.

Discussion

This study leads to the conclusion that the teaching modality (e-Learning vs. b-Learning) may affect the results in different elements of PBL¹⁷^,¹⁸. In future studies, this aspect will be explored in greater depth to see if the same pattern is found with students from other (particularly health sciences) courses, since this is the subject of this protocol. In contrast, no differences were found in the total learning outcomes or in student satisfaction with the teaching methodology. This is in line with results from other studies⁴^,¹³^,¹⁴^,¹⁵^,¹⁸^,²⁹.

In summary, the use of a PBL teaching methodology has been shown to be effective both for students' learning outcomes and their motivation for learning, which is consistent with the findings from other studies²^,³^,⁴. However, the students in the b-Learning group expressed an interest in more in-person teaching. This aspect supports a hybrid use of the methodology⁸. Similarly, the students in both groups appreciated that this methodology guides them toward the practical aspects of their profession⁴^,¹³. It was also found that the social situation in which the teaching was delivered affected the students' satisfaction and critical reflections about their learning. The learning was more participative in the second part of the pandemic, where there were greater possibilities for in-person learning⁴^,¹⁴^,¹⁵^,¹⁸^,²⁹. Not all students were found to respond in the same way or experience the same amount of participation²². This is an important aspect to be considered by teachers with respect to using this methodology, and supporting the methodology requires teachers with digital competencies along with the use of EDM techniques³²^,³³^,³⁶ and the analysis of students' open responses using mixed techniques³⁹^,⁴⁰^,⁴¹.

Finally, one limitation of this study is related to the PBL methodology, which was used only with students studying for a single degree-occupational therapy. The methodology was not implemented with multi- or interdisciplinary teaching designs³¹.

Subscription Required. Please recommend JoVE to your librarian.

Disclosures

The authorizations for recording the images and videos presented in this work are attached.

A model of a practical assumption that is applied in the work with PBL is also attached (see Supplementary Material Appendix 3: An example of a case study). The authors declare that they have no competing financial interests.

Acknowledgments

The study was carried out as part of the research project "SmartLearnUni", funded by the Spanish Ministry of Science and Innovation 2020 I+D+i Projects - RTI Type B. Reference: PID2020-117111RB-I00. The authors also gratefully acknowledge the collaboration of the health sciences students at the University of Burgos, especially the students studying for their Occupational Therapy and Nursing degrees.

Materials

Name	Company	Catalog Number	Comments
Atlas.ti v.9	Atlas.ti
Orange v. 3.30	Orange
SPSS v.24	SPSS
UBUVirtual	UBU

DOWNLOAD MATERIALS LIST

References

Mann, L., et al. et al. problem-based learning to practice-based education: A framework for shaping future engineers. European Journal of Engineering Education. 46 (1), 27-47 (2021).
Alonso Martínez, L., Puente Alcaraz, J. The importance of global health in nursing, what do we know. Nursing Index. 29 (3), 160-164 (2020).
enyuva, E., Kaya, H., Bodur, G. Effect social skills of nursing students of the project based teaching methods. Florence Nightingale Journal of Nursing. 23 (2), 116-125 (2014).
Sáez López, J. M. Estilos de Aprendizaje y Métodos de Enseñanza [Learning Styles and Teaching Methods]. Universidad Nacional de Educación a Distancia - UNED. , Madrid, Spain. 35-56 (2018).
Holm, M. Project-based instruction: A review of the literature on effectiveness in prekindergarten through 12th grade classrooms. InSight: Rivier Academic Journal. 7 (2), 1-13 (2011).
Kokotsaki, D., Menzies, V., Wiggins, A. Project-based learning: A review of the literature. Improving Schools. 19 (3), 267-277 (2016).
Jay, J. Problem based learning-A review of students' perceptions in an occupational therapy undergraduate curriculum. South African Journal of Occupational Therapy. 44 (1), 56-61 (2014).
Fortune, T., et al. Transformative learning through international project-based learning in the global south: Applying a students-as-partners lens to a "high-impact" Capstone. Journal of Studies in International Education. 23 (1), 49-65 (2019).
Sáiz-Manzanares, M. C., et al. Teaching and learning styles in Moodle: An analysis of effectiveness of use in STEM and Non-STEM qualifications from a gender perspective. Sustainability. 13 (3), 1166 (2021).
Sáiz-Manzanares, M. C., et al. Improve teaching with modalities and collaborative groups in an LMS: An analysis of monitoring using visualisation techniques. Journal of Computing in Higher Education. 33, 747-778 (2021).
Yazici, H. J. Project-based learning for teaching business analytics in the undergraduate curriculum. Decision Sciences Journal of Innovative Education. 18 (4), 589-611 (2020).
Sáiz-Manzanares, M. C., Garcia-Osorio, C. I., Díez-Pastor, J. F. Differential efficacy of the resources used in b-learning environments. Psicothema. 31 (2), 170-178 (2019).
Sáiz-Manzanares, M. C. E-Project Based Learning in Occupational Therapy: An Application in the Subject "Estimulación Temprana". , Servicio de Publicaciones de la Universidad de Burgos. Burgos, Spain. 15-250 (2018).
Sáiz-Manzanares, M. C., Marticorena, R., García-Osorio, C. I., Díez-Pastor, J. F. How do b-learning and learning patterns influence learning outcomes. Frontiers in Psychology. 8, 745 (2017).
Chen, C. H., Yang, Y. C. Revisiting the effects of project-based learning on students' academic achievement: A meta-analysis investigating moderators. Educational Research Review. 26, 71-81 (2019).
Spikol, D., Ruffaldi, E., Dabisias, G., Cukurova, M. Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning. 34 (4), 366-377 (2018).
Guo, P., Saab, N., Post, L. S., Admiraal, W. A review of project-based learning in higher education: Student outcomes and measures. International Journal of Educational Research. 102, 101586 (2020).
Almulla, M. A. The effectiveness of the project-based learning (PBL) approach as a way to engage students in learning. SAGE Open. 10 (3), (2020).
Sáiz-Manzanares, M. C., Escolar-Llamazares, M. C., Arnaiz, Á Effectiveness of blended learning in nursing education. International Journal of Environmental Research and Public Health. 17 (5), 1589 (2020).
Sáiz-Manzanares, M. C., Marticorena-Sanchez, R., Ochoa-Orihuel, J. Effectiveness of using voice assistants in learning: A study at the time of COVID-19. International Journal of Environmental Research and Public Health. 17 (15), 5618 (2020).
Simonton, K. L., Layne, T. E., Irwin, C. C. Project-based learning and its potential in physical education: an instructional model inquiry. Curriculum Studies in Health and Physical Education. 12 (1), 36-52 (2021).
Yadav, R. L., et al. Attitude and perception of undergraduate medical students toward the problem-based learning in Chitwan Medical College, Nepal. Advances in Medical Education and Practice. 9, 317-322 (2018).
Brinkman, D. J., et al. Switching from a traditional undergraduate programme in (clinical) pharmacology and therapeutics to a problem-based learning programme. European Journal of Clinical Pharmacology. 77 (3), 421-429 (2021).
de la Puente-Pacheco, M. A., Guerra-Florez, D., de Oro-Aguado, C. M., Llinas-Solano, H. Does project-based learning work in different local contexts? A Colombian Caribbean case study. Educational Review. 73 (6), 733-752 (2019).
Cavicchia, M. L., Cusumano, A. M., Bottino, D. V. Problem-based learning implementation in a health sciences blended-learning program in Argentina. International Journal of Medical Education. 9, 45-47 (2018).
Wu, T. T., Huang, Y. M., Su, C. Y., Chang, L., Lu, Y. C. Application and analysis of a mobile E-Book system based on project-based learning in community health nursing practice courses. Journal of Educational Technology & Society. 21 (4), 143-156 (2018).
Avrech Bar, M., et al. Problem-based learning in occupational therapy curriculum-implications and challenges. Disability and Rehabilitation. 40 (17), 2098-2104 (2018).
DeFillippi, R., Milter, R. G. Chapter 18: Problem-based and project-based learning approaches: Applying knowledge to authentic situations. The SAGE Handbook of Management Learning, Education and Development. Armstrong, S. J., Fukami, C. V. , SAGE Publications Ltd. London, United Kingdom. (2009).
Theobald, K. A., Ramsbotham, J. Inquiry-based learning and clinical reasoning scaffolds: An action research project to support undergraduate students' learning to 'think like a nurse'. Nurse Education in Practice. 38, 59-65 (2019).
AlHaqwi, A. I., et al. Problem-based learning in undergraduate medical education in Saudi Arabia: Time has come to reflect on the experience. Medical Teacher. 37 (1), S61-S66 (2015).
Wright, S. L., Duncana, M. J., Savin-Badenb, M. Student perceptions and experiences of problem-based learning in first year undergraduate sports therapy. Journal of Hospitality, Leisure, Sport & Tourism Education. 17, 39-49 (2015).
Baker, R. S., Inventado, P. S. Chapter 4. Educational data mining and learning analytics. Learning Analytics: From Research to Practice. Larusson, J. A., White, B. , Springer: Science+Business Media. New York, United States. 61-75 (2014).
Yücel, ÜA., Usluel, Y. K. Knowledge building and the quantity, content and quality of the interaction and participation of students in an online collaborative learning environment. Computers & Education. 97 (1), 31-48 (2016).
Gibson, D., de Freitas, S. Exploratory analysis in learning analytics. Technology, Knowledge and Learning. 21 (1), 5-19 (2016).
Romero, C., Ventura, S. Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications. 33 (1), 135-146 (2007).
Romero, C., Espejo, P. G., Zafra, A., Romero, J. R., Ventura, S. Web usage mining for predicting final marks of students that use Moodle courses. Computer Applications in Engineering Education. 21 (1), 135-146 (2013).
Arnaiz-González, Á, Díez-Pastor, J. F., Rodríguez, J. J., García-Osorio, C. Instance selection for regression: Adapting DROP. Neurocomputing. 201 (12), 66-81 (2016).
García, S., Luengo, J., Herrera, F. Data Preprocessing in Data Mining, Volume 72 of Intelligent Systems Reference Library. , Springer. New York, United States. (2015).
Verd Pericás, J. M. Introducción a la investigación cualitativa [Introduction to qualitative research]. , Síntesis. Madrid, Spain. 370 (2016).
Castañer, M., Camerino, O., Anguera, M. T. Métodos mixtos en la investigación de las ciencias de la actividad física y el deporte. Apunts. Educación física y deportes. 2 (112), 31-36 (2013).
Ji, Y. P., Marticorena-Sánchez, R., Pardo-Aguilar, C. UBUMonitor: Monitoring of students on the Moodle platform. , Available from: https://github.com/yjx0003/ (2018).
IBM Corp. SPSS Statistical Package for the Social Sciences (SPSS) (Version 24). IBM Corp. , Madrid, Spain. (2016).
Software Package. Orange. , Available from: https://orange.biolab.si/docs/ (2021).
Software Package Qualitative Data Analysis (Version 9). Atlas.ti. , Scientifc Software Development GmbH. Berlin, Germany. Available from: https://atlasti.com/es/ (2020).

Behavior