Mixed Reality for Education (MRE) Implementation and Results in Online Classes for Engineering

Summary

In this work, a mixed reality system called MRE was developed to help students develop laboratory practices complementing online classes. An experiment was carried out with 30 students; 10 students did not use MRE, 10 used MRE, and 10 more used MRE with teacher feedback.

Abstract

The COVID-19 pandemic has changed many industries, empowering some sectors and causing many others to disappear. The education sector is not exempt from major changes; in some countries or cities, classes were taught 100% online for at least 1 year. However, some university careers need laboratory practices to complement learning, especially in engineering areas, and having only theoretical lessons online could affect their knowledge. For this reason, in this work, a mixed reality system called mixed reality for education (MRE) was developed to help students develop laboratory practices to complement online classes. An experiment was carried out with 30 students; 10 students did not use MRE, 10 used MRE, and 10 more used MRE with teacher feedback. With this, one can see the advantages of mixed reality in the education sector. The results show that using MRE helps to improve knowledge in engineering subjects; the students obtained qualifications with grades 10% to 20% better than those who did not use it. Above all, the results show the importance of feedback when using virtual reality systems.

Introduction

Technology has always been present in the education sector; profound changes have occurred in the devices used to teach classes. However, face-to-face classes remain the preferred option for students and teachers. When the pandemic came, it changed all sectors, and education was no exception. In 2018, before the pandemic, only 35% of students who studied a degree reported having taken at least one class online; that is, 65% of students completed their studies in person¹. As of April 2020, by government order (Mexican), all public and private schools were prohibited from teaching face-to-face classes; for this reason, 100% of the students had to take distance classes. Universities were the first to act, using tools for video calling, preparing classes, homework management, etc. This makes sense, since people of university age (between 18 and 25 years old) are people who have been in contact with technology since birth.

Some classes can be fully adapted virtually; however, laboratory practices are complex to perform remotely, and students do not have the necessary material, which is often expensive. The impact that online classes have on the quality of knowledge is unclear, and some studies show that online courses generally yield worse student performance than in-person coursework². But one thing is certain, not carrying out laboratory practices that bring students closer to what they will experience in the industry will negatively affect their professional performance. Therefore, the importance of real-scale experiences becomes necessary in the current teaching of engineering^3,4,5. For these reasons, new technologies are being used to mitigate these problems. Among them are virtual reality (VR), augmented reality (AR), and mixed reality (MR). It is important to mention that VR is a technology that allows the creation of a totally immersive digital environment, whileAR overlays virtual objects in the real-world environment. On the other hand, MR does not just use virtual objects, but also anchors these objects to the real world, making it possible to interact with them. Thus, MR is a combination of VR and AR⁶. On the other hand, some organizations have also made efforts to develop remote laboratories, where real equipment exists but can be controlled remotely⁷.

The term MR dates to 1994; however, in the last 5 years, it has taken on special importance, thanks to large companies that have focused their efforts on developing environments, such as the Metaverse⁶. MR can be applied in different areas; two of the most common are training and education. Training has been one of the great drivers of MR; it is very expensive for a company to stop a production line to train new employees, or in dangerous environments, and it isn't easy to carry out training in the field. Education is not far behind; although face-to-face classes have changed very little, there are great efforts to incorporate MR into classes^8,9. For education, there are professional careers where it is necessary to carry out laboratory practices to have complete training. Many existing studies and research are in medicine, with VR, AR, and MR playing a key role. Multiple papers show how MR surpasses traditional teaching methods in surgical and medical subjects, where the practice is a clear advantage for developing students^{10,11,12,13,14}.

However, there is not the same amount of research on engineering issues. Normally in engineering careers, a student has theory classes complemented by practices. In this way, there are studies on MR and VR showing the benefits in engineering pedagogy¹². However, some of these studies focus on analyzing the complexity of the environment and the tools used^8,15. Tang et al. devised a study where students from different areas and with different knowledge used MR to improve their understanding of geometric analysis and creativity¹⁶. In a subsequent test, people who took their classes using MR finished faster, making it clear that MR positively affects learning¹⁶. Moreover, Halabi showed the use of VR tools in engineering education. Although it is not MR, it shows tools that can be used for teaching. It makes a real case study to show that it is possible to introduce VR in engineering classes¹⁷.

On the other hand, remote laboratories (RLs) are technological tools composed of software and hardware that allow students to remotely carry out their practices as if they were in a traditional laboratory. RLs are generally accessed through the internet, and are normally used when students are required to autonomously put into practice what they have learned as many times as they require¹⁸. However, with the arrival of COVID-19, its use has been to replace traditional laboratories and to be able to carry out practices during online classes¹⁸. As mentioned above, an RL needs a physical space (traditional laboratory) and elements that allow it to be controlled remotely. With the arrival of VR, laboratories have been modeled virtually, and through physical mechanisms, the elements of the laboratory can be controlled¹⁹. However, having an RL is very expensive, impeding many schools especially in developing countries. Some studies mention that costs can vary between $50,000 and $100,000^20,21.

Moreover, since the pandemic began, changes have had to be made quickly; in the case of RLs, attempts were made to send kits to the homes of each student to replace the traditional laboratories. However, there was a cost problem, as studies showed that each kit cost around $700^18,22. Nevertheless, the studies used expensive and difficult-to-obtain components. The pandemic affected education worldwide, and not many people could spend thousands of dollars to automate a lab or buy a kit. Most studies consider face-to-face classes and complement them with MR. However, in recent years, classes have been online due to COVID-19, and only some works show the improvement of virtual classes using MR and affordable devices^23,24.

The research that exists so far is mainly focused on medicine, with little information on engineering. However, without a doubt, we believe that the greatest contribution and difference is that our experiment was carried out for 6 months and was compared with subjects with the same characteristics who did not use virtual models, whereas most previous works carried out short experiments to compare single technologies or procedures; they did not apply them over several months. Therefore, this paper shows the difference in learning that can be made using MR in a university subject.

For this reason, this work shows the development and results of an MR system to help carry out laboratory practices in universities focused on electronic engineering. It is important to mention that special emphasis is placed on keeping the cost of the device low, making it accessible to the general population. Three groups use different teaching methods, and an exam is conducted on the class topics. In this way, it is possible to obtain results on understanding the topics in distance education using MR.

The project explained in this work is called mixed reality for education (MRE) and is proposed as a platform where students use VR glasses with a smartphone (i.e., no special VR glasses are used). A workspace is created where students can interact with virtual environments and real objects simply by using their own hands, due to the use of virtual and real objects, a mixed reality system. This workspace consists of a base with an image where all the virtual objects are displayed and interacted with. The environment created focuses on conducting laboratory practices to show electronic components and physics for engineering careers. It is important to highlight the need to provide feedback to students. For this reason, MRE incorporates a feedback system where an administrator (normally the teacher) can see what is being done to rate the activity. In this way, feedback can be given on the work done by the student. Finally, the scope of this work is to check if there are advantages in using MR in online classes.

To achieve this, the experiment was carried out with three groups of students. Each group consisted of 10 students (30 students in total). The first group did not use MRE, only taking theory (online classes) on the momentum conservation principle and electronic components. The second group used MRE without feedback, and the third group used MRE with feedback from a teacher. It is important to mention that all students have the same school level; they are university students in the same semester and with the same career, studying mechatronics engineering. The experiment was applied in a single course called Introduction to Physics and Electronics, in the second semester of the degree; that is, the students had been in university for less than 1 year. Therefore, the topics covered in the class can be considered basic from an engineering point of view. The experiment was carried out on 30 students, as this was the number of students who enrolled in the class where the experiment was authorized. The selected class (Introduction to Physics and Electronics) had theory and laboratory practices, but due to the pandemic, only theory classes were being taught. The students were separated into three groups to see the impact that the practices have on general learning and if MR classes could be a substitute for face-to-face practices.

Protocol

The protocol follows the guidelines of the Panamerican University ethics committee. The experiment was conducted with a total of 30 students, between 18 and 20 years old; eight students were female and 22 were male, and they all attended the Panamerican University in Guadalajara, Mexico (the second largest city in Mexico). All participants completed the informed consent process and provided written permission for photos to be taken and published during data collection. The only requirement was that the students needed to have a smart phone, which was no problem. Therefore, there was no exclusion criteria for the experiment.

1. VR system setup and calibration

NOTE: This step takes ~10 min.

1. Ensure the system includes all the components: an Android phone with operating system version 10 or higher, VR box glasses, and a wooden base with a calibration image (Figure 1) (see Table of Materials).
2. Open the MRE application on the cell phone and load the Unity, AR Foundation, Google Cardboard, and ManoMotion services^25,26,27,28. The MRE application has been developed by ourselves; it was developed for Android and it is not public.
3. Insert the cell phone into the VR glasses and put on the glasses.
4. Visually locate the center of the base of the MRE prototype (the blue square in Figure 1).
5. When the simulation appears, raise an outstretched hand to place it in the center of the view.
   NOTE: From this moment, the users can make hand gestures to interact with the simulated environment.

2. User preparation

NOTE: This step takes ~5 min.

1. Without VR glasses, open the MRE application, as shown in Figure 2.
2. Ensure that the application starts in user mode so it is only necessary to log in.
3. Select the scenario that the user wants to perform. There are two scenarios: electronic components and physics.
4. Press Play; the user will have 30 s to put on the VR glasses.

3. Execution of scenarios

NOTE: This step takes ~15 min.

1. Scenario 1: electronic components
   1. Locate the areas to position components, by means of red, green, and blue colors. This delimits the six interaction zones of this scene: three zones to take the virtual electronics components and three zones to drop the components, as shown in Figure 3.
   2. Take the component and position it in the right place. The right place depends on the component and what is seen in theory; for example, in theory, it is explained how to place a heat sink, and in MRE said placement is practiced.
   3. Continue until all components are in place.
2. Scenario 2: physics
   1. Locate the two cars involved in the scenario (Figure 4).
   2. Select the speed of each car.
   3. Visualize the graphs after the collision.

4. Administration view

1. On the main screen, press MRE modes (see Figure 2) and select the administrator option.
2. Log in to verify if the account has permission to access as admin.
   ​NOTE: It becomes possible to view the list of students and the grades obtained in each scenario.

5. Student results

1. Logging in as an administrator, click on the name of the desired student and view the table with the information of the grades of their scenarios.
2. Click on a student's name and select download grades as CSV. This will display all the results in a comma separated file.

Representative Results

This section shows the results obtained from the experiment. First, some details of how the experiment was carried out are explained, then the tests carried out on the students of the experiment are shown, and moreover, the results of the tests are presented. Finally, an analysis using one student of each group is described.

One of the biggest problems that the pandemic brought to engineering education was that it was not possible to carry out face-to-face laboratory practices, which has a direct impact on the knowledge acquired by students. To analyze whether the project developed in this article has an impact, an experiment was carried out with three groups of students. Each group consisted of 10 students; the first group did not use MRE, instead only taking theory (online classes) on the momentum conservation principle and electronic components. The second group used MRE without feedback, and finally, the third group used MRE with feedback from a teacher. It is important to mention that all the students had the same school level. They were all university students in the same semester and with the same career, studying mechatronics engineering. They were all students at the Panamerican University in Guadalajara, Mexico (the second largest city in Mexico). The experiment was applied in a single course called Introduction to Physics and Electronics, in the second semester of the degree (i.e., they were students who had been in university for less than 1 year. Therefore, the topics covered in class could be considered basic from an engineering point of view¹⁷.

The course (Introduction to Physics and Electronics) in which the experiment was carried out had the following characteristics: (1) the duration of the course was one semester; (2) there were two exams throughout the semester (i.e., a test was held every 10 weeks of classes), And each of these tests, or 10 week period, is called "partial"; and (3) each week had 6 h of classes, divided into 3 days of 2 h per class. During the week, 4 h of theory and 2 h of practice were taught. It is very important to mention that the characteristics mentioned above is what was done before the pandemic; during the pandemic, online classes were held. Therefore, the 2 h of practices per week could not be carried out, and were replaced by counseling and problem solving. For this reason, in the online classes, no practices were carried out.

Our experiment tried to modify as little as possible what was established in the class; the MRE system was introduced during practice hours (2 h per week), and the students who did not use the system continued with advice and problem solving. The 4 h of theory was not modified at all by our experiment. Similarly, the students who used MRE used one of the practice classes to explain the operation of the system. Moreover, MRE has two environments, one for electronic components and one for physics concepts. The experiment was carried out during one partial (10 weeks), which involved physics practices and electronic component practices. In this period, six practices were carried out in MRE (three practices of physics and three of electronic components). Finally, there were two groups that used MRE; one did not have feedback from the teacher and the other did. Those who did not have feedback were given a script of the practice to be carried out, and at the end the teacher assigned a grade from 0 to 10 in the MRE system, but no further explanation was given. On the other hand, in the group that had feedback, the teacher guided them during the practice. The teacher could observe the simulation at the same time as the students, since the system does not contain sound and their ears are uncovered, so the teacher guided the student by speaking to them during the simulation, indicating their errors and the reasons of said errors.

It is important to mention that the test was not edited for this experiment. In other words, the test would have be the same for the students if the current experiment was not carried out. The test had 14 questions, listed in Supplementary File 1 in the same order as they were presented.

Each question on the test had the same weight in the grade, however the teacher could assign fractions of points to each question based on the student's response. This was at the discretion of the teacher. Table 1 shows the grades of each of the students, with 0 being the worst grade and 10 the best. At the end, the average of each group is shown.

On the other hand, Figure 5 graphically shows the scores of each student separated by the group. In this way it is easier to visualize the results obtained from the experiment. Table 2 shows the results of each question, taking one student from each group.

Figure 1: MRE main materials. The MRE system consists of a simple 8 in x 8 in square piece of wood, onto which a base image is glued. The image consists of a central logo that is 3 in x 3 in in size; the rest of the space consists of randomly placed 1 in x 1 in icons using dark blue colors on a light blue background. Additionally, a VR box and an Android cell phone are inserted into the box. Please click here to view a larger version of this figure.

Figure 2: MRE application. (A) The button to select between user or administrator; it starts as the user by default. (B) Option to register/login. (C) Button to continue configuring the scenario. (D) Return to the previous screen. (F) Qualification at the moment; if it is the first time it is "played", it will appear at 0. (G) Start with the selected scenario. Please click here to view a larger version of this figure.

Figure 3: Electronic components scenario. The colors delimit the six interaction zones of this scene: three zones to take the components and three zones to drop the components. Please click here to view a larger version of this figure.

Figure 4: Physics scenario. Two cars are created facing each other, in addition to a spherical start button (green color) and a cubic one (light blue color) to modulate the force with which the second car is pushed. Please click here to view a larger version of this figure.

Figure 5: Score of each student and standard deviation separated by group. Grades per student and technology used; the standard deviation of each group is shown next to it. There are 30 students in total, 10 for each learning approach, and each student in each group was assigned a number from 1 to 10. It is important to mention the typical deviation, where it is clearly seen that without the use of MRE, the scores are much more dispersed. This may be logical, since these students only received online classes, so the attention that each student payed is very variable, and this is seen in the scores obtained. On the other hand, there is much less dispersion when MRE is used. Further, when feedback is added to MR technology, there is less dispersion, which indicates a better understanding by all students, not just by some students. Please click here to view a larger version of this figure.

Table 1: Knowledge test results for the three groups. This table shows all the results of the exams taken by the students. There are 30 students in total, 10 for each learning approach, and each student in each group was assigned a number from 1 to 10. It can be clearly seen that the best average obtained was when MRE was used and there was feedback from the teacher. Even if there was no feedback, it is still a better option in general terms to use MRE for a better understanding of the topics. When using MRE, there was no score lower than 7.5 in any of the students; so, it can be deduced that there was in general a better understanding of the topics. Finally, using MRE and with feedback from the teacher, there were no scores below 8.0, and the highest scores of the 30 students were also seen, 9.3 and 9.5. Therefore, one can clearly see the benefits that students have in understanding topics when using MRE, but above all, when feedback is given on the work done in the practices. Please click here to download this Table.

Table 2: Results per question using one student from each group. Grades to the answers of a student of each group. Students whose grade was close to the group average were selected. The teacher could award points to partially correct answers. The students who used MRE had better results with the electronic components questions, suggesting that knowing the components in their real dimensions and shapes (using MRE) helped to improve theoretical knowledge. The students who used MRE with feedback, in addition to being able to observe the components as they would be seen in reality, received help from the teacher in the practices of physics and electronic components. Therefore, it can be said that in addition to practicing, they had advice hours, and this is clearly reflected in the results. Please click here to download this Table.

Supplementary File 1: Questions presented to the students. Please click here to download this File.

Discussion

The MRE system allows different scenarios for students to learn about electronic components or physics topics. An important point is the possibility of the teacher providing feedback. In this way, the students can know what they did wrong and why. With the MRE system developed, an experiment was carried out with 30 students, where 10 students did not use MRE, 10 used MRE, and finally another 10 used MRE and received feedback from the teacher. At the end of the classes, a general knowledge test was given to all the students. The test was not modified for the experiment (i.e., the same test is applied if the classes are purely theory or if laboratory practices are carried out. The practices are only a complement to better understand the theory and thus have a better general understanding of the subject. The test is written answers showing calculations, and the teacher can mark with half points in case the answer is partially correct.

Thanks to the use of MRE, the students obtained a better general average, the best average seen when there was feedback from the teacher. In the same way, an important point is the standard deviation. The objective of a class is that the majority of the students, or ideally all of them, get the greatest amount of knowledge. Due to the use of MRE, a smaller dispersion of the scores can be observed, which proves that the knowledge about the topics was understood by a greater number of students.

When observing the scores of each question in detail, MRE has a smaller effect when the questions are focused on problems that can be analyzed 100% from the theory. However, in engineering topics, it is important to know both the equipment and components, therefore MRE had a positive impact, and the students who used MRE responded better to the questions that covered these topics. Moreover, In the case of theoretical questions (such as physics), MRE is helpful when one has feedback from the teacher, as the teacher can clarify these issues supported by a virtual environment. Teacher feedback is nothing new; it occurs in face-to-face classes, so it is clear that this feedback is still just as important in virtual environments.

The MRE system helps engineering students to carry out laboratory practices remotely. The world has changed, and although it is currently returning to face-to-face classes, every day more schools open 100% online courses²⁹. To face these changes, applications have been created using emerging technologies. One such technology is MR, in which it is possible to visualize study environments to improve learning. However, most of these applications are used in medical environments, with few in engineering^9,12. On the other hand, RLs have been hailed as the solution for distance engineering classes, but it is necessary to have a physical space and the components are very expensive. Therefore, the investment for a RL is very high, and they are not included as a possibility for many schools in Latin America^19,20.

In the same way, other works have discussed how virtual and remote laboratories can help in distance education. For instance, they agree that the costs are lower than setting up a traditional laboratory. Vergara et al. analyzed data from more than 400 students asking about their experience with the use of VR and MR in laboratories; 89% of the students mentioned that they are adequate to complement a teacher's explanation, but only 11% said that the use by itself is adequate. This technology alone is enough to understand the subject, although the work does not carry out any analysis on the impact that the use of this technology has on the understanding of the subject beyond asking the student's feelings³⁰. Moreover, Wu et al. analyzed multiple works that mention VR using head-mounted displays (HMDs; as we use it in this work). They conclude that HMD-based immersive learning has a better effect on learning performance than non-immersive learning approaches³¹. Despite this, Wu et al. also do not present how much the understanding of the subject can improve using VR or MR; they only mention that there is better learning, especially in science subjects, again as the case presented in this paper.

On the other hand, Makarova et al. experimented to find the effect of VR in teaching automotive services. Although the number of students mentioned is 344, these students are from different grades, so they have different knowledge and skills. The students in their study range from 19 to 30 years old, unlike what is presented here, where all the students have the same level of studies and are between 18 to 20 years of age. On the other hand, Makarova et al. analyzed students using physical and virtual equipment, where 35 students used virtual equipment (a number of students not very different from our experiment). They conclude that VR and MR technologies are much more effective than traditional methodologies, increasing students' interest in learning³². Additionally, other works mention that the use of virtual systems helps teach science and languages, even analyzing the usability of different approaches and ergonomics, which is out of the scope of this work^33,34.

Other works, such as Loetscher et al., analyzed the correct VR tool that should be used depending on the test type, especially for behavioral tests, in which the response time is often essential for data analysis. They mention that VR systems on cell phones have a low response time³⁵, although for the experiment shown in this study, the response time does not influence the exam applied to the students. Additionally, it is necessary to analyze the cost of setting up a laboratory with specialized equipment against the time of desired response to obtain feasibility. It is clear that some experiments will be crucial to reduce the limitations of the hardware, but it is not the case for this work.

Therefore, we firmly believe that this work complements the studies that have been carried out so far. Many works have shown that using virtual technologies helps in learning and interest, however, they have not tried to demonstrate the real impact that it can have on learning. Although the number of students used in the experiment is low, we made sure that everyone had the same level of knowledge and skills (as much as possible) and that the same topic was taught to all, attempting to eliminate any external component that could have affected the results. The exam applied was the same, allowing to quantify (in a small sample) the improvement that students have using virtual technologies to complement the theory seen in class.

Thanks to MRE, it is possible to carry out laboratory practices for engineering at a low cost and with a minimum investment for schools. One only needs an Android cell phone from 2019 or later and a wooden base for calibration, making it much more accessible for schools in developing countries. It is worth mentioning that it is necessary to follow a series of steps to use the MRE system. Undoubtedly, the critical step for the correct operation of the system is the configuration and calibration of the VR system (step 1). Because MRE uses the hands as application tools, an error in the calibration would prevent being able to continue with the execution of the scenarios. Additionally, it is important to use the base with the image for calibration. The image is used to dimension the environment and detect the hand in space.

Therefore, it is clear that a limitation of the presented project is to have a base with the image for the calibration. For the experiment presented, it was necessary to manufacture a base for each student. Although once calibrated it was quite easy to reproduce and play the scenarios, it is worth mentioning that it is complex to create new scenarios. Therefore, a long development time is needed for each practice that is required to be developed.

However, a differentiating point with RLs or other MR technologies is the low cost of equipment and material needed. Any Android phone can be used as a tool to carry out the practices, although one limitation is obtaining the calibration image; still, it can be printed in the traditional way and no special equipment is necessary. Therefore, access to the already developed scenarios has a low cost. By using such accessible technology, MRE can be used in other areas too, not just laboratory practices. Mainly, during the training of personnel for companies, when a new employee joins, it is often necessary to stop or lower production to teach the use of machinery. Therefore, MRE can be adapted to develop production line environments.

Materials

Name	Company	Catalog Number	Comments
MRE application for Andorid			The application was developed for the experiment, it was made by us. It is NOT public, and there are no plans for publication.
Non-slip fabric (20 x 20 cm)
Printing of our base image
Self-adhesive paper (1 letter size sheet)
Virtual Reality Glasses	Meta Quest 2		We use the Meta Quest 2, which is a virtual reality headset with two displays of 1832 x 1920 pixels per eye, with this headset you could play video games, or try simulators with a 360 view. Also, the headset has two controls, in which the virtual hands feel like your real ones and this is thanks to the hand-tracking technology. https://www.meta.com/quest/products/quest-2/tech-specs/#tech-specs
Wooden plate (20 x 20 cm)