Online Virtual Reality Networked Control Laboratory Applied in Control Engineering Education

Summary

This study describes a WebVR-based online virtual reality (VR) laboratory system that provides users with immersive and interactive experimentation capabilities supported by VR devices. The proposed system not only helps to enhance the realism of user participation in online experiments but is also applicable to a wide range of online laboratory frameworks.

Abstract

Online laboratories play an important role in engineering education. This work discusses a WebVR-based virtual laboratory system. The user enters the simulated laboratory environment through a virtual reality (VR) device and interacts with the experimental equipment, similar to hands-on experiments in a physical laboratory. In addition, the proposed system allows users to design their own control algorithms and observe the effects of different control parameters to enhance their understanding of the experiment. To illustrate the features of the proposed virtual laboratory, an example is provided in this paper, which is an experiment on a double inverted pendulum system. The experimental results show that the proposed system allows users to conduct experiments in an immersive and interactive manner and provides users with a complete experimental process from principal design to experimental operation. A solution is also provided to change any virtual laboratory into a WebVR-based virtual laboratory for education and training.

Introduction

With the advancement of the Internet and the popularity of mobile devices, the demand for online education is increasing¹. In particular, during periods of widespread epidemics, traditional educational institutions often face challenges in conducting in-person instruction effectively, which highlights the importance of online education as an important pedagogical approach². Theoretical courses are relatively easy to transfer to online platforms. They can be conducted with the help of tools such as remote video conferencing software and massive open online courses (MOOCs)³. However, practical courses face greater challenges as they require users to perform hands-on experiments in traditional laboratories.

Researchers have made significant contributions to addressing the challenge of making experimental equipment available online. Over the past two decades, extensive studies have been conducted on the concepts and technologies of online laboratories^4,5. Online laboratories typically encompass remote laboratories⁶, virtual laboratories⁷, and hybrid laboratories⁸. These online laboratory approaches have found widespread application in various engineering disciplines, including control engineering⁹, mechanical engineering¹⁰, and software engineering¹¹.

While significant progress has been made in terms of the convenience of experimental operations in online laboratories¹², users still perceive a lack of realism and similar hands-on practical operations compared to traditional laboratory environments, which affects their overall experience¹³. This discrepancy in user experience motivates further research and development efforts to enhance realism and engagement in online laboratory environments.

To solve the above problems, virtual reality (VR) technology has been applied in virtual laboratories¹⁴ to improve the immersiveness and interactivity of virtual laboratories¹⁵. VR-based virtual laboratories provide users with a close-to-realistic experimental experience. Users can complete group assignments in the architectural education process through avatars¹⁶, performing the architectural surveying process together immersively, just as they would in a traditional classroom environment. Furthermore, the VR-based virtual laboratories allow users to enter the immersive environment of virtual laboratories and interact with virtual experimental equipment by wearing VR headsets and handles¹⁷, improving users' hands-on abilities¹⁸. For different educational purposes, we can design different virtual environments. For example, VR can be combined with gamification theory to enhance engineering education for the general public and to improve the efficiency of disseminating difficult-to-understand knowledge such as sustainable development¹⁹.

Similar to online laboratories, particularly virtual laboratories, WebVR-based virtual laboratories have many advantages. Firstly, they break through the time and space limitations of traditional laboratories, and users can conduct experiments anytime and anywhere²⁰. Secondly, online laboratories can provide a safer experimental environment to avoid possible dangers and accidents in experimental operations²¹. Thirdly, virtual laboratories can also provide more experimental resources and simulation situations to extend users' experimental scope and experience²². Most importantly, WebVR-based virtual laboratories can stimulate users' learning interest and initiative and enhance their experimental experience and participation²³.

Compared with other VR-based virtual laboratories, WebVR-based virtual laboratory seamlessly combines the merits of VR-based virtual laboratories with web-based online laboratories. Virtual Instrument Systems in Reality (VISIR)²⁴ builds a basic analog electronic remote laboratory by constructing real circuit boards. Users can perform simulated experiments on the web interface to complete real circuit board experiments. Weblab-Deusto⁸ builds the water tank Field Programmable Gate Array (FPGA) laboratory where users can interact with the three-dimensional (3D) model of the water tank in the web platform without relying on other plug-ins. The system proposed in this paper introduces the capability to seamlessly integrate WebVR as a modular component into the existing virtual laboratory infrastructure. This integration can be achieved without destroying the original architectural framework of the laboratory, thus preserving the basic structure and function of the laboratory. This integration is also applicable to the framework of an online laboratory with separate front end and back end.

The system proposed in this paper is implemented based on Networked Control System Laboratory (NCSLab)²⁵, which inherits the flexibility, interactivity, modularity, and cross-platform features of the NCSLab system. Users can conduct experiments according to different modules and can also customize algorithms and configuration interfaces, providing users with enough space for self-realization. Online experiments are driven in real-time according to the algorithms run by the user. Users can interact with the virtual model to change the inputs of the experimental algorithm when conducting VR experiments and can even change the parameters of the control algorithm through the components so that users can experience the principle of the control algorithm more realistically.

WebVR-based virtual laboratories bring great potential for online education. It can provide an immersive experimental experience, overcome the limitations of traditional laboratories, and promote hands-on practical skills and innovative thinking among users.

Protocol

This study met the guidelines of the Human Research Ethics Committee at Wuhan University, and informed consent was obtained for all experimental data. In this paper, the experimental steps for the double-inverted pendulum system are discussed, and all the steps are performed in the WebVR-based NCSLab.

1. Access WebVR-based NCSLab system

1. Open a web browser that supports WebVR. Enter the Uniform Resource Locator (URL) of the WebVR-based NCSLab to access the system.
2. Click the Start Experiment button to log into the NCSLab system. If it is the first-time logging into the system, do an account registration.
3. Log into the NCSLab system, select different experiments from the left menu bar, and choose the double inverted pendulum experiment in this case.
4. Access the 3D sub-page on the main page.
   NOTE: There are five sub-pages on the main page, beginning with the first one, which is the introduction of the equipment model. It contains a 3D model animation as well as documentation. By visiting this page, users can grasp the principle of the double inverted pendulum system, enabling convenient execution of subsequent experiments.
5. Apply for experiment control by clicking the Request Control button to ensure efficient use of resources. This will grant users 30 min of experiment time.
   NOTE: For virtual experiments, 500 users can be allowed to conduct experiments at the same time.
6. Enter the plant information sub-page to gain access to comprehensive details regarding the experimental apparatus. This encompasses information on equipment that is currently in use, equipment that remains unused, and maintenance-related equipment.
7. Choose the system default control algorithm to download on the Experimental Algorithm sub-page. Alternatively, proceed to the Algorithm Design sub-page to design a different algorithm.
   1. To design a new control algorithm, click the Create New Model button on the algorithm design sub-page to enter the design interface.
      NOTE: The process of algorithm design closely mirrors that of MATLAB/Simulink, whereby users construct the control algorithm block diagram through an intuitive drag-and-drop approach, employing various modules to craft the desired control logic.
   2. Build the complete control algorithm block diagram, as depicted in Figure 1, and follow the steps described below.
   3. Select the Double Inverted Pendulum System Module from the device model on the left.
   4. Choose the Gain Module to design the feedback matrix for the Linear Quadratic Regulator (LQR) controller.
   5. Select the Step Signal as input and add other modules. Double-click the module to view detailed information and modify the parameter configuration. For example, double-click the Constant Signal Module to modify the value of the constant signal.
8. Click the Start Simulation button upon completing the control algorithm design. Upon completion of the simulation, observe the control effectiveness of the designed algorithm. If unsatisfied with the simulation results, fine-tune the parameters of the LQR controllers until a control algorithm with improved performance is achieved.
9. Click the Compile button to generate the control algorithm. After compilation, the algorithm is stored in the private algorithm area of the experimental algorithm sub-page and the algorithm design sub-page.
10. Download the control algorithm on the experimental algorithm sub-page by clicking the Download Algorithm button located on the right side of the control algorithm section.
11. Select an experimental configuration and conduct experiments on the Monitoring Configuration sub-page. The system provides a pre-defined configuration to meet the general experimental requirements of users.
    NOTE: Users have the flexibility to click the Create New monitor button to craft a customized monitoring setup tailored to their specific experimental demands.
12. Customize the monitoring configuration and choose from a variety of components available in the editing interface of the monitoring configuration sub-page, as depicted in Figure 2. These components include input variable components, output variable curve display components, and 3D model components.
13. For VR experiments, select the 3D model component. The 3D model component allows users to integrate a 3D model into the monitoring configuration.
14. To facilitate parameter configuration, adjust the parameters for each component, which are directly linked to system parameter variables. Double-click on a component and access the window to select the relevant optional parameters within the experimental system.
15. Users have the flexibility to optimize the layout of the monitoring configuration by resizing components. To do this, drag the edges of the respective components to the desired dimensions.
16. Click the Save button to save the designed monitoring configuration for future use in subsequent experiments, saving time and effort for setting up the monitoring system repeatedly.
    NOTE: The monitoring configuration can only be performed after the control algorithm has been correctly downloaded.
17. Click the Start Experiment button on the window to initiate the experiment. Click the VR Button in the bottom right corner of the 3D model component to launch the VR experiment.
    NOTE: The VR experiment is embedded on the web page. When users use it for the first time, the browser may prompt them in the top left corner to allow the browser to use VR functionality, select Allow to proceed.

2. Selecting the access method

1. Use a WebVR emulator extension. To engage in experimentation using this method, install the WebVR emulator extension, which is readily available for search and download from the browser's extension store.
   NOTE: The WebVR emulator extension helps users run WebVR content in a web browser and provides the virtual VR headset and the handles controller environment without the need to use the real VR device.
2. Use VR devices that support WebVR. If VR devices are used for the first time, the basic environment configuration is needed. First, turn ON the power of the headset and controller to start the system. Set up the initial ROOM program in the headset. Following the visual cues displayed on the headset screen, use the handle controllers to carefully calibrate the boundaries and orientation of the virtual space environment. Finally, establish a streaming connection between the headset and the computer.
   NOTE: This is the second method to access the proposed system. VR devices generally include a headset and a pair of handle controllers. VR devices have built-in stores where users can download WebVR-enabled browsers. Alternatively, users can use the built-in browser, which generally supports WebVR. It is noteworthy that various VR devices may employ distinct methods for connectivity.

3. Experimental procedure

1. Adjust the perspective to find the optimal position for conducting the double inverted pendulum system experiment.
   1. For users utilizing the WebVR emulator extension, open the Developer Tools, locate the WebVR extension, and manipulate the virtual VR device using the mouse to adjust the perspective, as shown in Figure 3.
   2. For users employing VR devices, immerse in the virtual experimental environment and ascertain the optimal experimental position through physical movements.
2. Interact with the double inverted pendulum system using the handle controller as described below.
   1. Move the handle closer to the cube. Press the Trigger button to pick up the cube and the double-inverted pendulum system will stop moving.
   2. By moving the handle, control the position of the cube. Release the cube once it is in the desired position by releasing the trigger button. The position is now designated as the subsequent setpoint for the cart, as depicted in Figure 3.
3. Observe the motion process of the double inverted pendulum system. By manipulating the Alternating Current (AC) servo motor, set the belt in motion. Under the impetus of the belt, the inverted pendulum can move along the guide rail, The system structure of the double inverted pendulum is elucidated in Figure 4. Eventually, the double inverted pendulum will stabilize at the setpoint.
4. Encourage users to iteratively manipulate the cube's position, continuously adjust the cart's setpoint, and meticulously observe the dynamic behavior of the double inverted pendulum system.

Results

The VR experiment system presented provides users with the capability to engage in immersive experiments using VR devices, thereby enhancing the interaction between users and the experimental equipment. Furthermore, the system is web-based, eliminating the need for users to configure local environments. This design allows for the system's scalability, making it suitable for large-scale applications and training and educational purposes.

In traditional laboratory environments, users are required to personally configure and install software and hardware devices, which can consume a significant amount of time and resources²⁶. However, virtual laboratories leverage cloud computing and virtualization technologies to move the laboratory environments to the cloud. Users can simply access the corresponding website through a web browser to utilize the functionalities and resources offered by the laboratories.

Figure 3 demonstrates that users can engage in WebVR experiments using different approaches. Users who do not have readily available VR devices can quickly conduct experiments through browser extensions. Users who have access to VR devices can immerse themselves in the experiments and interact directly with the experimental equipment, enhancing realism when exploring the experimental process. These two different ways of conducting WebVR experiments provide users with more options and enable a wider range of users to utilize the proposed system.

The double inverted pendulum examples demonstrate that the proposed WebVR-based virtual laboratory can run directly in a web browser without the need for additional software installations or configurations. This approach not only reduces user inconvenience but also greatly enhances the system's scalability. Additionally, users have the option to use VR devices for immersive interaction with the experimental equipment. By using handle controllers to adjust system parameters, users not only enhance their hands-on experience but also improve their theoretical knowledge and practical skills.

A total of 21 students participated in the experiment, where a questionnaire was conducted to further validate the applicability and effectiveness of the proposed system. We included students with backgrounds in automation and control engineering, and all of these students had previously participated in virtual experiments in NCSLab and had some basic knowledge of virtual experiments but had not participated in VR experiments in WebVR-based NCSLab. By adopting anonymous statistical data, we guarantee the privacy and security of the participants when filling out the questionnaire, thus ensuring the reliability of the questionnaire data.

The results of the questionnaire are shown in Figure 5, and the data indicate that the system proposed in this paper performs well in terms of realism and interaction with the device and achieves significant improvement compared to the traditional mouse-keyboard virtual experiment. In addition, participant feedback showed that the system not only increased students' interest and experimental skills in learning but also helped them better understand the experimental content, thus enhancing learning outcomes.

It is worth noting that most of the students believed that this type of experimentation is not only applicable to the current course and experiment but also has the potential to be applied in other courses and experiments.

The system proposed in this paper uses 3DS Max software for modeling the experimental equipment, which renders the experimental scenes using Unity engine software²⁷ and allows users to interact with the equipment using VR devices. Finally, the experimental scenes are packaged into Web Graphics Library (WebGL) format and seamlessly integrated into the online laboratory system in the form of modularized components to construct a WebVR-based virtual laboratory system.

Figure 1: Control algorithm design for the double inverted pendulum system. Users can select different modules from the module library on the left to build the control algorithm for the double-inverted pendulum system. The selection and connection of modules are similar to those in MATLAB/Simulink. In the realm of double-inverted pendulum systems, a plethora of control methods abound. For the present system, the chosen strategy is the Linear Quadratic Regulator (LQR) control approach, and the figure illustrates the feedback matrix fashioned in accordance with the LQR controller. Please click here to view a larger version of this figure.

Figure 2: Configuration design for monitoring the double-inverted pendulum system. Users can select components from the component library above to design the monitoring configuration. If a VR experiment is desired, the 3D Model component must be selected. Users have the flexibility to opt for the Chart component to visually track alterations in the angular orientation and position of the double inverted pendulum or the input component for making adjustments to controller parameters. The double-click on the component permits users to establish associations between system variables for parameter configuration. Within the double-inverted pendulum system, the parameters of the chart are configured to encompass both the set and actual positions of the cart, along with the angles of the first order and double pendulums. After the completion of the monitoring configuration design, users should first activate the experiment by clicking the Start Experiment button. Following this, they can initiate the VR experiment by clicking the VR button located in the lower right corner of the 3D Model component. Please click here to view a larger version of this figure.

Figure 3: Conducting the double inverted pendulum system experiment using VR headset and the WebVR emulator extension. Users can conduct WebVR experiments through VR devices or the WebVR emulator extension. The cube is controlled to set the setpoint for the double inverted pendulum using a handle. Once the position of the cube is determined, the double inverted pendulum will steadily move towards the setpoint direction until it eventually stabilizes at the set position. On the right side of the 3D model is a chart that records the cart position and the angles of the first order and double pendulums. The chart also allows for the observation of the trend of changes in key system parameters. Please click here to view a larger version of this figure.

Figure 4: Structure of the double inverted pendulum system. There is a cube above the base, and the position of the cube is the setpoint of the cart. Users can pick up the cube and adjust the position by the handle. Once the alternating current (AC) servo motor propels the belt into rotation, the cart will proceed along the guide rail under the impetus of the belt. In concert with this motion, the first-order pendulum and the double pendulum will also undergo corresponding displacement and rotation. Please click here to view a larger version of this figure.

Figure 5: Data results of the survey questionnaire. The questionnaire comprised six questions, each meticulously detailed here. Each question had five options, roughly meaning strongly disagree, disagree, neutral, agree, and strongly agree, on a scale of 1 to 5. A total of 21 valid responses were collected. The mean values and standard deviation were calculated from these scores and graphically presented in the figure for clarity and interpretation. Please click here to view a larger version of this figure.

Discussion

The presented protocol describes a virtual laboratory system that enables users to conduct VR experiments online but also uses a low-cost PC controller²⁸, which is conducive to large-scale application promotion. Users can gain knowledge about the entire experimental process, from principles and algorithms to practical experimental operations. This system allows users to immerse themselves in the experiments, eliminating the reliance on traditional mouse and keyboard input. This system provides an immersive experience for observing the experimental process and hands-on manipulation of experimental devices.

This system goes beyond traditional interfaces and provides users with a more intuitive and engaging way to interact with experimental equipment. Similar to hands-on experiments in a physical laboratory, this virtual laboratory strives to recreate the experimental operations as faithfully as possible. This online access approach provides virtual laboratories with the following advantages.

Flexibility and convenience: Users can access virtual laboratories anytime and anywhere through a web browser without being limited to specific physical laboratory locations and schedules. This approach greatly enhances the convenience of remote learning².

Scalability and cost-effectiveness: Virtual laboratories can easily scale and provide additional computing resources and experimental equipment to meet the demands of large-scale applications. Users do not need to purchase and maintain expensive hardware devices themselves but can conduct experiments using web-based resources, reducing their cost investments³.

Security: Virtual laboratories can offer enhanced security measures. Users do not need to worry about accidents resulting from mishandling during experiments, which helps ensure their safety to a certain extent²⁹.

By leveraging VR technology, users can enter a simulated laboratory environment where they can interact with objects and conduct experiments using the handle, similar to a physical laboratory. As shown in Figure 3, users can use the handle to pick up and move the cube to set the setpoint for the cart in the double inverted pendulum system. This form of interaction not only adds a new level of realism and interactivity to the virtual laboratory experience but also enhances users' understanding of the experiment.

Additionally, this system provides users with opportunities to explore experiments. They can design their own control algorithms and observe the effects of different control parameters, which helps them gain a deeper understanding of the experimental principles³⁰. It cultivates a sense of participation and active learning among users.

Currently, VR laboratories are primarily designed and utilized for specific scenarios, lacking a framework for large-scale applications. Users are often limited to conducting experiments according to predefined steps, with limited opportunities for implementing their own ideas. In contrast, a WebVR-based virtual laboratory seamlessly integrates experimental content as component modules into the virtual laboratory. This approach is not only versatile, fitting into a wide range of application frameworks, but also empowers users to interact with experimental equipment and carry out customized experiments according to their preferences and needs.

Nonetheless, certain issues deserve attention and resolution. These include the need for a more extensive repository of virtual resources, as well as the requirement for enhanced precision in simulating the dynamic behavior of virtual devices in comparison to their physical counterparts. We plan to expand our Virtual Resource Repository by working with subject matter experts in different fields, which will ensure that we cover a wide range of experimental content from natural sciences to engineering to meet the needs of different users. In addition, we encourage users to actively participate in the construction of our system. In addition to providing suggestions for the repository, we also plan to conduct user surveys and interviews in the future to gain a deeper understanding of the types and areas of resources that users expect. To address the precision challenge of simulating dynamic behaviors, advanced modeling techniques, such as machine learning-based methods or more complex mathematical models, are used to improve the precision of virtual device representations. Additionally, real-world systems are often characterized by uncertainty, which needs to be incorporated into the approach to virtual device simulation while maintaining accuracy, allowing for a more realistic representation of the real world.

In summary, the proposed virtual laboratory system enables users to participate in VR experiments in an immersive and interactive manner. By providing an experimental experience that is as realistic as possible, it enhances users' understanding of the experimental process, from principles and design to experimental operations. The online accessibility of the system also offers a flexible, convenient, and safe experimental environment, making it a promising solution for scientific research and educational training on a large scale.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
3DS Max	Autodesk		3ds Max professional 3D modeling, rendering, and animation software enables you to create expansive worlds and premium designs. https://www.autodesk.com/ca-en/products/3ds-max/overview
Meta Quest 2	Meta Platforms	10036728220341	meta quest 2 is a standalone virtual reality headset that allows users to experience WebVR content. https://www.meta.com/it/quest/products/quest-2/
Unity	Unity Technologies		Unity is the platform for real-time 3D interactive content creation and operation. All creators, including game developers, artists, architects, automotive designers, film and television, use Unity to bring their ideas to life. The Unity platform offers a complete suite of software solutions for creating, operating, and realizing any real-time interactive 2D and 3D content on cell phones, tablets, PCs, game consoles, augmented reality, and virtual reality devices. https://unity.com/cn