Analysis, Design and Practical Validation of an Augmented Reality Teaching System Based on Microsoft HoloLens 2 and Edge Computing ^†

Abstract

In recent years, the education sector has incorporated the use of new technologies and computing devices into classrooms, which allowed for implementing new ways for enhancing teaching and learning. One of these new technologies is augmented reality (AR), which enables creating experiences that mix reality and virtual elements in an attractive and visual way, thus helping teachers to foster student interest in learning certain subjects and abstract concepts in novel visual ways. This paper proposes to harness the potential of the latest AR devices in order to enable giving AR-enabled lectures and hands-on labs. Specifically, it proposes an architecture for providing low-latency AR education services in a classroom or a laboratory. Such a low latency is achieved thanks to the use of edge computing devices, which offload the cloud from the traditional tasks that are required by dynamic AR applications (e.g., near real-time data processing, communications among AR devices). Depending on the specific AR application and the number of users, the wireless link (usually WiFi) could be overloaded if the network has not been properly designed, and the overall performance of the application can be compromised, leading to high latency and even to wireless communication failure. In order to tackle this issue, radio channel measurements and simulation results have been performed by means of an in-house developed 3D ray-launching tool, which is able to model and simulate the behaviour of an AR-enabled classroom/laboratory in terms of radio propagation and quality of service. To corroborate the obtained theoretical results, a Microsoft HoloLens 2 teaching application was devised and tested, thus demonstrating the feasibility of the proposed approach.

1. Introduction

Education has thoroughly made use of augmented reality (AR) [1] for teaching students different subjects and skills like language spelling, high-school chemistry or to learn to play piano. AR can bring many benefits to the mentioned applications, including inmersiveness, collaboration promotion, remote guidance and gesture interaction. Moreover, some AR frameworks include voice recognition, which usually works fine on adults, but still struggles with kid voices [2]. Despite certain challenges, AR is really useful in education, since it increases student motivation, improves learning performance/retention, increases student engagement and encourages exploration and interaction [3].

Prior AR teaching works have proposed applications for engaging students during face-to-face or remote labs, regular lectures, storytelling and outdoor activities. In such scenarios, students frequently make use of their own smartphones or tablets, but some of the latest works have tested the usability of Head-Mounted Display (HMD) devices like Google Glass or Microsoft HoloLens. This article is focused on the latter, whose first generation (i.e., Microsoft HoloLens 1) has already been suggested for implementing educational applications for neurosurgical training, robot programming, or for teaching how to play the guitar. In spite of the previously mentioned works, a recent survey has found that there is barely any research on the use of AR in real environments and that most of the experiments of such works are difficult to replicate in other educational environments [4].

In contrast to the previously mentioned articles, this paper presents an AR teaching system for Microsoft HoloLens 2, one of the latest HMD devices, which was released recently in a limited edition in November 2019. The proposed AR system architecture makes use of edge computing devices so as to reduce latency response, to offload traditional cloud-based systems and to enable creating AR experiences that are shared among the teacher and the students. In addition, the system is designed having in mind the potential interactions that may happen with Internet of Things (IoT) devices in next-generation smart classrooms and campuses. For such scenarios, the paper proposes to perform theoretical simulations prior to the deployment of the AR educational system in order to guarantee its Quality of Service (QoS).

2. Design and Implementation of the System

2.1. Communications Architecture

Figure 1 illustrates the communications architecture of the proposed system. At the bottom is the AR-IoT device layer, which is divided into two sublayers: the AR device sublayer, which is composed by the AR devices used by the students and the teacher, and the IoT device sublayer, which includes the IoT devices (i.e., smart objects like sensors and actuators) that are present in a smart education environment.

Both AR and IoT devices are connected either through a WiFi access point (AP) or a gateway to the edge computing layer, where the AR educational services are executed. Specifically, the edge computing layer is conformed by two sublayers:

• AR Service Sublayer: it makes use of edge computing devices like cloudlets or fog computing gateways based on single-board computers (SBCs) to provide fast distributed responses. Teachers can act as local administrators of the devices on this layer to control which content is delivered to the student AR devices. Moreover, the edge computing devices of this layer enable sharing AR experiences, thus being able to synchronize in time and location the content shared simultaneously among multiple AR users.
• AR-IoT Service Sublayer: it allows for interconnecting AR and IoT devices seamlessly. The system presented in this paper uses Message Queue Telemetry Transport (MQTT) and a REpresentational State Transfer (REST) Application Programming Interface (API) to ease data exchanges among the multiple heterogeneous devices. Regarding the AR-IoT Bridge Service, it is a software component that connects MQTT and REST API requests, making use of certain auxiliary services when necessary.

When the edge computing layer is not able to respond to the requests from an AR or IoT device, it can ask the top layer (the institutional cloud) for help. The institutional cloud allows remote users to view and to manage the stored data and the provided services. Moreover, the institutional cloud can collaborate with other third-party services and external information repositories in order to fulfill the data requests from AR users and IoT devices.

2.2. Implementation

For the sake of brevity, the implementation described in this paper is limited to the components of the AR device sublayer, the AR service sublayer and the institutional cloud. No AR-IoT device interactions are demonstrated, but the interested reader can find in [5] further details on a similar implementation of the IoT device and the AR-IoT service sublayers.

AR Device Sublayer: HoloLens 2 Teaching Application

The core of the AR device sublayer is a HoloLens 2 AR teaching application that uses the following technologies:

• Universal Windows Platform (UWP) is a platform that allows for implementing video-based features like on-demand video-conferencing. For such a purpose, UWP can create a solid window where videos can be displayed, providing an easy way to move and resize the window, so that the user can position the videos on the most convenient place.
• Unity and Microsoft Mixed Reality Toolkit (MRTK) are used to implement the 3D-model visualization part of the application. MRTK provides certain features like gesture recognition and built-in buttons that simplify the creation of interaction events with the 3D model.

The following are the main features of the implemented HoloLens 2 application:

• Educational video watching. Videos can be automatically downloaded and stored on the device. The student can place the video-player window in the real world, wherever he/she finds it more appropriate. The user can play, stop and go forwards or backwards, visualizing the video as many times as needed.
• Remote guidance. Students can make direct video calls to remote teachers to ask questions and receive further guidance on course labs or exercises.
• Step-by-step instructions. Besides instructional videos, the developed software provides guidance through step-by-step multimedia content that includes the visualization of animations and 3D models, which can be moved, rotated and scaled. This feature is really useful during labs that require one to make use of certain tools or to assemble/disassemble hardware components.

Figure 2 shows some of the moments when an engineering student was making use of the HoloLens 2 application. Specifically, the pictures in Figure 2 show the main panel of the application (upper left), one of the instants when the student watched a teacher’s video on the procedure to be performed during a remote lab (upper right) and a step-by-step guidance menu (at the bottom) that illustrates the assembly of a part of an engine.

2.3. Edge Computing Layer and Institutional Cloud

The AR service sublayer of the edge computing layer implemented for this paper is based exclusively on a cloudlet (no fog computing gateways were used), which makes use of hardware that can be usually found in high-end PCs aimed at performing fast video processing tasks. Regarding the cloud, it ran on a Dell Power Edge R415 server with 16 3.1 GHz AMD Opteron cores and 128 GB of RAM.

These both layers enable implementing the most advanced features of the developed system:

• Teacher guidance: the teacher is able to control remotely when and how the AR content is displayed on the HoloLens 2 glasses used by the students.
• Student collaboration: students can share the same AR experience and thus collaborate among them to achieve a common goal.
• Environment interaction: the developed system can interact with surrounding IoT objects, allowing AR users to obtain information on them or to actuate on them.
• User interaction and collaboration monitoring. Since the HoloLens 2 application detects the actions performed by each student, it is possible to record them and then determine the student performance or his/her contributions during collaboration tasks.

3. Towards a Real Practical Use Case

Although the implemented AR application has been successfully tested and validated in a real classroom, a realistic practical use case will present significantly different conditions (mainly due to the presence of a high number of students that use AR devices simultaneously), which will affect the performance of the proposed system and application. This increase in the number of AR devices could be very demanding in terms of wireless communication capacity and latency. Therefore, the current WiFi infrastructure of the classrooms and labs may not support the QoS required for AR-based smart teaching applications.

Many technologies and solutions have been proposed for next generation teaching applications and interactive smart classrooms (e.g. IoT, edge computing, AR, blockchain) [6,7], and the mentioned potential and challenging requirements of capacity and latency of the wireless communication has led to proposals based on mmWave and 5G communication systems [8]. However, nowadays, the reality is that 5G networks are unavailable in the vast majority of teaching environments, and therefore, radio planning studies and/or modification of the current infrastructure (usually WiFi) is the common procedure to implement this kind of AR-based solutions when the number of required AR devices is high.

To tackle the previous issues, an in-house developed 3D ray launching (3D-RL) simulator was tested and adjusted in order to obtain QoS estimations within the proposed scenario. This tool, which has been previously used and validated for different environments and wireless communication technologies (including mmWave [9] and 5G systems [10]), could provide accurate results in order to analyze the feasibility of the proposed AR system when used at the same time by a high number of users (i.e., students). In this way, the results provided by this simulation tool can lead to optimized radio planning for the developed AR-based application. As an example of the performed tests, Figure 3 shows the scenario created by the 3D-RL tool and simulation results that illustrate the estimated received signal power for both 2.4 GHz and 5 GHz operating frequency bands of the HoloLens 2 Wi-Fi for an access point located at the middle of the scenario, on the ceiling.

4. Conclusions

This paper presented an AR teaching system based on Microsoft HoloLens 2. The AR system architecture was designed to provide low latency services thanks to the use of edge computing devices, and to ease the interaction between IoT and RA devices. In order to tackle potential QoS limitations when the AR system is deployed with a high number of students, preliminary WiFi radio channel measurements and simulations have been obtained through an in-house developed 3D-RL simulator.