**2. Methods**

The methodology followed in this study includes the following steps:


students. A methodology based on a typical flow of the instructional design [23] has been used. The first assumption made to start the application of instructional design is that the setting up of a laboratory is not an isolated task. It should be integrated into the subject objectives as another element of the instructional design. It should be a way to acquire a competence or skill related to the subject. In addition to this, the methodology involved in a teaching/learning process on distance, as our case is, implies much more periodical virtual attendance and interaction with/among students than with a traditional methodology. All these learning resources presented to students have to be available to them all time, and they expect innovation teaching approaches to improve the quality of courses. This expectation is more noticeable in Engineering subjects than only theoretical subjects such as maths when practical skills have to be acquired by students. Additionally, supporting many students becomes a real challenge when technologies are implemented and deployed in virtual courses. The instructional design methodology is made up of the four phases, as it can be observed in the Figure 1:


**Figure 1.** Instructional design phases for a new laboratory.

#### **3. State of the Art**

To understand the complexity associated with the development of IoT solutions, it is important to understand the organization of these systems, usually in a set of layers that implement specific functionalities [3,24], as it can be observed in Figure 2. Usually, these layers are classified using a criterion of physical proximity to the environment and processing capacity of the components that integrate it [25]:


**Figure 2.** Fog computing approximation for Internet of Things (IoT) solutions. Figure available on [34].

Other important aspects to have into account when analyzing IoT applications are the communication technologies and protocols (such as HTTP, MQTT [25], CoAP [35] and others [36–38]), and cybersecurity.

Analyzing the literature, most of the educational IoT labs are for hands-on experimentation. Among those designed to experiment online out of laboratory facilities, many were just pure simulations or virtual labs [39]. An example is the work of Patil et al. [40], who describe an IoT virtual lab to allow students sensing and retrieving simulated data from the cloud using Python, as part of the modeling and simulation lab.

Only a few remote labs can be found to allow remote experimentation. An example is the work of Tunc et al. [41], who presented an IoT remote laboratory designed only for cybersecurity experimentation.

El-Hasan [42] introduces an IoT mobile dashboard to allow off-campus practices through a system including sensors, controlling and interfacing kits, cameras and others. Basically it only allows the modification of certain parameters to switch the direction of rotation of a motor by changing predefined values of voltage, current and power as well as other required parameters, such as speed and torque.

Fernandez-Pacheco et al. [43] describe an Arduino remote lab using a Raspberry Pi as a server, but it is only intended for microcontroller programming (Arduino), not for IoT purposes (cloud, Python programming, IoT protocols, cybersecurity, etc.).

Leisenberg [44] presents a remote lab based on Raspberry Pi for movement analysis. Students should write the code to analyze real time images coming from a webcam. Again this system is not intended for full-cycle IoT purposes.

Rajurikar et al. [45] present a system for IoT protocols experimentation, mainly REST and CoAP. They connected several sensors through an Arduino board and a Beagle Bone device to a cloud platform. This cloud environment is only intended for End point Data Acquisition and decision-making.

From the previous work, we can conclude the essential characteristics to be covered in a IoT laboratory/environment are:


It can also be observed that none of the analyzed works allow the implementation of all the essential features required for learning the complete development cycle of the IoT applications. Only Rajurikar et al. [45] complete the implementation and support of three out of the five features. As a consequence, it is necessary to have an environment that complies with all the characteristics.

Table 1 compares the functionality implemented in each of the IoT remote labs found on the literature and the authors' proposed system. It can be observed that our proposal covers all the features included in the study (edge programming, fog programming, cloud dashboards and analytics programming, protocol experimentation and cybersecurity), whereas other approaches only cover one or several functionalities. This way, the development of LoT@UNED implements the full set of features, advancing in the development/research of this type of environments.


**Table 1.** Functionality comparison of the state of the art on IoT remote labs.

To check how these features are implemented in the authors' proposed system, the following section describes the LoT@UNED platform more in detail.

## **4. Solution Description**

## *4.1. Hardware Architecture*

The LoT@UNED platform implements the edge layer through a set of IoT devices (i.e., Raspberry Pi boards). Each device is connected to the services of layer 2 (Cloud IoT Layer) using the MQTT protocol. This way, students can develop the skills and abilities corresponding to layers 1 and 2. The services of the Cloud IoT layer are provided through the IBM cloud provider, and specifically using the IBM Watson IoT service. The service for storing sensor/device data is also implemented in this provider. The non-relational database Cloudant is used for this purpose [46]. This cloud storage service is used in the dashboard and assisted decision layer (Cloud Layer). Again, the IBM Watson Studio service from IBM Cloud is used for the development of the analysis and machine learning algorithms based on the data stored in the Cloudant service. LoT@UNED has been designed flexibly to be able to use other specific services from other providers for the cloud layers, so that experiences can be designed for the development of IoT solutions using the AWS or Google services (AWS IoT, Cloud IoT, S3 or Cloud Storage). The basis of a previous generation of our Web of Things (WoT) platform was presented in [10].

The structure of the platform focuses on the availability of low-cost Raspberry Pi devices, which are connected through a cluster that eases the device connection to the Internet. There are two logical groupings available that facilitate the connection of new devices. The first group uses a specific rack for the managemen<sup>t</sup> and connection of the devices, as shown in Figure 3a. The characteristics of the rack can be found on the website of the BitScope provider [47]. It allows grouping up to 40 Raspberry Pi (Model B) devices, facilitating the connection to the electricity grid, the connection among devices and the Internet connection. It can also be installed in a traditional rack, facilitating the managemen<sup>t</sup> of the cluster itself. Its high cost and the inability to add specific sensors in an individual way for each device can be noticed as its main drawbacks.

The second grouping, as shown in Figure 3b, uses cheaper and more flexible components in terms of device separation. This fact allows us to add specific sensors (cameras, GPS, temperature sensors, etc.) without storage problems or cluster connectivity. In fact, the storage can be increased by lateral fixings that support the structure of the cluster. Proof of this is the specific configuration that is used in the example specified in the following section. This type of configuration is deployed outside of the clusters to ease the replacement and managemen<sup>t</sup> of the sensors in order to provide the necessary redundancy for the services that uses this configuration.

**Figure 3.** Devices clustering: blade rack and cheap setup. (**a**) Blade rack. (**b**) Cheap setup.

The variety of setups (clustered or individual) allows the logical grouping of the services offered by LoT@UNED and, also, the redundancy necessary to provide a stable learning service for students. For example, in the case of the setup of Figure 5, there are three exact replicas that are managed by the software developed, installed on the base image of the Raspberry Pi and integrated transparently within the service availability (a service, three concurrent accesses). The base image of each Raspberry Pi card comes with the connection services to the IoT service in the cloud, which allows the self-registration of the devices. This self-registration allows us to automatically have the inventory of the devices and the available setups for the entire IoT environment of LoT@UNED. Each device will add specific information about the type of educative service offered (it can be more than one), which will allow the activity manager to decide on the assignment of each environment for the student who requests it.
