*4.2. Communication*

Resources in LoT@UNED are understood as a standard communication channel using the MQTT protocol (for interaction commands) and the required software to "control" and "program" the device (a Python distribution, sensor access libraries, etc.). All these resources define a run-time environment that depends on the service that the end-user wants to offer.

The MQTT protocol has been selected due it its popularity and specific way of working. This paradigm uses the message as a fundamental unit of communication. The participants in the solution are the ones who give meaning to the message. The roles of subscriber, editor (publisher) and broker are usually assigned. Subscribers register their interest in certain messages from one or more editors/publishers. This interest is handled through the broker, which is the responsible for managing the flow of messages between publishers and subscribers. Once the editor generates a message, it is delivered through the broker so that it sends it to the interested subscribers. Hence, the MQTT protocol is able to simplify and facilitate the synchronization between all nodes and jobs available in the IoT platform.

Each message with MQTT is associated with a topic, so the broker and the subscriber can identify the message. The usual topics are "data", "status" or "alarm"; and they act as semantic labels of the information carried by the messages. The targeting of different topics allows administrator to check the health of the IoT solution and to monitor its network communication in a fast way.

An example of a message's flow for MQTT is shown in Figure 4. In this particular case, a temperature sensor (publisher) is sending the temperature using the topic data to the MQTT broker. The MQTT broker delivers the topic messages to the two subscribers (computer and mobile device), which previously registered their interest in the data topic.

**Figure 4.** MQTT flow of messages for topics and subscriptions.

#### *4.3. Software Architecture*

#### 4.3.1. Virtualization and Orchestration

The run-time environment can be "packaged" using already known virtualization technologies [41], such as Docker [48,49]. Docker is based on the use of containers that define a prefabricated execution environment. Docker can be deployed in any infrastructure that supports this technology. The definition of a service is based on the execution of one or more Docker containers, although usually only one of them is necessary. Specifically, in the case of services associated with experimental sessions with IoT devices, the container is executed on the same device which provides sensors and runtime. However, in more advanced practices it is possible to run several containers on the same device or several at the same time. This orchestration of containers allows identifying scenarios of collaborative use where the sensors of several devices [50,51] are used in coordination to obtain a specific purpose (traffic control at crossings with several traffic lights, data from the environmental sensors of several drones flying over aerial areas for pollution indices, etc.).

Regardless of whether the service requires the execution of one or more containers, it is essential to provide an orchestration layer of those containers providing:


There are several orchestration system solutions available, such as Docker Compose [53], Docker Swarm [54] or Kubernetes [55]. For the orchestration and control layer of LoT@UNED, Kubernetes has been selected since it eases the managemen<sup>t</sup> of the device containers and the supervision of all the executed containers in the infrastructure through its dashboard [56]. This characteristic is essential to provide a continuous service delivery of the IoT laboratories in the infrastructure and an availability close to 24 × 7.

The main drawback of the Kubernetes deployment model is associated with the dedicated use of one of the infrastructure devices as the master node of the orchestration layer. This makes the orchestration layer vulnerable to the fall of this node and, therefore, it is essential to monitor it in real time, and to include automatic restart procedures.

#### 4.3.2. Execution Services

The complete architecture of the execution services into the LoT@UNED infrastructure is shown in Figure 5. It shows how each IoT device contains a Docker run-time environment and acts as a slave node of the Kubernetes cluster. On the control plane, there is a device (a Raspberry Pi) that acts as a master of the cluster and it communicates with the broker (IBM Watson IoT) to ease the communication channel (MQTT) with the IoT devices to be used during the interactive sessions (Shell Service).

There are currently three base containers that are identified with the "services" offered by the LoT infrastructure:


**Figure 5.** Technical solution for Labs of Things at UNED (LoT@UNED).

With these computing services (of execution) it is possible to build and define laboratory activities in different areas using the flexible infrastructure of LoT@UNED. The fundamentals of the definition of learning services and the workload protocol to define them using the tools/applications provided by LoT@UNED are detailed in the next subsection.
