Smart Home Automation—Use Cases of a Secure and Integrated Voice-Control System

Abstract

Smart home automation is expected to improve living standards with the evolution of internet of things (IoT) that facilitate the remote control of residential appliances. There are, however, several factors that require attention for broader successful consumer adoption. This paper focusses on three key barriers: (i) different underlying technologies requiring an integrated voice-based control for ease of use, (ii) lack of trust due to security and privacy concerns, and (iii) unawareness of the use of machine intelligence by users for exploiting the full potential of smartness. Voice-controlled home environments are possible with cloud-based solutions that are being deployed commercially. However, there are drawbacks due to non-standard voice channels and commands with delays in meeting the required response time for real-time services. Adoption is also required to meet with the expected goals of simplicity, security, and integration. To address these barriers, we propose a model integrating IoT services and wireless technologies for developing a secure smart home automation with a voice-controlled artificial intelligence system. We demonstrate the model’s application in a variety of practical use cases, by implementing a secure and smart voice-based system for an integrated control of several home devices seamlessly.

1. Introduction

The momentous evolution of Internet of Things (IoT) has enabled the realisation of smart homes of the future [1]. Ideally, IoT devices in a smart home environment can communicate seamlessly with one another via the internet, with intelligent capabilities for big data-based decision making [2,3]. While there is an expected improvement in the quality of human life, the current gaps in practical implementation form barriers in the adoption of smart devices in home environments [4,5].

Studies show that the potential value of smart homes can be realised if existing barriers such as interoperability among IoT systems, security and privacy concerns, and usability of IoT data gaps are addressed [6,7]. Further, there is lack of awareness for fully exploiting the capabilities of different IoT devices to use machine intelligence for a voice-based integrated smart home automation [8,9]. Currently, users consider health and fitness as the two main categories of applications for successful IoT adoption and a range of devices and appliances are still emerging towards establishing smart homes of the future [10,11]. Greater economic considerations are given to human productivity in the office environment as well as safety in factories or worksites for workplace automation. Hence, research and commercial development have focused on business-to-business applications rather than consumer applications [12]. However, some of the smartness with interconnected thermostat devices, self-guided vacuum cleaners, automated lighting, door entry/security systems, kitchen and laundry appliances could be employed to achieve chore automation, energy efficiency and other areas of economic value in a home environment [13]. It is important to have a customer focus in providing any smart service in home automation systems to enhance user adoption [14]. This forms the main motivation for this paper. Our aim is to propose a secure and integrated smart home automation model using a voice-based command and control. Further, to enable the perceived value for consumers, it is important to implement any proposed model, with typical use cases. We consider this approach for demonstrating how an easy-to-use voice-based smart home environment could be achieved.

In recent years, users prefer to interact with digital devices using voice as an effective and natural mode of communication [3,9]. Voice technologies have emerged to employ machine intelligence for a smart control of home appliances offering hands-free and eye-free operations along with using the IoT pervasive environments more effectively [15,16]. However, commercially available voice-assisted technologies lack personalisation, standardisation, security, and integration as they focus on only specific applications. Such off-the-shelf solutions are not capable of providing a common platform for all types of users, including the elderly, to interact with several disparate IoT devices in a home environment [17,18,19]. Recent studies discuss design implications and the requirement of voice-assisted features tailored to meet the needs of visually impaired people as well as older adults for a better adoption of smart home technologies [20,21]. Further, smart homes of today exhibit gaps in privacy norms with the deployment of several third-party smart services that can encroach beyond the boundaries of home, leading to security and privacy threats [22].

In this paper, we take a modest step to fill the gap in the literature with the aim to achieve a smart home automation system with essential features sustainable for the future. We propose a secure and integrated model, connecting IoT devices and appliances, to implement a smart home automation with a voice-based command and control approach. Our system leverages the seamless connectivity of IoT platforms to automatically identify devices over a securely deployed network. The main consideration in proposing the system model is to allow users to operate remotely, via their voice, with addressing the concerns related to usability, trust, and the overheads of managing several IoT devices in a typical home environment. Furthermore, by implementing a prototype of the system with a variety of use cases, our aim is to demonstrate the simplicity and viability of our proposed model for any smart home environment.

Overall, the key contributions of our work are as follows:

(i) Proposal of a unique model that combines low-cost, scalable, and simple to configure end-to-end security of IoT along with voice-controlled command operations of various devices in a smart home.

(ii) Application of a customised virtual private network (VPN) using the Raspberry Pi development platform with sensors, as a single embedded system to connect securely with personalised user authentication, granting access to authorised devices within a IoT network of a smart home.

(iii) Development of a voice-control system to recognise speech commands, using the smartphone of an authenticated user, to operate smart devices intelligently and securely via the customised VPN that connects the Raspberry Pi platform to IoT devices in a smart home.

(iv) Demonstration of our proposed model’s implementation and practical viability with devices such as a lighting system, music player, computer, and water tank, upon which to execute different speech commands to switch on/off certain sensors, to play a particular song, to perform a voice-based online search or to even monitor and control the water flow from a water tank remotely in a home environment.

The rest of the paper is organized as follows: Section 2 provides the background, highlighting previous research and contributions of this work. In Section 3, we describe our proposed model for a secure home automation system that is voice-enabled. In Section 4, we demonstrate the use cases of our proposed system and their implementation. Finally, we provide concluding remarks and future research directions in Section 5.

2. Background and Previous Work

In the past couple of decades, wireless communication technologies have evolved rapidly [23,24,25,26]. In the late 2000s, we implemented a remote-controlled irrigation system which established wireless communication among three different farms (vineyard, apple orchard and flood irrigated pasture) located at widely dispersed locations in the Goulbourn Valley of Australia [27]. During that time, 3G wireless technology was developed and our remote-controlled irrigation system was deployed in a new 3G network infrastructure. Real-time communication and remote access on such a large-scale deployment was achieved using remote desktop sessions through port 3389. Subsequently, as technologies evolve, we witness security breaches and vulnerabilities, and this type of remote access and control is no longer considered secure. In a more recent work, we deployed a remote-controlled water tank system, which allowed access to IoT devices via the smart phone with secure communication to the sensors of several water tanks as a large-scale deployment through a virtual private network (VPN) [28]. Sustainable development and successful user acceptance of such smart technologies depend much on the ease-of-use, operational integration, and establishment of trust, apart from economy and robustness. Hence, this paper considers the next level of our research advancement in developing an integrated voice-control system that offers a secure and personalised service to consumers. We take an initial step in this direction with the development of a smart home automation system in this paper.

A literature review of recent studies indicates that voice technology combined with the latest wireless technologies capable of seamless connectivity of IoT devices is trending towards successful consumer adoption [9,10,14,29]. The most popular context of its application is in Smart Home Automation, to serve as a personalised assistant [30]. Recently, a voice control system for smart home implemented with cloud data storage was studied [31]. The system was based upon Alexa Voice Service connected to Amazon Echo in a cloud environment. It has come to light that Alexa is “always on” and records all voice activity in the home even when it is not intended to be activated. Such data collection may raise privacy concerns, even though this is in line with the terms of use agreement that users click and agree to. In addition, breaches of these voice data have occurred. Over a period of time, technological advancements have opened up many ways of how this endless stream of personal voice information can be accessed and used, not in a way that users had intended, and the possibility of misuse has started to raise trust concerns [32,33]. Further, recent capabilities of artificial intelligence (AI) to interpret large swaths of an individual’s voice along with big data technologies have led to real concerns as to how this information might be used [34]. One impact could be to push direct targeted marketing campaigns for influencing an individual’s purchasing choices. AI-induced voice recognition has enabled many banking and government organisations to switch to online voice identification. Voice profiling and the storage of this biometric data with an individual’s profile can even lead to identity theft. Overall, we find that the “always on” voice data collection in such cloud-based systems has drawn the spectre of public opinion and concerns [5,15,35].

The solution to the above-mentioned concerns is to leverage the positive aspects of voice-based AI engines that have become very efficient in terms of software development and can now be run by local processing elements rather than in the cloud data centres. Since home automation systems require only small-scale deployments unlike commercial industry automation systems, voice command datasets are relatively small and can even be run by devices such as Raspberry Pi [7]. This is advantageous as it brings the control back to the individual user and mitigates the leakage of personal voice information. Hence, this research project aims to develop a voice-controlled AI application to run on a Raspberry Pi to control various IoT devices over the internet using a mobile phone in a secure home environment, including the water tank in a remote backyard location. This paper brings together several key elements from earlier work [27,28]. In this research study, we advance further to securely integrate various IoT devices wirelessly at home along with the enhanced features of a light-weight voice-assisted AI system. Thereby, we develop the initial steps to pave way for a sustainable and acceptable home automation towards enabling remote command and control capabilities at the local home computing level without the need for large-scale AI capabilities from a cloud environment.

3. Proposed Model for a Secure Smart Home Automation System

In recent years, technological innovations have provided tremendous growth in IoT. This combined with advances in ubiquitous computing are converging technologies for smart homes of the future [1,3]. However, security and privacy (and trust), form the key barriers in the expected adoption. While research shows several ways of implementing security in the IoT context, the main concerns are lack of simplicity, adaptability, integration, and low-cost deployment in a smart home environment. We propose a secure and integrated smart home automation system as a modest initial step to address these existing limitations. The proposed model aims to:

(i)

Combine low-cost, scalable, simple to configure end-to-end security;

(ii)

Apply a customised virtual private network;

(iii)

Employ a voice-controlled AI system to recognise speech commands;

(iv)

Demonstrate the practical viability through an easy-to-use implementation with typical use cases in a smart home environment.

For every device in a smart home, the proposed model is aimed at providing an integrated voice control system with a secure connection to the IoT network. This is achieved by establishing end-to-end security using uniquely customized VPN technology. A practically viable deployment is implemented with smartphone-based speech recognition, for command and control of the devices. More importantly, we aim to demonstrate the practical application of the model by implementing a smart home automation system in a typical home environment with selected appliances and IoT devices to securely connect and operate with voice-assisted commands.

The proposed smart home automation system ensures the security and privacy of each user by connecting and configuring the IoT devices using OpenVPN and with a smartphone to control the devices, enhancing our previous work [28]. Users will be able to interact with the system with non-native English using a machine intelligent software platform. This is programmed in Python language with API commands to communicate with simple mobile phones that do not require a sophisticated voice system. We employ opensource libraries to develop the required voice recognition component of the system. This is used to detect different speech sounds and voice-based commands to operate various IoT devices to realize the unique advantage of having both convenience and security. Low-cost embedded controllers process each command and then send it to the appropriate device within the smart home environment through its relay controller. We make use of the Raspberry Pi platform as it provides several options to connect IoT devices, home appliances and sensors in a simple and efficient way. A pictorial representation of the proposed model is shown in Figure 1. This shows how a customized VPN is deployed to securely connect a mobile phone to the Raspberry Pi wirelessly to send the user’s speech commands. The proposed integrated voice-recognition system executes user commands to operate, monitor and control various devices in a smart home.

As shown in Figure 1, the proposed smart home architecture consists of three main components: (i) a secure VPN customized to connect smart devices to the Raspberry Pi, establishing end-to-end security as one embedded unit in a smart home; (ii) an integrated Smart Home connecting the Raspberry Pi breadboard directly through a relay driver to control the sensors and devices such as a lighting system, music player, water tank, air-conditioning unit, computer and other external smart systems via the internet; (iii) a Mobile Interface with a voice-based recognition module for authorized users to give speech commands through their smartphone’s microphone to communicate securely to the Smart Home devices. These components work together via the secure IoT network established using the Raspberry Pi platform to cater to the user-based command and control system.

4. Implementation of a Secure Smart Home Automation System

We demonstrate the application of the proposed system to a unique context of a scalable IoT infrastructure within a smart home environment. Privacy and security are the main barriers for a practical implementation of home automation systems and several studies have identified the key requirement of IoT security [36,37]. With the background of a comprehensive study on the vulnerabilities and security attacks of an IoT network, we advance further in this research work to address these [38,39,40]. Firstly, we implement an end-to-end security model that we had developed previously [28], enhancing it to use a uniquely customized VPN technology for the smart home use case in this work. We address the high overheads and complex configurations of OpenVPN by employing smart mobile apps that readily support VPN client interfaces to access the smart home devices instantly in a securely authenticated manner, as shown in Figure 2; we demonstrate the securely established connection to the IoT devices, with the water tank use case as an illustration in the home environment.

In the next step, the Raspberry Pi controller is connected via the secure VPN to the display device and the IoT devices such as temperature sensors, lighting systems, music players, mobile phones, water tank, and other smart home appliances. Figure 3 shows the actual deployment of the Raspberry Pi breadboard and selected sensors and IoT devices. As shown in Figure 3, we integrate a relay channel, humidity and temperature sensor, distance sensor and voice input sensor with the Raspberry Pi breadboard. It illustrates the connections of our system to the IoT server to facilitate control of these devices via VPN wireless technologies.

In this work, the required programming codes to serve as a mobile app for controlling the sensors that were connected to Raspberry Pi. Figure 4 shows the deployment of a water tank’s sensors and valves connected to the Raspberry Pi via a breadboard in the initial prototype testing phase. We tested the codes using an opensource software tool called NodeRed using NodeJS and JavaScript code. Using the mobile application, we could control the flow of water in and out of the tank. Once the water tank was successfully deployed and tested within our secure home smart automation system, we used the breadboard to extend the number of ports on the Raspberry Pi to connect other devices of the smart home. We customized the codes for each sensor and employed voice-based commands developed in Python to operate various devices in the smart home. The AI engine for voice recognition was then trained to learn various home user voices to personalize the home automation system.

In the final evaluation phase, the voice recognition module (robot) of the AI engine was implemented using a machine learning approach. This was tested to identify the user voice as well as undergo supervised training iterations after each evaluation, to enhance its robustness. We tested different scenarios by sending speech commands to perform specific actions to control various devices in the smart home. The speech recognition robot was initially trained using music searches in a browser. This was then extended using a pre-defined set of keywords such as on, off, open, close, date, water level, and others relating to weather parameters as well. Figure 5 demonstrates the output of the training underwent by the robot to display the current date when voice-based command was used. For the voice-based command given by the user such as “What is the date today?”, the robot recognizes the command. The robot interprets the command and the current date is displayed as the result. These tests performed show that the speech was correctly translated to text in English. After the learning process, the robot could obtain the stored data from the computer connected in the secure IoT network in the smart home for the correct voice recognition of different users. Figure 5 also shows the results of speech recognition training, including user-friendly spare time conversation of the robot by saying “I’m Listening” to invite the user to make further commands.

We implemented the use case of searching the web browser for various needs of the home user. For instance, Figure 6 shows how a voice-based command is used to search in the Web browser to play any music based on the user’s requirement. The robot is also intelligent to respond suitably with user errors. For instance, if some music is already in play mode when the user prompts to play any music, the robot will respond to the user that the music is playing, as illustrated in Figure 6. Another application of the robot that is useful for people with visual impairment or any disability is using voice to browse the Web for shopping. Figure 7 shows the result of voice-based search for an auto-recognised user to seek some information about furniture from the Web. The page located on the Web could be even printed or saved in the mobile phone as required by the user.

We established the robustness of the voice-based control of the smart home automation system by training the robot to recognize and differentiate homophones or words that sound similar. We tested this with homophones related to the water tank device operations to perform the correct actions. For instance, we tested the system by asking an incorrect question “What is the bottle label?”, which sounds similar to the desired question “What is the water level?”. The robot demonstrated that it could learn well to identify homophones present in a user command that were incorrectly used. When the correct question was asked, the system activated the data from the right sensor in the water tank to output the water level. For the water tank use case, we have installed a sensor for detecting the water level and two smart sensor valves, one for water inlet (Valve 1) and the other for water outlet (Valve 2). The program code was written in Python language for setting various rules and to calculate the percentage of water level from the water tank sensor. For instance, the speech recognition robot was connected to the water sensor to output the data as the calculated percentage of water. Figure 8 demonstrates the results of the tests performed to establish the robustness of the system via a mobile app. It shows that the speech recognition robot could output the correct result of 67% as the level of water in the tank.

Further, we explored alternative ease-of-use approaches in addition to voice recognition AI by employing smartphone controls effectively. To enhance the usability of the smart home automation, we deployed the user interface in a mobile app. This would cater to end-users who would prefer to monitor the water level and operate the water tank remotely in the home environment system using the mobile app. Figure 9 demonstrates a dashboard display to illustrate how the status of the second valve outlet (Valve 2) sensor of the smart water tank is automatically set to “on” and is shown by Water Out toggle and the current water level in real-time. The display continuously shows the water level reducing as the water is let out remotely from the water tank. By having this control setting, the smart home automation system allows the user to open the water tank valve remotely using the smartphone’s app. This feature will allow water to be used for situations such as watering plants in the garden when the user is in a remote location. The configuration for the controls could be customised by the user such that “if the water level is greater than 15% of tank capacity, turn ON the valve”. In addition, using Python libraries, the smart home automation system is able to gather external weather parameters outside the home environment such as atmospheric pressure, humidity and temperature. Integrating these with the water tank controls would lead to more intelligent control settings and automatic actions.

Overall, the proposed smart home automation system is developed to securely adapt for deploying different IoT use case scenarios in a home environment as demonstrated in this work and to support our future ongoing research in this direction. The proposed framework is sustainable, catering to different user interfaces and communication preferences, as a home user’s requirements could change over time and with the evolution of new wireless technologies and IoT devices in smart homes. Future research considerations could also include the use of suitable energy management schemes to embrace the green energy agenda and to address socioeconomic challenges in improving the consumer service satisfaction of smart homes for successful adoption [41,42].

5. Conclusions and Future Research

This paper proposed the convergence of IoT in a secure infrastructure model, distributed across multiple platforms, in remote locations with AI voice control. We demonstrated an integrated implementation of the model for smart home automation system by considering several challenges, principally:

(i)

The requirement of different underlying technologies to integrate voice-based control for ease-of-use;

(ii)

Lack of trust and usability due to security/privacy concerns and affordability;

(iii)

Unawareness of the application of machine intelligence by users for exploiting the full potential of smartness to control IoT devices in a home environment.

This original research work resulted in three key contributions. Firstly, we developed a voice-recognition AI engine as a personal assistant robot to successfully become trained with user voice and to remotely control various IoT sensors and devices using a smartphone. Secondly, the proposed unique model provided a low-cost, scalable, and simple to configure end-to-end security of IoT devices addressing privacy threats that could be associated with any third-party services. Thirdly, several use cases in a typical home environment were illustrated to demonstrate the practical viability and deployment of a secure and easy-to-use system, paving the way for a sustainable smart home consumer adoption.

Future work would explore several other use cases to arrive at a more comprehensive suite of integrated IoT devices in a smart home. The voice AI engine will also undergo deep learning approaches for its training. Further, user experiences with the system to enhance the user interface and service satisfaction will be explored.