IoT-Based Human Fall Detection System

Abstract

Human falls are an issue that especially affects elderly people, resulting in permanent disabilities or even in the person’s death. Preventing human falls is a social desire, but it is almost impossible to achieve because it is not possible to ensure full prevention. A possible solution is the detection of human falls in near real-time so that help can quickly be provided. This has the potential to greatly reduce the severity of the fall in long-term health consequences. This work proposes a solution based on the internet of things devices installed in people’s homes. The proposed non-wearable solution is non-intrusive and can be deployed not only in homes but also in hospitals, rehabilitation facilities, and elderly homes. The solution uses a three-layered computation architecture composed of edge, fog, and cloud. A mathematical model using the Morlet wavelet and an artificial intelligence model using artificial neural networks are used for human fall classification; both approaches are compared. The results showed that the combination of both models is possible and brings benefits to the system, achieving an accuracy of 92.5% without false negatives.

1. Introduction

Human falls can occur at all ages, provoked by the loss of balance or the inability to regain balance. The World Health Organization (WHO, www.who.int, accessed on 1 February 2022) and the Prevention of Falls Network Europe (Pro-FaNE) define a fall as an event that inadvertently takes a person to the ground or to a lower level, excluding intentional situations [1]. In [2], it was defined that a fall is “an unexpected event in which the participant comes to rest on the ground, floor, or lower without known loss of consciousness”. Although human falls are common at young ages, it is at advanced ages that they can be a significant health risk. Falls can result in dependence, loss of autonomy, confusion, immobilization, and depression, negatively affecting the person’s day-to-day activities [3]. The health issues resulting from human falls affect all countries, but especially countries where there is a significant rate of elderly people. Moreover, the share of elderly people in the population is increasing. In Europe, people aged 65 or above represented 20.2% of the population in 2019 and are predicted to reach 29.4% in 2050 [4].

According to [5], falls can be classified according to their cause, as being accidental falls, unpredictable physiological falls, and predictable physiological falls. Accidental falls are unpredictable falls that result from external factors that affect people with no risk of falling. Unpredictable physiological falls represent about 8% of falls and affect people with no risk of falling, but they are the result of physiological factors, such as pathological fractures or seizures. Predictable physiological falls are 80% of falls and affect people with a risk of falling, mainly resulting from physiological changes. To identify people with a risk of falling and to evaluate the likelihood of falling, the Morse Falls Scale (NDE) can be applied to patients and elderly persons [5].

To minimize the negative impact of human falls, it is important to act fast when the fall occurs and to supply the proper warning to health support services [6]. It is impossible to completely prevent falls, but with physical exercise and technological solutions, falls can be prevented to a certain degree [7]. The use of internet of things (IoT) devices can benefit fall detection solutions by enabling the continuous and remote monitoring of environments and contexts [8]. The wearable IoT devices can go with the elderly person and create walking profiles and detect falls in near- or real-time [9,10].

This paper proposes an IoT-based solution for fall detection, using non-wearable and non-intrusive devices and a layered architecture to process the data near-real-time. This work is an extension and evolution of the work published in [11], where a Morlet wavelet is used for classification in the fog-layer. In this paper, a new classification model is proposed using artificial neural networks (ANN) to be deployed in the system. The main goal is to evaluate the two classification models. This paper also proposes a new IoT device using the ESP8266 microcontroller. The case study presented in the paper uses a dataset produced with the proposed IoT device and compares the two classification models.

The paper is structured as follows: after this first introductory section, Section 2 presents related works on fall detection systems, considering the use of artificial intelligence and IoT devices. Section 3 describes the adopted method and the proposed solution, including the proposed main architecture, the proposed IoT device, and the proposed classification models. The use of the proposed solution is illustrated in Section 4, where the main results and discussion are presented. Section 5 presents the main conclusions of this work.

2. Related Works

This section describes related works on fall detection systems, namely focusing on solutions that use IoT devices (i.e., wearables and non-wearables) and classification models based on artificial intelligence models or mathematical models.

Fall detection systems can be classified as wearable solutions, non-wearable solutions, or fusion solutions [12]. Each of these approaches has its pros and cons. While the wearable solutions stand for most of the proposed solutions, enabling a continuous tracking of the user [12], they raise some issues when targeting the elderly population, namely the required charging as well as daily usage issues. The non-wearable solutions can overcome these issues, enabling the monitoring of spaces without the need for people to use the devices on them. However, non-wearable devices are limited in space, not being able to detect human falls beyond their monitoring radius. The fusion solutions can be described as a combination of several solutions able to fuse several data sources. Although this paper proposes a non-wearable solution, it uses an IoT-based architecture that can handle wearable devices. Additionally, the classification model in the fog layer can handle data provided by several IoT devices.

A wearable approach is proposed in [13], where a 3D accelerometer is used to monitor data that is then transmitted to a big data server using the Ipv6 over a low power wireless persona area network (6LowPAN). In [14], the data acquired by a 3D accelerometer is used to feed a support vector machine (SVM) trained to classify human falls with an accuracy of 80%. A solution using the edge-computing layer is proposed in [15], where multiple sensors were used in different positions on the human body. The collected data was used to train and evaluate a long short-term memory (LSTM) neural network that the authors used to detect the ideal body positions to deploy sensors. The authors identified the best complementary positions as the waist and the wrist. The combination of these two sensors enabled the training of the LSTM, achieving an accuracy of 95.8%. Trying to improve the results of previous literature, the proposed solution in this paper will combine several sensors along-side accelerometers.

In non-wearable solutions, it is common to find solutions based on computer vision [16] and ambient sensors [17]. In [18], a computer vision solution using the k-Nearest Neighbors (KNN) algorithm is proposed to classify video streaming. The authors proposed this solution to address the common issue of elderly people forgetting to wear wearable devices. The proposed classification model resulted in an accuracy of 97.6%. A computer vision solution based on deep learning classifiers is proposed in [19] by combining a convolutional neural network (CNN) and an LSTM neural network to detect human falls. This solution used kinematic data recorded videos and achieved accuracies slightly above 90%. A non-wearable and non-intrusive solution is proposed in [20], where the IEEE (Institute of Electrical and Electronics Engineers) 802.11 signals are monitored to detect distortions in the environment. This solution was tested in a controllable environment but showed remarkable results by achieving a detection precision of 90%. The results published in previous studies are truly relevant, achieving high numbers of precision and accuracy. The solution proposed in this paper will evaluate and compare two classification models, one mathematical model and one artificial intelligence model, to be able to detect human falls near real-time and to drop false negatives.

The use of artificial intelligence, such as machine learning algorithms, in fall detection solutions, can boost the solutions and allow them to adjust to a variety of environments. However, future developments should also address the need for the expandability of artificial intelligence algorithms [21]. Explainable artificial intelligence is becoming a concern and is an aspect that must be considered in future fall detection systems. Going in this direction, the continuous tracking of persons can be conducted to better understand, identify, and record changes in the persons’ moves or poses. In [22], the authors proposed a solution based on computer vision that can pave the way in the direction of better identifying and understanding human structural information.

Although artificial intelligence models are widely explored in the literature, it is also possible to use mathematical models in human fall detection systems. In [23], a solution for fall detection using a wavelet transform approach is proposed to analyze the impact moment. This solution achieved a 90% accuracy when using a dataset with 1170 signals of acceleration records. The analysis of vibration signals was conducted in [24], where a comparison was made between wavelet transform, the Fourier transform, and the mel-frequency cepstrum. The proposed solution applied an SVM-based classifier. To avoid false positives, the authors proposed the use of passive infrared (PIR) sensors due to their ability to detect the human body’s radiant heat. The use of mathematical models can improve human fall classification, but they usually need more computational processing. Therefore, in this paper, an alternative artificial intelligence model will be compared against a classification model based on the Morlet wavelet.

Fall detection solutions can be divided into two main components, the hardware that enables the acquisition of data and the software that enables the classification of events. The use of hardware and software is interconnected, so the choice must be made in a coordinated way. The classification model, as seen in this related work section, can be conducted using data mining algorithms and machine learning models. In this paper, a non-wearable solution using multiple sensors will be adopted. For the classification, two models are proposed and implemented, one based on the Morlet wavelet and one based on artificial neural networks. The wavelet transform allows the analysis of time series with nonstationary signals [25].

3. Method and Proposed Solution

This section presents the method applied in our research and proposes the solution that was conceived, developed, and tested to address near-real-time human fall detection.

3.1. Method

To address the issue of human falls, the authors started by exploring existing datasets with falling and activities of daily living (ADL) data. These datasets, such as UMAFall [26], SisFall [27], DLR [28], MobiFall [29], and UP-FALL [30], allowed the study of signals acquired by a different type of hardware, excluding the collection of images. Image processing was not explored in this work due to its invasive nature. This paper proposes a non-invasive and non-wearable solution for human fall detections inside facilities. This solution considers an IoT device that was prototyped and two classification models.

The prototype of the IoT device enables the real acquisition of data that will allow the testing and validation of the classification models. To do so, two different datasets were created using a human-body model to generate fall data and ADL activities.

The datasets created had raw data from sensors. These data were cleaned, and new data were added based on raw data, such as the spectral analysis that considered the accelerometers data. The dataset also has the corresponding activity: human-body model fall, chair fall, cellphone fall, water bottle fall, and noise values.

Using the created datasets, the authors created two classification models, one based on the Morlet wavelet and one based on artificial neural networks. These two models were conceived considering the distributed architecture of the proposed solution, where a three-computational-layered architecture is promoted.

The results of the proposed solution, together with the proposed classification models, were evaluated according to the accuracy and false negatives rate. Regarding human falls, the existence of false negatives can create dangerous undetected situations where the system was not able to detect a fall. On the other hand, false positives, if not constant, are not harmful in the proposed solution as it calls local human help.

3.2. Proposed Architecture

The proposed solution is based on IoT architecture, where three computational layers coexist and complete each other, as can be seen in Figure 1. These layers enable data filtering and treatment among layers and allow scalable solutions where new nodes can be added. The solution can be seen as a hierarchical tree where a node can have multiple leaves. For deployment, the edge- and fog-layers should be installed in the facility (e.g., residential house and elderly home) while the cloud-layer can be unique to all the facilities. The present work developed and tested in this paper considers the edge-layer (i.e., IoT device) together with the functionalities of fog-layer that were developed in Python using a standard desktop.

The edge-layer together with the hardware sensors, compose the IoT device that must be placed in each room of the facility (e.g., bedroom, hallway, and bathroom). The main goal of this layer is to acquire in real-time and have continuous monitoring of the environment. The IoT device can detect off-normal values and collect a time window of data. The IoT device uses a local buffer to enable the collection of a time window composed of three events: before the off-normal event, during the off-normal event, and after the off-normal event. These three events will allow the following classification of the off-normal event. The edge-layer of the proposed solution should be implemented with a microcontroller able to read sensor data. More details on this layer will be provided in Section 3.1.

The fog-layer is used for the classification of events. The edge-layer, when detecting an off-normal event, will send the time series on the collected time window to the fog-layer for classification. The location of the fog-layer in the end-user facility will make the classification faster and will not need an external communication connection. According to the proposed solution, the fog-layer device will handle the classification of events produced in several IoT devices. The classification models proposed in this paper were implemented on a standard desktop. However, for real pilot deployments, the authors recommend the use of a microprocessor, such as a small server of single-board computers (e.g., the NVIDIA^® Jetson Nano™ and the Raspberry Pi 4 Model B). For positive classifications, the microprocessor must warn the facility health supporters to indicate a possible human fall in a given room. The fog-layer also has the functionality of storing the events that were classified as falls. After the following human validation of these events, the fog-layer will send the events to the cloud-layer to be considered in future classification model training.

The cloud-layer is responsible for the system evolution and classification model updates. The data sent from the fog-layer to the cloud-layer will be used to improve the classification models by training new models. The training will be conducted in the cloud, as it has more computational resources, and it will be sent to all fog units. Additionally, the cloud-layer should implement a push notification server and a storage server to communicate with the users and to store anonymous data on fall events. Although the cloud-layer is not used to detect falls, it is a key feature in the proposed solution that will enable the continuous improvement of the system.

3.3. Edge-Side IoT Device

The edge-side IoT device that is represented in Figure 1 is the eye of the solution and is the equipment that will enable the collection of data. The development of this IoT device was conducted using two prototypes with different microcontrollers while keeping the same sensors.

The microcontroller used in the first prototype version was Arduino UNO R3′s Atmega328. The first prototype enabled the data acquisition and testing using the Morlet wavelet. These first results can be seen in [11]. The first IoT device prototype used a USB connection to transmit the data to a Python script that processed and classified the fall data. In this work, we used a second version prototype using an ESP8266 (NodeMCU) microcontroller to facilitate the data transmission over Wi-Fi. However, it can be easily updated to the new ESP32 microcontroller.

The sensors used in this new prototype are the same as the previous one: an analog sound sensor, an MPU-9250 accelerometer, a GY-521 accelerometer, and an HB-100 Doppler sensor. The proposed IoT device is intended to be deployed in contact with the room’s floor, and the accelerometers are used to measure the floor vibrations. Both accelerometers are in close contact with the floor surface.

The sensors are read without any delays in the microcontroller, being the readings period defined by the internal clock speed of the hardware, in this case, ESP8266. In this new prototype, the sensor data is transmitted using the Message Queuing Telemetry Transport (MQTT) protocol. As stated before, only off-normal events are transmitted for the following classification.

The off-normal events were detected using level thresholds in each sensor. If such thresholds are triggered, the IoT device creates a time window where each sensor data is stored and transmitted. These thresholds can be communicated from the cloud-layer and monitored each time a sensor is read. The time window is created using a buffer where previous values are stored (i.e., before the trigger) and a buffer that stores data for half a second after the trigger. The complete time window is created with around 700 milliseconds and has around 70 data samples of each sensor.

3.4. Classification Models

In this work, two classification models were conceived, developed, and tested. One is based on the mathematical approach of the Morlet wavelet, and the other is based on the artificial intelligence approach of artificial neural networks (ANN).

The mathematical approach considers the use of a threshold to evaluate the Morlet wavelet of the accelerometer time series. To start, the accelerometer data, concerning three axes, are used to calculate the spectral analyses as

$$D=\sqrt[2]{a_x^2+a_y^2+a_z^2}$$

(1)

where $a_x$, $a_y$, and $a_z$ represent the acceleration measures in the $xx$, $yy$, and $zz$ axes, respectively. The spectral analyses of each time series, with the classification of the dataset, are then used to create a profile, average of values, concerning human falls and ADLs.

The mathematical approach proposed for human fall detection uses the continuous wavelet transform (CWT) to decompose the time series into components of different scales, comparing the decomposition with the wavelet of varied sizes, as

$$c\left(a,b\right)={\displaystyle \int }f\left(t\right)\psi \left(at+b\right)dt$$

(2)

where a set of smaller waves is represented by $\psi \left(at+b\right)$, $a$ stands for the real value for dilatation, and $b$ stands for the displacement [31]

The CWT was applied to the two profiles created: human falls and ADLs. After the creation of the mother wave, new event signals are compared with the mother wave of the human fall in the other to evaluate its similarity degree according to the calculation of the coefficients of CWT [25] (the higher the coefficient, the greater the similarity), as

$$CW{T}_{coeff\left(a,b\right)}=\frac{1}{\sqrt{x}}{{\displaystyle \int }}_{-\infty }^{+\infty }S{V}_{candidate}\left(t\right){\psi }_{fall}\left(\frac{t-b}{a}\right)dt$$

(3)

where $a$ and $b$ represent the scale and translation parameters. After the application of Equation (3), the maximum value of $CW{T}_{coeff\left(a,b\right)}$ is chosen [32].

In the proposed classification model, the Morlet wavelet is used to calculate the wavelet transform of the time series D due to its recommendation for geophysical signals analysis [33], which are similar to the nonstationary signals of accelerometers, when used to read floor vibrations. The Morlet wavelet can be represented as

$${\psi }_0\left(\eta \right)={\pi }^{-\frac{1}{4}}{e}^{i{w}_0\eta }{e}^{-{\eta }^2/2}$$

(4)

where $w_0$ stands for the number of oscillations within the wavelet.

For the artificial intelligence classification model, a multi-layer ANN with eight input neurons and one output neuron is used. The two hidden layers are composed of 12 and 8 neurons, respectively. The input layer and hidden layers are configured to use the rectified linear activation function (ReLu).

The ANN was configured using Keras API (Application Programming Interface) to run in the TensorFlow platform. The ANN was configured with eight inputs: x-axis acceleration value, y-axis acceleration value, z-axis acceleration value, x-axis gyroscope value, y-axis gyroscope value, z-axis gyroscope value, sound level, and $D$ value from Equation (1). The output layer has only one neuron that indicates if the event is classified as a human fall or as an ADL. A sigmoid was used as an activation function for the last layer (i.e., output). The ANN was trained and validated using a dataset with 170 events (40 human-body falls and 130 different ADLs). The dataset was created using the proposed IoT device, and the ANN was implemented on a standard desktop. However, the use of the ANN in pilot sites must be conducted using a single-board computer to enable easy deployment.

The ANN model was implemented using the binary cross-entropy loss function as the classification model should supply a binary output. Regarding the optimization function, the Adam algorithm was used for the ANN training. The training used 250 epochs and a batch size of 32 samples.

4. Dataset and Case Study

To test and evaluate the proposed solutions, a dataset with 170 events was created using the IoT device prototype. Each event can be of one of the following types: human-body model falling (40 records), chair falling (30 records), cellphone falling (30 records), water bottle falling (30 records), and noise environment (40 records). Each event represented in the dataset is classified according to the activity recorded. This dataset has records, with a 100-millisecond period, of readings regarding: two accelerometers using 6 degrees of freedom (6 DoF) each, a microphone, and a Doppler sensor. To support this data, the spectral analyses of Equation (1) were also calculated for each accelerometer data.

Figure 2 show a record of a human fall using the spectral analyses of Equation (1). The use of a vectorial sum enables the visualization and classification of a single value that represents the three axes of the accelerometer.

The exclusive use of the Morlet wavelet showed particularly satisfactory results with an accuracy of 92.5% (that was previously reported by this paper’s authors in [11]). The ability of this method to not produce false negatives, producing only false positives, is important. In fact, the model was able to correctly identify 100% of the human falls.

The same dataset used for the Morlet wavelet was also used to build a classification model using an ANN. To do so, the dataset was split into two datasets (80–20%) to enable the training and validation of the model. The trained model was able to reach 79.01% accuracy. However, the use of ANN produced false negatives, meaning that there were falls that were not identified by this model.

Although different datasets were used, the obtained results were compared to proposed classification models previously published by other authors. The main differentiation between the proposed models and the already existing models is the ability to not have false negatives. The models proposed in [14,19,20] had accuracies between 80% and 92% but had false negatives, indicating that they were not able to identify all the human falls. The results of classification models are related to the quality of data and data collection. To address these issues, this paper proposed a dedicated IoT device to monitor and acquire several sources of data. This may have benefited the classification models and allowed better results.

The results of the Morlet wavelet surpass the results of the ANN model. However, they require higher computational effort. The use of ANN enables the faster classification and the integration of such a classifier in the IoT device, using frameworks such as the TensorFlow Lite for Microcontrollers. Additionally, the use of ANN allows the fast detection of human falls, enabling a near-real-time detection system.

Although an evaluation and comparison of the Morlet wavelet and ANN models was the goal of this paper, by analyzing the results, the authors defend the use of both models to complement each other. The deployment of the ANN model in the proposed IoT device will allow the fast classification of potential falls, eliminating the need for hardcoded triggers. The use of the Morlet wavelet model should be deployed in the fog-layer to effectively detect human falls. This will enable the proposed solution to have two classification steps: one in the microcontroller of the IoT device and another in the microprocessor of the fog-layer.

To enable the deployment of the ANN model in the IoT device, the adjustment of the sigmoid activation function is needed to enable the detection of all the off-normal events in the room. This will trigger the communication of the event from the edge-layer to the fog-layer. At this moment, the system can use the Morlet wavelet to better classify the event. The newly proposed solution, focusing on the use of both classification models, is shown in Figure 3.

5. Conclusions

Human fall detection systems have significant social importance and can make significant changes in people’s lives, especially for the elderly. It is therefore important to study solutions that prevent and minimize the negative impact of human falls. This paper proposes a fall detection solution based on internet of things devices that uses a layered architecture for human fall classification. The proposed internet of things device uses a low-cost microcontroller with low-cost sensors that can be deployed in residential homes and other facilities, using a non-wearable and non-intrusive approach.

The final proposed solution uses two classification models that are used at the edge and in the fog layers. Using the artificial intelligence-based model in the edge-layer enables the detection of off-normal events in the data. This enables data transmission to the fog-layer where the Morlet wavelet is used to accurately classify if it is a human fall. If this is the case, the solution raises alarms in the building and warns caregivers and health assistants. The results showed an accuracy of 92.5% without having false negatives, meaning that the system was able to detect all the falls.

Studying a non-wearable solution enables installation in the building and avoids the need for elderly people to use the device and track its battery state. However, it is confined to the physical premises of the building. Therefore, the authors conclude that the best approach is a combination of wearable and non-wearable solutions to enable continuous operation.