3.1. Evaluation Metrics
Due to the highly imbalanced data of each class, traditional classification accuracy is not suitable for analyzing the system’s effectiveness. To measure the effectiveness of the proposed architecture, we calculated the metrics individually for each event, where . That allowed us to compute accuracy using an unbalanced number of Non-Fall, Pre-Fall, and Fall events in a single test. Three evaluation metrics were used in the method, which is presented as follows:
Accuracy: Accuracy is one of the most fundamental evaluation metrics. It can be formally defined as the ratio of accurateness over all experiments. It can be defined as:
Sensitivity: Sensitivity is also known as recall score or true positive rate. It refers to the correctness of the true positive events of each available class. It can be mathematically defined as
Specificity: Specificity is also known as the true negative rate. It refers to the percentage of all negative samples that the model correctly predicts as negative. It can be represented as
where and . For a specific event, c, is the number of events correctly classified as the event; is the number of events incorrectly classified as the event; is the number of events misclassified as the event; and is the number of events correctly classified as not the event.
3.3. Result Analysis
In the result comparison, we split the section into three subsections. First, we analyzed the proposed architecture with different hyperparameters, such as the number of skipped convolution blocks, the number of recurrent layers, and the width of the network, in terms of the sensitivity of the architecture. Secondly, we analyzed the baseline architecture from hyperparameter tuning with different convolution layers, such as convolutions and separable convolutions [
32], pooling layers, such as max-pooling and average pooling, activation layers, such as Relu and Swish [
33], and recurrent layers, such as LSTM, GRU [
34], and bidirectional [
35]. Finally, we compared our proposed architectural results with state-of-the-art real-time FDS detection works.
Table 2 summarizes the result of the hyperparameter tuning experiment on testing data. The proposed architecture receives the sensory data through the head model. The head model is used as a feature extractor of the input data to generate sequential embedding. Furthermore, the embeddings are passed to the ensemble model for sequential processing to classify events. As the experiments show, increasing the number of skipped convolution blocks 3 to 4 in the head model decreases the architecture’s performance. The ensemble model consists of recurrent layers for processing sequential data. Increasing the number of recurrent layers from 1 to 2 in the ensemble model increases the architecture’s performance by 2%. Increasing the number of recurrent layers from 2 to 3 has a performance gain of 0.05%. However, increasing the recurrent layers results in a considerable parameter expansion, and sequential processing requires time. As our process is in real-time, the increasing processing time is not considered for the work. Our experiments found that three skipped convolution blocks for the head model and two recurrent layers for the ensemble model are suitable for the architecture.
Traditional deep learning architecture increases the width of the network for each block. However, our experiment shows that the same architectural width produces better results than increasing the width of the architecture. Furthermore, the optimal width of the network is set at 16.
Table 2 shows that expanding the network’s width decreases the architecture’s performance because, in the ensemble model, the width is greater than the data’s sequentiality. However, the low-level width of the network has one other advantage; the model is less parameterized and requires fewer computational costs.
We further investigated our proposed baseline architecture in the table with different layer configurations.
Table 3 shows the performance analysis with varying compositions of layers. First, the head model was analyzed using two different convolutions, pooling, and activation functions: the general convolution and the separable convolution with the ReLU and the Swish activation functions, along with max pooling and average pooling. Max pooling retains the most prominent features of the feature map and average pooling retains the average values of features of the feature map. Due to the high peaks of sensor data during the fall, the max pool works better than the average pool because it helps the architecture preserve the sensor data’s peak values. From
Table 3, we saw that the swish activation function gives a superior performance boost of 3% to the ReLU activation function in all different configurations. Swish is a smooth continuous function that allows a small number of negative weights to be propagated through the network while ReLu thresholds all negative weights to zero. Additionally, the trainable parameters allow better tuning of the function to maximize information propagation and push for a smoother gradient. It helps to optimize the network to be more accessible, thus generalizing better and faster.
On the other hand, compared to general convolution with swish activation, separable convolution with swish activation gives a more acceptable result, with a performance gain of 1.5%. The reason lies in the configuration of separable convolution. While standard convolution computes channel-wise and spatially in a single phase, separable convolution divides the calculation into two steps: depthwise convolution uses a single convolutional filter for each channel. Moreover, pointwise convolution generates a linear combination of the depthwise convolution output. Therefore, separable convolutions work on multiple filters simultaneously and tend to understand the information characteristics better. Additionally, separable convolution provides better nonlinearity than traditional convolutions, with excellent convergence speed, minor accuracy gain, and fewer trainable parameters than standard convolution.
Secondly, the ensemble model was analyzed using three types of recurrent layers: LSTM, GRU, and bidirectional. From
Table 3, we chose LSTM as the baseline recurrent layer in our proposed architecture. Although bidirectional LSTM achieved a performance gain of 0.5% over LSTM, bidirectional recurrent layers increase the network’s computational complexity and memory occupancy, making such performance improvements not worthwhile for real-time. We also considered GRU in our experiment. However, due to the absence of the output gate, GRU required slightly fewer computations and parameters than LSTM but entailed a reduction in sensitivity of over 4%. From the experiment, we found that a dropout rate of 0.5 was suitable for our architecture.
Table 4 presents a comparison of our proposed architecture with the state-of-the-art systems. The results demonstrate that our proposed architecture outperforms the state-of-the-art in all aspects. Two literature reviews [
10,
11] represent the current state-of-the-art, respectively. Ref. [
10] achieves mean accuracies of 0.93, 0.90, and 0.97 for Non-Fall, Pre-Fall, and Fall events, respectively. Ref. [
11] outperformed this result using a convolutional recurrent architecture with mean accuracies of 0.94, 0.93, and 0.97. However, our proposed ensemble-based convolutional recurrent architecture achieves an even higher accuracy, with mean accuracies of 0.95, 0.97, and 0.99 for all classes.
In terms of specificity, our proposed architecture achieves a mean specificity of 0.98, 0.97, and 0.99 for Non-Fall, Pre-Fall, and Fall events, respectively, which is higher than that of the state-of-the-art architectures. With respect to sensitivity, our proposed ensemble model achieves a mean sensitivity of 0.96, 0.94, and 0.98, which is higher than that of the state-of-the-art systems’ sensitivity; Ref. [
10]’s 0.88, 0.91, and 0.97 and [
11]’s 0.93, 0.93, and 0.96 for Non-Fall, Pre-Fall, and Fall events, respectively.
Figure 5 illustrates the losses, accuracy, confusion matrix, and ROC curve of the resultant architecture, respectively.
Figure 5a and b indicate that the model did not overfit during training, as the training and validation loss decreased consistently and the training and validation accuracy increased consistently without diverging from each other. However, it should be noted that these graphs are based on the performance of the deep learning model on simulated data, and the actual hardware implementation of the model has not been developed in the current scope of the study. Therefore, the results of the model’s performance on real-world data may differ from the simulated data presented in the figure.