Next Article in Journal
Facilitating Conditions as the Biggest Factor Influencing Elementary School Teachers’ Usage Behavior of Dynamic Mathematics Software in China
Next Article in Special Issue
Auto-Encoders in Deep Learning—A Review with New Perspectives
Previous Article in Journal
An Adapted Multi-Objective Genetic Algorithm for Healthcare Supplier Selection Decision
Previous Article in Special Issue
Breast Abnormality Boundary Extraction in Mammography Image Using Variational Level Set and Self-Organizing Map (SOM)
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Analysis and Recognition of Human Gait Activity Based on Multimodal Sensors

by
Diego Teran-Pineda
1,*,
Karl Thurnhofer-Hemsi
1,2 and
Enrique Dominguez
1
1
Department of Computer Languages and Computer Science, University of Málaga, Bulevar Louis Pasteur, 35, 29071 Málaga, Spain
2
Biomedical Research Institute of Málaga (IBIMA), C/ Doctor Miguel Díaz Recio, 28, 29010 Málaga, Spain
*
Author to whom correspondence should be addressed.
Mathematics 2023, 11(6), 1538; https://doi.org/10.3390/math11061538
Submission received: 20 February 2023 / Revised: 17 March 2023 / Accepted: 19 March 2023 / Published: 22 March 2023

Abstract

:
Remote health monitoring plays a significant role in research areas related to medicine, neurology, rehabilitation, and robotic systems. These applications include Human Activity Recognition (HAR) using wearable sensors, signal processing, mathematical methods, and machine learning to improve the accuracy of remote health monitoring systems. To improve the detection and accuracy of human activity recognition, we create a novel method to reduce the complexities of extracting features using the HuGaDB dataset. Our model extracts power spectra; due to the high dimensionality of features, sliding windows techniques are used to determine frequency bandwidth automatically, where an improved QRS algorithm selects the first dominant spectrum amplitude. In addition, the bandwidth algorithm has been used to reduce the dimensionality of data, remove redundant dimensions, and improve feature extraction. In this work, we have considered widely used machine learning classifiers. Our proposed method was evaluated using the accelerometer angles spectrum installed in six parts of the body and then reducing the bandwidth to know the evolution. Our approach attains an accuracy rate of 95.1% in the HuGaDB dataset with 70% of bandwidth, outperforming others in the human activity recognition accuracy.

1. Introduction

Human gait is a natural activity that people do every time to move from one point to another, involving muscles, nerves and brain activities. Human joints are a fundamental part of human movement, and therefore, a gait analysis is needed to study kinetics and kinematics [1,2], which are examined by physiotherapists, orthopedists, and neurologists to analyze and assess the status, treatment, and rehabilitation of patients [3]. Extrinsic and intrinsic factors (both psychological and physical) influence daily human activities; hence, determining normal gait parameters is very difficult [4]. In addition, there are a wide range of applications in different fields, such as neurology for monitoring neurological symptoms [5], or rehabilitation and physical therapy for the detection of gait disorders [6,7].
Physical activity monitoring via body-worn devices has recently been increased by sensor technologies (multimodal fusion sensors). They help vulnerable people maintain or increase the quality of individual and social lives through activity tracking [8]. The development of automatic information systems and improved methods to analyze biosignals with AI in this area is a way to contribute to more efficient health care.
The devices used to acquire body signals are classified into three approaches: non-wearable sensor (NWS), wearable sensor (WS), and hybrid system [2]. Nevertheless, WS is most commonly used due to its low cost, small dimensions, and high precision. These sensors are installed in the body to acquire the gait biosignal information during personal daily activities. WS includes force sensors, accelerometers, gyroscopes, extensometers, inclinometers, goniometers, active markers, electromyography, etc. To optimize the functionalities of such sensors (accelerometers, gyroscopes, and magnetometers), they are fused into a single unit called Inertial Measurement Units (IMUs) with multimodal fusion sensors technologies.
Extensive research has used body-worn inertial sensors and fortified the development of original Human Activity Recognition (HAR) applications. These applications include health rehabilitation, well-being assistance [9], smart homes and biofeedback systems [8], gait analysis [10,11,12,13], motion symmetry study [14], or for monitoring human activities [15,16]. Each of these applications requires continuous monitoring and tracking [17,18,19,20,21].
Feature extraction and selection algorithms are meant to sort pertinent features or suppress redundant information to increase activity recognition accurately and efficiently. These relevant features are commonly based on time-domain, wavelet and statistical analysis, with several IMU sensors installed in the body [22]. The frequency spectral of accelerometers has helped researchers predict vibrations in building structures [23] or turbines [24] and recognize the running path of dogs [25]. In addition, selection techniques to identify the most relevant features in datasets are needed to simplify the learned models and decrease the computational complexity and improve the model’s efficiency for recognition tasks. Using expensive industrial IMU sensors and extracting more complex features such as entropy [26,27] or frequency measures [20,28] brings promising results.
In this paper, we propose a novel methodology based on the frequency domain and bandwidth reduction of IMU accelerometer signals for HAR applications. The main objective is to analyze the minimum amount of features necessary to obtain good performance in the model training and decrease the computational time of the signal processing and classification. For that purpose, a new signal preprocessing methodology is presented based on the frequency domain analysis, and functional transforms are included to reduce the computational complexity due to the high dimensionality of features. Only accelerometer signals are considered, since it is the most relevant feature that allows distinguishing between activities, but also, we want to demonstrate the effectiveness of our proposal with less information, which could be of interest in some treatments where a simple sensor is needed to be implanted for a treatment. The proposed method is applied successfully to a public benchmark dataset named the Human Gait Database (HuGaDB) for performance evaluation.
The rest of the paper is organized as follows. Section 2 addresses briefly a literature review. In Section 3, our proposed methodology is presented. Section 4 analyzes and discusses the experimental results. Finally, the conclusion and future research are presented in Section 5.

2. Related Work

Wearable sensors (WS) are used in recent advances because researchers have successfully implemented body-worn devices to monitor personal locomotion behaviors and recognize human activity. The most common WS uses an accelerometer and gyroscope integrated into one wearable inertial mobile unit (IMU). Another type of WS based on the electrical current associated with muscular actions is also used in combination with IMUs for HAR [22]. These WS named electromyography (EMG) measure the myoelectric signals produced by muscular actions, hence their importance in activity recognition. A study on the fusion of EMG and IMU sensors for HAR is presented in [22], showing the potential of incorporating EMG signals in activity recognition.
A motoring real-time personal locomotion is introduced in [8], wielding three inertial sensors at different body locations (wrist, thigh, and chest). Data were processed through Gaussian and zero-phase filters. A hierarchical feature-based technique is used to extract features based on stochastic gradient descent optimization methods, achieving an accuracy rate of 92.50% in their experiments using the HuGaDB dataset. In [5], a technique to extract features using Discrete Fourier Transforms is proposed to estimate the mean power in selected frequency bands for ataxic gait assessment recognition. The accelerometric data were acquired by 31 time-synchronized sensors (perception neuron system) located at different body parts. Different classifiers were used for evaluation, such as support vector machines, Bayesian, nearest neighbors, and neural network methods, with the highest accuracy of 98.5%. The data comprised 13 normal and 12 ataxic individuals, and the entire study was conducted in a clinical environment. Deep learning techniques were applied to predict falls in older adults [9]. Data were collected on fall risk factors in the elderly using WS (accelerometers), questionaries, and physical tests. The dataset consisted of 296 older adults wearing a triaxial accelerometer on their lower back for a week and the following six months in which fall incidences and descriptions were obtained. Researchers used the raw accelerometer data (without making the preprocessing step) as input to an LSTM classifier, obtaining a time reduction and an AUC (Area Under the Curve) of 0.75.
In summary, most of the works conclude that the more sensors and extracted features from data, the better the accuracy of the computer classification algorithm. Nevertheless, sensors installed on the human body produce less comfort for the patient (ergonomic), make it more challenging to perform human activities, increase noise, and require more time to (pre/post)process data and analyze the activities in real time [29].

3. Methodology

This section presents the mathematical techniques we propose to classify human gait signals. Figure 1 illustrates the proposed method workflow. The first step is collecting the raw data to preprocess it, removing noise and undesired signals, then extracting relevant features, and finally training and testing a human activity classifier.
This work intends to start from the raw accelerometer signal in our proposed methodology. This approach was used by other works, such as in [9], where they obtained good results in processing time, or in [30], where nuances could be unintentionally removed from the raw features when the parameters are extracted. We intend to model a classifier with less data and less processing time. For instance, Figure 2a illustrates a raw signal containing two activities, and Figure 2b presents only one signal corresponding to the walking activity from the right shin. As in [9], we use the raw acceleration data directly as input to study and transform the signal.
Feature extraction is a commonly used technique to clear data and focus only on the relevant features by reducing the dimensionality. Several methods proposed in the literature focused on extracting signal features, such as those based on time–frequency domain, and many other techniques that allow the reduction of data dimensionality [31].
In this work, a total of six sensors were installed on different parts (feet, shins, and thighs) of the legs, each containing a 3-axis accelerometer X, Y and Z (18 total signals). All information of the signals is converted into the frequency domain using FFT to reduce the size, eliminate useless data, and facilitate the training process of the classifiers.

3.1. Signal Preprocessing

There are methods for feature extraction based on the time, frequency, and time-frequency domain, which are usually applied to the raw data. In this work, we propose a set of signal processing steps to obtain relevant features, which are summarized in Algorithm 1.
Algorithm 1 Features extraction from acceleration signal
Input: acc = acceleration data (x,y,z) from each part of the body
f c 2 = max bandwidth frequency
Output: Feature vector of the selected frequency bandwidth
Method: IMU_features(x,y,x)
  1:
raw_data ← acc
  2:
while iteration=1,2, , N p a t i e n t s do
  3:
      filtered_data ← Highpass_filter(raw_data)
  4:
      angle_data ← Resultant_angle(filtered_data)
  5:
      fft_data ← ExtractFFT(angle_data)
  6:
      norm_data ← Normalize(fft_data)
  7:
      extrap_data ←Freq_norm(norm_data)
  8:
       f c 1 ← QRS(extrap_data)
  9:
      feature_vector ← get(extrap_data, f c 1 , f c 2 )
10:
end while
11:
return: feature_vector

3.1.1. Noise Filtering

First, the high-pass Chebyshev filter filters the direct current (DC) noise. The approximation of the gain response as a function of angular frequency w is indicated in Equation (1). The high-pass filter of eighth order, cutoff frequency of 0.4 Hz, and attenuation of 80 are applied.
G n ( ω ) = 1 1 + 1 ε 2 T n 2 ( ω 0 / ω )
where ε is the ripple factor, ω 0 is the cutoff frequency and T n is a Chebyshev polynomial of the n-th order.
As an example, in Figure 3a, we can see the signal spectrum with one pulse at 0 Hz (DC noise), and the pass-high filter removes the noise (Figure 3b).

3.1.2. Resultant Angle

The number of features is a crucial stage of feature selection, where the information in the feature matrix will influence the discrimination ability of the deep features. If the signal is considerably large, then a data reduction technique is helpful to simplify the input while preserving the properties of the signal. In those cases, besides complicated data analysis, the proposed method may be incompatible with the dataset, and the proposed classifier may be generalized to a limited dataset.
As was denoted at the beginning of the Section, we have 18 signals, 3 axes for each of the 6 sensors. A common practice is to create the feature vector F k containing the filtered signal for the 3 axes:
F k = [ f X k , f Y k , f Z k ] , k = 1 , , 6 .
However, there is much redundant information that slows down the training of the classification model. For that reason, we computed the resultant angle of the accelerometer’s specific forces [32]:
A n g l e X = arctan f X 2 f Y 2 + f Z 2
where f X , f Y , f Z are the gravity forces, i.e., the accelerometer signals. f X is in the numerator since the principal signal would be the X-axis.
Applying Equation (3) to our filtered signals and translating into Equation (2), we reduce the size of the feature vector of each body accelerometer.
F k = A n g l e X k

3.1.3. Frequency Domain Analysis

In this research, the Fast Fourier Transform (FFT) has been used to convert the received resultant signals into the frequency domain. Specifically, the Discrete Fourier Transform (DFT) [31,33,34] is applied to improve the computational complexity. The frequency content of the input signal x is then extracted using the following transformation:
X ( k ) = n = 0 N 1 x n e x p i 2 π n k N
where X = F F T ( x ) , x n is the n-th element of an input signal, k is the frequency sample, and N is the transformation length.
FFT’s performance depends on the signal’s length, and to improve its speed, the Cooley/Tukey algorithm is used through the Matlab fft function, which exploits the symmetries to reduce a large DFT into smaller DFTs. This process helps reduce computational complexity from O ( N 2 ) to O ( N l o g N ) , improving speed [33]. Our research used the entire IMU signal for the FFT operation and transformed it separately for each patient.

3.1.4. Power and Bandwidth Normalization

After applying the FFT, the signal is normalized in the range [0,1] by applying
X p o w e r _ n o r m = | X f f t | max ( | X f f t | )
Figure 3c depicts the resulting signal.
In the same way, the signal was normalized in the power spectrum, and it is convenient to equalize the signal in the frequency axis. This way, all samples fall into the power spectrum, helping the classifier have homogeneous training. For that purpose, the maximum frequency for all the data samples is found, and then, linear extrapolation is computed:
F M A X = max { F j , j = 1 , , M } X f r e q _ n o r m = linear _ extrap ( X p o w e r _ n o r m , F M A X )
where F represents the set of frequency values of a sample, and M represents the total number of samples.
Figure 4 and Figure 5 present a detailed spectral analysis (evolution of spectral components) as the relative power along the frequency. This figure shows an example of the processed accelerometer signal of the walking activity.

3.1.5. Mean Relative Power QRS Detection

As it was exposed in [31], the first and second dominant frequency amplitudes hold the most information about the signal, so we propose a method to extract those amplitudes based on QRS complexes and use them as the inputs of the classification models.
The key element of extracting these frequency amplitudes is setting, and appropriate threshold tolerance, which is used by the QRS algorithm [35]. This technique is used in spectrum analysis to recognize QRS complexes, reducing false detections caused by interferences in ECG signals. Figure 6 shows an example of the QRS detection, where R is the peak of the principal spectrum, Q is the start of the peak signal, and S is the end of the peak signal, which will be unnecessary for our application, as explained later.
However, peak detection is a very critical step, since information can be lost if an adequate threshold is not set up correctly. For instance, in [36], a modified version of the QRS algorithm with an adaptative threshold is presented, defining two kinds of peaks: signal and noise peak. If a peak value is larger than the threshold, it is marked as a QRS complex, and then, the signal peak is updated.
This solution has a problem with the noise peak because it detects some spectra as noise, and the signal amplitude is not constant, such as the ECG signal. For instance, a threshold of 0.257 when calculated with the data in Figure 7 loses the information of other spectrums.
The Pan and Tompkins algorithm inspires our proposal to detect the QRS complex [35]. First, a differentiation provides complex slope information, and then, an amplitude squaring function and a moving window integrator are applied. The following paragraphs will describe the process in detail:
1.
Differentiation. After filtering and normalizing, the signal is differentiated as follows:
y ( n T ) = 1 8 [ x ( n T 2 T ) 2 x ( n T T ) + 2 x ( n T + T ) + x ( n T + 2 T ) ]
where T is the sampling period.
2.
Squaring. It amplifies the slope of the frequency response.
y ( n T ) = [ x ( n T ) ] 2
3.
Moving-Windows Integration. To obtain waveform feature information:
y ( n T ) = 1 N [ x ( n T ( N 1 ) T ) + x ( n T ( N 2 ) T ) + + x ( n T ) ]
After that, the Mean Relative Power signal is used to initialize the threshold. This measure aims to vary the bandwidth of our spectral signal so that only the desired signal information is considered. Equation (11) calculates the average power in a window selected by the desired frequencies:
P w = f f X ( f ) 2 d f
As shown in [5], each of the selected spectral features of a signal segment of N samples is evaluated and divided by the first half-part signal ( N / 2 samples) in order to keep the most significant information. Equation (12) computes the discrete Mean Relative Power in a specified frequency band:
M P r e l = k ϕ | X ( k ) | 2 k = 0 N / 2 | X ( k ) | 2
where ϕ is the set of indices for which the frequency values f k = k N f s [ f c 1 , f c 2 ] . In the given case, we study the accelerometric spectral features in the [ 0 % , 100 % ] range.
Therefore, the M P r e l is used as signal peaks instead of using both the integration waveform signal peak and the noise-filtered signal peak for the threshold initialization as in [35].
M V A L = M P r e l ( s i g n a l ) S P K I = 0.13 M V A L T H R E S H O L D = 0.25 S P K I + 0.75 N P K I
where s i g n a l refers to the integration waveform and filtered signal, respectively; S P K I is the estimate of the signal peak, N P K I is the noise peak, and T H R E S H O L D is the applied threshold.
The rest of the detection procedure follows the original QRS algorithm. Once the peaks are detected, the initial frequency f c 1 is set, and the rest of the bandwidth is the signal extracted to be used for the next step:
f c 1 = Q
where Q is the Q-element of the QRS complex of the first peak. As an example, in Figure 6, the point Q is automatically found by the proposed methodology, and we set the value to f c 1 . On the other hand, f c 2 will define the last point of the bandwidth frequency. Figure 7 shows a f c 2 reaching 100 % bandwidth.

3.2. Sliding Bandwidth Analysis

The selected frequency range is varied to reduce the number of features and sensors. The bandwidth is located within the range f c 1 to f c 2 of each frequency analysis, as shown in Figure 7, and these sets are expressed in Equation (15):
B W = f c 2 f c 1 f o r f c 1 f c 2 100 %
Therefore, the power spectra analyzed vary according to the length of B W :
P S = [ X ( k ) ] θ
where θ [ f c 1 , f c 2 ] is the set of indices for which the frequency values vary within the bandwidth.

4. Experimental Results

4.1. Dataset Description

The Human Gait Database [37] is commonly used in research for human activity recognition (HAR), gesture recognition (GR), and gait analysis (GA). This dataset has been used in many works because they represent the inertial movement of a person and help to develop models before installing them in wearable sensor systems. The dataset comprises 18 healthy participants who were split into two groups: 15 participants to create and validate the model and 3 participants to test the model. This way, the information the classification models test will be completely independent of the one used for training, assuring a fair comparison.
In total, six inertial sensors (IMU) (each IMU sensor has a three-axis accelerometer and three-axis gyroscope) and electromyography (sEMG) sensors were placed on the right and left thighs, shins, and feet. The total number of signals collected is 38:36 from the IMU sensor and 2 from the sEMG sensor. This database contains 12 activities: Walking with ID (1), Running with ID (2), Going up with ID (3), Going down with ID (4), Sitting with ID (5), Sitting down with ID (6), Standing up with ID (7), Standing with ID (8), Bicycling with ID (9), Up by elevator with ID (10), Down by elevator with ID (11), and Sitting in the car with ID (12).

4.2. Classification Experiments

The proposed method was applied to accelerometric raw data on the HuGaDB dataset. After the initial spectral preprocessing, we set the frequency f c 2 to 100% to check the performance of the classifiers with the total bandwidth. For the experiments, Discriminant Analysis (DA), Support Vector Machine (SVM) and Neural Network (NN) classification models were employed. The code was written in Matlab (R2021b). The method was applied on each axis, receiving signals from the right and left legs.
As mentioned, the dataset contains information on six different sensors, which is summarized in Table 1.
In order to carry out the classification experiments, all preprocessed accelerometer signals are concatenated into an activity vector.
A j = [ { F k } ] j , k = 1 , , 6 .
where j represents the number of sample activities, the training data comprises a set of signals for N = 15 persons that carried out several activities. These activities are previously identified in the whole signal and separated into different M samples. The set of samples { A j } j = 1 , , M and their respective true class labels will form the training data. After preprocessing the dataset, the total number of training samples after preprocessing the dataset is 152, where M = 122 (from 15 participants) will be used for training and 30 will be used for testing (from 3 participants).
First, the performance achieved using the maximum bandwidth was compared. Here, 10-fold cross-validation was carried out to train the models and make a preliminary selection among several classifiers, using 75% of the data for training and 25% for testing (from 15 participants, as indicated in Section 4.1). After that, the unseen test set (from three participants) was used to compute the real performance. Each instance is represented by the patient, for a total of 18 patients within each activity, meaning that each class keeps the same amount of class instances for both the train and test sets.
The results of the different classifiers are compared in Table 2. In addition, we compared our performance with a state-of-art method that used a similar sensor and data configurations to our experiments [8] to have a fair comparison. We did not compare our approach with other published work on a combination of sensor signals. Hence, our methodology yielded an accuracy of 95.10% using NN and 91% using SVM compared to the 5% and 4% less accuracy, respectively, obtained by the competing method. In addition, our method was applied with a Discriminant Analysis classifier, achieving 95.5% accuracy, being chosen as the best for our proposal. The methodology proposed in [8] is still worse with 92.5% accuracy.
The rest of the experiments will focus on modifying the bandwidth to analyze the impact of applying the proposed method with less information.

4.3. Bandwith Analysis

This analysis aims to reduce the sample data (8022 features per sample) used to train the classifiers. Each activity contains 56.35 samples per second, and the acquisition time varies. We study the accelerometric power spectral in [ f c 1 , f c 2 ] , where f c 1 is detected automatically and f c 2 will vary for training. The bandwidth extension has been studied in these three classifiers, reducing the bandwidth by 10% each time, starting from the endpoint of the spectrum, f c 2 = 100 % .
Figure 8 summarizes the accuracy obtained for each bandwidth. It can be observed that the precision of the DA classifier does not change up to a bandwidth of 70%. In addition, it can be observed that the NN classifier has good precision in the range of 90% to 50%, although it slowly decreases when the bandwidth is shortened. Lastly, the SVM classifier presents the worst overall performance, although it is quite robust to the reduction of training data, having even better performance with only 30–50% of the bandwidth. From these results, we can extract that using 70% of the bandwidth with the DA classifier produces the same accuracy as if 100% bandwidth is used. Therefore, data can be reduced to 5615 features to improve the training time of the classifiers, obtaining the same outcomes.
Figure 9 depicts the behavior of the classifiers when changing the bandwidth for each type of activity. Each of the classifiers has obtained excellent performances for most of the activities. For instance, walking, going up the stairs, and going up with the elevator only needed a primary spectrum in the 20% window of bandwidth, independently of the classifier; however, other activities need a wider bandwidth to be recognized. However, to obtain excellent general performance, more data are required. The DA classifier, shown in Figure 9a, can perfectly recognize up to nine activities with 70% bandwidth. SVM and NN (Figure 9b,c) reflected similar performances except for the elevator and going downstairs. However, they improve the recognition of the car-sitting activity, reaching 100% accuracy with 40% and 50% bandwidth, respectively.
Focusing on the type of activity, the elevator is the most critical one. It is interesting to see that for the SVM, 60% of the bandwidth is necessary to reach 80% accuracy, but for NN, 70% of the data is needed. The DA cannot detect this activity properly, but the action of sitting has a perfect detection with only 30% of the bandwidth. On the other hand, the run and bicycle are better classified by the NN, which means that it works better with complex signals.

5. Conclusions

This paper presents a new methodology based on the frequency domain. Features were extracted from the raw data of IMUs by varying the frequency window to optimize the number of relevant features. All the information provided by the raw accelerometer signals was optimized and transformed using the proposed methodology. In addition, a modified QRS algorithm is proposed to automatically find the first frequency and the beginning of the bandwidth.
The standard HuGaDB dataset was used with the best classifiers found in the literature for HAR problems (DA, SVM, and NN). Experimental results show the improved accuracies achieved (Table 2) when the proposed methodology is applied. These results were achieved using the whole frequency spectrum. Additionally, the performance of our proposal was analyzed for reduced bandwidths and studied in detail for the different activities to be recognized. SVM is the most robust classifier, producing a similar performance for reduced bandwidths, although DA achieves the best accuracy for reduced bandwidths (up to 70%). Regarding the recognition of the different activities, both Up Elevator and Down Elevator are the activities with the worst accuracy. In the end, activity recognition varies depending on the bandwidth used due to where the most significant information is placed in the frequency spectrum. For instance, Walking and Running activities are easily recognized, even with 20% of the bandwidth, because the relevant information of these activities is in the first power spectrum.
Future works will include more types of signals as part of the input for training by creating an adequate methodology to preprocess and join the signals to achieve better outcomes than state-of-art methods. In addition, another important study to carry out is the reduction of the number of sensors, determining which positions and how many of them are enough to provide good classification results.

Author Contributions

Conceptualization, D.T.-P. and E.D.; methodology, D.T.-P.; software, D.T.-P.; validation, D.T.-P. and K.T.-H.; formal analysis, K.T.-H.; investigation, D.T.-P.; writing—original draft preparation, D.T.-P.; writing—review and editing, K.T.-H. and E.D.; supervision, K.T.-H. and E.D. All authors have read and agreed to the published version of the manuscript.

Funding

This research was partially funded by the Autonomous Government of Andalusia (Spain), grant number UMA20-FEDERJA-108. It is also partially supported by the University of Málaga (Spain), grant numbers B1-2019_01, B1-2019_02, B1-2021_20, and B4-2022. Partial funding for open access charge: Universidad de Málaga.

Data Availability Statement

HuGaDB https://github.com/romanchereshnev/HuGaDB (accessed on 13 November 2021).

Acknowledgments

The authors thankfully acknowledge the computer resources, technical expertise and assistance provided by the SCBI (Supercomputing and Bioinformatics) center of the University of Málaga. They also gratefully acknowledge the support of NVIDIA Corporation with the donation of a RTX A6000 GPU with 48Gb. The authors also thankfully acknowledge the grant of the Universidad de Málaga and the Instituto de Investigación Biomédica de Málaga y Plataforma en Nanomedicina-IBIMA Plataforma BIONAND.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
HuGaDBHuman Gait Database
NWSNon-Wearable Sensor
WSWearable Sensor
IMUInertial Measurement Units
HARHuman Activity Recognition
EMGElectromyigraphy
PAMAP2Physical Activity Monitoring for Aging People
SGDStochastic Gradient Descent
FARAOFall Risk Assessment in Older Adults
AUCArea Under the Curve
GRGesture Recognition
GAGait Analysis
FFTFast Fourier Transform
DFTDiscrete Fourier Transform
DSPDigital Signal Process
DCDirect Courrient
PSPower Spectral
BWBandwidth
DTData Transform
DADiscriminant Analysis
SVMSupport Vector Machine
NNNeural Network

References

  1. Senanayake, C.; Senanayake, S. Human assisted tools for gait analysis and intelligent gait phase detection. In Proceedings of the 2009 Innovative Technologies in Intelligent Systems and Industrial Applications, Kuala Lumpur, Malaysia, 25–26 July 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 230–235. [Google Scholar]
  2. Teran, P.D.; Dominguez, E. Human gait model based on a machine learning and filtering noisy signals with recursive algorithm. In Proceedings of the 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Seoul, Republic of Korea, 16–19 December 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 1142–1145. [Google Scholar]
  3. Prakash, C.; Kumar, R.; Mittal, N. Recent developments in human gait research: Parameters, approaches, applications, machine learning techniques, datasets and challenges. Artif. Intell. Rev. 2018, 49, 1–40. [Google Scholar] [CrossRef]
  4. Song, Y.; Zhang, J.; Cao, L.; Sangeux, M. On discovering the correlated relationship between static and dynamic data in clinical gait analysis. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Prague, Czech Republic, 23–27 September 2013; Springer: Berlin/Heidelberg, Germany, 2013; pp. 563–578. [Google Scholar]
  5. Dostál, O.; Procházka, A.; Vyšata, O.; Ťupa, O.; Cejnar, P.; Vališ, M. Recognition of motion patterns using accelerometers for ataxic gait assessment. Neural Comput. Appl. 2021, 33, 2207–2215. [Google Scholar] [CrossRef]
  6. Buckley, E.; Mazzà, C.; McNeill, A. A systematic review of the gait characteristics associated with Cerebellar Ataxia. Gait Posture 2018, 60, 154–163. [Google Scholar] [CrossRef] [PubMed]
  7. Joukov, V.; Bonnet, V.; Karg, M.; Venture, G.; Kulić, D. Rhythmic extended Kalman filter for gait rehabilitation motion estimation and segmentation. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 26, 407–418. [Google Scholar] [CrossRef] [PubMed]
  8. Gochoo, M.; Tahir, S.B.U.D.; Jalal, A.; Kim, K. Monitoring real-time personal locomotion behaviors over smart indoor-outdoor environments via body-worn sensors. IEEE Access 2021, 9, 70556–70570. [Google Scholar] [CrossRef]
  9. Nait Aicha, A.; Englebienne, G.; Van Schooten, K.S.; Pijnappels, M.; Kröse, B. Deep learning to predict falls in older adults based on daily-life trunk accelerometry. Sensors 2018, 18, 1654. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Allseits, E.; Kim, K.J.; Bennett, C.; Gailey, R.; Gaunaurd, I.; Agrawal, V. A novel method for estimating knee angle using two leg-mounted gyroscopes for continuous monitoring with mobile health devices. Sensors 2018, 18, 2759. [Google Scholar] [CrossRef] [Green Version]
  11. Del Din, S.; Hickey, A.; Hurwitz, N.; Mathers, J.C.; Rochester, L.; Godfrey, A. Measuring gait with an accelerometer-based wearable: Influence of device location, testing protocol and age. Physiol. Meas. 2016, 37, 1785–1797. [Google Scholar] [CrossRef]
  12. Staab, W.; Hottowitz, R.; Sohns, C.; Sohns, J.M.; Gilbert, F.; Menke, J.; Niklas, A.; Lotz, J. Accelerometer and gyroscope based gait analysis using spectral analysis of patients with osteoarthritis of the knee. J. Phys. Ther. Sci. 2014, 26, 997–1002. [Google Scholar] [CrossRef] [Green Version]
  13. Zilani, T.A.; Al-Turjman, F.; Khan, M.B.; Zhao, N.; Yang, X. Monitoring movements of ataxia patient by using UWB technology. Sensors 2020, 20, 931. [Google Scholar] [CrossRef] [Green Version]
  14. Procházka, A.; Vyšata, O.; Charvátová, H.; Vališ, M. Motion symmetry evaluation using accelerometers and energy distribution. Symmetry 2019, 11, 871. [Google Scholar] [CrossRef] [Green Version]
  15. Ebara, T.; Azuma, R.; Shoji, N.; Matsukawa, T.; Yamada, Y.; Akiyama, T.; Kurihara, T.; Yamada, S. Reliability of smartphone-based gait measurements for quantification of physical activity/inactivity levels. J. Occup. Health 2017, 59, 17–0101. [Google Scholar] [CrossRef] [PubMed]
  16. Voicu, R.A.; Dobre, C.; Bajenaru, L.; Ciobanu, R.I. Human physical activity recognition using smartphone sensors. Sensors 2019, 19, 458. [Google Scholar] [CrossRef] [Green Version]
  17. Castillejo, P.; Martinez, J.F.; Rodriguez-Molina, J.; Cuerva, A. Integration of wearable devices in a wireless sensor network for an E-health application. IEEE Wirel. Commun. 2013, 20, 38–49. [Google Scholar] [CrossRef] [Green Version]
  18. Chatterjee, R.; Maitra, T.; Islam, S.H.; Hassan, M.M.; Alamri, A.; Fortino, G. A novel machine learning based feature selection for motor imagery EEG signal classification in Internet of medical things environment. Future Gener. Comput. Syst. 2019, 98, 419–434. [Google Scholar] [CrossRef]
  19. Gravina, R.; Alinia, P.; Ghasemzadeh, H.; Fortino, G. Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges. Inf. Fusion 2017, 35, 68–80. [Google Scholar] [CrossRef]
  20. Uddin, M.Z.; Hassan, M.M. Activity recognition for cognitive assistance using body sensors data and deep convolutional neural network. IEEE Sensors J. 2018, 19, 8413–8419. [Google Scholar] [CrossRef]
  21. Fortino, G.; Giannantonio, R.; Gravina, R.; Kuryloski, P.; Jafari, R. Enabling effective programming and flexible management of efficient body sensor network applications. IEEE Trans. -Hum.-Mach. Syst. 2012, 43, 115–133. [Google Scholar] [CrossRef]
  22. Badawi, A.A.; Al-Kabbany, A.; Shaban, H.A. Sensor type, axis, and position-based fusion and feature selection for multimodal human daily activity recognition in wearable body sensor networks. J. Healthc. Eng. 2020, 2020, 7914649. [Google Scholar] [CrossRef]
  23. Beltran-Carbajal, F.; Abundis-Fong, H.F.; Trujillo-Franco, L.G.; Yañez-Badillo, H.; Favela-Contreras, A.; Campos-Mercado, E. Online frequency estimation on a building-like structure using a nonlinear flexible dynamic vibration absorber. Mathematics 2022, 10, 708. [Google Scholar] [CrossRef]
  24. Kuo, J.Y.; You, S.Y.; Lin, H.C.; Hsu, C.Y.; Lei, B. Constructing Condition Monitoring Model of Wind Turbine Blades. Mathematics 2022, 10, 972. [Google Scholar] [CrossRef]
  25. Hayati, H.; Mahdavi, F.; Eager, D. Analysis of agile canine gait characteristics using accelerometry. Sensors 2019, 19, 4379. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Nguyen, N.; Phan, D.; Pathirana, P.N.; Horne, M.; Power, L.; Szmulewicz, D. Quantification of axial abnormality due to cerebellar ataxia with inertial measurements. Sensors 2018, 18, 2791. [Google Scholar] [CrossRef] [Green Version]
  27. Phan, D.; Nguyen, N.; Pathirana, P.N.; Horne, M.; Power, L.; Szmulewicz, D. Quantitative assessment of ataxic gait using inertial sensing at different walking speeds. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 4600–4603. [Google Scholar]
  28. Caliandro, P.; Conte, C.; Iacovelli, C.; Tatarelli, A.; Castiglia, S.F.; Reale, G.; Serrao, M. Exploring risk of falls and dynamic unbalance in cerebellar ataxia by inertial sensor assessment. Sensors 2019, 19, 5571. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Antoniades, A.; Spyrou, L.; Martin-Lopez, D.; Valentin, A.; Alarcon, G.; Sanei, S.; Took, C.C. Detection of interictal discharges with convolutional neural networks using discrete ordered multichannel intracranial EEG. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 25, 2285–2294. [Google Scholar] [CrossRef]
  30. Tunca, C.; Salur, G.; Ersoy, C. Deep learning for fall risk assessment with inertial sensors: Utilizing domain knowledge in spatio-temporal gait parameters. IEEE J. Biomed. Health Informatics 2019, 24, 1994–2005. [Google Scholar] [CrossRef]
  31. Taylan, O.; Sattari, M.A.; Elhachfi Essoussi, I.; Nazemi, E. Frequency Domain Feature Extraction Investigation to Increase the Accuracy of an Intelligent Nondestructive System for Volume Fraction and Regime Determination of Gas-Water-Oil Three-Phase Flows. Mathematics 2021, 9, 2091. [Google Scholar] [CrossRef]
  32. Tjhai, C.; O’Keefe, K. Using step size and lower limb segment orientation from multiple low-cost wearable inertial/magnetic sensors for pedestrian navigation. Sensors 2019, 19, 3140. [Google Scholar] [CrossRef] [Green Version]
  33. Kang, H.; Lee, J.; Kim, D. Hi-fft: Heterogeneous parallel in-place algorithm for large-scale 2D-fft. IEEE Access 2021, 9, 120261–120273. [Google Scholar] [CrossRef]
  34. Adámek, K.; Novotnỳ, J.; Thiyagalingam, J.; Armour, W. Efficiency near the edge: Increasing the energy efficiency of FFTs on GPUs for real-time edge computing. IEEE Access 2021, 9, 18167–18182. [Google Scholar] [CrossRef]
  35. Pan, J.; Tompkins, W.J. A Real-Time QRS Detection Algorithm. IEEE Trans. Biomed. Eng. 1985, BME-32, 230–236. [Google Scholar] [CrossRef] [PubMed]
  36. Lu, X.; Pan, M.; Yu, Y. QRS detection based on improved adaptive threshold. J. Healthc. Eng. 2018, 2018, 5694595. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  37. Chereshnev, R.; Kertész-Farkas, A. Hugadb: Human gait database for activity recognition from wearable inertial sensor networks. In Proceedings of the International Conference on Analysis of Images, Social Networks and Texts, Moscow, Russia, 5–7 July 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 131–141. [Google Scholar]
Figure 1. Flow chart of the proposed methodology.
Figure 1. Flow chart of the proposed methodology.
Mathematics 11 01538 g001
Figure 2. Raw signal from X-axis accelerometer of the right shin: (a) whole raw data, (b) walking subsample.
Figure 2. Raw signal from X-axis accelerometer of the right shin: (a) whole raw data, (b) walking subsample.
Mathematics 11 01538 g002
Figure 3. Signal in the frequency domain of X-axis accelerometer in right foot (walk activity). (a) Raw data. (b) Spectrum after pass-band filter. (c) Spectrum after FFT and normalization.
Figure 3. Signal in the frequency domain of X-axis accelerometer in right foot (walk activity). (a) Raw data. (b) Spectrum after pass-band filter. (c) Spectrum after FFT and normalization.
Mathematics 11 01538 g003
Figure 4. Right Walk Activity Spectrum. (a) Foot Spectrum. (b) Shin Spectrum. (c) Thigh Spectrum.
Figure 4. Right Walk Activity Spectrum. (a) Foot Spectrum. (b) Shin Spectrum. (c) Thigh Spectrum.
Mathematics 11 01538 g004
Figure 5. Left Walk Activity Spectrum. (a) Foot Spectrum. (b) Shin Spectrum. (c) Thigh Spectrum.
Figure 5. Left Walk Activity Spectrum. (a) Foot Spectrum. (b) Shin Spectrum. (c) Thigh Spectrum.
Mathematics 11 01538 g005
Figure 6. Example of the QRS calculation in the first spectrum.
Figure 6. Example of the QRS calculation in the first spectrum.
Mathematics 11 01538 g006
Figure 7. Example of the selected bandwidth, where f c 1 = Q and f c 2 = 100 % .
Figure 7. Example of the selected bandwidth, where f c 1 = Q and f c 2 = 100 % .
Mathematics 11 01538 g007
Figure 8. Evolution of accuracy over different bandwidths.
Figure 8. Evolution of accuracy over different bandwidths.
Mathematics 11 01538 g008
Figure 9. Human Activities Accuracy with Bandwidth Select. (a) DA—Discriminant Analysis Classification. (b) SVM—Support Vector Machine Classification. (c) NN—Neural Network Classification.
Figure 9. Human Activities Accuracy with Bandwidth Select. (a) DA—Discriminant Analysis Classification. (b) SVM—Support Vector Machine Classification. (c) NN—Neural Network Classification.
Mathematics 11 01538 g009aMathematics 11 01538 g009b
Table 1. Acceleromether used in the body.
Table 1. Acceleromether used in the body.
LeftRight
DataFeetShinsThighsFeetShinsThighs
k123456
Table 2. Accuracy Experiments using HuGaDB and Frequency Features.
Table 2. Accuracy Experiments using HuGaDB and Frequency Features.
MethodDA (%)SVM (%)NN (%)
Gochoo et al. [8]-85.6891.23
Ours95.5091.0095.10
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Teran-Pineda, D.; Thurnhofer-Hemsi, K.; Dominguez, E. Analysis and Recognition of Human Gait Activity Based on Multimodal Sensors. Mathematics 2023, 11, 1538. https://doi.org/10.3390/math11061538

AMA Style

Teran-Pineda D, Thurnhofer-Hemsi K, Dominguez E. Analysis and Recognition of Human Gait Activity Based on Multimodal Sensors. Mathematics. 2023; 11(6):1538. https://doi.org/10.3390/math11061538

Chicago/Turabian Style

Teran-Pineda, Diego, Karl Thurnhofer-Hemsi, and Enrique Dominguez. 2023. "Analysis and Recognition of Human Gait Activity Based on Multimodal Sensors" Mathematics 11, no. 6: 1538. https://doi.org/10.3390/math11061538

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop