A New Time–Frequency Feature Extraction Method for Action Detection on Artificial Knee by Fractional Fourier Transform

Abstract

With the aim of designing an action detection method on artificial knee, a new time–frequency feature extraction method was proposed. The inertial data were extracted periodically using the microelectromechanical systems (MEMS) inertial measurement unit (IMU) on the prosthesis, and the features were extracted from the inertial data after fractional Fourier transform (FRFT). Then, a feature vector composed of eight features was constructed. The transformation results of these features after FRFT with different orders were analyzed, and the dimensions of the feature vector were reduced. The classification effects of different features and different orders are analyzed, according to which order and feature of each sub-classifier were designed. Finally, according to the experiment with the prototype, the method proposed above can reduce the requirements of hardware calculation and has a better classification effect. The accuracies of each sub-classifier are 95.05%, 95.38%, 91.43%, and 89.39%, respectively; the precisions are 78.43%, 98.36%, 98.36%, and 93.41%, respectively; and the recalls are 100%, 93.26%, 86.96%, and 86.68%, respectively.

1. Introduction

The knee joint is an important supporting joint of the lower limbs. For a knee joint injury that seriously impedes the movement of the lower limbs, the best treatment is to replace the joint with an artificial joint. Among patients, a large number need amputation, and prosthesis is an important guarantee for the physical and physiological recovery of amputees. In the lower limb prosthesis, the artificial knee joint is the core component and it can help the human body to achieve supported standing and flexible walking so that patients show good gait [1].

Before the 1990s, the movement of prosthetics could not keep pace with the speed of amputees and road conditions, and the movement lacked stability [2]. With the continuous development of bionic intelligent knee prosthesis technology, there are many intelligent knee prostheses, which can achieve dynamic adjustment, real control of gait, and other functions. The intelligent lower limb usually refers to the unpowered intelligent lower limb. The flexion and extension movement of the prosthetic knee joint is driven by the leg stump. The intelligent control system only adjusts the size of the damping torque of the knee joint [3]. For the knee joint of the unpowered intelligent lower limb prosthesis that controls gait by adjusting the damping torque, the accuracy of damping adjustment directly affects the comfort, convenience and safety of the knee joint in question. Generally, the damping adjustment is based on the recognition of the current action state. Among the existing action detection methods of intelligent prosthetics, methods that detect the motion frequency or the position of the piston of a cylinder and the pressure at the bottom of the foot have long lag times and low accuracy. Methods that extract the features of acceleration or angular velocity from the time domain have few features from which to choose, hence they are susceptible to interference and low robustness. So, an accurate, fast, low-cost and stable action detection method is needed.

The action detection often needs to go through the process of data collection, data preprocessing, feature extraction, the establishment of an action detection model, and the use of the model for action detection [4]. Currently, sensors used to collect data can be divided into two major categories of wearable and nonwearable. The nonwearable methods include motion detection methods based on radar [5], and machine vision [6]. The wearable methods include motion detection methods based on accelerometers [7], accelerometers and gyroscopes [8], accelerometers and other sensors [9], and myoelectricity [10]. The nonwearable sensors are not suitable for human motion recognition on prosthesis due to their severe constraints of scene and high cost. In the wearable methods, myoelectricity is inconvenient to wear on a daily basis due to the need to install electrodes in multiple places on the wearer’s legs. The microelectromechanical systems (MEMS) inertial measurement unit (IMU) integrated with accelerometers, gyroscopes and barometers can be integrated into the control circuit and does not require additional operation. So, it is more suitable for human action data acquisition on commercial products.

For feature extraction, the preprocessed data are usually divided into a series of windows by the sliding window mechanism of fixed length first-in, first-out, and the feature values are extracted by a series of methods. Aziz et al., for example, calculated the mean and variance of data in each window to construct 18 dimensional features [11]; Pierleoni et al. discovered that the RMS (Root Mean Square) of acceleration was no less than 2.5 g when the impact process of motion occurs [12]; Cheng et al. judged the state of motion according to an amplitude of acceleration higher than a given threshold [13].

For the establishment of an action detection model, the commonly used methods are the threshold method and the machine learning method. The threshold method determines motion state by comparing the extracted features with a setting threshold; generally, it can be divided into two types: a single state recognition and a multiple state recognition. The machine learning method regards action detection as a typical classification problem and constructs a motion detection model based on the training set composed of various motion data. Typical machine learning algorithms such as support vector machine (SVM) [11,14,15,16], decision tree (DT), Naive Bayes [17], deep learning [18], artificial neural network (ANN) [19], and K nearest neighbor (K-NN) [20] can be used for model construction. The preferable effective algorithms include convolutional neural network (CNN), I support vector machine (1SVM), CNN+1SVM [21], hidden Markova model [22], K-NN and ANN [23].

However, in most action detection methods the machine learning method can achieve higher precision, but it needs lots of features and is a complicated detection model with a slow detection speed and high calculate cost. The threshold method is simple in calculation and fast in speed, which is very suitable for long-term operation on micro-wearable devices with limited resources. However, the selection of its threshold value often depends on the simulated motion data, and there are few features from which to choose. When the distribution of test data and training data is greatly different, it will seriously affect the detection accuracy of the model, so it lacks certain applicability. Therefore, a detection method that can provide sufficient features for selection, at the same time as satisfying high computing speed and low model complexity, is needed. Considering that human action is a time-varying movement, and we cannot only look for features in the time domain or frequency domain, we came up with the idea of using the fractional Fourier transform (FRFT) in order to provide sufficient features from the fractional domain, and also in order to obtain suitable features for low power consumption wearable devices.

Fractional Fourier transform (FRFT) is a representation method on the fractional-order Fourier domain by the signal on the time–frequency plane after the coordinate axis rotates counterclockwise around the origin at any angle. It is a time–frequency analysis method and a generalized Fourier transform. Fractional Fourier transform has many properties that traditional Fourier transform does not have, and is widely applied in scientific research and engineering technology [24]. In this paper, firstly the effect of FRFT is analyzed, and the results of transformation with the same action in different orders, or different actions in the same order, are compared. Then, eight features are selected from the commonly used features to construct a feature vector space. The variation of each feature after FRFT in different orders is analyzed. Then, the dimension of the feature vector is further reduced, and the results of different features after FRFT in different orders are analyzed, according to which we design a classifier. Finally, a prototype of an intelligent knee joint is designed, and the method is verified by experiments.

2. Theory

2.1. Fractional Fourier Transform (FRFT)

In the field of signal processing, the traditional Fourier transform is a mature and widely used mathematical tool. The fractional order Fourier transform (FRFT) is proposed in the form of pure mathematics by V. Namias from the view of feature and feature function [25]. Then, researchers proposed the concept of FRFT from an optical point of view. It can be proved that these definitions are completely equivalent [26]. FRFT is first applied to optical signal processing because it can be implemented by simple optical devices. In recent years, several rapid algorithms for FRFT have been discovered, so that FRFT has received attention in multiple areas of signal processing.

2.1.1. Definition

Generally, the p-order fractional Fourier transform of a function $f_p(u)$ can be expressed as: $f_p(u)$ or $F^{p}f(u)$, where, $F^{p}f(u)$ can be interpreted as the operator $F^p$ on the function $f(u)$ whose result is in the domain $u$.

The definition of the fractional Fourier transform [27] is

$$f_p(u)={\displaystyle {\int }_{-\infty }^{+\infty }{K}_p(u,t)f}(t)dt$$

(1)

where $K_p(u,t)=\{\begin{array}{l}{A}_{\alpha }\exp [j\pi (u^2\cot \alpha -2ut\csc \alpha +t^2\cot \alpha )],\alpha \ne n\pi \hfill \\ \delta (u-t),\alpha =2n\pi \hfill \\ \delta (u+t),\alpha =(2n+1)\pi \hfill \end{array}$ is the kernel function of the fractional Fourier transform, $A_{\alpha }=\frac{\exp [-j\pi \mathrm{sgn}(\sin \alpha )/4+j\alpha /2]}{{\left|\sin \alpha \right|}^{1/2}}$, $\alpha =\frac{p\pi }{2}$, $n$ is integer.

After sorting, it is shown as follows:

$$f_{\alpha }(u)=A_{\alpha }{T}_t(u){\displaystyle {\int }_{-\infty }^{+\infty }{T}_s(u-x)[T_t(x)f(x)]dx}$$

(2)

where $T_t(x)=\exp (-j\pi t{x}^2)$, $t=\tan (\alpha /2)$, $s=-\csc (\alpha )$.

Noticing that, $F^{4n}$ and $F^{4n\pm 2}$ are equivalent to identity operator $\tau$ and parity operator $P$ respectively. For $p=1$, there are $\alpha =\frac{\pi }{2}$, $A_a=1$ and $f_1(u)={\displaystyle {\int }_{-\infty }^{+\infty }{e}^{-j2\pi ut}f(t)dt}$.

Apparently, $f_1(u)$ is the Fourier transformation of $f(u)$. The 0-order transformation is defined as the function itself, and the definition by $p$ or $\alpha$ is periodic in 4 or $2\pi$ due to $\alpha =\frac{p\pi }{2}$ only appearing in the parameter position of the trig function.

Except for comparison with the time domain or the frequency domain, to explain the reason that choose FRFT to extract features, Li Chao derived the relationship between the amplitude mean $\overline{A}$ in the time domain and the amplitude B in the fractional domain by the timewidth–bandwidth in theory [28].

From the fraction domain sampling theorem, the relationship between the fraction domain bandwidth $Bu$ and the frequency domain bandwidth B is:

$$Bu=B\sin \alpha$$

(3)

where $\alpha$ is the rotation angle of the fraction domain.

The timewidth and bandwidth of the fraction domain signal are defined as follows:

$$\Delta t^2\triangleq {{\displaystyle {\int }_{-\infty }^{+\infty }\left|(t-t_0)x(t)\right|}}^{2}dt$$

(4)

$$\Delta u_a{}^2\triangleq {{\displaystyle {\int }_{-\infty }^{+\infty }\left|(u_a-u_a{}_0)x(u_a)\right|}}^{2}d{u}_a$$

(5)

where $u_a$ is the frequency of the fractional domain and $x(u_a)$ is the FRFT of $x(t)$.

From the uncertainty principle of the signal frequency domain:

$$\Delta t^2\cdot \Delta u^2\ge \frac{1}{4}\mathrm{is}(\Delta t\cdot \Delta u\ge \frac{1}{2})$$

(6)

where $\Delta t$ is the timewidth and $\Delta u$ is the bandwidth.

The uncertainty principle of the fractional domain can be derived from the three Equations (4)–(6):

$$\Delta t^2\cdot \Delta u_a{}^2\ge \frac{{\sin }^2\alpha }{4}is(\Delta t\cdot \Delta u_a\ge \frac{\sin \alpha }{2})$$

(7)

From Parseval, the energy of the signal in the time domain is the same as that in the fractional domain:

$$E=E_{\alpha }$$

(8)

where $E$ is the total energy in the time domain, and $E_{\alpha }$ is the total energy in fraction domain. Furthermore:

$$E=\frac{1}{2}\overline{A}\cdot \Delta t^2,E_{\alpha }=\frac{1}{2}\overline{B}\cdot \Delta u_{\alpha }{}^2$$

(9)

Combining Equations (7)–(9), it can be observed that the signal’s amplitude average $\overline{A}$ in the time domain and the $\overline{B}$ of the signal show a non-linear change, and it is related to the changed angle in the fractional domain [29,30].

Therefore, a sequence of values with no significant difference in amplitude in time domain can be converted into a fractional sequence with significant differences in amplitude by choosing an appropriate FRFT order [28].

2.1.2. Discrete FRFT

Equation (2) is the calculation method in the continuous domain. Such continuous transformation cannot be calculated in practice, and it is usually necessary to carry out numerical calculation by sampling and interpolating continuous signals. According to the Shannon interpolation equation and numerical integration operation, the discrete calculation equation of FRFT is shown in Equation (10):

$$f_{\alpha }(\frac{k}{2\Delta })\approx \frac{A_{\alpha }}{2\Delta }\exp [-j\pi \tan (\alpha /2){(\frac{k}{2\Delta })}^2]\times {\displaystyle \sum _{l=-N}^{N-1}\exp [j\pi {(\frac{k-l}{2\Delta })}^2\csc \alpha ]\left\{\exp [-j\pi {(\frac{l}{2\Delta })}^2\tan (\alpha /2)f(\frac{l}{2\Delta })]\right\}}$$

(10)

After sorting, it is shown as follows:

$$f_{\alpha }(x_k)\approx \frac{x_k{C}_{\alpha }}{k}{E}_t(u_k){\displaystyle \sum _{l=-N}^{N-1}\{E_s(x_{k-l})[E_t(x_l)f(x_l)]\}}$$

(11)

where $E_t(u_k)=\exp (-j\pi t{u}_k^2)$, $t=\tan (\alpha /2)$, $s=-\csc (\alpha )$ and $x_k=u_k=k/2\Delta$. $\Delta$ is the dimension normalized time domain or frequency domain scale.

2.1.3. Transform Effect

According to the theory above, the inertial data collected by MEMS can be processed by FRFT. In contrast to the time domain and the frequency domain, the fractional domain has a better diversity and can provide more optional features. Figure 1 and Figure 2 are the results of the same data after FRFT in different orders. Figure 3a is the results of the same class data after FRFT in the same order. Figure 3b is the results of different data after FRFT in the same order. It can be observed that the results of the same data after FRFT in different orders show significant differences, and the results of different data after FRFT in the same order also show differences. Therefore, different actions can be distinguished by selecting an appropriate order and an appropriate feature.

2.2. Feature

After the performance of FRFT on the collected data, the appropriate features should be extracted and then classified according to the features. The commonly used features are shown in

Table 1. The commonly used features and the calculation method.

Features	Calculation Method
Maximum	$max(A)$
Minimum	$min(A)$
Mean	$mean(A)={\displaystyle \sum _i\frac{A_i}{n}}$
Extreme Difference	$range(A)=max(A)-min(A)$
Variance	$var(A)=\frac{{\displaystyle \sum _i{(A_i-mean(A))}^2}}{n}$
Standard Deviation	$std(A)=\sqrt{var(A)}$
Root Mean Square	$rms(A)=\sqrt{\frac{{\displaystyle \sum _i{A}_i^2}}{n}}$
Absolute Value	$abs(A_i)=\left|A_i\right|$
Signal Amplitude Area	$sma(A)=\frac{1}{t}({\displaystyle {\int }_0^t\left|A_x\right|dt+}{\displaystyle {\int }_0^t\left|A_y\right|dt+}{\displaystyle {\int }_0^t\left|A_z\right|dt})$
Correlation Coefficient	$cc(A)=\frac{cov(A_x,A_y)}{var(A_x,A_y)}$
Interquartile Range	$IQR=Q_3-Q_1$
Number of Peaks	The number of peaks in signal ($pksNum$)
Mean of Peaks	$pksMean=\frac{{\displaystyle \sum _{\mathrm{i}}pk{s}_i}}{n}$

. However, not all the features are appropriate for the data in MEMS IMU, besides, too many features will lead to excessive computing costs, time costs, storage costs, and reduce the endurance of the device. However, too few features will not reach enough accuracy to classify different actions. Therefore, it is necessary to select appropriate features for action recognition.

In this paper, the extreme difference $range$, standard deviation $std$, variance $var$, interquartile range $IQR$, mean $mean$, mean of peaks $pksMean$, and number of peaks $pksNum$ were selected to form the $feature\_vector$:

$$feature\_vector=[range,std,var,rms,IQR,mean,pksMean,PksNum]$$

(12)

Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 are the analysis of each feature during the order of FRFT changed from 0 to 1. Figure 4, Figure 6, Figure 8 and Figure 10 are the FRFT results of the eight features of four actions, namely walk, run, upstairs, and dwstairs (downstairs), in different orders. Figure 5, Figure 7, Figure 9 and Figure 11 are the standard deviation and mean of eight features of each action.

Table 2. Mean and min of standard deviation of each feature of walk.

Feature	STD
	Min		Mean
	Value	Order
Range	0.6996	0.390	1.1189
Standard Deviation (Std)	0.1494	0.453	0.1854
Variance (Var)	0.4785	0.453	0.6612
Root Mean Square (RMS)	0.0382	0.635	0.0387
Interquartile Range (IQR)	0.2171	0.310	0.3105
Mean	0.2799	1	0.3076
Mean of Peaks (pksMean)	0.6857	0.043	1.1725
Number of Peaks (pksNum)	0.5430	0.992	2.6806

,

Table 3. Mean and min of standard deviation of each feature of upstairs.

Feature	STD
	Min		Mean
	Value	Order
Range	1.9913	0.981	3.0821
Std	0.4898	0.997	0.5659
Var	2.4833	0.428	3.1291
RMS	0.0527	0.607	0.0535
IQR	0.3743	0.130	0.7365
Mean	0.3249	1	0.7060
pksMean	1.9208	0.701	2.8684
pksNum	0.5633	1	2.7401

,

Table 4. Mean and min of standard deviation of each feature of dwstairs.

Feature	STD
	Min		Mean
	Value	Order
Range	0.6453	0.400	1.2644
Std	0.1589	0.281	0.1978
Var	0.4816	0.308	0.7028
RMS	0.0209	0.624	0.0213
IQR	0.2369	0.212	0.3018
Mean	0.1681	1	0.2500
pksMean	0.6681	0.194	1.3048
pksNum	0.6485	0.997	2.9343

and

Table 5. Mean and min of standard deviation of each feature of run.

Feature	STD
	Min		Mean
	Value	Order
Range	2.3835	0.803	3.3444
Std	0.4612	0.168	0.5488
Var	3.6646	0.169	4.9012
RMS	0.0783	0.663	0.0803
IQR	0.6780	0.318	0.8944
Mean	0.6471	0.020	0.8000
pksMean	0.9936	0.031	1.2464
pksNum	2.8274	0.991	4.4601

are the mean, minimum standard deviation and corresponding order of standard deviation of each feature of each action. It can be observed that, except for $rms$, all the other features change significantly, but this does not mean that $rms$ cannot be used for classification. Besides, it can be observed that it might be useful to classify walk and dwstairs by $rms$ and $mean$, to classify upstairs and dwstairs by $range$ and $pksNum$, and to classify walk, dwstairs and run by $std$ and $pksNum$.

Different features have different classification effects for the inertia data in MEMS IMU. Figure 12 shows the effect of distinguishing two actions by two kinds of features. It can be observed that to classify walk and dwstairs, the combination of $pksNum$ and $std$ is better than the combination of $range$ and $pksMean$, and the order = 0.67 is better than order = 0.20. To classify upstairs and dwstairs, the combination of $range$ and $pksNum$ is better than the combination of $rms$ and $IQR$, and the order = 0.64 is better than order = 0.20. To classify walk and upstairs, the combination of $rms$ and $mean$ is better than the combination of $std$ and $IQR$, and the order = 0.71 is better than order = 0.20. To separate run from the other actions, the combination of $rms$ and $mean$ is better than the combination of $range$ and $IQR$, and it is better when order = 0.75. Figure 13 shows the effect of classification with two features extracted directly in the time domain without FRFT. It can be observed that the effect was not good for classification.

2.3. Classifier

According to the conclusion above, different features and different orders have different effects for classification. The optimal order and feature of each action are different. A binary classifier was designed which classifies by changing the order and feature value of each sub-classifier. Only one action is separated at a time, and then all motion gestures are recognized by multiple classifications. The structure of the classifier is shown in Figure 14.

3. Experiment

3.1. Experiment Design

According to the conclusion above, an experiment for verification is designed. The information of the subject is shown in

Table 6. Information of the subject.

Project	Value
Age	24
Weight	65 kg
Position of prosthesis	Left
High	174 cm
Gender	M

. To protect the privacy of amputees, the photos of amputees will not be shown here. The artificial knee used in the experiment is shown in Figure 15, the MEMS IMU used in the experiment is shown in Figure 16, and the main specifications of the devices are given in

Table 7. Main specifications of devices.

Parameters		Main Specifications
Sustainable Working Hours		≥ 24 h
Operating Voltage		3.3 to 30 V
Sustainable Working Temperature		−40 °C to 85 °C
Power Consumption		550 [email protected] V
Core Circuit Board Dimensions		33 mm × 20 mm × 22 mm
Weight		<100 g
Sensing Range		0° to 360°
Static Accuracy		±0.5° (roll, pitch); ±1° (yaw)
Dynamic Accuracy		±1° RMS
Resolution		0.05°
Output Frequency		0.01 to 100 Hz
Sensor	Accelerometer	Gyroscope	Parameters	Barometer
Measure range	±10 g	±1000°/s	Measure range	10 to 1200 mbar
Nonlinear	<0.2% of FS	<0.1% of FS	Resolution	10 cm
Bias stability	±4 mg	9.2°/h	Bias stability	±1 mbar/year

.

3.2. Results and Analysis

After the experimental data collection, the same processing method is used to extract the feature vector through FRFT after dividing the cycle, and the classifier designed above is used for classification, the results are shown in Figure 17 and the performance of the sub-classifier is detailed in

Table 8. Performance of each sub-classifier.

Sub-Classifier	b	c	d	e
Accuracy	0.9505	0.9538	0.9143	0.8939
Precision	0.7843	0.9836	0.9836	0.9341
Recall	1	0.9326	0.8696	0.8667
F1–Score	0.3053	0.7018	0.7157	0.6753

. The comparison of the accuracy in fractional domain and time domain is in

Table 9. Accuracy of each sub-classifier in fractional domain and time domain.

Sub-Classifier	Fractional Domain	Time Domain
b	0.9505	0.9011
c	0.9538	0.8259
d	0.9143	0.7424
e	0.8939	0.7629

. It can be seen that the effect of fractional domain is better than the effect of time domain.

4. Conclusions

With the aim of designing an action detection method on artificial knee, a new time–frequency feature extraction method is proposed. This method is targeted at four common actions of the artificial knee wearer, and extracted features from the inertia data are measured by MEMS IMU, using fractional Fourier transform (FRFT) to magnify the diversity of features. FRFT is employed to extract the appropriate feature vectors and construct a feature vector composed of eight features.

By analyzing the results of these features after FRFT in different orders, it was discovered that it may have a good effect to classify walk and dwstairs by $rms$ and $mean$, to classify upstairs and dwstairs by $range$ and $pksNum$, and to classify walk, dwstairs and run by $std$ and $pksNum$.

The results of the classification with different features and orders are also analyzed. It was discovered that in order to classify walk and dwstairs, the combination of $pksNum$ and $std$ is better than the combination of $range$ and $pksMean$, and it is better when order = 0.67 than order = 0.20. To classify upstairs and dwstairs, the combination of $range$ and $pksNum$ is better than the combination of $rms$ and $IQR$, and it is better when order = 0.64 than order = 0.20. To classify walk and upstairs, the combination of $rms$ and $mean$ is better than the combination of $std$ and $IQR$, and it is better when order = 0.71 than order = 0.20. To separate run from the other actions, the combination of $rms$ and $mean$ is better than the combination of $range$ and $IQR$, and it is better when order = 0.75.

Finally, the verification experiment is designed with the artificial lower limb knee prototype. The results show that the method that we proposed has a good classification performance while also reducing the requirements of hardware calculation. The accuracies of each sub-classifier are 95.05%, 95.38%, 91.43%, and 89.39%, respectively; the precisions are 78.43%, 98.36%, 98.36%, and 93.41%, respectively; and the recalls are 100%, 93.26%, 86.96%, and 86.68%, respectively.