A Novel Activity Recognition System for Alternative Control Strategies of a Lower Limb Rehabilitation Robot

Abstract

Robot-aided training strategies that allow functional, assist-as-needed, or challenging training have been widely explored. Accurate activity recognition is the basis for implementing alternative training strategies. However, some obstacles to accurate recognition exist. First, scientists do not yet fully understand some rehabilitation activities, such as abnormal gaits and falls; thus, there is no standardized feature for identifying such activities. Second, during the activity identification process, it is difficult to reasonably balance sensitivity and specificity when setting the threshold. Therefore, we proposed a multisensor fusion system and a two-stage activity recognition classifier. This multisensor system integrates explicit information such as kinematics and spatial distribution information along with implicit information such as kinetics and pulse information. Both the explicit and implicit information are analyzed in one discriminant function to obtain a detailed and accurate recognition result. Then, alternative training strategies can be implemented on this basis. Finally, we conducted experiments to verify the feasibility and efficiency of the multisensor fusion system. The experimental results show that the proposed fusion system achieves an accuracy of 99.37%, and the time required to prejudge a fall is approximately 205 milliseconds faster than the response time of single-sensor systems. Moreover, the proposed system also identifies fall directions and abnormal gait types.

1. Introduction

Rehabilitation robots have received considerable attention in recent years due to the increasing tendency toward rehabilitation and the shortage of rehabilitation physicians. According to statistics, one out of every six people in the world may suffer from stroke [1], making it a global problem that seriously endangers human health. In China, approximately 2.5 million new brain injuries and spinal cord injuries occur each year [2]. These patients need rehabilitation treatment after surgery to restore normal function, but few rehabilitation medical resources exist. There is an urgent need for approximately 350,000 rehabilitation technicians, but fewer than 20,000 professionals currently work on rehabilitation. In addition, traditional physical therapy is labor intensive and repetitive [3]; therefore, rehabilitation robots have emerged.

A rehabilitation robot is a device that assists rehabilitation therapists in repetitive rehabilitation training tasks based on neurological rehabilitation mechanisms. In addition to undertaking intensive, repetitive rehabilitation training, modern rehabilitation robots should also have the ability to recognize the patient’s motor intent and physical condition. If given such recognition abilities, a high-level rehabilitation strategy could be achieved. In recent years, robot-aided training strategies that support functional, assist-as-needed, or challenging training regimens have been widely explored. These strategies can better provide motor plasticity. The primary goal of alternative robot-aided training strategies is to accurately recognize the various patient states in real time to provide an auxiliary judgment basis for the top-level control system and to provide the necessary control input for the bottom controller. These states include the patient’s motor intent and tolerance degree, the physical interaction (i.e., asynchronous error) between the patient and the robot, the patient’s abnormal motor pattern (e.g., abnormal gait) and the tasks that must be completed for rehabilitation assessment [4], and patient fall detection. Among these, fall detection is critical, because there is a high probability of falls during rehabilitation, and these falls can cause severe injuries. Statistically, the probability of a stroke survivor falling within six months after surgery is 25%–37%, and subsequently, the fall probability is 23%–50% [5,6,7,8]. Accidental falls and fall-related injuries, such as hip fractures, often lead to severe disabilities and affect the patient’s overall health [9]. Therefore, timely and effective fall prediction and protection during rehabilitation training are beneficial for stroke patients.

The detection of abnormal gait (patient-specific gait) [10] and the fall methods currently used in rehabilitation training can be roughly divided into three methods: image sensing, environmental sensors, and wearable sensors. Image detection works mainly through cameras installed in the environment, and uses image-processing algorithms to detect whether a patient has fallen [11]. This approach does not require patients to wear testing equipment; however, the image-processing algorithms are sophisticated and easily affected by light and noise, which degrades the accuracy and applies only to fixed locations, providing little applicability to a wider range of activities. Infrared sensors and acoustic sensors are generally installed [12,13] to detect whether a fall occurs in a specific environment. This approach is inexpensive, but it is limited to environments where the sensors are installed. Wearable sensor methods [14,15,16,17,18,19] mainly analyze the acceleration sensor data of limbs and then use an algorithm to identify falls. Wearable sensors have been widely studied and applied because of their small volume and low cost.

In terms of detection algorithms, the methods used in fall and abnormal behavior detection are mainly threshold-based and rely on machine learning (pattern matching) classifiers, such as the support vector machine (SVM) [20,21,22], k-nearest neighbor (KNN) [14], backpropagation (BP) and single-class minimax probability machine (SCMPM) [23]. These methods usually require a threshold value to be set manually or through a semisupervised process to distinguish abnormal from standard samples. Threshold-based systems are simple to implement and computationally inexpensive, but they do not provide a good trade-off between false positives (if the threshold value is too low, the system could generate false alarms, specificity <100%) and false negatives (if the threshold value is too high, the system could miss real alarms, sensitivity <100%) because soft falls due to paralysis or fatigue of the gluteus medius are characterized by kinematic peaks similar to those of normal gaits, and small perturbations or “wiggly duck gait” are characterized by kinematic peaks similar to those of falls. In other words, an extrinsic fall [9,24] can be identified by a high threshold because extrinsic falls caused by trips and slips are often accompanied by sharp changes in acceleration. In contrast, intrinsic falls caused by dizziness, impaired balance, or gait [9,24] usually do not exhibit a sharp change in acceleration, and can easily go undetected. In addition, these methods cannot perform detailed classifications of falls or abnormal gaits. In addition, existing abnormal gait and fall detection research faces other problems, such as missed alarms and false alarms [25,26,27] and inadequate understanding of falls and abnormal gait [28,29]. Currently, no standardized methodology in terms of sensor types, feature engineering [24], or standard classification samples exists for identifying these problems. Although many studies on activity recognition and fall detection exist, these studies tend to use a single sensor (usually a kinematic sensor) for pattern recognition, largely for cost reasons. However, the potential problem of this approach is that it is difficult for a single dimension input to fully reflect the gait characteristics; thus, there is some risk when using a single type of sensor for pattern recognition. In addition, the sensors are easily disturbed by noise or unexpected situations, which inevitably results in uncertainty. Under interference from noise, it is difficult to achieve accurate determination and identification. Kinematic features such as falling speed, stride time, acceleration coordinates, posture information, inactivity periods, and angular velocity vary by individual, especially for different age groups. Moreover, most of these studies are intended to raise alarms only after a fall occurs. Some studies [19] determined that a fall occurs when the acceleration threshold is triggered, and the wearer subsequently remains in a quiet position for a short duration. These studies have been applied to improve the safety of elderly people, but they are not applicable during rehabilitation training, because when a patient is at risk of falling, the robot cannot make a timely judgment; however, if the robot continues to operate, it is likely to cause serious harm to the patient. Therefore, from the perspective of fall injury prevention, timeliness is as crucial as accuracy [30]. During the identification process, a faster yet accurate judgment allows better subsequent measures to be implemented. However, kinematic-sensor-based approaches often require high thresholds to achieve high accuracy, and high thresholds correspond to later predictions. To make accurate judgments in the minimum time, it is worth studying sensor selection and other features to compensate for the defects of kinematic sensors for recognizing complex activities.

Given these problems, we have conducted some investigations and hope to find a sensor system that can be applied to weight support rehabilitation training and conduct basic research on criteria for activity recognition in patients with hemiplegia. To accurately recognize various patient and robot states in real time, we propose a multisensor fusion system that recognizes three dimensions: kinematics, kinetics, and position distribution. Our system distinguishes between periods of training and rest, detects events such as normal gait and abnormal gait, and achieves a reasonable degree of accuracy. In a previous study, we proposed a two-stage activity recognition classifier, in which a probabilistic neural network (PNN) is first trained to identify a fall in real time, and then data are transferred to a second level for detailed classification.

Based on this recognition system, our previous paper also proposed an adaptive neural sliding-mode controller (ANSMC) that provides a continuum of either assistive or challenging rehabilitation training. This controller was applied to a mobile chaperone lower limb rehabilitation-training robot (MCLLRTR). The main contributions of this paper are summarized as follows:

• We propose a multisensor fusion system that provides accurate activity and motor capability recognition fall prediction and physical fitness assessment during the rehabilitation training process. It uses the multidimensional features provided by multiple sensors with differing properties to achieve a reasonable degree of identification accuracy. To overcome the shortcomings of the kinematic sensor, we propose multisensor activity identification based on kinematics, kinetics, and spatial distribution.
• We introduce a two-stage activity recognition classifier in which a probabilistic neural network (PNN) is first trained on regular activities to filter out falls. Then, the fall data are input to an SVM-RBF-KNN algorithm, which classifies specific falls or an abnormal gait in a supervised manner. The first stage predicts falls as early as possible, and the second stage determines fall details and abnormal gait types. These detailed classifications are provided to the therapist for functional assessment and the subsequent formulation of the therapeutic schedule.
• We have tried to address the contradictions of sensitivity and specificity. By selecting appropriate sensors whose features can compensate for each other’s shortcomings, we can achieve both high sensitivity and reasonable specificity.

Additionally, to provide proper safety protection for rehabilitation patients, the motor locking commands are triggered when the algorithm predicts falls, and the bodyweight support sling provides protection.

The rest of this article is organized as follows. Section 2 introduces the methods and system setup. Section 3 presents the experimental results. Finally, conclusions are summarized in Section 4.

2. Methods and System Setup

We cooperated with the China Rehabilitation Research Center (CRRC) to develop a mobile chaperone lower-limb rehabilitation-training robot (MCLLRTR). The goal was to integrate active assistance and challenge-based training control strategies into the robot’s control system. The robot is intended to assist patients recovering from poststroke hemiplegia and who have an exceptional ability to move freely from the synergy pattern. Additionally, the robot can provide patients with real-time protection and provide doctors with a physical fitness assessment, which promotes subsequent diagnosis. Therapists can utilize a selection of training modes and parameters to adjust the challenge level to provide scalable therapy that is catered to each patient’s needs. For example, therapists can steer the MCLLRTR at any time and support gait initiation or nudge patients in a specific direction.

2.1. System Introduction

The MCLLRTR activity recognition process is shown in Figure 1. First, the height and weight of the patient are measured, and the therapist makes a functional assessment of the patient. Then, the essential data are input into MATLAB and compared with the standard model, and a detection threshold is specified for each patient off-chip. In step two, zero-point correction of the tension sensor is performed, the abnormal gait trigger position settings for the photoelectric sensors are set, and preparations are made to begin training. In the third step, the training data are transmitted to the STM32 main control board after Kalman filtering and analog/ digital (AD) conversion. Based on the sample features and offline thresholds from the first step, the proposed first-stage classifier (PNN) determines whether the data indicate a ‘fall’ or a ‘normal gait’. If a fall has occurred, protection measures are launched, and the robot initiates safety protection measures; otherwise, rehabilitation training continues. The rehabilitation control strategies adaptively adjust based on the patient’s performance, and all data are transmitted to the host computer through the wireless serial port in real time. As the final step, the data are uploaded for detailed analysis and classification during training, training reports are generated, new patient data are sent to the database, the neural network training parameters are updated, and the fall determination thresholds are updated.

2.2. Proposed Methods

A block diagram of the MCLLRTR’s control system is shown in Figure 2. First, the control architecture includes a local sensing and recognition system in which local variables are adopted for state detection and activity pattern recognition. Second, the asynchronous force and pulse are employed as inputs to identify the tolerance degree of the patients and training task completion differences in the mid-level control; we designed a transition law suitable for active assistance, challenge-based, or regular training. Third, we designed an adaptive RBF neural network sliding mode (ARNNSM) controller that executes a selected control strategy. Furthermore, a patient’s training state, which is obtained from gait, force, and kinematics data, is monitored by the photoelectric sensor, tension sensor, and acceleration sensor, providing security protection that is as accurate and immediate as is possible. We developed a two-stage activity recognition neural network algorithm to perform activity recognition and fall detection using the collected data. We hypothesized that the new algorithm would achieve a reduction in the time delay between a detected event and an actual fall such that it could guarantee enough time to institute fall protection. In the low-level motion controller, the MCLLRTR controls its velocity and trajectory based on either the potentiometer signal of the control handle or the specific value set by the rehabilitation therapist. When the detection system predicts a fall, the MCLLRTR brakes immediately and deploys safety protection for the patient through a body weight support sling.

2.3. Sensor Locations

It is challenging to identify diverse falls and abnormal motor patterns from routine activities because there are many similar features between falls and the activities of daily life (ADLs). Thus, it is challenging to achieve a good trade-off between false positives (when the threshold value is too low, the system may generate false alarms, specificity <100%) and false negatives (when the threshold value is too high, the system may miss real alarms, sensitivity <100%) when setting thresholds using a single sensor. To solve this problem, we proposed a multisensor system (as shown in Figure 3) that obtains multiple features for classifying activities from different dimensions. This system board is designed with an STM32 microprocessor powered by a 5-V battery. The photoelectric sensors collect the spatial distribution of the gait data, the tension sensors collect the interactive forces between the patient and the rehabilitation robot, and the accelerometer sensor collects the patient’s kinematic features.

As shown in Figure 3, the multisensor system includes an accelerometer, a pulse sensor, and two photoelectric sensors, which include four tension sensors and seven photoelectric sensors. The photoelectric sensors include a speed-detecting sensor. The accelerometer monitors the patients’ spatial motion from the kinematic dimension. The pulse sensor is used to detect physical load. The tension sensor monitors the motion homogeneity between the robot and patient from the kinetics dimension, and the photoelectric sensors monitor gait from the spatial distribution characteristics. At the same time, to meet the needs of the rehabilitation assessment, we introduce asynchronous deviation and pulse monitoring based on the multisensor system. The tension sensors mainly obtain the synchronous deviation, and the tolerance degree of the patients is mainly obtained by synthesizing the data of the pulse sensor and the idle duration. The activity recognition system also provides data support for dynamic control.

The spatial distributions of normal and abnormal gaits (such as the wiggly duck gait) are different. Based on this characteristic, the OMRON (E3JK-RAMA, Japan) photoelectric sensors are applied to this type of detection. We located the photoelectric sensors below the ankle and above the knee in a vertical direction to detect whether the lower leg is beyond the normal gait range or at a healthy gait boundary in the horizontal direction (as shown in Figure 1 and Figure 3). These sensors detect whether the foot’s landing point is beyond the range of a normal gait. The photoelectric sensors include a transmitter, a receiver, and a detection circuit. The transmitter continuously emits light; that is, the light is received by the receiver or blocked by patients. The receiver converts the received optical signal into an electric signal, and the detection circuit is used to detect the electrical signals received by the receiver. Signal impurities are removed, leaving the valid signals; then, the valid electric signals are transmitted to the main control board.

The spatial distribution of a normal gait is shown in Figure 4 (the blue area). Six photoelectric sensors were installed on the left and front side of the rehabilitation robot. The second set of photoelectric sensors was installed in the middle position on the left side of the robot, and is used to detect the number of training steps. The first and third sets of photoelectric sensors were installed on the front and back sides of the second set of photoelectric sensors, and are used to determine whether the patient’s gait exhibits a normal travel direction. The fourth, fifth, and sixth sets of photoelectric sensors were installed on the front side of the robot, and are used to detect whether the patient’s gait is normal in the lateral direction. The second set of photoelectric sensors is triggered when the patient walks and counts the normal gait steps. The other sets of photoelectric sensors are triggered when an abnormal gait occurs. The locations of the photoelectric sensors are flexible and can be adjusted according to the rehabilitation training needs.

The tension sensor (Micro tension sensor, ZNLBS-100KG, China) is used to detect the interactive force and the consistency of movement between the robot and the patient. This sensor converts force into a microvoltage, which is then amplified by an amplifier to a value between 0–5 volts. Four sets of tension sensors were installed on the joints between the bodyweight support sling and the robot in four diagonal directions (as shown in Figure 1 and Figure 3). An abnormal gait or fall will cause tension force changes between the robot and the patient; thus, classifications can be made based on tension changes. The tension sensor installed between the bodyweight support sling and the main robot frame provides vertical and horizontal interaction force data in real time.

The kinematic sensor chosen is a micro-inertial measurement unit (μ-IMU, a three-axis acceleration sensor that is shown in Figure 3). To acquire real-time motion data, the accelerometer is installed on the patient’s upper trunk, i.e., below the neck and above the waist, as this region is the most suitable for distinguishing falls from other movements during rehabilitation training [18].

A pulse sensor (FengQi, DC 3.3 V, China) was attached to the patient’s ring finger to detect the tolerance degree of the patients (as shown in Figure 3).

2.4. Feature Extraction and Two-Stage Activity Recognition Method

To predict falls as early as possible to ensure that subsequent protection steps can be taken and for detailed functional assessment needs, we proposed a two-stage activity recognition method in which a probabilistic neural network (PNN) is first trained on routine activities. This network identifies potential falls and passes their data to a supervised SVM-RBF-KNN algorithm that can classify falls and abnormal gaits.

The feature extraction procedure involves extracting one element (x_i) from three sets of sensing data (photoelectric sensor data (P_i=[P₁,P₂, P₃,P₄,P₅]), tension sensor data (T_i=[T₁,T₂,T₃,T₄]), and accelerometer sensor data (A_i=[A₁,A₂,A₃])) during a sampling period (t_s). Then, the three elements make up the input data (X_i) that describe the features of patient activity during rehabilitation training. The sensor sampling period in our experiment is $t_s=20\mathrm{ms}$.

Since accelerometer values change dramatically during falls or abnormal gaits, they are often used to predict abnormal conditions. Since the angle of acceleration is related to the sensor’s orientation when worn, it is generally not used for prediction. To eliminate the accelerometer gravity effect and avoid complex threshold comparisons from every direction ($A_x$, $A_y$, and $A_z$), we employed the signal magnitude vector (SMV) algorithm [31] to discriminate between abnormal gaits and falls during normal gait; the SMV algorithm is the most frequently employed algorithm for distinguishing motion:

$$A_{svm}(SignalMagnitudeVector)=\sqrt{{(A_x)}^2+{(A_y)}^2+{(A_z)}^2}$$

(1)

To predict falls, the value of the tension sensor is significant. In particular, the value of the tension sensor can help make a sensitive determination of a flexible fall. The combination of the photoelectric sensor and the acceleration pull force is also valid for abnormal gait determination.

The tension sensors, which are located between the robot and the patient’s waist in different directions, can reflect the force characteristics between the robot and the patient, such as the force direction and the uniformity and continuity of target movement. We adopt d (T₁, T₄) and d (T₂, T₃) to represent the forces in the left and right directions, respectively, and d (T₃, T₁) and d (T₄, T₂) to represent the frontward and backward force directions, respectively. We adopt T_svm to predict falls.

$$d\left(T_1,T_4\right)=T_1-T_4$$

(2)

$$d\left(T_3,T_2\right)=T_3-T_2$$

(3)

$$d\left(T_3,T_1\right)=T_3-T_1$$

(4)

$$d\left(T_4,T_2\right)=T_4-T_2$$

(5)

$$T_{svm}=T_1+T_2+T_3+T_4$$

(6)

2.5. Asynchronous Deviation

To determine the completion of rehabilitation training and the tolerance degree of the patient in real time, we need to detect the consistency and continuity between actual and target movements. Therefore, we introduce a state variable to represent the asynchronous difference. Ideally, the patient and the robot move synchronously without mutual interference, but in most cases, it is difficult for the patients to synchronize with the predefined training target set on the robot because of their gait disorders. We defined this type of interference as an external disturbance in the dynamic model and employed asynchronous deviation to indicate the interference degree. The asynchronous deviation in kinematics can be obtained through the four tension sensors. These tension sensors were installed on the joints between the bodyweight support sling device and the robot in four diagonal positions (as shown in Figure 3).

Asynchronous deviation is disadvantageous for dynamic control, but it can be used to enhance the training effect.

This algorithm is detailed in our previous paper, which mainly describes the process of activity recognition.

2.6. Activity Recognition Method

Activity recognition is performed in two stages (as shown in Figure 5). In Stage 1, the probabilistic neural network (PNN) is first trained on routine activities to identify potential falls. This stage of the classifier is mainly intended to perform rapid prejudgments of possible falls to enable sufficient time to protect patients; therefore, timeliness is the most important factor. As mentioned above, we selected acceleration A_svm and T_svm to predict falls.

PNN is a feed-forward neural network based on the Bayesian criterion and a Parzen window for probability distribution function estimation that also exhibits good pattern recognition accuracy. The most important advantage of PNN is that training it is easy and nearly instantaneous. Other advantages offered by PNN are that the user needs to set only one parameter (the so-called smoothing parameter). This model can achieve reasonable accuracy, even with small numbers of training samples. Moreover, the network is tolerant to erroneous data and operates entirely in parallel without requiring feedback from the individual neurons to the inputs [32].

The PNN consists of nodes allocated in three layers after the inputs:

Pattern layer: In this layer, one pattern node exists for each training example. Each pattern node forms a product of the weight vector and the given example for classification, where the weights entering a node stem from a particular example. Next, the vector is passed to the summation layer.

$$e^{\left[\left(x^T\ast {\omega }_{ki}-1\right)/{\sigma }^2\right]}$$

(7)

Summation layer: This layer weights the output of the hidden neurons belonging to the same class in the hidden layer.

$$\sum _{i=1}^{NK}{e}^{\left[\left(x^T\ast {\omega }_{ki}-1\right)/{\sigma }^2\right]}$$

(8)

Output layer: The output nodes are binary neurons that produce the classification decision:

$$\sum _{i=1}^{NK}{e}^{\left[\left(x^T\ast {\omega }_{ki}-1\right)/{\sigma }^2\right]}>\sum _{i=1}^{NK}{e}^{\left[\left(x^T\ast {\omega }_{kj}-1\right)/{\sigma }^2\right]}$$

(9)

where X is the input data, NK is the number of samples, k is the dimensionality of the training samples, and σ is a smoothing factor, which is the deviation of the Gaussian functions. An appropriate deviation is chosen through experimentation.

In Stage 2, the SVM-RBF-KNN classifier is trained to classify specific falls or an abnormal gait in a supervised manner. The goal of this classifier is to provide the rehabilitation physician with a detailed gait status for functional assessment. Supplying a classification that includes as much detail as possible is the most important factor.

During the rehabilitation process, activity recognition is critical, because in addition to fall prediction, we need to identify and record gait situations so that therapists fully understand the rehabilitation process and provide essential data for designing reasonable and scientific rehabilitation therapy schedules. Traditional machine learning techniques such as perception, backpropagation (BP) neural network, and one-class support vector machines (OC-SVM) have limitations in modeling human behavior because they lack the reference points of human decision-making [33]. They discriminate a target class from other data by calculating the optimal hyperplane with a bias term such that all training data patterns are classified to the target class [34]. However, for many real-life datasets, such a hyperplane may not exist, and thus, the input space must be mapped to a new feature space with a specific structure (the same as for OC-SVM). Therefore, we proposed a multiple classifier system that combines the SVM, radial basis function (RBF), and KNN classifiers to form an ensemble, which significantly improves fall detection and activity recognition systems.

The k-nearest neighbor (KNN) algorithm is a simple and effective technique for object classification based on finding the closest training examples in the feature space [35]. The radial basis function (RBF) has been widely applied in approximation and classification tasks because of its structural simplicity; its learning process converges quickly, and no local minimum problem occurs. Therefore, we proposed a fusion algorithm (SVM- RBF-KNN) based on extending SVM, KNN, and RBF to recognize and classify complex rehabilitation activities. We set up seven cluster centers based on evaluation criteria for abnormal and normal gaits and falls provided by a rehabilitative therapist, and combine these with real data. Let x₁, x₂,... x_n $\in R^d$ be the n training samples; C_i is the ith cluster center; and the representation $y_i$ of the hidden layer, which maps the input x_k into y_i, is calculated using the following nonlinear function:

$$y_i=\Phi \left(X_k,X_i\right)=\omega G\left(X_k,X_i\right)=\omega G\left(X-X_i\right)=\omega e^{\left(-\frac{1}{2{\sigma }^2}{X}_k-X_i\right)}$$

(10)

$$\sigma =\frac{d_{max}}{\sqrt{2n}}$$

(11)

where $\sigma$ is a parameter scaling the Euclidean distance between $\Phi \left(X_k)\mathrm{and}\Phi (X_i\right)$, and $d_{\mathit{max}}$ is the maximum distance between the selected centers, $X_i\in C_i$, where $C_i$ denotes the neighborhood of x_i in the feature space Ƒ. To obtain reliable clustering centers, we adopted a supervised learning algorithm. The RBF function center $C_i$, expansion constant ${\sigma }_j$, and output weight ${\omega }_j$ should be trained by a supervised learning algorithm and undergo a process of error-corrected learning to obtain their optimal values. The gradient descent method is used to define the objective function as follows:

$$e_k=d_k-\sum _{i=1}^I{\omega }_{j}G\left({\Vert X_k-t_j\Vert }_{C_j}\right)$$

(12)

$$E=\frac{1}{2}\sum _{i=1}^n{e}_k{}^2$$

(13)

$$\Delta t_j=-\eta \frac{\partial E}{\partial t_j}$$

(14)

$$\Delta {\omega }_j=-\eta \frac{\partial E}{\partial {\omega }_j}$$

(15)

$$\Delta {\sigma }_j=-\eta \frac{\partial E}{\partial {\sigma }_j}$$

(16)

The algorithm iterates until the cluster centers are obtained.

The weight $\mathsf{\omega }$ can be obtained as follows:

$$\frac{\partial E\left(n\right)}{\partial {\omega }_i\left(n\right)}=\sum _{k=1}^N{e}_k\left(n\right)G\left({\Vert X_k-t_i\Vert }_{C_i}\right)$$

(17)

$$\omega \left(n+1\right)={\omega }_i\left(n\right)-{\eta }_1\frac{\partial E\left(n\right)}{\partial {\omega }_i\left(n\right)},i=1,2,\cdots ,I$$

(18)

The SVM-RBF-KNN Algorithm 1 for acquiring the cluster centers is shown below:

Algorithm 1: SVM-RBF-KNN algorithm

<Sample ID#>   Input samples: {photoelectric sensors signal: P1, P3, P4, P6}, {tension sensors signal: Tsvm, d (T1, T4), d (T2, T3), d (T1, T3), d(T4, T2)}, {IMU sensors signal: Asvm}.   Transform IMU sensor signal to SVM value:   $A_{svm}(SignalMagnitudeVector)=\sqrt{{(A_x)}^2+{(A_y)}^2+{(A_z)}^2}$   Initialization;   Randomly select eight cluster centers [$X_1,X_2,\dots ,X_i$];   for i = 1: maximum iterations     if E<=0.00001       break;   for i = 1 to N do,   Forward propagation:   compute output and the error:     $e_k=d_k-y_i$;   compute sum error:     $E=\frac{1}{2}{\displaystyle \sum _{i=1}^n}{e}_k{}^2$;   Backward propagation:   update parameters:   $\Delta t_j=-\eta \frac{\partial E}{\partial t_j}$   $\Delta {\omega }_j=-\eta \frac{\partial E}{\partial {\omega }_j}$   $\Delta {\sigma }_j=-\eta \frac{\partial E}{\partial {\sigma }_j}$   Iterate until cluster centers [$C_1,C_2,\dots ,C_i$] are obtained.   end for   Generate clustering centers and weights. end for

3. Experiment with More Detailed Information

To verify the feasibility and efficiency of our proposed system, we designed fall prediction experiments that were conducted using the MCLLRTR. A total of 54 young, healthy student volunteers (including 36 males and 18 females; 22 ± 4 years old; weight: 64 ± 20 kg) performed simulated falls and abnormal gaits as well as some other training activities (the numbers of falls, abnormal gaits, and normal gait events are presented in

Table 1. Statistical information on simulated behavior of rehabilitation training performed by each participant.

Type	Activity	Number of Activity
Fall	(c1) Vertical fall (soft fall)	5
	(c2) Fall to left	5
	(c3) Fall to right	5
	(c4) Fall frontward	5
	(c5) Fall backward	5
(b) Abnormal gait	Rond de jembe a terre at left	5
	Rond de jembe a terre at right	5
	Dragging gait	5
	Forward gait	5
(a) Normal gait	Normal gait	5

). All volunteers provided informed consent before the simulated experiments, and were paid for their participation. Exclusion criteria included severe hypertension (BP >180/100 mmHg) and severe cardiopulmonary disease. Each participant was informed about how to operate the rehabilitation robot, and each participant completed the entire experiment. They performed various fall-type actions and rehabilitation tasks using the protective robot.

Figure 6 shows the simulation of different activities in the training process, such as (a) normal gait, (b1–b2) abnormal gait, and (c1–c5) falls. Accelerations along the x, y, and z axes are denoted as ${\mathrm{A}}_{\mathrm{x}}$, ${\mathrm{A}}_{\mathrm{y}}$, and ${\mathrm{A}}_{\mathrm{z}}$, respectively, and their norm (the resultant acceleration) is $A_{svm}=\sqrt{A_X^2+A_y^2+A_z^2}$. The tension signal is stored on the control board using an A/D converter with a sampling interval of Δt = 20 ms. The photodiode signals consist of digital signals, either a “0” or “1”, which reflect the foot landing point distribution.

After the experiment, the data must be processed. First, the two-stage activity recognition classifier mentioned above needs to be trained; then, the trained classifier can be used to classify the rehabilitation training activities. During neural network classifier training, the output needs to be labeled. After training, the classification threshold is determined.

When setting the threshold, the values of the inertia sensor and tension sensor were transformed by SVM kernel functions (Asvm and Tsvm) and drawn by MATLAB (as shown in Figure 6). When setting the thresholds of the first-level PNN network for training to reduce the number of calculations, we used only accelerometer and tension sensor data for fall prediction to simplify the rules for fall prediction. These rules can be expressed as follows: if |A_svm | exceeds a threshold b₁ and T_svm exceeds a threshold b₂, then, the event is determined to be a 1 (a fall event); otherwise, the event is determined to be a 0 (a nonfall event). The trigger thresholds for a falling event are shown in Figure 6. To achieve high accuracy, the threshold value determined by a single sensor must be 28.6% higher than that determined by multisensor fusion. Moreover, the prediction time by multisensor fusion is approximately 205 milliseconds faster than that of a single sensor. As Figure 6 shows, setting a single threshold value for the acceleration and tension features fails. The multisensor system incorporates multiple thresholds to avoid the effects of single-sensor noise. By integrating multiple single weak classifiers into a robust classifier, the multisensor fusion approach also achieves a high detection rate, while reducing the false alarm rate. From Figure 6, we can see that a single sensor’s data indicate only some activity features, and are sensitive in only some aspects.

Table 2. Comparison between the proposed algorithm and state-of-the-art fall prevention systems. PNN: probabilistic neural network.

	Sensor	Sensor Location	Prediction Method	Computational Complexity	Sensitivity
Fallarm [36]	Inertial sensor	Wrist	Traffic light-like alert system	Low	NA
iCane [37]	Force sensor + laser rangefinders	Robot	Center of gravity of the user in conjunction with sensors	Medium	NA
SmartPrediction [38]	Accelerometer and pressure sensor	Pocket and shoe	Decision tree of triggering an alarm	Medium	97.2%
Our work	Accelerometer and tension sensor	Chest and robot	PNN	Low	99.37%

compares our proposed method with state-of-the-art fall prediction algorithms. The proposed algorithm uses an accelerometer attached to the chest and waist belt force sensors in conjunction with the rehabilitation robot for data acquisition. The prediction algorithm identifies a fall or other gait abnormalities based on the off-line threshold value set by the PNN neural network. The prediction time when using the multisensor fusion approach is approximately 205 milliseconds ahead of approaches using a single sensor. The sensitivity is 99.37%, and the threshold is lower than that of single-sensor approaches.

The curves for the accelerometer data show that each type of activity has significant distinctions. The tension data curves have abrupt changes, and the peak sequence has an implicit law. After a long period of experimental study, we found that the most significant peak value of the tension sensor in one direction indicates that the fall is occurring in the opposite direction.

Detailed and correct functional assessment is the basis of formulating a therapeutic schedule [4]; therefore, further detailed identification concerning falls and abnormal gaits is required. The second SVM-RBF-KNN classifier is intended to classify specific types of falls or abnormal gaits in a supervised manner. The data extracted by the first-stage classifier are transmitted to the second-stage classifier. Let X_i = [A_svm, T_svm, d (T₁, T₄), d (T₂, T₃), d (T₃, T₁), d (T₄, T₂), P₁, P₃, P₄, P₆] denote the input data, and C_i denotes the corresponding cluster center. We set seven types for the final output; “c1” denotes a normal gait, “c2” denotes an abnormal gait, “c3” denotes a vertical fall, “c4” denotes a fall to the left, “c5” denotes a fall to the right, “c6” denotes a backward fall, and “c7” denotes a frontward fall. First, we adopted a gradient descent algorithm for the obtained data cluster center, which starts with few group data. Then, other samples are treated as falls or abnormal patterns, and RBF-BP is trained on each of them. This step is followed by an iterative procedure that involves merging a similar cluster (reclassifying) and retraining the remaining RBF-BP until no more merging occurs. Finally, the RBF is applied to train the neural network, and some randomly selected samples are used to test the performance of the neural network. This study was conducted with the MATLAB-2016a 64-bit version installed on Windows 7 Ultimate 64-bit with an Intel Core i7 (8 M cache, up to 2.7 GHz) and 8 GB of RAM. The final clustering centers are as follows:

$$\left[\begin{array}{c}{\mathrm{C}}_1\\ {\mathrm{C}}_2\\ {\mathrm{C}}_3\\ {\mathrm{C}}_4\\ {\mathrm{C}}_5\\ {\mathrm{C}}_6\\ {\mathrm{C}}_7\end{array}\right]=\left[\begin{array}{cccccccccc}948.47& 366.98& -19.91& -1.19& -5.25& -23.98& 0.25& 0.99& 0.99& 0.77\\ 952.35& 381.16& -24.20& -2.49& -6.65& -28.35& 0.21& 0.99& 0.99& 0.78\\ 952.14& 467.32& -35.67& -8.24& -7.39& -34.82& 0.24& 0.96& 0.92& 0.63\\ 953.27& 428.85& -27.19& -13.84& -18.74& -32.10& 0.23& 0.93& 0.92& 0.75\\ 951.72& 447.61& -32.50& -0.27& 3.56& -28.65& 0.24& 0.98& 0.95& 0.67\\ 953.45& 484.96& -43.75& -8.26& -5.05& -40.53& 0.27& 0.98& 0.87& 0.74\\ 950.32& 413.71& -29.59& 0.44& -10.45& -40.49& 0.22& 0.99& 0.99& 0.71\end{array}\right]$$

$$\mathsf{\omega }=\left[\begin{array}{c}-1.06\mathrm{e}+38\\ -7.68\mathrm{e}+45\\ -3.35\mathrm{e}+48\\ -2.30\mathrm{e}+120\\ -5.75\mathrm{e}+122\\ -7.02\mathrm{e}+122\\ -3.11\mathrm{e}+108\\ 2.70\end{array}\right]$$

Through data training, we obtained weight ω, which could also be calculated by bringing the data into Equations (17) and (18).

The primary purpose of the second-level classifier is to carry out detailed category identification. We selected several classifiers commonly used in MATLAB and compared them with the classifiers we built, using accuracy and time consumption as evaluation indicators. All the data collected by our homemade rehabilitation robot platform were compared in terms of accuracy and time consumption.

Table 3. Comparison between the proposed algorithm and state-of-the-art fall detection systems. KNN: k-nearest neighbors, RBF: radial basis function, SVM: support vector machine.

	Bilgin [39]	SAFE [40]	Baek [41]	Our Work
Sensor type	Tri-accelerometer	Tri-accelerometer and Gyro	Tri-accelerometer and Gyro	kinematic, tension, and photoelectric
Kinematic sensor location	Waist	Chest and thigh	Neck	Chest and robot
Fall type	×	○	×	○
Recognition expect fall	×	×	×	○
Learning	KNN	Decision tree	Threshold-based	SVM-RBF-KNN
Specificity	NA	NA	100%	100%
Sensitivity	89.4%	99.45%	80%	99.37%

compares the proposed fall detection method with state-of-the-art methods. The proposed method is the most accurate, and it can identify other gait features to enhance functional assessments. However, the proposed activity recognition method took approximately 41.79 seconds. In our study, after quickly predicting falls and instituting protection, the robot needs to perform more detailed activity recognition; therefore, the accuracy of its performance is more critical than its time consumption.

The data from the three sensors are supplied to three weak classifiers, which are merged into a robust classifier. The experiment shows that the tension is most sensitive in the gait direction, with a high weight value, the photoelectric sensors are sensitive to abnormal gaits, and the acceleration data are most sensitive to sudden abnormalities. Since different people have different kinematic features and different weights, the cluster weights ω and, consequently, the classifier weights are also different for different people; therefore, cluster weight assessment is of critical importance to ensure correct weight values for each person. In general, for weight loss gait training, the higher the weight loss ratio is, the greater the weight of the tension sensor (as shown in ω).

4. Discussion

In rehabilitation training, activity recognition is essential for the implementation and control of a rehabilitation robot and in the functional assessments of patients. However, there is currently no standard and detailed classification identification method for rehabilitation activity recognition. In addition, to prevent injuries due to falls, timely and accurate fall prediction is of the utmost importance. Unfortunately, very few studies have been conducted in this area [42,43]. Achieving timely prediction and making good trade-offs between sensitivity and specificity are also crucial for activity recognition, and thus are worth studying. To improve the accuracy of activity recognition and fall prediction during rehabilitation, we proposed a multisensor fusion system by combining heterogeneous sensor data for reasoning and analysis to solve complex activity recognition problems. We employed an asynchronous deviation obtained via tension sensors and individual conversion to estimate the motor capabilities of patients and adopted the deviation as a modulating factor to compute a corresponding assistance torque with ultimate adjustable torque bounds. Then, an assistance strategy can be generated. We also introduced a two-stage fall detection method in which a probabilistic neural network (PNN) is trained on routine activities. Subsequently, it identifies potential falls and passes them to an SVM-RBF-KNN algorithm, which predicts specific fall types or abnormal gaits in a supervised manner. The first stage allows fall prediction to occur as early as possible, while the second stage determines the fall details or the type of abnormal gait. These detailed classifications are stored and later provided to a therapist for functional assessment and the subsequent formulation of the therapeutic schedule.

During rehabilitation training, falls are most commonly caused by intrinsic factors such as muscle weakness, impaired balance, and declining cognition. These intrinsic falls usually do not show dramatic changes in acceleration values, and are therefore difficult to identify with general kinematic sensors. To solve this problem, we implemented a unique design in the multisensor system, which utilizes force dimensional information to avoid missed alarms caused by the lack of dramatic changes in kinematics values. This dimensional force information comes from the bodyweight support sling, which can reflect the potential falls caused by intrinsic factors in a timely manner. Previous studies [19,21,22,26,27] have also employed multicriteria algorithms to detect falls, but these algorithms perform fall identification in a parallel manner, and involve only explicit kinematics and posture information. In contrast, our multisensor system processes not only explicit information such as kinematics and spatial distribution information, but also implicit information such as kinetics and pulse information. Both the explicit and implicit information are analyzed in one discriminant function to obtain a detailed and accurate recognition result.

The results of the experiments indicated that it is possible to set lower thresholds that perform accurate classification by fusing information from the two signal sources. The proposed first-stage classifier can provide timely and accurate fall detection. The second stage SVM-RBF-KNN algorithm performed favorably in terms of activity recognition, and achieved higher accuracy and fewer false warnings than did the other classifiers. The results also revealed that separate classification problems could be solved by providing a different set of features from multisensor data. We also found that the value obtained from an accelerometer on a single axis has large amounts of randomness, and needs to be transformed into a suitable feature space by SVM algorithms. The multisensor system incorporates multiple thresholds to avoid the effects of noise in a single sensor, and it integrates multiple single weak classifiers into a robust classifier. This multisensor fusion method also achieves a high detection rate while reducing the false alarm rate. The proposed multisensor activity recognition system can effectively meet a physician’s needs for rehabilitation training. The sampling rate, number, and locations of sensors, and the employed algorithm and its corresponding ranges for such variables affected the detection results. The tension sensor data are more sensitive than the accelerometer data in the second stage of classification. The experiments demonstrate that 205 milliseconds can be saved compared to models that use a single sensor. The accuracy of the fusion method is almost 99.37%, and no false warnings occurred. These results verify the feasibility and efficiency of multisensor fuzzy data fusion algorithms and the calculation process. We also found that the tension signal is most reliable for identification, and the tension signal peak sequence reflects the fall direction (the largest peak value of the tension sensor in one direction indicates that the fall is occurring in the opposite direction).

The proposed recognition system coupled with a new type of mobile chaperone lower limb rehabilitation training robot exhibits excellent control performance, good interaction performance, and accurate monitoring, which promotes the enthusiasm and confidence of patients during rehabilitation training.

However, this new type of mobile chaperone lower limb rehabilitation training robot cannot improve recovery efficiency, because that efficiency is related to the pathological physical condition and medication of the patients; instead, our rehabilitation robots improve the efficiency of physicians by providing them with reliable data and allowing them to adjust the training strategies and training difficulty levels. Currently, neural network classifiers require more data for further training. Additionally, the experiments were performed by young students who were not hemiplegic patients. In future work, we plan to conduct experiments using hemiplegic patients from the China Rehabilitation Research Center (CRRC).

5. Patents

An Intelligent Safety Protection Robot for Rehabilitation Training of Human Lower Limbs (201611102656.X).