Using Wearable Sensors to Estimate Vertical Ground Reaction Force Based on a Transformer

Abstract

In this paper, we present a new method to estimate ground reaction forces (GRF) from wearable sensors for a variety of real-world situations. We address the drawbacks of using force plates with limited activity range and high cost in previous work. We use a transformer encoder as a feature extractor to extract temporal and spatial features from wearable sensors more efficiently. Using the Mean Absolute Percentage Error (MAPE) as the evaluation criterion, the experimental results show that the average error of the predicted values using the transformer as a feature extractor improved by 32% compared to the RNN architecture and by 25% compared to the LSTM architecture. Finally, we use Gate_MSE to solve the problem of a large peak error in GRF prediction. Meanwhile, this paper explores the effect of the number of wearable sensors or wearable modes on GRF prediction.

1. Introduction

Kinetic data on human movements, such as ground reaction forces (GRF) and joint movement have become indicators of injury risk and pain during exercise. GRF are often used to assess gait dynamics and balance measures, with the vertical ground reaction force (vGRF) being the most significant component of GRF during walking. The curve of vGRF consists of two peaks: a passive (weight acceptance) peak and an active (push-off) peak. The passive peak (PP) results from the foot colliding with the ground, while the active peak (AP) is generated by the foot’s primary force applied to the ground as it pushes off. The magnitude and timing of these peaks can affect the joints and muscles of the lower limb. They can lead to the development or progression of musculoskeletal overuse injuries and conditions such as osteoarthritis [1,2].

For healthy people, the vGRF at the bottom of the foot remains almost constant. However, for patients with an asymmetrical gait, the vGRF at the bottom of the foot changes as the right and left feet alternate. In previous work, motion analysis was often used and was achieved in the laboratory using a precision motion tracker and treadmill force plates that could be embedded in an instrumented treadmill or ground walkway. The disadvantages of this measurement method included the cost of equipment, space requirements, and the limitations of testing in a laboratory setting rather than a natural environment [3].

Recent work has used wearable sensors for gait analysis to address this limitation [4,5,6,7]. Alternatively, inertial measurement units (IMUs; wireless wearable devices that measure magnetism, linear acceleration, and angular velocity) have been used to measure athletes’ leg joint angles, stride kinematics, and segmental accelerations during competitive events. Although IMUs cannot directly measure GRFs, previous studies have used algorithms to estimate discrete biomechanical variables like peak vertical GRF, ground contact time, vertical impulse, and vertical loading rate from IMU data. With the wider adoption of machine learning approaches to the monitoring of human movement using wearable sensors, it is possible to estimate ground reaction forces from IMUs using several different machine learning algorithms. Jiang et al. [8] wore four IMUs on the foot, lower leg, distal thigh, and proximal thigh and moved at different speeds on a treadmill equipped with force plates. A random forest model was used to estimate vertical ground reaction forces from the data collected from each IMU. Wouda et al. [9] used two artificial neural networks to predict ground reaction forces. The first artificial neural network mapped data (rotation and acceleration) from three inertial sensors (located at the left and right calf and pelvis) to lower limb joint angles. The estimated joint angles were combined with the measured vertical accelerations and input to a second artificial neural network, which estimated the vertical ground reaction forces. Refai et al. [10] used an IMU mounted on the pelvis to estimate the GRF during everyday gait, a calibration procedure, and an error-state extended Kalman filter (EEKF) for converting the acceleration at the center of mass (CoM) into a 3D GRF. When GRF are used in conjunction with the calculated kinematics, the GRF can calculate inverse dynamics and estimate the internal forces and interactions between different joints, muscles, and bones in the subject [11,12]. In the field of biomechanics, LSTM networks have been used to make frame-by-frame predictions of GRF waveforms using motion capture data and predictions of the center of mass position relative to the center of pressure from IMU data during walking [13,14,15]. The method used in this paper differs from previous work in two main ways. Firstly, the model of the neural network is different. Previous works have used RNN, LSTM [16,17], or other neural network models. Transformer is a deep learning model with an encoder-decoder architecture that has been widely used in natural language processing and computer vision. It relies on a self-attentive mechanism to capture long-range dependencies, and the multi-head mechanism allows it to compute in parallel and reduce running time, where different heads will learn different levels of knowledge, and the use of residual networks can solve the problem of gradient disappearance and degradation [18]. In this paper, the transformer is used to capture spatial and temporal features between inertial sensor data.

Second, the treadmill force plate limits the spatial and temporal range of the activity. It cannot be measured outside the laboratory environment, such as measuring the GRF when going up and down stairs, making it impossible to apply in practical situations. Therefore, this paper uses five inertial sensors worn on the pelvis, left thigh, left ankle, right thigh, and right ankle to collect human motion information. A pair of pressure insoles with eight force-sensitive resistors (FSR) are used as a tool to collect vGRF. We used the transformer as a feature extractor to estimate the vGRF using human motion information.

2. Dataset

In this section, datasets for estimating vGRF from inertial data are presented. In this domain, there is no large publicly available dataset containing raw inertial GRF data. Therefore, we will construct our own dataset as training data for the model. Motion data from the human lower limbs is captured via Notion’s Perception Legacy full-body motion capture system, and pressure insoles are used to collect vGRF. Figure 1 left shows the compression insoles used in this paper, each of which consists of eight FSRs mounted at different locations to capture the pressure of the foot. Although pressure insoles are prone to deflection, causing measurement errors. Figure 1 right shows where the IMU should be worn.

The IMU and the pressure insole collect data at a frequency of 100 Hz and are synchronized via the ESP-NOW data transfer protocol to ensure that our data is reliable.

Ten volunteers were recruited to participate in a data collection experiment that included fast running, jogging, slow walking, brisk walking, and walking up and down stairs. Each movement was captured twice for five minutes. Subjects were given a three-minute adaptation period between the completion of the instrument set-up and the start of the measurements, to feel comfortable in the wearable device. Throughout the experiment, subjects remained barefoot, wore no footwear, and were not allowed to use handrails or touch any other external objects to prevent additional external forces from being generated. Precautions were taken prior to data collection, and the experiment was not harmful in any way.

2.1. Data Acquisition System

We use a data acquisition system to synchronize inertial motion data of the lower extremity with vGRF at eight locations on the sole of the foot. Figure 2 illustrates our data acquisition system, incorporating data calibration, synchronized acquisition, and data visualization. The right side of the figure shows the real-time GRF visualization of the left and right foot. The red circle at the bottom left represents the contact between the sole and FSR, and the black color represents the sole not in contact with FSR. The light blue color represents the real-time pressure center point, which can be observed in real time during the acquisition process to avoid insole detachment.

2.2. Collection Process

First, click on “Open pressure insoles” to start transferring raw data, then click on “Pressure cushion calibration” to convert ADC values into pressure data, which will be described in detail in the next section. “Calibrations 1” and “Calibrations 2” are the IMU calibration processes, as each sensor has its own coordinate system. First, the raw inertia measurements are converted to the same reference frame, called calibration. Then the leaf joint inertia is converted to root space and rescaled to fit the network input, called normalization. The sensors can be arbitrarily rotated during setup, and our method automatically calculates the matrix of each sensor before converting it to capture motion. This process requires the subject to hold the t-position for a few seconds [19]. Finally, by clicking “IMU data collection”, the system will collect both GRF data and inertial motion data simultaneously.

2.3. Pressure Sensor Calibration

Since our pressure insoles based on FSR sensors cannot accurately measure the pressure applied, we defined the pressure data of each pressure sensor on foot in the t-pose state as the base data. Here we assume that the FSR sensors on each pressure insole can be fully pressed by the foot when the subject does the t-pose and applies the average body weight to each foot. Furthermore, in order to obtain stable pressure data, we ask the subject to do t-pose and stand in place for 3 s to obtain N frames of data, and we obtain the average of these 3 s of pressure data as the baseline data. In this way, we obtain the baseline pressure of 16 sensors on a pair of shoes.

$$F_s^i=\frac{1}{N}{\displaystyle {\sum }_{j=0}^N}{F}^i, i=1, 2,\dots ,15,16$$

(1)

the data collected by the FSR sensor are ADC values, not pressure data that can be used directly. However, there is a linear relationship between body weight and ADC values, so we used body weight to convert the ADC data to pressure data. The vGRF for each foot can be calculated as the sum of the data collected by the eight FSR sensors and the weight of the subject is defined as $W_s$, from which the scale factor $S_f^i$ and the pressure $F^i$ applied by each FSR sensor corresponds to the weight.

$$S_f^i=\frac{W_s}{2\times {{\displaystyle \sum }}_{i=1}^{N=16}{F}_s^i}$$

(2)

$$F^i=F_s^i\times S_f^i$$

(3)

Finally the pressure data converted to weight is recorded for post-processing analysis.

3. Methods

The transformer is a neural network architecture using a self-attentive mechanism that allows the model to focus on the relevant parts of the time series to improve prediction quality. The self-attentive mechanism can interconnect all time series steps simultaneously to produce long-term dependency understanding. Transformer architectures that combine the self-attentive mechanism, parallelization, and positional coding often have advantages over LSTM and CNN models. Our proposed model is shown in Figure 3 and contains a temporal embedding layer, a transformer’s encoder layer, and a linear mapping layer.

3.1. Time Feature

When dealing with time series data, time is an essential feature. However, when using the transformer for time series data, the sequences are forwarded all at once through the transformer architecture, thus making it difficult to extract the time sequence dependencies. To overcome the time discrepancy of the transformer, this paper uses the approach proposed in [20], where the vector representation can be used as a standard embedding layer that can be added to the neural network architecture to improve the model’s performance. The authors determined that a meaningful temporal representation must include periodic and non-periodic patterns, and secondly, the temporal representation should be invariant to temporal rescaling. The following mathematical definition can suggest this.

$$\mathrm{t}2\mathrm{v}\left(\tau \right)\left[i\right]=\left\{\begin{array}{c}{\omega }_i\tau +{\psi }_i, if i=0\\ F\left({\omega }_i\tau +{\psi }_i\right), if 1\le i\le k\end{array}\right.$$

(4)

3.2. Loss Function

Using mean squared error (MSE) as a loss function is a standard approach to predicting GRF tasks. MSE loss tends to produce a smoothed or averaged approximation of ground truth. However, in GRF prediction, the peak instantaneous impact of the GRF signal is more important than the average accuracy of the entire sequence. The peak information represents the maximum GRF during motion, which is often associated with knee injury or injury risk. Therefore, the same loss function Gate-MSE as in [21] is used in this paper, which prioritizes the regions with a high impact on the prediction results during the training period and significantly improves the accuracy of the peak estimation for ground reaction forces. The Gate-MSE is described as follows and can be viewed as a linear combination of weighted MSE under the T threshold.

$$L_f={{\displaystyle \sum }}_T{w}_t·\frac{1}{\left|F_{{\delta }_t}\right|}||F_{{\delta }_t},-,{\widehat{F}}_{{\delta }_t}||\begin{array}{c}2\\ 2\end{array}$$

(5)

We define $F_{{\delta }_t}=\{F_1,F_2,F_3\dots F_f;\forall |F_f|<{\delta }_t\}$, which serves as an indexed array for all elements in the ground truth signal F, with an absolute value below the threshold ${\delta }_n$. In practice, T serves as a hyper-parameter where each threshold belongs to the sequence $\delta$ = [0, 1, 5, 10, 15, …] in terms of N/kg. The summed loss is equivalent to the MSE when T = 1, and we weigh each contribution with $w_t$ = 1/T, weighted by the number of thresholds.

3.3. Data Format

We compared the impact of three different data formats on the predictions. In the first, we used raw input data from five inertial sensors (containing acceleration data and rotation data). Next, we consider the two features of acceleration and joint position. When the GRF reaches its peak, with the ankle at its lowest position, the acceleration of the foot should be 0. When the ground reaction force is 0, it means that the foot has left the ground, and the acceleration of the foot should be 0 when the joint position is at its highest point. We use inverse kinematics from joint rotation to calculate the joint position.

In order to predict the result, using Principal Component Analysis (PCA) reduces the dimensionality of features in the model. It does this by constructing the so-called principal components (PCs) from multiple features. The PC is constructed so that the PC1 direction explains as many features as possible for maximum variation. PC2 then explains the remaining features as much as possible for the maximum variation, and the other PCs do the same. Generally speaking, PC1 and PC2 explain a significant fraction of the overall feature variation.

4. Results

We show the results on our dataset, using MSE and MAPE as the evaluation metrics of the model. Multiple sets of comparison experiments are conducted to highlight the effectiveness of the method proposed in this paper.

In

Table 1. Effect of model and loss on the results.

Model	MSE	MAPE
RNN	0.0365	84.4239
LSTM	0.0293	82.7453
Transformer	0.0218	79.2751
Transformer + Gate_MSE	0.0227	80.0136

, we compare the effectiveness of the classical neural network models RNN, LSTM, and the transformer used in this paper for predicting GRF and the effect of Gate_MSE on the final experimental results. The use of Gate_MSE does not improve the overall effect, and its effect is slightly worse than that of the MSE loss function. The primary function of Gate_MSE is to predict the peak of the GRF, and we will demonstrate the improvement effect of Gate_MSE for predicting the peak of the GRF in the following section.

During the movement, the foot is the first touch until entirely off the ground, and the GRF will appear in two peaks. The moment the heel first makes contact with the ground, the GRF quickly reaches a higher value; this process occurs quickly and steeply, indicating that the impact force occurs quickly, and the first peak force loading rate is considered to be highly correlated with the occurrence of injury. Then it rises more slowly to the second peak. To demonstrate the effectiveness of the transformer, we show in Figure 4 the prediction results of the transformer and the LSTM, which also uses MSE as the loss function.

Table 1 illustrates that the overall effectiveness of using Gate_MSE is not as good as MSE. However, by comparing the curves of the ground truth with the predicted results in Figure 5, we can see that Gate_MSE is more effective in predicting the peak ground reaction force, which is often used to assess injury risk, muscle loss, etc., so that effective peak prediction is more important.

In this paper, we use the transformer’s encoders as feature extractors. More encoders tend to mean deeper network layers and extract more valuable features, so in

Table 2. Effect of the number of encoders on the results.

Number of Encoders	MSE	MAPE
1	0.0325	90.5268
2	0.0237	81.5452
3	0.0218	79.2751
4	0.0216	78.4532

, we show the effect of stacking different numbers of encoders on the prediction effect of the model. In Table 2, we can see that using two encoders improves the prediction effect by 18% compared to using one encoder. When using three encoders, the effect improves by only 7%, and when the number of encoders is increased to four, the effect improves by only 5%. With the addition of encoders, there is a greater amount of computation and a longer computation time, which cannot be used in practice.

In

Table 3. Effect of the data format on results.

Data format	MSE	MAPE
Acceleration + Rotation	0.0325	90.5268
Acceleration + Joint position	0.0237	81.5452
Acceleration + Joint position + PCA	0.0218	79.2751

, we verified the model’s prediction effect for different data formats. We used acceleration and rotation, acceleration and joint position, and acceleration and joint position with dimensionality reduction by PCA as the input data for the model. We can see from Table 3 that the best prediction results are obtained by using acceleration and joint position with dimensionality reduction by PCA, which we believe is because joint position contains more human information than rotation data, while using PCA not only makes the model input data less but also more generalizable.

Since too many inertial sensors are cumbersome to wear, we compare the effects of different wearing methods on the prediction results in

Table 4. Effect of the wearing method on results.

Wearing Method	MSE	MAPE
Ankle	0.0294	83.1910
Ankle + Pelvis	0.0218	79.2751
Ankle + Pelvis + Thigh	0.0205	78.7815

to see how they apply to different usage scenarios. Through the experimental results, we can see that the MSE is 0.0294 when using only the two inertial sensors worn at the ankle. When we add the inertial sensor data at the pelvis, the effect is improved by 25%. However, when we add the inertial data that is at the thigh, the effect is only improved by 5%. However, the final effect will be improved with the increased number of nodes, but the wearing of too many nodes is not convenient and invasive, so you can judge which sensors to use according to the usage scenario.

5. Discussions

Using the transformer as a feature extractor requires more training data to make the model avoid overfitting or underfitting problems, and the transformer has twice the number of parameters as the RNN architecture, so it requires longer training time. To address the limited range of motion caused by the use of force plates in previous work, in this paper, vGRF is measured by pressure insoles. Kinetic data measured by pressure insoles only includes vertical components, and the thickness and size of insoles are also limited to some extent, and their structure and material properties also have significant effects on the force patterns of the foot with different structures. All our testers were healthy and disease-free and between the ages of 18 and 23; however, due to factors such as gender, age, and physical fitness, the distribution of input parameters is not consistent across different populations and cannot be used in this category. In future research, the diversity of the training set samples should be increased, and different populations and different movement patterns should be included in the learning model.

6. Conclusions

In this work, we use a transformer as a feature extractor to address the problem of predicting GFR using wearable sensors. First, we introduce the dataset used in this paper, which contains lower extremity motion data and vGRF in multiple motion situations, solving the problems of high cost and limited range of motion associated with the use of ergometers in the past and can be applied to outdoor situations. Then we introduce the advantages of using transformer as a feature extractor, which performs better than the traditional neural network models RNN and LSTM. Finally, we refer to Gate_MSE to solve the problem of large peak errors in the GRF prediction task. Meanwhile, this paper explores the effects of the number of transformer encoders, the data format, and the wearing mode or number of inertial sensors worn on GRF prediction, which can be replaced flexibly in practice.