GaitAE: A Cognitive Model-Based Autoencoding Technique for Gait Recognition

Abstract

Gait recognition is a long-distance biometric technique with significant potential for applications in crime prevention, forensic identification, and criminal investigations. Existing gait recognition methods typically introduce specific feature refinement modules on designated models, leading to increased parameter volume and computational complexity while lacking flexibility. In response to this challenge, we propose a novel framework called GaitAE. GaitAE efficiently learns gait representations from large datasets and reconstructs gait sequences through an autoencoder mechanism, thereby enhancing recognition accuracy and robustness. In addition, we introduce a horizontal occlusion restriction (HOR) strategy, which introduces horizontal blocks to the original input sequences at random positions during training to minimize the impact of confounding factors on recognition performance. The experimental results demonstrate that our method achieves high accuracy and is effective when applied to existing gait recognition techniques.

1. Introduction

Gait recognition, a long-distance technique, holds significant application potential in crime prevention, forensic identification, and criminal investigation [1]. The problem is substantial due to the complexity and diversity of gait patterns in real-world scenarios, where even slight variations can lead to significant challenges in accurate identification. While existing methods perform well on controlled experimental data, their applicability in challenging real-world scenarios is limited. In real-world environments, gait sequences are often non-ideal and fragmented, as depicted in Figure 1. These low-quality sequences can severely hinder efficient gait recognition, emphasizing the need for models to extract and highlight valuable information while mitigating the impact of confounding factors. When confronted with low-quality input sequences, deep neural networks often exhibit instability [2]. Addressing the impact of such sequences on gait recognition remains an open question, as most prevalent models primarily focus on minimizing training errors during sample encoding [3,4,5,6]. Current methods [5,7,8] have made progress by emphasizing relevant information in input sequences. For instance, Wei et al. [5] proposed a multi-scale gait recognition network (GMSN), which integrates a multi-scale parallel convolution network and a local-based horizontal mapping module to prioritize critical gait regions. Hou et al. [7] introduced a gait quality-aware network (GQAN), which explicitly evaluates contour and part quality through quality blocks. Wang et al. [8] proposed cost-effective quality assessment strategies that maximize connected regions and use template matching to remove background noise and unidentifiable outlines, along with alignment techniques for non-standard poses. In addition, relation-based methods engage in spatio-temporal modeling to emphasize biologically plausible data [3]. These methods have made certain progress, but they focus solely on optimizing the final output of gait feature descriptors without considering gait reconstruction, making them vulnerable to noise interference in real-world scenarios. In this paper, we address these challenges by introducing the GaitAE method, which employs an autoencoder (AE) to enhance the reconstruction abilities of gait feature descriptors, significantly improving model performance without increasing computational complexity. Furthermore, we propose the horizontal occlusion restriction (HOR) strategy to minimize the impact from unexpected covariates. These innovations ensure that our approach remains efficient and practical for real-world applications, as they are only introduced during training and do not add extra parameters or computational complexity during the testing phase. Experimental validation on benchmark datasets showcases the effectiveness and robustness of our method. The significance of this work lies in its ability to improve gait recognition accuracy under challenging conditions while maintaining computational efficiency, making it highly applicable for large-scale deployment. Our key contributions are as follows:

(1)

Designing an AE mechanism capable of adapting gait feature descriptors to different quality sequences.

(2)

Introducing the HOR strategy to enhance model robustness by minimizing the impact of confounding factors.

(3)

Extensive experiments on benchmark datasets, including CASIA-B [9], OU-MVLP [10], and SUSTech1K [11], validate the effectiveness and robustness of our proposed method.

(4)

Our method demonstrates flexibility and effectiveness when integrated into existing gait recognition methods.

In the following sections, we present the related work in Section 2, describe the proposed GaitAE method in Section 3, detail the datasets and experiments in Section 4, and conclude the paper in Section 5.

2. Related Work

2.1. Video-Based Methods

Current gait recognition methods commonly rely on video-based approaches. Techniques like LSTMs [12,13] and time aggregation [14,15] are used to maintain temporal gait information. For instance, Zhang et al. [12] separated pose and appearance features from RGB gait sequences using an LSTM-based network, while Zhao et al. [13] integrated multi-view features through memory and capsule modules. However, these methods can be computationally inefficient due to unnecessary constraints for temporal sequences [14]. To address this, utilizing gait video sequences directly as inputs and mapping features into horizontal strips for spatio-temporal preservation [14,15] has gained popularity. Approaches diverged into 2D and 3D convolution directions. The former one focuses on modeling spatio-temporal features by treating the human body as parts [3,4,5,15,16,17]. For example, Fan et al. [15] emphasized local short-range spatio-temporal features as discriminative for periodic gaits. On the other hand, 3D convolution extracts gait features from global to local perspectives [18,19,20]. Wei et al. [5] introduced a multi-scale gait recognition network (GMSN) that enhances the discriminative representation of gait by combining parallel convolutional networks with a locally based horizontal mapping module. Furthermore, some multimodal gait recognition methods integrate skeletons and silhouettes to capture rich spatio-temporal features [21,22]. For instance, Peng et al. [22] proposed a bimodal fusion network (BiFusion) combining skeleton and silhouette representations effectively. Building on this, Hus et al. [23] introduced GaitTAKE, a method combining temporal attention-based global and local appearance features with time-aggregated human pose features. Despite advancements in accuracy, these methods do not explicitly address modeling low-quality or non-ideal sequences.

2.2. Gait Quality Restoration Methods

Most gait quality restoration methods derive inspiration from adaptive weighting approaches based on image quality in set-based person reidentification [24]. In gait recognition, Hou et al. [7] introduced a gait quality-aware network (GQAN) which explicitly evaluates the quality of each silhouette and each segment. Wang et al. [8] proposed a series of cost-effective quality assessment strategies, including maximizing connected regions and employing template matching to eliminate background noise and unidentifiable outlines, alongside alignment techniques to handle non-standard poses. Inspired by these methods, we introduce an AE capable of reconstructing both typical and low-quality sequences, enhancing the reconstruction capabilities of gait feature descriptors for both types of sequences.

3. GaitAE

3.1. Gait Autoencoder

For a given gait recognition problem, we follow a three-step method to handle video-based gait recognition tasks, as follows:

$$f\left(X^{\left(n\right)}\right)=\mathcal{H}\left(\mathcal{G}\left(h\left(X^{\left(n\right)}\right)\right)\right)$$

(1)

where $X^{\left(n\right)}$ represents the $n^{th}$ input sample, the h is a sequence encoder used to extract frame-level features, the $\mathcal{G}$ operation is used to aggregate video features from different frames, and $\mathcal{H}$ is used to learn a discriminative representation of the training data distribution from extracted features. We refer to the operations $\mathcal{G}$ and $\mathcal{H}$ as gait refinement.

Equation (1) outlines the standard steps in existing gait recognition methods. Unlike traditional approaches, we incorporate a decoder g within the encoder h to form an autoencoder (AE) framework, illustrated in Figure 2. This AE framework, with $n=1,\cdots ,N$, is designed to reconstruct input gait sequences, capturing essential details while filtering out noise. The reconstruction task is crucial for improving gait recognition accuracy and serves as a regularization mechanism to prevent overfitting. The model, by learning both reconstruction and recognition, develops shared representations that are generalizable and discriminative. The AE is typically trained by minimizing the mean squared error (MSE) between the input and output as follows:

$$R_{MSE}={\displaystyle \frac{1}{N}}\sum _{n=1}^N{(X^{\left(n\right)}-g\left(h\left(X^{\left(n\right)}\right)\right))}^2$$

(2)

The triplet loss is defined as:

$$L_{triplet}=\sum _{n=1}^N{\left[|f\left(X_{anchor}^{\left(n\right)}\right)-f\left(X_{pos}^{\left(n\right)}\right){|}^2-{|f\left(X_{anchor}^{\left(n\right)}\right)-f\left(X_{neg}^{\left(n\right)}\right)|}^2+\alpha \right]}_{+}$$

(3)

where $X_{anchor}^{\left(n\right)}$, $X_{pos}^{\left(n\right)}$, and $X_{neg}^{\left(n\right)}$ represent the anchor, positive, and negative samples for the n-th instance, respectively. The function $f(·)$ denotes the embedding function that maps inputs to the feature space, and $\alpha$ is a margin that enforces a separation between positive and negative pairs. The triplet loss ensures that the distance between the anchor and the positive sample is smaller than the distance between the anchor and the negative sample by at least the margin $\alpha$. The notation $[·]+$ indicates that only positive values are considered, ensuring that the loss is non-negative. By combining the MSE with the triplet loss, the total loss function becomes:

$$L_{total}=R_{MSE}+R_{triplet}$$

(4)

3.2. Horizontal Occlusion Restriction Strategy

The impact of external changes during walking, such as carrying a bag and wearing a coat, on recognition accuracy is one of the major factors that restrict the practical application of gait recognition. The recognition accuracy of previous studies [3,5,16] is relatively low in these unexpected conditions, possibly due to the lack of an information-sifting ability of these algorithms. We assume that adding occlusion to the area which is likely to be affected by the covariate (e.g., upper body and lower middle body) may allow the algorithm to pay more attention on the other area which is less likely to be affected by covariates (e.g., foot, leg, and head) and thus enhance the information-sifting ability. Existing occlusion strategies mostly use simple rectangular occlusions [1], which tend to separate the human body and result in a lack of correlation between body parts during movement. Unlike these existing strategies, the proposed HOR strategy employs multiple horizontally adjustable rectangular occlusions, which better simulate the complexity of real-world occlusions, thereby maintaining the correlation between body parts and preventing the loss of important features.

For the tth frame in gait sequences with the width and height as $H^{\prime }$ and $W^{\prime }$, respectively, the matrix of the corresponding horizontal occlusion restriction is:

$$M_t={\left[\begin{array}{cccc}0& 0& \cdots & 0\\ 0& 0& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & 0\end{array}\right]}_{H_1:H_2,W^{\prime }}$$

(5)

with the 0 in the matrix representing the brightness of the pixel being black. Generally, the HOR is the best when the widths of the occlusion are the same as the original sequences since the occlusion can completely eliminate the covariates in this condition. The height of the occlusion is adjusted by two parameters, $H_1$ and $H_2$ ($H_1<H_2$). Therefore, the occlusion is zero for the area of a height from $H_1$ to $H_2$, and a width of $W^{\prime }$, as is shown in Figure 3. When only using one occlusion area to eliminate the covariates, the continuous information loss in the height dimension would drop the connection between the remaining body parts. To retain the connection between body parts, we propose to use multiple horizontal occlusion restrictions, as is shown in Figure 3. This is achieved by generating a random number $j=1,2,\cdots ,J$ occlusion with the width as $W^{\prime }$, located from $H_3$ to $H_4$, so as to construct $M_t^j$, with j as the number of occlusions. The multiple horizontal occlusion restrictions can be represented as follows:

$$M=\left\{M_t^j{\in }^{H_3:H_4,W^{\prime }}\right|j=1,2,\cdots ,J;t=1,2,\cdots ,T;H_3,H_4\in H_1,H_2\}$$

(6)

where the four parameters, $H_1$, $H_2$, $H_3$, and $H_4$ are hyperparameters. $H_1$ and $H_2$ are generally set in the middle of the image for better removal of covariates.

4. Experiments

We evaluate the effectiveness of our proposed method through extensive assessments using the CASIA-B dataset [9], OU-MVLP dataset [10], and sustech1k dataset [11], which offer a diverse range of gait samples. In this section, we first introduce these datasets, following which we integrate our method into existing state-of-the-art gait recognition models for comparison with the original models. Furthermore, we conduct comprehensive ablation and parameter sensitivity experiments on the CASIA-B dataset to validate the efficacy of the proposed components. Results are presented in terms of average rank-1 accuracy, excluding identical views.

4.1. Datasets

4.1.1. CASIA-B

The CASIA-B dataset [9] is a comprehensive benchmark for gait analysis, featuring video sequences from 124 subjects captured at 25 frames per second. Each subject has 110 video sequences recorded from 11 different viewing angles (ranging from 0 to 180 in 18 increments) under three walking conditions: normal walking (NM), carrying a bag (BG), and wearing a coat (CL). The complexity of these walking conditions increases from NM to BG to CL.

For our experiments, we employ the large-sample training (LT) setup. In the LT configuration, the first 74 subjects (001–074) are used for training, while the remaining 50 subjects (075–124) are reserved for testing. During the testing phase, the first four sequences under the NM condition (NM #1–4) are used as the gallery set. The remaining six sequences are divided into three probe subsets: NM (NM #5–6), BG (BG #1–2), and CL (CL #1–2).

4.1.2. OU-MVLP

The OU-MVLP dataset [10] is the largest publicly available gait database, featuring video sequences from 10,307 subjects. Each subject has 28 video sequences captured from 14 different views (0, 15, …, 90; 180, 195, …, 270 ), with two samples (indexes #00 and #01) per view. According to the standard protocol, sequences from 5153 subjects are used for training, while the remaining 5154 subjects are reserved for testing. During the testing phase, sequences with index #01 are used as the gallery set, and those with index #00 are used as the probe set.

4.1.3. SUSTech1K

The SUSTech1K dataset [11] is a comprehensive and synchronized multimodal gait dataset that captures a wide range of variations, including normal walking, carrying a bag, changing clothes, different views, object carrying, occlusion, illumination changes, uniform wearing, and using an umbrella. The data are collected using a mobile robot equipped with a 128-beam LiDAR scanner and a monocular camera, allowing for synchronized multimodal data acquisition. The dataset consists of 1050 identities, 25,239 sequences, 763,416 point cloud frames, and 3,075,575 RGB images, each paired with corresponding silhouettes. This dataset serves as a valuable resource for research in the field of gait recognition.

4.2. Training and Testing

During preprocessing, gait silhouette sequences are rescaled to 64 × 44 pixels (height × width) using the technique proposed by Takemura et al. [10]. An Adam optimizer [25] is selected with a learning rate of $1\times {10}^{-4}$. The margin in the Batch All (BA+) triplet loss function [26] is set to 0.2. In the training stage, the batch size of the input training data is given by $\mathtt{P}\times \mathtt{K}=8\times 12$, where $\mathtt{P}$ is the number of subjects and $\mathtt{K}$ is the size of the training sample for each subject within the batch. The number of frames $\mathtt{f}=30$. For the decoder g, we design it with a structure similar to that of the encoder h, utilizing nearest-neighbor upsampling in the upsampling layers.

For CASIA-B and SUSTech1K, the model is trained for 160 k iterations. For OU-MVLP, the model is trained for 550 k iterations. The learning rate is reduced to $1\times {10}^{-5}$ at 450 k iterations. In the test stage, all video frames are selected and utilized as test data.

4.3. Effectiveness of GaitAE

We assess the effectiveness and generalizability of our proposed GaitAE by integrating it into existing models and evaluating its impact on their performance. We introduced GaitAE to other cutting-edge gait recognition models conducted LT on the CASIA-B gait dataset. The models are GaitSet [27], GaitPart [15], GaitGL [18], GaitSlice [3], and GMSN [5]. As seen in

Table 1. The average rank-1 accuracies for other models and their combination with GaitAE in CASIA-B, OU-MVLP, and SUSTech1K datasets.

Model	CASIA-B		OU-MVLP		SUSTech1K
	Origin	GaitAE	Origin	GaitAE	Origin	GaitAE
GaitSet [27]	84.2	86.4 ↑2.2	87.9	88.3 ↑0.4	48.4	49.0 ↑0.6
GaitPart [15]	88.8	90.3 ↑1.5	88.7	89.6 ↑0.9	47.6	48.1 ↑0.5
GaitGL [18]	91.8	92.6 ↑0.8	89.7	90.0 ↑0.3	47.3	47.6 ↑0.3
GaitSlice [3]	90.2	91.6 ↑1.4	89.3	89.7 ↑0.4	47.8	48.3 ↑0.5
GMSN [5]	93.7	94.1 ↑0.4	89.7	90.1 ↑0.4	50.0	50.3 ↑0.3

, when GaitAE is introduced, the performance of most existing models surpassed their original version. This shows the generalizability and effectiveness of our proposed GaitAE method. Furthermore, we utilize weights trained on the CASIA-B dataset to reconstruct gait sequence images, as depicted in Figure 4. This demonstrates that our method is capable of reconstructing both typical and low-quality sequences across the CASIA-B, OU-MVLP, and SUSTech1K datasets.

4.4. Ablation Study

As seen in

Table 2. The average rank-1 accuracies for other models and their combination with AE or HOR in CASIA-B, OU-MVLP, and SUSTech1K datasets.

Model	CASIA-B			OU-MVLP			SUSTech1K
	Origin	AE	HOR	Origin	AE	HOR	Origin	AE	HOR
GaitSet [27]	84.2	85.2 ↑1.0	85.7 ↑1.5	87.9	88.0 ↑0.1	88.2 ↑0.3	48.4	48.6 ↑0.2	48.9 ↑0.5
GaitPart [15]	88.8	89.4 ↑0.6	89.4 ↑0.6	88.7	89.1 ↑0.4	89.5 ↑0.8	47.6	47.9 ↑0.3	48.0 ↑0.4
GaitGL [18]	91.8	92.0 ↑0.2	92.2 ↑0.4	89.7	89.8 ↑0.1	89.9 ↑0.2	47.3	47.4 ↑0.1	47.5 ↑0.2
GaitSlice [3]	90.2	90.7 ↑0.5	91.2 ↑1.0	89.3	89.6 ↑0.3	89.6 ↑0.3	47.8	48.1 ↑0.3	48.2 ↑0.4
GMSN [5]	93.7	94.0 ↑0.3	93.9 ↑0.2	89.7	89.8 ↑0.1	89.0 ↑0.2	50.0	50.1 ↑0.1	50.1 ↑0.1

, the two components of the proposed GaitAE framework, namely the AE and HOG, consistently enhance the rank-1 accuracy across various models and datasets. Comparatively, HOG tends to yield greater improvements than AE. This suggests that addressing occlusion and other covariates during training significantly contributes to better recognition performance. While the extent of these improvements varies, they collectively demonstrate the effectiveness of our method in enhancing existing gait recognition techniques and strengthening their robustness against low-quality sequences.

Furthermore, to assess the impact of the occlusion count J, the occlusion range $H_1$ and $H_2$, and the occlusion height $H_4-H_3$ on recognition accuracy, we conduct ablation experiments on HOR using the GaitSlice model, as shown in

Table 3. The impact of the number of occlusions J, the occlusion range $H_1$ and $H_2$, and the occlusion height $H_4-H_3$ on recognition accuracy. Bold indicates the best result.

Group	${\mathit{H}}_{\mathbf{1}}$	${\mathit{H}}_{\mathbf{2}}$	${\mathit{H}}_{\mathbf{4}}\mathbf{-}{\mathit{H}}_{\mathbf{3}}$	J	Acc
1	12	40	4	1	91.0
	12	40	4	2	91.2
	12	40	4	3	91.1
2	8	44	4	2	90.8
	12	40	4	2	91.2
	16	35	4	2	91.0
3	12	40	2	2	90.9
	12	40	4	2	91.2
	12	40	6	2	90.5

. Group 1 shows that the best performance is achieved when $J=2$, possibly because using two occlusions is most effective in eliminating the impact of covariates, thereby improving accuracy. Group 2 indicates that the optimal performance is achieved when $H_1$ and $H_2$ are set to 12 and 40, respectively, suggesting that this range may contain the most covariates. Group 3 demonstrates that an occlusion height of 4 yields the best results. This is because smaller occlusions are insufficient to eliminate covariates, while larger occlusions may remove useful features. Therefore, we ultimately set $J=2$, $H_1=12$, $H_2=40$, and $H_4-H_3=4$ as the final HOR configuration.

4.5. Practicality of GaitAE

In real-world applications of gait recognition, occlusions can be a common challenge. We analyze the average rank-1 accuracy decrease under different settings, as shown in

Table 4. The average rank-1 accuracy decreases of different models when adding various occlusions of different sizes (lower is better).

Model	4 × 4		8 × 8		16 × 16
	Origin	GaitAE	Origin	GaitAE	Origin	GaitAE
GaitSet [27]	1.9	1.6 ↑0.3	5.4	5.2 ↑0.2	14.3	14.1 ↑0.2
GaitPart [15]	1.5	1.1 ↑0.4	4.4	4.1 ↑0.3	12.4	12.0 ↑0.4
GaitGL [18]	0.6	0.4 ↑0.2	1.7	1.6 ↑0.1	6.5	6.2 ↑0.3
GaitSlice [3]	1.2	1.0 ↑0.2	3.9	3.6 ↑0.3	13.0	12.8 ↑0.2
GMSN [5]	0.3	0.2 ↑0.1	1.2	1.0 ↑0.2	5.9	5.8 ↑0.1

. The results indicate that models utilizing GaitAE outperform the original models when dealing with square occlusions of side lengths 4, 8, and 16, as shown in Figure 5. These findings emphasize the advantages of GaitAE in occluded scenarios.

4.6. Robustness of GaitAE

To assess the robustness of GaitAE, we introduce Gaussian noise to the CASIA-B dataset in the GaitSlice and GMSN models. The introduction of Gaussian noise allows us to simulate common random interferences present in real-world scenarios. In our specific gait sequence instances, we add Gaussian noise with a mean of 0 and standard deviations selected from the set 0.05, 0.1, 0.15, 0.2, 0.25, 0.3 to each frame. The experimental results, depicted in Figure 6, show varying degrees of accuracy improvement when integrating GaitAE into these models, affirming the robustness of our approach. Notably, when the standard deviation of Gaussian noise is set to 0.200, the rank-1 accuracy of GaitAE improves in both the GaitSlice and GMSN models, reaching 1.4 and 0.9, respectively. This indicates that at moderate noise levels, GaitAE’s reconstruction capability effectively filters out noise while preserving key gait features, demonstrating the model’s optimal robustness.

Furthermore, we compare the performance of the original model with the model incorporating GaitAE under three walking conditions (NM, BG, CL) on the CASIA-B dataset. The experimental results, as shown in

Table 5. The impact of introducing GaitAE on the Rank-1 accuracy of existing methods.

Model	NM		BG		CL
	Origin	GaitAE	Origin	GaitAE	Origin	GaitAE
GaitSet [27]	95.0	96.1 ↑1.1	87.2	88.8 ↑1.6	70.4	74.2 ↑3.8
GaitPart [15]	96.0	96.7 ↑0.7	91.5	92.8 ↑1.3	78.7	81.1 ↑2.4
GaitGL [18]	97.4	97.4 ↑0.3	94.5	95.6 ↑0.9	83.6	84.9 ↑1.3
GaitSlice [3]	96.7	97.4 ↑0.7	92.4	93.4 ↑1.0	81.6	84.2 ↑2.6
GMSN [5]	98.2	98.3 ↑0.1	96.0	96.5↑0.5	87.0	87.7 ↑0.7

, clearly demonstrate that accuracy improved under all three conditions after the introduction of GaitAE. Notably, as the walking conditions become more complex, from NM to BG to CL, the magnitude of accuracy improvement increases. This is because GaitAE effectively refines subtle gait features under normal conditions, handles partial occlusions caused by carrying a bag, and mitigates the impact of clothing variations, thereby significantly enhancing the model’s robustness and recognition capability in more challenging scenarios.

5. Conclusions and Future Work

In this paper, we introduced GaitAE, a novel framework that significantly enhances gait recognition accuracy and robustness through two key innovations. The first innovation is the use of an autoencoder mechanism to effectively reconstruct and refine gait features, allowing the model to capture essential details while filtering out noise. The second innovation is the horizontal occlusion restriction (HOR) strategy, which improves performance by reducing the impact of covariates through the introduction of a random number of horizontally placed blocks during training. Our experimental results on the CASIA-B, OU-MVLP, and SUSTech1K datasets validate that GaitAE can be effectively integrated into existing silhouette-based gait recognition models. Experiments simulating real-world occlusion and noise demonstrate that GaitAE can also improve performance under these conditions, proving its practicality and robustness.

Moreover, GaitAE is only applied during the training phase, adding no additional parameters or computational complexity during testing, highlighting its potential for practical applications. However, whether GaitAE can be integrated with other modalities, such as pose, skeleton, and point cloud data, remains a direction for future research.