MFCF-Gait: Small Silhouette-Sensitive Gait Recognition Algorithm Based on Multi-Scale Feature Cross-Fusion
Abstract
:1. Introduction
2. Related Works
2.1. Takemura Method
2.2. Super-Resolution
2.3. Set Pooling
3. Materials and Methods
3.1. Dataset Detail
3.1.1. Dataset
3.1.2. Dataset Splitting and Rank-1
3.2. Methods
3.2.1. Model Overview
3.2.2. Multi-Scale Feature Cross-Fusion Module
3.2.3. Loss
4. Experimental Results
4.1. Implementation Details
4.2. Super-Resolution Algorithm Comparison Experiments
4.3. Feature Fusion Module Ablation Experiments
4.4. Comparison with the Existing Methods
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Wan, C.; Wang, L.; Phoha, V.V. A survey on gait recognition. ACM Comput. Surv. (CSUR) 2018, 51, 1–35. [Google Scholar] [CrossRef]
- Mogan, J.N.; Lee, C.P.; Lim, K.M. Advances in vision-based gait recognition: From handcrafted to deep learning. Sensors 2022, 22, 5682. [Google Scholar] [CrossRef] [PubMed]
- Sepas-Moghaddam, A.; Etemad, A. Deep gait recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 45, 264–284. [Google Scholar] [CrossRef] [PubMed]
- Liao, R.; Yu, S.; An, W.; Huang, Y. A model-based gait recognition method with body pose and human prior knowledge. Pattern Recognit. 2020, 98, 107069. [Google Scholar] [CrossRef]
- Teepe, T.; Khan, A.; Gilg, J.; Herzog, F.; Hörmann, S.; Rigoll, G. Gaitgraph: Graph convolutional network for skeleton-based gait recognition. In Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA, 19–22 September 2021; pp. 2314–2318. [Google Scholar]
- Li, X.; Makihara, Y.; Xu, C.; Yagi, Y.; Yu, S.; Ren, M. End-to-end model-based gait recognition. In Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan, 30 November–4 December 2020. [Google Scholar]
- Kanazawa, A.; Black, M.J.; Jacobs, D.W.; Malik, J. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7122–7131. [Google Scholar]
- Chao, H.; He, Y.; Zhang, J.; Feng, J. Gaitset: Regarding gait as a set for cross-view gait recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; Volume 33, pp. 8126–8133. [Google Scholar]
- Fan, C.; Peng, Y.; Cao, C.; Liu, X.; Hou, S.; Chi, J.; Huang, Y.; Li, Q.; He, Z. Gaitpart: Temporal part-based model for gait recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 14225–14233. [Google Scholar]
- Lin, B.; Zhang, S.; Yu, X. Gait recognition via effective global-local feature representation and local temporal aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 14648–14656. [Google Scholar]
- Fan, C.; Liang, J.; Shen, C.; Hou, S.; Huang, Y.; Yu, S. Opengait: Revisiting gait recognition towards better practicality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
- Takemura, N.; Makihara, Y.; Muramatsu, D.; Echigo, T.; Yagi, Y. Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ Trans. Comput. Vis. Appl. 2018, 10, 4. [Google Scholar] [CrossRef]
- Hou, S.; Cao, C.; Liu, X.; Huang, Y. Gait lateral network: Learning discriminative and compact representations for gait recognition. In European Conference on Computer Vision; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
- Makihara, Y.; Mori, A.; Yagi, Y. Temporal super resolution from a single quasi-periodic image sequence based on phase registration. In Proceedings of the Computer Vision–ACCV 2010: 10th Asian Conference on Computer Vision, Queenstown, New Zealand, 8–12 November 2010; Revised Selected Papers, Part I 10. Springer: Berlin/Heidelberg, Germany, 2011. [Google Scholar]
- Zhang, J.; Cheng, Y.; Chen, C. Low resolution gait recognition with high frequency super resolution. In Proceedings of the PRICAI 2008: Trends in Artificial Intelligence: 10th Pacific Rim International Conference on Artificial Intelligence, Hanoi, Vietnam, 15–19 December 2008; Proceedings 10. Springer: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
- Han, J.; Bhanu, B. Individual recognition using gait energy image. IEEE Trans. Pattern Anal. Mach. Intell. 2006, 28, 316–322. [Google Scholar] [CrossRef] [PubMed]
- Wang, C.Y.; Bochkovskiy, A.; Liao, H.Y.M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
- Nguyen, K.; Fookes, C.; Sridharan, S.; Tistarelli, M.; Nixon, M. Super-resolution for biometrics: A comprehensive survey. Pattern Recognit. 2018, 78, 23–42. [Google Scholar] [CrossRef]
- Blu, T.; Thévenaz, P.; Unser, M. Linear interpolation revitalized. IEEE Trans. Image Process. 2004, 13, 710–719. [Google Scholar] [CrossRef] [PubMed]
- Keys, R. Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 1981, 29, 1153–1160. [Google Scholar] [CrossRef]
- Chung, M.; Jung, M.; Kim, Y. Enhancing Remote Sensing Image Super-Resolution Guided by Bicubic-Downsampled Low-Resolution Image. Remote Sens. 2023, 15, 3309. [Google Scholar] [CrossRef]
- Song, C.; Huang, Y.; Wang, W.; Wang, L. CASIA-E: A large comprehensive dataset for gait recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 45, 2801–2815. [Google Scholar] [CrossRef] [PubMed]
- Yu, S.; Tan, D.; Tan, T. A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China, 20–24 August 2006; Volume 4, pp. 441–444. [Google Scholar]
- Gao, W.; Cao, B.; Shan, S.; Chen, X.; Zhou, D.; Zhang, X.; Zhao, D. The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Trans. Syst. Man Cybern.-Part A Syst. Hum. 2007, 38, 149–161. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Han, X.; Wang, L.; Wang, X.; Zhang, P.; Xu, H. A multi-scale recursive attention feature fusion network for image super-resolution reconstruction algorithm. Sensors 2023, 23, 9458. [Google Scholar] [CrossRef] [PubMed]
- Fu, Y.; Wei, Y.; Zhou, Y.; Shi, H.; Huang, G.; Wang, X.; Yao, Z.; Huang, T. Horizontal pyramid matching for person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; Volume 33, pp. 8295–8302. [Google Scholar]
- Yu, D.; Wang, H.; Chen, P.; Wei, Z. Mixed pooling for convolutional neural networks. In Rough Sets and Knowledge Technology: Proceedings of the 9th International Conference, RSKT 2014, Shanghai, China, 24–26 October 2014, Proceedings 9; Springer International Publishing: Berlin/Heidelberg, Germany, 2014; pp. 364–375. [Google Scholar]
- Wang, J.; Sun, K.; Cheng, T.; Jiang, B.; Deng, C.; Zhao, Y.; Liu, D.; Mu, Y.; Tan, M.; Wang, X.; et al. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 43, 3349–3364. [Google Scholar] [CrossRef] [PubMed]
- Liu, Z.; Gao, G.; Sun, L.; Fang, Z. HRDNet: High-resolution detection network for small objects. In Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China, 5–9 July 2021. [Google Scholar]
- Schroff, F.; Kalenichenko, D.; Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 815–823. [Google Scholar]
- Mao, A.; Mohri, M.; Zhong, Y. Cross-entropy loss functions: Theoretical analysis and applications. In Proceedings of the International Conference on Machine Learning, ICML 2023, Honolulu, HI, USA, 23–29 July 2023; pp. 23803–23828. [Google Scholar]
- Lin, T.Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on COMPUTER vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2117–2125. [Google Scholar]
Environment Name | Configure Parameters |
---|---|
Operating system | Windows10 |
GPU | NVIDIA RTX 3060 |
CUDA | 11.8 |
Compiled language | Python 3.9 |
Open source framework | Pytorch 1.12 |
Hyperparameters | Value |
---|---|
iter | 40,000 |
Optimizer | SGD |
Learning Rate | 0.1 |
Momentum | 0.9 |
Weight Decay | 0.0005 |
Learning Rate Schedule | MultiStep |
Milestones | 10,000 20,000 30,000 |
gamma | 0.1 |
Resolution | Nearest Neighbor | Bilinear | Bicubic | ACC/% | ||
---|---|---|---|---|---|---|
NM | BG | CL | ||||
64 × 64 | ✓ | 95.0 | 87.7 | 71.4 | ||
✓ | 96.1 | 90.7 | 76.0 | |||
✓ | 96.5 | 91.4 | 78.2 | |||
32 × 32 | ✓ | 86.5 | 75.4 | 53.7 | ||
✓ | 87.0 | 77.8 | 57.1 | |||
✓ | 94.2 | 87.7 | 71.6 |
Resolution | MGB | FPN | MFCF (Ours) | ACC/% | ||
---|---|---|---|---|---|---|
NM | BG | CL | ||||
64 × 64 | 95.9 | 90.2 | 76.8 | |||
✓ | 96.4 | 91.6 | 75.5 | |||
✓ | 96.0 | 90.4 | 76.4 | |||
✓ | 96.5 | 91.4 | 78.2 | |||
32 × 32 | 83.8 | 75.8 | 54.6 | |||
✓ | 85.9 | 77.5 | 54.9 | |||
✓ | 90.3 | 82.6 | 63.4 | |||
✓ | 94.2 | 87.7 | 71.6 |
Resolution | Type | Gallery | 0~180° | Mean | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Probe | 0° | 18° | 36° | 54° | 72° | 90° | 108° | 126° | 144° | 162° | 180° | |||
64 × 64 | NM | GaitSet | 90.8 | 97.9 | 99.4 | 96.9 | 93.6 | 91.7 | 95.0 | 97.8 | 98.9 | 96.8 | 85.8 | 95.0 |
GaitPart | 94.1 | 98.6 | 99.3 | 98.5 | 94.0 | 92.3 | 95.9 | 98.4 | 99.2 | 97.8 | 90.4 | 96.2 | ||
GaitGL | 96.0 | 98.3 | 99.0 | 97.9 | 96.9 | 95.4 | 97.0 | 98.9 | 99.3 | 98.8 | 94.0 | 97.4 | ||
GaitBase | 95.6 | 99.2 | 100 | 99.0 | 97.6 | 95.4 | 97.5 | 99.4 | 100 | 99.1 | 94.2 | 97.9 | ||
MFCF-Gait (Ours) | 93.8 | 99.2 | 99.6 | 98.1 | 94.9 | 93.3 | 96.6 | 98.7 | 98.6 | 98.5 | 90.1 | 96.5 | ||
BG | GaitSet | 83.8 | 91.2 | 91.8 | 88.8 | 83.3 | 81.0 | 84.1 | 90.0 | 92.2 | 94.4 | 79.0 | 87.2 | |
GaitPart | 89.1 | 94.8 | 96.7 | 95.1 | 88.3 | 94.9 | 89.0 | 93.5 | 96.1 | 93.8 | 85.8 | 91.5 | ||
GaitGL | 92.6 | 96.6 | 96.8 | 95.5 | 93.5 | 89.3 | 92.2 | 96.5 | 98.2 | 96.9 | 91.5 | 94.5 | ||
GaitBase | 92.5 | 95.7 | 96.1 | 95.7 | 92.1 | 90.2 | 92.2 | 95.3 | 97.1 | 95.4 | 89.9 | 93.8 | ||
MFCF-Gait (Ours) | 89.8 | 94.8 | 95.0 | 94.0 | 89.4 | 85.5 | 88.8 | 93.2 | 94.9 | 94.7 | 85.5 | 91.4 | ||
CL | GaitSet | 61.4 | 75.4 | 80.7 | 77.3 | 72.1 | 70.1 | 71.5 | 73.5 | 73.5 | 68.4 | 50.0 | 70.4 | |
GaitPart | 70.7 | 85.5 | 86.9 | 83.3 | 77.1 | 72.5 | 76.9 | 82.2 | 83.8 | 80.2 | 66.5 | 78.7 | ||
GaitGL | 76.6 | 90.0 | 90.3 | 87.1 | 84.5 | 79.0 | 84.1 | 87.0 | 87.3 | 84.4 | 69.5 | 83.6 | ||
GaitBase | 68.0 | 79.9 | 82.7 | 81.5 | 77.4 | 75.5 | 76.8 | 79.9 | 80.6 | 78.7 | 67.5 | 77.1 | ||
MFCF-Gait (Ours) | 72.3 | 84.0 | 85.2 | 81.5 | 77.3 | 74.0 | 76.8 | 79.8 | 81.4 | 81.2 | 67.1 | 78.2 | ||
32 × 32 | NM | GaitSet | 64.4 | 79.6 | 88.1 | 88.9 | 80.0 | 75.8 | 81.1 | 86.3 | 86.8 | 79.8 | 60.7 | 79.2 |
GaitPart | 70.6 | 87.3 | 92.9 | 91.0 | 84.3 | 79.8 | 86.7 | 89.8 | 92.6 | 85.0 | 67.1 | 84.3 | ||
GaitGL | 73.5 | 90.6 | 93.2 | 91.5 | 86.9 | 83.9 | 86.8 | 91.6 | 94.3 | 90.5 | 67.9 | 86.4 | ||
GaitBase | 84.1 | 96.5 | 98.5 | 95.4 | 90.1 | 88.7 | 92.0 | 96.1 | 97.7 | 94.4 | 80.2 | 92.2 | ||
MFCF-Gait (Ours) | 86.5 | 95.8 | 98.9 | 97.6 | 94.0 | 91.8 | 95.4 | 98.1 | 97.5 | 95.3 | 85.5 | 94.2 | ||
BG | GaitSet | 55.7 | 70.2 | 76.4 | 79.9 | 69.4 | 64.2 | 70.0 | 81.3 | 77.6 | 74.8 | 51.9 | 70.1 | |
GaitPart | 62.0 | 77.8 | 83.1 | 81.6 | 74.5 | 68.6 | 77.5 | 81.6 | 84.2 | 74.6 | 54.6 | 74.6 | ||
GaitGL | 68.1 | 86.0 | 88.5 | 86.2 | 81.5 | 77.3 | 80.2 | 86.6 | 88.6 | 86.9 | 60.3 | 80.9 | ||
GaitBase | 74.2 | 86.6 | 91.5 | 88.9 | 86.1 | 82.8 | 86.2 | 89.4 | 91.5 | 86.7 | 80.2 | 85.2 | ||
MFCF-Gait (Ours) | 80.1 | 89.8 | 93.5 | 93.1 | 88.1 | 83.0 | 87.0 | 92.8 | 92.0 | 87.9 | 77.2 | 87.7 | ||
CL | GaitSet | 35.0 | 48.4 | 58.5 | 57.4 | 55.3 | 50.9 | 55.9 | 56.9 | 52.8 | 46.8 | 32.8 | 50.0 | |
GaitPart | 38.3 | 60.2 | 64.8 | 66.3 | 62.1 | 56.9 | 61.7 | 67.7 | 61.1 | 49.4 | 36.1 | 56.8 | ||
GaitGL | 45.0 | 71.8 | 77.5 | 72.6 | 69.9 | 63.1 | 64.1 | 69.3 | 67.8 | 62.3 | 39.4 | 63.9 | ||
GaitBase | 45.0 | 64.3 | 70.1 | 69.6 | 67.1 | 61.6 | 64.4 | 67.5 | 63.7 | 61.2 | 43.1 | 61.6 | ||
MFCF-Gait (Ours) | 58.7 | 73.9 | 79.8 | 75.7 | 74.3 | 70.6 | 73.2 | 76.7 | 76.6 | 72.8 | 55.0 | 71.6 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Song, C.; Yun, L.; Li, R. MFCF-Gait: Small Silhouette-Sensitive Gait Recognition Algorithm Based on Multi-Scale Feature Cross-Fusion. Sensors 2024, 24, 5500. https://doi.org/10.3390/s24175500
Song C, Yun L, Li R. MFCF-Gait: Small Silhouette-Sensitive Gait Recognition Algorithm Based on Multi-Scale Feature Cross-Fusion. Sensors. 2024; 24(17):5500. https://doi.org/10.3390/s24175500
Chicago/Turabian StyleSong, Chenyang, Lijun Yun, and Ruoyu Li. 2024. "MFCF-Gait: Small Silhouette-Sensitive Gait Recognition Algorithm Based on Multi-Scale Feature Cross-Fusion" Sensors 24, no. 17: 5500. https://doi.org/10.3390/s24175500