Gait Energy Response Functions for Gait Recognition against Various Clothing and Carrying Status
Abstract
:1. Introduction
2. Related Work
2.1. Spatial Metric Learning-Based Approaches to Gait Recognition
2.2. Intensity Transformation-Based Approaches to Gait Recognition
2.3. CNN-Based Approaches to Gait Recognition
3. Gait Recognition Using GERF
3.1. Representation of Global GERF
3.2. Representation of SD-GERF
3.3. Training of GERF
3.4. Score-Level Fusion of Multiple SD-GERFs
3.5. Post-Processing
4. Experiments
4.1. Datasets
4.2. Parameter Setting
4.3. Evaluation Metrics
4.4. Comparison with Intensity Transformation-Based Methods
4.5. Comparison on Clothing and Carrying Status Variations
4.5.1. OU-TD-B and OU-LP-Bag
4.5.2. CASIA-B
4.6. Discussion
4.6.1. Comparison on Speed Variation
4.6.2. Consideration for Real-World Applications
4.7. Evaluation of Computational Time
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Nixon, M.S.; Tan, T.; Chellappa, R. Human Identification Based on Gait; International Series on Biometrics; Springer: Berlin/Heidelberg, Germany, 2005. [Google Scholar]
- Bouchrika, I.; Goffredo, M.; Carter, J.; Nixon, M. On Using Gait in Forensic Biometrics. J. Forensic Sci. 2011, 56, 882–889. [Google Scholar] [CrossRef] [PubMed]
- Iwama, H.; Muramatsu, D.; Makihara, Y.; Yagi, Y. Gait Verification System for Criminal Investigation. IPSJ Trans. Comput. Vis. Appl. 2013, 5, 163–175. [Google Scholar] [CrossRef]
- Lynnerup, N.; Larsen, P. Gait as evidence. IET Biom. 2014, 3, 47–54. [Google Scholar] [CrossRef]
- Urtasun, R.; Fua, P. 3D Tracking for Gait Characterization and Recognition. In Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea, 17–19 May 2004; pp. 17–22. [Google Scholar]
- Wagg, D.; Nixon, M. On Automated Model-Based Extraction and Analysis of Gait. In Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea, 17–19 May 2004; pp. 11–16. [Google Scholar]
- Yam, C.; Nixon, M.; Carter, J. Automated Person Recognition by Walking and Running via Model-based Approaches. Pattern Recognit. 2004, 37, 1057–1072. [Google Scholar] [CrossRef]
- Zhao, G.; Liu, G.; Li, H.; Pietikainen, M. 3D gait recognition using multiple cameras. In Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, Southampton, UK, 10–12 April 2006; pp. 529–534. [Google Scholar]
- Sarkar, S.; Phillips, J.; Liu, Z.; Vega, I.; Grother, P.; Bowyer, K. The HumanID Gait Challenge Problem: Data Sets, Performance, and Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 162–177. [Google Scholar] [CrossRef] [PubMed]
- Wang, L.; Tan, T.; Ning, H.; Hu, W. Silhouette analysis-based gait recognition for human identification. IEEE Trans. Pattern Anal. Mach. Intell. 2003, 25, 1505–1518. [Google Scholar] [CrossRef] [Green Version]
- Han, J.; Bhanu, B. Individual Recognition Using Gait Energy Image. IEEE Trans. Pattern Anal. Mach. Intell. 2006, 28, 316–322. [Google Scholar] [CrossRef] [PubMed]
- Bashir, K.; Xiang, T.; Gong, S. Gait recognition using gait entropy image. In Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP), London, UK, 3 December 2009; pp. 1–6. [Google Scholar]
- Bashir, K.; Xiang, T.; Gong, S. Gait recognition without subject cooperation. Pattern Recognit. Lett. 2010, 31, 2052–2060. [Google Scholar] [CrossRef] [Green Version]
- Castro, F.M.; Marín-Jiménez, M.J.; Mata, N.G.; Muñoz-Salinas, R. Fisher Motion Descriptor for Multiview Gait Recognition. Int. J. Pattern Recognit. Artif. Intell. 2017, 31, 1–40. [Google Scholar] [CrossRef]
- Marín-Jiménez, M.J.; Castro, F.M.; Mata, N.G.; de la Torre, F.; Muñoz-Salinas, R. Deep Multi-Task Learning for Gait-based Biometrics. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017; pp. 106–110. [Google Scholar]
- Makihara, Y.; Sagawa, R.; Mukaigawa, Y.; Echigo, T.; Yagi, Y. Gait Recognition Using a View Transformation Model in the Frequency Domain. In Proceedings of the 9th European Conference on Computer Vision (ECCV), Graz, Austria, 7–13 May 2006; pp. 151–163. [Google Scholar]
- Wang, C.; Zhang, J.; Wang, L.; Pu, J.; Yuan, X. Human Identification Using Temporal Information Preserving Gait Template. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 2164–2176. [Google Scholar] [CrossRef] [PubMed]
- Tao, D.; Li, X.; Wu, X.; Maybank, S.J. General Tensor Discriminant Analysis and Gabor Features for Gait Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 1700–1715. [Google Scholar] [CrossRef] [PubMed]
- Liu, Z.; Sarkar, S. Simplest Representation Yet for Gait Recognition: Averaged Silhouette. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR), Cambridge, UK, 26 August 2004; Volume 4, pp. 211–214. [Google Scholar]
- Li, X.; Makihara, Y.; Xu, C.; Muramatsu, D.; Yagi, Y.; Ren, M. Gait Energy Response Function for Clothing-Invariant Gait Recognition. In Proceedings of the 13th Asian Conference on Computer Vision (ACCV), Taipei, Taiwan, 20–24 November 2016; pp. 257–272. [Google Scholar]
- Makihara, Y.; Mannami, H.; Tsuji, A.; Hossain, M.; Sugiura, K.; Mori, A.; Yagi, Y. The OU-ISIR Gait Database Comprising the Treadmill Dataset. IPSJ Trans. Comput. Vis. Appl. 2012, 4, 53–62. [Google Scholar] [CrossRef] [Green Version]
- Makihara, Y.; Suzuki, A.; Muramatsu, D.; Li, X.; Yagi, Y. Joint Intensity and Spatial Metric Learning for Robust Gait Recognition. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 5705–5715. [Google Scholar]
- Yu, S.; Tan, D.; Tan, T. A Framework for Evaluating the Effect of View Angle, Clothing and Carrying Condition on Gait Recognition. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR), Hong Kong, China, 20–24 August 2006; Volume 4, pp. 441–444. [Google Scholar]
- Lee, S.; Liu, Y.; Collins, R. Shape Variation-Based Frieze Pattern for Robust Gait Recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8. [Google Scholar]
- Xu, D.; Yan, S.; Tao, D.; Zhang, L.; Li, X.; Zhang, H.J. Human gait recognition with matrix representation. IEEE Trans. Circuits Syst. Video Technol. 2006, 16, 896–903. [Google Scholar]
- Guan, Y.; Li, C.T.; Hu, Y. Robust Clothing-Invariant Gait Recognition. In Proceedings of the Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), Piraeus-Athens, Greece, 18–20 July 2012; pp. 321–324. [Google Scholar]
- Guan, Y.; Li, C.T.; Roli, F. On Reducing the Effect of Covariate Factors in Gait Recognition: A Classifier Ensemble Method. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 1521–1528. [Google Scholar] [CrossRef] [PubMed]
- Hossain, M.A.; Makihara, Y.; Wang, J.; Yagi, Y. Clothing-Invariant Gait Identification using Part-based Clothing Categorization and Adaptive Weight Control. Pattern Recognit. 2010, 43, 2281–2291. [Google Scholar] [CrossRef]
- Rokanujjaman, M.; Islam, M.; Hossain, M.; Islam, M.; Makihara, Y.; Yagi, Y. Effective Part-Based Gait Identification using Frequency-Domain Gait Entrophy Features. Multimedia Tools Appl. 2015, 74, 3099–3120. [Google Scholar] [CrossRef]
- Boulgouris, N.; Chi, Z. Human gait recognition based on matching of body components. Pattern Recognit. 2007, 40, 1763–1770. [Google Scholar] [CrossRef]
- Iwashita, Y.; Uchino, K.; Kurazume, R. Gait-Based Person Identification Robust to Changes in Appearance. Sensors 2013, 13, 7884–7901. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wu, Z.; Huang, Y.; Wang, L. Learning Representative Deep Features for Image Set Analysis. IEEE Trans. Multimedia 2015, 17, 1960–1968. [Google Scholar] [CrossRef]
- Wolf, T.; Babaee, M.; Rigoll, G. Multi-view gait recognition using 3D convolutional neural networks. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 4165–4169. [Google Scholar]
- Shiraga, K.; Makihara, Y.; Muramatsu, D.; Echigo, T.; Yagi, Y. GEINet: View-Invariant Gait Recognition Using a Convolutional Neural Network. In Proceedings of the 8th IAPR International Conference on Biometrics (ICB), Halmstad, Sweden, 13–16 June 2016; pp. 1–8. [Google Scholar]
- Zhang, C.; Liu, W.; Ma, H.; Fu, H. Siamese neural network based gait recognition for human identification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016; pp. 2832–2836. [Google Scholar]
- Wu, Z.; Huang, Y.; Wang, L.; Wang, X.; Tan, T. A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 209–226. [Google Scholar] [CrossRef] [PubMed]
- Castro, F.M.; Marín-Jiménez, M.J.; Guil, N.; López-Tapia, S.; de la Blanca, N.P. Evaluation of CNN Architectures for Gait Recognition Based on Optical Flow Maps. In Proceedings of the International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 20–22 September 2017; pp. 1–5. [Google Scholar]
- Makihara, Y.; Yagi, Y. Silhouette Extraction Based on Iterative Spatio-Temporal Local Color Transformation and Graph-Cut Segmentation. In Proceedings of the 19th International Conference on Pattern Recognition (ICPR), Tampa, FL, USA, 8–11 December 2008; pp. 1–4. [Google Scholar]
- Papafitsoros, K.; Schönlieb, C.B. A Combined First and Second Order Variational Approach for Image Reconstruction. J. Math. Imaging Vis. 2014, 48, 308–338. [Google Scholar] [CrossRef]
- Phillips, P.; Blackburn, D.; Bone, M.; Grother, P.; Micheals, R.; Tabassi, E. Face Recogntion Vendor Test. 2002. Available online: https://www.nist.gov/itl/iad/image-group/face-recognition-vendor-test-frvt-2002 (accessed on 5 August 2017).
- Kittler, J.; Hatef, M.; Duin, R.P.W.; Matas, J. On Combining Classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 1998, 20, 226–239. [Google Scholar] [CrossRef]
- Xu, D.; Huang, Y.; Zeng, Z.; Xu, X. Human Gait Recognition Using Patch Distribution Feature and Locality-Constrained Group Sparse Representation. IEEE Trans. Image Process. 2012, 21, 316–326. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Yang, J.; Zhang, D.; Frangi, A.F.; Yang, J.Y. Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 131–137. [Google Scholar] [CrossRef] [PubMed]
- Li, X.; Maybank, S.; Yan, S.; Tao, D.; Xu, D. Gait Components and Their Application to Gender Recognition. IEEE Trans. Syst. Man Cybern. Part C 2008, 38, 145–155. [Google Scholar] [Green Version]
- Aggarwal, H.; Vishwakarma, D. Covariate conscious approach for Gait recognition based upon Zernike moment invariants. IEEE Trans. Cogn. Dev. Syst. 2018, 10, 397–407. [Google Scholar] [CrossRef]
- Lombardi, S.; Nishino, K.; Makihara, Y.; Yagi, Y. Two-Point Gait: Decoupling Gait from Body Shape. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Sydney, Australia, 1–8 December 2013; pp. 1041–1048. [Google Scholar]
- Otsu, N. Optimal Linear and Nonlinear Solutions for Least-square Discriminant Feature Extraction. In Proceedings of the 6th International Conference on Pattern Recognition (ICPR), Munich, Germany, 19–22 October 1982; pp. 557–560. [Google Scholar]
- Liu, K.; Cheng, Y.; Yang, J. Algebraic feature extraction for image recognition based on an optimal discriminant criterion. Pattern Recognit. 1993, 26, 903–911. [Google Scholar] [CrossRef]
- Xu, D.; Yan, S.; Zhang, L.; Zhangh, H.J.; Shum, H.Y. Concurrent Subspaces Analysis. In Proceedings of the International Conference Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, 20–25 June 2005; pp. 203–208. [Google Scholar]
- Martin-Felez, R.; Xiang, T. Uncooperative gait recognition by learning to rank. Pattern Recognit. 2014, 47, 379–3806. [Google Scholar] [CrossRef]
- Iwama, H.; Okumura, M.; Makihara, Y.; Yagi, Y. The OU-ISIR Gait Database Comprising the Large Population Dataset and Performance Evaluation of Gait Recognition. IEEE Trans. Inf. Forensics Secur. 2012, 7, 1511–1521. [Google Scholar] [CrossRef] [Green Version]
- Chen, X.; Weng, J.; Lu, W.; Xu, J. Multi-gait Recognition based on Attribute Discovery. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 1697–1710. [Google Scholar] [CrossRef] [PubMed]
- Alotaibi, M.; Mahmood, A. Improved Gait recognition based on specialized deep convolutional neural networks. In Proceedings of the IEEE Applied Imagery Pattern Recognition Workshop (AIPR), Washington, DC, USA, 13–15 October 2015; pp. 1–7. [Google Scholar]
- Kusakunniran, W. Recognizing Gaits on Spatio-Temporal Feature Domain. IEEE Trans. Inf. Forensics Secur. 2014, 9, 1416–1423. [Google Scholar] [CrossRef]
- Xu, C.; Makihara, Y.; Li, X.; Yagi, Y.; Lu, J. Speed Invariance vs. Stability: Cross-Speed Gait Recognition Using Single-Support Gait Energy Image. In Proceedings of the 13th Asian Conference on Computer Vision (ACCV), Taipei, Taiwan, 20–24 November 2016; pp. 52–67. [Google Scholar]
- Makihara, Y.; Tsuji, A.; Yagi, Y. Silhouette Transformation based on Walking Speed for Gait Identification. In Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010; pp. 717–722. [Google Scholar]
- Kusakunniran, W.; Wu, Q.; Zhang, J.; Li, H. Gait Recognition Across Various Walking Speeds Using Higher Order Shape Configuration Based on a Differential Composition Model. IEEE Trans. Syst. Man Cybern. Part B 2012, 42, 1654–1668. [Google Scholar] [CrossRef] [PubMed]
- Guan, Y.; Li, C.T. A robust speed-invariant gait recognition system for walker and runner identification. In Proceedings of the 6th IAPR International Conference on Biometrics (ICB), Madrid, Spain, 4–7 June 2013; pp. 1–8. [Google Scholar]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R.B. Mask R-CNN. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017. [Google Scholar]
Dataset | OU-TD-B | OU-LP-Bag | ||
---|---|---|---|---|
Methods | z-EER | Rank-1 | z-EER | Rank-1 |
GEI [11] | 16.12 | 52.8 | 19.59 | 24.6 |
GEnI [12] | 12.81 | 59.0 | 18.82 | 29.5 |
Masked GEI [13] | 28.15 | 28.0 | 61.95 | 0.1 |
Global GERF [20] | 11.68 | 60.6 | 16.22 | 33.5 |
SD-GERF (proposed) | 7.41 | 71.5 | 12.99 | 36.1 |
Dataset | OU-TD-B | OU-LP-Bag | ||
---|---|---|---|---|
Methods | z-EER | Rank-1 | z-EER | Rank-1 |
SVB-frieze pattern [24] | 19.81 | - | - | - |
Component-based [44] | 18.25 | - | - | - |
Whole-based [16] | 14.88 | 58.1 | - | - |
Part-based [28] | 10.26 | 66.3 | - | - |
Part-EnDFT [29] | - | 72.8 | - | - |
AESI+ZNK [45] | - | 72.7 | - | - |
GEI+RSM [26] | N/A | 80.4 | N/A | N/A |
Gabor+RSM-HDF [27] | N/A | 90.7 | N/A | N/A |
Gabor GEI [18] | 11.80 | 62.3 | 10.48 | 46.4 |
TPG+GEI [46] | 7.10 | - | - | - |
GEI w/LDA [47] | 15.63 | 54.3 | 8.10 | 54.6 |
GEI w/2DLDA [48] | 8.91 | 70.7 | 11.47 | 43.3 |
GEI w/CSA [49] | 16.00 | - | - | - |
GEI w/DATER [25] | 8.72 | - | - | - |
GEI w/Ranking SVM [50] | 10.75 | 58.4 | 10.81 | 28.3 |
JIS-ML [22] | 6.66 | 74.5 | 5.45 | 57.4 |
GEINet [34] | 8.38 | 60.2 | 9.75 | 40.7 |
Gabor+Global GERF [20] | 5.14 | 82.7 | 6.67 | 58.3 |
Gabor+SD-GERF (proposed) | 4.61 | 87.4 | 5.60 | 64.3 |
Methods | Set-A | Set-B | Set-C | Average |
---|---|---|---|---|
GEI [11] | 99 | 60 | 30 | 63.0 |
GEnI [12] | 98.3 | 80.1 | 33.5 | 70.6 |
STIP+NN [54] | 95.4 | 60.9 | 52 | 69.4 |
AESI+ZNK [45] | 100 | 93.1 | 81.3 | 91.5 |
L-CRF [52] | 98.6 | 90.2 | 85.8 | 91.5 |
GEINet [34] | 97.5 | 84.5 | 71.8 | 84.6 |
DCNN [53] | 95.6 | 88.3 | 76.2 | 86.7 |
Gabor+Global GERF [20] | 99 | 91 | 92 | 94.0 |
Gabor+SD-GERF (proposed) | 99 | 100 | 96 | 98.3 |
Methods | Small Speed | Large Speed | Average (All Speed) |
---|---|---|---|
STM [56] | 90 | 58 | - |
DCM [57] | 98 | 82 | 92.44 |
RSM [58] | 100 | 95 | 98.07 |
SSGEI [55] | 100 | 98 | 99.33 |
Gabor+Global GERF [20] | 100 | 92 | 96.89 |
Gabor+SD-GERF (proposed) | 100 | 96 | 98.11 |
Running Stage | Machine Specification | Training Time | Query Time of Each Sequence (#Gallery Sequences) | |
---|---|---|---|---|
Method | ||||
Gabor+RSM-HDF [27] | Intel Core i5 3.10 GHz processor | 320.090 | 0.600 (122) | |
Proposed method | Intel Core i7 4.00 GHz processor | 13.330 | 0.016 (48) | |
Proposed method (estimated) | 75% computing power | 17.773 | 0.054 (122) |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, X.; Makihara, Y.; Xu, C.; Muramatsu, D.; Yagi, Y.; Ren, M. Gait Energy Response Functions for Gait Recognition against Various Clothing and Carrying Status. Appl. Sci. 2018, 8, 1380. https://doi.org/10.3390/app8081380
Li X, Makihara Y, Xu C, Muramatsu D, Yagi Y, Ren M. Gait Energy Response Functions for Gait Recognition against Various Clothing and Carrying Status. Applied Sciences. 2018; 8(8):1380. https://doi.org/10.3390/app8081380
Chicago/Turabian StyleLi, Xiang, Yasushi Makihara, Chi Xu, Daigo Muramatsu, Yasushi Yagi, and Mingwu Ren. 2018. "Gait Energy Response Functions for Gait Recognition against Various Clothing and Carrying Status" Applied Sciences 8, no. 8: 1380. https://doi.org/10.3390/app8081380