Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images
Abstract
:1. Introduction
- A novel training approach to train accurate posture estimation neural networks using solely IR and depth images;
- A method to reduce the amount of manually annotated samples required to train posture estimation networks, thereby reducing the costs and effort required to create posture-estimation datasets.
2. Related Work
3. Methodology
3.1. Problem Formulation
3.2. Data Collection
3.3. Domain-Aware Posture Estimation
3.4. Model Training Parameters and Hardware
4. Results
4.1. Results on Simulation Data
4.2. Results on Real Data
5. Conclusions and Scope for Extension
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Appendix A
Appendix A.1. Per-Joint Average Error for Simulation Data
Joint | Average Train Error (cm) | Average Test Error (cm) |
---|---|---|
Pelvis | 0.84 | 0.88 |
Abdomen | 1.15 | 1.19 |
Thorax | 2.14 | 2.31 |
Neck | 1.98 | 2.97 |
Head | 2.78 | 3.72 |
Left Hip | 0.82 | 0.86 |
Left Knee | 4.85 | 5.23 |
Right Hip | 0.92 | 0.96 |
Right Knee | 6.29 | 6.41 |
Left Shoulder | 3.30 | 4.10 |
Left Elbow | 3.33 | 5.24 |
Left Wrist | 3.68 | 6.38 |
Right Shoulder | 3.32 | 4.37 |
Right Elbow | 3.48 | 5.14 |
Right Wrist | 3.50 | 5.87 |
Appendix A.2. Per-Joint Average Error on Real Data
Joint | OpenPose-Trained Model | Fine-Tuned Model |
---|---|---|
Pelvis | 0.06 | 0.04 |
Abdomen | 0.90 | 0.76 |
Thorax | 4.21 | 2.37 |
Neck | 16.60 | 6.17 |
Head | 15.62 | 7.37 |
Left Hip | 0.82 | 0.38 |
Left Knee | 11.41 | 7.99 |
Right Hip | 0.79 | 0.42 |
Right Knee | 12.93 | 8.89 |
Left Shoulder | 15.10 | 7.05 |
Left Elbow | 15.65 | 10.14 |
Left Wrist | 13.59 | 8.85 |
Right Shoulder | 11.82 | 6.84 |
Right Elbow | 14.77 | 7.65 |
Right Wrist | 15.98 | 9.19 |
Appendix A.3. Additional Examples of Predictions on Real Data
References
- Yang, Y.; Gerlicher, M.; Bengler, K. How does relaxing posture influence take-over performance in an automated vehicle? Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2018, 62, 696–700. [Google Scholar] [CrossRef]
- Köhler, A.L.; Pelzer, J.; Seidel, K.; Ladwig, S. Sitting Postures and Activities in Autonomous Vehicles–New Requirements towards Occupant Safety. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2019, 63, 1874–1878. [Google Scholar] [CrossRef]
- Wan, J.; Wang, N. A method of motion-based immersive design system for vehicle occupant package. In Proceedings of the Volume 1B: 38th Computers and Information in Engineering Conference, Quebec City, QC, Canada, 26–29 August 2018; American Society of Mechanical Engineers: New York, NY, USA, 2018. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar] [CrossRef]
- Cao, Z.; Hidalgo Martinez, G.; Simon, T.; Wei, S.; Sheikh, Y.A. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 43, 172–186. [Google Scholar] [CrossRef] [PubMed]
- Lin, T.Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C.L.; Dollár, P. Microsoft COCO: Common Objects in Context. arXiv 2015, arXiv:1405.0312. [Google Scholar]
- Bartol, K.; Bojanić, D.; Petković, T.; D’Apuzzo, N.; Pribanic, T. A Review of 3D Human Pose Estimation from 2D Images. In Proceedings of the 3DBODY. TECH 2020-11th International Conference and Exhibition on 3D Body Scanning and Processing Technologies, Online, 17–18 November 2020. [Google Scholar] [CrossRef]
- Hu, P.; Zhao, R.; Dai, X.; Munteanu, A. Predicting high-fidelity human body models from impaired point clouds. Signal Process. 2022, 192, 108375. [Google Scholar] [CrossRef]
- Supančič, J.S.; Rogez, G.; Yang, Y.; Shotton, J.; Ramanan, D. Depth-Based Hand Pose Estimation: Methods, Data, and Challenges. Int. J. Comput. Vis. 2018, 126, 1180–1198. [Google Scholar] [CrossRef]
- Ko, K.L.; Yoo, J.S.; Han, C.W.; Kim, J.; Jung, S.W. Pose and Shape Estimation of Humans in Vehicles. IEEE Trans. Intell. Transp. Syst. 2024, 25, 402–416. [Google Scholar] [CrossRef]
- Ionescu, C.; Papava, D.; Olaru, V.; Sminchisescu, C. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1325–1339. [Google Scholar] [CrossRef] [PubMed]
- Reed, M.P.; Park, B.K.; Kim, K.H.; Raschke, U. Creating Custom Human Avatars for Ergonomic Analysis using Depth Cameras. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2014, 58, 1590–1594. [Google Scholar] [CrossRef]
- Park, B.K.D.; Wan, J.; Kozak, K.; Reed, M.P. Model-based characterisation of vehicle occupants using a depth camera. Int. J. Veh. Des. 2020, 83, 23–37. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. arXiv 2015, arXiv:1512.03385. [Google Scholar]
- Nair, V.; Hinton, G.E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, Madison, WI, USA, 21–24 June 2010; ICML’10. pp. 807–814. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the Advances in Neural Information Processing Systems; Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R., Eds.; Curran Associates, Inc.: Nice, France, 2015; Volume 28. [Google Scholar]
- Dai, J.; Li, Y.; He, K.; Sun, J. R-FCN: Object detection via region-based fully convolutional networks. In Proceedings of the NIPS’16: 30th International Conference on Neural Information Processing Systems, Barcelona Spain, 5–10 December 2016; Curran Associates Inc.: Nice, France, 2016; pp. 379–387. [Google Scholar]
- Xu, B.; Wang, N.; Chen, T.; Li, M. Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv 2015, arXiv:1505.00853. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- Gastaldi, X. Shake-Shake regularization of 3-branch residual networks. In Proceedings of the International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Reddi, S.J.; Kale, S.; Kumar, S. On the Convergence of Adam and Beyond. In Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018. [Google Scholar]
- Hoffman, J.; Wang, D.; Yu, F.; Darrell, T. FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation. arXiv 2016, arXiv:1612.02649. [Google Scholar]
Component | Type of Input Y | Z | Number of Data Points | Function |
---|---|---|---|---|
Simulated | Exact | 50,000 | Pre-training | |
Real | Approximate | 3000 | Fine-tuning | |
Real | Exact | 81 | Fine-tuning |
ID | Sex | Age | Stature | BMI | Sitting Height | Waist Circ. |
---|---|---|---|---|---|---|
1 | F | 21 | 1590 | 25.157 | 855 | 745 |
2 | F | 36 | 1530 | 25.610 | 836 | 832 |
3 | F | 30 | 1650 | 23.380 | 884 | 826 |
4 | M | 20 | 1698 | 22.580 | 909 | 775 |
5 | M | 28 | 1853 | 25.630 | 983 | 878 |
6 | F | 28 | 1607 | 38.684 | 910 | 786 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tambwekar, A.; Park, B.-K.D.; Kusari, A.; Sun, W. Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images. Sensors 2024, 24, 5530. https://doi.org/10.3390/s24175530
Tambwekar A, Park B-KD, Kusari A, Sun W. Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images. Sensors. 2024; 24(17):5530. https://doi.org/10.3390/s24175530
Chicago/Turabian StyleTambwekar, Anuj, Byoung-Keon D. Park, Arpan Kusari, and Wenbo Sun. 2024. "Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images" Sensors 24, no. 17: 5530. https://doi.org/10.3390/s24175530