Estimation of Fine-Grained Foot Strike Patterns with Wearable Smartwatch Devices
Abstract
:1. Introduction
- With the hypothesis that different types of ground contact during walking and running would result in differences in whole-body movements, we propose an intelligent system that can indirectly observe and recognize the FS patterns based on MTS signals measured from a smartwatch.
- We conducted two experiments (i.e., walking and running) to validate the proposed approach, which aims to investigate whether captured motion signals from the wrist deliver enough information to differentiate FS patterns.
2. Related Work
2.1. Human Activity Recognition
2.2. Recognition of Foot Motions from Other Body Parts Apart from Feet
2.3. Sensor-Based Recognition of Foot Strike Patterns
3. Proposed Approach
3.1. Activity Definition
3.2. System for Data Collection
3.3. Models
3.3.1. Baseline: Feature-Based Machine Learning
3.3.2. End-to-End Machine Learning
4. Experiment
4.1. Data Acquisition
4.1.1. Walking
4.1.2. Running
4.2. Classification
4.2.1. Feature-Based Classification
4.2.2. End-to-End Machine Learning
4.3. Results
5. Discussion and Limitations
5.1. Classification Performance
5.2. Data Imbalance
5.3. Effect of Waveform Length
5.4. Applications & Explorations
5.5. Limitations and Future Work
5.5.1. Inter-User Variability
5.5.2. Scalability to a Large Dataset
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Acknowledgments
Conflicts of Interest
Appendix A
References
- Gjoreski, H.; Bizjak, J.; Gjoreski, M.; Gams, M. Comparing deep and classical machine learning methods for human activity recognition using wrist accelerometer. In Proceedings of the IJCAI 2016 Workshop on Deep Learning for Artificial Intelligence, New York, NY, USA, 9–15 July 2016. [Google Scholar]
- Alsheikh, M.A.; Selim, A.; Niyato, D.; Doyle, L.; Lin, S.; Tan, H.-P. Deep activity recognition models with triaxial accelerometers. In Proceedings of the Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, 12–13 February 2016. [Google Scholar]
- Han, B.-K.; Ryu, J.-K.; Kim, S.-C. Context-Aware Winter Sports Based on Multivariate Sequence Learning. Sensors 2019, 19, 3296. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kim, S.; Han, B. Emulating Touch Signals from Multivariate Sensor Data Using Gated RNNs. In Proceedings of the 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Korea, 11–14 March 2019; pp. 628–629. [Google Scholar]
- Ravi, D.; Wong, C.; Deligianni, F.; Berthelot, M.; Andreu-Perez, J.; Lo, B.; Yang, G.-Z. Deep Learning for Health Informatics. IEEE J. Biomed. Health Inform. 2016, 21, 4–21. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Weiss, G.M.; Lockhart, J.W.; Pulickal, T.T.; McHugh, P.T.; Ronan, I.H.; Timko, J.L. Actitracker: A smartphone-based activity recognition system for improving health and well-being. In Proceedings of the 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Montreal, QC, Canada, 17–19 October 2016; pp. 682–688. [Google Scholar]
- Reiss, A.; Stricker, D. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the 2012 16th International Symposium on Wearable Computers, Newcastle, UK, 18–22 June 2012; pp. 108–109. [Google Scholar]
- Scott, J.; Dearman, D.; Yatani, K.; Truong, K.N. Sensing foot gestures from the pocket. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA, 3–6 October 2010; pp. 199–208. [Google Scholar]
- Morales, J.; Akopian, D. Physical activity recognition by smartphones, a survey. Biocybern. Biomed. Eng. 2017, 37, 388–400. [Google Scholar] [CrossRef]
- Shoaib, M.; Bosch, S.; Incel, O.D.; Scholten, H.; Havinga, P.J. A survey of online activity recognition using mobile phones. Sensors 2015, 15, 2059–2085. [Google Scholar] [CrossRef]
- Maurer, U.; Smailagic, A.; Siewiorek, D.P.; Deisher, M. Activity recognition and monitoring using multiple sensors on different body positions. In Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06), Cambridge, MA, USA, 3–5 April 2006; pp. 4–116. [Google Scholar]
- Bao, L.; Intille, S.S. Activity recognition from user-annotated acceleration data. In Proceedings of the International Conference on Pervasive Computing, Vienna, Austria, 21–23 April 2004; pp. 1–17. [Google Scholar]
- Davis, I.S.; Rice, H.M.; Wearing, S.C. Why forefoot striking in minimal shoes might positively change the course of running injuries. J. Sport Health Sci. 2017, 6, 154–161. [Google Scholar] [CrossRef] [Green Version]
- Daoud, A.I.; Geissler, G.J.; Wang, F.; Saretsky, J.; Daoud, Y.A.; Lieberman, D.E. Foot strike and injury rates in endurance runners: A retrospective study. Med. Sci. Sports Exerc. 2012, 44, 1325–1334. [Google Scholar] [CrossRef] [Green Version]
- Pohl, M.B.; Hamill, J.; Davis, I.S. Biomechanical and Anatomic Factors Associated with a History of Plantar Fasciitis in Female Runners. Clin. J. Sport Med. 2009, 19, 372–376. [Google Scholar] [CrossRef]
- Almonroeder, T.; Willson, J.D.; Kernozek, T.W. The Effect of Foot Strike Pattern on Achilles Tendon Load During Running. Ann. Biomed. Eng. 2013, 41, 1758–1766. [Google Scholar] [CrossRef]
- Telford, R.; Sly, G.J.; Hahn, A.G.; Cunningham, R.B.; Bryant, C.; Smith, J.A. Footstrike is the major cause of hemolysis during running. J. Appl. Physiol. 2003, 94, 38–42. [Google Scholar] [CrossRef] [Green Version]
- Lippi, G.; Sanchis-Gomar, F. Epidemiological, biological and clinical update on exercise-induced hemolysis. Ann. Transl. Med. 2019, 7, 270. [Google Scholar] [CrossRef]
- Hamill, J.; Gruber, A.H. Is changing footstrike pattern beneficial to runners? J. Sport Health Sci. 2017, 6, 146–153. [Google Scholar] [CrossRef] [PubMed]
- Ruder, M.; Jamison, S.T.; Tenforde, A.; Hannan, M.; Davis, I. Relationship of Footstrike Pattern and Landing Impacts During a Marathon Race. Med. Sci. Sports Exerc. 2017, 49, 136. [Google Scholar] [CrossRef]
- Kuska, E.C.; Barrios, J.A.; Kinney, A.L. Multi-segment foot model reveals distal joint kinematic differences between habitual heel-toe walking and non-habitual toe walking. J. Biomech. 2020, 110, 109960. [Google Scholar] [CrossRef] [PubMed]
- Clinic, M. Toe Walking in Children. Available online: https://www.mayoclinic.org/diseases-conditions/toe-walking/symptoms-causes/syc-20378410 (accessed on 17 December 2021).
- Ruzbarsky, J.J.; Scher, D.; Dodwell, E. Toe walking: Causes, epidemiology, assessment, and treatment. Curr. Opin. Pediatrics 2016, 28, 40–46. [Google Scholar] [CrossRef]
- Kleindienst, F.; Campe, S.; Graf, E.; Michel, K.; Witte, K. Differences between fore-and rearfoot strike running patterns based on kinetics and kinematics. In Proceedings of the ISBS-Conference Proceedings Archive, Ouro Preto, Brazil, 23–27 August 2007. [Google Scholar]
- Altman, A.R.; Davis, I.S. A kinematic method for footstrike pattern detection in barefoot and shod runners. Altman, A.R.; Davis, I.S. A kinematic method for footstrike pattern detection in barefoot and shod runners. Gait Posture 2012, 35, 298–300. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Demrozi, F.; Pravadelli, G.; Bihorac, A.; Rashidi, P. Human Activity Recognition Using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey. IEEE Access 2020, 8, 210816–210836. [Google Scholar] [CrossRef]
- Coskun, H.; Achilles, F.; DiPietro, R.; Navab, N.; Tombari, F. Long short-term memory kalman filters: Recurrent neural estimators for pose regularization. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 5525–5533. [Google Scholar]
- Cho, K.; van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. arXiv 2014, arXiv:1406.1078. [Google Scholar]
- Kim, Y. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 25–29 October 2014; pp. 1746–1751. [Google Scholar]
- LeCun, Y.; Bengio, Y. Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 1995, 3361, 1995. [Google Scholar]
- Wang, J.; Chen, Y.; Hao, S.; Peng, X.; Hu, L. Deep learning for sensor-based activity recognition: A survey. Pattern Recognit. Lett. 2019, 119, 3–11. [Google Scholar] [CrossRef] [Green Version]
- Weiss, G.M.; Yoneda, K.; Hayajneh, T. Smartphone and Smartwatch-Based Biometrics Using Activities of Daily Living. IEEE Access 2019, 7, 133190–133202. [Google Scholar] [CrossRef]
- Kwapisz, J.R.; Weiss, G.; Moore, S.A. Activity recognition using cell phone accelerometers. ACM SIGKDD Explor. Newsl. 2011, 12, 74–82. [Google Scholar] [CrossRef]
- Laput, G.; Harrison, C. Sensing Fine-Grained Hand Activity with Smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; pp. 1–13. [Google Scholar]
- Nho, Y.-H.; Lim, J.G.; Kim, D.-E.; Kwon, D.-S. User-adaptive fall detection for patients using wristband. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daegeon, Korea, 9–14 October 2016; pp. 480–486. [Google Scholar]
- Papavasileiou, I.; Qiao, Z.; Zhang, C.; Zhang, W.; Bi, J.; Han, S. GaitCode: Gait-based Continuous Authentication Using Multimodal Learning and Wearable Sensors. Smart Health 2020, 19, 100162. [Google Scholar] [CrossRef]
- Strohrmann, C.; Harms, H.; Tröster, G.; Hensler, S.; Müller, R. Out of the lab and into the woods: Kinematic analysis in running using wearable sensors. In Proceedings of the 13th International Conference on Ubiquitous Computing, Vienna, Austia, 19–20 March 2011; pp. 119–122. [Google Scholar]
- Bulling, A.; Blanke, U.; Schiele, B. A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 2014, 46, 1–33. [Google Scholar] [CrossRef]
- Reyes-Ortiz, J.-L.; Oneto, L.; Samà, A.; Parra, X.; Anguita, D. Transition-Aware Human Activity Recognition Using Smartphones. Neurocomputing 2016, 171, 754–767. [Google Scholar] [CrossRef] [Green Version]
- Haque, M.N.; Mahbub, M.; Tarek, M.H.; Lota, L.N.; Ali, A.A. Nurse Care Activity Recognition: A GRU-based approach with attention mechanism. In Proceedings of the Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, London, UK, 9–13 September 2019; pp. 719–723. [Google Scholar]
- Ma, H.; Li, W.; Zhang, X.; Gao, S.; Lu, S. AttnSense: Multi-level Attention Mechanism For Multimodal Human Activity Recognition. In Proceedings of the IJCAI, Macao, China, 10–16 August 2019; pp. 3109–3115. [Google Scholar]
- Kim, H.; Kim, H.-J.; Park, J.; Ryu, J.-K.; Kim, S.-C. Recognition of Fine-Grained Walking Patterns Using a Smartwatch with Deep Attentive Neural Networks. Sensors 2021, 21, 6393. [Google Scholar] [CrossRef] [PubMed]
- Mc Ardle, R.; Morris, R.; Hickey, A.; Del Din, S.; Koychev, I.; Gunn, R.N.; Lawson, J.; Zamboni, G.; Ridha, B.; Sahakian, B.J.; et al. Gait in Mild Alzheimer’s Disease: Feasibility of Multi-Center Measurement in the Clinic and Home with Body-Worn Sensors: A Pilot Study. J. Alzheimer’s Dis. 2018, 63, 331–341. [Google Scholar] [CrossRef]
- Moore, S.R.; Kranzinger, C.; Fritz, J.; Stöggl, T.; Kröll, J.; Schwameder, H. Foot Strike Angle Prediction and Pattern Classification Using LoadsolTM Wearable Sensors: A Comparison of Machine Learning Techniques. Sensors 2020, 20, 6737. [Google Scholar] [CrossRef]
- DeJong, A.F.; Hertel, J. Validation of Foot-Strike Assessment Using Wearable Sensors During Running. J. Athl. Train. 2020, 55, 1307–1310. [Google Scholar] [CrossRef]
- Mo, L.; Zeng, L. Running gait pattern recognition based on cross-correlation analysis of single acceleration sensor. Math. Biosci. Eng. 2019, 16, 6242–6256. [Google Scholar] [CrossRef]
- Chan, Z.Y.; Zhang, J.H.; Ferber, R.; Shum, G.; Cheung, R.T. The effects of midfoot strike gait retraining on impact loading and joint stiffness. Phys. Ther. Sport 2020, 42, 139–145. [Google Scholar] [CrossRef]
- Kasmer, M.E.; Liu, X.-C.; Roberts, K.G.; Valadao, J.M. Foot-Strike Pattern and Performance in a Marathon. Int. J. Sports Physiol. Perform. 2013, 8, 286–292. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Rish, I. An empirical study of the naive Bayes classifier. In Proceedings of the IJCAI 2001 Workshop on Empirical Methods in Artificial Intelligence, Seattle, WA, USA, 4–6 August 2001; pp. 41–46. [Google Scholar]
- Liaw, A.; Wiener, M. Classification and regression by randomForest. R News 2002, 2, 18–22. [Google Scholar]
- Cortes, C.; Vapnik, V. Support-vector networks. Mach. Learn. 1995, 20, 273–297. [Google Scholar] [CrossRef]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: Berlin, Germany, 2006. [Google Scholar]
- Gislason, P.O.; Benediktsson, J.A.; Sveinsson, J.R. Random Forests for land cover classification. Pattern Recognit. Lett. 2006, 27, 294–300. [Google Scholar] [CrossRef]
- Christ, M.; Braun, N.; Neuffer, J.; Kempa-Liehr, A.W. Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package). Neurocomputing 2018, 307, 72–77. [Google Scholar] [CrossRef]
- Benjamini, Y.; Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 2001, 29, 1165–1188. [Google Scholar] [CrossRef]
- Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef]
- Han, B.-K.; Kim, S.-C.; Kwon, D.-S. DeepSnake: Sequence Learning of Joint Torques Using a Gated Recurrent Neural Network. IEEE Access 2018, 6, 76263–76270. [Google Scholar] [CrossRef]
- Ryu, S.; Kim, S.-C. Knocking and Listening: Learning Mechanical Impulse Response for Understanding Surface Characteristics. Sensors 2020, 20, 369. [Google Scholar] [CrossRef] [Green Version]
- Perol, T.; Gharbi, M.; Denolle, M. Convolutional neural network for earthquake detection and location. Sci. Adv. 2018, 4, e1700578. [Google Scholar] [CrossRef] [Green Version]
- Ryu, S.; Kim, S.-C. Embedded identification of surface based on multirate sensor fusion with deep neural network. IEEE Embed. Syst. Lett. 2020, 13, 49–52. [Google Scholar] [CrossRef]
- Lin, M.; Chen, Q.; Yan, S. Network in network. arXiv 2013, arXiv:1312.4400. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Maaten, L.v.d.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. [Google Scholar]
- Zheng, Y.; Liu, Q.; Chen, E.; Ge, Y.; Zhao, J.L. Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Front. Comput. Sci. 2015, 10, 96–112. [Google Scholar] [CrossRef]
- Neumann, M.; Vu, N.T. Attentive convolutional neural network based speech emotion recognition: A study on the impact of input features, signal length, and acted speech. arXiv 2017, arXiv:1706.00612. [Google Scholar]
- Xu, Q.; Liu, C.; Wang, X.; Cao, Y.; Yu, H.; Li, W.; Guo, L. Machine learning classification of flow regimes in a long pipeline-riser system with differential pressure signal. Chem. Eng. Sci. 2021, 233, 116402. [Google Scholar] [CrossRef]
Total Duration (min) | Time per Subject (Mean ± SD) | |
---|---|---|
Standby | 144.45 | 35.83 ± 7.67 |
Forefoot strike | 80.16 | 7.13 ± 12.21 |
Midfoot strike | 116.57 | 9.38 ± 16.18 |
Rearfoot strike | 107.95 | 9.39 ± 16.07 |
Total Duration (min) | Time per Subject (Mean ± SD) | |
---|---|---|
Standby | 319.3 | 79.38 ± 59.84 |
Forefoot strike | 270.21 | 29.36 ± 46.25 |
Midfoot strike | 529.96 | 38.86 ± 47.78 |
Rearfoot strike | 65.82 | 10.83 ± 5.67 |
Feature Name | Descriptions |
---|---|
change_quantiles | the average, absolute value of consecutive changes of the time series inside the corridor |
cwt_coefficients | a continuous wavelet transform for the Ricker wavelet, also known as the “Mexican hat wavelet” |
fft_coefficient | the Fourier coefficients of the one-dimensional discrete Fourier Transform for real input by fast Fourier transformation algorithm |
agg_linear_trend | a linear least-squares regression for values of the time series |
quantile | the q quantile of time series |
permutation_entropy | the permutation entropy |
autocorrelation | the autocorrelation of the specified lag |
ar_coefficient | the unconditional maximum likelihood of an autoregressive process |
fourier_entropy | the binned entropy of the power spectral density of the time series |
number_peaks | the number of peaks of the time series |
fft_aggregated | the spectral centroid (mean), variance, skew, and kurtosis of the absolute Fourier transform spectrum |
ratio_beyond_r_sigma | ratio of values that are more than r ∗ std (time series) away from the mean of time series |
agg_autocorrelation | the autocorrelation of the time series |
partial_autocorrelation | the value of the partial autocorrelation function at the given lag |
spkt_welch_density | the cross power spectral density of the time series at different frequencies |
Type | Signal Length | Feature-Based Learning | Deep Neural Network | ||||
---|---|---|---|---|---|---|---|
NB | RF | SVM | LSTM | GRU | Conv1D | ||
walk | 50 | 68.220 | 78.042 | 90.103 | 92.880 | 93.189 | 95.536 |
75 | 68.929 | 80.282 | 92.305 | 93.847 | 93.378 | 95.436 | |
100 | 71.197 | 80.964 | 93.469 | 94.306 | 94.280 | 96.379 | |
150 | 72.277 | 83.816 | 93.927 | 94.064 | 94.409 | 96.313 | |
run | 50 | 47.014 | 74.896 | 93.578 | 96.270 | 96.050 | 96.772 |
75 | 51.495 | 77.820 | 94.493 | 96.704 | 96.715 | 97.150 | |
100 | 57.143 | 78.465 | 94.632 | 96.720 | 97.013 | 97.596 | |
150 | 56.129 | 78.082 | 95.303 | 96.881 | 97.426 | 98.056 |
Type | Signal Length | Feature-Based Learning | Deep Neural Network | ||||
---|---|---|---|---|---|---|---|
NB | RF | SVM | LSTM | GRU | Conv1D | ||
walk | 50 | 145.54(3.02) | 153.22(9.23) | 149.56(4.42) | 35.34 | 36.39 | 27.70 |
75 | 139.91(3.13) | 141.74(9.47) | 135.13(3.83) | 38.51 | 38.41 | 27.58 | |
100 | 131.48(3.13) | 138.24(9.43) | 131.86(3.37) | 42.14 | 41.49 | 28.10 | |
150 | 127.52(3.09) | 132.93(9.52) | 129.17(3.39) | 46.53 | 46.00 | 28.95 | |
run | 50 | 143.64(3.03) | 150.45(9.39) | 145.59(5.04) | 35.89 | 36.27 | 28.39 |
75 | 147.81(3.09) | 154.36(9.38) | 147.26(4.26) | 38.42 | 38.78 | 28.18 | |
100 | 141.39(2.98) | 151.23(9.50) | 145.17(4.06) | 41.58 | 41.56 | 27.82 | |
150 | 130.96(3.11) | 139.13(9.42) | 134.80(3.70) | 46.94 | 46.31 | 28.10 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Joo, H.; Kim, H.; Ryu, J.-K.; Ryu, S.; Lee, K.-M.; Kim, S.-C. Estimation of Fine-Grained Foot Strike Patterns with Wearable Smartwatch Devices. Int. J. Environ. Res. Public Health 2022, 19, 1279. https://doi.org/10.3390/ijerph19031279
Joo H, Kim H, Ryu J-K, Ryu S, Lee K-M, Kim S-C. Estimation of Fine-Grained Foot Strike Patterns with Wearable Smartwatch Devices. International Journal of Environmental Research and Public Health. 2022; 19(3):1279. https://doi.org/10.3390/ijerph19031279
Chicago/Turabian StyleJoo, Hyeyeoun, Hyejoo Kim, Jeh-Kwang Ryu, Semin Ryu, Kyoung-Min Lee, and Seung-Chan Kim. 2022. "Estimation of Fine-Grained Foot Strike Patterns with Wearable Smartwatch Devices" International Journal of Environmental Research and Public Health 19, no. 3: 1279. https://doi.org/10.3390/ijerph19031279