Application of Texture Descriptors to Facial Emotion Recognition in Infants
Abstract
:Featured Application
Abstract
1. Introduction
2. Materials and Methods
2.1. Pain Perception in Babies
2.2. Feature Extraction
2.2.1. Local Binary Patterns
2.2.2. Local Ternary Patterns
2.2.3. Radon Barcodes
Algorithm 1: Radon Barcode Generation [53] |
|
2.3. Classification: Support Vector Machines
- Linear kernel:
- P-Grade polynomial kernel:
- Gaussian kernel:
2.4. The Proposed Method
3. Results
3.1. Results on LBP
3.2. Results on LTP
3.3. Results on RBC
3.4. Final Results and Discussion
4. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Ekman, P. Facial expression and emotion. Am. Psychol. 1993, 48, 384–392. [Google Scholar] [CrossRef]
- Pujol, F.A.; Mora, H.; Martínez, A. Emotion Recognition to Improve e-Healthcare Systems in Smart Cities. In Proceedings of the Research & Innovation Forum 2019, Rome, Italy, 24–26 April 2019; Springer: Cham, Switzerland, 2019; pp. 245–254. [Google Scholar] [CrossRef]
- Brahnam, S.; Chuang, C.F.; Sexton, R.S.; Shih, F.Y. Machine assessment of neonatal facial expressions of acute pain. Decis. Support Syst. 2007, 43, 1242–1254. [Google Scholar] [CrossRef]
- Roy, S.D.; Bhowmik, M.K.; Saha, P.; Ghosh, A.K. An Approach for Automatic Pain Detection through Facial Expression. Procedia Comput. Sci. 2016, 84, 99–106. [Google Scholar] [CrossRef] [Green Version]
- Lucey, P.; Cohn, J.F.; Prkachin, K.M.; Solomon, P.E.; Matthews, I. Painful data: The UNBC-McMaster shoulder pain expression archive database. In Face and Gesture 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 57–64. [Google Scholar] [CrossRef]
- Hammal, Z.; Cohn, J.F. Automatic detection of pain intensity. In Proceedings of the 14th ACM International Conference on Multimodal Interaction—ICMI ’12, Santa Monica, CA, USA, 22–26 October 2012; p. 47. [Google Scholar] [CrossRef] [Green Version]
- Lucey, P.; Cohn, J.F.; Matthews, I.; Lucey, S.; Sridharan, S.; Howlett, J.; Prkachin, K.M. Automatically Detecting Pain in Video Through Facial Action Units. IEEE Trans. Syst. Man Cybern. Part B Cybern. 2011, 41, 664–674. [Google Scholar] [CrossRef]
- Rodriguez, P.; Cucurull, G.; Gonzàlez, J.; Gonfaus, J.M.; Nasrollahi, K.; Moeslund, T.B.; Roca, F.X. Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification. IEEE Trans. Cybern. 2017, 1–11. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ilyas, C.M.A.; Haque, M.A.; Rehm, M.; Nasrollahi, K.; Moeslund, T.B. Facial Expression Recognition for Traumatic Brain Injured Patients. In Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Funchal, Madeira, Portugal, 27–29 January 2018; SCITEPRESS—Science and Technology Publications: Setúbal, Portugal, 2018; pp. 522–530. [Google Scholar] [CrossRef]
- Brahnam, S.; Chuang, C.F.; Shih, F.Y.; Slack, M.R. Machine recognition and representation of neonatal facial displays of acute pain. Artif. Intell. Med. 2006, 36, 211–222. [Google Scholar] [CrossRef] [PubMed]
- Naufal Mansor, M.; Rejab, M.N. A computational model of the infant pain impressions with Gaussian and Nearest Mean Classifier. In Proceedings of the 2013 IEEE International Conference on Control System, Computing and Engineering, Penang, Malaysia, 29 November–1 December 2013; pp. 249–253. [Google Scholar] [CrossRef]
- Nanni, L.; Lumini, A.; Brahnam, S. Local binary patterns variants as texture descriptors for medical image analysis. Artif. Intell. Med. 2010, 49, 117–125. [Google Scholar] [CrossRef]
- Celona, L.; Manoni, L. Neonatal Facial Pain Assessment Combining Hand-Crafted and Deep Features. In Proceedings of the New Trends in Image Analysis and Processing—ICIAP 2017, Catania, Italy, 11–15 September; Springer: Cham, Switzerland, 2017; pp. 197–204. [Google Scholar] [CrossRef]
- Zamzmi, G.; Goldgof, D.; Kasturi, R.; Sun, Y. Neonatal Pain Expression Recognition Using Transfer Learning. arXiv 2018, arXiv:1807.01631. [Google Scholar]
- Zamzmi, G.; Paul, R.; Goldgof, D.; Kasturi, R.; Sun, Y. Pain assessment from facial expression: Neonatal convolutional neural network (N-CNN). In Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1–7. [Google Scholar]
- Sun, Y.; Shan, C.; Tan, T.; Long, X.; Pourtaherian, A.; Zinger, S.; With, P.H.N.d. Video-based discomfort detection for infants. Mach. Vis. Appl. 2019, 30, 933–944. [Google Scholar] [CrossRef]
- Lisetti, C.L.; Nasoz, F. Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals. EURASIP J. Adv. Signal Process. 2004, 2004, 929414. [Google Scholar] [CrossRef] [Green Version]
- Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 2018, 8, 1–15. [Google Scholar] [CrossRef] [PubMed]
- Miranda Calero, J.A.; Marino, R.; Lanza-Gutierrez, J.M.; Riesgo, T.; Garcia-Valderas, M.; Lopez-Ongil, C. Embedded Emotion Recognition within Cyber-Physical Systems using Physiological Signals. In Proceedings of the 2018 Conference on Design of Circuits and Integrated Systems (DCIS), Lyon, France, 14–16 November 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Chen, M.; Ma, Y.; Li, Y.; Wu, D.; Zhang, Y.; Youn, C.H. Wearable 2.0: Enabling Human-Cloud Integration in Next, Generation Healthcare Systems. IEEE Commun. Mag. 2017, 55, 54–61. [Google Scholar] [CrossRef]
- Kwon, J.; Kim, D.H.; Park, W.; Kim, L. A wearable device for emotional recognition using facial expression and physiological response. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; pp. 5765–5768. [Google Scholar] [CrossRef]
- Washington, P.; Voss, C.; Haber, N.; Tanaka, S.; Daniels, J.; Feinstein, C.; Winograd, T.; Wall, D. A Wearable Social Interaction Aid for Children with Autism. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’16. San Jose, CA, USA, 7–12 May 2016; ACM: New York, NY, USA, 2016; pp. 2348–2354. [Google Scholar] [CrossRef]
- Kwon, J.; Kim, L. Emotion recognition using a glasses-type wearable device via multi-channel facial responses. arXiv 2019, arXiv:1905.05360. [Google Scholar]
- Dias, D.; Paulo Silva Cunha, J. Wearable Health Devices—Vital Sign Monitoring, Systems and Technologies. Sensors 2018, 18, 2414. [Google Scholar] [CrossRef] [Green Version]
- Chen, W.; Dols, S.; Oetomo, S.B.; Feijs, L. Monitoring Body Temperature of Newborn Infants at Neonatal Intensive Care Units Using Wearable Sensors. In Proceedings of the Fifth International Conference on Body Area Networks, BodyNets ’10, Corfu Island, Greece, 10–12 September 2010; ACM: New York, NY, USA, 2010; pp. 188–194. [Google Scholar] [CrossRef]
- Mahmud, M.S.; Wang, H.; Fang, H. Design of a Wireless Non-Contact Wearable System for Infants Using Adaptive Filter. In Proceedings of the 10th EAI International Conference on Mobile Multimedia Communications, Chongqing, China, 13 July 2017. [Google Scholar] [CrossRef]
- Lobo, M.A.; Hall, M.L.; Greenspan, B.; Rohloff, P.; Prosser, L.A.; Smith, B.A. Wearables for Pediatric Rehabilitation: How to Optimally Design and Use Products to Meet the Needs of Users. Phys. Ther. 2019, 99, 647–657. [Google Scholar] [CrossRef]
- Cattani, L.; Saini, H.P.; Ferrari, G.; Pisani, F.; Raheli, R. SmartCED: An Android application for neonatal seizures detection. In Proceedings of the 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Benevento, Italy, 15–18 May 2016; pp. 1–6. [Google Scholar] [CrossRef]
- Bonafide, C.P.; Jamison, D.T.; Foglia, E.E. The Emerging Market of Smartphone-Integrated Infant Physiologic Monitors. JAMA 2017, 317, 353–354. [Google Scholar] [CrossRef]
- King, D. Marketing wearable home baby monitors: Real peace of mind? BMJ 2014, 349. [Google Scholar] [CrossRef] [Green Version]
- Wang, J.; O’Kane, A.A.; Newhouse, N.; Sethu-Jones, G.R.; de Barbaro, K. Quantified Baby: Parenting and the Use of a Baby Wearable in the Wild. Proc. Acm Hum. Comput. Interact. 2017, 1, 1–19. [Google Scholar] [CrossRef] [Green Version]
- Roofthooft, D.W.E.; Simons, S.H.P.; Anand, K.J.S.; Tibboel, D.; Dijk, M.V. Eight Years Later, Are We Still Hurting Newborn Infants? Neonatology 2014, 105, 218–226. [Google Scholar] [CrossRef]
- Cruz, M.D.; Fernandes, A.M.; Oliveira, C.R. Epidemiology of painful procedures performed in neonates: A systematic review of observational studies. Eur. J. Pain 2016, 20, 489–498. [Google Scholar] [CrossRef]
- Goksan, S.; Hartley, C.; Emery, F.; Cockrill, N.; Poorun, R.; Moultrie, F.; Rogers, R.; Campbell, J.; Sanders, M.; Adams, E.; et al. fMRI reveals neural activity overlap between adult and infant pain. eLife 2015, 4, e06356. [Google Scholar] [CrossRef] [PubMed]
- Eriksson, M.; Campbell-Yeo, M. Assessment of pain in newborn infants. Semin. Fetal Neonatal Med. 2019, 24, 101003. [Google Scholar] [CrossRef] [PubMed]
- Pettersson, M.; Olsson, E.; Ohlin, A.; Eriksson, M. Neurophysiological and behavioral measures of pain during neonatal hip examination. Paediatr. Neonatal Pain 2019, 1, 15–20. [Google Scholar] [CrossRef] [Green Version]
- Stevens, B.; Johnston, C.; Petryshen, P.; Taddio, A. Premature Infant Pain Profile: Development and Initial Validation. Clin. J. Pain 1996, 12, 13. [Google Scholar] [CrossRef] [PubMed]
- Krechel, S.W.; Bildner, J. CRIES: A new neonatal postoperative pain measurement score. Initial testing of validity and reliability. Paediatr. Anaesth. 1995, 5, 53–61. [Google Scholar] [CrossRef]
- Lawrence, J.; Alcock, D.; McGrath, P.; Kay, J.; MacMurray, S.B.; Dulberg, C. The development of a tool to assess neonatal pain. Neonatal Netw. NN 1993, 12, 59–66. [Google Scholar] [CrossRef]
- Grunau, R.V.E.; Craig, K.D. Pain expression in neonates: Facial action and cry. Pain 1987, 28, 395–410. [Google Scholar] [CrossRef]
- Grunau, R.V.E.; Johnston, C.C.; Craig, K.D. Neonatal facial and cry responses to invasive and non-invasive procedures. Pain 1990, 42, 295–305. [Google Scholar] [CrossRef]
- Peters, J.W.B.; Koot, H.M.; Grunau, R.E.; de Boer, J.; van Druenen, M.J.; Tibboel, D.; Duivenvoorden, H.J. Neonatal Facial Coding System for Assessing Postoperative Pain in Infants: Item Reduction is Valid and Feasible. Clin. J. Pain 2003, 19, 353–363. [Google Scholar] [CrossRef] [Green Version]
- Sumathi, C.P.; Santhanam, T.; Mahadevi, M. Automatic Facial Expression Analysis A Survey. Int. J. Comput. Sci. Eng. Surv. 2012, 3, 47–59. [Google Scholar] [CrossRef]
- Kumari, J.; Rajesh, R.; Pooja, K.M. Facial Expression Recognition: A Survey. Procedia Comput. Sci. 2015, 58, 486–491. [Google Scholar] [CrossRef] [Green Version]
- Arias, M.C.C.; Guinsburg, R. Differences between uni-and multidimensional scales for assessing pain in term newborn infants at the bedside. Clinics 2012, 67, 1165–1170. [Google Scholar] [CrossRef]
- Witt, N.; Coynor, S.; Edwards, C.; Bradshaw, H. A Guide to Pain Assessment and Management in the Neonate. Curr. Emerg. Hosp. Med. Rep. 2016, 4, 1–10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ahmed, M.; Shaukat, A.; Akram, M.U. Comparative analysis of texture descriptors for classification. In Proceedings of the 2016 IEEE International Conference on Imaging Systems and Techniques (IST), Chania, Greece, 4–6 October 2016; pp. 24–29. [Google Scholar] [CrossRef]
- Ahonen, T.; Hadid, A.; Pietikäinen, M. Face Recognition with Local Binary Patterns. In Computer Vision—ECCV 2004; Pajdla, T., Matas, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2004; pp. 469–481. [Google Scholar]
- Shan, C.; Gong, S.; McOwan, P. Robust facial expression recognition using local binary patterns. In Proceedings of the IEEE International Conference on Image Processing 2005, Genoa, Italy, 11–14 September 2005; Volume 2, p. II-370. [Google Scholar] [CrossRef]
- Liu, L.; Fieguth, P.; Wang, X.; Pietikäinen, M.; Hu, D. Evaluation of LBP and Deep Texture Descriptors with a New Robustness Benchmark. In Lecture Notes in Computer Science, Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands, 11–14 October 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 69–86. [Google Scholar] [CrossRef]
- Tan, X.; Triggs, B. Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions. In Analysis and Modeling of Faces and Gestures; Springer: Berlin/Heidelberg, Germany, 2007; pp. 168–182. [Google Scholar] [CrossRef] [Green Version]
- Hoang, T.V.; Tabbone, S. Invariant pattern recognition using the RFM descriptor. Pattern Recognit. 2012, 45, 271–284. [Google Scholar] [CrossRef] [Green Version]
- Tizhoosh, H.R. Barcode annotations for medical image retrieval: A preliminary investigation. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 818–822. [Google Scholar] [CrossRef] [Green Version]
- Smyser, C.D.; Dosenbach, N.U.F.; Smyser, T.A.; Snyder, A.Z.; Rogers, C.E.; Inder, T.E.; Schlaggar, B.L.; Neil, J.J. Prediction of brain maturity in infants using machine-learning algorithms. NeuroImage 2016, 136, 1–9. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Guenther, N.; Schonlau, M. Support vector machines. Stata J. 2016, 16, 917–937. [Google Scholar] [CrossRef] [Green Version]
Algorithm | Recognition Rate (%) | Cross-Validation (%) | Execution Time Per Frame (ms) |
---|---|---|---|
LBP | 89.43 | 7.96 | 20 |
LTP | 89.43 | 19.23 | 300 |
RBC | 95.12 | 11.54 | 30 |
Article | Algorithm | Recognition Rate (%) |
---|---|---|
Brahnam et. al. [3] | PCA+SVM | 82.55 |
Brahnam et. al. [10] | PCA+SVM | 88 |
Mansor and Rejab [11] | LBP+Gaussian | 87.74–88 |
LBP+K-NN | 76–80 | |
Nanni et. al [12] | LBP+SVM | 82.6 |
Celona and Maloni [13] | LBP+HOG+ CNN | 82.95 |
Zamzmi et al. [14] | CNN+Strain+NB | 92.71 |
Zamzmi et al. [15] | N-CNN | 84.5 |
The proposed method | LBP+SVM | 89.43 |
The proposed method | LTP+SVM | 89.43 |
The proposed method | RBC+SVM | 95.12 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Martínez, A.; Pujol, F.A.; Mora, H. Application of Texture Descriptors to Facial Emotion Recognition in Infants. Appl. Sci. 2020, 10, 1115. https://doi.org/10.3390/app10031115
Martínez A, Pujol FA, Mora H. Application of Texture Descriptors to Facial Emotion Recognition in Infants. Applied Sciences. 2020; 10(3):1115. https://doi.org/10.3390/app10031115
Chicago/Turabian StyleMartínez, Ana, Francisco A. Pujol, and Higinio Mora. 2020. "Application of Texture Descriptors to Facial Emotion Recognition in Infants" Applied Sciences 10, no. 3: 1115. https://doi.org/10.3390/app10031115