Towards Facial Expression Recognition for On-Farm Welfare Assessment in Pigs
Abstract
:1. Introduction
Contributions
- A first attempt to automate stress detection using facial expression in pigs on the farm via machine vision using a convolutional neural network.
- It is demonstrated that we can do so with >90% accuracy on animals that are not part of the model’s training set.
2. Background
3. Methods
3.1. Data Collection and Organisation
3.1.1. Ethical Approval
3.1.2. Animals and Housing
3.1.3. Image Collection and Social Stress Application
3.1.4. Image Identification and Cleaning
3.2. Dataset and Image Preprocessing
- To remove extraneous information from the images that might provide reliable but undesired discriminatory information (i.e., different objects in the background or different ambient illumination on different days);
- To remove secondary animals from the automatically detected masks (i.e., there may be a second pig which is behind a gate/fence which the instance segmentation may detect);
- To remove any animals that are too far from the camera and therefore too small for any sort of useful facial analysis to be performed.
3.3. Description of our CNN and the Leave-One-Out Cross-Validation Paradigm
4. Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Alonso, M.E.; González-Montaña, J.R.; Lomillos, J.M. Consumers’ Concerns and Perceptions of Farm Animal Welfare. Animals 2020, 10, 385. [Google Scholar] [CrossRef] [Green Version]
- Dawkins, M.S. Animal welfare and efficient farming: Is conflict inevitable? Anim. Prod. Sci. 2017, 57, 201–208. [Google Scholar] [CrossRef]
- Martínez-Miró, S.; Tecles, F.; Ramón, M.; Escribano, D.; Hernández, F.; Madrid, J.; Orengo, J.; Martínez-Subiela, S.; Manteca, X.; Cerón, J.J. Causes, consequences and biomarkers of stress in swine: An update. BMC Vet. Res. 2016, 12, 171. [Google Scholar] [CrossRef] [Green Version]
- Serpell, J.A. How happy is your pet? The problem of subjectivity in the assessment of companion animal welfare. Anim. Welf. 2019, 28, 57–66. [Google Scholar] [CrossRef]
- Tuyttens, F.A.M.; de Graaf, S.; Heerkens, J.L.; Jacobs, L.; Nalon, E.; Ott, S.; Stadig, L.; Van Laer, E.; Ampe, B. Observer bias in animal behaviour research: Can we believe what we score, if we score what we believe? Anim. Behav. 2014, 90, 273–280. [Google Scholar] [CrossRef]
- Ekman, P. Universal facial expressions of emotions. Calif. Ment. Health Res. Dig. 1970, 8, 151–158. [Google Scholar]
- Kaya, H.; Gürpınar, F.; Salah, A.A. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 2017, 65, 66–75. [Google Scholar] [CrossRef]
- Ekman, R. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS); Oxford University Press: Oxford, UK, 1997. [Google Scholar]
- Lien, J.J.; Kanade, T.; Cohn, J.F.; Li, C.C. Automated facial expression recognition based on FACS action units. In Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 14–16 April 1998; pp. 390–395. [Google Scholar]
- Waller, B.M.; Julle-Daniere, E.; Micheletta, J. Measuring the evolution of facial ‘expression’ using multi-species FACS. Neurosci. Biobehav. Rev. 2020, 113, 1–11. [Google Scholar] [CrossRef]
- Camerlink, I.; Coulange, E.; Farish, M.; Baxter, E.M.; Turner, S.P. Facial expression as a potential measure of both intent and emotion. Sci. Rep. 2018, 8, 17602. [Google Scholar] [CrossRef] [PubMed]
- Vullo, C.; Barbieri, S.; Catone, G.; Graïc, J.M.; Magaletti, M.; Di Rosa, A.; Motta, A.; Tremolada, C.; Canali, E.; Dalla Costa, E. Is the Piglet Grimace Scale (PGS) a Useful Welfare Indicator to Assess Pain after Cryptorchidectomy in Growing Pigs? Animals 2020, 10, 412. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Di Giminiani, P.; Brierley, V.L.M.H.; Scollo, A.; Gottardo, F.; Malcolm, E.M.; Edwards, S.A.; Leach, M.C. The Assessment of Facial Expressions in Piglets Undergoing Tail Docking and Castration: Toward the Development of the Piglet Grimace Scale. Front. Vet. Sci. 2016, 3, 100. [Google Scholar] [CrossRef] [Green Version]
- Koolhaas, J.M.; Bartolomucci, A.; Buwalda, B.; de Boer, S.F.; Flügge, G.; Korte, S.M.; Meerlo, P.; Murison, R.; Olivier, B.; Palanza, P.; et al. Stress revisited: A critical evaluation of the stress concept. Neurosci. Biobehav. Rev. 2011, 35, 1291–1301. [Google Scholar] [CrossRef] [PubMed]
- Moberg, G.P.; Mench, J.A. The Biology of Animal Stress: Basic Principles and Implications for Animal Welfare; CABI: Oxon, UK, 2000. [Google Scholar]
- Cook, N.J. Minimally invasive sampling media and the measurement of corticosteroids as biomarkers of stress in animals. Can. J. Anim. Sci. 2012, 92, 227–259. [Google Scholar] [CrossRef]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. arXiv 2017, arXiv:1703.06870. [Google Scholar]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Ison, S.H.; Donald, R.D.; Jarvis, S.; Robson, S.K.; Lawrence, A.B.; Rutherford, K.M.D. Behavioral and physiological responses of primiparous sows to mixing with older, unfamiliar sows. J. Anim. Sci. 2014, 92, 1647–1655. [Google Scholar] [CrossRef]
- Jarvis, S.; Moinard, C.; Robson, S.K.; Baxter, E.; Ormandy, E.; Douglas, A.J.; Seckl, J.R.; Russell, J.A.; Lawrence, A.B. Programming the offspring of the pig by prenatal social stress: Neuroendocrine activity and behaviour. Horm. Behav. 2006, 49, 68–80. [Google Scholar] [CrossRef]
- Ison, S.H.; D’Eath, R.B.; Robson, S.K.; Baxter, E.M.; Ormandy, E.; Douglas, A.J.; Russell, J.A.; Lawrence, A.B.; Jarvis, S. ‘Subordination style’in pigs? The response of pregnant sows to mixing stress affects their offspring’s behaviour and stress reactivity. Appl. Anim. Behav. Sci. 2010, 124, 16–27. [Google Scholar] [CrossRef]
- Rutherford, K.; Donald, R.; Arnott, G.; Rooke, J.; Dixon, L.; Mehers, J.; Turnbull, J.; Lawrence, A. Farm animal welfare: Assessing risks attributable to the prenatal environment. Anim. Welf. 2012, 21, 419–429. [Google Scholar] [CrossRef]
- Rutherford, K.M.D.; Piastowska-Ciesielska, A.; Donald, R.D.; Robson, S.K.; Ison, S.H.; Jarvis, S.; Brunton, P.J.; Russell, J.A.; Lawrence, A.B. Prenatal stress produces anxiety prone female offspring and impaired maternal behaviour in the domestic pig. Physiol. Behav. 2014, 129, 255–264. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Otten, W.; Kanitz, E.; Tuchscherer, M. The impact of pre-natal stress on offspring development in pigs. J. Agric. Sci. 2015, 153, 907–919. [Google Scholar] [CrossRef]
- Lin, T.Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C.L.; Dollár, P. Microsoft COCO: Common Objects in Context. arXiv 2015, arXiv:1405.0312. [Google Scholar]
- Hansen, M.F.; Smith, M.L.; Smith, L.N.; Salter, M.G.; Baxter, E.M.; Farish, M.; Grieve, B. Towards on-farm pig face recognition using convolutional neural networks. Comput. Ind. 2018, 98, 145–152. [Google Scholar] [CrossRef]
- Biewald, L. Experiment Tracking with Weights and Biases, 2020. Available online: wandb.com (accessed on 10 June 2020).
- Selvaraju, R.; Das, A.; Vedantam, R.; Cogswell, M.; Parikh, D.; Batra, D. Grad-CAM: Why Did You Say That? arXiv 2016, arXiv:1611.07450. [Google Scholar]
- Wemelsfelder, F.; Hunter, E.A.; Mendl, M.T.; Lawrence, A.B. Assessing the’whole animal’: A free-choice-profiling approach. Anim. Behav. 2001, 62, 209–220. [Google Scholar] [CrossRef] [Green Version]
- Telkänranta, H.; Marchant-Forde, J.N.; Valros, A. Tear staining in pigs: A potential tool for welfare assessment on commercial farms. Animal 2016, 10, 318–325. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Larsen, M.L.V.; Gustafsson, A.; Marchant-Forde, J.N.; Valros, A. Tear staining in finisher pigs and its relation to age, growth, sex and potential pen level stressors. Animal 2019, 13, 1704–1711. [Google Scholar] [CrossRef] [Green Version]
- Fureix, C.; Jego, P.; Henry, S.; Lansade, L.; Hausberger, M. Towards an Ethological Animal Model of Depression? A Study on Horses. PLoS ONE 2012, 7, e39280. [Google Scholar] [CrossRef] [PubMed]
- de Oliveira, D.; Keeling, L.J. Routine activities and emotion in the life of dairy cows: Integrating body language into an affective state framework. PLoS ONE 2018, 13, e0195674. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Veenstra, L.; Schneider, I.K.; Koole, S.L. Embodied mood regulation: The impact of body posture on mood recovery, negative thoughts, and mood-congruent recall. Cogn. Emot. 2017, 31, 1361–1376. [Google Scholar] [CrossRef]
Batch | N Pigs | Condition | N Images |
---|---|---|---|
1 | 2 | Stressed | 5957 |
Unstressed | 1980 | ||
2 | 3 | Stressed | 3588 |
Unstressed | 1222 | ||
3 | 2 | Stressed | 1282 |
Unstressed | 443 | ||
4 | 3 | Stressed | 1170 |
Unstressed | 672 | ||
5 | 2 | Stressed | 1501 |
Unstressed | 346 |
Unstressed | Stressed | ||||||||
---|---|---|---|---|---|---|---|---|---|
Omitted | Acc. | Prec. | Recall | F1 | N | Prec. | Recall | F1 | N |
None | 0.99 | 0.99 | 0.98 | 0.98 | 4663 | 0.99 | 1.00 | 0.99 | 13,498 |
1 | 0.98 | 0.97 | 0.94 | 0.95 | 1980 | 0.98 | 0.99 | 0.98 | 5957 |
2 | 0.94 | 0.91 | 0.83 | 0.87 | 1222 | 0.94 | 0.97 | 0.96 | 3588 |
3 | 0.96 | 0.92 | 0.93 | 0.92 | 443 | 0.98 | 0.97 | 0.97 | 1282 |
4 | 0.91 | 0.88 | 0.86 | 0.87 | 672 | 0.92 | 0.93 | 0.93 | 1170 |
5 | 0.98 | 0.96 | 0.91 | 0.93 | 346 | 0.98 | 0.99 | 0.99 | 1501 |
Cumulative | 0.96 | 0.93 | 0.90 | 0.91 | 4663 | 0.96 | 0.98 | 0.97 | 13,498 |
Predicted | |||
---|---|---|---|
Stressed | Unstressed | ||
GT | Stressed | 0.90 | 0.10 |
Unstressed | 0.02 | 0.98 |
Omitted Batch | Full Pig Acc. | Eyes Only Acc. |
---|---|---|
None | 0.99 | 0.98 |
Batch 1 | 0.98 | 0.92 |
Batch 2 | 0.94 | 0.90 |
Batch 3 | 0.96 | 0.95 |
Batch 4 | 0.91 | 0.94 |
Batch 5 | 0.98 | 0.95 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hansen, M.F.; Baxter, E.M.; Rutherford, K.M.D.; Futro, A.; Smith, M.L.; Smith, L.N. Towards Facial Expression Recognition for On-Farm Welfare Assessment in Pigs. Agriculture 2021, 11, 847. https://doi.org/10.3390/agriculture11090847
Hansen MF, Baxter EM, Rutherford KMD, Futro A, Smith ML, Smith LN. Towards Facial Expression Recognition for On-Farm Welfare Assessment in Pigs. Agriculture. 2021; 11(9):847. https://doi.org/10.3390/agriculture11090847
Chicago/Turabian StyleHansen, Mark F., Emma M. Baxter, Kenneth M. D. Rutherford, Agnieszka Futro, Melvyn L. Smith, and Lyndon N. Smith. 2021. "Towards Facial Expression Recognition for On-Farm Welfare Assessment in Pigs" Agriculture 11, no. 9: 847. https://doi.org/10.3390/agriculture11090847