Train Fast While Reducing False Positives: Improving Animal Classification Performance Using Convolutional Neural Networks
Abstract
:1. Introduction
2. Materials and Methods
2.1. Data Acquisition
2.2. Image Pre-Processing
2.3. Proposed Approach
2.3.1. Metric
2.3.2. Workflow
Proposed Hard-Negative Mining (HNM)
Training Times
3. Results
3.1. Hard-Negative Mining
3.2. Class Activation Maps
3.3. Training Times
4. Discussion
4.1. Training the Models
4.2. Hard-Negative Mining
4.3. Perspectives on Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Seed | MCC Val | ValPool MCC | ValPool FP | MCC Test | Test FP |
---|---|---|---|---|---|
123 | 0.9891 | 0.6241 | 1390 | 0.6164 | 1903 |
2167 | 0.9771 | 0.5311 | 2217 | 0.4603 | 4303 |
1545 | 0.9803 | 0.5829 | 1702 | 0.5549 | 2620 |
8297 | 0.9879 | 0.6568 | 1158 | 0.6060 | 1992 |
1487 | 0.9858 | 0.5724 | 1806 | 0.5546 | 2576 |
7725 | 0.9847 | 0.5914 | 1630 | 0.5459 | 2746 |
8780 | 0.9913 | 0.6096 | 1498 | 0.5660 | 2504 |
5027 | 0.9836 | 0.5857 | 1683 | 0.4648 | 4249 |
970 | 0.9858 | 0.6285 | 1359 | 0.5328 | 2945 |
960 | 0.9923 | 0.7149 | 854 | 0.6142 | 1760 |
Average | 0.9858 | 0.6098 | 1529.7 | 0.5516 | 2759.8 |
SD | 0.0047 | 0.0506 | 373.6252 | 0.0553 | 886.6702 |
Seed | MCC Val | ValPool MCC | ValPool FP | MCC Test | Test FP |
---|---|---|---|---|---|
123 | 1.0000 | 0.9791 | 51 | 0.9883 | 34 |
2167 | 0.9983 | 0.9791 | 51 | 0.9879 | 31 |
1545 | 0.9992 | 0.9752 | 61 | 0.9879 | 34 |
8297 | 0.9983 | 0.9772 | 56 | 0.9925 | 20 |
1487 | 0.9983 | 0.9729 | 67 | 0.9867 | 38 |
7725 | 0.9992 | 0.9791 | 51 | 0.9886 | 31 |
8780 | 0.9992 | 0.9748 | 62 | 0.9854 | 42 |
5027 | 0.9992 | 0.9868 | 32 | 0.9931 | 20 |
970 | 0.9983 | 0.9633 | 92 | 0.9807 | 59 |
960 | 0.9992 | 0.9701 | 74 | 0.9863 | 36 |
Average | 0.9989 | 0.9758 | 59.7 | 0.9877 | 34.5 |
SD | 0.0006 | 0.0063 | 16.042 | 0.0006 | 11.138 |
References
- Jachmann, H. Estimating Abundance of African Wildlife: An Aid to Adaptive Management, 1st ed.; Springer Science & Business Media: Berlin, Germany, 2012; ISBN 978-1-4615-1381-0. [Google Scholar]
- Linchant, J.; Lisein, J.; Semeki, J.; Lejeune, P.; Vermeulen, C. Are Unmanned Aircraft Systems (UASs) the Future of Wildlife Monitoring? A Review of Accomplishments and Challenges: A Review of UASs in Wildlife Monitoring. Mamm. Rev. 2015, 45, 239–252. [Google Scholar] [CrossRef]
- Goode, M.J.; Beaver, J.T.; Muller, L.I.; Clark, J.D.; van Manen, F.T.; Harper, C.A.; Basinger, P.S. Capture—Recapture of White-Tailed Deer Using DNA from Fecal Pellet Groups. Wildl. Biol. 2014, 20, 270–278. [Google Scholar] [CrossRef]
- Corlatti, L.; Gugiatti, A.; Pedrotti, L. Spring Spotlight Counts Provide Reliable Indices to Track Changes in Population Size of Mountain-Dwelling Red Deer Cervus Elaphus. Wildl. Biol. 2016, 22, 268–276. [Google Scholar] [CrossRef] [Green Version]
- Hodgson, J.C.; Baylis, S.M.; Mott, R.; Herrod, A.; Clarke, R.H. Precision Wildlife Monitoring Using Unmanned Aerial Vehicles. Sci. Rep. 2016, 6, 22574. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Witmer, G.W. Wildlife Population Monitoring: Some Practical Considerations. Wildl. Res. 2005, 32, 259. [Google Scholar] [CrossRef]
- Lhoest, S.; Linchant, J.; Quevauvillers, S.; Vermeulen, C.; Lejeune, P. How Many Hippos (Homhip)—Algorithm for Automatic Counts of Animals with Infrared Thermal Imagery from UAV. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2015, XL-3/W3, 355–362. [Google Scholar] [CrossRef] [Green Version]
- Rey, N.; Volpi, M.; Joost, S.; Tuia, D. Detecting Animals in African Savanna with UAVs and the Crowds. Remote Sens. Environ. 2017, 200, 341–351. [Google Scholar] [CrossRef] [Green Version]
- Vermeulen, C.; Lejeune, P.; Lisein, J.; Sawadogo, P.; Bouché, P. Unmanned Aerial Survey of Elephants. PLoS ONE 2013, 8, e54700. [Google Scholar] [CrossRef] [Green Version]
- Ransom, J.I. Detection Probability in Aerial Surveys of Feral Horses. J. Wildl. Manag. 2012, 76, 299. [Google Scholar] [CrossRef]
- Burke, C.; Rashman, M.; Wich, S.; Symons, A.; Theron, C.; Longmore, S. Optimizing Observing Strategies for Monitoring Animals Using Drone-Mounted Thermal Infrared Cameras. Int. J. Remote Sens. 2019, 40, 439–467. [Google Scholar] [CrossRef] [Green Version]
- Wang, D.; Shao, Q.; Yue, H. Surveying Wild Animals from Satellites, Manned Aircraft and Unmanned Aerial Systems (UASs): A Review. Remote Sens. 2019, 11, 1308. [Google Scholar] [CrossRef] [Green Version]
- Van Gemert, J.C.; Verschoor, C.R.; Mettes, P.; Epema, K.; Koh, L.P.; Wich, S. Nature Conservation Drones for Automatic Localization and Counting of Animals. In Proceedings of the ECCV Workshops, Zurich, Switzerland, 6–7 September 2014; pp. 255–270. [Google Scholar]
- Gonzalez, L.; Montes, G.; Puig, E.; Johnson, S.; Mengersen, K.; Gaston, K. Unmanned Aerial Vehicles (UAVs) and Artificial Intelligence Revolutionizing Wildlife Monitoring and Conservation. Sensors 2016, 16, 97. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bondi, E.; Jain, R.; Aggrawal, P.; Anand, S.; Hannaford, R.; Kapoor, A.; Piavis, J.; Shah, S.; Joppa, L.; Dilkina, B.; et al. BIRDSAI: A Dataset for Detection and Tracking in Aerial Thermal Infrared Videos. In Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, CO, USA, 1–5 March 2020; pp. 1736–1745. [Google Scholar]
- Chrétien, L.-P.; Théau, J.; Ménard, P. Visible and Thermal Infrared Remote Sensing for the Detection of White-Tailed Deer Using an Unmanned Aerial System: Detection of White-Tailed Deer Using an UAS. Wildl. Soc. Bull. 2016, 40, 181–191. [Google Scholar] [CrossRef]
- Longmore, S.N.; Collins, R.P.; Pfeifer, S.; Fox, S.E.; Mulero-Pázmány, M.; Bezombes, F.; Goodwin, A.; De Juan Ovelar, M.; Knapen, J.H.; Wich, S.A. Adapting Astronomical Source Detection Software to Help Detect Animals in Thermal Images Obtained by Unmanned Aerial Systems. Int. J. Remote Sens. 2017, 38, 2623–2638. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on Imagenet Classification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 2015, 115, 211–252. [Google Scholar] [CrossRef] [Green Version]
- Christin, S.; Hervet, É.; Lecomte, N. Applications for Deep Learning in Ecology. Methods Ecol. Evol. 2019, 10, 1632–1644. [Google Scholar] [CrossRef]
- Sun, C.; Shrivastava, A.; Singh, S.; Gupta, A. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. arXiv 2017, arXiv:1707.02968. [Google Scholar]
- Maire, F.; Alvarez, L.M.; Hodgson, A. Automating Marine Mammal Detection in Aerial Images Captured During Wildlife Surveys: A Deep Learning Approach. In Proceedings of the Australasian Joint Conference on Artificial Intelligence, Canberra, ACT, Australia, 30 November–4 December 2015; pp. 379–385. [Google Scholar]
- Gray, P.C.; Fleishman, A.B.; Klein, D.J.; McKown, M.W.; Bézy, V.S.; Lohmann, K.J.; Johnston, D.W. A Convolutional Neural Network for Detecting Sea Turtles in Drone Imagery. Methods Ecol. Evol. 2018, 10, 345–355. [Google Scholar] [CrossRef]
- Gray, P.C.; Bierlich, K.C.; Mantell, S.A.; Friedlaender, A.S.; Goldbogen, J.A.; Johnston, D.W. Drones and Convolutional Neural Networks Facilitate Automated and Accurate Cetacean Species Identification and Photogrammetry. Methods Ecol. Evol. 2019, 10, 1490–1500. [Google Scholar] [CrossRef]
- Bowler, E.; Fretwell, P.T.; French, G.; Mackiewicz, M. Using Deep Learning to Count Albatrosses from Space: Assessing Results in Light of Ground Truth Uncertainty. Remote Sens. 2020, 12, 2026. [Google Scholar] [CrossRef]
- Smith, L.N. A Disciplined Approach to Neural Network Hyper-Parameters: Part 1—Learning Rate, Batch Size, Momentum, and Weight Decay. arXiv 2018, arXiv:1803.09820. [Google Scholar]
- Kellenberger, B.; Marcos, D.; Tuia, D. Detecting Mammals in UAV Images: Best Practices to Address a Substantially Imbalanced Dataset with Deep Learning. Remote Sens. Environ. 2018, 216, 139–153. [Google Scholar] [CrossRef] [Green Version]
- Leevy, J.L.; Khoshgoftaar, T.M.; Bauder, R.A.; Seliya, N. A Survey on Addressing High-Class Imbalance in Big Data. J. Big Data 2018, 5, 42. [Google Scholar] [CrossRef]
- Mazurowski, M.A.; Habas, P.A.; Zurada, J.M.; Lo, J.Y.; Baker, J.A.; Tourassi, G.D. Training Neural Network Classifiers for Medical Decision Making: The Effects of Imbalanced Datasets on Classification Performance. Neutral Netw. 2008, 21, 427–436. [Google Scholar] [CrossRef] [Green Version]
- Chan, P.K.; Stolfo, S.J. Toward Scalable Learning with Non-Uniform Class and Cost Distributions: A Case Study in Credit Card Fraud Detection. In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining (KDD’98), New York, NY, USA, 27–31 August 1998; p. 12. [Google Scholar]
- Mac Namee, B.; Cunningham, P.; Byrne, S.; Corrigan, O.I. The Problem of Bias in Training Data in Regression Problems in Medical Decision Support. Artif. Intell. Med. 2002, 24, 51–70. [Google Scholar] [CrossRef]
- Buda, M.; Maki, A.; Mazurowski, M.A. A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks. Neutral Netw. 2018, 106, 249–259. [Google Scholar] [CrossRef] [Green Version]
- Sung, K.-K.; Poggio, T. Example-Based Learning for View-Based Human Face Detection. IEEE Trans. Pattern Anal. Mach. Intell. 1998, 20, 39–51. [Google Scholar] [CrossRef] [Green Version]
- Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems; Touretzky, D.S., Mozer, M.C., Hasselmo, M.E., Eds.; MIT Press: Cambridge, MA, USA, 2019; pp. 8026–8037. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Liu, L.; Jiang, H.; He, P.; Chen, W.; Liu, X.; Gao, J.; Han, J. On the Variance of the Adaptive Learning Rate and Beyond. arXiv 2020, arXiv:1908.03265. [Google Scholar]
- Zhang, M.R.; Lucas, J.; Hinton, G.; Ba, J. Lookahead Optimizer: K Steps Forward, 1 Step Back. arXiv 2019, arXiv:1907.08610. [Google Scholar]
- Prechelt, L. Early Stopping—But When? In Neural Networks: Tricks of the Trade; Lecture Notes in Computer Science; Orr, G.B., Müller, K.-R., Eds.; Springer: Berlin/Heidelberg, Germany, 1998; Volume 1524, pp. 55–69. ISBN 978-3-540-65311-0. [Google Scholar]
- Smith, L.N. Cyclical Learning Rates for Training Neural Networks. arXiv 2015, arXiv:1506.01186. [Google Scholar]
- Chicco, D.; Jurman, G. The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation. BMC Genom. 2020, 21, 6. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning Deep Features for Discriminative Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2921–2929. [Google Scholar]
- Norouzzadeh, M.S.; Nguyen, A.; Kosmala, M.; Swanson, A.; Palmer, M.S.; Packer, C.; Clune, J. Automatically Identifying, Counting, and Describing Wild Animals in Camera-Trap Images with Deep Learning. Proc. Natl. Acad. Sci. USA 2018, 115, E5716–E5725. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Willi, M.; Pitman, R.T.; Cardoso, A.W.; Locke, C.; Swanson, A.; Boyer, A.; Veldthuis, M.; Fortson, L. Identifying Animal Species in Camera Trap Images Using Deep Learning and Citizen Science. Methods Ecol. Evol. 2019, 10, 80–91. [Google Scholar] [CrossRef] [Green Version]
Initial Training Set | Initial Validation Set | Training Pool | Validation Pool | Test Set | |
---|---|---|---|---|---|
Summer | |||||
Deer | 3307 | 911 | 3307 | 911 | 1209 |
Negative | 3307 | 911 | 627,131 | 85,923 | 325,756 |
Imbalance factor | 1 | 1 | |||
Winter | |||||
Deer | 3434 | 1203 | 3434 | 1203 | 1518 |
Negative | 3434 | 1203 | 424,647 | 77,100 | 426,625 |
Imbalance factor | 1 | 1 | 193 | 64 | 281 |
Training Set | % of Training Pool | Training Pool | Validation Set | % of Validation Pool | Validation Pool | ||
---|---|---|---|---|---|---|---|
Summer | |||||||
Round 0 | Deer | 3307 | 100 | 3434 | 911 | 100 | 911 |
Negative | 3307 | 0.53 | 627,131 | 911 | 1.06 | 85,923 | |
Round 1 | Deer | 3307 | 100 | 3307 | 911 | 100 | 911 |
Negative | 6144 | 0.98 | 627,131 | 1765 | 2.06 | 85,923 | |
Imbalance factor | 2 | 190 | 2 | 94 | |||
Winter | |||||||
Round 0 | Deer | 3434 | 100 | 3434 | 1203 | 100 | 1203 |
Negative | 3434 | 0.81 | 426,747 | 1203 | 1.56 | 77,100 | |
Round 1 | Deer | 3434 | 100 | 3434 | 1203 | 100 | 1203 |
Negative | 3487 | 1.44 | 426,747 | 1254 | 1.63 | 77,100 | |
Imbalance factor | 1 | 124 | 1 | 64 |
Round 0 | Round 1 | ||||||
---|---|---|---|---|---|---|---|
Predicted Label | Predicted Label | ||||||
Deer | Negative | MCC | Deer | Negative | MCC | ||
Validation /True label | Deer | 911 | 0 | 0.7149 | 902 | 9 | 0.9928 |
Negative | 854 | 85,069 | 4 | 85,919 | |||
Test /True label | Deer | 1157 | 52 | 0.6142 | 1192 | 17 | 0.9859 |
Negative | 1760 | 323,996 | 17 | 325,739 |
Round 0 | Round 1 | ||||||
---|---|---|---|---|---|---|---|
Predicted Label | Predicted Label | ||||||
Deer | Negative | MCC | Deer | Negative | MCC | ||
Validation /True label | Deer | 1203 | 0 | 0.9791 | 1202 | 1 | 0.9975 |
Negative | 51 | 77,049 | 5 | 77,095 | |||
Test /True label | Deer | 1516 | 2 | 0.9883 | 1502 | 16 | 0.9940 |
Negative | 34 | 426,591 | 2 | 426,623 |
Total Number of Epochs | Total Training Time (min) | Average Training Time Per Model (min) | |
---|---|---|---|
Summer | |||
HNM | 1150 | 1068 | 106.8 |
Full dataset | 10 | 932 | 932 |
Winter | |||
HNM | 1135 | 872 | 87.2 |
Full dataset | 14 | 295 | 295 |
Validation Pool MCC | Test Set MCC | |
---|---|---|
Summer | ||
Best HNM model | 0.993 | 0.986 |
Full dataset | 0.877 | 0.923 |
Winter | ||
Best HNM model | 0.998 | 0.994 |
Full dataset | 0.996 | 0.990 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Moreni, M.; Theau, J.; Foucher, S. Train Fast While Reducing False Positives: Improving Animal Classification Performance Using Convolutional Neural Networks. Geomatics 2021, 1, 34-49. https://doi.org/10.3390/geomatics1010004
Moreni M, Theau J, Foucher S. Train Fast While Reducing False Positives: Improving Animal Classification Performance Using Convolutional Neural Networks. Geomatics. 2021; 1(1):34-49. https://doi.org/10.3390/geomatics1010004
Chicago/Turabian StyleMoreni, Mael, Jerome Theau, and Samuel Foucher. 2021. "Train Fast While Reducing False Positives: Improving Animal Classification Performance Using Convolutional Neural Networks" Geomatics 1, no. 1: 34-49. https://doi.org/10.3390/geomatics1010004