Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment
Abstract
:1. Introduction
- The lightweight SE-YOLOv3-MobileNetV1 network is proposed to distinguish four types of maturity: <60%, 61%_70%, 71%_80%, and 81%_100% [23] under nature greenhouse environment.
- The backbone network of YOLOv3 is replaced with MobileNetV1 [24] to improve the speed of network detection.
- The Squeeze-and-Excitation [25] attention mechanism module, mosaic data augmentation, and K-means clustering algorithm are used to improve the classification accuracy of tomato maturity.
2. Materials and Methods
2.1. Data Acquisition and Annotation
2.2. Tomato Maturity Classification Network Selection and Improvement
2.2.1. Network Selection
2.2.2. Network Speed Improvement
2.2.3. Network Accuracy Improvement
2.3. Loss Function Improvement
2.4. Network Visualization
3. Results
3.1. Evaluation Indices of the Model
3.2. Performance Comparison of the Models on the Test Set
3.3. Visualization of the SE-YOLOv3-MobileNetV1 and YOLOv3-MobileNetV1 Models
3.4. Analysis of Test Results
4. Discussion
5. Conclusions
- (1)
- Speed: Replace the feature extraction network with the lightweight MobileNetV1 network with a model size of only 46.7 MB.
- (2)
- Accuracy: Due to the small parameters of lightweight network, the detection accuracy decreased. Therefore, the attention mechanism is added, which can ensure the accuracy of detection while maintaining lightweight. Through this operation, the accuracy of the model on the test set is 87.5% for category <60% tomatoes, 88.9% for 61%_70% tomatoes, 81.1% for 71%_80% tomatoes, and the accuracy of tomatoes in the 81%_100% category is 93.4%. The accuracy of tomato maturity classification is above 80%.
- (3)
- Compared with the traditional target detection, the proposed model is robust and can accurately detect tomatoes and judge the maturity of tomatoes under the condition that branches, leaves, and fruits are shielded from each other. Compared with the deep learning model detection research, the proposed model has advantages in reasoning speed, which is of great significance to the automatic picking and embedded research of tomato picking robot.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Magalhes, S.A.; Castro, L.; Moreira, G.; Santos, F.N.; Cunha, M.; Dias, J.; Moreira, A.P. Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse. Sensors 2021, 21, 3569. [Google Scholar] [CrossRef] [PubMed]
- Sun, J.; He, X.; Wu, M.; Wu, X.; Lu, B. Detection of tomato organs based on convolutional neural network under the overlap and occlusion backgrounds. Mach. Vis. Appl. 2020, 31, 31. [Google Scholar] [CrossRef]
- Moreira, G.; Magalhães, S.A.; Pinho, T.; dos Santos, F.N.; Cunha, M. Benchmark of Deep Learning and a Proposed HSV Colour Space Models for the Detection and Classification of Greenhouse Tomato. Agronomy 2022, 12, 356. [Google Scholar] [CrossRef]
- Guan, Z.; Wu, F.; Sargent, S.A. Labor Requirement and Costs for Harvesting Tomatoes. EDIS 2018, 2018. [Google Scholar] [CrossRef]
- Clark, B.; Jones, G.; Kendall, H.; Taylor, J.; Cao, Y.; Li, W.; Zhao, C.; Guijun, Y.; Chen, L.; Li, Z.; et al. A proposed framework for accelerating technology trajectories in agriculture: A case study in China. Front. Agric. Sci. Eng. 2018, 5, 485–498. [Google Scholar] [CrossRef]
- Lili, W.; Bo, Z.; Jinwei, F.; Xiaoan, H.; Shu, W.; Yashuo, L.; Zhou, Q.; Chongfeng, W. Development of a tomato harvesting robot used in greenhouse. Int. J. Agric. Biol. Eng. 2017, 10, 140–149. [Google Scholar] [CrossRef]
- Sembiring, A.; Budiman, A.; Lestari, Y.D. Design and control of agricultural robot for tomato plants treatment and harvesting. J. Phys. Conf. Ser. 2017, 930, 012019. [Google Scholar] [CrossRef]
- Zhu, A.; Bian, B.; Jiang, Y.; Hu, J. Integrated Tomato Picking and Distribution Scheduling Based on Maturity. Sustainability 2020, 12, 7934. [Google Scholar] [CrossRef]
- Zhao, Y.; Gong, L.; Huang, Y.; Liu, C. A review of key techniques of vision-based control for harvesting robot. Comput. Electron. Agric. 2016, 127, 311–323. [Google Scholar] [CrossRef]
- Zhao, Y.; Gong, L.; Zhou, B.; Huang, Y.; Liu, C. Detecting tomatoes in greenhouse scenes by combining AdaBoost classifier and colour analysis. Biosyst. Eng. 2016, 148, 127–137. [Google Scholar] [CrossRef]
- Liu, G.; Mao, S.; Kim, J.H. A mature-tomato detection algorithm using machine learning and color analysis. Sensors 2019, 19, 2023. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Chen, J.; Wu, J.; Wang, Z.; Qiang, H.; Cai, G.; Tan, C.; Zhao, C. Detecting ripe fruits under natural occlusion and illumination conditions. Comput. Electron. Agric. 2021, 190, 106450. [Google Scholar] [CrossRef]
- Kamilaris, A.; Prenafeta-Boldú, F.X. Deep learning in agriculture: A survey. Comput. Electron. Agric. 2018, 147, 70–90. [Google Scholar] [CrossRef] [Green Version]
- Gai, R.; Chen, N.; Yuan, H. A detection algorithm for cherry fruits based on the improved YOLO-v4 model. Neural Comput. Appl. 2021. [Google Scholar] [CrossRef]
- Liu, J.; Pi, J.; Xia, L. A novel and high precision tomato maturity recognition algorithm based on multi-level deep residual network. Multimed. Tools Appl. 2019, 79, 9403–9417. [Google Scholar] [CrossRef]
- Xu, Z.F.; Jia, R.S.; Liu, Y.B.; Zhao, C.Y.; Sun, H.M. Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots. IEEE Access 2020, 8, 55289–55299. [Google Scholar] [CrossRef]
- Fernqvist, F. Consumer Experiences of Tomato Quality and the Effects of Credence. Ph.D. Thesis, Swedish University of Agricultural Sciences, Uppsala, Sweden, 2014; p. 65. [Google Scholar]
- Syahrir, W.M.; Suryanti, A.; Connsynn, C. Color grading in tomato maturity estimator using image processing technique. In Proceedings of the 2009 2nd IEEE International Conference on Computer Science and Information Technology, Beijing, China, 8–11 August 2009; pp. 276–280. [Google Scholar]
- Xiao, Q.; Niu, W.; Zhang, H. Predicting fruit maturity stage dynamically based on fuzzy recognition and color feature. In Proceedings of the 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 23–25 September 2015; pp. 944–948. [Google Scholar]
- Zhang, L.; Jia, J.; Gui, G.; Hao, X.; Gao, W.; Wang, M. Deep Learning Based Improved Classification System for Designing Tomato Harvesting Robot. IEEE Access 2018, 6, 67940–67950. [Google Scholar] [CrossRef]
- Wan, P.; Toudeshki, A.; Tan, H.; Ehsani, R. A methodology for fresh tomato maturity detection using computer vision. Comput. Electron. Agric. 2018, 146, 43–50. [Google Scholar] [CrossRef]
- Yu, Y.; Zhang, K.; Yang, L.; Zhang, D. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Comput. Electron. Agric. 2019, 163, 104846. [Google Scholar] [CrossRef]
- Kaur, K.; Gupta, O. A machine learning approach to determine maturity stages of tomatoes. Orient. J. Comput. Sci. Technol. 2017, 10, 683–690. [Google Scholar] [CrossRef]
- Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv 2017, arXiv:1704.04861. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2011–2023. [Google Scholar]
- Pék, Z.; Helyes, L.; Lugasi, A. Color Changes and Antioxidant Content of Vine and Postharvest-ripened Tomato Fruits. HortScience 2010, 45, 466–468. [Google Scholar] [CrossRef]
- Lin, Z.; Sun, J.; Davis, A.; Snavely, N. Visual Chirality. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 12292–12300. [Google Scholar]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Zheng, Z.; Wang, P.; Ren, D.; Liu, W.; Ye, R.; Hu, Q.; Zuo, W. Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation. IEEE Trans. Cybern. 2021. [Google Scholar] [CrossRef] [PubMed]
- Zeiler, M.D.; Fergus, R. Visualizing and Understanding Convolutional Networks. In Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 818–833. [Google Scholar]
- Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning Deep Features for Discriminative Localization. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2921–2929. [Google Scholar]
- Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int. J. Comput. Vis. 2020, 128, 336–359. [Google Scholar] [CrossRef] [Green Version]
- Chen, J.; Wang, Z.; Wu, J.; Hu, Q.; Zhao, C.; Tan, C.; Teng, L.; Luo, T. An improved Yolov3 based on dual path network for cherry tomatoes detection. J. Food Process Eng. 2021, 44, e13803. [Google Scholar] [CrossRef]
- Lawal, M.O. Tomato detection based on modified YOLOv3 framework. Sci. Rep. 2021, 11, 1447. [Google Scholar] [CrossRef]
- Rupanagudi, S.R.; Ranjani, B.S.; Nagaraj, P.; Bhat, V.G. A cost effective tomato maturity grading system using image processing for farmers. In Proceedings of the 2014 International Conference on Contemporary Computing and Informatics (IC3I), Mysore, India, 27–29 November 2014; pp. 7–12. [Google Scholar]
Index | Type of Loss | The Meaning of Loss |
---|---|---|
1 | Box loss | Compare the position of the prediction box with the true box |
2 | Confidence loss | Confidence prediction of the target in the prediction box |
Confidence prediction without target in the prediction box | ||
3 | Classes loss | Comparison between the predicted target category and the actual category |
Model | [email protected] (%) | F1-Score (%) | Model Size (MB) |
---|---|---|---|
YOLOv3 | 97.9 | 95.2 | 117.0 |
YOLOv3-MobileNetV1 | 97.2 | 94.4 | 46.4 |
YOLOv5 | 98.4 | 96.6 | 88.5 |
SE-YOLOv3-MobileNetV1 | 97.5 | 94.9 | 46.7 |
Model | [email protected] (%) | Detection Speed (ms) |
---|---|---|
YOLOv3 | 88.7 | 505.7 |
YOLOv3-MobileNetV1 | 87.5 | 220.2 |
YOLOv5 | 86 | 463.9 |
SE-YOLOv3-MobileNetV1 | 87.7 | 227.1 |
Source | Method | Environment | Maturity Level | mAP |
---|---|---|---|---|
Zhao et al. [10] | Adaboost | Greenhouse | Ripe tomato | 96.5% |
Liu et al. [15] | DenseNet | Various film and television works, etc. | Ripe tomato | 91.26% |
Wan et al. [21] | BPNN | Laboratory | Three maturity levels | 99.31% |
Chen et al. [33] | YOLOv3-DPN | Greenhouse | All tomatoes | 96.83% |
Lawal et al. [34] | YOLO-Tomato-C | Greenhouse | Two maturity levels | 99.5% |
Rupanagudi et al. [35] | image processing | Laboratory | Six maturity levels | >98% |
Proposed method | SE-YOLOv3-MobileNetV1 | Greenhouse | Four maturity levels | 97.5% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Su, F.; Zhao, Y.; Wang, G.; Liu, P.; Yan, Y.; Zu, L. Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment. Agronomy 2022, 12, 1638. https://doi.org/10.3390/agronomy12071638
Su F, Zhao Y, Wang G, Liu P, Yan Y, Zu L. Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment. Agronomy. 2022; 12(7):1638. https://doi.org/10.3390/agronomy12071638
Chicago/Turabian StyleSu, Fei, Yanping Zhao, Guanghui Wang, Pingzeng Liu, Yinfa Yan, and Linlu Zu. 2022. "Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment" Agronomy 12, no. 7: 1638. https://doi.org/10.3390/agronomy12071638