A Novel Fusion Perception Algorithm of Tree Branch/Trunk and Apple for Harvesting Robot Based on Improved YOLOv8s
Abstract
:1. Introduction
2. Materials and Methods
2.1. Acquisition and Preprocessing of Image Data
2.1.1. Image Acquisition Method
2.1.2. Preprocessing of Image Data
2.2. Network Architecture and the Improved Design of YOLOv8s
2.2.1. YOLOv8s Network Architecture
2.2.2. Inserted Design of SE Module in Backbone Network
2.2.3. Inserted Design of Dynamic Snake Convolution Module
3. Training and Evaluation of Model
3.1. Network Training
3.1.1. Training Platform
3.1.2. Training Results
3.2. Evaluation of Detection and Segmentation Model
3.2.1. Evaluation Indicators for Apple Target Detection
3.2.2. Evaluation Indicators for Apple Tree Trunk Segmentation
4. Results and Discussion
4.1. Results and Analysis of Apple Detection and Branch/Trunk Segmentation Based on Improved YOLOv8s
4.2. Comparison of Apple Detection and Branch/Trunk Segmentation Performance of Different Models
5. Conclusions
- (1)
- Considering the fact that most existing algorithms only recognize and detect fruit targets on apple trees without integrating segmentation perception for apple tree branches and trunks, the spindle-shaped fruit trees, which are widely planted in standard modern apple orchards, were focused on, and an intelligent perception algorithm for apple tree fruit detection and branch segmentation for picking robots was proposed based on an improved YOLOv8s model design, providing technical support for intelligent obstacle avoidance picking of apples using harvesting robots.
- (2)
- The Backbone and Neck architectures of the original YOLOv8s network were improved by embedding the SE module visual attention mechanism behind the C2f module of the Backbone structure, and then, the dynamic snake convolution module was embedded into the Neck structure, achieving the enhancement of the ability to extract deep detail features from images and better extracting the features of different apple targets and detailed information of tree branches and trunks, with the perceptual performance of the deep learning models being optimized.
- (3)
- The proposed improved network model can effectively recognize apple targets in images and segment tree branches and trunks. The experimental results of the test set showed that the recall for apple recognition was 96.8%, the precision was 99.6%, and the mAP value was 98.3%. The Dice value for branch and trunk segmentation was 77.8%, the IoU was 63.7%, and the mAP value was 81.6%.
- (4)
- The proposed improved YOLOv8s algorithm was compared with the original YOLOv8s, YOLOv8n, and YOLOv5s algorithms in the recognition of apple targets and segmentation of tree branches and trunks on the test set. The results showed that compared with the other three algorithms, the improved YOLOv8s algorithm increased the mAP values for apple recognition by 1.5%, 2.3%, and 6%, respectively, and increased the mAP values for branch and trunk segmentation by 3.7%, 15.4%, and 24.4%, respectively.
6. Future Work
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Yan, B.; Fan, P.; Wang, M.; Shi, S.; Lei, X.; Yang, F. Real-time apple picking pattern recognition for picking robot based on improved YOLOv5m. Trans. CSAM 2022, 53, 28–38+59. [Google Scholar] [CrossRef]
- Yan, B.; Fan, P.; Lei, X.; Liu, Z.; Yang, F. A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5. Remote Sens. 2021, 13, 1619. [Google Scholar] [CrossRef]
- Ma, H.; Li, Y.; Zhang, X.; Li, Y.; Li, Z.; Zhang, R.; Zhao, Q.; Hao, R. Target Detection for Coloring and Ripening Potted Dwarf Apple Fruits Based on Improved YOLOv7-RSES. Appl. Sci. 2024, 14, 4523. [Google Scholar] [CrossRef]
- Liu, J.; Zhao, G.; Liu, S.; Liu, Y.; Yang, H.; Sun, J.; Yan, Y.; Fan, G.; Wang, J.; Zhang, H. New Progress in Intelligent Picking: Online Detection of Apple Maturity and Fruit Diameter Based on Machine Vision. Agronomy 2024, 14, 721. [Google Scholar] [CrossRef]
- Sekharamantry, P.K.; Melgani, F.; Malacarne, J.; Ricci, R.; de Almeida Silva, R.; Marcato Junior, J. A Seamless Deep Learning Approach for Apple Detection, Depth Estimation, and Tracking Using YOLO Models Enhanced by Multi-Head Attention Mechanism. Computers 2024, 13, 83. [Google Scholar] [CrossRef]
- Wu, X.; Qi, Z.; Wang, L.; Yang, J.; Xia, X. Apple Detection Method Based on Light-YOLOv3 Convolutional Neural Network. Trans. CSAM 2020, 51, 17–25. [Google Scholar] [CrossRef]
- Zhao, H.; Qiao, Y.; Wang, H.; Yue, Y. Apple fruit recognition in complex orchard environment based on improved YOLOv3. Trans. CSAE 2021, 37, 127–135. [Google Scholar] [CrossRef]
- Lu, S.; Chen, W.; Zhang, X.; Karkee, M. Canopy-attention-YOLOv4-based immature/mature apple fruit detection on dense-foliage tree architectures for early crop load estimation. Comput. Electron. Agric. 2022, 193, 106696. [Google Scholar] [CrossRef]
- Kang, H.; Zhou, H.; Chen, C. Visual Perception and Modeling for Autonomous Apple Harvesting. Ieee Access 2020, 8, 62151–62163. [Google Scholar] [CrossRef]
- Kang, H.; Chen, C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Comput. Electron. Agric. 2020, 171, 105302. [Google Scholar] [CrossRef]
- Fu, L.; Majeed, Y.; Zhang, X.; Karkee, M.; Zhang, Q. Faster R-CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting. Biosyst. Eng. 2020, 197, 245–256. [Google Scholar] [CrossRef]
- Gao, F.; Fu, L.; Zhang, X.; Majeed, Y.; Li, R.; Karkee, M.; Zhang, Q. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Comput. Electron. Agric. 2020, 176, 105634. [Google Scholar] [CrossRef]
- Long, Y.; Li, N.; Gao, Y.; He, M.; Song, H. Apple fruit detection under natural condition using improved FCOS network. Trans. CSAE 2021, 37, 307–313. [Google Scholar] [CrossRef]
- Gao, F.; Wu, Z.; Suo, R.; Zhou, Z.; Li, R.; Fu, L.; Zhang, Z. Apple detection and counting using real-time video based on deep learning and object tracking. Trans. CSAE 2021, 37, 217–224. [Google Scholar] [CrossRef]
- Zhang, Z.; Jia, W.; Shao, W.; Hou, S.; Ze, J.; Zheng, Y. Green Apple Detection Based on Optimized FCOS in Orchards. Spectrosc. Spectr. Anal. 2022, 42, 647–653. [Google Scholar] [CrossRef]
- Sun, J.; Qian, L.; Zhu, W.; Zhou, X.; Dai, C.; Wu, X. Apple detection in complex orchard environment based on improved RetinaNet. Trans. CSAE 2022, 38, 314–322. [Google Scholar] [CrossRef]
- Wang, Z.; Wang, J.; Wang, X.; Shi, J.; Bai, X.; Zhao, Y. Lightweight Real-time Apple Detection Method Based on Improved YOLO v4. Trans. CSAM 2022, 53, 294–302. [Google Scholar]
- Hu, G.; Zhou, J.; Chen, C.; Li, C.; Sun, L.; Chen, Y.; Zhang, S.; Chen, J. Fusion of the lightweight network and visual attention mechanism to detect apples in orchard environment. Trans. CSAE 2022, 38, 131–142. [Google Scholar] [CrossRef]
- Yang, F.; Lei, X.; Liu, Z.; Fan, P.; Yan, B. Fast Recognition Method for Multiple Apple Targets in Dense Scenes Based on CenterNet. Trans. CSAM 2022, 53, 265–273. [Google Scholar] [CrossRef]
- Song, H.; Jiang, M.; Wang, Y.; Song, L. Efficient detection method for young apples based on the fusion of convolutional neural network and visual attention mechanism. Trans. CSAE 2021, 37, 297–303. [Google Scholar] [CrossRef]
- Song, H.; Ma, B.; Shang, Y.; Wen, Y.; Zhang, S. Detection of Young Apple Fruits Based on YOLO v7-ECA Model. Trans. CSAM 2023, 54, 233–242. [Google Scholar] [CrossRef]
- Zhong, H.; Zhang, Z.; Liu, H.; Wu, J.; Lin, W. Individual Tree Species Identification for Complex Coniferous and Broad-Leaved Mixed Forests Based on Deep Learning Combined with UAV LiDAR Data and RGB Images. Forests 2024, 15, 293. [Google Scholar] [CrossRef]
- Zhao, X.; Zhang, W.; Zhang, H.; Zheng, C.; Ma, J.; Zhang, Z. ITD-YOLOv8: An Infrared Target Detection Model Based on YOLOv8 for Unmanned Aerial Vehicles. Drones 2024, 8, 161. [Google Scholar] [CrossRef]
- Ye, R.; Gao, Q.; Qian, Y.; Sun, J.; Li, T. Improved YOLOv8 and SAHI Model for the Collaborative Detection of Small Targets at the Micro Scale: A Case Study of Pest Detection in Tea. Agronomy 2024, 14, 1034. [Google Scholar] [CrossRef]
- Yang, S.; Yao, J.; Teng, G. Corn Leaf Spot Disease Recognition Based on Improved YOLOv8. Agriculture 2024, 14, 666. [Google Scholar] [CrossRef]
- Wang, C.; Wang, H.; Han, Q.; Zhang, Z.; Kong, D.; Zou, X. Strawberry Detection and Ripeness Classification Using YOLOv8+ Model and Image Processing Method. Agriculture 2024, 14, 751. [Google Scholar] [CrossRef]
- Sun, D.; Zhang, K.; Zhong, H.; Xie, J.; Xue, X.; Yan, M.; Wu, W.; Li, J. Efficient Tobacco Pest Detection in Complex Environments Using an Enhanced YOLOv8 Model. Agriculture 2024, 14, 353. [Google Scholar] [CrossRef]
- Niu, S.; Nie, Z.; Li, G.; Zhu, W. Early Drought Detection in Maize Using UAV Images and YOLOv8+. Drones 2024, 8, 170. [Google Scholar] [CrossRef]
- Ma, N.; Su, Y.; Yang, L.; Li, Z.; Yan, H. Wheat Seed Detection and Counting Method Based on Improved YOLOv8 Model. Sensors 2024, 24, 1654. [Google Scholar] [CrossRef]
- Liu, M.; Cui, M.; Wei, W.; Xu, X.; Sun, C.; Li, F.; Song, Z.; Lu, Y.; Zhang, J.; Tian, F.; et al. Sorting of Mountage Cocoons Based on MobileSAM and Target Detection. Agriculture 2024, 14, 599. [Google Scholar] [CrossRef]
- Lian, X.; Li, Y.; Wang, X.; Shi, L.; Xue, C. Research on Identification and Location of Mining Landslide in Mining Area Based on Improved YOLO Algorithm. Drones 2024, 8, 150. [Google Scholar] [CrossRef]
- He, C.; Wan, F.; Ma, G.; Mou, X.; Zhang, K.; Wu, X.; Huang, X. Analysis of the Impact of Different Improvement Methods Based on YOLOV8 for Weed Detection. Agriculture 2024, 14, 674. [Google Scholar] [CrossRef]
- Yu, S.; Xue, G.; He, H.; Zhao, G.; Wen, H. Lightweight Detection of Ceramic Tile Surface Defects on improved YOLOv8. Comput. Eng. Appl. 2024, 1–19. [Google Scholar] [CrossRef]
- Yao, J.; Qi, J.M.; Zhang, J.; Shao, H.M.; Yang, J.; Li, X. A Real-Time Detection Algorithm for Kiwifruit Defects Based on YOLOv5. Electronics 2021, 10, 1711. [Google Scholar] [CrossRef]
- Li, G.; Shi, G.; Zhu, C. Dynamic Serpentine Convolution with Attention Mechanism Enhancement for Beef Cattle Behavior Recognition. Animals 2024, 14, 466. [Google Scholar] [CrossRef]
- Bai, Z.; Pei, X.; Qiao, Z.; Wu, G.; Bai, Y. Improved YOLOv7 Target Detection Algorithm Based on UAV Aerial Photography. Drones 2024, 8, 104. [Google Scholar] [CrossRef]
- Ma, B.; Hua, Z.; Wen, Y.; Deng, H.; Zhao, Y.; Pu, L.; Song, H. Using an improved lightweight YOLOv8 model for real-time detection of multi-stage apple fruit in complex orchard environments. Artif. Intell. Agric. 2024, 11, 70–82. [Google Scholar] [CrossRef]
- Firozeh, S.; Angelo, C.; Giovanni, D.; Angelo, P.; Stephan, S.; Francesco, C.; Vito, R. Optimizing tomato plant phenotyping detection: Boosting YOLOv8 architecture to tackle data complexity. Comput. Electron. Agric. 2024, 218, 108728. [Google Scholar] [CrossRef]
Perception Model | Year | Achieve Apple Detection and Branch/Trunk Segmentation Simultaneously | Reference |
---|---|---|---|
Improved YOLOv7-RSES | 2024 | N | [3] |
Improved YOLOv5s | 2024 | N | [4] |
Improved YOLOv7 | 2024 | N | [5] |
Improved YOLOv3 | 2020 | N | [6] |
Improved YOLOv3 | 2021 | N | [7] |
CA-YOLOv4 | 2022 | N | [8] |
Improved YOLOv5s | 2021 | N | [2] |
DaSNet | 2019 | Y | [9] |
DaSNet-v2 | 2020 | Y | [10] |
Faster R-CNN (VGG16) | 2020 | N | [11] |
Faster R-CNN (VGG16) | 2020 | N | [12] |
Improved FCOS | 2021 | N | [13] |
YOLOv4-tiny | 2021 | N | [14] |
Improved FCOS | 2022 | N | [15] |
Improved RetinaNet | 2022 | N | [16] |
Improved YOLOv4 | 2022 | N | [17] |
Lad-YXNet | 2022 | N | [18] |
Improved Centernet | 2022 | N | [19] |
Improved YOLOv5m | 2022 | N | [1] |
YOLOv4-SENL | 2021 | N | [20] |
YOLOv7-CEA | 2023 | N | [21] |
Model | Depth | Width | Layer | Parameters | Size (MB) |
---|---|---|---|---|---|
YOLOv8n | 0.33 | 0.25 | 225 | 3.16 × 106 | 6.23 |
YOLOv8s | 0.33 | 0.5 | 225 | 1.12 × 107 | 21.54 |
YOLOv8m | 0.67 | 0.75 | 295 | 2.59 × 107 | 49.7 |
YOLOv8l | 1 | 1 | 365 | 4.37 × 107 | 83.73 |
YOLOv8x | 1 | 1.25 | 365 | 6.82 × 107 | 130.5 |
Improved YOLOv8s | Apple detection | Precision (%) | Recall (%) | mAP (%) |
99.6 | 96.8 | 98.3 | ||
Branch/trunk segmentation | mAP (%) | Dice (%) | IoU (%) | |
81.6 | 77.8 | 63.7 |
Perception Model | Apple Recognition | Branch/Trunk Segmentation | Average Perception Speed (ms/pic) | ||||
---|---|---|---|---|---|---|---|
mAP (%) (mAP 0.5) | mAP 0.5–0.95 (%) | F1(%) | Precision (%) | Recall (%) | mAP (%) | ||
YOLOv5s | 92.3 | 83.3 | 91.6 | 98.1 | 86.4 | 57.2 | 73.1 |
YOLOv8n | 96 | 86.3 | 95.6 | 97.3 | 93.9 | 66.2 | 9.8 |
YOLOv8s | 96.8 | 90.8 | 96.6 | 99.5 | 93.9 | 77.9 | 16.1 |
Improved YOLOv8s | 98.3 | 94.8 | 98.2 | 99.6 | 96.8 | 81.6 | 17.7 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yan, B.; Liu, Y.; Yan, W. A Novel Fusion Perception Algorithm of Tree Branch/Trunk and Apple for Harvesting Robot Based on Improved YOLOv8s. Agronomy 2024, 14, 1895. https://doi.org/10.3390/agronomy14091895
Yan B, Liu Y, Yan W. A Novel Fusion Perception Algorithm of Tree Branch/Trunk and Apple for Harvesting Robot Based on Improved YOLOv8s. Agronomy. 2024; 14(9):1895. https://doi.org/10.3390/agronomy14091895
Chicago/Turabian StyleYan, Bin, Yang Liu, and Wenhui Yan. 2024. "A Novel Fusion Perception Algorithm of Tree Branch/Trunk and Apple for Harvesting Robot Based on Improved YOLOv8s" Agronomy 14, no. 9: 1895. https://doi.org/10.3390/agronomy14091895