Segmentation Network for Multi-Shape Tea Bud Leaves Based on Attention and Path Feature Aggregation
Abstract
:1. Introduction
2. Materials and Methods
2.1. Data Acquisition and Processing
2.2. Unet-Enhanced Model
2.2.1. Segmentation Network Architecture
- (1)
- A hierarchical Transformer block is designed based on a self-attention mechanism in the encoder network. It captures dependencies between input features during the generation of high-resolution coarse features and low-resolution fine features, enhancing the extraction of common features of tea buds and leaves with different shapes.
- (2)
- A multi-path feature aggregation module is designed to fuse feature information from multiple different levels, alleviating the loss of spatial detail information caused by downsampling.
- (3)
- A polarized attention mechanism is introduced to endow the model with a more comprehensive perception capability for fine-grained targets within the field of view.
2.2.2. Transformer Block
2.2.3. Multi-Path Feature Aggregation Module
2.2.4. Polarized Attention Mechanism
2.3. Model Training
2.3.1. Training Details
2.3.2. Evaluation Metrics
3. Results and Analysis
3.1. Training Results
3.2. Comparison Experiment of Different Backbones
3.3. Ablation Experiments
3.3.1. Path Feature Aggregation Module
3.3.2. Attention Mechanism
3.4. Comparison with Advanced Segmentation Models
3.5. Segmentation Performance
3.6. Feature Extraction Analysis
3.7. Feature Attention Analysis
3.8. Limitations and Future Work
4. Conclusions
- (1)
- The target of one bud–one leaf presents a variety of different shapes under different angles of imaging vision due to the occlusion of stems or overlapping of buds and leaves. Utilizing a hierarchical Transformer block based on a self-attention mechanism facilitates the capture of long-range dependencies between features, enhancing the expression of common features.
- (2)
- Using multi-path feature aggregation to fuse features and introducing a polarization self-attention mechanism can effectively enhance the fine-grained features of tea bud leaves. The experimental results show that the IoU for background, Tea_Y, Tea_V, and Tea_I are 99.62%, 88.69%, 86.46%, and 89.94%, respectively. The mIoU was 91.18% and the mPA was 95.10%. These performance values verify that the designed model can effectively and accurately segment the tea bud leaf target with different shapes and different fine-grain sizes in the field of view.
Author Contributions
Funding
Institutional Review Board Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Xie, S.; Sun, H. Tea-YOLOv8s: A tea bud detection model based on deep learning and computer vision. Sensors 2023, 23, 6576. [Google Scholar] [CrossRef]
- Wang, J.; Li, X.; Yang, G.; Wang, F.; Men, S.; Xu, B.; Xu, Z.; Yang, H.; Yan, L. Research on Tea Trees Germination Density Detection Based on Improved YOLOv5. Forests 2022, 13, 2091. [Google Scholar] [CrossRef]
- Zhu, Y.; Wu, C.; Tong, J.; Tong, J.; Chen, J.; He, L.; Wang, R.; Jia, J. Deviation tolerance performance evaluation and experiment of picking end effector for famous tea. Agriculture 2021, 11, 128. [Google Scholar] [CrossRef]
- Xu, W.; Zhao, L.; Li, J.; Shang, S.; Ding, X.; Wang, T. Detection and classification of tea buds based on deep learning. Comput. Electron. Agric. 2022, 192, 106547. [Google Scholar] [CrossRef]
- Zhang, S.; Yang, H.; Yang, C.; Zhang, S.; Yang, H.; Yang, C.; Yuan, W.; Li, X.; Wang, X.; Zhang, Y.; et al. Edge device detection of tea leaves with one bud and two leaves based on ShuffleNetv2-YOLOv5-Lite-E. Agronomy 2023, 13, 577. [Google Scholar] [CrossRef]
- Yang, H.; Chen, L.; Ma, Z.; Chen, M.; Zhong, Y.; Deng, F.; Li, M. Computer vision-based high-quality tea automatic plucking robot using Delta parallel manipulator. Comput. Electron. Agric. 2021, 181, 105946. [Google Scholar] [CrossRef]
- Li, Y.; Wu, S.; He, L.; Tong, J.; Zhao, R.; Jia, J.; Chen, J.; Wu, C. Development and field evaluation of a robotic harvesting system for plucking high-quality tea. Comput. Electron. Agric. 2023, 206, 107659. [Google Scholar] [CrossRef]
- Tang, Y.; Han, W.; Hu, A.; Wang, W.; Anguo, H. Design and Experiment of Intelligentized Tea plucking Machine for Human Riding Based on Machine Vision. Nongye Jixie Xuebao/Trans. Chin. Soc. Agric. Mach. 2016, 47, 15–20. [Google Scholar]
- Zhang, L.; Zhang, H.; Chen, Y.; Dai, S.; Li, X.; Kenji, I.; Liu, Z.; Li, M. Real-time monitoring of optimum timing for harvesting fresh tea leaves based on machine vision. Int. J. Agric. Biol. Eng. 2019, 12, 6–9. [Google Scholar] [CrossRef]
- Karunasena, G.; Priyankara, H. Tea bud leaf identification by using machine learning and image processing techniques. Int. J. Sci. Eng. Res. 2020, 11, 624–628. [Google Scholar] [CrossRef]
- Zhang, L.; Zou, L.; Wu, C.; Jia, J.; Chen, J. Method of famous tea sprout identification and segmentation based on improved watershed algorithm. Comput. Electron. Agric. 2021, 184, 106108. [Google Scholar] [CrossRef]
- Chen, S.; Zou, X.; Zhou, X.; Xiang, Y.; Wu, M. Study on fusion clustering and improved yolov5 algorithm based on multiple occlusion of camellia oleifera fruit. Comput. Electron. Agric. 2023, 206, 107706. [Google Scholar] [CrossRef]
- Wu, W.; He, Z.; Li, J.; Chen, T.; Luo, Q.; Luo, Y.; Wu, W.; Zhang, Z. Instance Segmentation of Tea Garden Roads Based on an Improved YOLOv8n-seg Model. Agriculture 2024, 14, 1163. [Google Scholar] [CrossRef]
- Chen, Y.T.; Chen, S.F. Localizing plucking points of tea leaves using deep convolutional neural networks. Comput. Electron. Agric. 2020, 171, 105298. [Google Scholar] [CrossRef]
- Li, J.; Li, J.; Zhao, X.; Su, X.; Wu, W. Lightweight detection networks for tea bud on complex agricultural environment via improved YOLO v4. Comput. Electron. Agric. 2023, 211, 107955. [Google Scholar] [CrossRef]
- Meng, J.; Wang, Y.; Zhang, J.; Tong, S.; Chen, C.; Zhang, C.; An, Y.; Kang, F. Tea Bud and Picking Point Detection Based on Deep Learning. Forests 2023, 14, 1188. [Google Scholar] [CrossRef]
- Cao, M.; Fu, H.; Zhu, J.; Cai, C. Lightweight tea bud recognition network integrating GhostNet and YOLOv5. Math. Biosci. Eng. 2022, 19, 12897–12914. [Google Scholar] [CrossRef] [PubMed]
- Yan, L.; Wu, K.; Lin, J.; Xu, X.; Zhang, J.; Zhao, X.; Tayor, J.; Chen, D. Identification and picking point positioning of tender tea shoots based on MR3P-TS model. Front. Plant Sci. 2022, 13, 962391. [Google Scholar] [CrossRef]
- Li, Y.; Ma, R.; Zhang, R.; Cheng, Y.; Dong, C. A tea buds counting method based on YOLOV5 and Kalman filter tracking algorithm. Plant Phenomics 2023, 5, 0030. [Google Scholar] [CrossRef]
- Fang, M.R.; Lu, J.; Ruan, J.Y.; Bian, L.; Wu, C.Y.; Yao, Q. Tea Buds Detection Model Using Improved YOLOv4-tiny. J. Tea Sci. 2022, 42, 549–560. [Google Scholar]
- Yan, C.; Chen, Z.; Li, Z.; Liu, R.; Li, Y.; Xiao, H.; Lu, P.; Xie, B. Tea sprout picking point identification based on improved deepLabV3+. Agriculture 2022, 12, 1594. [Google Scholar] [CrossRef]
- Chen, T.; Li, H.; Chen, J.; Zeng, Z.; Han, C.; Wu, W. Detection network for multi-size and multi-target tea bud leaves in the field of view via improved YOLOv7. Comput. Electron. Agric. 2024, 218, 108700. [Google Scholar] [CrossRef]
- Qian, C.; Li, M.; Ren, Y. Tea sprouts segmentation via improved deep convolutional encoder-decoder network. IEICE Trans. Inf. Syst. 2020, 103, 476–479. [Google Scholar] [CrossRef]
- Lu, J.; Yang, Z.; Sun, Q.; Gao, Z.; Ma, W. A Machine Vision-Based Method for Tea Buds Segmentation and Picking Point Location Used on a Cloud Platform. Agronomy 2023, 13, 1537. [Google Scholar] [CrossRef]
- Wang, T.; Zhang, K.; Zhang, W.; Wang, R.; Wan, S.; Rao, Y.; Jiang, Z.; Gu, L. Tea picking point detection and location based on Mask-RCNN. Inf. Process. Agric. 2021, 10, 267–275. [Google Scholar] [CrossRef]
- Zhang, F.; Sun, H.; Xie, S.; Dong, C.; Li, Y.; Xu, Y.; Zhang, Z.; Chen, F. A tea bud segmentation, detection and picking point localization based on the MDY7-3PTB model. Front. Plant Sci. 2023, 14, 1199473. [Google Scholar] [CrossRef] [PubMed]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; Springer: Berlin, Germany, 2015; pp. 234–241. [Google Scholar] [CrossRef]
- Qian, L.; Wen, C.; Li, Y.; Hu, Z.; Zhou, X.; Xia, X.; Kim, S.H. Multi-scale context UNet-like network with redesigned skip connections for medical image segmentation. Comput. Methods Programs Biomed. 2024, 243, 107885. [Google Scholar] [CrossRef] [PubMed]
- Lin, G.; Wang, C.; Xu, Y.; Wang, M.; Zhang, Z.; Zhu, L. Real-time guava tree-part segmentation using fully convolutional network with channel and spatial attention. Front. Plant Sci. 2022, 13, 991487. [Google Scholar] [CrossRef]
- Li, Y.; Jing, B.; Feng, X.; Li, Z.; He, Y.; Wang, J.; Zhang, Y. nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance. arXiv 2023, arXiv:2309.16967. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- Iandola, F.; Moskewicz, M.; Karayev, S.; Girshick, R.; Darrell, T.; Keutzer, K. Densenet: Implementing efficient convnet descriptor pyramids. arXiv 2014, arXiv:1404.1869. [Google Scholar]
- Liang, Z.; Zhao, K.; Liang, G.; Li, S.; Wu, Y.; Zhou, Y. MAXFormer: Enhanced transformer for medical image segmentation with multi-attention and multi-scale features fusion. Knowl.-Based Syst. 2023, 280, 110987. [Google Scholar] [CrossRef]
- Li, X.; Sun, X.; Meng, Y.; Liang, J.; Wu, F.; Li, J. Dice loss for data-imbalanced NLP tasks. arXiv 2019, arXiv:1911.02855. [Google Scholar]
- Howard, A.; Sandler, M.; Chu, G.; Chen, L.C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1314–1324. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Liu, H.; Liu, F.; Fan, X.; Huang, D. Polarized self-attention: Towards high-quality pixel-wise regression. arXiv 2021, arXiv:2107.00782. [Google Scholar]
- Li, X.; Wang, W.; Hu, X.; Yang, J. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 510–519. [Google Scholar]
- Oktay, O.; Schlemper, J.; Folgoc, L.L.; Lee, M.; Heinrich, M.; Misawa, K.; Mori, K.; McDonagh, S.; Hammerla, N.Y.; Kainz, B.; et al. Attention u-net: Learning where to look for the pancreas. arXiv 2018, arXiv:1804.03999. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 801–818. [Google Scholar]
- Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2881–2890. [Google Scholar]
- Sun, K.; Xiao, B.; Liu, D.; Wang, J. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 5693–5703. [Google Scholar]
- Xie, E.; Wang, W.; Yu, Z.; Anandkumar, A.; Alvarez, J.M.; Luo, P. SegFormer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inf. Process. Syst. 2021, 34, 12077–12090. [Google Scholar]
Datasets | Image Samples | Label Category | ||
---|---|---|---|---|
Tea_Y | Tea_V | Tea_I | ||
Raw dataset | 1548 | 4829 | 2644 | 1238 |
Test set | 154 | 523 | 241 | 138 |
Training–validation dataset | 1394 | 4306 | 2403 | 1100 |
Augmented dataset | 2668 | 6459 | 5478 | 4612 |
Training set | 2402 | 5781 | 4944 | 4123 |
Validation set | 266 | 678 | 534 | 489 |
Operation | Input | Output | Kernel | Step | Head |
---|---|---|---|---|---|
Patch embedding | (512, 512, 3) | (128, 128, 32) | (7, 7) | 4 | |
(Transform block) × 2 | (128, 128, 32) | (16,384, 32) | 1 | ||
Feature conversion | (16,384, 32) | (128, 128, 32) | |||
Patch embedding | (128, 128, 32) | (64, 64, 64) | (3, 3) | 2 | |
(Transform block) × 2 | (64, 64, 64) | (4096, 64) | 2 | ||
Feature conversion | (4096, 64) | (64, 64, 64) | |||
Patch embedding | (64, 64, 64) | (32, 32, 160) | (3, 3) | 2 | |
(Transform block) × 2 | (32, 32, 160) | (1024, 160) | 5 | ||
Feature conversion | (1024, 160) | (32, 32, 160) | |||
Patch embedding | (32, 32, 160) | (16, 16, 256) | (3, 3) | 2 | |
(Transform block) × 2 | (16, 16, 256) | (256, 256) | 8 | ||
Feature conversion | (256, 256) | (16, 16, 256) |
Configuration | Parameters |
---|---|
Operating system | Windows 10 |
GPU | Nvidia RTX 3090 |
CPU | Intel I9-10920X |
Library | Pytorch 1.10.0; |
Accelerated environment | CUDA 11.3; CUDNN 8.2.1 |
Model | Backbone | IoU (%) | mIoU (%) | mPA (%) | |||
---|---|---|---|---|---|---|---|
Background | Tea_Y | Tea_V | Tea_I | ||||
Mobilenetv3–Unet | Mobilenetv3 | 96.75 | 31.29 | 30.44 | 44.37 | 50.71 | 55.55 |
Vgg–Unet | VGGNet | 99.04 | 77.83 | 67.73 | 80.04 | 81.16 | 85.36 |
Resnet–Unet | Resnet | 99.08 | 77.18 | 78.95 | 77.77 | 83.24 | 87.11 |
Transformer–Unet | Transformer | 99.41 | 80.49 | 80.71 | 80.95 | 85.39 | 90.83 |
Tag | Transformer–Unet | Path Features Aggregation | Polarized Attention | SK Attention | Gates Attention | CBAM Attention | mIoU (%) | mPA (%) |
---|---|---|---|---|---|---|---|---|
1 | ✓ | 85.39 | 90.83 | |||||
2 | ✓ | ✓ | 89.10 | 93.76 | ||||
3 | ✓ | ✓ | ✓ | 91.18 | 95.10 | |||
4 | ✓ | ✓ | ✓ | 90.36 | 94.52 | |||
5 | ✓ | ✓ | ✓ | 87.94 | 92.60 | |||
6 | ✓ | ✓ | 88.07 | 93.10 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, T.; Li, H.; Lv, J.; Chen, J.; Wu, W. Segmentation Network for Multi-Shape Tea Bud Leaves Based on Attention and Path Feature Aggregation. Agriculture 2024, 14, 1388. https://doi.org/10.3390/agriculture14081388
Chen T, Li H, Lv J, Chen J, Wu W. Segmentation Network for Multi-Shape Tea Bud Leaves Based on Attention and Path Feature Aggregation. Agriculture. 2024; 14(8):1388. https://doi.org/10.3390/agriculture14081388
Chicago/Turabian StyleChen, Tianci, Haoxin Li, Jinhong Lv, Jiazheng Chen, and Weibin Wu. 2024. "Segmentation Network for Multi-Shape Tea Bud Leaves Based on Attention and Path Feature Aggregation" Agriculture 14, no. 8: 1388. https://doi.org/10.3390/agriculture14081388