3.4.2. Improved YOLOv8 Network
In natural environments, chili pepper fruits are prone to occlusion, requiring models to extract image feature information from different scales and levels to identify objects of varying sizes and shapes. The traditional YOLOv8n model uses simplistic feature extraction and fusion methods in its backbone and head, making it challenging to fully utilize feature information across different scales. Therefore, to ensure that the recognition algorithm for automated chili pepper picking maintains high accuracy and real-time performance in complex, unstructured environments, this study optimizes and improves the YOLOv8n model in the following four aspects: First, to accelerate network detection speed, the backbone network is redesigned, replacing it with a more lightweight and improved HGNetV2. Second, to enhance the network’s ability to recognize occluded images, an effective SEAM attention mechanism for occluded images is introduced into the network’s detection head. Third, the redundancy in the C2f network structure is analyzed, and to reduce the model’s floating-point operations and computational load, some C2f structures are optimized. Lastly, CARAFE upsampling operators are introduced to replace traditional upsampling methods, further improving detection accuracy. This paper presents an improved YOLOv8 structure, as shown in
Figure 12, replacing the enhanced HGNetV2 as the backbone network, introducing the SEAM attention mechanism into the detection head, optimizing parts of the C2f structure, and using the CARAFE upsampling operator instead of traditional upsampling to effectively enhance model detection capability and speed while reducing missed detections and false alarms. When chili pepper images are inputted into this model, they undergo feature extraction through the HGNetV2 network first, aiming to achieve high detection accuracy and speed with minimal model parameters. Subsequently, feature fusion is conducted via the neck network, followed by the detection head integrated with the SEAM attention mechanism to further enhance the model’s ability to handle occluded chili pepper features, thereby improving the overall network’s expressive power.
This article focuses on improvements in four main aspects:
- (1)
Improved HGNetV2 Model
RT-DETR is the first real-time vision transformer (ViT) model introduced by Baidu, claiming superiority over the YOLO series in real-time detection. It utilizes two backbones, one being HGNet and the other ResNet. The HGNetV2 network is more lightweight and accurate. Therefore, this study attempts to replace the YOLOv8 backbone with HGNetV2 to meet the real-time and efficient requirements for pepper fruit picking.
The main idea of the HGNetV2 model is to extract features in a hierarchical manner, where complex patterns can be learned at different scales and abstraction levels, enhancing the network’s ability to process complex image data. This hierarchical and efficient processing is particularly beneficial for tasks like image classification. The overall structure of the HGNetV2 model backbone network is shown in
Figure 13:
Stem layer: This is the initial preprocessing layer of the network, typically consisting of convolutional layers to start extracting features from the raw input data.
HG block (hierarchical graph block): These blocks are the core components of the network, designed to process data hierarchically. Each HG block may handle data at different abstraction levels, allowing for the network to learn from low-level and high-level features.
LDS (learnable downsampling) layer: These layers located between HG blocks may perform downsampling operations, reducing the spatial dimensions of the feature maps, decreasing computational load, and potentially increasing the receptive field of subsequent layers.
GAP (global average pooling): Before final classification, the GAP layer reduces the spatial dimensions of the feature maps to a single vector per feature map, aiding in improving the network’s robustness to spatial transformations of input data.
Final convolution and fully connected (FC) layers: These include a 1 × 1 convolutional layer to combine features and fully connected layers to map these features to the required number of output classes.
The HGNetV2 is composed of multiple HG blocks, the structure of which is shown in
Figure 14. It includes several convolutional layers with different filter sizes to capture diverse features.
From the above HGNetV2 model structure, it can be seen that there is a large number of convolution operations in this network. Therefore, we can optimize HGNetV2 by using a more lightweight depthwise separable convolution, DWConv (depthwise convolution).
DWConv, a lightweight convolution operation proposed by Google in 2018, achieves comparable effects to traditional convolutions with fewer parameters and computational complexity. It is widely used in deep learning models on mobile and edge devices. The main difference between DWConv and traditional convolutions lies in the separation of its convolution kernel, which is divided into two parts corresponding to the depth and spatial dimensions. For each input channel, DWConv performs convolution with a kernel equal to the number of channels, and then aggregates the results along the depth dimension to obtain the final output. This operation can maintain output quality with a small number of convolution kernels, thus reducing parameters and computational complexity. Specifically, for a convolution kernel of size k, the parameter count is k ∗ k ∗ c_in ∗ c_out, where c_in represents the number of input channels and c_out represents the number of output channels. With DWConv, the parameter count is reduced to k ∗ k ∗ c_in since different channels share the same convolution kernel. This reduction in parameter count can improve the efficiency of deep learning models on mobile and edge devices.
The improved HGNetV2 model is presented in
Figure 15.
- (2)
SEAM Attention Mechanism
The SEAM (spatially enhanced attention module) is an attention mechanism proposed by YOLO-Face to improve object occlusion detection. Its goal is to compensate for the loss of response from occluded objects by enhancing the response of unoccluded objects. The SEAM aims to improve object occlusion detection by learning the relationship between occluded and unoccluded areas. The structure of the SEAM is as shown in
Figure 16: The first part of SEAM consists of depthwise separable convolution with residual connections. Depthwise separable convolution operates layer by layer, separating the convolution channel by channel. While depthwise separable convolution can learn the importance of different channels and reduce the number of parameters, it ignores the information relationship between channels. To compensate for this loss, the outputs of different depth convolutions are subsequently combined through pointwise (1 × 1) convolution. Then, a two-layer fully connected network is used to merge information from each channel, enabling the network to enhance connections between all channels. It is hoped that this model can compensate for the loss mentioned above in occlusion scenarios by learning the relationship between occluded and unoccluded pepper fruits in previous steps. Next, the output learned by the fully connected layers is processed by an exponential function to expand the value range from [0,1] to [1,e]. This exponential normalization provides a monotonic mapping relationship, making the result more tolerant of position errors. Finally, the output of the SEAM is used as attention multiplied by the original feature, allowing for the model to more effectively handle occlusions of pepper fruits and diminish background areas.
During the pepper harvesting process, it is common for fruits to be occluded. In order to improve the capability of extracting pepper target features under such circumstances, this study integrates the SEAM attention mechanism into the YOLOv8 detection head. The 3 × 3 convolutional layers in the YOLOv8 detection head are replaced with SEAMs, as illustrated in
Figure 17. With this improvement, the YOLOv8 detection head is capable of multi-scale pepper detection, placing greater emphasis on the pepper fruit regions in the image while weakening background areas. Additionally, the original 3 × 3 convolutional layers in the YOLOv8 detection head would increase the computational load of the module. By substituting the 3 × 3 convolutional layers with SEAMs, it is possible to further reduce the floating-point operations and computational load during the convolution process.
- (3)
Dilated Reparam Block
The UniRepLKNet model innovatively reconsiders the design of using multiple small convolutional kernels in traditional models by introducing an architecture based on large convolutional kernels. One significant innovation is the introduction of the dilated reparam block module, which reparameterizes convolutional layers with large kernels to enhance performance without increasing inference costs. Reparam block uses dilated small-kernel convolution layers to enhance undilated large-kernel layers. From a parameter perspective, these dilated layers are equivalent to undilated convolution layers with larger sparse kernels; thus, the entire block can effectively be seen as transforming into a large-kernel convolution layer. The structure of dilated reparam block is shown in
Figure 18.
YOLOv8’s C2f module employs numerous bottleneck structures, which extract more features but lead to excessive redundancy in channel information, resulting in increased computational overhead. This paper utilizes the dilated reparam block from UniRepLKNet, integrating it into the feature extraction C2f module through a class inheritance approach, replacing the bottleneck as the primary gradient flow branch. This reduces model parameters and computational costs. The new module is named C2f-DRB. The structure of the C2f-DRB module is shown in
Figure 19.
- (4)
CARAFE
The CARAFE (content-aware reassembly of features) upsampling operation has a large receptive field during reassembly, guiding the reassembly process based on input features, while keeping the operator itself lightweight.
CARAFE consists of two steps: In the first step, a reassembly kernel is predicted for each target position based on the content of the input. The second step involves reassembling features using the predicted kernels. Assuming a feature map X of size C × H × W and an upsampling rate σ (assumed to be an integer), CARAFE generates a new feature map X’ of size C × σH × σW. For any target position = (,) in the output X’, there exists a corresponding source position l = (i, j) in the input X, where i = , j = .
In the first step, the kernel prediction module Ψ predicts an appropriate kernel W
l’ for each position l’ based on the neighborhood of X
l, as shown in Equation (1). The reassembly step in the second step is formulated as shown in Equation (2), where Ψ is the content-aware reassembly module that reassembles the neighborhood of X
l with kernel W
l’.
Here, N(Xl, k) represents the k × k sub-region of X centered at position l, denoted as Xl’s neighborhood.