Object Detection in High-Resolution UAV Aerial Remote Sensing Images of Blueberry Canopy Fruits

Abstract

Blueberries, as one of the more economically rewarding fruits in the fruit industry, play a significant role in fruit detection during their growing season, which is crucial for orchard farmers’ later harvesting and yield prediction. Due to the small size and dense growth of blueberry fruits, manual detection is both time-consuming and labor-intensive. We found that there are few studies utilizing drones for blueberry fruit detection. By employing UAV remote sensing technology and deep learning techniques for detection, substantial human, material, and financial resources can be saved. Therefore, this study collected and constructed a UAV remote sensing target detection dataset for blueberry canopy fruits in a real blueberry orchard environment, which can be used for research on remote sensing target detection of blueberries. To improve the detection accuracy of blueberry fruits, we proposed the PAC3 module, which incorporates location information encoding during the feature extraction process, allowing it to focus on the location information of the targets and thereby reducing the chances of missing blueberry fruits. We adopted a fast convolutional structure instead of the traditional convolutional structure, reducing the model’s parameter count and computational complexity. We proposed the PF-YOLO model and conducted experimental comparisons with several excellent models, achieving improvements in mAP of 5.5%, 6.8%, 2.5%, 2.1%, 5.7%, 2.9%, 1.5%, and 3.4% compared to Yolov5s, Yolov5l, Yolov5s-p6, Yolov5l-p6, Tph-Yolov5, Yolov8n, Yolov8s, and Yolov9c, respectively. We also introduced a non-maximal suppression algorithm, Cluster-NMF, which accelerates inference speed through matrix parallel computation and merges multiple high-quality target detection frames to generate an optimal detection frame, enhancing the efficiency of blueberry canopy fruit detection without compromising inference speed.

1. Introduction

Blueberries are considered one of the fruits that are rich in a variety of nutrients. They are rich in antioxidants, which can help humans reduce the risk of chronic diseases such as cardiovascular disease and cancer [1]. Blueberries are also rich in vitamin C and vitamin E, which help protect the eyes from free radical damage and prevent eye diseases. In orchard cultivation, blueberries are one of the fruit crops with high economic returns for orchard farmers, and the implementation of blueberry fruit monitoring during cultivation is economically important for the estimation of blueberry yields or the prediction of blueberry fruit maturity. However, due to disturbing factors such as outdoor light variations, color similarity of blueberry fruits to their own fruit tree canopy, and imaging distance and occlusion in real outdoor orchard scenes, it is still extremely challenging to investigate reliable visual detection methods to identify fruits in blueberry fruit tree canopies. In recent years, due to the rapid development of computer deep learning technology, visual detection systems have combined with deep learning technology to achieve the detection of blueberry fruit as a new research direction.

It is well known that blueberry fruit is relatively small, making it challenging to detect small objects using target detection algorithms in machine learning. Yang F proposed a cluster detection (ClusterNet) network, which will be used to detect each scale of the extracted input image through the scale estimation sub-network (ScaleNet). The normalized clustering region of the extracted input image will be detected through the dedicated detection network (DetecNet), which effectively improves the detection efficiency of small objects [2]. Xiaolin F proposed a small-target detection method based on the improved Faster R-CNN algorithm in remote sensing images, which improves the detection of small targets by introducing the improved Anchor generation strategy and multi-scale feature fusion mechanism [3]. Gong Y proposed a new concept, ‘fusion factor’, to control the amount of information passed from the deeper to the shallower layers of the feature pyramid (FPN), so that the FPN can adapt to the small-target detection, and when the FPN is configured with the appropriate fusion factor, the network can obtain a significant performance improvement on the small-target detection dataset [4].

Small-target detection in UAV remote sensing images is also an important part of the research on target detection. Zhu X explored the prediction potential of the self-attention mechanism based on the YOLOv5 algorithm by adding more prediction heads to detect objects at different scales and by replacing the original prediction heads with those of the Transformer structure. The prediction potential of the self-attention mechanism was explored on the small target dataset. Experimental results conducted on VisDron2021 showed a 7% improvement in accuracy over the baseline model YOLOv5 [5]. Li C proposed a density map-guided target detection network (DMNet) that determines whether there is an object in a region based on the change in pixel intensity distribution of an image, and achieved an excellent performance [6]. Lu W constructed a hybrid backbone neural network model based on the combination of Convolutional Neural Network (CNN) and Transformer (ViT) for for UAV remote sensing image detection, and applied the Masked Image Modeling (MIM) method to the UAV remote sensing dataset with a volume lower than that of the mainstream natural scene dataset, to generate a pre-trained model for the proposed method to further improve the target detection accuracy of UAV imagery [7]. Liao J et al. proposed an unsupervised cluster-guided detection framework (UCGNet) in order to solve the problem of compressing the resolution of high-resolution images prior to training by focusing on the densely distributed regions of objects [8]. Buters T used low-altitude UAV imagery and an automated object-based image analysis software that detects and counts target seeds and seedlings in non-target grass substrates from a variety of reflective localized restoration substrates [9]. Melnychenko O proposed a new approach combining Artificial Intelligence (AI), Deep Learning (DL), and Unmanned Aerial Vehicles (UAVs) to demonstrate superior real-time capabilities in fruit detection and counting using a combination of AI techniques and multi-UAV systems [10]. Shiu Y S used high-resolution UAVs with a ground sampling distance (GSD) of 3 cm for (UAV) images for pineapple plant body detection using Faster R-CNN to locate and count the number of ripe fruits based on input image tiles of a size of 256 × 256 pixels [11]. Khan S et al. implemented and evaluated high-resolution UAV images captured on two different target fields to study and develop a deep learning system for identifying weeds and crops in agricultural field weeds and crops [12]. Gallo I combined the YOLOv7 target detection algorithm with an unmanned aircraft system [13]. Wittstruck L developed an image processing method for Unmanned Aerial Vehicles (UAVs) with high-resolution RGB data and applied it to the fields of a pumpkin farmer in Hokkaido, northwestern Germany [14]. Gai R presented a blueberry recognition algorithm TL-YOLOv8 based on the YOLOv8 algorithm. Feature extraction during training is enhanced by introducing an improved MPCA (Multiplexed Coordinated Attention) in the last layer of the backbone network. By replacing the C2f module with OREPA (Online Convolutional Reparameterisation) module, not only is the training accelerated, but the representation is also enhanced. The fruit occlusion problem was effectively solved [15]. Xiao F proposed a lightweight detection method based on the improved YOLOv5 algorithm. A lightweight deep convolutional neural network was implemented using the ShuffleNet module, and the effectiveness of the method was evaluated using the blueberry fruit dataset; it effectively detected blueberry fruits in an orchard environment and identified their ripening stages [16].

Currently, the growth monitoring process of blueberries is still mainly carried out by on-site manual inspection, which not only requires professional knowledge and experience, but also requires the use of a large number of human resources, so advanced optical sensors assembled by UAVs to take remotely sensed images have been gradually applied in the detection of the growth of fruit trees hanging fruit, the detection of crop diseases, yield prediction of fruit trees fruit, and fruit ripeness detection. Lu C et al. proposed a practical workflow based on YOLOv5 and UAV images to detect maize plant seedlings [17]. Li W designed a set of processes combining improved U-net and VGG-19 nets to solve the problem of difficulty in obtaining grape leaves for identification in UAV images in complex environments [18]. Bao W proposed an unmanned remote sensing method based on the DDMA-YOLO method for the effective detection of tea leaf blight [19]. Wang C constructed a YOLO-BLBE model combined with an innovative I-MSRCR method to accurately identify blueberry fruits with a different ripeness [20]. Yang W designed a blueberry recognition model based on the improved YOLOv5, and designed a new attention module NCBAM to improve the backbone network’s ability to extract blueberry features [21]. Huang H et al. improved the accuracy of detecting citrus fruit images captured by UAVs by extending the target detection algorithm [22]. Yong Z X proposed a method for detecting individual grapefruit fruit trees by combining deep learning with UAV remote sensing [23].

The You Only Look Once (YOLO) series, a widely used and excellent single-stage target detection method (the better ones in recent years are YOLOv5, YOLOv7, and YOLOv8 [24,25]), which can obtain the category and position of the object through the single-stage detection method. Compared with two-stage target detection algorithms such as RCNN [26,27,28], single-stage target detection algorithms can reduce a large amount of computational time and cost, and are more suitable for UAVs to quickly detect and identify blueberry fruits in the canopy, and the YOLO series of algorithms has been widely used in remote sensing and agriculture.

The detection and counting of fruit tree canopies is important for orchard management, yield estimation, and phenotyping; however, so far there have been relatively few studies using UAV remote sensing imagery and deep learning methods to monitor blueberry canopy fruits in orchards. In this study, UAV remote sensing images are used to detect blueberry fruit in the canopy in the orchard, but three problems will be encountered: the first problem is that the UAV images cover a wide area, the image as a whole has a high resolution, and most of the pixel space is allocated to useless background information, and there is very little pixel space allocated to the small objects of the blueberry fruits, which leads to the difficulty of the neural network in accurately detecting the small target fruit. The second problem is that the blueberry fruits in the UAV image are densely distributed and similar in shape, which the neural network uses to produce false detections. The third problem is that when the YOLO algorithm is used to train UAV remote sensing images, many target prior frames will be generated in advance in the images, and the use of non-greatly suppressed NMS [29] to filter the most optimal prior frame inference is slow, and the use of ClusterNMS [30] will improve the inference speed, but it is difficult to select the optimal prior frame.

To address the above problems, this research paper proposes a PF-YOLO-based UAV remote sensing monitoring method for blueberry canopy fruits, which can monitor blueberry canopy fruits a timely in and accurate manner. This research made the following contributions:

1. High-resolution remote sensing images for detecting blueberry crown fruits were collected based on UAV photography in real blueberry orchards, and a high-resolution remote sensing blueberry crown fruit target detection dataset was constructed.

2. Combined with the position information coding structure fused to the target detection feature extraction part, the PAC3 structure is proposed, which pays more attention to the spatial position information features of the detected targets, in order to solve the problem of the missed detection of small targets in the smoke sensing images of UAVs.

3. Using fast convolution instead of conventional convolution so as to reduce the number of parameters and computational complexity of the neural network model, with the YOLO framework as the basis, it fused the PAC3 structure and fast-convolution proposed UAV blueberry canopy fruit remote sensing image target detection model PF-YOLO.

4. The cluster-NMF algorithm is proposed to speed up the inference stage of blueberry canopy fruit cluster detection and improve the detection efficiency by optimizing the redundant target detection frame screening process.

2. Experimental Equipment and Materials

2.1. Data Source

The image data used in this study were collected and photographed on the afternoon of 13 June 2023, in Shimen Town, Tongxiang City, Zhejiang Province, China, under clear weather conditions with wind speeds of 2 to 3 levels. A DJI PHANTOM4 PRO drone equipped with a 20-megapixel visible light camera was used to capture the images (Shenzhen Dajiang Innovation Technology Co., Ltd., Shenzhen, China). The camera model is FC6310, with a focal length of 8 mm; an aperture range from f/2.8 to f/11; and features autofocus. In motion mode, the drone has a maximum ascent speed of 6 m/s, a maximum descent speed of 4 m/s, a maximum horizontal flight speed of 72 km/h, and a maximum flight altitude of 6000 m, with a maximum endurance time of 30 min. The captured images are set to a resolution of 5472 × 3078 in JPEG format. Due to the complexity of the geographic environment, during the image acquisition period, the flight speed and direction were manually controlled, with the camera positioned vertically at a 90-degree angle to the ground. The drone is equipped with a GPS positioning function, allowing it to record the latitude, longitude, and altitude of each image in detail.

In this study, the flight height of the UAV captured is about 5 m from the ground, and because the resolution of the images captured by the UAV is too large, applying them directly to neural network training requires great computational resources, and the same problem of insufficient samples of the training images will be faced. In this study, 10 images out of 96 collected are used as a test inference, and 86 high-resolution images of a 5472 × 3078 resolution are cut into 1032 images of a 1536 × 1536 resolution, of which 840 are used as the training set and 192 are used as the validation set. To ensure consistency in sample labeling, all annotations during the data labeling process were carried out by a single person. For an image label annotation tool, LabelImg software 1.8.6. was used; in this experiment, the design of the label was based on the size of the cluster set of the number of blueberry fruits, and the label category was divided into the three categories large_cluster, middle_cluster, and small_cluster, as shown in Figure 1a, which shows a UAV image of the blueberry crown fruits with a resolution of 5472 × 3078 pixels, where the collected UAV images cover a relatively wide space and are too large in resolution. Using the originally captured data directly to train the neural network model does not consume a large amount of computational resources and takes a very long time, and the number of sample images used for training in this way is insufficient. b shows the cut-up drone blueberry canopy fruit image with a resolution of 1536 × 1536, c shows the original image of the trained model recognizing the blueberry fruit, and d shows the cut-up trained model image of recognizing blueberry fruit.

2.2. Data Processing

In order to increase the diversity of the training data, we used Mosaic data augmentation [31], where four randomly selected samples were spliced together by random scaling, random cropping, random alignment, and adjusting the brightness. This improves the richness of the dataset and increases the robustness of the network. This is shown in Figure 2 below.

3. Experiments and Methods

3.1. Location Information Encoding

The SE structure compresses global spatial information into a single channel descriptor, using global average pooling to transform the feature map containing global information into a one-dimensional feature vector, alleviating inter-channel information dependency. The CBAM structure effectively integrates the spatial and channel dimensions of the feature map, learning adaptive channel weights to focus on specific channel location information.

SE only considers the internal channel information and ignores the importance of location information, but in target detection, the spatial location information of the detected target object is extremely important. Although CBAM tries to introduce the location information by global pooling on the channels, this way it can only capture the local information, and cannot obtain the information of the long-distance dependence, because the input image feature maps of each location contain information about a local region of the input image after the operation of the convolutional layer; as such, CBAM is used as a weighting coefficient by taking the maximum and average of multiple channels for each region, so this weighting knowledge takes into account the information of the local region.

Standard two-dimensional convolution cannot associate channel relationships with input image features, and constructing links between channels can improve the sensitivity of neural network models for classification decisions to information channels. The global pooling operation is usually used to encode the global spatial information channel note, which can make up for the shortcomings of the convolution operation, but this will directly compress the global spatial information into the channel descriptor, which will lose the position information, and have a great impact on capturing the spatial structure in the visual task. In the positional information encoding mechanism, the channel and remote dependencies of the input features are encoded in two steps, and then positional information is embedded and positional attention is generated to produce a feature map with accurate positional information. This is shown in Figure 3 below:

The location information encoding uses three 1D feature encoding operations to realize the spatially captured remote interaction dependence of the attention block with the precise location information. Let the input be $X$. Using the horizontal spatial range pooling kernel $(X,1)$ and the vertical spatial range pooling kernel $(1,Y)$, each channel within the horizontal and vertical coordinates of $X$ is encoded, respectively. So the output of the cth channel with height h is represented as follows:

$$z_c^h\left(h\right)={\displaystyle \frac{1}{W}}\sum _{0\le i<W}{x}_c\left(h,i\right)$$

(1)

The output of the $\mathrm{c}$th channel with width $w$ is denoted as follows:

$$z_c^w\left(w\right)={\displaystyle \frac{1}{H}}\sum _{0\le j<H}{x}_c\left(j,w\right)$$

(2)

These two transformations aggregate features in two spatial directions to obtain a pair of direction-aware feature maps, which allows the attention block to capture long-range dependencies in one spatial direction and retains precise location information in the other, helping the neural network to more accurately localize the location of the feature region of the object of interest in the input image.

After the aggregated feature maps are generated by the above two processes, respectively, we stitch them together and feed them into the shared 1 × 1 convolution $F_1$ for feature integration to obtain $\mathrm{f}$:

$$\mathrm{f}=\delta \left(F_1\left(\left[{\mathrm{z}}^h,{\mathrm{z}}^w\right]\right)\right)$$

(3)

$\delta$ denotes a nonlinear activation, $\mathrm{f}\in {\mathbb{R}}^{C/r\times (H+W)}$ is an intermediate feature map which encodes spatial information in the horizontal and vertical directions, and $\mathrm{r}$ is a hyperparameter controlling the block size. Then, $\mathrm{f}$ is split along the spatial dimension to obtain two independent tensors, ${\mathrm{f}}^h\in {\mathbb{R}}^{C/r\times H}$, ${\mathrm{f}}^w\in {\mathbb{R}}^{C/r\times W}$, which are later obtained by 1 × 1 convolutional feature integration:

$$\begin{array}{c}{\mathrm{g}}^h=\sigma \left(F_h\left({\mathrm{f}}^h\right)\right)\\ {\mathrm{g}}^w=\sigma \left(F_w\left({\mathrm{f}}^w\right)\right)\end{array}$$

(4)

Attention is weighted in both directions, and the final output positional attention y is denoted as follows:

$$y_c\left(i,j\right)=x_c\left(i,j\right)\times g_c^h\left(i\right)\times g_c^w\left(j\right)$$

(5)

Positional information coding focuses on the spatial information of the input image, aggregating feature extraction in both horizontal and vertical directions. This allows for the precise localization of targets located in different spatial directions. During training, the neural network model can more accurately concentrate on the features of the target’s location, enabling it to learn more about the target’s characteristics in that area.

3.2. PAC3 Structure

In the UAV remote sensing images, the number of blueberry canopy fruits is high, the location is similar, and the shape and color are closer, resulting in a dense distribution of blueberry fruits. It is very easy for a neural network facing this dense distribution problem to produce leakage and misdetection. The C3 module in yolov5 is the main module for learning the residual features, which is used for feature extraction from the input image to capture the feature information of different scales and semantic features, including edges, textures, shapes, and so on. Its structure is divided into two branches: one uses multiple bottleneck stacks and three standard convolutional layers as specified above, and the other goes through only one basic convolutional module, and finally the two branches are subjected to concat operation.

However, the standard convolution and standard pooling operations used in the C3 module do not capture the positional features of the target to be detected in the input image, and incorporate a large amount of useless background information, and these redundant features also greatly increase the difficulty of detection. Therefore, we integrate the positional attention module into C3, and after channel feature splicing, the feature map with captured edges, textures, shapes, and other feature information is input into the positional information encoding, which pays more attention to the exact location of the blueberry fruit in the canopy by integrating the horizontal and vertical spatial information features, and reduces the influence of redundant background information. As shown in Figure 4 below:

3.3. Fast Convolutional Structure

FLOPs represent the total number of floating-point operations needed during model execution, while MAC indicates the total number of multiply accumulated operations. FLOPs provide an overall estimate of the model’s computational load, whereas MAC focuses more on describing the specific operational requirements within the model. These two metrics can be used to evaluate and compare the efficiency and performance of different models. We know that when forward propagation is performed in a neural network model, operations such as convolution, pooling, BatchNorm, Relu, upsampling, etc. are performed. All of these operations have corresponding computational power consumption, and convolution corresponds to the highest percentage of computational power consumption. Therefore, we introduce a fast convolution which extracts image spatial features more efficiently by simultaneously reducing redundant computations and memory accesses. It is shown in Figure 5:

Instead of performing a convolution operation on all the pass tubes on the input channel, it performs a convolution operation on a specific portion of the paraphernalia to extract the spatial features of the image, while the rest of the channel remains unchanged. The channel/r channel is considered as a representation of the whole feature map and is computed in the memory access process. The FLOPs and MAC (Memory Access Cost) at this time are as follows:

$$FLOPs=h\times w\times k^2\times {\left({\displaystyle \frac{c}{r}}\right)}^2$$

(6)

$$MAC=h\times w\times 2(c/r)+k^2\times {\left({\displaystyle \frac{c}{r}}\right)}^2\approx h\times w\times 2\left({\displaystyle \frac{c}{r}}\right)$$

(7)

FLOPs and MAC for regular convolution are as follows:

$$FLOPs=h\times w\times k^2\times c^2$$

(8)

$$MAC=h\times w\times 2c+k^2\times (c/r{)}^2\approx h\times w\times 2c$$

(9)

When $r$ = 1/4, the FLOPs for fast convolution are only 1/16 of those for regular convolution, and the memory accesses for fast convolution are only 1/4 of those for regular convolution.

3.4. ClusterNMF (Non-Maximal Fusion)

In the target detection task, the single-stage target detection neural network model based on the YOLO framework generates a large number of prediction frames in the pre-task period; however, the vast majority of prediction frames are redundant frames that do not correctly contain the target detector, so an algorithm is needed to filter them, and NMS (non-maximal suppression) is an algorithm used for the target detection to remove highly redundant prediction frames in the reasoning stage. The process of prediction frame generation generates a large number of predictions at the same target location, and these candidate frames may overlap with each other; at this time we need to find the best target prediction frames using non-maximal suppression to eliminate redundant prediction bounding boxes.

The steps of the NMS are shown in Figure 6: (1) first, initially set a confidence threshold, and initially screen to remove the bounding boxes that are lower than this confidence threshold; (2) sort the initially screened prediction boxes according to the confidence scores in ascending order; (3) find out the boxes with the highest confidence scores in the current category and save them; (4) delete all the boxes that are higher than the IoU threshold retained in step (2); and (5) repeat steps (3) and (4) until all boxes are processed.

However, the NMS algorithm has some drawbacks: (1) the need to manually set the threshold, as if the threshold is set too large it will cause false detection, or if too small it will not achieve the desired effect (recognition accuracy decreases); (2) if the nms is below the threshold, the box score is set to 0, and is directly removed—this practice is too hard; (3) for the cycle step, parallel processing for the GPU is difficult, and so the operational efficiency is low, resulting in a slow model inference speed; (4) evaluated by predicting the IoU of the box, the IoU practice has different effects on the target box scale and distance.

To address the above problem, Bodla N proposed the Soft-NMS algorithm, which uses a reduced detection score to suppress the confidence score of each box through a penalized Gaussian function, for which the confidence score is not high; after the suppression of the penalized function, its confidence score will be lower than the threshold and thus will be removed, and for those boxes with a high confidence score, even after the suppression of the penalized function, its confidence is still higher than the threshold and is retained, thus reducing the leakage rate due to the overlapping of multiple detected targets and the strong removal of some of these boxes. If its function is suppressed, its confidence score is still higher than the threshold and is retained, thus reducing the number of boxes that are forcefully removed because multiple detected targets overlap together, i.e., reducing the leakage rate [32]. Ning C proposed Weighted NMS, where the confidence scores of the prediction boxes are weighted and normalized according to the confidence scores of the prediction boxes to obtain the new bounding box, and the bounding box is used as the final predicted bounding box, and then the other boxes are eliminated [33]. Zheng Z proposed DIoU-NMS to filter the predicted bounding boxes by combining the area overlap region and the centroid distance of the two boxes through the calculation of DIoU [34].

The above NMS has a common bottleneck, which is the computation of IoU and sequential iterative suppression. If there are n detection frames in an image, according to the sequential processing, a certain frame M and the other frames have to compute the IoU at least once, at most n−1 times. Together with the sequential iterative suppression, the above NMS algorithm calculates the IoU at least n−1 times, up to $(n-1)+(n-2)+\cdots +1={\displaystyle \frac{1}{2}}{n}^2-{\displaystyle \frac{n}{2}}$ times, so if you want to speed up the process of NMS, you should parallelize the calculation of IoU. As such, Zheng Z et al. proposed Cluster-NMS, which utilizes the GPU matrix operations of pytorch for NMS, but at the same time ensures the performance remains the same as Traditional NMS [30].

However, Cluster-NMS only speeds up the inference by guaranteeing the same accuracy as Traditional NMS, but does not further improve the accuracy of inference. For such a problem, we propose Cluster-NMF (Cluster Non-maximum fusion). Firstly, we parallelize the computation of IoUs in the same matrix format, and only need to compute the IoUs of the set of detection frames $B=\{B_i{\}}_{i=1 \mathrm{to} \mathrm{n}}$ with itself, and the set of detection frames $B$ will be firstly sorted by the descending order of the confidence scores, i.e., the score of $B_1$ is the highest, and the score of $B_n$ is the lowest. The following matrix is obtained; $X_{ij}$ represents the IoUs of detection frames $B_i$ and detection frames $B_j$.

$$X=IoU\left(B,B\right)=\left(\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{nn}\end{array}\right),x_{ij}=IoU\left(B_i,B_j\right)$$

(10)

Due to the heap symmetry of the IoU matrix $X$., $IoU(B_i,B_j)=IoU(B_j,B_i)$, and the fact that a detection frame is meaningless with respect to calculating the IoUs on its own, the $X$ is then up-triangulated to obtain an up-triangulated matrix $X^{\prime }$ with elements of 0 on the diagonal and below.

$$X=\left(\begin{array}{ccccc}{x}_{11}& x_{12}& x_{13}& \cdots & x_{1n}\\ x_{21}& x_{22}& x_{23}& \cdots & x_{2n}\\ x_{31}& x_{32}& x_{33}& \cdots & x_{3n}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& x_{n3}& \cdots & x_{nn}\end{array}\right)\stackrel{upper}{\underset{triangulation}{\to }}{X}^{\mathrm{\prime }}=\left(\begin{array}{ccccc}0& x_{12}& x_{13}& \cdots & x_{1n}\\ 0& 0& x_{23}& \cdots & x_{2n}\\ 0& 0& 0& \cdots & x_{3n}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& \cdots & 0\end{array}\right)$$

(11)

Then, take the maximum value of $X^{\prime }$ by column and binarize it according to the NMS threshold to obtain a one-dimensional tensor $b=[b_1,b_2,\cdots ,b_n]$; $b_i$ represents the maximum value of the element on the $i$th column of the new matrix $X$. Then, binarize the tensor $b$ according to the threshold set by the NMS, and assign a value of 0 to those greater than the threshold, and of 1 for those smaller than the threshold. An element of $b$ with a value of 0 denotes a suppressed prediction box, and elements with a value of 1 indicate retained boxes. The b is then expanded into a diagonal matrix $E$. The matrix $E$ is then left-multiplied by the IoU matrix $X$. The new tensor $b^1$ is obtained by again taking the maximum value by column and binarizing it according to the NMS threshold.

$$\begin{array}{ll}{b}^1=& 1 \hspace{0.33em} 0 \hspace{0.33em}1 0 \hspace{0.33em}0\\ & \uparrow Threshold-binarization\\ b=& 0 0.54 0.32 0.68 0.88\\ & \uparrow Column-maximum\\ X^{\prime }=& \left(\begin{array}{ccccc}0& 0.54& 0.11& 0.40& 0.88\\ & 0& 0.32& 0.68& 0.10\\ & & 0& 0.45& 0.25\\ & & & 0& 0.28\\ & & & & 0\end{array}\right)\end{array}$$

(12)

Repeat the above steps until after two iterations $b^i=b^j$ is found, and the final IoU matrix $C^{final}$ is obtained at this point to determine which detection frames should be retained and which detection frames should be suppressed. Then, for each column of the matrix $C^{final}$, according to a new fusion IoU threshold, determine which detection boxes are retained for fusion, determine the need to fuse the detection box through the classification confidence value and the IoU of similar objects of all the border coordinates of the weighted average to update the coordinates of the new prediction box.

$$Scores=f\left(C_{i,j}^{final},newthres\right)$$

(13)

$$Weight{s}_{ij}=C_{ij}^{final}\times Score{s}_i$$

(14)

$$Box{|}_{i,j=1}^n={\displaystyle \frac{\sum _{i=1,j}^{n}Weight{s}_{ij}\times B_{ij}}{\sum _{i=1,j}^{n}Weight{s}_{ij}}}$$

(15)

$$Box=Box{|}_{i,j}^n+{Box}^{final}$$

(16)

The pseudo-code of the Cluster-NMF algorithm is shown in Algorithm 1; $C^{final}$ denotes the IoU matrix of the retained and suppressed detection frames screened by Cluster-NMF; $newthres$ is the new fusion IoU threshold, greater than which it should be fused, less than which it is not fused; $Scores$ is the new list of scores for the test box obtained after the function f determines whether the test box should be fused or not; $B_{ij}$ denotes the coordinate matrix of the retained detection frame; $Weights$ is the weight matrix; $Weight{s}_{ij}$ denotes the score weight of the ${Box}_{ij}$ box; $C_{ij}^{final}$ denotes the IoU of the $\mathrm{i}$-th box and $\mathrm{j}$-th box; $Score{s}_i$ is the score of the $\mathrm{i}$-th box; ${Box}^{final}$ denotes the prediction frames that were retained before fusion; $Box{|}_{i,j=1}^n$ indicates the prediction frame that was retained after fusion; and $Box$ indicates the prediction box that is ultimately retained.

Algorithm 1: Cluster-NMF

Input: N detected boxes $B=[B_1,B_2,\cdots ,B_n{]}^T$ with non-ascending sorting by classification score-descending order. Output: $Box=\{b_i{\}}_{1\times N},b_i\in \{0,1\}$ encodes final detection result, where 1 denotes reservation and 0 denotes supression. Initialize $T=N,t=1,t^{\ast }=T$$\mathrm{and} t^0=1$. Compute IoU matrix $\mathit{X}=\{x_{ij}{\}}_{N\times N} \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}{ x}_{ij}=IoU\left(b^i,b^j\right)$. $X^{\prime }=\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{u}\left(\mathit{X}\right)$ Upper traingular matrix with $x_i=0,{\forall }_i$ While $t\le T$ do         $\begin{array}{c}{\mathit{E}}^t=\mathrm{diag}\left(b^{t-1}\right)\hfill \\ C^t={\mathit{E}}^t\times X\hfill \\ \mathrm{g}\leftarrow \max C^t\hfill \\ b^t\leftarrow \mathrm{find}\{g<thres\}.\mathrm{i}\mathrm{f} g_i<thres, b_i=1 \mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e} b_i=0\hfill \\ \mathbf{i}\mathbf{f}:\hfill \\ t^{\ast }=t,\mathrm{break}\hfill \\ t=t+1\hfill \end{array}$ end if end While $C^{final}=C^t$ $\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s}=f(C_{ij}^{final},newthres)$ ${\mathrm{W}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{t}\mathrm{h}\mathrm{s}}_{ij}=C_{ij}^{final}\times Score{s}_i$ $\begin{array}{c}\mathrm{Box}{|}_{i,j=1}^n={\displaystyle \frac{\sum _{i=1,j}^{n}Weight{s}_{ij}\times B_{ij}}{\sum _{i=1,j}^{n}Weight{s}_{ij}}}\\ \mathrm{Box}=\mathrm{Box}{|}_{i,j}^n+{\mathrm{B}\mathrm{o}\mathrm{x}}^{final}\end{array}$ Return Box

3.5. PF-YOLO Network Model

The YOLO framework is a popular method for single-stage target detection based on deep learning, which has a high detection accuracy and fast inference speed. And to realize the neural network model deployed in UAV embedded devices for real-time monitoring, the weight file needed should be small, so considering the accuracy and efficiency of the detection model, this study combines the above improved modules and algorithms with the YOLO framework, and proposes a PF-YOLO neural network model for detecting the blueberry canopy fruits using remote sensing from UAVs.

As shown in Figure 7 below, the PF-YOLO network model is divided into three parts: Backbone, Neck, and Head. In Backbone, the PAC3 module is used to improve the ability of extracting the exact positional features of the blueberry crown fruit target from the input image, in order to reduce the leakage of the blueberry crown fruit detection problem. In the Neck, the PF-YOLO network model is used by using the fast convolution in order to reduce the number of parameters and speed up the memory access speed of the overall model, and to maintain the original feature extraction ability number and speed up memory access, and maintain the original feature extraction capability. In the Head, four scales of detection heads are applied to the feature maps of blueberry crown fruit clusters of different sizes to generate information of different categories, coordinates, and confidence levels.

4. Experimental Results and Analysis

4.1. Experimental Configuration and Evalution Index

In this study, the PyTorch deep learning framework was used to implement and train the PF-YOLO neural network model. The configuration of the experimental hardware and software environments is shown in the following

Table 1. Experimental hardware and software environment configuration.

Platform	Configuration
Operating system	Windows 10 professional 64 bits
CPU	13th Gen Intel(R) Core(TM) i9-13900K
GPU	NVIDIA RTX A5000
GPU accelerator	CUDA 11.6 & cuDNN 8.3.2
Deep learning frame	Pytorch 1.13.0
Compilers	Pycharm & Anaconda
Scripting language	Python 3.9

. The training parameters of PF-YOLO during the training process are configured, as shown in

Table 2. Configuration of training parameters during training with PF-YOLO.

Parameters	Configuration
Input size	1536 × 1536 × 3
Optimization algorithm	Adam
Batch size	4
Training epochs	160
Initiation learning rate	0.001
Momentum	0.9
Weight decay	0.1

.

The performance evaluation metrics for a model and method in a target detection task mainly include the detection accuracy, detection completeness, average precision (AP), category-wide average accuracy (mAP), number of parameters, and computation. The detection rate is the proportion of all correctly predicted targets in the prediction frame, or the proportion of correctly predicted targets in all true detection frames, the AP is the average precision of a single target category, and the mAP is used to measure the performance of target detection models and algorithms, and is obtained by weighting the average correctness rate of all categories of detection. The number of parameters and the amount of computation is used to evaluate the complexity of the model and algorithm, combined with the above five indicators. Here, the higher the detection rate and average accuracy, the lower the number of parameters and the amount of computation, which means that the performance of a model is better. The definitions of the formulas is as follows:

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(17)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(18)

$$AP={\int }_0^{1}P\left(R\right)dR$$

(19)

$$mAP={\displaystyle \frac{\sum _{i=1}^{n}A{P}_i}{n}}$$

(20)

In small-object detection, mAP provides a comprehensive evaluation of the model’s performance across different thresholds, making it suitable for assessing the overall detection capability for small targets. Precision emphasizes the accuracy of the model, ensuring that detected objects are correct. Recall focuses on the model’s ability to capture all relevant instances, ensuring that small objects are not missed. Average Precision combines both Precision and Recall, effectively capturing the model’s performance across different classes and IoU thresholds. Therefore, these metrics collectively reflect the model’s performance in small-object detection tasks.

4.2. Comparing Different Feature Extraction Modules

In this study, the PAC3 structure is proposed to extract image target features while paying more attention to the location information of the target, which is used to realize the training and detection of blueberry fruit clusters in blueberry canopies from remote sensing images of blueberry canopies captured by unmanned aerial vehicles (UAVs). Comparing a variety of excellent feature extraction modules, PAC3-Yolo indicated that with the use of the PAC3 module, it pays more attention to the exact location of the blueberry fruits in the canopy by integrating the spatial information features in the horizontal and vertical directions, and reduces the omission and false detection of the blueberry fruits; as shown in

Table 3. Comparison result data of different feature extraction modules.

Yolo	Rrecision (%)	Recall (%)	[email protected] (%)	[email protected]–0.95
+C3	51.5	58.4	51.9	22.5
+SE	51.6	56.2	50.5	22.5
+CBAM	46.6	66.9	52.1	22.6
+Ghost	57.2	54.5	52.3	21.6
+PAC3	51.4	59.1	54.0	23.7

below, in the case of the same backbone network, the PAC3 module has a performance 2.1% higher than the C3 module on the mAP and 3.5% higher than the SE module, 1.9% higher than the CBAM module, and 1.7% higher than the Ghost module.

4.3. Comparative Experiment of Different Models

In this study, we proposed the PF-Yolo target detection model using the YOLO model as a framework. In order to verify the effectiveness of our proposed model PF-Yolo in blueberry canopy fruit detection, we compared our model with some representative state-of-the-art models, as shown in

Table 4. Comparison of data from experimental results of different models.

Models	Precision (%)	Recall (%)	[email protected] (%)	Parameters (M)	Flops (G)
Yolov5s	45.5	57.6	48.9	7.02	16.0
Yolov5l	38.8	67.9	47.6	46.12	107.7
Yolov5s-p6	51.5	58.4	51.9	12.36	16.9
Yolov5l-p6	52.3	60.2	52.3	76.15	110.6
Tph-Yolov5	46.8	59.8	48.7	7.43	36.8
Yolov8n	52.8	55.8	51.5	3.01	8.9
Yolov8s	51.6	57.5	52.9	11.12	28.8
Yolov9c	53.8	57.5	51.0	25.32	102.3
PAC3-Yolo	51.4	59.1	54.0	12.41	16.9
PF-Yolo	54.4	58.9	54.4	11.50	15.5

. The PAC3-Yolo is a model that uses the PAC3 feature extraction module we proposed. Compared with Yolov5s, Yolov5l, Yolov5s-p6, Yolov5l-p6, Tph-Yolov5, Yolov8n, Yolov8s, and Yolov9c, the maps were improved by 5.1%, 6.4%, 2.1%, 1.7%, 5.3%, 2.5%, 1.1%, and 3.0% respectively. PF-Yolo represents the improvement in the PAC3-Yolo in order to further reduce the number of parameters and computation of the model, while improving the detection accuracy of blueberry canopy fruit clusters, which is achieved by performing spatial feature extraction with channel-specific convolutional operations. Compared with the PAC3-Yolode map, the accuracy of the model is improved by 0.4% and the number of parameters and computation amount are reduced by 0.91 M and 1.4 G, respectively. The detection and recognition results for blueberries were visualized, as shown in Figure 8.

4.4. Comparative Experiment of Different NMS Algorithms

In the target detection task, it is generally divided into two processes: the training process and the inference process; in the training process, the target detection algorithm tunes the learning parameters of the model to fit the target features of the dataset according to the specific true values, and after the training is completed, the parameters of the neural network model are fixed to be able to make target prediction directly on new images. However, in the actual detection, a very large number of target detection frames are generated, in which there are many duplicate frames localized to the same target, and the NMS is used to remove these duplicate frames to obtain the true target detection frames. In this study, based on the Cluster-NMS algorithm, Cluster-NMF is proposed to improve the inference speed and at the same time further improve the detection efficiency of blueberry crown fruits by fusing the several relatively better target detection frames to obtain the optimal target detection frame. In order to verify the effectiveness of Cluster-NMF, we conducted experimental comparisons with some excellent NMS algorithms. Fps denotes the number of images that can be processed per second when processing the remote sensing images of blueberry canopy fruits taken by UAVs during the inference process, and a larger value indicates that the processing algorithm is more excellent. Inference denotes the time consumed in processing each image, and a smaller value is better. As shown in

Table 5. Data comparison of experimental results of different NMS algorithms.

NMS	Fps	Inference (ms)	[email protected] (%)	[email protected]–0.95
Traditional NMS	62.8	15.9	53.1	26.3
Soft-NMS	41.9	23.8	53.7	26.8
Weighted-NMS	61.9	16.2	53.5	26.4
Cluster-NMS	63.7	15.7	53.7	26.7
Cluster-NMF(ours)	63.6	15.7	54.4	27.1

below, our improved algorithm Cluster-NMF has a certain superior performance over other NMS algorithms.

5. Discussion

In this study, we collect and construct a blueberry dataset for agricultural remote sensing high-resolution target detection in a real blueberry orchard scenario, using UAV remote sensing combined with a deep learning target detection method to detect blueberry fruits in the crown layer of a blueberry orchard. This not only saves a lot of manual labor and workload, but also improves picking efficiency by monitoring the blueberry fruit clusters on blueberry trees in time for timely picking. We constructed a PF-YOLO algorithm, which reduces the omission and misdetection of blueberry fruits by integrating the spatial information features in horizontal and vertical directions to pay more attention to the exact location of blueberry fruits in the canopy while feature extraction is performed on the input image by using the proposed PAC3 module. The use of fast convolution for spatial feature extraction with some channel-specific convolution operations aims to reduce the number of parameters and computational complexity of the model for subsequent practical deployment without losing the accuracy of blueberry target detection. We also propose an improved Cluster-NMS algorithm named Cluster-NMF, which also utilizes matrix parallel operations to speed up the computation of IOUs, after initially filtering out the redundant target prediction frames, then filtering out the better target prediction frames based on a new threshold, and then fusing the above prediction frames to obtain the optimal target prediction frame. This not only accelerates the inference speed of detecting blueberry fruit clusters in the canopy, but also improves the accuracy of detection. The experimental results show that the model and method proposed in this study can help orchard farmers to monitor blueberry fruits in real-time, and according to the detected situation, farmers can pick on time, thus avoiding a delay to the best picking time which causes a certain degree of economic loss. However, all our experiments were conducted in good conditions. Although we preprocessed the collected data by adding noise and applying blurring, the recognition of blueberries would be significantly affected under harsh weather conditions. We found that there is an issue with the leaf obstruction of the fruits. Leaves can block parts of the fruits, leading to partial or complete occlusion, which affects the recognition capability of the detection algorithm. Additionally, occlusion may cause the detection model to misclassify leaves as fruits, reducing accuracy. Therefore, we will focus on tackling this challenge in the future.

6. Conclusions

Aiming at the shortcomings of the current small-target detection algorithms in high-resolution images in terms of detection effect and inference speed, we collected and constructed a blueberry canopy fruit dataset dedicated to the study of remote sensing target detection by using a UAV in a real blueberry orchard environment. In the process of feature extraction, we combined the encoding of positional information, and designed the PAC3 structure, which is intended to pay more attention to target positional information when feature extraction is performed on the input image, so as to effectively reduce the false detection and omission of target detection in remotely sensing targets to pay more attention to the location information of the target during feature extraction, so as to effectively reduce the misdetection and omission of blueberry fruit targets. In order to reduce the number of parameters and computational complexity of the model, we used a fast convolutional structure instead of the conventional convolutional structure. Based on this, we propose the PF-YOLO model, which significantly improves the detection accuracy of blueberry canopy fruits. In addition, we proposed the Cluster-NMF algorithm, which further optimizes the final detection frame by fusing several better target detection frames in the detection inference stage, improving the detection efficiency of blueberry crown fruits. We also conducted a critical analysis of the results and discussed the challenges encountered, as well as potential directions for future research.