Image Segmentation of UAV Fruit Tree Canopy in a Natural Illumination Environment

Abstract

Obtaining canopy area, crown width, position, and other information from UAV aerial images and adjusting spray parameters in real-time according to this information is an important way to achieve precise pesticide application in orchards. However, the natural illumination environment in the orchard makes extracting the fruit tree canopy difficult. Hereto, an effective unsupervised image segmentation method is developed in this paper for fast fruit tree canopy acquisition from UAV images under natural illumination conditions. Firstly, the image is preprocessed using the shadow region luminance compensation method (SRLCM) that is proposed in this paper to reduce the interference of shadow areas. Then, use Naive Bayes to obtain multiple high-quality color features from 10 color models was combined with ensemble clustering to complete image segmentation. The segmentation experiments were performed on the collected apple tree images. The results show that the proposed method’s average precision rate, recall rate, and F1-score are 95.30%, 84.45%, and 89.53%, respectively, and the segmentation quality is significantly better than ordinary K-means and GMM algorithms.

1. Introduction

Precise spraying is a spraying strategy. It uses modern technology to obtain crop information to guide the spraying machinery to achieve target spraying and variable spraying, so as to maximize the benefits of plant protection and reduce damage to the ecological environment. Refs. [1,2,3]. Image information is highly significant among the quantitative data on precise pesticide applications. Specifically, it is that the crop shape, structure, position, and other information of crops are obtained through images, and the spray parameters are adjusted according to specific crop information [4,5].

For precise spraying in orchards, the canopy is the main object of spraying, and parameters such as canopy volume, area, and crown width are important indicators to determine the spraying in orchards. Low-altitude remote sensing images that are based on UAV photography are one of the important ways to obtain fruit tree canopy parameters [6,7]. Compared with airborne radar imaging [8] and high-altitude satellite remote sensing [9], the use of UAVs to obtain images is more convenient and requires less data processing. The first process in extracting canopy parameters from drone images is to extract the canopy from the background [10,11,12]. In this process, one of the challenges is how to overcome various natural illumination and weather conditions in the orchard, which sometimes cause a shadow over the vegetation, for example (Figure 1a,b), hindering the computation of correct segmentations.

For efficient segmentation, effective features need to be employed to discriminate between plants and background. In previous studies, color has been identified as a critical feature for separating plants from non-plant backgrounds [13,14,15,16]. Frequently used color features can be grouped into two categories [17,18]. The first category is the color indices that are calculated in the RGB color space, such as excess green index (ExG) [19], color index of vegetation (CIVE) [20], and Modified Excess Green Index (MExG) [21], etc. The second category is the color channel of various color spaces, such as HSI, HSV, Lab, etc. [22,23,24,25]. Yaxiao et al. [26] effectively segmented cotton images at seedling and bud stages based on the differences between vegetation and non-vegetation pixels in the Lab color space, combined with a fixed classification threshold. Zhai et al. [27] extracted the HSI color space’s H and I channel to effectively segment the field rapeseed plant image that was captured under different lighting conditions. The research results of Pouria et al. [28] show that using multiple color spaces is more robust for background noise and outdoor lighting changes than using a single color space. High-quality color channels from different color spaces were combined to form the optimal color set (OCS) for vegetation segmentation to extract plants from the images in some studies [29,30,31]. Alwaseela et al. [32] selected the optimal channel of each color space according to the minimum classification error, then combined them into the supreme color feature, and segmented the Infield oilseed rape images using the feature. Yingli et al. [33] applied the optimal subset selection algorithm to extract seven characteristic parameters that were suitable for segmenting rice ear images of Northeast japonica rice from the RGB and HSV color spaces and good segmentation results were obtained.

Existing vegetation segmentation methods can be grouped into color index-based segmentation, threshold-based segmentation, and learning-based segmentation [34]. The threshold-based and color index-based segmentation can deal with uniform background and do not perform well under variable illumination conditions [35]. Machine learning (ML) is an image segmentation method that can overcome the insufficiency of color index-based and threshold-based. ML methods can be grouped into two categories: supervised learning and unsupervised learning, both of which can be applied to image segmentation in field conditions [36,37,38]. Wei et al. [39] proposed a wheat image segmentation method that was based on a machine learning process, decision tree, and image noise reduction filters for the problem of natural light interference and achieved good segmentation results. Chen et al. [40] proposed a tree segmentation method that was based on a support vector machine (SVM) algorithm and monocular machine vision technology to segment citrus trees precisely under different brightness and weed coverage conditions, and the results show that the citrus tree segmentation accuracy reached 85.27% ± 9.43%. Ferreira et al. [41], to obtain information on the spatial distribution of palm trees in forests, used ResNet-18 as the backbone network of the DeepLabv3+ model and combined morphological post-processing to obtain palm tree canopy layers. The authors compared the proposed method with the results of FCN segmentation and the results showed that that the model can improve the detection rate of canopy by 34.7%. In a word, supervised learning can achieve high-precision segmentation, but it relies on complex feature engineering or large-size labeled data [42]. Yuzhen et al. [43] summarized 34 data sets of machine vision in agriculture, among which there are no open datasets for fruit tree segmentation as yet. Therefore, supervised learning which based on data markers can consume a lot of labor costs.

In contrast, unsupervised learning algorithms can divide the data according to the similarity between the data without knowing the data classification in advance. Many unsupervised models have been used for image segmentation under controlled (single background, sufficient and uniform lighting, etc.) or uncontrolled conditions. Yingli et al. [44] proposed an unsupervised Bayesian method that was based on Lab color space for rice UAV image segmentation. The method is somewhat robust to variable illumination. Zhenzhen et al. [45] proposed a canopy segmentation algorithm that was based on M-SP feature weighted clustering for the problem of background and weed interference. However, this method only applies to fruit tree images that were taken on the ground. GUO et al. [46] proposed an effective segmentation of tree images that was based on Lab color distance and GMM. This method combines color and spatial correlation in the Lab space to achieve good segmentation results. In conclusion, unsupervised image segmentation methods have been widely used in controlled conditions and less in uncontrolled conditions. In addition, most unsupervised learning algorithms are susceptible to noise and initial conditions, tend to converge to local optimal, and are only suitable for specific data structures. It leads to unsupervised learning-based image segmentation methods with low robustness under uncontrolled conditions, poor segmentation quality, and insufficient generalization ability.

This paper aims to develop an unsupervised image segmentation method with excellent segmentation quality, high robustness and strong generalization ability for fast extraction of fruit canopy from aerial fruit tree images that are taken by UAV in a natural illumination environment. For this purpose, the following research work has been done: (1) a shadow region luminance compensation method (SRLCM) is developed to preprocess fruit tree images, which can reduce the difference between shadow and non-shadow canopy areas; (2) inspired by ensemble clustering and OCS methods, a fruit tree image segmentation method that is based on ensemble OCC-K (Optimal color channel-K-means) is proposed in this study. The algorithm can accurately segment canopy images without supervision.

2. Materials and Study Area

The test objects in this paper are apple trees in the orchard research and demonstration garden (117°7′8.9′′ E and 36°12′17.8′′ N) in Taian, Shandong Province, China. The terrain of the experimental area is flat, without slope and stepped ground, with an average elevation of 153 m. Apple trees in the test area were planted at a fixed 3 × 4 m spacing.

Images were collected on 2–3 October, 2018 (sunny, no wind) and 8–9 October 2018 (cloudy, no wind). The image acquisition platform was a DJI Phantom 3 four-axis aerial photography aircraft which was equipped with a Sony EXMOR 1/2.3-inch CMOS digital camera that can output RGB orthophotos. The UAV filmed horizontally at a constant speed (5 m/s) and a height range of 15–30 m. The flight path that was adopted by the drone was in the shape of an “n” (Figure 2b). Since the difference between each frame of video image is not big, one picture is extracted every 40 frames and the blurred images were remove. A total of 200 images were extracted, with a pixel of 1830 × 2279. Among them included 100 cloudy images (Figure 1b), 100 sunny images (Figure 1c).

All the algorithms that are mentioned in this paper were developed in Python using OpenCV [47] and scikit-learn [48] packages. The computer processor is Intel (R) Core (TM) i7-9750H CPU @2.6 GHz 2.59 GHz whose operating system is Windows 10 64 bits.

3. The Fruit Tree Segmentation Method

The proposed approach consists of two main processes, as shown in Figure 2 and Figure 3 image preprocessing:

▪

(1) luminance compensation of fruit trees in shaded areas using the SRLCM method.

▪

(2) fruit tree image segmentation based on OCC-K:

• Firstly, ten standard color spaces along with all possible combinations of their channels are evaluated by using accuracy (A), precision (P), F1-score (F1), and recall (R) as evaluation indexes; then according to the evaluation results, the color channel with the highest A value (AOCC), the color channel with the highest p value (POCC), the color channel with the highest R value (ROCC), the color channel with the highest F1 value(FOCC), and the color channel with the highest mean value of four indicators(MOCC) are extracted as color features.
• Secondly, one standard K-means and four Mini Batch K-means are used to cluster AOCC, POCC, ROCC, FOCC, and MOCC, respectively.
• Finally, the clustering results are combined to obtain the final segmentation result.

3.1. Image Preprocessing

In the fruit tree images that were collected by the UAV under natural illumination conditions, there is a big difference in the luminance of the canopy in the shaded and non-shaded areas, leading to misclassification and omission of the canopy. Therefore, SRLCM is proposed to compensate for the luminance of shadow areas to reduce the contrast between the canopy in shadow and the non-shadow regions. The SRLCM uses the luminance histogram (L color channel representing luminance information in the Lab color space) of the foreground area as the target histogram. Then, histogram matching on the brightness histograms of the shadow area and the foreground area was performed to achieve the purpose of compensating the luminance of the shadow area. Where the foreground area is the canopy area that was obtained by the CIVE + Otsu method, the shaded area is the area of pixels in the entire image whose luminance value is less than the first decile. After SRLCM processing, the shadow area and the foreground area have the same luminance distribution, which reduces the contrast between the shadow and foreground area. The SRLCM algorithm only changes the luminance information of the shadow area. It does not change the a and b channels representing color information in the Lab color space, so the canopy and the background can still be distinguished by color.

3.2. Fruit Tree Image Segmentation Method Based on Ensemble OCC-K

The K-means clustering algorithm is an unsupervised real-time clustering algorithm that was proposed by Mac Queen in 1967 [49]. This algorithm has a faster convergence speed than other unsupervised learning algorithms. The Mini Batch is an improved algorithm for K-means [50,51]. The algorithm uses randomly selected sample subsets instead of all sample points to find the optimal solution. Therefore, this method significantly reduces the convergence time of the K-means algorithm, but the clustering effect is also slightly lower than the standard K-means algorithm. Ensemble clustering is an idea of combining the clustering results of multiple clustering algorithms through a consensus function [52]. Combining multiple clusters can obtain significantly superior generalization performance, higher robustness, and better clustering quality than a single clustering algorithm [53]. It has been shown that the ensemble is most effective when there is a specific difference between the results output by different cluster members [54].

Inspired by ensemble clustering and considering the algorithm response time and the segmentation quality, a fruit tree image segmentation method that is based on ensemble OCC-K (optimal color channel-K-means) is proposed in this study: (1) AOCC, POCC, ROCC, FOCC, and MOCC features are extracted from the image. (2) One standard K-means and four Mini Batch K-means are used to cluster the five features, respectively. (3) The result of the standard K-means output is used as the reference partition to re-label the partition of Mini Batch K-means, and then use voting to combine the results after re-labelling to get the final result. In the process of clustering an ensemble, the reference partition’s clustering quality significantly influences the final ensemble result [55]. As such, we use one standard K-means as the ensemble member and its output result as the reference partition. Using Mini Batch K-means as the ensemble member can reduce not only the computational cost of the algorithm but also increase the difference in the output results of different clustering algorithms.

3.2.1. Color Feature Extraction

To select the optimal color features for the fruit tree image segmentation task, we use the optimal color space selection method that was proposed by Hernández-Hernández et al. [31]. This method consists of the following steps: (1) Convert an image to a specific color space and get all the channels and channel combinations for that color space; (2) in each channel combination, foreground and background masks are created; (3) each pixel is classified into foreground or background based on a posteriori probability value using Bayes naïve classifier. (4) The optimal channel combinations from each color space were selected based on the minimum classification error (err). To improve the diversity of the features, we introduce three more evaluation indices (precision, recall, and F1-score) based on the original method and use 1-err (accuracy, A) instead of err as an evaluation indicator. Accuracy (A) is the percentage of correctly classified pixels to all pixels, which can measure the classification performance of the algorithm on pixels. Recall (R) is the percentage of canopy pixels that are extracted by the algorithm to all canopy pixels, which can measure the segmentation completeness. Precision (P) is the correctly extracted percentage of the canopy pixels with the segmentation method, indicating the segmentation accuracy. The F1-score (F1) value considers both the precision rate and the recall rate of the segmentation model. It is the harmonic mean of precision rate and recall rate. These four indicators can all evaluate the segmentation performance of the color channel, and their evaluation focuses are different., which ensures both the diversity of ensemble members’ outputs and the quality of their clusters. These indices are defined as follows:

$$P=\frac{T_P}{T_P+F_P}\times 100\%$$

(1)

$$R=\frac{T_P}{T_P+F_N}\times 100\%$$

(2)

$$F1=\frac{2P\times R}{P+R}\times 100\%$$

(3)

$$A=\frac{T_p+T_N}{T_P+F_N+T_N+F_{{}_P}^{}}\times 100\%$$

(4)

where T_p is the number of pixels of the real canopy that are contained in the canopy that were extracted by the algorithm. F_p represents the number of pixels where the background is mistaken for the canopy. F_N represents the number of pixels where the canopy is mistaken for the background in the segmentation result. T_N represents the number of pixels of the real background that are contained in the background segmented by the algorithm.

In this study, ten standard color spaces were analyzed with all possible combinations of their channels. They are RGB, HSV, HSI, Lab, Luv, XYZ, YCrCb, YUV, I1I2I3, and TSL. All these color spaces are converted from RGB color spaces. Simultaneously, to ensure the ensemble members’ output diversity, AOCC, ROCC, POCC, FOCC, and MOCC should come from different channel combinations. MOCC features are preferentially selected and used as input to standard K-means to ensure the quality of reference partitions.

3.2.2. Clusters Initialization

In the K-means clustering algorithm, the effect of clustering largely depends on the number of clusters (K). An improper K-value may lead to over-segmentation or insufficient segmentation of the image. The Elbow method is a K-value selection algorithm that is based on the sum of squared errors (SSE) [56,57]. This method holds that when the K-value is less than the optimal cluster number, the increase of the K-value will lead to a sharp decline in SSE. When the K-value exceeds the optimal cluster number, SSE will no longer decrease significantly with the continuous increase of the K-value [58,59]. Therefore, the relationship between K and SSE is a broken line graph of elbow shape, and the K-value corresponding to the bending point in the graph is the optimal number of clusters. The SEE corresponding to the input of different K-values for the standard K-means in the ensemble members is calculated to choose the optimal K-value. The relationship between the K-value and SSE is shown in Figure 4. According to the picture, K = 5 is the bending point of the broken line graph. Therefore, we chose 5 as the number of clusters of ensemble members. In addition, Mini Batch K-means also needs to set the batch size, that is the number of samples that are drawn each time. Considering the algorithm response time and segmentation quality comprehensively, we take one-fifth of the number of image pixels as the batch size.

3.2.3. Combining Clustering Results

Plurality voting is a simple and efficient combination strategy that has proven effective for classifiers [60,61]. However, K-means clustering is an unsupervised method without any prior information that can be referenced. Therefore, K-means cannot output labels with semantic information, which leads to a problem of inconsistent cluster labels among partitions for ensemble member output. Therefore, the labels of each partition need to be unified before combining the clustering results. The general idea of cluster-label unification is as follows: taking cluster labels of the reference partition as the standard category labels, then matching the cluster-labels output by ensemble members with the standard category labels and re-labels the cluster labels based on matching results. Matching the cluster labels to the standard category labels is equivalent to a maximum weight bipartite matching problem [53]. This starts with creating a K × K (Where K is the number of clusters) contingency matrix between to be re-labeled partition πg and the reference partition πr. A contingency matrix contains entry Ω (l, l′) which denotes the co-occurrence statistics between label l ∈ πr and label l′∈ πg. Each entry Ω (l, l′) is defined by:

$$\mathsf{\Omega }(l,l\prime )={\displaystyle \sum _{\forall x_i\in X}\omega (x_i)}$$

(5)

where $\omega (x_i)=$1 if $(C^r(x_i)=l)\cap (C^g(x_i)=l\prime )$, otherwise $\omega (x_i)=0$, $C^r(x_i)$ and $C^g(x_i)$ represent the cluster-label of sample x_i in a πr and πg partitions, respectively, and X represents the sample set. Having obtained the contingency matrix, the problem of label matching can be solved by maximization of the weight of complete bipartite matching:

$$\underset{\theta (l,l\prime )}{\max }{\displaystyle \sum _{l=1}^{\mathrm{K}}{\displaystyle \sum _{l\prime =1}^{\mathrm{K}}\mathsf{\Omega }(l,l\prime )}}\theta (l,l\prime )$$

(6)

$$\mathrm{Subject} \mathrm{to} {\displaystyle \sum _{l=1}^{\mathrm{K}}\theta (l,l\prime )}={\displaystyle \sum _{l\prime =1}^{\mathrm{K}}\theta (l,l\prime )}=1,\underset{}{}\theta (l,l\prime )\in \{0,1\}$$

(7)

where $\theta (l,l\prime )$ represents correspondences amongst labels of partitions πr and πg, $\theta (l,l\prime )=1$ if label l ∈ πr corresponds to l′ ∈ πg, otherwise $\theta (l,l\prime )=0$. This optimization problem can be solved using the Hungarian algorithm [62]. Figure 5 shows an example of cluster label unification, which covers the various steps of label unification.

After all the partitions are re-labeled, plurality voting can be used to combine the clustering results. Specifically, it counts the number of times that each data object is divided into different categories and marks each data object as the cluster label with the highest number of statistics.

3.2.4. Evaluation of Image Segmentation Methods

To validate the proposed method’s crown segmentation performance on fruit trees, this study compared it with the K-means clustering algorithm and GMM (Gaussian mixture model) algorithm, which were unsupervised image segmentation methods that are widely used for fruit crown segmentation [42,44,63]. AOCC, POCC, FOCC, MOCC, and ROCC are combined into a feature matrix M as the segmentation feature of K-means and GMM algorithms. P, R, and F1 are introduced to evaluate the segmentation results of fruit tree canopy images and take the manually segmented canopy image as the real canopy. The manual segmentation tool is Adobe Photoshop 2022. The specific steps are: select the lasso tool in Photoshop to draw the outline of the canopy and use the selected area as the foreground of the image, then select the fill function and set the color of the area outside the foreground to black.

4. Results and Discussion

4.1. Image Preprocessing Test Results

Figure 6 shows the results of preprocessing fruit tree images that were captured under different weather conditions using SRLCM., As shown in the figure, the contrast of the shaded and non-shaded areas in the image is significantly reduced after SRLCM processing compared to before SRLCM processing, and the color of the canopy and background is not changed. Therefore, we believe that the SRLCM algorithm can effectively compensate for the luminance of the shadow area.

To evaluate the effect of applying the SRLCM, we obtained R, G, and B channels of shaded and non-shaded canopy regions before and after SRLCM processing, as shown in the box plot in Figure 7. Figure 7a shows the R, G, and B components of a fruit tree image that was taken under sunny conditions. Before being processed by SRLCM, the gray average values of the R, G, and B channels in the shaded canopy area are 61, 65, and 56, respectively. The gray average values of the non-shaded canopy area are 101, 115, and 100, respectively. The grayscale differences between the shaded and non-shaded areas are 40, 50, and 44, respectively. After being processed by SRLCM, the gray average values of the R, G, and B channels in the shaded canopy area are 87, 91, and 82, respectively, and the gray average values of the non-shaded canopy area are 106, 120, and 105, respectively. The grayscale differences between shaded and non-shaded areas are 19, 28, and 23, respectively. Obviously, the grayscale difference of R, G, and B channels in shady and non-shaded canopy regions is significantly reduced after SRLCM processing. Similar results were seen in images of fruit trees that were taken on cloudy days (as shown in Figure 7b), in which the grayscale differences of R, G, and B channels in shaded and non-shaded canopy regions before and after SRLCM processing are 29, 32, 43 and 20, 23, 35, respectively. Therefore, we believe that the SRLCM algorithm can reduce the difference between shaded and non-shaded canopy regions, thereby reducing the difficulty of fruit tree canopy segmentation.

4.2. Results of Color Space Evaluation

Table 1. Recall rate ®, precision rate (P), accuracy rate (A), and F1-score (F1) of background/fruit tree canopy classification that was selected by different color spaces and channels, in%.

Channels	Indicators	RGB	HSV	Lab	HSI	XYZ	Luv	YCrCb	YUV	I1I2I3	TSL
1	R	68.26	85.27	64.04	64.21	64.80	64.12	64.03	64.01	63.90	74.76
	P	79.15	87.83	68.66	52.39	73.49	71.93	71.94	68.78	72.10	80.40
	A	72.67	84.04	66.87	63.59	68.52	67.58	67.56	66.88	67.47	77.84
	F	73.30	86.53	66.27	57.70	68.87	67.80	67.75	66.31	67.75	77.48
2	R	62.11	50.59	79.51	56.51	63.54	81.02	67.86	82.04	78.07	65.36
	P	67.94	31.10	84.59	29.51	70.67	84.09	70.84	88.82	76.51	74.80
	A	65.16	52.94	82.36	55.84	66.87	83.12	70.28	85.47	78.68	69.23
	F	64.89	38.52	81.97	38.78	66.92	82.53	69.32	85.30	77.28	69.76
3	R	60.06	64.49	67.60	63.91	60.59	68.33	82.64	64.66	70.82	63.95
	P	65.46	73.63	78.79	72.12	66.17	72.37	87.89	72.49	75.80	72.20
	A	62.98	68.27	72.04	67.48	63.55	71.00	83.41	68.17	73.71	67.53
	F	62.65	68.76	72.77	67.77	63.26	70.29	85.18	68.35	73.22	67.83
12	R	66.31	84.84	80.20	63.16	64.24	79.35	72.43	80.34	77.12	74.99
	P	75.33	87.97	86.87	53.11	72.26	85.90	76.87	89.98	82.01	80.03
	A	70.15	86.86	83.60	63.05	67.75	82.72	75.17	84.80	79.92	77.89
	F	70.53	86.38	83.40	57.70	68.01	82.49	74.59	84.88	79.49	77.43
13	R	66.12	82.32	71.79	65.28	63.26	72.56	79.40	69.85	74.34	74.85
	P	75.20	84.16	75.27	74.64	70.75	77.00	85.87	72.42	81.68	78.74
	A	69.98	83.93	74.27	69.15	66.67	75.28	82.74	72.10	77.95	77.37
	F	70.37	83.23	73.49	69.65	66.80	74.72	82.51	71.11	77.84	76.74
23	R	61.08	64.18	83.07	63.30	62.28	82.98	80.98	81.75	80.67	64.78
	P	66.93	73.09	86.01	69.39	68.81	83.35	83.52	88.17	80.87	73.37
	A	64.10	67.89	85.06	66.42	65.45	82.82	82.89	85.06	81.73	68.47
	F	63.87	68.34	84.52	66.21	65.38	83.16	82.23	84.84	80.77	68.80
123	R	64.99	82.08	82.19	64.59	63.38	79.58	79.57	80.08	79.07	71.91
	P	73.15	83.79	87.66	71.87	70.75	84.97	85.32	88.90	84.12	77.16
	A	68.61	83.64	85.05	67.98	66.74	82.52	82.65	84.26	81.93	74.89
	F	68.83	82.93	84.83	68.04	66.87	82.19	82.35	84.26	81.52	74.44

According to the table that the top color channels in index F1 are H (in HSV), HS (in HSV), U (in YUV), CrCb (in YCrCb), YU (in YUV), and Lab in turn; the top color channels in index A are HS (in HSV), U (in YUV), ab (in Lab), Lab, UV (in YUV), and YU (in YUV) in turn; the top color channels in index P are YUV (YUV), U (YUV), UV (YUV), HS (in HSV), CrCb (in YCrCb) in turn and the top color channels in index R, are H (in HSV), HS (in HSV), ab (in Lab), UV (in Luv), Lab, and HSV in turn. According to the evaluation results, HS (in HSV) was selected as the MOCC, H (in HSV) was selected as the FOCC, U (in YUV) was selected as the AOCC, YU (in YUV) was selected as the POCC, and ab (in Lab) was selected as the ROCC, which were marked in bold in the table.

shows the accuracy rate (A), recall rate (R), precision rate (P), and F1-score (F1) of background/fruit tree canopy classification that was selected by different color spaces and channels. In the table, the number 1 represents the first color channel of the color model (such as the R channel of the RGB color model), and the number 12 represents the combination of the first l of the color model (such as the RG channel combination of RGB, and other numbers have the same meaning.

4.3. The Results of Image Segmentation Using the Proposed Method

After clustering is complete, the category label of each pixel is taken as the pixel value to form a grayscale image, as shown in Figure 8. The figure shows the whole process of combining of the clustering results. The first two lines in the figure show the results of each ensemble member clustering and the unified cluster labels. As seen from the figure, the segmentation results of the standard K-means have an obvious under-segmentation phenomenon, manifested in the mulch pixels that were mistakenly classified as a canopy, and the segmentation results of other ensemble members have an obvious over-segmentation phenomenon.

Figure 8g shows the result of combining the outputs of each ensemble member. As seen from the figure, the result of the combination effectively avoids the problems of the ensemble members. Specifically, there is no obvious over-segmentation and under-segmentation; the result of the combination is significantly better than the output of the ensemble members. Therefore, we believe that combining multiple clustering algorithms can dramatically improve the image segmentation quality. This result is similar to that of Navid et al. [64].

4.4. Image Segmentation Method Evaluation Results

The lighting environment on sunny and cloudy days are two representative natural lighting environments, which represent the lighting environment with direct sunlight and the lighting environment without direct sunlight, respectively. The ideal image segmentation algorithm should adapt to the changes of lighting environment. In this regard, we randomly select 30 images of fruit trees that were taken in sunny days and 30 images that were taken in cloudy days from the collected images as experimental samples to test the performance of the segmentation method. The segmentation results were binarized to evaluate the obtained canopy, and the artificially segmented canopy was used as a reference.

Figure 8 shows an example of image segmentation using the method that is proposed in this paper (the proposed method), K-means algorithm, and GMM algorithm, respectively. It can be seen from the figure that the common K-means and GMM image segmentation methods have obvious over-segmentation. It is because the difference in the pixel values between shaded and non-shaded canopy areas is significant, resulting in the canopy in shaded areas being misclassified as background. In addition, the fruit tree canopy is a porous medium, and the inter-leaf pores can also lead to over-segmentation. In contrast, the method that is proposed in this paper is significantly better than the comparison method. Specifically, the canopy pixels in the shadow area are entirely preserved, and the canopy contour is close to the manually segmented canopy. Obviously, the proposed method also cannot completely avoid the influence of canopy holes, but these tiny holes can be removed entirely by image post-processing methods (such as hole filling, morphological processing, etc.). Figure 9a,b show the segmentation results of the algorithm on fruit tree that were images taken under sunny and cloudy conditions, respectively. It can be seen from the figure that the change in illumination conditions has a more significant impact on the K-means method, while the GMM and the proposed method are less sensitive to illumination changes.

The same conclusion can also be reached when we compare these algorithms using indicators in image segmentation.

Table 2. Comparison of the evaluation indicators (P, R, F1) of different segmentation algorithms.

Algorithm	Weather	P	Means of P	R	Means of R	F1	Means of F1	Time
K-means	sunny	93.05%	95.17%	64.98%	71.91%	76.52%	80.78%	7.1 s
	cloudy	96.84%		75.83%		85.05%
GMM	sunny	93.35%	94.56%	73.09%	73.55%	81.98%	82.63%	51.4 s
	cloudy	95.76%		74.00%		83.27%
The proposed method	sunny	94.29%	95.30%	81.15%	84.45%	87.22%	89.53%	7.9 s
	cloudy	96.31%		87.75%		91.83%

shows the results of the quantitative evaluation of different segmentation algorithms using P, R, and F1. Where the P of all the algorithms is relatively high, so it cannot be used as an effective indicator to compare the performance of each algorithm. According to Table 2, the K-means algorithm has all lower recall rates under two illumination conditions, 64.98%, and 75.84%, respectively, which indicates that the canopy integrity that was obtained by the K-means algorithm is low. In addition, changes in illumination conditions can significantly affect the recall rate and F1-score of K-means, which shows that the algorithm is not robust enough to illumination changes. The recall rates of the GMM algorithm in the two illumination conditions were 73.09% and 74.00%, respectively. Compared with the K-means algorithm, it is less affected by illumination changes, but its recall rate is still very low. In contrast, the method that is proposed in this paper has a higher recall rate under the two illumination conditions, 81.15%, and 87.75%, respectively. It shows that the canopy that was obtained in this paper has high integrity and is robust to changes in illumination conditions. In addition, the proposed method’s average F1 and recall rates are 8.75% and 12.54% higher than the K-means algorithm, respectively; and 6.9% and 10.9% higher than the GMM algorithm, respectively. In a word, it can be considered that the segmentation quality of the proposed method is significantly better than the common K-means and GMM algorithms.

Table 2 also shows the response time of each algorithm. It is worth noting that the response time that is shown in the table is only used for comparison, and the algorithm can be very fast if the computer’s performance is higher or the picture is smaller. According to Table 2, the response time of the proposed algorithm is slightly higher than that of the K-means algorithm and significantly lower than that of the GMM algorithm.

5. Conclusions

This paper proposes an unsupervised method for canopy segmentation of fruit trees, and segmentation experiments are carried out. The following conclusions are drawn based on the test results: (1) SRLCM can effectively reduce the difference between shaded and unshaded canopy regions, thus reducing the difficulty of segmentation. (2) Combining multiple clustering algorithms can significantly improve the quality of image segmentation. (3) Compared with commonly used unsupervised methods, the method that is proposed in this paper has the highest average P, R, and F1-score, which are 95.30%, 84.45%, and 89.53%, respectively. The average F1-score and recall rates are 8.75% and 12.54% higher than the K-means algorithm, respectively, and 6.9% and 10.9% higher than the GMM algorithm, respectively. The response time of the proposed algorithm is slightly higher than that of the K-means algorithm and significantly lower than that of the GMM algorithm. (4) The research in this paper also proves that computer vision technology has great application potential in agriculture. The algorithm that is proposed in this paper can provide a reference for the application of computer vision technology in agriculture.