Uniformity Detection for Straws Based on Overlapping Region Analysis

Abstract

Nowadays, the advanced comprehensive utilization and the complete prohibition of burning fully covered straws in croplands have become increasingly important in agriculture engineering. As a kind of direct straw-mulching method in China, conservation tillage with straw smashing is an effective method to reduce pollution and enhance fertility. In view of the high straw-returning yields, complicated manual operation, and the poor performance of straw detection with machine vision, this study introduces a novel form of uniformity detection for straws based on overlapping region analysis. An image-processing technology using a novel overlapping region analysis was proposed to overcome the inefficiency and low precision resulting from the manual identification of the straw uniformity. In this study, the debris in the gray map was removed according to region characteristics. Through using morphological theory with overlapping region analysis in low-density cases, straws of appropriate length can be identified and then uniformity detection can be accomplished. Compared with traditional threshold segmentation methods, the advantages of an accurate identification, fast operation, and high efficiency contribute to the better performance of the innovative overlapping region analysis. Finally, the proposed algorithm was verified through detecting the uniformity in low-density cases, with an average accuracy rate of 97.69%, providing a novel image recognition solution for automatic straw-mulching systems.

1. Introduction

To reduce the influence of straw burning and utilize the resource sufficiently, some countries have published targeted policies in the comprehensive utilization of straw and achieved positive implementation effects, especially in terms of new energy and conservation tillage with straw mulching [1]. As the leading method of straw recycling in European and North American countries, straw mulching consists of two modes (i.e., direct straw-mulching during harvesting or straw mulching after feeding livestock). Specifically, the amount of straw returned to the field is about 68% of the total straw produced in the United States. More than 60% of the total straw produced in Canada, 73% of the total straw produced in the United Kingdom, more than 66% of the rice straw in Japan, and almost all the rice-wheat straw in South Korea are directly returned to the field.

Due to high rice-wheat production in China, a large quantity of field straw will seriously affect sowing quality during the rice–wheat rotation period [2,3]. As a kind of direct straw-mulching method in China, conservation tillage with straw smashing is an effective method to reduce pollution and enhance fertility in agriculture. This is the main existing production pattern among the grain-producing areas in the middle and lower reaches of the Yangtze River [4,5]. In practice, nonuniform straw mulch will affect the growth of seedlings and hinder the smooth operation of fertilizer and sowing machines. Therefore, the uniform coverage of straw mulching in the field is still an urgent practical problem. Moreover, there is a lack of technology and equipment for straw crushing and returning satisfying such agronomic requirements. In recent years, reasonable mechanized smashing and returning technology has attracted the attention of researchers, where the straw uniformity was the index of the smashing level [6,7,8,9,10]. As shown in Figure 1, the straw-mulch machine, which was developed by Nanjing Institute of Agricultural Mechanization (Ministry of Agriculture and Rural Affairs, China), adjusts the spraying device to realize straw mulching with smashed straw after the manual identification of the shredded-straw uniformity. This machine includes a straw-crushing device, a straw-spraying device and a straw-dispersing device [11]. The straw-spraying device is mainly composed of the throwing impeller and the throwing pipeline (Figure 2). The crushed straw is blown along the pipeline under the action of the rotation of the impeller. During operation, the straw-crushing device and the straw-spraying device are powered by the tractor. After the residual straw in the field is picked and smashed by the straw-crushing device, the straw is lifted by the straw-spraying device, and it is evenly thrown back under the action of the dispersing device. Through this, the ultimate intention is to maintain shredded-straw uniformity in the coverage region. However, this traditional straw-mulching method requires manual operation, which not only consumes manpower and material resources, but also has a low working efficiency [6,7,8].

With the development of intelligent equipment and Internet of Things technology [12,13], machine vision and image-processing technologies have received extensive attention, especially for crop disease detection [14,15], spraying machines [16], and weed robots [17]. Advanced image-processing methods can greatly reduce costs and improve productivity [18]. From the perspective of agricultural product harvesting, the quantity and quality parameters can be considered as important indicators of agricultural products. By taking photos with electronic cameras, straws can be accurately identified to reduce artificial errors and provide technical solutions for the intelligent straw-mulching system.

Over the past few years, researchers have studied the advantages of agricultural image-processing technology. Tian et al. [19] applied machine vision technology to estimate weed density and size in real time, which can effectively reduce the amount of herbicide in corn and soybean fields. Coy et al. [20] proposed an unsupervised segmentation algorithm for the accurate estimation of vegetation coverage, which converted the original image into a certain color space (brightness, red-green, and blue-yellow), and the results showed that the accuracy rate was more than 85%. Hamuda et al. [21] summarized image-processing techniques for the extraction and segmentation of wild plants, including color-based, threshold-based, and learning-based segmentation. Hernández et al. [22] proposed a novel color-processing probability method, which can not only create a best optimal color model for plant/soil segmentation, but can also select the most appropriate color space in different situations.

However, studies on straw recognition technology are lacking. Kujawa et al. [23] utilized computer image analysis and a neural model to construct a classification of sewage sludge composted with maize straw and identify the early maturity stage. Jia et al. [24] introduced machine vision to obtain the numbers of corn plants and demarcated the geometric centers. Liu et al. [25] established a high-throughput quantitative image-processing method to analyze the decomposition degree of straws based on their color degradation in soil over a few days. Sun et al. [26] proposed a multi-scale retinal-enhanced image technology to obtain images of a cucumber canopy and improve crop performance. Based on a binary image-processing method, Eisa et al. [27] estimated the coverage rate of rice and wheat straws. Liu et al. [28] detected straw coverage by a multi-threshold segmentation algorithm, which combined the searching mechanism of the gray wolf algorithm with the differential evolution algorithm.

Although experts have made progress in exploring the straw-coverage rate as described above, no published research to estimate straw uniformity using a vision-based process is available at present. In practice, uniformity detection for straws is a challenging task in view of the overlapping problems, which makes the detection of targets difficult and cumbersome when using machine vison. Considering the high straw-returning yields, complicated manual operation, and the poor performance of straw detection with machine vision, this study focused on a novel morphological image recognition technology for solving overlapping problems in low-density cases, which can offer a value reference for investigating the high-density overlapping issues. A fast visual-processing method based on color vector and shape feature, with the ability to remove the debris according to region characteristics, was designed. Notably, straws of appropriate length were identified by a mathematical morphological operation algorithm, and the uniformity was then calculated immediately. Above all, the straw detection approach in this study provides a novel image analysis solution for automatic straw-mulching systems, which can improve recognition accuracy and reduce work time.

2. Materials and Methods

Field experiments in an artificial laboratory were performed at the Nanjing Institute of Agricultural Mechanization, Jiangsu Province, China, in the daytime, with dim sunlight or on a cloudy day without wind. The experimental soil was a loam. Shredded rice straws in the low-density case were investigated under uniform illumination. The Intel-RealSense D435 camera is a stereo-tracking device that provides high-quality color and depth data for a variety of applications. With the advantages of low cost, compact size, and wide field of view, it is ideal for robotics or applications such as augmented reality and virtual reality. This digital camera, with a maximum resolution of 1920 × 1080 pixels, was fixed at a height of about 30 cm from the targets, and sixty RGB images were processed in Matlab 2019b (The MathWorks, Inc., Natick, MA, USA). Images were captured vertically if possible. A maximum contrast between the straws and the loam guaranteed the target extraction. In this study, the exposure parameter was set within 80~150, and the RGB frame rate of the camera was 30 fps.

3. Digital Image Pre-Processing

An RGB image is an original full-color model, which is synthesized through the three primary colors of red, green, and blue. A typical original RGB image of the straws with a resolution of 960 × 720 pixels is shown in Figure 3. Utilizing gray image analysis can significantly reduce the storage space and accelerate the subsequent recognition. However, a gray image merely contains brightness without color information. The weighted average method is suitable for gray transformation from the full-color RGB image, and human eyes can easily distinguish targets [29].

For convenience, the color value of any pixel in the RGB model is represented by:

$$C(i,j)=\left\{x|x\in [R(i,j) G(i,j) B(i,j)]\right\}$$

(1)

where i(j) denotes the row (column) index of the pixel matrix, $R(i,j)$ denotes the red component value, $G(i,j)$ denotes the green component value, and $B(i,j)$ denotes the blue component value.

According to the indicator of the pixel components in RGB images, the weighted average value of the three primary color components is taken as the gray value, and its luminance formula is established as follows [29]:

$$C_g(i,j)=0.299\times R(i,j)+0.587\times G(i,j)+0.114\times B(i,j)$$

(2)

Based on the weighted average method, the straw image in grayscale was obtained (Figure 4).

Usually, HSV and HSI are powerful tools to distinguish targets, which can present the color intensity intuitively [30]. The hue and saturation are regarded as the main indices in such models. However, this is not suitable for the lighting model. To compare the identification accuracy, the HSV and HSI models were adopted to test the performance in this study, and the results are shown in Figure 5.

As presented in Figure 5, in the images, the luminance variation between the background and the straws in an artificial laboratory is relatively obvious, and this may serve as an effective tool for detecting the straws. The H and S component values indicate that the straws in the map are similar to the background, and the characteristics of the crops cannot be highlighted obviously. Then, the I component shows that the luminance variation can enhance the contrast effect, which was regarded as the evaluation criterion in this study.

4. Image Smoothing by Spatial Filtering

Generally, random noise exists in images and the pixel values have random mutations in this field. To eliminate such random noise and thus smooth the image, spatial filtering is necessary [31].

To compare the filtering results, various methods were used and these are listed in

Table 1. Common filters in practice.

Method	Classification	Transfer Function
Average filter	linear	/
Classical low-pass filter	frequency domain	$H(u,v)=\{\begin{array}{c}1, when D(u,v)\le D_0\\ 0, when D(u,v)>D_0\end{array}$
Gaussian low-pass filter	frequency domain	$H(u,v)=e^{-D^2(u,v)/2{\sigma }^2}$
Adaptive median filter	nonlinear	/

. Average filtering is a popular linear method to average pixel values in the whole window range. However, it fails to protect image details well and makes the image blurry, as shown in Figure 6a. Furthermore, frequency-domain filtering is a widespread method used in engineering and is carried out based on Fourier transform [32]. The flow diagram of frequency-domain filtering is presented in Figure 7, and the filter transfer function is defined as $H(u,v)$. In order to eliminate the noise in an image, low-frequency filtering is usually adopted, removing the high-frequency signal. As shown in Table 1, the filtering parameters of the frequency-domain approaches can be defined as follows: $D_0>0$, $D(u,v)$ represents the distance between the point $(u,v)$ and the center of the filter template, and $\sigma$ is the standard deviation. The classical low-pass filtering result is shown in Figure 6b. Moreover, Gaussian filtering is widely used in image processing to reduce Gaussian noise [15]. Specifically, the results are presented in Figure 6c.

In brief, linear spatial filtering is used to calculate the sum of the product (i.e., a linear operation). Nevertheless, nonlinear spatial filtering is used to order the pixel values, and then update the center pixel based on the sorted values. For example, nonlinear adaptive median filtering, by using a filter window of adaptive size, puts the median value of the neighborhood in the center [33], and this approach can protect the sharp edges and preserve the original information reasonably. Therefore, the adaptive median filter was remarkably effective in this study, and the filtering result is shown in Figure 8, where $S_{\max }$ denotes the maximum allowed size of the adaptive filter window.

5. Image Threshold Segmentation Algorithm

Image thresholding is one of many active segmentation methods, which divides the image pixels into several classes. Due to the simple calculation, high efficiency, and stable performance, image thresholding has been widely applied in image-processing techniques.

5.1. Threshold Segmentation by Setting Manually

In fact, different targets are separated by setting the grayscale threshold, and then image binarization processing can be carried out. To distinguish the target and the background, the straw image threshold was assumed to be Y, and the binarization image after thresholding was defined as:

$$B(x,y)=\{\begin{array}{c}a, when B(x,y)>Y\\ b, when B(x,y)\le Y\end{array}$$

(3)

where $a=1$ and $b=0$. By setting the threshold value manually, the original typical straw image was segmented as shown in Figure 9. Obviously, the segmentation accuracy depends on manual operation rather than automatic generation.

5.2. Adaptive Global Thresholding

In an adaptive global threshold method, the custom and iterative optimization algorithms are combined, and the basic steps can be described as follows:

(1)

Select an initial global threshold $Y_{ini}=\frac{Y_{\min }+Y_{\max }}{2}$, where $Y_{\min }$ and $Y_{\max }$ denote the maximum and minimum grayscale values of the image.

(2)

Segment the image to produce two sets of pixels according to the threshold Y: $C_{g1}=\left\{C_g(x,y)>Y_{ini}\right\}$ and $C_{g2}=\left\{C_g(x,y)\le Y_{ini}\right\}$.

(3)

Calculate the average value of each pixel set: $A{V}_1=\frac{{\displaystyle \sum _{(x,y)\in C_{g1}}{C}_g(x,y)}}{num(C_{g1})}$ and $A{V}_2=\frac{{\displaystyle \sum _{(x,y)\in C_{g2}}{C}_g(x,y)}}{num(C_{g2})}$.

(4)

According to the updated threshold, $Y_{up}=\frac{A{V}_1+A{V}_2}{2}$, repeat the previous steps and stop iterations until the variation of $Y_{up}$ is less than the setting, so as to obtain the final global threshold.

Adaptive global threshold methods perform threshold segmentation based on the variation of gray levels, and then execute the binarization procedure. This method fully considers the characteristics of each region in an image and can easily highlight the target as distinct from the background. The segmentation result with adaptive global thresholding is shown in Figure 10.

5.3. Otsu’s Optimal Global Thresholding

In addition, Otsu’s optimal global threshold processing has been adopted in this study [34], and the basic principle of the algorithm is described as follows:

(1)

According to the chosen threshold Y, the pixel sets $C_1$ and $C_2$ can be divided with levels [1, 2, …, k] and [k + 1, k + 2, …, L], where L denotes the color intensity. The occurring probabilities of sets $C_1$ and $C_2$ can be denoted as $P_1(k)={\displaystyle \sum _{i=1}^k{p}_i}$ and $P_2(k)={\displaystyle \sum _{i=k+1}^L{p}_i}=1-P_1(k)$, where $p_i$ is the occurring probability of pixels with level i.

(2)

Define the within-class variance, between-class variance, and total variance as ${\sigma }_w^2(k)=P_1(k){\sigma }_1{}^2+P_2(k){\sigma }_2{}^2$, ${\sigma }_b^2(k)=P_1(k){\left[{\eta }_1-{\eta }_t\right]}^2+P_2(k){\left[{\eta }_2-{\eta }_t\right]}^2$ and ${\sigma }_t^2(k)={\displaystyle \sum _{i=1}^L{(i-{\eta }_t)}^2{p}_i}$, where ${\sigma }_1={\displaystyle \sum _{i=1}^k{(i-{\eta }_1)}^2{p}_i/P_1(k)}$, ${\sigma }_2={\displaystyle \sum _{i=k+1}^L{(i-{\eta }_2)}^2{p}_i/P_2(k)}$, ${\eta }_1={\displaystyle \sum _{i=1}^{k}i{p}_i/P_1(k)}$, ${\eta }_2={\displaystyle \sum _{i=k+1}^{L}i{p}_i/P_2(k)}$, and ${\eta }_t={\displaystyle \sum _{i=1}^{L}i{p}_i}$, the global average gray intensity.

Through searching for the optimal threshold Y, Otsu’s threshold method maximizes the objective functions $\lambda ={\sigma }_b^2/{\sigma }_w^2$, $\kappa ={\sigma }_t^2/{\sigma }_w^2$ and $\gamma ={\sigma }_b^2/{\sigma }_t^2$. In previous functions, the discriminant $\gamma$ corresponding to the variable k can serve as the divisibility indicator to evaluate the threshold value with level k. By using Otsu’s threshold algorithm described above, the processing result shown in Figure 11 was obtained.

6. Segmentation Algorithm Based on Overlapping Region Analysis

6.1. Target Extraction with Region Characteristics

The image segmentation algorithm based on region characteristics divides the image into a series of independent regions based on the similar features of the pixels, and then each region will possess the consistency characteristics.

Furthermore, the area-based filter of connected regions can remove the miscellaneous noise. According to the morphology of the straws, the target recognition can be completed through calculating the external rectangular aspect ratio of the targets [35,36]. The whole image region can be represented by R, which is divided into n regions (R₁, R₂, …, R_n). If adjacent pixel values in the image are nonzero, these pixels can be considered as a connected region. According to the shape characteristic of the pixel matrix, the connected directions of two adjoining pixels can be defined as horizontal, vertical, and diagonal directions, as shown in Figure 12.

Obviously, discontinuous parts remain in the straw image and do not contribute to the objects. Based on the shape feature of the straws, binary denoising can be used to obtain the slender target. In this study, the 8-connected template was applied to mark the grayscale image, where the background pixels were denoted as 0 and the other pixels were denoted as 1, 2, ..., n [37]. According to the area sizes and region shapes of segmented targets, the discontinuous parts were excluded to obtain the result in Figure 13.

6.2. Morphological Operation Based on Overlapping Region Analysis

According to the operational specifications and requirements provided in the Chinese National Standard GB/T 24675.6-2009 “Conservation tillage equipment-Smashed straw machine”, the length qualification rate of shredded straws can be a critical factor affecting the uniformity level [4]. Traditional region segmentations are inefficient, particularly in occluded situations. As a result, this study concentrates on separating the straws one by one with a novel morphological approach.

To apply the overlapping region algorithm, a three-step process on the binary image was proposed:

(i)

Thinning-based skeletonization algorithms are devoted to the extraction of skeletons.

(ii)

Endpoints can be identified by the Harris–Stephens algorithm in skeleton images.

(iii)

Each straw skeleton can be extracted by using the overlapping region optimization, and the corresponding straw length can be calculated.

The first step is to skeletonize the straws by adopting the one-pixel thick skeleton method [38], and a representative skeleton image is shown in Figure 14. Subsequently, the second step is to find all endpoints by the Harris–Stephens algorithm in skeleton images [39]. The feature point, which has only one pixel in the adjacent positions, was regarded as an endpoint. The identified endpoints were marked with red dots in Figure 14.

The third step is critical, that is, to extract each skeleton by a novel overlapping region optimization. As shown in Figure 14, nine straw skeletons are intersecting, of which four endpoints can be denoted as A, B, C, and D. To split each skeleton, the line segments AB and CD should be extracted. For instance, the endpoint A can be the initial searching point and its adjacent pixels are taken as candidate points in the 8-connected neighborhood. If the number of candidate points is more than two, the most suitable candidate point should be selected as the follow-up searching point based on the optimization equation, Equation (6).

As shown in Figure 15, the slant angle of the line that passes through the initial searching point and the ith candidate point can be established by:

$$angl{e}_i={\tan }^{-1}(k_i), k_i=\frac{y_i-y_{initial}}{x_i-x_{initial}}$$

(4)

where $(x_{initial},y_{initial})$ represents the initial searching pixel coordinate; $(x_i,y_i)$ denotes the ith candidate point coordinate around the previous searching point; $k_i$ and $angl{e}_i$ denote the slope and the slant angle of the line that passes through the initial searching point and the ith candidate point, respectively.

In Figure 15, the slant angle of the line that passes through the previous searching point and the initial searching point can be obtained by:

$$angl{e}_{ps}={\tan }^{-1}(k_{ps}), k_{ps}=\frac{y_{ps}-y_{initial}}{x_{ps}-x_{initial}}$$

(5)

where $(x_{ps},y_{ps})$ represents the previous searching pixel coordinate; $k_{ps}$ and $angl{e}_{ps}$ denote the slope and the slant angle of the line that passes through the previous searching point and the initial searching point, respectively.

Notably, the future searching point can be established with the optimization approach as follows:

$$\mathrm{If} \left|angle\_erro{r}_k\right|==\min (\left|angle\_error\right|) (k=1, \dots , n),$$

(6)

$$(x_k,y_k) \mathrm{will} \mathrm{be} \mathrm{chosen} \mathrm{as} \mathrm{the} \mathrm{follow}\text{-}\mathrm{up} \mathrm{searching} \mathrm{point}.$$

(7)

where $angle\_error=angl{e}_i-angl{e}_{ps} (i=1,\dots ,n)$, n is the total number of candidate points around the previous searching point, $angle\_erro{r}_k$ denotes the kth element of $angle\_error$, $|\ast |$ denotes the absolute value of $\ast$, $\min (\ast )$ is the minimum of $\ast$.

Similarly, the searching point around the branchpoint can be determined, and the searching program will stop once one of the other endpoints is found. As shown in Figure 16a, the line segment AB can be obtained by using the novel overlapping region analysis, and the length of AB can be calculated at once. Then, the rest of the straw skeletons in this image can be precisely extracted by using the same principle, as shown in Figure 16b,c. Finally, the length qualification rate of the shredded straws, as well as the uniformity of the straws, can be easily estimated.

7. Results and Discussion

In order to verify the novel overlapping region method, shredded straws in low-density overlapping situations were considered. In each of the three cases introduced in

Table 2. Identification accuracy rate in low-density overlapping cases.

Cases	Branchpoint Number	Accuracy Rate %
Seven targets	7	98.23
Eight targets	10	97.68
Nine targets	12	97.16

, Figure 17 shows that the length value of each straw was exactly calculated with the proposed optimization algorithm, and the results demonstrate that the approach used is effective for the uniformity detection of straws, with an average accuracy rate of 97.69%.

The pattern recognition technology described here is a new and simple approach; however, it is quite effective. Therefore, one can conclude that the detection of slender bodies is an efficient approach to overcome the low-density overlapping problem, because it is possible to calculate and approximate the nonvisible part of the objects in an accurate manner. Additionally, one can easily eliminate the problems of the target intersection by utilizing the proposed scheme. Determining the correct searching points is essential before starting the following optimization step; one should remember that in this step the algorithm attempts to find the best searching point that belongs to the same body. This step prevents the merging of pixels that do not belong to the same body, thereby ensuring high-quality detection. In practice, the high-performance approach presented in this study can be applied for solving the low-density overlapping problems of slender objects, and will have comprehensive application in straw uniformity detection for automatic straw-mulching systems.

8. Conclusions

This study investigated a novel image recognition technique for straw uniformity in low-density cases based on overlapping region analysis. The adaptive global threshold method and Otsu’s optimal global threshold segmentation, which are convenient means to separate the objects from the background, were adopted to extract the straws. The debris in the grayscale map was removed according to region characteristics. Additionally, a novel morphological optimization approach was developed to demonstrate the pattern information, and the results verified the effectiveness of the proposed algorithm, which can provide a novel image recognition solution for automatic straw-mulching systems.

Compared with traditional threshold segmentation methods, the advantages of an accurate identification, fast operation, and high efficiency contribute to the better performance of the innovative overlapping region analysis. In conclusion, this approach is suitable for the high-quality detection of slender bodies in low-density overlapping cases, and this technology offers a value reference for investigating the high-density overlapping issues. In future, uniformity detection in high-density overlapping cases will be investigated for better applications in actual production.