Real-Time Monitoring System of Seedling Amount in Seedling Box Based on Machine Vision

Abstract

Conventional mat-type seedlings are still widely used in autonomous rice transplanters and automatically supplying seedling devices suited to conventional mat-type seedlings is difficult to develop. Thus, an autonomous rice transplanter carries at least one person to load the seedling pieces into the seedling box, which has led to an increase in the labor force and low operational efficiency. To solve this problem from another point of view, a machine vision-based system for the real-time monitoring of the seedling amount in a seedling box is developed. This system aims to achieve the monitoring of the fault of seedlings and seedling amount in the seedling box. According to the real-time and accuracy requirements of the image, the image acquisition platform is designed based on a previously developed autonomous rice transplanter. A camera model was developed and camera parameters for correcting the image distortion is obtained. The image processing method and segment method of seedling rows are presented. The algorithms for fault diagnosis and the calculation of the number of remaining seedlings are proposed by image analysis. The software is developed for seedling box fault diagnosis and monitoring the remaining number of seedlings. Field experiments are carried out to test the effectiveness of the developed monitoring system. The experimental results show that the image processing time is less than 1.5 s and the relative error of the seedling amount is below 3%, which indicates that the designed monitoring system can accurately realize the fault diagnosis of the seedling pieces and monitor for the remaining amount of each row. By combining the navigation information, the developed monitoring system can predict the distance from which the remaining seedlings in the seedling box can be planted, which can guarantee remaining seedlings in a seedling box are enough for transplanting until the rice transplanter returns to the supplying seedling site. This implies that one person can provide seedlings for multiple autonomous rice transplanters. This study was limited to supplying the seedling when the rice transplanter passed through the place of the seedling storage situated at the headland. In the future, we decide to conduct a study on path planning of breakpoint endurance so that the rice transplanter can automatically return to the supplying seedling place when the seedling amount in the seedling box is not enough.

1. Introduction

With the transfer of the rural labor force to cities and the increase in young and middle-aged laborers unwilling to engage in agricultural production, autonomous agricultural machinery covering plowing, planting, managing, and harvesting has been developed in China [1] and several runs of field demonstration tests have been carried out. However, it has not yet reached the full coverage of the field and long-term continuous and reliable operations are limited by the complex and dynamic farmland environment [1].

Many studies have been conducted on autonomous rice transplanters [2,3,4,5,6]. Straight line tracking, head-turning, and the automatic splicing of rows have been achieved. The lateral deviation is less than 5 cm, which meets the operating accuracy and agronomic requirements of rice transplanting. However, the automatic device which can load rice seedlings for seedling boxes has not been developed. The rice seedling is still needed to load into the seedling box manually, thus the operation of the autonomous rice transplanters continues to need a manual intervention. The complete autonomous operation of the rice transplanter has not been fully developed. An unmanned transplanter currently carries at least one person to load the seedling pieces into the seedling box. The mat-shaped rice seedling was widely applied in rice transplanting. Since the fresh seedlings is the flexible and easily broken and the seedling tray is needed to be removed before loading the seedling pieces into the seedling box, it is difficult to develop the automatic supplying seedling device. Therefore, automatic supply seedling technology is the common key technical problem facing the existing unmanned transplanter.

Conventional rice seedlings were used in the autonomous rice transplanter, where 20 seedling mats were required to transplant an area of 1000 m². The seedling box of a six-row rice transplanter can carry 12 seedling mats at one time [7]. This implies that the seedlings have to be supplied during the process of transplanting, which is a problem for the fully autonomous rice-transplanting operation. To solve this problem, a novelty strategy was proposed in [2]. Long mat-type hydroponic rice seedlings were cultivated and no longer require supplying the additional seedlings during the transplanting operation. Meanwhile, the seedling box in a conventional rice transplanter was modified by adding an appendance to carry long-mat seedlings. The long-mat seedling and the modified seedling box were as shown in Figure 1 [2].

To investigate the effect of hydroponically grown long-mat seedlings (HLMS) on rice yields and growth characteristics, a few comparison studies about HLMS and seedlings via the traditional nutritive soil method (CK) were carried out. The growth and yield of long-mat-type hydroponic rice seedlings were surmised to be almost the same as those of ordinary mat-type rice seedlings cultivated in a soil bed [8]. Subsequently, the quality and field growth characteristics of hydroponically grown long-mat seedlings were investigated and a comparison between CK and HLMS was conducted [9]. The results indicated that the yield and benefit/cost ratio of HLMS were equivalent or superior to those of CK. In addition, the seedling quality of HLMS was better than that of CK, which enabled high-yield and high-efficiency rice production. Regrettably, conventional rice seedlings were mostly used in autonomous rice transplanters despite the advantages of the HLMS. The reasons for this need further research.

Based on advanced precision agriculture and Internet technologies, some relatively mature and feature-rich unmanned farm cloud platforms such as the operations center network [10], farm operation platform called Farmer core [11], TELEMATICS portal and mobile applications [12], an IoT-enabled cloud management platform [13], a remote monitoring platform for agricultural machinery [14], an ecological unmanned farm intelligent cloud platform [15], and an intelligent agricultural machinery operation integrated management and monitoring information platform [16] were developed. These platforms, which were online farm management systems in essence, could be used to manage users, agricultural machinery, and farmland information. The index, such as the equipment performance, future crop yield, operating environment quality, and collaborative operation efficiency of agricultural machinery, was generated, which can be convenient for an examination by the users. Unfortunately, none of these platforms involved the real-time monitoring of fault diagnosis and the seedling remaining amount in the seedling box. Therefore, based on the current situation that an automatic seedling supply device is difficult to develop and an unmanned transplanter needs at least one person to monitor and supply seedlings in the seedling box, it is of urgence to develop a real-time, low-cost, and labor-saving automatic monitoring system of fault diagnosis and measuring the seedling amount in the seedling box.

Machine vision technology has been widely used in many agriculture applications such as weed detection [17], disease detection of crops [18,19], obstacle detection [20], crop row extraction [21,22], detection of crop growth state and yields [23], defect non-destructive detection of horticultural products [24], visual navigation [25,26,27,28], and so on. The navigation control of the crawler tractor based on the integration of satellite and vision was proposed [29]. The image segmentation is achieved using the ExG(2G-R-B), Otsu, and mask method. Then, the region of each rice row is classified by the region search algorithm. A real-time dual-spectral camera system was developed for paddy rice seedling row detection [30]. By applying the AND operation between two thresholded images made from an NIR image and a subtracted image, the noise originating from the reflection of the sky and tree crowns could be eliminated. The crop row detection of maize fields was achieved by a computer vision system [31]. A rice seedling row recognition method based on row vector grid classification was developed [32]. An end-to-end CNN model composed of seedling feature extraction and row vector grid classification was built, and seedling row recognition was converted into a grid classification problem based on the row vectors. These studies indicated that it is feasible to conduct the monitoring of the seedling amount in the seedling box based on machine vision.

Some studies of crop row detection based on deep learning-based methods and YOLOv3 were performed [33,34], which were effective but had high computational costs and many hardware requirements. This has undoubtedly increased the cost of agricultural equipment. The comparison of the main literatures is shown in

Table 1. Comparison of main literatures.

Methods	Source Image	Applications	Results or Accuracy	References
Row vector grid classification	Color images	Recognition of rice seedling rows	89.22%	[32]
System geometry	RGB images	Crop rows and weeds detection in maize fields	Maximum deviation for path following: 7.7 cm for crops with height 14 cm	[31]
Standardization thresholding AND operation	NIR images Red images	Rice seedling row detection	It can successfully detect the seedling rows under cloudy conditions	[30]
Deep learning-based method	Video stream	Bending diagnosis of rice seedling lines and guidance line extraction	/	[29]
Gabor filter PCA K-means clustering algorithm	Raw color image	Orchard robot navigation	RMSE for the lateral deviation was 45.3 cm; maximum trajectory tracking error: 14.6 mm; SD: 6.8 mm.	[28]
Yolov3 and deep SORT	RGB images	Moving obstacle detection	/	[20]
Multi-feature fusion	RGB images	Tree trunk detection and orchard mobile robot localization	Average localization error: 62 mm; recall rate: 92.14%; Accuracy: 95.49%.	[27]

.

The contribution of this study aims to develop a real-time, low-cost, and labor-saving automatic monitoring system for fault diagnosis and measuring the seedling amount in the seedling box. The main contribution is as follows: (1) based on the actual operation requirements, the image acquisition platform of seedlings was developed, which moved as the seedling box moved. Because the seedling box and the platform are relatively stationary, our designed platform can easily acquire the high-quality image and avoid applying the complex image processing method. (2) By the information fusion of navigation data and the remaining seedling amount in the seedling box, the distance can be planted based on the remaining seedlings amount in each row that can be obtained in real-time. According to this distance, the person supplying the seedlings can decide whether there is a need to reload seedlings into the seedling box or not when the rice transplanter travels across the seedling site in the field. (3) To the problem of the seedlings in the seedling box shielding the partition of the lattice, the combination of the background subtraction method and image morphological processing was used to segment the seedling image, which could separate the region of interest from the background. Thus, the individual region of each row was obtained to facilitate the calculation of the remaining seedling amount and the fault diagnosis of each row. (4) One person can provide seedlings for multiple autonomous rice transplanters by using the developed monitoring system.

2. Design of Image Acquisition Platform

A six-row autonomous rice transplanter (2ZG-6DM, Nantong Fulaiwei Agricultural Equipment Co., Ltd., Nantong, China), as shown in Figure 2, was used as the experimental platform.

The proper visual distance and view angle are the premises for collecting high-quality seedling images. Therefore, the image acquisition platform of the seedling box needs to be adjustable in the height and shooting angle, and the platform should have a good stability.

A set of a 3D universal joints from the perspective of practicality and cost-saving was chosen to realize the adjustable angle of the camera. If the platform is too high and too heavy, it will increase the shaking amplitude of the seedling box due to inertia when the seedling box moves horizontally. To meet the requirements of strength and lightness, some seamless steel pipes with a diameter of 20 mm are used. At the bottom of the platform, the triangular principle is used for the reinforcement to improve the overall stability of the platform. Figure 3 shows the structure diagram designed by Solid Works software. The designed image acquisition platform and components are shown in Figure 4.

The camera determines whether the whole monitoring system can meet the working requirements. To improve the diagnosis and calculation accuracy of the proposed monitoring system, it is necessary for a high-resolution camera. However, the high image resolution means a longer image processing time, which is not conducive to the real-time performance of the system. By compromising the real-time performance and resolution, the combination of the high-speed camera (GE200GC-T, Suzhou Mingo Automation Technology Co., Ltd., Suzhou, China) and lens (FA0401-5MP, Guangzhou Long Walk Photoelectric Technology Co., Ltd., Guangzhou, China) is determined. The main performance parameters of the camera and lens FA0401-5MP are shown in

Table 2. Main performance parameters of camera GE200GC-T and lens FA0401-5MP.

Type	Parameters
Camera dimension (mm)	29 × 29 × 40
Size of the target area	1/1.8″
Pixel size (μm)	4.5 × 4.5
Shutter mode	Global shutter
Effective pixel	2 million pixels (600 × 1200)
Resolution @ frame rate	1600 × 1200@60 fps/1280 × 1024@70 fps/800 × 600@118 fps
Data interface	RJ45
Trigger mode	Continuous/soft/hard trigger
Exposure control (ms)	0.016~91
Data format	Mono8p;RGB8p;YUV422p; BayerRG8/10p, BayerGB8/10p;
power supply mode (V)	DC 12
Compatible target area	1/1.8″
Len interface	C-Mount
Focus length (mm)	6
Focusing range (mm)	100~$\infty$
Lens size (mm)	$\phi$38.2 × 27.9

.

3. Camera Calibration

The concavity of the camera lens will distort the image information. To reduce the distortion degree of the image and improve the detection accuracy of the system, it is necessary to calibrate the camera. To obtain the three-dimensional spatial relationship between an object and its image, it is necessary to establish a camera model to solve the camera parameters. The whole process of obtaining camera parameters is called camera calibration.

3.1. Development of Camera Model

The imaging process of the camera is the conversion of the relation between the four reference coordinate systems. The four coordinate systems are the world coordinate system $\left(X_w,Y_w,Z_w\right)$, which can take any point as the origin, the camera coordinate system $\left(X_c,Y_c,Z_c\right)$, which takes the optical center of the camera as the origin, the image coordinate system $\left(uv{o}_0\right)$, which takes the upper left corner of the image as the origin, and the pixel coordinate system $\left(xyo\right)$.

3.1.1. Conversion of Pixel Coordinate System to Image Coordinate System

An image cartesian coordinate system, as shown in Figure 5, was established, where $u$ and $v$ represent the rows and columns of the image, respectively. Pick any point $p\left(x,y\right)$ in the coordinate system. $dx$ and $dy$ represent the true size of a pixel on a photosensitive chip. The relation between the pixel coordinate and the real size of the image can be described as

$$\left\{\begin{array}{l}u=\frac{x}{dx}+u_0\\ v=\frac{y}{dy}+v_0\end{array}\right.$$

(1)

Equation (1) can be rewritten as:

$$\left[\begin{array}{l}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{l}\frac{1}{dx}\hspace{1em}0\hspace{1em} u_0\\ 0\hspace{1em} \frac{1}{dy}\hspace{1em} v_0\\ 0\hspace{1em} 0\hspace{1em} 1\end{array}\right]\left[\begin{array}{l}x\\ y\\ 1\end{array}\right]$$

(2)

3.1.2. Conversion from Camera Coordinate System to Image Coordinate System

A coordinate system $\left(X_c,Y_c,Z_c\right)$ is established with the origin of the photocenter coordinate of the camera (Figure 6). Axis $X_c$ and $Y_c$ are parallel to axis $x$ and $y$, respectively, and axis $z$ is the optical axis of the camera. $f$ is the focal length of the camera, which is determined by the internal parameters of the camera. It can be seen from Figure 6 that $\Delta AB{O}_c~\Delta oC{O}_c$ and $\Delta PB{O}_c~\Delta pC{O}_c$; therefore, the following expression can be obtained.

$$\frac{AB}{oC}=\frac{A{O}_c}{o{O}_c}=\frac{PB}{pC}=\frac{X_c}{x}=\frac{Z_c}{f}=\frac{Y_c}{y}\Rightarrow \left\{\begin{array}{l}x=f\frac{X_c}{Z_c}\\ y=f\frac{Y_c}{Z_c}\end{array}\right.$$

(3)

The corresponding matrix expression can be written as:

$$Z_c\left[\begin{array}{l}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{l}f\hspace{1em}0\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}f\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}0\hspace{1em}1\hspace{1em}0\end{array}\right] \left[\begin{array}{l}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]$$

(4)

3.1.3. Conversion from World Coordinate System to Camera Coordinate System

The conversion from the world coordinate system to the camera coordinate system can be achieved using a rotation matrix $R$ and a translation vector $t$. In homogeneous coordinates, the rotation matrix $R$ can be transformed into a rotation vector of three independent variables by Rodrigues transformation. R is a $3\times 3$ orthogonal identity matrix, and t is the translation vector of the camera with respect to the world coordinate system. Therefore, the rigid body transformation can be represented by three rotation vectors and three translation vectors with a total of six parameters, which are also known as the camera’s external parameters.

The relationship between the world coordinate system and camera coordinate system is described as:

$$\left[\begin{array}{l}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]=\left[\begin{array}{l}R\hspace{1em}t\\ 0\hspace{1em}1\end{array}\right] \left[\begin{array}{l}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(5)

The matrix expression of Equations (2), (4), and (5) can be summarized as:

$$Z_c\left[\begin{array}{l}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{l}\frac{1}{dx}\hspace{1em}0\hspace{1em}{u}_0\\ 0\hspace{1em} \frac{1}{dy}\hspace{1em} v_0\\ 0\hspace{1em} 0\hspace{1em} 1\end{array}\right] \left[\begin{array}{l}f\hspace{1em}0\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}f\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}0\hspace{1em}1\hspace{1em}0\end{array}\right] \left[\begin{array}{l}R\hspace{1em}t\\ 0\hspace{1em}1\end{array}\right] \left[\begin{array}{l}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=\left[\begin{array}{l}{f}_x\hspace{1em}0\hspace{1em}{c}_x\hspace{1em}0\\ 0\hspace{1em}{f}_y\hspace{1em}{c}_y\hspace{1em}0\\ 0\hspace{1em}0\hspace{1em} 1\hspace{1em} 0\end{array}\right] \left[\begin{array}{l}R\hspace{1em}t\\ 0^T\hspace{1em}1\end{array}\right] \left[\begin{array}{l}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(6)

where $f_x$ and $f_y$ are the scaling factors of axles $u$ and $v$, respectively. $\left(c_x,c_y\right)$ is the optical center.

3.1.4. Lens Distortion

Camera lens distortion mainly includes radial distortion along the radius direction of the lens and tangential distortion due to the non-parallelism between the lens and the camera sensor plane or image plane.

The mathematical model of radial and tangential distortions can be written as:

$$\left\{\begin{array}{l}{x}^{″}=x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+x+\left(2p_{1}xy+p_2\left(r^2+2x^2\right)\right)\\ y^{″}=y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+y+\left(2p_{1}xy+p_2\left(r^2+2y^2\right)\right)\end{array}\right.$$

(7)

where $\left(x^{″},y^{″}\right)$ is the coordinate of the distortion point; $k_1$,$k_2$, and $k_3$ are the radial distortion coefficients; and $p_1$ and $p_2$ are the tangential distortion parameters.

3.1.5. Camera Calibration Method

Compared with other calibration methods, Zhang’s calibration method [35] has the advantages of simplicity and a high calibration accuracy, which has been widely used in the field of camera calibration in recent years. Therefore, this calibration method is applied in this study.

Zhang’s calibration method [35] takes the calibration plate as the plane of the spatial coordinate system $X_w{O}_w{Y}_w$. The rotation matrix R is represented using the rotation vector and $Z_w$ is set as 0. The relationship between the world coordinate system and the pixel coordinate system is as follows [35]:

$$Z_c\left[\begin{array}{l}u\\ v\\ 1\end{array}\right]=K\left[r_1\hspace{1em}{r}_2\hspace{1em}{r}_3\hspace{1em}t\right] \left[\begin{array}{l}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=K\left[r_1\hspace{1em}{r}_2\hspace{1em}t\right] \left[\begin{array}{l}{X}_w\\ Y_w\\ 1\end{array}\right]$$

(8)

where $K$ is the internal parameter matrix and $\left[r_1 r_2 t\right]$ is the rotation and translation matrix, denoted as:

$$H=\lambda K\left[r_1\hspace{1em}{r}_2\hspace{1em}t\right]=\left[h_1\hspace{1em}{h}_2\hspace{1em}{h}_3\right]$$

(9)

where $\lambda$ is the scaling factor of $H$ and the homography matrix.

Because $r_1$ is orthogonal to $r_2$, the constraint conditions can be concluded as:

$$\left\{\begin{array}{l}{h}_1{K}^{-T}{K}^{-1}{h}_2=0\\ h_1^T{K}^{-T}{K}^{-1}{h}_1=h_2{K}^{-T}{K}^{-1}{h}_2\end{array}\right.$$

(10)

According to the above formula, the internal parameters of the camera can be deduced as follows:

$$\left\{\begin{array}{l}f=dx\sqrt{\lambda /b_1}\\ c_x=-b_3{f}^2/\left(\lambda k_2\right)\\ c_y=\left(b_2{b}_3-b_1{b}_6\right)/\left(b_1{b}_5-b_2{}^2\right)\\ \lambda =b_9-\left[b_3{}^2+c_y\left(b_2{b}_3-b_1{b}_6\right)\right]/b_1\end{array}\right.$$

(11)

$$\left[\begin{array}{l}{b}_1\hspace{1em}{b}_2\hspace{1em}{b}_3\\ b_4\hspace{1em}{b}_5\hspace{1em}{b}_6\\ b_7\hspace{1em}{b}_8\hspace{1em}{b}_9\end{array}\right]=K^{-T}{K}^{-1}$$

(12)

The external parameter matrix, which is achieved by solving the internal parameter matrix, can be written as:

$$\left\{\begin{array}{l}{r}_1=\lambda K^{-1}{h}_1\\ r_2=\lambda K^{-1}{h}_2\\ r_3=r_1\times r_2\\ t=\lambda K^{-1}{h}_3\end{array}\right.$$

(13)

4. Image Processing

4.1. Image Calibration

The checkerboard grid with the size of 28.5 mm $\times$ 28.5 mm, which is black and white and evenly distributed, was used to calibrate the camera. The checkerboard grid (Figure 7) was printed on A4 paper and laid flat on a stool. The ten pieces of images were selected by adjusting the angle and distance between the stool and the printed checkerboard grid.

The camera calibrator of Matlab 2017b is applied to calibrate the camera. The principle used is Zhang’s calibration method [35] and the calibration results are shown in

Table 3. Calibration results of the camera.

Parameters	Output Results
Internal parameter matrix	[763.1827 0 0; −0.3839 814.2960 0; 631.7540 445.5892 1]
Radial distortion	$k_1$ = −0.0957, $k_2$ = 0.2147, $k_3$ = −0.1740
Tangential distortion	$p_1$ = −0.0100, $p_2$ = 0.0025
Mean error of pixel	0.2430

.

4.2. Image Preprocessing

The image quality of the seedlings directly determines the detection accuracy of the whole monitoring system, and the feature information is the key to accurately identifying the seedlings. The color feature information of the seedlings is different from that of the background. Therefore, this paper conducts relevant research based on the color feature of the seedlings. In this paper, the image processing equipment is a personal PC, and the image processing software is Matlab 2017b.

4.2.1. Grey Scale Processing

The component method and weighted average method are the several grayscale methods widely used at present. The images of each seedling row in the seedling box collected in this paper are color images with a resolution of 1280 × 1024, which are superimposed by three channels R/G/B. The expressions of the component method and the weighted average method can be described by Equations (14) and (15), respectively.

$$g\left(x,y\right)=max\{R\left(x,y\right),G\left(x,y\right),B\left(x,y\right)\}$$

(14)

$$g\left(x,y\right)=a\ast R\left(x,y\right)+b\ast G\left(x,y\right)+c\ast B\left(x,y\right)$$

(15)

where $R\left(x,y\right)$, $G\left(x,y\right)$, and $B\left(x,y\right)$ are the red, green, and blue color component, respectively. $a$, $b$, and $c$ are the weighed coefficients.

The fifteen images were randomly selected from the collected images, and the average values of the R/G/B components in the images $f\left(x,y\right)$ were calculated.

$$f\left(i,j\right)\left\{\begin{array}{l}{R}_a=75.95\\ G_a=90.79\\ B_a=48.88\end{array}\right.$$

(16)

Obviously, the G component is the largest and R is the second largest. The gray-scale effect is enhanced by adjusting the weight of the R/G/B component trial and error. The expression of the grayscale image of the seedlings in the seedling box can be written as:

$$\mathrm{g}\left(\mathrm{x},\mathrm{y}\right)=\left\{\begin{array}{l}0 (\mathrm{G}<\mathrm{R} \mathrm{or} \mathrm{G}<\mathrm{B})\\ 2\mathrm{G}-1.2\mathrm{R}-0.5\mathrm{B} \mathrm{others}\end{array}\right.$$

(17)

Figure 8 shows the grayscale comparison of the G component method and the weighted average method.

The unmanned rice transplanter, which belongs to outdoor operation machinery, needs to work under a natural light source for a long time. The image quality is easily affected by the intensity change in outdoor light, which interferes with the subsequent image processing. Aiming at the problem that farmland image acquisition is easy to be affected by natural light, a combined strategy feature gray method was proposed [36], which has a strong adaptability to outdoor uneven lighting. The gray-scale effect is enhanced by adjusting the weight of the strategy combination. The expression of the combined strategy feature gray method can be written as:

$$\left\{\begin{array}{l}R=\frac{R}{R_{max}}\\ G=\frac{G}{G_{max}}\\ B=\frac{B}{B_{max}}\end{array}\right.\Rightarrow \left\{\begin{array}{l}r=\frac{R}{R+G+B}\\ g=\frac{G}{R+G+B}\\ b=\frac{B}{R+G+B}\end{array}\right.$$

(18)

$$ExG=2g-r-b$$

(19)

$$ExR=1.4r-g$$

(20)

$$CIVE=0.441r-0.811g+0.385b+18.78745$$

(21)

$$VEG=\frac{g}{r^a{b}^{1-a}}, a=0.667$$

(22)

Compared with entropy, uniformity is easier to calculate. Therefore, uniformity is chosen as the detection method in this study. The uniformity is expressed as:

$$U={{\displaystyle \sum }}_{i=0}^{L-1}{P}^2\left(z_i\right)$$

(23)

where $U$ denotes the uniformity; $z_i$ is the random variable of the grayscale level; $P\left(z_i\right)$ represents the histogram; and $L$ is the number of different gray levels.

The uniformity $U_{G_k}$ of each image can be written as:

$$U_{G_k}=\frac{1-U_{G_k}}{{{\displaystyle \sum }}_{G_h\in \Omega }{U}_{G_h}}$$

(24)

where $\Omega =\left\{G_k|k\equiv ExG,ExGR,CIVE,VEG\right\}$, $w_{G_k}$ is a combination strategy coefficient.

$$w_{G_k}=\frac{U_{G_k}}{{{\displaystyle \sum }}_{G_h\in \Omega }{U}_{G_h}}$$

(25)

$$G={\displaystyle \sum _{\Omega }{w}_{G_k}{G}_k}$$

(26)

The effective combination strategy is obtained by acquiring image uniformity:

$$g\left(i,j\right)=0.32\ast ExG+0.29\ast CIVE+0.09\ast VEG+0.3\ast ExGR$$

(27)

The effect comparison for two kinds of grayscale methods is shown in Figure 9.

It was found that the combination strategy feature gray method was better than the ultra-green feature gray method at the edge of the seedlings and in dark light conditions. Therefore, the combination strategy feature gray method is selected to the grayscale image.

4.2.2. Image Enhancement

The outdoor image information collected not only includes the color information of the seedling in the seedling box but also the interference noise caused by various factors such as mechanical jitter, electromagnetic equipment, and the transmission process, which will affect the subsequent processing of the image. Therefore, it is necessary to suppress the noise.

To analyze and compare the adaptability of different filtering methods to the farmland environment, various noise models were added to verify the filtering effect. Gaussian noise and salt and pepper noise, which are common in farmland environments, are analyzed in this study.

The filter processing requirements of the monitoring system of an unmanned rice transplanter in the farmland environment are a high real-time performance and keeping the image as clear as possible. To obtain an effective filter method, the comparison of mean filtering, median filtering, and adaptive median filtering is conducted. The comparison results are as shown in

Table 4. Comparison of filtering methods.

Filtering Method	Gaussian Noise (s)	Salt and Pepper Noise (s)	Enhancement Effect
Mean filtering	0.022	0.062	poor
Median filtering	0.148	0.101	good
Adaptive median filtering	1.538	1.271	optimal

and Figure 10, Figure 11 and Figure 12.

The effect and time of several filtering methods are comprehensively compared. The adaptive median filtering process relies on a constant comparison and iteration update, so the whole processing process is longer than the mean filtering and median filtering, which does not meet the real-time requirement. In summary, median filtering is selected in this paper.

4.2.3. Background Segmentation

Background separation is actually to classify the pixel gray level of the image and then binarize it by setting a threshold to achieve the separation of the target and the background area. For the seedling in the seedling box, the difference between the target area seedling and the background seedling box is obvious, which can separate the seedling region from the background through the threshold segmentation. The iterative selection threshold (IST) method and Otsu algorithm are used, and the segment results as shown in Figure 13 are compared.

The optimum segmentation threshold and average computing time for the IST method are 70.96 and 0.086 s, respectively. However, the corresponding parameters for the Ostu algorithm are 69.74 and 0.007 s, respectively. Although the thresholds obtained by the two segmentation methods are similar, the processing time of the Ostu algorithm is shorter, which is more conducive to the real-time implementation of the system. Hence, the Ostu algorithm is selected in this study.

4.3. Segmentation Algorithm of Rice Seedling Row

Through the above image processing process, the target seedlings were successfully separated as a whole from the seedling box background. However, the height of the seedlings is greater than that of the partition of the lattice of the seedling box. That is to say, the seedlings in the seedling box shield the partition of the lattice (Figure 14). This causes a problem that the seedling amount in each seedling lattice cannot be obtained directly. Therefore, for the convenience of the subsequent separation diagnosis of the seedling pieces and calculation of the remaining seedling amount, it is necessary to separate the seedlings into six rows so that the seedling in each lattice is separated into individual blocks.

4.3.1. Motion Region Segmentation Algorithm Based on Background Subtraction Method

The image acquisition platform is installed above the seedling box. During the transplanting operation of the transplanter, the rice seedling box moves transversely so that the transplanting mechanism can pick the seedling in the seedling box. The image acquisition platform and seedling box are relatively static in the same direction so that the image background information remains unchanged. Thus, we can separate the moving object from the background by using the background subtraction method.

The background subtraction method establishes a suitable background model based on the characteristic of the monitoring object such as a static background or simple movement [37]. Through image matching, all the information containing the background model is removed from the current frame image, and the remaining information is classified into different regions through judgment. Thus, the moving object and the background can be separated. The detailed expression of the background subtraction method can be described as:

$$I{B}_t\left(x,y\right)=\left|I_t\left(x,y\right)-B_t\left(x,y\right)\right|$$

(28)

$$R_t\left(x,y\right)=\left\{\begin{array}{l}1\hspace{1em}I{B}_t\left(x,y\right)>T_1\\ 0\hspace{1em}else\end{array}\right.$$

(29)

where $I_t\left(x,y\right)$, $B_t\left(x,y\right)$, and $I{B}_t\left(x,y\right)$ are the image of the current frame, background image, and background differential image, respectively. The values 1 and 0 denote the moving region and background zone, respectively.

Establishing a suitable background model is the key and difficult problem of the background subtraction method algorithm. The adaptive hybrid Gaussian model [38] provides a reliable idea for model development, which regards the values of the pixels in a video sequence as a time series $\left\{X_1,X_2,...., X_{T_2}\right\}=\left\{I\left(x,y,t\right):1\le t\le T_2\right\}$ and expresses the features of the pixel point $\left(x,y\right)$ by multiple independent Gaussian distributions. The pixels are then matched with the mixed Gaussian model. The pixel point is the background point when $\left|X_{t-\mu ,t-1}\right|\le 2.5{\sigma }_{k,t-1}$, otherwise it is the foreground point. The weights and mean variances of the new images are used to update the parameters of the Gaussian model.

In this study, the image after preprocessing only remains the target region, which reduces the complexity of the interframe image changes. For each pixel, 3~5 Gaussian models are usually required. However, the seedling is simply described as $\left\{1, seedling region; 0, backgrond region\right\}$ by binarization because only the color characteristics of the seedlings were retained in this study. The seedling region of each row, which is considered a background model, is obtained based on the baffle of the seedling box. The background model can be expressed as:

$$g_f\left(x,y\right)=\left\{\begin{array}{l}1\hspace{1em}{g}_{temp}\left(x,y\right)-g_e\left(x,y\right)=0\\ 0\hspace{1em}{g}_{temp}\left(x,y\right)-g_e\left(x,y\right)=1\end{array}\right.$$

(30)

where $g_f\left(x,y\right)$, $g_e\left(x,y\right)$, and $g_{temp}\left(x,y\right)$ are the remaining seedling region of each row, the preprocessed image of the current frame, and the background image model, respectively. When the pixel points of the background model are the same as that of the current frame, it means that the current pixel points are the seedling areas. When the pixel difference between the background model and the current frame is 1, it implies that the area is the background area. A lawn was used to cover the seedling area on the seedling box, and the processed image was used as the background model (Figure 15). The raw seedling image and the segment results obtained by the background subtraction algorithm are shown in Figure 16.

4.3.2. Image Morphological Processing

From the segmented renderings, as shown in Figure 16b, there are some narrow cracks in the seedling image of each area and the shielding of the pressing seedling rod. Therefore, it is necessary to perform morphological processing on these slits and the occlusions to restore the image information. The morphological processing of images mainly includes corrosion, expansion, open and close operation, and cavity filling. Expansion: for the sets $S_1$ and $S_2$ of the elements on $Z^2$, use $S_1$ to expand $S_2$, denoted as $S_2\oplus S_1$, so the detailed expression can be described as:

$$S_2\oplus S_1=\left\{z|{(\stackrel{\wedge }{S})}_z\cap S_2\ne \varphi \right\}$$

(31)

Corrosion: for the element sets $S_1$ and $S_2$ of $Z^2$, use $S_1$ to corrode $S_2$, denoted as $S_2\otimes S_1$, so the detailed expression can be written as:

$$S_2\otimes S_1=\left\{z|{(S)}_z\subseteq S_2\right\}$$

(32)

The expansion was applied to Figure 16b to connect the occlusion areas and the slit region, and then corrosion was used to smooth the edges of the areas. The result was as shown in Figure 17. The narrow cracks in the seedling image of each area were effectively eliminated by using the morphological processing method.

4.4. Image Analysis

4.4.1. Diagnosis of Separation of Seedling Pieces

After the above treatment, the whole seedling region of the seedling box has been completely separated from the background and the individual seedling rows can be seen. The seedlings in the seedling box presents three kinds of states such as the fault, normal, and missing row (Figure 18). The missing row can also be regarded as a special case of the fault state. The leakage transplanting area will be generated if the fault of the seedling pieces (Figure 18a) cannot be found in time.

Through the analysis of the three states, it is found that, under the normal state, the seedlings in the seedling box form a complete region in terms of the units of the row. Based on this phenomenon, the status of each row seedling was monitored by marking the centroid position of each zone. When a fault occurs, two or more areas appear in a row. If the distance between the mass centers of the upper and lower regions is increasing, which implies that the seedling fault state is worsening, the operator must be reminded to eliminate the fault. Otherwise, when the distance between the center of the mass of the upper and lower regions decreases, it indicates that the fault state is weakening and can continue to be observed. When the center of the mass in the lower region is 10 cm away from the bottom of the seedling box, it is necessary to remind the operators to eliminate it if the fault still does not disappear. Based on the above analysis, the flowchart (Figure 19) was designed for fault diagnosis.

4.4.2. Calculation of Seedling Residual Amount in Seedling Box

For the six-row rice transplanters, the seedling box can be regarded as six rows separated by partitions. The remaining number of seedlings in each row determines the operating distance of the rice transplanter. The distance to be planted is related to the seedling area of each row in the seedling box.

Based on the minimization idea, each area is regarded as a square with an area of $\xi$. However, the image has a resolution limit and the minimum unit area is the pixel size, thus the area of each zone can be expressed as:

$$\xi =dx\times dy$$

(33)

The seedling area of each row can be written as:

$$S_i={{\displaystyle \sum }}_{i=1}^{l_i}{\xi }_i$$

(34)

where $S_i$ is the seedling area of each row and $l_i$ is the pixel number of each row seedling.

Supposing $S_m$ is the seedling area of each row when the seedling box is full, then the proportion of the real-time surplus of each row can be expressed as:

$$L_i=\frac{S_i}{S_m}\times 100\%$$

(35)

During the transplanting of the rice transplanter, the seedling amount consumed for planting a row is related to plant spacing, the number of seedlings per grab, and the working distance. The relationship can be written as:

$$d_s=\frac{S_D}{q_i}\times d\times \lambda .$$

(36)

$$L_s=\frac{S_D}{S_m}\times 100\%.$$

(37)

where $S_D$, $d_s$, and $d$ are the consumption amount of seedlings for one-way working, the length of the operating row, and the spacing of the seedlings, respectively, and $q$, $\lambda$, and $L_s$ are the amount of seedlings taken by the planting claw at one time, the transplanting coefficient, and the consumption ratio of single-way transplanting seedlings, respectively.

The transplanting efficiency is:

$$\frac{L_s}{d_s}=\frac{q}{S_m\times d\times \lambda }$$

(38)

The distance can be planted of the remaining seedlings amount in each row, and $d_i$ can be expressed as:

$$d_i=\frac{S_i}{S_m}÷\frac{L_s}{d_s}=L_i\times \frac{S_m\times d\times \lambda }{q}$$

(39)

4.5. Design of Monitoring Software

4.5.1. Function Analysis of Software

The monitoring process of the seedling fault and seedling remaining amount based on the machine vision is shown in Figure 20. First, the seedling box image is acquired and directly transmitted to the PC through the Vision Gige interface. The acquired image is displayed on the PC monitoring interface to complete the system calibration. The calibration parameters of the camera in Table 3 are used to calibrate the system. Second, the fault state and the information on the seedling remaining amount are obtained by the processing and analysis of the image. Then, the image information is sent to the vehicle control unit through the USB_232 serial port for data fusion, and the result is presented. Finally, the raw image, the image after correction, the image after being processed, and the key information of the image (seedling status and seedling remaining amount information) are displayed in the monitoring interface. Moreover, the key information of the image was saved as a .txt document for the convenience of the subsequent system stability analysis. Meanwhile, the seedling information of the seedling box is sent to the vehicle control unit (STM32ZET6) through the serial port.

4.5.2. Image Development Tools and Environment

The image processing algorithm programs in this paper are implemented based on the Matlab platform, and the system monitoring interface is designed and written using the GUI tool in Matlab R2017b software.

4.5.3. Design of Software Data Processing Process

According to software function analysis, the whole fault diagnosis of the seedling pieces and seedling remaining amount monitoring can be divided into two parts. One part is based on Matlab software for system calibration, image acquisition, processing, and analysis. The other is the data fusion and the output of the processing results based on Keil software, as shown in Figure 21.

First, the system is calibrated by using Zhang’s calibration method [35]. After setting the camera’s acquisition parameters, as shown in Table 3, and configuring the camera driver, the seedling images are acquired. Then, to obtain real-time seedling regional information of each row of the seedling box, the collected seedling images are processed by a series of algorithms such as distortion correction, combined strategy grayscale, median filtering, Otsu threshold segmentation, background subtraction method seedling row segmentation, and morphological processing.

Based on the processed images, the fault diagnosis of the seedling pieces and the calculation remaining seedling amount are conducted for each row of seedlings. Then, the image information is converted into digital information for the convenience of the subsequent data fusion.

The vehicle control unit receives the navigation data and the processed seedling image data. The planting distance of the remaining seedling quantity in the seedling box is calculated by calculating the seedling transplanting efficiency.

When the seedling fault occurs, the transplanter stops to travel, and the alarm signal generates. Otherwise, the transplanter will continue to operate if the available seedlings in the seedling box are enough for transplanting to the supplying seedling site in the field. Moreover, when the transplanter travels across the supplying seedling site, if the available seedlings in the seedling box are not enough for transplanting back to the supplying seedling site next time, it will stop to travel and the alarms. The rice transplanter resumes transplanting until the seedling box is fully reloaded and the alarm automatically eliminates.

5. Field Test and Data Analysis

To validate the effectiveness of the proposed monitoring system, the paddy field tests are carried out in the grain industrial park of XINGHUA City, JIANGSU Province.

A 100 m × 30 m rectangular field was selected for the experiment. The unevenness of the paddy soil was less than 5 cm. The mud foot depth in the area is about 25 cm. During the paddy field experiments, the vehicle track with a depth of about 25 cm was generated after the vehicle traveled across the planned paths.

The four longitude and latitude coordinates of the field vertices were collected by the navigation module and transformed into the current field coordinates. According to the operating width of the rice transplanter, the field operation path was independently planned and autonomously generated.

5.1. Performance Test for the Seedling Amount

Before the experiments, the parameters such as the height and view angle of the camera, the spacing of the seedlings, and the number of seedlings taken by the planting claw at one time were previously set. Twelve pieces of conventional mat-type seedlings are carefully loaded into the seedling box. Once the transplanter starts to travel along previously planned paths, the proposed monitoring system starts to work. First, the tests of the seedling amount are conducted. During the experiments, the seedling amount of each row in the seeding box is manually measured with tape from the top of the seedling. To ensure data synchronization from the manual measurement and the monitoring system, the transplanting operation of the autonomous transplanter is stopped during the manual measurement. For each sample, the measurement data with three replications are obtained by manual measurement (real value) and the proposed monitoring system (measurement value). Then, the data with three replications are averaged and the ratio of the current seedling amount of each row to the seedling amount of the full row is calculated. The real value and the measurement value for each row are compared.

Table 5. Real value and the measurement value of the seedling amount.

Real Value (%)	Measurement Value (%)	Relative Error (%)
13.72	12.89	−2.54
24.59	25.13	2.15
35.86	36.61	2.05
46.78	47.08	0.64
53.63	52.98	−1.22
62.69	61.77	−1.49
71.21	73.28	2.82
80.96	79.65	−1.64
88.76	89.01	0.28
96.56	95.89	−0.70

shows the results of the fourth row. Due to space constraints, the measurements of other rows are not provided.

Moreover, the seedling information was saved to the specified path. Txt document to facilitate the subsequent analysis. The experiment results are shown in Figure 22, Figure 23 and Figure 24. The monitoring interface, as shown in Figure 22, mainly displays the original image, corrected image, and processed image as well as the seedling information such as the seedling state in the seedling box and real-time surplus of each seedling row. The seedling information was saved to the specified path. txt document (Figure 23). Figure 24 shows that the navigation controller can receive and display the seedling amount of each row in real-time.

Table 5 shows that the relative error between the real value and the measurement value is below 3%, which implies that the proposed monitoring system achieves enough accuracy.

5.2. Performance Test for Seedling Fault

To verify the diagnosis ability for the seedling fault, the fault of the seedling in each row was artificially set by wrinkling the seedling pieces, which makes the seedling pieces hunch. It is found that the alarm device immediately alarms once the seedling pieces are in a state of fault. The results of many tests indicate that the accuracy of diagnosis is 100%. This signifies that the system has a strong ability to the diagnosis of the seedling fault.

5.3. Performance Test of the Plantable Distance of the Seedling

To examine whether the proposed monitoring system can predict the planting distance based on the number of seedlings in the seedling box, several run field experiments are conducted. The autonomous rice transplanter travels along the previously planned paths. The alarm device of the lack of seedlings is observed when the rice transplanter approaches the seedling storage place in the field. It is found that the alarm device almas when the seedling in the seedling box is insufficient for transplanting the round trips (about 180 m). Meanwhile, the plantable distance of the seedling can be observed from the monitoring interface. This implies that the good performance of the system in terms of predicting the planting distance based on the number of seedlings in the seedling box is achieved.

5.4. Real-Time Test of the System

To investigate the real-time performance of the system, 20 seedling images were analyzed, and the processing time is shown in

Table 6. Image processing time of 20 seedling images.

No.	Time (s)	No.	Time (s)
1	1.455876	11	1.387314
2	1.321593	12	1.399905
3	1.314241	13	1.176554
4	1.297413	14	1.400023
5	1.421732	15	1.441549
6	1.384999	16	1.335629
7	1.405213	17	1.364401
8	1.438917	18	1.365961
9	1.401312	19	1.396413
10	1.404978	20	1.401021

.

Table 6 shows that the image processing time is less than 1.5 s. Based on the velocity of the transplanter and agronomy requirements, the system has a good real-time performance.

6. Conclusions

The current autonomous transplanter needs to carry at least one person for loading the seedling. For this problem, a real-time monitoring system of the seedling fault and seedling amount in the seedling box was presented based on the seedling image information. We focused on the practical operation situation, performed the research on the monitoring system of the seedling fault and seedling amount based on machine vision, and conducted experiments and analyses to verify the effectiveness of the developed monitoring system. Through the above research work and experiments, the following conclusion can be drawn:

(1)

Based on the actual operation requirements, the image acquisition platform of the seedlings was developed, which moved as the seedling box moved. Because the seedling box and the platform are relatively stationary, our designed platform can easily acquire a high-quality image and avoid applying the complex image processing method.

(2)

By the information fusion of navigation data and the remaining seedling amount in the seedling box, the distance can be planted based on the remaining seedlings amount in each row that can be obtained in real-time. According to this distance, the person supplying the seedlings can decide whether there is a need to reload seedlings into the seedling box or not when the rice transplanter travels across the seedling site in the field.

(3)

The seedlings in the seedling box shield the partition of the lattice. To this problem, the combination of the background subtraction method and image morphological processing was used to segment the seedling image, which could separate the region of interest from the background. Thus, the individual region of each row was obtained to facilitate the calculation of the remaining seedling amount and the fault diagnosis of each row.

(4)

The combination of the median filter and Otsu method was determined by comparing it with other filter methods and background separation methods. Our proposed method has a good real-time performance and robustness.

(5)

The experimental results show that the image processing time is less than 1.5 s and the relative error of the seedling amount is below 3%, which indicates that the designed monitoring system can accurately realize the fault diagnosis of the seedling pieces and monitor the remaining amount of each row.

Some limitations in this study still need to be overcome. For example, this study was limited to supplying the seedlings when the rice transplanter passed through the place of the seedling storage situated at the headland. In the future, we decide to conduct a study on the path planning of the breakpoint endurance so that the rice transplanter can automatically return to the supplying seedling place when the seedling amount in the seedling box is not enough. The rice transplanters can go back to the breakpoint and continue to operate when the supplying seedling finishes.