Next Article in Journal
Implications of Non-Farm Work for Clean Energy Adoption: Evidence from Rural China
Previous Article in Journal
Method Research and Structural Design of Segmented Shrimp Clamping
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Design of a Machine Vision-Based Automatic Digging Depth Control System for Garlic Combine Harvester

1
School of Management and Engineering, Nanjing University, Nanjing 210093, China
2
Nanjing Institute of Agricultural Mechanization, Ministry of Agriculture and Rural Affairs, Nanjing 210014, China
3
College of Artificial Intelligence, Nanjing Agricultural University, Nanjing 210031, China
*
Authors to whom correspondence should be addressed.
Agriculture 2022, 12(12), 2119; https://doi.org/10.3390/agriculture12122119
Submission received: 29 September 2022 / Revised: 7 December 2022 / Accepted: 8 December 2022 / Published: 9 December 2022
(This article belongs to the Section Digital Agriculture)

Abstract

:
The digging depth is an important factor affecting the mechanized garlic harvesting quality. At present, the digging depth of the garlic combine harvester (GCH) is adjusted manually, which leads to disadvantages such as slow response, poor accuracy, and being very dependent on the operator’s experience. To solve this problem, this paper proposes a machine vision-based automatic digging depth control system for the original garlic digging device. The system uses the improved YOLOv5 algorithm to calculate the length of the garlic root at the front end of the clamping conveyor chain in real-time, and the calculation result is sent back to the system as feedback. Then, the STM32 microcontroller is used to control the digging depth by expanding and contracting the electric putter of the garlic digging device. The experimental results of the presented control system show that the detection time of the system is 30.4 ms, the average accuracy of detection is 99.1%, and the space occupied by the model deployment is 11.4 MB, which suits the design of the real-time detection of the system. Moreover, the length of the excavated garlic roots is shorter than that of the system before modification, which represents a lower energy consumption of the system and a lower rate of impurities in harvesting, and the modified system is automatically controlled, reducing the operator’s workload.

1. Introduction

Garlic is one of the most important cash crops in China, with a vast cultivation area [1]. As the world’s largest producer, exporter, and consumer of garlic, while China’s garlic production has accounted for more than 60% of the world’s total production, while the export share accounts for about 80% to 90% of the trade volume in the global market [2,3,4]. Garlic production includes tillage, sowing, field management, and harvesting, of which the harvesting process is the most labor-intensive [5]. To improve the harvesting efficiency and reduce the labor cost, researchers have developed a variety of garlic combine harvesters (GCH). These GCHs include the baling and seedling cutting garlic harvesters produced by ERME in France [6], the single-row and double-row garlic baling combine harvesters produced by Poch in Spain [7,8], the HZ-1 garlic harvester developed by Yanmar in Japan [9], and the 4DLB-2 [10] and 4DSL-7 [11] GCHs in China. In garlic harvesting, the control of digging depth is an important prerequisite to ensure the quality of harvesting [12]. While the deep digging depth increases the energy consumption of the harvesters and makes the excavated garlic stay beard bringing more impurities, which leads to additional costs (e.g., cleaning and logistics). The shallow digging depth may damage the garlic bulb and cause unexpected economic losses. To date, the digging device of GCH mainly relies on manual control, which requires experienced operators. Meanwhile, the operators may choose a deeper digging depth to reduce the rate of garlic bulb damage, even if they know it will raise the overall costs. Therefore, it is worth developing an automatic digging depth control system for GCH.
In recent years, scholars have performed some studies on the digging depth limit technologies of the fruit under the soil. The KMC2002 peanut digging and harvesting machine produced by KMC (United States) has an advanced step-less speed regulation digging shovel control mechanism that can adjust the height of the digging shovel in real-time with the change of ground heaving during work to ensure stable digging depth, and its equipped torsion bar spring suspension setting ensures the smoothness during operation [13]. The GT170 series potato digging and harvesting machine from Grimme, Germany, is equipped with depth-limiting profiling wheels, displacement sensors, hydraulic systems, and controllers that can keep the digging depth constant during the operation [14]. You et al. [15] designed a ground profiling device by using the ultrasonic to measure the digging depth and completed the automatic control of the digging depth of the peanut harvester with a fuzzy PID controller. Dai et al. [16] designed a peanut harvester automatic depth limiting system based on the STM32 microcontroller and rotary encoder. Xiong et al. [17] integrated sensor technology, PLC control technology, and hydraulic control technology, and used an integral separation fuzzy PID algorithm to enable the automatic control of the digging depth of the cassava harvester. Li et al. [18] designed a potato harvester digging control system consisting of a front profiling mechanism and a digging mechanism. The angle sensor and displacement sensor were used to monitor the angle between the profiling wheel bracket and the ground and the expansion and contraction of the digging depth cylinder in real-time. The adjustment of the digging depth was completed by driving the hydraulic cylinder through the microprocessor, which improved the account potato rate and reduced the injury and leakage rates. However, most of the research on digging depth-limiting technologies are focused on peanut and potato harvesters, and garlic harvesters have only mechanical structure research in this aspect [1,19,20]. So, there is no research about the automatic digging depth control for garlic harvesters. Moreover, many existing technologies still focus on keeping the digging depth constant, while the digging depth set by them is artificially set before the harvester operation, which is not necessarily the most reasonable digging depth and has certain limitations.
The goal of the depth limit is to choose the appropriate digging depth with the changes in the surrounding environment. This paper proposes a machine vision-based automatic digging depth control system for the GCH. By using the intelligent seeders, the planting depth of garlic is more consistent in the same area [21], the digging depth can be judged according to the length of garlic roots dug out of the soil, and thus the digging mechanism can be controlled to adjust to an appropriate digging depth. This paper modified the original digging mechanism of the 4DSL-7 GCH developed by the Nanjing Institute of Agricultural Mechanization, Ministry of Agriculture and Rural Affairs. The digging mechanism comes with a depth-limiting wheel, and the modified digging device can not only reduce the influence of digging depth brought on by ground heaving but also adjust the digging shovel to an adequate depth position in real-time through the data fed by machine vision. The experiment shows that the system can automatically control the digging depth in the garlic harvesting process. The average root length of the excavated garlic is shorter, which reduces the energy consumption of the system and reduces the rate of impurities in the harvesting process. This paper also provides a reference for the application of machine vision in the field of depth limitation in harvesters.

2. Materials and Methods

2.1. Overall System Structure and Working Principle

The digging depth automatic control system of 4DSL-7 GCH (as shown in Figure 1) mainly consists of digging mechanism, industrial camera, and control system. The overall structure and working principle of the system are shown in Figure 2. When the machine is in operation mode, the lifter holds the garlic seedlings straight, and the digging shovel breaks the main root of the garlic and loosens the soil. Then, the plant enters the aligned seedling-cutting device through the clamping conveyor chain, and the depth-limiting wheel ensures that the excavation depth is not affected by the undulation of the ground. When the digging depth automatic control system starts to work, the industrial camera collects the garlic image information at the front of the clamping conveyor chain and uploads it to the upper computer system. The upper computer system calculates the length of the garlic root out of the soil in real-time according to the image processing algorithm, and thus obtains the expected value of the digging depth before sending the command to the control board in the control box through RS485. The control board sends the control signal to the motor to enable the electric putter to do the telescopic movement (e.g., the up and down adjustment of the digging shovel). During the digging depth regulation process, the data acquisition board monitors the movement status of the electric putter in real-time through the displacement sensor and sends the status to the upper computer system through RS232, thus forming a closed-loop control system for digging depth and ensuring the accurate control of digging depth.

2.2. Main Material Selection

2.2.1. Industrial Camera

The color industrial camera (MV-SUA131GC-T, MindVision) is used in the proposed system. The camera adopts a global shutter, and the image element size is 3.75 × 3.75 μm, the maximum resolution is 1280 × 960, and the adjustable focal length range is 6~12 mm. The camera is connected to the upper computer system through the USB 3.0 port.

2.2.2. Microcontroller

The STM32F103VET6 microcontroller is used to design the control board and data acquisition board. The control board is responsible for sending the control signals based on the sensor dataset collected by the data acquisition board. The microcontrollers used in this paper is a 32-bit RISC processor based on the ARM core framework with a maximum operating frequency of 72 Mhz, and it is widely used in many embedded systems due to its low cost, low power consumption, and high performance [22].

2.2.3. Electric Putter

The electric putter used has a 50 mm stroke and a 2000 N thrust force, which is small and suitable for mounting on the digging support. The main board switch is implemented here to ensure the safety of the motor.

2.2.4. Displacement Sensor

The KPZ miniature linear displacement sensor is selected with a 0~50 mm detection range and 0–5 V analog output signal with 0.015 independent linear accuracies. It is installed on the electric putter to monitor its movement status.

2.3. Software System Design

2.3.1. Machine Vision Measurement Principle

As shown in Figure 1, the industrial camera is installed on the right side of the clamping conveyor chain, the camera’s optical axis is parallel to the ground, and perpendicular to the plane, where the clamping conveyor chain is located. The small-aperture imaging model is established as shown in Figure 3.
Through the principle of similar triangles, Equation (1) can be listed as follows:
A B f = A B d
where f is the camera focal length, d is the object distance.
Since the camera focal length and object distance are constant, the pixel length hpix of AB can be calculated by finding the pixel length of A′B′ in the imaging plane, then the actual physical length h of AB can be obtained by Equation (2) as follows:
h = h p i x ρ
where ρ is the image element size of the camera.

2.3.2. Camera Calibration

Since the MV-SUA131GC-T industrial camera used is a 6~12 mm focusable lens with inherent error, it is necessary to calibrate it to obtain the actual focal length before it was applied. In this paper, the camera is calibrated by the Zhang Zhengyou calibration method [23]. The calibration plate used is an 8 × 6 checkerboard grid with each grid length of 22.5 mm. Twenty sets of calibration plate images with a resolution size of 1280 × 960 are collected, and the calibration results are shown in Table 1.
The average reprojection error of the calibration is 0.12 pixels. From the internal reference matrix of the camera, the pixel focal length of the camera is about 1914 pixels, and the 7.18 mm physical focal length of the camera can be calculated from the image element size.

2.3.3. Target Detection

Before calculating the length of garlic roots, it is necessary to extract the area where the roots are located from the image to determine the location of its upper and lower boundaries. Figure 4 shows an example of the image acquired by the industrial camera.
As shown in Figure 4, the background of the image to be processed is complex. To extract the target area accurately, deep learning algorithms need to be used. Since YOLO v5 is the latest YOLO algorithm, its detection speed may better meet the needs of real-time detection [24], this paper adopts the YOLOv5 algorithm based on deep learning for target detection.
Figure 5 shows the following four parts of the YOLOv5 algorithm [24]: Input, Backbone, Neck, and Prediction. The input side uses Mosaic data enhancement for image stitching to enrich the training dataset and reduce GPU usage. The adaptive anchor box is used to calculate the best anchor box value for different training sets. To improve the target detection speed, adaptive image scaling is applied to add the least black edge to the scaled image. Backbone is composed of Focus structure and CSP structure. While the Focus structure is mainly used for image-slicing operations, the CSP structure borrowed from the CSPNet method is used to ensure the accuracy of detection with reduced computation power by dividing the feature mapping of the base layer into two parts, before merging them through the cross-stage hierarchy. To improve the network feature fusion ability, Neck adopts the FPN + PAN structure, which aggregates parameters from different backbone layers to the detection layers accordingly. Prediction considers three important geometric factors, namely, overlap area, centroid distance, and aspect ratio. While the CIOU_Loss is used in the Prediction phase as the target box regression function to accelerate the speed of model training convergence, nms non-maximum suppression is applied to remove the redundant target box.
To further reduce the space for model deployment, the BN layer in the YOLOv5 network is pruned [25]. The BN layer structure is shown in Figure 6.
The mean and variance of the sample set μSa and σ2Sa are expressed by Equation (3) and Equation (4) as follows:
μ S a = 1 m i = 1 m x i
σ S a 2 = 1 m i = 1 m x i μ S a 2
where Sa = {x1, ..., xm } is the sample set.
Standardization of the sample set x i ^   is calculated by Equation (5) as follows:
x i ^ = x i μ S a σ S a 2 + ϵ
Scaling and translation of the sample are expressed by Equation (6) as follows:
y i = γ x i ^ + β
where γ and β are the parameters trained in the network. The channels with γ close to 0 can be removed since the activation value yi is very small when γ is close to 0.
Since the training parameters of the BN layer are similar to a normal distribution during normal training of the network, the values around 0 are too small to be pruned, and the sparsification of the training parameters is needed. Therefore, the L1 regular constraint is added to the loss function, as shown in Equation (7) as follows:
L = x , y l f x , W , y + λ γ ϵ Γ γ
where λ is the regular term coefficient. Equation (7) can make the BN layer parameters gradually converge to near 0 as the training proceeds. The parameters with smaller weights can be removed by setting the pruning rate to prune the model.

2.3.4. Calculation of Garlic Root Length

The target area is extracted from the raw image to avoid the influence of the ground and the digging mechanism supports the garlic root pose. The extracted target area (see Figure 7) ensures that the garlic roots are all in a natural downward state when the image is processed.
The bounding box predicted by the YOLOv5 algorithm returns four positional parameters, which are the upper left point pixel coordinates (x1, y1) and the lower right point pixel coordinates (x2, y2) of the bounding box. The coordinates of the bulb bounding box are (xg1, yg1) and (xg2, yg2), and the coordinates of the garlic root bounding box are (xr1, yr1) and (xr2, yr2). Combined with the principle of machine vision measurement, the physical length h of the garlic root can be calculated by Equation (8) as follows:
h = y r 2 y r 1 ρ d f
where f is the camera focal length, d is the object distance, and ρ is the image element size of the camera.
Due to the presence of marking errors and the fact that the garlic plant in the test field may have a tilt angle, some corrections to Equation (8) are required.
Figure 8 shows four positional relationships between the bulb bounding box and the garlic root bounding box that can be derived from the sample experiments.
Position 1 is the ideal location of the bounding box, and the garlic root length can be calculated by Equation (8). Position 2 is the location of the bounding box when there is a marking error, as thus Equation (9), which is modified based on Equation (8).
h 2 = y r 2 y r 1 + y r 2 y g 2 ρ d 2 f
Position 3 and position 4 are the cases where the tilt angle of the garlic plant exists. Based on field testing, it is known that the tilt angle of the garlic plant is generally within ±15°, and the diagonal inclination ω of the target rectangular box of the bulb bounding box and the garlic root bounding box increases as the tilt angle of the garlic plant increases. The inclination ω can then be used as a correction factor with Equation (10) as follows:
h 3 = h 2 cos ω
where the inclination angle ω can be obtained by combining the coordinate values with the inverse trigonometric transformation.
A certain number of sample pictures are selected, and the measurement error of the corrected equation is kept within 5 mm to meet the requirements of the control system in this paper.

2.4. Control System Analysis and Design

2.4.1. System Analysis

As shown in Figure 9, the digging depth control system is used to adjust the digging bracket to make rotational movement around the pivot point O through the extension length of the electric putter, so as to indirectly adjust the digging shovel’s digging depth. Since the feedback quantity of the image processing algorithm is the digging depth, and the direct control quantity of the system is the extended length of the electric putter, the correspondence between them needs to be determined first to ensure the normal operation of the closed-loop control system.
In Figure 9, a right-angle coordinate system is established with O’ as the coordinate origin, where the angles α and β and the line segments a, b, c, d, e, and f are known quantities. The relationship between the total length l of the electric putter and the extension s is shown in Equation (11) as follows:
l = 155 + s
where the value of s is in the range of 0–50 mm.
According to the triangle cosine theorem, the relationship between the total length l of the electric putter and ANB can be obtained by Equation (12) as follows:
A N B = a 2 + b 2 l 2 2 a b
The value of ONB and the length of the line segment MO can be calculated according to the value of the ANB. Then, through a series of inverse trigonometric transformations, the value of the inclination angle γ of CO and the coordinates of point C can be obtained by Equation (13) as follows:
C x = O x f cos γ C y = O y f s i n γ + ε
where the coordinates of point O are known as O (1204,245) and ε is the value of the longitudinal coordinates of point C at the initial moment, which is related to the position of the digging shovel mounted on the support.
Take the initial position of the digging shovel as (0, −29), and take a sample for every 0.5 mm change in the electric putter extension s. The displacement transformation curve of point C is shown in Figure 10 below.
With the change of the electric putter extension, the digging shovel makes an approximately circular motion. Since the horizontal displacement of the digging shovel does not affect the control process, only the transformation of the vertical coordinates of the digging shovel is considered. Sampling plots of electric putter extension versus digging depth are plotted in Figure 11 as follows:
As shown in Figure 11, the relationship between the electric putter extension and the digging depth is approximately a straight line due to the small range of variation, so the least squares method is used to fit a straight line, and the fitting results are as shown in Figure 12.
The final parameters of the fitted straight line are k = −1.66 and b = −29.01, therefore, the relationship between the electric putter extension s and the digging depth y can be expressed as Equation (14) as follows:
y = 1.66 s 29.01

2.4.2. System Design

The image processing result and the displacement sensor are used as the feedback, and the extension length of the electric putter is used as the control quantity to form a closed-loop control system for digging depth. The system control flow chart is shown in Figure 13.
Since there is a certain distance between adjacent garlic plants, the garlic root length calculated by image processing cannot be fed back to the closed-loop control system in real-time, resulting in the electric putter not stopping in time when it moves to the desired position. So, a displacement sensor is added to calculate the desired extension of the electric putter according to the garlic root length, and then sends back the current electric putter extension value in real-time through the displacement sensor to control the digging depth precisely.
The results of image processing are divided into the following three cases: without bulb and garlic root, with bulb without garlic root, and with bulb and garlic root. Among them, without bulb and garlic root means that the garlic has not yet entered the clamping conveyor chain and the electric putter does not make any actions. With bulb without garlic root means that the digging depth is too shallow and the desired garlic root length hexp needs to be used as the digging depth adjustment value Δh. With bulb and garlic root means the garlic root length value is calculated normally and the error with the desired value is used as the digging depth adjustment value Δh. Combined with Equation (14), the desired electric putter extension sexp can be found as follows:
s e x p = s + Δ h 1.66
where s is the current electric putter extension, when Δh is negative, the electric putter is controlled to contract, and when Δh is positive, the electric putter is controlled to extend.
Among them, the stopping range of the electric putter is the desired garlic root length ± 5 mm or the desired electric putter extension ± 3 mm. The purpose of setting the stopping range is to reduce the influence of the measurement error on the control system and to prevent the equipment from being damaged by the frequent movement of the electric putter.

3. Results

3.1. Test Platform

The data acquisition board, control board, motor driver, and 24 V to 12 V buck module are installed in the electric control box, which together forms the lower computer control system.
The upper computer processor used in the experiment is an Intel® Core™ i7-9750H at 2.6 GHz with 16 GB of running memory, 512 GB of storage memory, and an NVIDIA GTX1650 GPU. Neural networks were established using the Pytorch1.7.0 deep-learning framework and PyCharm.

3.2. Model Training

In this paper, an industrial camera is used to capture videos of freshly emerged garlic plants. The videos contain garlic plants with different bulb shapes, different amounts of soil adhering to the exterior of the bulb and the garlic root, and different lengths of the garlic root. The valid images are intercepted, and the images are divided into the training set, the validation set, and the test set. The resolution of each image is 800 × 600 pixels. Then the mirror flip and rotation are used for data enhancement on the training and validation sets. The total size of the enhanced data set is 1590 images some of the dataset images are shown in Figure 14. The labeling tool was used to annotate the image dataset with the following two kinds of targets: bulbs and garlic roots. While 1260 images were selected as the training set, 330 images were used as the validation set.
Model training was performed using the server provided by AutoDL (GPU Rental Sites) with the following configuration: a RTX 3090 GPU and a 15-core AMD EPYC 7543 32-Core Processor CPU. Set the network input to 640 × 640, the batch_size to 16, and the number of iterations to 100. The five models of YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x, were trained, and the network depth and width of them gradually increased from YOLOv5n to YOLOv5x. The training results are shown in Figure 15.
The parameters of the model obtained from the training are shown in Table 2.
As shown in Figure 14 and Table 2, with the increase in network depth and width, the model converges faster. However, it will introduce more training parameters, which leads to an increase in model size and a decrease in detection speed. To ensure accuracy while keeping the model deployment space small, the YOLOv5s model is selected in this paper. After the training, the average accuracy of bulbs and garlic roots is 99.4%. The detection results using the trained YOLOv5s model are shown in Figure 16. The red box represents the bulb region and the green box represents the garlic root region. The trained model is able to detect the corresponding target accurately.

3.3. Model Pruning

The initial value of the regular term coefficient is set to 0.001, and its variation with the number of iterations epoch is shown in Equation (16) as follows:
λ = 0.001 1 0.9 e p o c h e p o c h s
where epochs are the total number of iterations.
After sparsifying the training parameters, the model is pruned by setting the pruning rate to 15%. The number of channels in each BN layer before and after pruning is shown in Figure 17, and the model is shown in Table 3.
The number of channels in the BN layer is greatly reduced after pruning, and the space occupied by the model is reduced by nearly 17% compared to that before pruning. Thus, the deployment space is greatly reduced using the pruned model.

3.4. Control System Experiment

Field experiments were conducted in May 2022 in the experimental field of garlic cultivation (Jinxiang County, Shandong Province, zip code 272200). Figure 18 shows the field conditions of the experimental field.
The upper computer monitoring interface designed using pyqt5 is shown in Figure 19.
Set the initial garlic root length to 10 mm, start the digging depth automatic control system, and the change in each state quantity is shown in Figure 20.
The electric putter can adjust the digging depth to a reasonable position in time according to the error between the desired garlic root length and the actual garlic root length. The automatic depth limiting system was turned on and off for comparison, and the harvested garlic root length curves under these two conditions were plotted separately in Figure 21.
As shown in Figure 21, the length of garlic roots harvested with the digging depth automatic control system is shorter compared to the length of garlic roots harvested with the initial system, which greatly reduces the energy consumption during the harvesting process of the GCH and also reduces the rate of impurities.

4. Discussion

Deep learning methods have been proposed for target detection and classification problems that are difficult to solve by traditional image processing methods [26,27,28]. Deep learning is an efficient network structure for feature extraction and target detection [29,30,31]. Currently, target detection algorithms such as Faster R-CNN [32], SDD [33], YOLOv2 [34], YOLOv3 [35], YOLOv4 [36], and YOLOv5 have been developed based on the deep learning framework. Traditional image processing methods have strict requirements on image brightness, background, and surface features of target objects. These methods are less applicable in terms of the large number of equipment needed and the high detection cost. Deep learning can learn the features of target objects under different environmental conditions through powerful convolutional neural networks and can accurately detect the target objects, which is suitable for more complex environments and has good robustness. In the field of agriculture, target detection algorithms based on deep learning have also been widely used, and these research results provide methods and ideas to solve the problem presented in this paper [37,38,39,40]. In this paper, we used the YOLOv5 algorithm to build the bulb and garlic root detection model, and compared the following five models of the YOLOv5 algorithm: YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The experiments showed that the comprehensive performance of the YOLOv5s model was the best. After that, the BN layer of the YOLOv5s model is pruned, which greatly reduces the deployment space of the model and makes it suitable for in-vehicle control systems while ensuring the average accuracy remains basically the same.
Unlike subsoil crops such as carrots, garlic root pan subsoil depth is more consistent, and garlic planting is becoming more and more standardized; therefore, garlic planting depth has good consistency within the same area. According to this characteristic of the garlic crop and the shortcomings of current harvesting machine depth-limiting technology, a machine vision-based method for limiting depth is proposed, and a comparison test is performed in the test field. The results show that the depth-limiting system designed in this paper can adjust the digging depth in real-time according to the detected garlic root length; thus, the harvested garlic leaves a shorter root length compared with the system before the modification, which reduces the workload of subsequent processing of impurities and also provides a certain reference for the application of machine vision in the field of automatic depth-limiting. However, there are also the following shortcomings: only the corresponding experiments were performed in the test field with a good environment, and the system’s operation in a complex environment was not verified; there is still a certain error in detecting the length of garlic roots; only one row of the harvesting channel of the harvester was intelligently modified, and the situation of multi-row harvesting was not considered. Therefore, future work will focus on the improvement of the image processing algorithm to overcome these shortcomings. Moreover, control algorithms such as fuzzy PID and adaptive PID could also be considered to improve the smoothness of the system’s operation.

5. Conclusions

To address the problem that the current automatic depth-limiting technology of the harvesting machines only guarantees a constant digging depth, a machine vision-based automatic digging depth control system for the GCH is designed. The improved YOLOv5 algorithm is used to build the bulb and garlic root detection model, which is used to extract the target area. The garlic root length value calculated from the target area is then used as the feedback value in combination with a displacement sensor to drive the electric putter to adjust the digging depth to the appropriate positions.
Although our method has achieved good results, there are still some spaces that need to be improved. Future work will focus on improving the generalization ability of the detection model to improve the detection accuracy in complex environments. In addition, in terms of the control system, in addition to the improvement of the device, it is necessary to analyze the control model and design a suitable control algorithm to ensure that the system can operate stably.

Author Contributions

A.D.: Conceptualization, Methodology, Software, Visualization, Writing—original draft. B.P.: Conceptualization, Methodology, Resources, Supervision, Writing—review and editing. K.Y.: Methodology, Resources, Supervision, Writing—review and editing. Y.Z.: Data Curation, Investigation, Project administration. X.Y.: Methodology. X.Z.: Writing—review and editing. Z.Z.: Funding acquisition, Writing—review and editing. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Projects funded by the Jiangsu Modern Agricultural Machinery Equipment and Technology Demonstration and Extension (NJ2020-24) and the National Key R&D Program of China (2017YFD0701305-02).

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data that support the findings of this study are available on request from all authors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Shen, S.; Wang, D.; Yin, Y.; He, X.; Shang, S.; Zhao, Z.; Zheng, X. Design and Experimental Analysis of Deep-limiting Excavation Device of Garlic Harvester. Agric. Eng. 2022, 12, 107–111. [Google Scholar] [CrossRef]
  2. Zhao, D.; Cai, D.; Qin, L.; Gao, X.; Huang, W.; Liu, C. Design and Experiment of Modularized Garlic Combine Harvester. Trans. Chin. Soc. Agric. Mach. 2020, 51, 95–102. [Google Scholar]
  3. Yang, K.; Hu, Z.; Yu, Z.; Peng, B.; Zhang, Y.; Gu, F. Design and Experiment of Garlic Harvesting and Root Cutting Device Based on Deep Learning Target Determination. Trans. Chin. Soc. Agric. Mach. 2022, 53, 123–132. [Google Scholar]
  4. Yu, Z.; Hu, Z.; Yang, K.; Peng, B.; Zhang, Y.; Yang, M. Operation Mechanism Analysis and Parameter Optimization of Garlic Root Floating Cutting Device. Trans. Chin. Soc. Agric. Mach. 2021, 52, 111–119. [Google Scholar]
  5. Ma, K.; Zhang, R.; Xia, J.; Huo, L.; Han, Y.; Lv, P. Design and Experimental Study of Garlic Full-automatic Combine Harvester. Agric. Eng. 2019, 9, 90–94. [Google Scholar]
  6. Wang, H.; Li, T.; Wu, Y.; Geng, A.; Li, Y.; Hou, J. Research Status and Prospects of Garlic Harvesting Machinery. J. Chin. Agric. Mech. 2018, 39, 102–107. [Google Scholar] [CrossRef]
  7. Niu, M.; Zhou, J.; Zhang, H.; Fan, B.; Zhu, Z.; Wei, X.; Li, N. Research Advances in Garlic Harvesting Machinery. Agric. Equip. Veh. Eng. 2019, 57, 182–186. [Google Scholar]
  8. Zhang, M.; Sun, Z.; Sun, Z.; Chen, Q. Research Progress of Crop Mechanical Harvesting Technology. J. Anhui Agric. Sci. 2017, 45, 199–203. [Google Scholar] [CrossRef]
  9. Cui, R.; Zhang, H.; Xu, W.; Ma, J.; Sun, G. Discussion on the Present Situation and Development of Garlic Mechanized Production in China. J. Agric. Mech. Res. 2015, 37, 264–268. [Google Scholar] [CrossRef]
  10. Yu, X.; Hu, Z.; Hu, L.; Peng, B. Working Performance Test and Analysis on 4DLB-2 Garlic Combine Harvester. J. China Agric. Univ. 2013, 18, 183–187. [Google Scholar]
  11. Yang, H.; Hu, Z.; Peng, B.; Wang, B.; Zhang, Y.; Yu, Z. Design and Test of Collecting and Conveying Device of Garlic Multi-row Combine Harvester. J. Chin. Agric. Mech. 2021, 42, 8–12. [Google Scholar] [CrossRef]
  12. Chen, Z.; Zhang, C.; Hu, Z.; Wu, H.; Zhang, Y.; Peng, B.; Wang, B.; You, Z. Research Status and Prospects of Digging Depth—Controlled Technology for Under—Soil Fruit. J. Agric. Mech. Res. 2019, 41, 9–14+33. [Google Scholar] [CrossRef]
  13. Yuan, J.; Guo, X. Self-adaptation Control of Chassis System and Digging Shovel in Peanut Harvester. J. Agric. Mech. Res. 2014, 36, 10–13. [Google Scholar] [CrossRef]
  14. Wu, Q. GT170 Series Traction Potato Harvester of Grimme Company. Agricultural Engineering 2015, 5, 161–164. [Google Scholar]
  15. You, Z.; Wu, H.; Hu, Z.; Peng, B. Fuzzy Control on Digging Depth of 4HLB-2 Peanut Harvester. J. Northwest A F Univ. (Nat. Sci. Ed.) 2015, 43, 221–227. [Google Scholar] [CrossRef]
  16. Dai, H.; Dou, Y.; Wang, G.; Zhu, Z. An Automatic Excavation Depth Control System for Semi-feeding Four-row Peanut Combine Harvester. In Proceedings of the 2019 11th International Conference on Intelligent Human-Machine Systems and Cybernetics: IHMSC 2019, Hangzhou, China, 24–25 August 2019; Volume 1, pp. 184–187. [Google Scholar]
  17. Xiong, C.; Zhou, D.; Deng, G.; Li, G.; Cui, Z.; He, F.; Li, L. Design and Test of Automatic Control System for Excavation Depth of Vibrating Chain Cassava Harvester. J. Huazhong Agric. Univ. 2022, 41, 217–226. [Google Scholar] [CrossRef]
  18. Li, T.; Li, N.; Liu, C.; Zhu, Z.; Zhou, J.; Zhang, H. Development of Automatic Depth Control System Employed in Potato Harvester. Trans. Chin. Soc. Agric. Mach. 2021, 52, 16–23. [Google Scholar]
  19. Wang, L.; Li, T.; Niu, Z.; Wu, Y.; Han, K.; Yang, Q.; Jiang, G.; Hou, J. Design of the Sectional Type Garlic Harvester. J. Agric. Mech. Res. 2020, 42, 86–90. [Google Scholar] [CrossRef]
  20. Wu, Y.; Li, T.; Lin, L.; Hou, J.; Zhu, S. Design and Experiment of Disk Digging Device for Garlic. J. Chin. Agric. Mech. 2017, 38, 17–23. [Google Scholar] [CrossRef]
  21. Wang, C. Research on the Intelligent Adjustment and Control of the Garlic Sowing Depth. J. Agric. Mech. Res. 2018, 40, 185–188. [Google Scholar] [CrossRef]
  22. Ding, L.; Song, Z.; Xu, M.; Tao, C. Design of Embedded Measurement and Control System Based on STM32. J. Cent. South Univ. (Sci. Technol.) 2013, 44, 260–265. [Google Scholar]
  23. Zhang, Z. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1330–1334. [Google Scholar] [CrossRef] [Green Version]
  24. Lv, J.; Xu, H.; Han, Y.; Lu, W.; Xu, L.; Rong, H.; Yang, B.; Zou, L.; Ma, Z. A visual identification method for the apple growth forms in the orchard. Comput. Electron. Agric. 2022, 197, 106954. [Google Scholar] [CrossRef]
  25. Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, Lille, France, 6–11 July 2015. [Google Scholar]
  26. Thuyet, D.Q.; Kobayashi, Y.; Matsuo, M. A robot system equipped with deep convolutional neural network for autonomous grading and sorting of root-trimmed garlics. Comput. Electron. Agric. 2020, 178, 105727. [Google Scholar] [CrossRef]
  27. Li, S.; Li, B.; Li, J.; Liu, B.; Li, X. Semantic Segmentation Algorithm of Rice Small Target Based on Deep Learning. Agriculture 2022, 12, 1232. [Google Scholar] [CrossRef]
  28. Zhang, C.; Ding, H.; Shi, Q.; Wang, Y. Grape Cluster Real-Time Detection in Complex Natural Scenes Based on YOLOv5s Deep Learning Network. Agriculture 2022, 12, 1242. [Google Scholar] [CrossRef]
  29. Sun, H.; Li, S.; Li, M.; Liu, H.; Qiao, L.; Zhang, Y. Research Progress of Image Sensing and Deep Learning in Agriculture. Trans. Chin. Soc. Agric. Mach. 2020, 51, 1–17. [Google Scholar]
  30. Xu, H.; Sun, Y.; Cao, X.; Ji, C.; Chen, L.; Wang, H. Apple Quality Detection Based on Photon Transmission Simulation and Convolutional Neural Network. Trans. Chin. Soc. Agric. Mach. 2021, 52, 338–345. [Google Scholar]
  31. Li, X.; Ding, Q.; Sun, J.-Q. Remaining useful life estimation in prognostics using deep convolution neural networks. Reliab. Eng. Syst. Saf. 2018, 172, 1–11. [Google Scholar] [CrossRef] [Green Version]
  32. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [Green Version]
  33. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single Shot MultiBox Detector; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
  34. Redmon, J.; Farhadi, A. YOLO9000: Better, Faster, Stronger. In Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6517–6525. [Google Scholar]
  35. Li, Y.; Guo, J.; Guo, X.; Zhao, J.; Tian, Y. Toward in situ zooplankton detection with a densely connected YOLOV3 model. Appl. Ocean Res. 2021, 114, 102783. [Google Scholar] [CrossRef]
  36. Ec, A.; Lin, T.B.; Zm, A.; Min, Z.C. ERF-YOLO: A YOLO algorithm compatible with fewer parameters and higher accuracy. Image Vis. Comput. 2021, 116, 104317. [Google Scholar]
  37. Choi, J.-Y.; Seo, K.; Cho, J.-S.; Moon, K.-D. Applying convolutional neural networks to assess the external quality of strawberries. J. Food Compos. Anal. 2021, 102, 104071. [Google Scholar] [CrossRef]
  38. Wu, D.; Lv, S.; Jiang, M.; Song, H. Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments. Comput. Electron. Agric. 2020, 178, 105742. [Google Scholar] [CrossRef]
  39. Ye, C.-w.; Yu, Z.-w.; Kang, R.; Yousaf, K.; Qi, C.; Chen, K.-j.; Huang, Y.-p. An experimental study of stunned state detection for broiler chickens using an improved convolution neural network algorithm. Comput. Electron. Agric. 2020, 170, 105284. [Google Scholar] [CrossRef]
  40. Zhao, G.; Quan, L.; Li, H.; Feng, H.; Li, S.; Zhang, S.; Liu, R. Real-time recognition system of soybean seed full-surface defects based on deep learning. Comput. Electron. Agric. 2021, 187, 106230. [Google Scholar] [CrossRef]
Figure 1. Automatic depth limiting structure diagram.
Figure 1. Automatic depth limiting structure diagram.
Agriculture 12 02119 g001
Figure 2. The working principle of the automatic control system of digging depth.
Figure 2. The working principle of the automatic control system of digging depth.
Agriculture 12 02119 g002
Figure 3. Schematic diagram of garlic root length measurement.
Figure 3. Schematic diagram of garlic root length measurement.
Agriculture 12 02119 g003
Figure 4. Image acquisition by the camera.
Figure 4. Image acquisition by the camera.
Agriculture 12 02119 g004
Figure 5. YOLOv5 algorithm structure diagram.
Figure 5. YOLOv5 algorithm structure diagram.
Agriculture 12 02119 g005
Figure 6. BN layer structure.
Figure 6. BN layer structure.
Agriculture 12 02119 g006
Figure 7. Target area extraction.
Figure 7. Target area extraction.
Agriculture 12 02119 g007
Figure 8. Location Relationship (a) Position l; (b) Position 2; (c) Position 3; (d) Position 4.
Figure 8. Location Relationship (a) Position l; (b) Position 2; (c) Position 3; (d) Position 4.
Agriculture 12 02119 g008
Figure 9. Digging depth control mechanism.
Figure 9. Digging depth control mechanism.
Agriculture 12 02119 g009
Figure 10. Coordinate change curve of point C.
Figure 10. Coordinate change curve of point C.
Agriculture 12 02119 g010
Figure 11. Sampling graph of electric putter extension and digging depth.
Figure 11. Sampling graph of electric putter extension and digging depth.
Agriculture 12 02119 g011
Figure 12. Electric putter extension and digging depth fitted straight line.
Figure 12. Electric putter extension and digging depth fitted straight line.
Agriculture 12 02119 g012
Figure 13. System control flow chart.
Figure 13. System control flow chart.
Agriculture 12 02119 g013
Figure 14. Some of the dataset images (a) Long garlic root; (b) Medium garlic root; (c) Short garlic root.
Figure 14. Some of the dataset images (a) Long garlic root; (b) Medium garlic root; (c) Short garlic root.
Agriculture 12 02119 g014
Figure 15. Training results (a) map curve; (b) validation set loss curve.
Figure 15. Training results (a) map curve; (b) validation set loss curve.
Agriculture 12 02119 g015
Figure 16. Detection results.
Figure 16. Detection results.
Agriculture 12 02119 g016
Figure 17. Change in the number of channels in the BN layer before and after pruning.
Figure 17. Change in the number of channels in the BN layer before and after pruning.
Agriculture 12 02119 g017
Figure 18. Field conditions.
Figure 18. Field conditions.
Agriculture 12 02119 g018
Figure 19. Upper computer monitoring interface.
Figure 19. Upper computer monitoring interface.
Agriculture 12 02119 g019
Figure 20. Variation of each parameter.
Figure 20. Variation of each parameter.
Agriculture 12 02119 g020
Figure 21. Garlic root length curves of garlic harvested with an automatic depth limiting system turned off and on.
Figure 21. Garlic root length curves of garlic harvested with an automatic depth limiting system turned off and on.
Agriculture 12 02119 g021
Table 1. Camera calibration results.
Table 1. Camera calibration results.
Camera ParametersParameter Value
Intrinsic Matrix 1914.08 0 0 0 1914.92 0 665.27 450.19 1
Radial Distortion 0.1402 0.4061 3.8165
Tangential Distortion 0 0.003
Table 2. Comparison of detection effects of different models.
Table 2. Comparison of detection effects of different models.
ModelsModel SizeDetection TimeAverage Accuracy
YOLOv5n3.69 MB23.4 ms98.9%
YOLOv5s13.7 MB27.0 ms99.4%
YOLOv5m40.2 MB40.1 ms99.3%
YOLOv5l88.5 MB60.9 ms99.0%
YOLOv5x165 MB110.4 ms99.4%
Table 3. Comparison of models before and after pruning.
Table 3. Comparison of models before and after pruning.
YOLOv5sModel SizeDetection TimeAverage Accuracy
Before pruning13.7 MB30.7 ms99.2%
After pruning11.4 MB30.4 ms99.1%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Ding, A.; Peng, B.; Yang, K.; Zhang, Y.; Yang, X.; Zou, X.; Zhu, Z. Design of a Machine Vision-Based Automatic Digging Depth Control System for Garlic Combine Harvester. Agriculture 2022, 12, 2119. https://doi.org/10.3390/agriculture12122119

AMA Style

Ding A, Peng B, Yang K, Zhang Y, Yang X, Zou X, Zhu Z. Design of a Machine Vision-Based Automatic Digging Depth Control System for Garlic Combine Harvester. Agriculture. 2022; 12(12):2119. https://doi.org/10.3390/agriculture12122119

Chicago/Turabian Style

Ding, Anlan, Baoliang Peng, Ke Yang, Yanhua Zhang, Xiaoxuan Yang, Xiuguo Zou, and Zhangqing Zhu. 2022. "Design of a Machine Vision-Based Automatic Digging Depth Control System for Garlic Combine Harvester" Agriculture 12, no. 12: 2119. https://doi.org/10.3390/agriculture12122119

APA Style

Ding, A., Peng, B., Yang, K., Zhang, Y., Yang, X., Zou, X., & Zhu, Z. (2022). Design of a Machine Vision-Based Automatic Digging Depth Control System for Garlic Combine Harvester. Agriculture, 12(12), 2119. https://doi.org/10.3390/agriculture12122119

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop