1. Introduction
Most crop growers face challenges from numerous agricultural insect pests, which may result in a reduction in crop production. To tackle this problem, integrated insect pest management has been developed and applied in practice. A strong integrated insect pest management system can help farmers apply appropriate treatments to crops and consequently reduce their economic losses. For example, as a crucial component in the integrated insect pest management, monitoring the number of insect pests can assist crop growers to spray the correct amount of pesticides on the correct field.
According to the literature, three methods can be used to obtain the number of insect pests in the field: (i) manual observation; (ii) using multi-dimensional data (e.g., soil temperature and leaf wetness) to estimate the order of magnitude of the insect pest number [
1,
2]; and (iii) capturing insect pest images with trapping devices followed by counting the insect pest numbers via computer-vision-based detection. The first method is too expensive and slow, and the second cannot output an exact value. In contrast, the third one overcomes the disadvantages of the first two methods and thus has become the main research direction of insect pest monitoring.
In the past decades, computer-vision-based insect pest detection has significantly advanced [
3,
4,
5]. Some researchers applied support vector machine using a sliding window to detect planthoppers in paddy fields [
3,
4]. Combined with the bag-of-features model, support vector machine was previously used to recognize vegetable insect pests [
5]. Although support vector machine is a popular machine learning technique, it requires careful engineering and considerable domain expertise for the design of the feature extractor. This hinders its application in practical crop production scenarios.
To overcome the limitations of conventional support vector machine, the emerging deep learning techniques can learn features automatically in an end-to-end manner. As a result, they have been widely applied in smart agriculture as well as other fields [
6], producing remarkable breakthroughs [
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34]. A brief introduction to these approaches is presented in the next section.
For the automatic monitoring of insect pests, researchers generally install sticky traps in the field to monitor insect pests, deploy a camera to obtain images of the installed sticky traps, and finally apply deep-learning-based object detectors to conduct insect pest detection and number counting, as illustrated in [
35]. Although desirable performance has been achieved, two problems are experienced by most state-of-the-art deep-learning-based insect pest detectors. In particular, most deep-learning-based insect pest detectors pay excessive attention to irrelevant background regions during the training phase and thus cannot accurately identify the beneficial and neutral insects from the insect pests in the testing phase. In addition, the non-maximum suppression (NMS) algorithm used in the existing deep-learning-based insect pest detectors ignores the relationship between detection bounding boxes; consequently, a false positive with a higher confidence is generally used to suppress a number of true positives that correspond to the same target but have smaller confidences.
To address these two problems, we developed a new insect pest counter using a saliency map and improved NMS (INMS). Specifically, as most easy negatives, such as background regions, are simple and humans are generally attracted by saliency components in images, we then exploited a saliency map [
24,
36,
37] to filter out most easy negatives of background regions, which in turn would guide the deep-learning-based detector to pay more attention to the hard negatives, such as the beneficial and neutral insects, and thus improve the detection accuracy. In addition, by considering the relationship between detection bounding boxes, we used a merging strategy arising from OverFeat [
13] to integrate the candidate detection bounding boxes with small confidence with those with high confidence, which consequently enhance the robustness of the conventional NMS. By combining the saliency-map-based region proposal generator and the INMS with the conventional general-purpose object detector, we thus constructed a new scheme for insect pest detection and counting. Moreover, as the developed insect pest counter may miscount insect pests that are close to each other, we further integrated the developed insect pest counter with the widely used Faster R-CNN [
15], yielding another insect pest counter with dual-path networks.
To illustrate the feasibility of the proposed scheme, we applied it to count the diamondback moth, which is a key insect pest of vegetables. Extensive experimental simulations showed that our two proposed insect pest counters achieve remarkable improvement in terms of score over the state-of-the-art deep-learning-based object detectors. The dual path network based scheme provides remarkable improvement over the other considered method, and the constructed insect pest counter provides a significant improvement compared to the state-of-the-art insect pest detection method. These findings demonstrate the feasibility and effectiveness of the proposed scheme in monitoring insect pest numbers.
The main contributions of this study are three-fold:
According to characteristics of yellow sticky traps, the saliency map, background–foreground classifier, and tune-up box techniques are used to remove most inessential background regions, which well facilitates the accurate detection of insect pests.
A merging strategy is developed to improve the conventional NMS that is adopted in most deep-learning-based insect pest detectors.
Two saliency-map- and INMS-based insect pest counters are constructed. Extensive simulations of the diamondback moth show that the developed counters achieve a desirable detection rate (DR) with a significantly lower false DR (FDR), providing a significant improvement in terms of score over most state-of-the-art deep-learning-based object detectors and the state-of-the-art insect pest detection methods.
For convenience,
Table 1 summarizes the main abbreviations in this paper. The rest of this paper is organized as follows:
Section 2 introduces related works in the literature.
Section 3 presents the experimental dataset and proposes an alternative insect pest counter. An enhanced dual-path-network-based version is presented in
Section 4. Experimental results and analyses are outlined in
Section 5.
Section 6 draws the conclusion to this study.
3. Materials and Methods
3.1. Dataset
In our work, we used yellow sticky traps installed in a vegetable field to monitor insect pests, and adopted a digital camera fixed in the designed monitoring equipment to obtain images of yellow sticky traps.
Figure 1 illustrates the monitoring equipment and the captured images.
The acquired images were stored in JPEG format with a resolution of 3120 × 4160 pixels. Given the trade-off between performance and computational complexity in model training, we randomly cropped the acquired images to , , , and so forth. In total, we constructed a dataset with 1789 images of yellow sticky traps.
Each image was annotated using the free software, LabelImg [
42], by two trained volunteers in entomology. The annotations including insect pest categories and bounding box coordinates were saved as XML files in PASCAL VOC format, where insect pest categories were the diamondback moth and others. In the initial stage, we mainly focused on the diamondback moth, which is a key insect pest in the vegetable field.
In the simulation, we used 650 images as the training set, 50 images as the validation set, and the other 1089 images as the test set.
3.2. Methodologies
It is reported that multi-stage object detectors, such as Faster R-CNN, usually perform better than the other kinds of object detectors [
19]. Therefore, in this study, we mainly exploited the architecture of multi-stage object detectors to build a more appropriate pest detector for pest counting.
As Faster R-CNN and OverFeat adopt a sliding window to generate candidate regions, a huge imbalance between easy negatives (e.g., regions only containing a patch of the yellow sticky trap) and hard negatives (e.g., regions including a neutral object) would occur. This imbalance would result in the training of the CNN-based object classifier being inefficient because easy negatives tend to overwhelm the training. Although Faster R-CNN employs a region proposal network to eliminate many easy negatives, the region proposal network is implemented in a sliding window fashion, which burdens the classification task with a large number of irrelevant regions.
As the background regions in our scenario were simple and humans are generally attracted by the salient components of a given image, we exploited a saliency map to remove most inessential background regions (i.e., easy negatives), forming the first stage of our proposed scheme. In the second stage, we deployed the widely-used CNN to conduct object classification for each region proposal generated in the first stage, and develop the INMS (i.e., improved non-maximum suppression) to enhance the detection and classification performance.
By following this idea, we constructed a region proposal generator to yield region proposals that probably contain insect pest objects, employed CNN as the classification module, designed an INMS-based module to remove redundant detection bounding boxes, and finally adopted a counter to calculate the number of insect pests on a given yellow sticky trap. The steps form the proposed scheme for insect pest counting, as illustrated in
Figure 2. Details of these modules are presented below.
3.2.1. Saliency-Map-Based Region Proposal Generator
As mentioned above, we exploited the saliency map to filter most of the inessential easy negatives. A saliency map consists of saliency components, such as the edges and textures of a given image, which may well mimic the most relevant regions captured by humans’ attention mechanism.
The yellow sticky trap background in our work is rather simple. Thus, we adopted the efficient method in [
37], which can be easily implemented, to generate the saliency map as illustrated in
Figure 2.
After obtaining the saliency map, we constructed the mask for the saliency map, exploited the mask to produce activation regions, and used a predefined threshold to remove non-target activation regions. Subsequently, we applied the CNN to classify each activation region as the foreground or background followed by imposing tune-up boxes on the foreground region to generate region proposals. These steps form the developed region proposal generator, which are described in detail as follows.
By following [
37], first convert image
of size
into LAB color space, yielding one luminance and two color channels:
,
, and
, respectively. Then, generate the saliency map
via the method in [
37] as:
where
is a
vector containing mean values of
,
, and
;
denotes the Gaussian blurred version of
,
and
, respectively; and
is the
norm.
Produce a mask with . That is, if is larger than or equal to a predefined threshold , is set to be 1; otherwise, is determined as 0.
Use mask to generate activation regions . Specifically, link pixels together to form a connected graph, and then take each connected enclosed graph as an activation region, namely , where .
Discard the activation region if its area is below the preset threshold . This is because these activation regions are usually a small non-target specie (i.e., an other than a diamondback moth).
Classify each
as the background or foreground using classifier
. Suppose that
denotes the bounding box of the ground-truth object, and then define the
as:
where
is the function computing the given area. If all
are smaller than the preset threshold, the activation region of
a is considered a background region; otherwise, it is regarded as a foreground region. The construction of
is presented in
Section 3.2.2.
Yield region proposals . Once the activation region a is classified as the background, remove a; otherwise, add a into , then calculate its central point , and impose k tune-up boxes of preset scales with respect to to . In essence, can be considered as an anchor, which would be much less than those generated by Faster R-CNN.
These steps are summarized in Algorithm 1.
Algorithm 1: Region proposal generator using a saliency map. |
![Insects 12 00705 i001]() |
Compared with the region proposal network of Faster R-CNN that takes all sliding windows of the entire image as candidate region proposals, our region proposal generator uses a saliency map to adaptively produce candidate region proposals. This treatment effectively decreases the inessential background regions and thus reduces the computational load in the object classification.
Compared with the method in [
24], the developed region proposal generator additionally integrates the background–foreground classifier and tune-up boxes for region proposal generation. The background–foreground classifier facilitates the further removal of small and easy negatives and thus contributes to improving the detection accuracy of the subsequent object classifier. The tune-up boxes help to make a fusion decision and thus prevent the whole detector from resulting in mistaken decisions in only one viewpoint. Moreover, the tune-up boxes also provide the benefits of refining the view field of the object classifier in the second stage and addressing the challenges discussed in [
24]: it is hard to properly determine the activation threshold
, that is, a small
may capture too much background noise while a large
cannot well highlight the whole target object, as illustrated in
Figure 3.
3.2.2. CNN-Based Background-Foreground and Object Classifiers
As mentioned in
Section 3.2.1, the background–foreground classifier
is used in the region proposal generator to classify an activation region as the background or the foreground. After region proposals are generated, each region proposal is then input to the object classifier
to determine the category, that is, a diamondback moth or background. Since CNN has powerful classification ability, it is exploited in our method to implement these two classifiers. As both classifiers classify a concerned region into different classes, they can be essentially considered equivalent. Thus, we adopted the same CNN model for both
and
.
As we mainly aimed to find a feasible CNN model instead of constructing a powerful one, we explored a number of popular CNNs and then choose one with desirable performance as the preferable CNN for
and
. As most inessential easy negatives were removed by the saliency map, we first considered a lightweight CNN with an acceptable performance and low computational complexity.
Table 2 lists a lightweight CNN model, CNN
, which was mainly modified from the popular CNNs such as OverFeat, R-CNN and YOLO. CNN
deploys a stride of two and max-pooling because insect pests are small objects and a small stride facilitates the capture of small insect pests.
The CNN model in
Table 2 consists of five layers. The first three layers are convolutional layers and the last two are the fully convolutional layers. Each of the first three layers includes convolution and max-pooling parts. The kernel of the max-pooling part is
, and the structure of the convolution part is represented as
, where
k,
s and
n denote the size of the convolution kernel, the filter stride, and the number of convolution kernels, respectively. The activation function of the CNN model is the sigmoid function.
Although the aforementioned lightweight CNN provides an acceptable performance with low computational complexity, we also considered deeper CNNs such as VGG [
11] to obtain better performance at the cost of acceptable computation time. VGG has two versions, that is, VGG-16 and VGG-19. As the insect pest object in our scenario is small, we adopted the VGG-16 in our work to obtain a suitable receptive field and reduce the computational load as much as possible. The details of VGG-16 are provided in [
11].
Although the same CNN structure is adopted for both the
and
, their hyper-parameters were trained with different data. Classifier
is trained using the input and output data described in Algorithm 1, whereas the input and output data of classifier
are formed as follows. As the
actually classifies a region proposal into a specific category, the input is any region proposal
generated by the region proposal generator (Algorithm 1) and the output is the category label, namely
ℓ, which is determined below. Specifically, suppose that
denotes a ground-truth bounding box. Then define the intersection-over-union value of
b and
as
If there is a such that is larger than or equal to the predefined threshold, the ℓ of b is then determined as the category label of .
3.2.3. Improved Non-Maximum Suppression
Via the object classifier , each region proposal generated by the region proposal generator is classified as one of the target objects or the background. The classified region proposal is then denoted as the detection bounding box. According to Algorithm 1, a number of region proposals may correspond to one insect pest object, and thus some detection bounding boxes may significantly overlap with each other, which causes redundancy. To eliminate redundant detection bounding boxes, the NMS (non-maximum suppression) technique is generally used. Specifically, it first calculates the intersection-over-union between two target detection bounding boxes, say and , then determines them to be overlapped if is larger than a predefined threshold, so the one with with higher confidence remains but the other is removed. This process continues until no overlapped detection bounding boxes can be found.
We adopted the conventional NMS algorithm in our pipeline and found that its performance over the validation set was unacceptable because the insect pests in our scenario have large inter-class similarity; thus, mistaken classification occurred with high probability, which is illustrated as follows: Suppose that denotes a target insect pest and and are two detection bounding boxes related to the . So, the contains the whole while the contains only part of . Due to the high inter-class similarity, has higher confidence than , and and are categorized as two different kinds of insect pests (i.e., is correct, but is wrong). In this situation, the conventional NMS suppresses , which clearly results in error suppression.
By observing that the object classifier could successfully identify the same insect pest at most angles (with respect to
k tune-up boxes), we improved the conventional NMS by considering the relationship between detection bounding boxes, yielding the INMS (improved NMS). Algorithm 2 presents the INMS, where the
function calculates the element number of a set. It first sorts the set of detection bounding boxes
in ascending order of the box’s area, and then denote the detection bounding box with the least area as
. Next, compute the intersection-over-union between
and
, subsequently seek the first
that has the same category label as
and an
is larger than the predefined threshold
, and finally merge
in
followed by resorting the merged result in ascending order of box’ area. If no
can be found, then
is thought to have no overlap with the other detection bounding boxes, and thus
is removed from
and then placed into set
. This process iterates until
is empty. Afterwards, the conventional NMS is imposed on the final set of
and the detection bounding box whose confidence is larger than or equal to the predefined threshold
is chosen as the INMS-handled result.
Algorithm 2: Improved non-maximum suppression using a merging strategy. |
![Insects 12 00705 i002]() |
Our INMS is an integration of the conventional NMS and the merging strategy of OverFeat [
13]. It overcomes the disadvantages of both NMS and OverFeat. That is, INMS tackles the issue of error suppression in the NMS by considering the relationship between detection bounding boxes. INMS also addresses the difficulty in the determination of the proper threshold
for the merging process of OverFeat. In more detail, OverFeat merges rather than suppresses the overlapped detection bounding boxes, and then removes the merged detection bounding box, whose confidence is smaller than a present threshold
. As the confidence of the merged detection bounding box varies widely, a confidence larger than
may still be a false positive. By involving the suppression of conventional NMS to tackle this problem, the false positives are eliminated by a true value with higher confidence.
3.2.4. Insect Pest Counter
After processing with INMS, the redundant detection bounding boxes are removed. Therefore, each insect pest is labeled by only one detection bounding box. By counting the number of these detection bounding boxes resulting from the INMS module, the number of insect pests can be easily obtained.
4. Insect Pest Counter Enhancement Based on Dual Path Network
As described in
Section 3, the proposed insect pest counter exploits a saliency map to remove most inessential easy negatives so that the object classifier can better handle the hard negatives. An improved version of NMS was designed to achieve a more accurate detection performance. These will be demonstrated in the experimental section.
The experimental simulation over the test set, however, showed that the proposed insect pest counter considers insect pests close to each other as one object, which decreases the DR (detection rate) accordingly, as illustrated in
Figure 4. We found that, although the proposed scheme can accurately detect insect pests, it mistook two diamondback moths located close to each other as one because the proposed scheme takes a connected graph constructed from the saliency map as an activation region (Algorithm 1). As a result, insect pests sufficiently close to each other are probably connected to form a connected graph; thus, they are determined as one activation region, so some of the insect pests are not counted.
Faster R-CNN deploys a sliding window to generate candidate region proposals followed by inputting all these region proposals to the object classifier to determine the object category. This scheme increases the DR at the cost of increased FDR (false detection rate). As shown in the bottom row of
Figure 4, Faster R-CNN successfully detects the diamondback moth that was missed by the proposed scheme. However, as Faster R-CNN generates many more region proposals, it would probably mistake a non-target as a diamondback moth and thus result in false detections.
In summary, the proposed insect pest counter achieves a lower FDR with a feasible DR, whereas Faster R-CNN obtains a higher DR at the cost of increased FDR. In other words, the proposed counter and Faster R-CNN are complementary to each other. As such, we integrated the proposed scheme in
Section 3 and Faster R-CNN to construct an enhanced insect pest counter using the dual-path network, as demonstrated in
Figure 5. For notational convenience, the proposed insect pest counter in
Section 3 is denoted
Proposed-I and the enhanced version in this method is called
Proposed-II.
In
Figure 5, the top path depicts the network in
Section 3.2 and the bottom path depicts Faster R-CNN, both of which yield the final detection bounding boxes. As these two paths may result in different detection bounding boxes, we finally designed a fusion module to obtain the final detection and classification results. In more detail, if any detection bounding box
generated by Faster R-CNN overlaps that
yielded by the proposed network in
Section 3.2, that is,
, then
is a reasonable detection bounding box and is retained; otherwise,
is considered to be a false detection bounding box and is eliminated.
As Proposed-II sophisticatedly involves Proposed-I and Faster R-CNN, it integrates the merits of both Proposed-I and Faster R-CNN while overcoming their disadvantages. Thus, a higher DR could be achieved at the cost of a smaller FDR, as demonstrated in the next section.
5. Experiments and Analysis
Next, we evaluated the Proposed-I and Proposed-II, which were described in
Section 3 and
Section 4, respectively. We first outline the experimental settings, then define evaluation metrics, and finally provide results for the ablation and comprehensive experimental simulations.
5.1. Experimental Settings
All experiments were run on a GPU server with 12 Intel® Core™ i7-8700 CPU with 3.20 GHz, 31.3 GiB memory, two GeForce GTX 1080 Ti GPUs and an Ubuntu 16.04 LTS operating system. Python, TensorFlow and PyTorch were employed to implement Proposed-I and Proposed-II as well as the other CNN-based object detectors in the literature.
In the implementation, we randomly initialized all layers of the light-weight CNNs in
Table 2 by drawing weights from a zero-mean Gaussian distribution with a standard deviation of 1, and used the adaptive moment estimation with
,
, and
to train the light weight CNNs. We deployed the Xavier initialization method to initialize all layers of VGG-16, and applied the min-batch stochastic gradient descent with momentum
to train VGG-16. In CNN model training, the cross entropy loss function was adopted for the light weight CNN, and that of entropy with
regularization (scale =
) was set for VGG-16.
In addition, when the receptive field of CNN extended beyond the image border, the missing portions were filled with white pixels [255,255,255], attempting to retain the object size.
5.2. Evaluation Metrics
As introduced in [
3,
4], the DR (detection rate) and FDR (false detection rate) are generally adopted to evaluate the performance of insect pest counting. In essence,
DR is equivalent to the metric of recall and FDR equals
. Recall,
R, and precision,
P, are defined as:
where
TP,
FN, and
FP denote the true positive, false negative and false positive values, respectively.
In our case, the TPs are determined as follows: as we pay more attention to counting accuracy than to locating performance, we did not develop a bounding-box regressor in our insect pest counter. Instead, we considered a detection bounding box b to be “correct” if and , where denotes the ground-truth bounding box. If several “correct” detection bounding boxes exist, only the one with the highest is chosen as the true positive while the others are treated as false positives.
As different approaches generally lead to various DRs and FDRs, it may be troublesome to assess the performance of given approaches. For instance, one method obtains
DR = 0.85 at
FDR = 0.2, and another achieves
DR = 0.92 and
FDR = 0.4, which are difficult to fairly compare. To tackle this issue, we then used the
score as the evaluation metric, which is defined as:
In evaluating the insect pest counting performance, the mean absolute error (
MAE) and mean squared error (
MSE) are generally used. They are defined as:
where
N,
and
denote the total number of test images, the insect pest number in each test image calculated via a certain method, and the true insect pest number in each test image, respectively.
5.3. Experimental Results
We first examined the effectiveness of different modules of Proposed-I via ablation experiments and explore the performance of different backbone networks. We then selected a CNN model with the best performance as the backbone network and conducted comprehensive simulations to compare our methods with the state-of-the-art CNN-based object detectors as well as insect pest detection methods, demonstrating the feasibility of Proposed-I and Proposed-II.
5.3.1. Ablation Experiments on Different Modules
Ablation Experiments on the Background–Foreground Classifier and Tune-up Boxes. As mentioned in
Section 3.2, we exploited a saliency map to mimic the mechanism through by humans select a candidate region, applied the background–foreground classifier to select candidate regions probably containing foreground objects, and imposed
k tune-up boxes to refine the candidate regions and thus yield the final region proposals. As the center of an activation region constructed from the saliency map is used as the anchor, most irrelevant background regions (i.e., easy negatives) are removed, as shown in
Figure 2 and
Figure 3. Below, we mainly focus on assessing the effectiveness of the background–foreground classifier and the tune-up boxes via ablation experiments.
In the simulation, we considered three cases: (1) removing the tune-up boxes; (2) removing the background–foreground classifier; and (3) removing both the tune-up boxes and the background–foreground classifier, which is essentially equivalent to the method in [
24]. For each case, only the concerned part in Proposed-I is eliminated while the others remained unchanged, and the CNN model in
Table 2 was employed as the backbone network of Proposed-I. Each modified network was then trained and tested accordingly.
Table 3 presents the results in terms of DR, FDR and
-score.
We found that, compared with Case 3, Case 1, which adds the background–foreground classifiers, produced a significantly improved score, which indicates that the multi-stage architecture contributes to the effective decrease in FDR. Similar results were also observed for the comparison between Cases 2 and 3, where the tune-up boxes module was inserted for Case 2. These two cases thus demonstrate well the effectiveness of the background–foreground classifier and tune-up boxes developed in our study.
The comparison between Proposed-I and Cases 1–3 showed that integrating both the background–foreground classifier and tune-up boxes considerably improved the
score further. As Case 3 is essentially the method in [
24], this result also demonstrated that Proposed-I actually produced a significant improvement of up to 27.7%(=(75.6 − 59.2)/59.2) over the state-of-the-art insect pest detection method in [
24].
Ablation Experiments on the INMS Algorithm. By considering the relationship between detection bounding boxes, we improved the conventional NMS (non-maximum suppression) via a merging strategy, as described in
Section 3.2.3. To demonstrate the effectiveness of the INMS (improved NMS) on the suppression of false and non-optimal detection bounding boxes, we modified the Proposed-I by replacing the INMS with the conventional NMS and then compared it with the original one, where the CNN model in
Table 2 served as the backbone network for creating an acceptable trade-off between performance and computational complexity.
Table 4 provides the simulation results. INMS is more robust to the false positives and thus helps the Proposed-I achieve remarkable improvements in accuracy.
Ablation Experiments on the CNN Architecture. In our work, we adopted the widely used CNN to build the background–foreground and object classifiers. Given the trade-off between performance and computational complexity, we explored the lightweight CNN in
Table 2 and VGG-16 [
11] via ablation experiments on the CNN architecture. The experimental results are provided in
Table 5. We observed that using a deeper network enabled Proposed-I to effectively increase the DR while decreasing the FDR. As the computational complexity of VGG-16 is acceptable, we thus chose VGG-16 as the CNN model for the backbone network of our proposed scheme.
5.3.2. Comprehensive Assessment of the Proposed Scheme
To further examine Proposed-I and Proposed-II, we compared them with the state-of-the-art CNN-based object detectors including SSD [
18], R-FCN [
16], Faster R-CNN [
15] and YOLOv4 [
38]. We also conduct a comparison with the state-of-the-art insect pest detection method in [
24], which is denoted as LIU for notational convenience. For fair comparison, VGG-16 served as their backbone networks. The experimental results are shown in
Table 6.
Proposed-I achieved the smallest FDR with feasible DR whereas the Faster R-CNN had the highest DR at the cost of a relatively large FDR. Proposed-I attained a larger score than SSD, R-FCN, Faster R-CNN, YOLOv4 and LIU, which well demonstrates the feasibility and effectiveness of Proposed-I. YOLOv4 being inferior to Faster R-CNN may be attributed to the overfitting in YOLOv4.
We investigated the insect pest counting performance. SSD, R-FCN, Faster R-CNN, YOLOv4, LIU, Proposed-I and Proposed-II were first employed to detect diamondback moths in each test image; the detected diamondback moths were then counted. The MAE and MSE in Equation (
2) with respect to the true number of diamondback moths were finally calculated via Equation (
2) and the results are shown in
Table 7. We observed that Proposed-I and Proposed-II achieved a significant improvement in terms of MAE and MSE compared with the other methods. Proposed-II, incorporating the dual networks, produced somewhat better results than Proposed-I, which was expected since Proposed-II simultaneously exploits the benefits of Proposed-I and Faster R-CNN.
5.4. Discussion
Figure 6 visualizes detection and classification results for LIU, SSD, YOLOv4, R-FCN, Faster R-CNN, Proposed-I and Proposed-II. We found that Proposed-I and Proposed-II deal with background samples more effectively using the saliency map and merging strategy. With the saliency map, a large number of easy negatives are eliminated so that the object classifier is able to powerfully handle the hard negatives. With the merging strategy, the confidence in any true positive is increased; consequently, the redundancy eliminator does not perform arbitrary suppression. As a result, significant improvement over the state-of-the-art methods was achieved.
Proposed-I, however, performed unsatisfactorily when insect pests were close to each other because the proposed scheme relies heavily on the saliency map, which does not handle occlusion and overlap situations well. To alleviate this problem, we combined Proposed-I with Faster R-CNN to yield Proposed-II.
Although Proposed-II is promising, it is somewhat time-consuming and the performance can be further improved. For example, instance segmentation [
43] may be applied to attain a better solution. In addition, the greedy merging strategy is sub-optimal, which may lead to false detections. To remedy this, contextual information [
44] and a graph neural network can be employed to better re-score the confidence of the detection bounding boxes.
6. Conclusions
In this paper, we present an insect pest counter using a saliency map and INMS that is composed of four modules, that is, the region proposal generator, object classifier, INMS and insect pest number computation. The region proposal generator exploits a saliency map to effectively discard anchors corresponding to easy negatives, employs a CNN-based background–foreground classifier to select candidate regions that probably contain insect pest objects, and imposes k tune-up boxes on the center of candidate regions to generate region proposals. The object classifier uses a CNN model to determine the category of each region proposal. The INMS is improved from the conventional NMS by considering the relationship between detection bounding boxes, which accurately eliminates the false or sub-optimal detection bounding boxes. The insect pest number computation simply accumulates the number of the final detection bounding boxes. To tackle the issue of the missing detection of objects close to each other, we further integrated the proposed insect pest counter with conventional Faster R-CNN to construct a new dual-path-based insect pest counter. Extensive experimental simulations showed that the proposed two insect pest counters perform significantly better than state-of-the-art methods, demonstrating the feasibility and effectiveness of the proposed two insect pest counters.
In this study, we mainly focused on the diamondback moth, one of the key insect pests in vegetable fields. In our future research, we will extend this work to other key insect pests in vegetable fields, for example, thrips, Bemisia tabaci, and so forth. In addition, as our region proposal generator yields the activation region by forming connected graphs from the saliency map, it may group objects close to each other in a region, resulting in missing detections. Thus, accurately detecting objects close to each other is a challenging problem deserving future research. The deep-learning-based instance segmentation may be a feasible method for addressing this problem.