Camouflaged Insect Segmentation Using a Progressive Refinement Network

Abstract

Accurately segmenting an insect from its original ecological image is the core technology restricting the accuracy and efficiency of automatic recognition. However, the performance of existing segmentation methods is unsatisfactory in insect images shot in wild backgrounds on account of challenges: various sizes, similar colors or textures to the surroundings, transparent body parts and vague outlines. These challenges of image segmentation are accentuated when dealing with camouflaged insects. Here, we developed an insect image segmentation method based on deep learning termed the progressive refinement network (PRNet), especially for camouflaged insects. Unlike existing insect segmentation methods, PRNet captures the possible scale and location of insects by extracting the contextual information of the image, and fuses comprehensive features to suppress distractors, thereby clearly segmenting insect outlines. Experimental results based on 1900 camouflaged insect images demonstrated that PRNet could effectively segment the camouflaged insects and achieved superior detection performance, with a mean absolute error of 3.2%, pixel-matching degree of 89.7%, structural similarity of 83.6%, and precision and recall error of 72%, which achieved improvements of 8.1%, 25.9%, 19.5%, and 35.8%, respectively, when compared to the recent salient object detection methods. As a foundational technology for insect detection, PRNet provides new opportunities for understanding insect camouflage, and also has the potential to lead to a step progress in the accuracy of the intelligent identification of general insects, and even being an ultimate insect detector.

1. Introduction

Segmenting an insect from its background is the necessary starting point for analyzing anything else about the insect. However, it is a challenging task to segment insects from complicated ecological images due to their various appearances, e.g., size, color and texture, even if they are of the same type [1]. Moreover, under natural selection, many insects have evolved an array of mechanisms that deceive the visual perceptual system of observers to avoid being detected [2]. In this regard, camouflaged insects perform best. However, their blurring of the boundary and lack of the intense contrast required for segmentation approaches further aggravate the difficulties of accurate insect detection [3].

Currently, a large number of image segmentations rely on extracted handcrafted features [4,5,6,7,8,9,10]. “Handcrafted” is a term in machine learning and computer vision referring to the application of some process, such as an algorithm or a manual procedure, to extract relevant features for identification from the raw data (image segmentation tasks) [1]. The handcrafted features are often low-level features, such as color, texture, shape, and appearance, which are sensitive to illumination changes, complicated backgrounds and different object positions [11,12]. Thus, these models tend to suffer from a high misdetection rate with regard to dealing with insect images shot in the wild.

Over the past few years, deep learning and convolutional neural networks (CNNs) have yielded a new generation of segmentation models [13,14,15,16,17,18]. Through the process of deep learning, CNNs extract the low-level features of the raw data via a series of nonlinear mappings and combine them into more abstract features (i.e., high-level features) [19]. These high-level features possess stronger representational ability than traditional handcrafted features and can express the comprehensive information of the original natural images. The full convolution neural network (FCN) [20] is the first method to employ fully convolutional networks for semantic segmentation. Subsequently, the improved methods based on FCN make great progress. Recent methods have achieved excellent results by using dilated convolution [21], deformable convolution [22], the attention mechanism [15,23], feature pyramid spatial pooling [24], and encoder–decoder structure [24,25]. Moreover, segmentation methods based on human hierarchy [26] or prototypes [27] have also received extensive attention. The results are significantly better than previous algorithms, and satisfactory results are achieved on object segmentation.

Methods for object segmentation based on deep learning show significant advantages; however, to distinguish the camouflaged insects from their backgrounds, three major challenges should be resolved: (1) Camouflaged insects have various shapes, scales, and positions, which are difficult to be accurately perceived from the whole images. (2) Due to similar texture or color, some noise from the multi-layer features extracted by CNNs is easily introduced to the fused features. (3) The boundaries of camouflaged insects are blurry, which interferes with segmentation refinement.

Specifically, camouflaged insects often have colors similar to their backgrounds, which makes them difficult to see. Bearing the colors and patterns similar to the background is most people’s conception of camouflage, termed “background picturing” by [28], or “background matching” by [29]. According to [30], camouflage is the ability of prey (or predators) to prevent (or facilitate) predation by changing their features (surface luminance, body pattern, color, texture, or edges) as per that of the environment. Insects have evolved diverse camouflage strategies [31,32], such as background matching, disruptive coloration, transparency, and masquerade [33,34,35]. An insect with background-matching camouflage reproduces the same distribution of simple features as found in the background. Disruptive coloration works at a later stage in visual perception, and suppresses the grouping of simple features into the attributes that could potentially be recognized. Transparency allows light to pass through the whole body or body parts, such as the wings of dragonflies. Masquerade has its effect after perceptual segregation of an object, through mimicry of specific objects within the background.

Examples of camouflaged insects are shown in Figure 1a. The lichen katydid is good at disguising itself as a lichen to adapt to the environment. Kallima inachus tends to rest on dead vegetation and closes its wings to disguise itself as a withered leaf. The stick insect is shaped like a withered or bamboo stem and remains motionless to confuse predators. Phyllium is shaped like a leaf, with bite marks and veins that are similar to those of a leaf. Biston betularia often falls on lichens or stones that are similarly colored to itself. Its larvae can even hold its body in posture to perfectly resemble the shape of twigs and also toggle its body color. Obviously, compared with salient insects (Figure 1b) that are the most attention-seeking to observers, camouflaged insects can hardly be captured rapidly by the naked eye due to their high similarities to twigs, leaves, flowers and other complex backgrounds.

To cope with the above challenges, we developed three novel modules: the asymmetric receptive field (ARF) module, self-refinement module (SRM) and reverse guidance (RG) module. The ARF modules aim to alleviate the effect of various camouflaged insects and improve the ability to distinguish camouflaged insects from backgrounds, in which three parallel and asymmetric convolution layers with multiple receptive fields are adopted to extract anisotropic context information. SRM applies an initial attention strategy on the fused the multi-layer features to suppress distractors of the background to obtain an initial coarse segmentation map. For further outline refinement, RG modules are adopted to enhance the attention to boundaries by erasing the predicted foreground from the side output. The ARF, SRM and RG will further be integrated into the encoder–decoder framework, yielding the progressive refinement network (PRNet). PRNet is expected to improve recognition efficiency, promote camouflage research, and even improve the odds for the development of the ultimate insect detector.

2. Dataset

The COD10K [36] and CAMO [37] benchmarks are the two largest camouflage datasets, covering artificial camouflage, animal camouflage and insect camouflage. After discarding the duplicate insect images, we assembled a total of 1900 ecological images and ground truth pairs from the COD10K and CAMO for this study, which included 10 orders of typical camouflaged insects, such as Coleoptera, Hemiptera, Odonata, Neuroptera, Hymenoptera, Diptera, Lepidoptera, Phasmatodea, Mantodea, and Orthoptera (Figure 2). The number distributions of each order are 8, 130, 65, 26, 24, 4, 629, 127, 140, and 747, respectively.

Examples of some ecological image and ground truth pairs are shown in Figure 3. In each pair, the image is a color picture that contains foreground (objects) and background, and the ground truth is an objective and standard object-level annotation that indicates the location, scale, and shape of the objects in the image. In each image, there exists at least one camouflaged insect. Notably, there exist many challenging properties that were often encountered in real-world shooting, such as shape complexity, indefinable boundaries, occlusions, multiple insects, large insects, small insects, and being out of view. The property description is shown in

Table 1. Property descriptions.

Attribute	Description
Multiple Objects	Image contains at least two insects, e.g., Figure 3b
Small object	Ratio between insect area and whole image area is lower than 0.1., e.g., Figure 3e
Big Object	Ratio between insect area and whole image area is higher than 0.5., e.g., Figure 3d
Complex shape	Insect has thin parts and holes. e.g., Figure 3g
Indefinable boundaries	Insect has a similar color appearance, e.g., Figure 3c
Occlusion	Insect is partially occluded, e.g., Figure 3j
Out-of-View	Insect is clipped by image boundaries, e.g., Figure 3h

.

3. Method

To ‘see’ an object clearly, in general, visual stimuli received by our eyes are processed into neural signals by the visual nervous system and interpreted by visual centers in our brains. Researchers have proven that the information processing of the visual system is hierarchical in the cerebral cortex, and it is a constantly iterative and abstract process [38], as shown in Figure 4. According to this, the progressive refinement network (PRNet) was proposed in this paper. PRNet is composed of the following parts: (a) a backbone network for feature extraction; (b) three asymmetric receptive field (ARF) modules for deriving discriminative features, such as size and direction, (c) a self-refinement module (SRM), for abstracting outline and shape features, and (d) three reverse guidance (RG) modules for paying more attention to insect regions for making sharp outlines.

Figure 5 shows the overall flowchart of our proposed model, which is built on an encoder–decoder architecture. Particularly, for the encoder, we use Res2Net [39] as the backbone, which is capable of capturing more multi-scale semantic features. We define the extracted multi-layer features as low-level features $\{X_0,X_1\}$ and high-level features $X_2,X_3,X_4$. The resolution of each feature $X_k$ is $H/2^{k+1}\times W/2^{k+1},k\in \{0,1,2,3,4\}$, covering diversified feature pyramids from high resolution, weak semantics to low resolution, strong semantics. Note that average pooling layers and the fully connected layers are removed to reduce the loss of details. For the decoder, we first introduce a novel ARF module, whose inputs are middle-level and high-level features, which is necessary to extract anisotropy contextual information in the horizontal, vertical and square kernel modes. Second, we propose a novel SRM with an initial attention strategy in order to aggregate the coarse information from the output of the proposed ARF module and to generate a refined camouflaged map based on the integrated features. Moreover, our SRM can be seen as a special self-attention module that makes full use of more fine-grained information. Finally, in the top-down decoding process, we use an RG module to supplement the missing regions and details of this refined map, which is essential to achieving a high-quality saliency map and clear boundaries. The details of the above are elaborated as follows.

3.1. Asymmetric Receptive Field

Since camouflaged insects often come from natural scenes, their sizes are varied and stochastic. Neuroscience experiments have verified that, in the human visual system, a set of various-sized receptive fields helps to highlight the area close to the retinal fovea, which is sensitive to small spatial information [40]. This motivated us to adopt receptive fields with various sizes to incorporate more discriminative camouflage cues after feature extraction. Additionally, the standard convolution operation of size $(2i-1)\times (2i-1)$ can be factorized as a sequence of two steps with $(2i-1)\times 1$ and $1\times (2i-1)$ kernels, speeding up the inference efficiency without decreasing the representation capabilities [41]. As the $1\times k$ and $k\times 1$ layers have non-square kernels, they are referred to as the asymmetric convolutional layers. Therefore, we proposed the asymmetric receptive field (ARF), which includes five branches $\{b_k,k=1,\dots ,5\}$. In each branch, the first convolutional (Bconv) layer had dimensions of $1\times 1$ to reduce the channel size to 32. This was followed by two other layers: a $(2k-1)\times (2k-1)$ Bconv layer and a $3\times 3$ Bconv layer with a specific dilation rate $(2k-1)$ when $k>2$. The first four branches were concatenated, and then their channel size was reduced to 32 with a $1\times 1$ Bconv operation. Finally, the 5th branch was added, and the whole module was fed to a ReLU [42] function to obtain the feature $\{e{f}_k,k=2,3,4\}$. In brief, compared to the standard receptive field block structure [40], ARF added one more branch with a larger dilation rate to enlarge the receptive field and further replaced the standard convolution with two asymmetric convolutional layers. By using the ARF modules, comprehensive information with integrated anisotropy context from three levels was generated, and the approximate scales of insects in images could be acquired.

3.2. Self-Refinement Module

After obtaining the informative features from the previous ARF modules, in the SRM, we need to preliminarily segment the camouflaged object. Due to repeated down-sampling operations, such as pooling and convolution, the resolution of the prediction map is greatly reduced, which leads to blurred insect outlines. According to the recent evidence [43], high-level features have more global semantic information, which helps to differentiate which features are camouflaged objects in an image. However, due to the lack of details, the camouflage regions are blurred. In contrast, low-level features have detailed information, but it is difficult to determine the camouflage regions. To accurately locate camouflaged insects and obtain sharper outlines simultaneously, it is necessary to integrate multi-level features together. However, feature fusion across multiple levels easily introduces redundant information, resulting in the inaccurate location of targets. Therefore, it is necessary to reduce the differences between the three coarse features. To this end, we used the partial decoder component (PDC) [44] to extract the fusion features that contained high-level information. Such features could then be directly used to generate a coarse camouflaged map $S_c$, via a simple convolutional operation. However, due to the low resolution, the coarse map generated by the fusion features was far from the ground truth mask. Inspired by [45], we formulate the generated coarse camouflaged map as an attention mask, where such an attention mechanism can help denoise the features and generate a preliminary coarse segmentation map. In detail, the inputs of this SRM are the three coarse features $\{e{f}_i,i=4,3,2\}$. We then use a PDC to integrate these features and use a $3\times 3$ Bconv and a $1\times 1$ Bconv to extract a coarse camouflaged map $S_c$ as follows:

$$\begin{array}{c}{f}_d=P_d(e{f}_4,e{f}_3,e{f}_2),\end{array}$$

(1)

$$\begin{array}{c}{S}_c=\mathrm{iBconv}\left(f_d\right),\end{array}$$

(2)

where $P_d$ uses multiplication and concatenation to gradually decrease the gap between different features, and $\mathrm{iBconv}$ denotes multiple Bconv operators. To generate a more accurate camouflage map, we multiply this map $S_c$ with the discriminative features to obtain a discriminative feature $f_r$, which can be described as follows:

$$\begin{array}{c}\hfill f_r=f_d\odot S_c,\end{array}$$

(3)

where ⊙ denotes element-wise multiplying. Finally, we also use a $3\times 3$ Bconv and a $1\times 1$ Bconv to extract a fine-grained camouflaged map $S_r$.

3.3. Reverse Guidance Module

The human vision system pays great attention to the outline of the object, and often obtains the specific shape of the target object through the edge information and interprets the target object and so on. However, a key factor for insect camouflage is edge disruption [46,47,48]. As previously described above, the coarse camouflage map $S_c$ was derived from the three highest layers, which could only capture a probable location of the camouflaged insect, ignoring boundary details. Moreover, direct up-sampling could further introduce more noise and make the boundary non-smooth. To this end, the RG module, which erases the predicted foreground from side-output, is proposed to refine such missing parts or details in the high-level prediction and apply residual architecture to further refine the predicted camouflaged map. As shown in Figure 5, the RG module aims to generate the corresponding edge attention map $W_i$ by using a reverse attention map $R_i$. We further split the feature $X_i$ with C channels into n groups (the number of channels in each group is c), and concatenate it with n reverse attention maps $R_i$ so as to guide the features to focus on boundaries. To obtain a more complete camouflaged map, we iteratively add the predicted result of the latter layer $S_{i+1}$ to the corresponding edge attention map $W_i$, which can be described as follows:

$$\begin{array}{c}{W}_i=R_i\odot X_i,\end{array}$$

(4)

$$\begin{array}{c}{x}_i^1,...,x_i^m,...,x_i^n=\mathrm{split}\left(X_i\right),\end{array}$$

(5)

$$\begin{array}{c}{F}_i=\mathrm{concat}(x_i^1,W_i,...,x_i^m,W_i,...,x_i^n,W_i),\end{array}$$

(6)

$$\begin{array}{c}{S}_i=\mathrm{iBconv}\left(F_i\right)+S_{i+1}\end{array}$$

(7)

Note that this reverse attention map $R_i$ is obtained by erasing the foreground in the prediction, and it can be formulated as

$$R_i=1-\sigma \left(U\left(S_{i+1}\right)\right),$$

(8)

where $\sigma$ is the sigmoid function, and U is the up-sampling operation. Having access to high-level information, lower levels can learn more powerful features for refining the camouflage map in a coarse-to-fine manner, and output the final segmentation result, which is a binary bitmap.

In short, ARF captures contextual information from multi-layer features for coarse-grained refinement of fusion features. For fine-grained refinement, SRM and RG modules cover more useful information by applying an initial attention strategy on fusion features, and erasing the foreground to pay more attention to boundaries, respectively. These three modules progressively refine features from coarse to fine so as to achieve accurate segmentation. Finally, we integrate the ARF, SRM and RG into the encoder–decoder architecture, and the entire network can be trained end to end.

3.4. Implementation and Evaluation

To evaluate the performance of PRNet, 5-fold cross validation was adopted in this research. The camouflaged insect dataset was randomly split into 1520 images (80%) for training and 380 images (20%) for validation in each fold. The proposed method was implemented on the PyTorch platform. We used the Adam optimizer with a learning rate $1\times {10}^{-4}$ for training, in which the epoch size was 40, and the batch size was 16. The encoder was initialized by the weights of Res2Net that is pre-trained on ImageNet. We resized all input images to $352\times 352$ and used images with three scale rates $\{0.75,1,1.25\}$ during training.

PRNet is a supervised segmentation network to predict each pixel to be the insect or background, thus it is trained by minimizing the pixel position-aware (PPA) loss [49] of camouflaged maps $\{S_2,S_3,S_4,S_c,S_r\}$. PPA loss assigns different weights to different positions and pays more attention to hard pixels. PPA loss $L_{ppa}$ is formulated as

$$L_{ppa}=L_{wbce}+L_{wiou},$$

(9)

where $L_{wbce}$ is a weighted binary cross entropy loss and $L_{wiou}$ is a weighted intersection over Union (IoU) loss. The $L_{wbce}$ loss function is formed as the following:

$$L_{wbce}=-\frac{1}{N}\frac{{\sum }_{i,j}\left(1+\gamma {\alpha }_{ij}\right)\left[g_{ij}log\left(p_{ij}\right)+\left(1-g_{ij}\right)log\left(1-p_{ij}\right)\right]}{{\sum }_{i,j}\gamma {\alpha }_{ij}},$$

(10)

where $g_{ij}$ and $p_{ij}$ represent the predicted values and ground truth of the pixel at location (i,j), respectively. N denotes the total number of pixels in an image, and $\gamma$ is a hyperparameter. The weight $\alpha$ is calculated according to the difference between the center pixel and its surroundings, which can be defined as follows:

$${\alpha }_{ij}=\left|\frac{{\sum }_{m,n\in A_{ij}}{g}_{mn}}{{\sum }_{m,n\in A_{ij}}1}-g_{ij}\right|,$$

(11)

where $A_{ij}$ is the area surrounding the pixel (i,j). If $a_{ij}$ is large, the pixel at (i,j) is very different from its surroundings, which may represent an important pixel (e.g., edge) and deserves more attention. Similarly, $\alpha$ is assigned to $L_{wiou}$ for emphasizing the importance of hard pixels, which can be defined as

$$L_{wiou}=1-\frac{1}{N}\frac{{\sum }_{i,j}\left(1+\gamma {\alpha }_{ij}\right)g_{ij}{p}_{ij}}{{\sum }_{i,j}\left(1+\gamma {\alpha }_{ij}\right)\left(g_{ij}+p_{ij}-g_{ij}{p}_{ij}\right)}$$

(12)

In this paper, all of the output segmentation maps are upsampled to the same size as the ground truth G. Thus, the total loss can be defined as

$$L_{total}=L\left(\mathit{G},\sigma \left({\mathit{R}}_g^{\mathrm{up}}\right)\right)+\sum _{t=1}^{5}L\left(\mathit{G},\sigma \left({\mathit{R}}_t^{\mathrm{up}}\right)\right)$$

(13)

4. Results

4.1. Evaluation Metrics

Four popular metrics were utilized to evaluate its performance in camouflaged insect segmentation, i.e., mean absolute error (MAE) [50], enhanced-alignment measure ($E_{\mathsf{\Phi }}$) [51], structural similarity measure ($S_{\alpha }$) [52], and weighted harmonic mean of precision and recall ($F_{\beta }^w$) [53]. For the predicted camouflage map C and its corresponding ground truth G, the image size is $W\times H$.

MAE calculates the pixel-wise absolute difference between the predicted camouflage map and ground truth, which is defined as

$$MAE=\frac{1}{W\times H}\sum _{i=1}^W\sum _{j=1}^H\left|C(i,j)-G(i,j)\right|,$$

(14)

where H and W are the height and the width of the map, respectively.

$E_{\mathsf{\Phi }}$ is defined as

$$E_{\mathsf{\Phi }}=\frac{1}{W\times H}\sum _{i=1}^W\sum _{j=1}^H{\mathsf{\Phi }}_{FM}(i,j),$$

(15)

which can obtain image-level statistics and local pixel-matching information, based on the enhanced alignment matrix ${\mathsf{\Phi }}_{FM}$.

$S_{\alpha }$ uses the object-aware and region-aware structure similarities, namely $S_o$ and $S_r$, to obtain structural similarity between the predicted camouflage map and ground truth, and is formulated as

$$S_{\alpha }=\alpha \ast S_o+(1-\alpha )\ast S_r,$$

(16)

where $\alpha$ = 0.5.

$F_{\beta }^w$ takes both the weighted precision ($Precisio{n}^w$) and weighted recall ($Recal{l}^w$) into account, which is formulated as

$$F_{\beta }^w=(1+{\beta }^2)\frac{Precisio{n}^w·Recal{l}^w}{{\beta }^2·Precisio{n}^w+Recal{l}^w},$$

(17)

where ${\beta }^2$ = 0.3.

4.2. Comparisons with State-of-the-Art Detection Methods

Our proposed PRNet was compared with seven recently published methods, including two detection methods designed for salient objects, i.e., BASNet [54] and F3Net [49], and five detection methods designed for camouflaged objects, i.e., SINet [44], PraNet [12], SINet-V2 [36], PFNet [55] and C2FNet [3]. For fair comparison, all results of these models were retrained with our camouflaged insect dataset, and the parameters were set as recommended in the corresponding papers. The open source codes were taken from http://dpfan.net/camouflage/ (accessed on 10 September 2022) and https://github.com/jiwei0921/SOD-CNNs-based-code-summary- (accessed on 10 September 2022). In addition, all the prediction maps are evaluated with the same code.

Quantitative Evaluation: The quantitative results of our PRNet against seven other object detection methods on the camouflaged insect dataset are presented in

Table 2. Quantitative comparison with state-of-the-art methods for camouflaged insect detection. $\uparrow /\downarrow$ indicates that larger or smaller is better. The three best results are in red, green and blue colors, respectively.

Method	Year	Field	Backbone	$\mathit{MAE}\downarrow$	${\mathit{E}}_{\mathit{\Phi }}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$
BASNet [54]	2019	Camouflage	ResNet-50	0.068	0.770	0.692	0.483
F3Net [49]	2020	Salient	ResNet-50	0.113	0.638	0.641	0.362
SINet [44]	2020	Camouflage	ResNet-50	0.050	0.872	0.803	0.567
PraNet [12]	2020	Camouflage	Res2Net-50	0.040	0.857	0.801	0.662
SINet-V2 [36]	2022	Camouflage	Res2Net-50	0.033	0.889	0.836	0.719
PFNet [55]	2021	Camouflage	ResNet-50	0.033	0.893	0.83	0.715
C2FNet [3]	2021	Camouflage	Res2Net-50	0.035	0.887	0.833	0.712
PRNet	2021	Camouflage	Res2Net-50	0.032	0.897	0.836	0.720

. Obviously, the performance of salient object detection methods, both BASNet and F3Net was not as good as that of all camouflaged object detection methods. Moreover, most models that employed Res2Net-50 as the backbone outperformed the models that employed ResNet-50. In terms of camouflaged object detection methods, PRNet was superior to PraNet and SINet which were also built on a customized UNet-based architecture. In particular, our method achieved the best performance on camouflaged insect detection with an MAE of $3.2\%$, $E_{\mathsf{\Phi }}$ of $89.7\%$, $S_{\alpha }$ of $83.6\%$ and $F_{\beta }^w$ of $72\%$, which were improved by $0.1\%$, $0.4\%$, $0.3\%$ and $0.1\%$, respectively, when compared to the second-best method.

Qualitative Evaluation: Furthermore, some visualization results are shown in Figure 6. Compared with other state-of-the-art models, our model achieved better visual effects by detecting more accurate and complete camouflaged objects with rich details. BASNet failed to detect some insects (see the 1st, 2nd and 3rd (last) rows). All boundaries of insects detected by F3Net were blurred. In particular, SINet and PraNet were weak at detecting the thin parts, such as the antennas (see 8th row) and feet (see 1st row). SINet-V2, PFNet and C2FNet interfused the noise from background in the last row and the last 4th row, in which the shapes of insects were complex.

4.3. Effectiveness Verification of Each Module

To verify the effectiveness of each key module, they were removed from the complete model for contrast tests. Note that three tests were performed under the same training settings, and the results are summarized in

Table 3. Results of the effectiveness verification. The best results are highlighted in bold fonts.

No.	Module			MAE↓	${\mathit{E}}_{\mathit{\Phi }}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$
	ARF	SRM	RG
1		√	√	0.033	0.89	0.834	0.715
2	√		√	0.040	0.897	0.831	0.657
3	√	√		0.161	0.830	0.672	0.326
4	√	√	√	0.032	0.897	0.836	0.720

. In the No.1 (SRM+RG) experiment, we replaced the ARF module with a convolution operation to adjust the features of the last three layers to a consistent number of channels. In the No.2 (ARF+RG) experiment, we removed the SRM module while keeping the ARF and RG modules, and then fused the features of the last three layers to generate an attention map via the combined operation of up-sampling and multiplication. In the No.3 (ARF+SRM) experiment, we replaced the RG with the combined operation of up-sampling and concatenation, and the ARF and SRM remained unchanged. No.4 (ARF+SRM+RG) was the complete model, consistent with the structure shown in Figure 5.

Compared with the complete model (No.4), removing the ARF module (No.1) slightly degraded the detection performance. The reason for this phenomenon was the fact that the asymmetric receptive field brought the semantic strength and location accuracy of high-level features into full play but also inevitably introduced noise and fuzzy edges for the target object. When removing the SRM, i.e., No.2, the performance declined in several respects as well, which suggested that the lack of details went against extracting coarse camouflage maps. In particular, comparing No.3 with No.4, the use of RG significantly improved by 12.9%, 6.7%, 16.4%, and 39.4% in terms of MAE, $E_{\mathsf{\Phi }}$, $S_{\alpha }$, and $F_{\beta }^w$, respectively. This result demonstrated that employing the RG module to mine edge information could help the model overcome the challenges in camouflaged object detection, such as complex shapes and indefinable boundaries. Thus, the contrast tests validated the rationality of integrating these modules into PRNet.

4.4. Generalization Verification

The generalization ability of the neural network model is of vital importance for its popularization and application. In other words, a good deep learning model should not only perform well on the existing images in academic datasets, but also perform well on fresh images. To further assess the generalization capability of the proposed PRNet, some camouflaged images from CHAMELEON. were directly fed into PRNet without retraining or any tune-up.

On the dataset CHAMELEON, our PRNet achieved the performance with an MAE of $3.8\%$, $E_{\mathsf{\Phi }}$ of $84.1\%$, $S_{\alpha }$ of $83.1\%$ and $F_{\beta }^w$ of $65.7\%$. According to the visual results shown in Figure 7, despite the outlines of segmentation maps being vague, PRNet could segment most parts of camouflaged objects, indicating that PRNet might provide more chances for the discovery and detection of various camouflaged objects. For further accurate segmentation performance on various camouflaged objects, retraining the PRNet on a large number of images of relevant objects is the most practical and effective method.

5. Discussion

This work proposed an end-to-end network PRNet that achieved the best performance on camouflaged insect detection compared with seven state-of-the-art detection methods designed for salient objects and camouflaged objects. PRNet integrated three key modules: ARF, SRM, and RG. The ARF module captured rich contextual information on the camouflaged images. SRM obtained a coarse segmentation map by suppressing distractors of the background. The RG module increased attention to insect edges by integrating reverse attention into multi-layer features.

As one of the toughest cases, camouflaged insect detection was often accompanied by weak boundaries, low contrast, and similar texture to the backgrounds, and thus, there were more challenges for camouflaged insect detection than salient insect or generic insect detection. Note that generic insects could be either salient or camouflaged, and camouflaged insects could be seen as difficult cases. Multi-level feature aggregation was explored for robust detection [56,57]. Recurrent and iterative learning strategies were also employed to progressively refine the prediction map [58,59]. Due to the effectiveness of feature enhancement, attention mechanisms [60,61] have also been applied to saliency detection [62,63]. In addition, edge cues were leveraged to refine the saliency map [54,64]. However, directly applying salient insect or generic insect detection methods to detect camouflaged insects may not yield the desired results, as the term “salient” was essentially the opposite of “camouflaged”, i.e., standout versus immersion. This view was underpinned by quantitative and qualitative results. The saliency methods that achieved superior performance on salient object detection, e.g., BASNet [54] and F3Net [49], were inapplicable to camouflaged insect detection since they highlighted the most attention-seeking part of an image and discarded the seemingly unimportant pixels.

The recently proposed camouflaged object segmentation approaches achieved performance improvement to some extent; however, their performance degraded significantly for a number of challenging cases, such as complex shape, small size, being out-of-view, etc. As shown in Figure 6, PraNet [12] and PFNet [55] missed some parts of the insect body in the cases of insects with indefinable boundaries or thin parts. As a human visual system, a set of various-sized receptive fields helped to highlight the area close to the retinal fovea that is sensitive to details [40], which was overlooked by PraNet and PFNet. In addition, cross-level feature fusion also played a vital role in the success of camouflaged object detection. The ARF and SRM modules jointly considered both rich context information and effective cross-level feature fusion, yielding superior detection performance. Nevertheless, SINet [44] and C2FNet [3] utilized region and boundary information simultaneously, but the relationship between them was not fully captured, hence failing to correctly identify camouflage regions and interfusing the noise from the background when the camouflaged insect was oversize or out of view. The region and boundary were two key characteristics that distinguished camouflaged insects and backgrounds. After aggregating features in high levels and then predicting coarse regions, SINet-V2 [36] and our PRNet leveraged a set of recurrent reverse attention modules to establish the relationship between the region and boundary cues, which enabled the models to calibrate some misaligned predictions.

Our results went beyond recent studies in terms of both quantitative evaluation and visualization evaluation. The verification of module rationality, as well as the model generalization, were further done in the previous section. Extensive experiments showed that our PRNet outperforms the state-of-the-art methods on both quantitative and qualitative evaluations, which made it possible to become an ultimate universal insect detector for not only the camouflaged insects, but also the salient or generic insects.

Despite the superior performance of PRNet, due to the limited sample size and long-tailed distribution of challenging cases (see Figure 8), such as small insects and multiple insects, the dataset studied in this work is unable to take full advantage of deep learning models. The effect of deep learning depends on a large amount of data to some extent, and the emergence of new datasets will lead to rapid progress in computer vision [65,66,67,68]. With this in mind, we encourage researchers to construct a camouflaged insect dataset of large size and with high-quality annotation. Moreover, some measures should be taken in the future to address the challenging problem caused by the imbalance of samples of the camouflage dataset. The performance of small insect segmentation can also be improved by data augmentation in the training set, such as oversampling those images with small insects and augmenting each of those images by copy-pasting small objects many times [69]. In addition, generative adversarial networks (GANs) [70,71] can also alleviate the problem arising from a few discriminative features of the small objects. GANs can be used to generate a super-resolved representation of a small object that is very similar to a large object. The super-resolved feature is superimposed on the feature map of the original small target so as to enhance the feature expression to improve the segmentation performance of the small object. In addition to the small object problem, multiple insects and the congestion or occlusion between multiple insects will lead to the loss of feature information, causing false or missed segmentation. The short-term transformer block [72], lifted edges [73] and repulsion loss [74] might be effective in addressing these constraints. Additionally, our model shows highly correlated statistical dependencies between the predictions and inputs, but datasets may have different distributions in different domains. If the new domain were directly input to validate the model, the prediction results on the new dataset CHAMELEON might be inaccurate [75]. For example, the spider’s feet cannot be completely distinguished from the background in Figure 7. Therefore, transfer learning is encouraged here as well to reduce the distribution difference between the new domain and the original domain.

6. Conclusions

In the present paper, we addressed the problem of segmenting insect objects. Camouflaged insects were the focus due to their extreme features in solving the detection problem. This was the first attempt and achieved success in applying deep learning techniques to camouflaged insect image detection. The main ingredients of our approach are the ARF, SRM and RG modules. ARF formulates a novel means of extracting contextual information that perceives the varied appearances of camouflaged insects. SRM fuses comprehensive features to suppress the distractors of the background exhibiting similar colors or textures to camouflaged insects. The refinement of insect outlines using the proposed RG module yields more robust segmentation results than recent approaches for the same task. Our method reaches state-of-the-art detection accuracy on 1900 images of camouflaged insects (MAE = 3.2%), approaching the performance of human experts. In future work, we will extend our camouflaged insect dataset as well as having high-quality annotations, and new techniques, such as weakly supervised learning, zero-shot learning, transfer learning, and multi-scale backbone, could also be explored.