ACA-Net: An Adaptive Convolution and Anchor Network for Metallic Surface Defect Detection

Abstract

Metallic surface defect detection is critical to ensure the quality of industrial products. Recently, human-advanced surface defect detection algorithms have been proposed. Most of these algorithms rely on convolutional neural networks (CNN) and an anchoring scheme. However, a convolution unit only samples the input feature maps at fixed shapes and locations. Similarly, a set of anchors are uniformly predefined with fixed scales and shapes, which increases the difficulties of bounding box regression. Therefore, we propose an adaptive convolution and anchor network for metallic surface defect detection, named ACA-Net. Specifically, an adaptive convolution and anchor (ACA) module is proposed, which mainly consists of adaptive convolution and an adaptive anchor. Firstly, an adaptive convolution module (ACM) is designed, which adaptively determines the location and shape of each convolution unit. In addition, a multi-scale feature adaptive fusion (MFAF) is proposed, which is used in ACM to extract and integrate multi-scale features. Then, an adaptive anchor module (AAM) is proposed to yield more suitable anchor boxes by adaptively adjusting shapes. Extensive experiments on NEU-DET dataset and GC10 dataset validate the performance of the proposed approach. ACA-Net achieves 1.8% on NEU-DET dataset higher Average Precision (AP) than GA-RetinaNet. Furthermore, the proposed ACA module is also adopted in GA-Faster R-CNN, improving the AP by 1.2% on NEU-DET dataset.

1. Introduction

Due to the influence of many factors, such as the production process, equipment, and other restrictions in the industrial production process, there will be some defects on the surface of some metallic products. As we know, metallic surface defects have a greatly adverse effect on the quality of industrial products. Thus, it is crucial to ensure the quality of industrial production with the use of surface defect detection technologies. At present, the methods of surface defect detection of metallic products are mainly still in the stage of artificial eye detection in the industrial production process. However, the artificial visual inspection is unreliable and time-consuming. Therefore, in order to meet growing demand, the desired way to guarantee the quality of products is to automatically detect surface defects by compute vision technologies.

The surface defect detection system mainly includes the image acquisition module, image processing module, image analysis module, data management, and the man-machine interface module. This paper only focuses on the surface defect detection algorithm. The founder of computer vision can be defined as the process discovering what presents in an image and where they are [1]. Similarly, surface defect detection can be categorized into image-level defect classification, region-level defect inspection, and pixel-level defect segmentation [2]. The different among image-level defect classification, region-level defect inspection, and pixel-level defect segmentation is shown in Figure 1. Image-level surface defect classification [3,4] is to recognize and label the surface defects, but not know where the defects are. Region-level defect inspection [5], similar to generic object detection [6] in computer vision, is able to determine simultaneously the classification and localization. And this localization information is crucial for the follow-up quality assessment system. Pixel-level defect segmentation [5,7] can not only recognize and locate the surface defects, but also describe the shapes of defects. However, the shortcoming of pixel-level defect segmentation is that pixel-wise annotation is very time-consuming, so it is not feasible in the industrial field. Under this background, we focus on region-level surface defect inspection. In this paper, surface defect detection is regard as a region-level defect inspection task. From the perspective of computer vision, the defect detection task is essentially object detection for defecting images.

The current surface defect detection methods can be mainly divided into two categories, i.e., traditional methods and based on deep learning methods. Most early surface defect detection methods use hand-crafted features to achieve good performance [8,9]. However, these methods heavily depend on expertise, and they are very sensitive to illumination, background, and other restrictions [10]. In recent years, with the rapid development of the convolutional neural network (CNN) [11], the surface detection methods based on deep learning have become the mainstream methods [12,13]. Although many remarkable breakthroughs have been made, there are still many challenges in current surface defect detectors. Under this background, we discuss three important problems, i.e., how to better extract and integrate the multi-scale features, how to better enhance CNNs’ capability of extracting features, and how to better generate more suitable anchors.

Feature fusion is a critical part of modern surface defect detectors, which could fuse the features from different layers, scales, or branches. Many recent works demonstrate that the performance of detectors can be effectively improved by using a feature fusion module [14,15,16]. However, these features are usually fused via simple operations, such as addition or concatenation. It is not the best choice, because not all features from different branches contribute equally to its response. Furthermore, these simple fusion operations fail to take both channel and spatial attention into account simultaneously. So these methods are unable to determine what and where to emphasize or suppress features effectively. In this work, a multi-scale feature adaptive fusion (MFAF) is proposed. MFAF can adaptively extract and integrate the features from different layers and scales, and take both channel and spatial attention into account. Specifically, the MFAF makes use of a cascade of different kernels and dilated convolution layers to capture multi-scale contextual information. Then, the channel attention weights of the features with different scales are extracted by the SEWeight [17], and these channel weights are re-calibrated by using the Softmax operation to establish long-range channel dependency. Finally, an improved spatial attention module SAM [18] is utilized. Thus, the channel-wise and spatial-wise features with different receptive fields can be acquired.

In recent years, CNN has achieved outstanding success for computer vision [6]. Meanwhile, many significant CNN architectures are proposed, such as, VGG [19], Inception-v4 [20], ResNet [21], and HRNet [22]. However, these CNN backbones still suffer from some drawbacks. For example, a convolution unit only samples the feature map at fixed shapes and locations. As a result, the receptive field of a convolution unit cannot be adjusted according to different surface defects. Inspired by Deformable convolutional networks (DCN) [23], we design an adaptive convolution module (ACM) with fewer parameters and lower computational cost to enhance CNNs’ capability of extracting features of surface defects. Specifically, 4-channel offsets are added in the traditional convolution, and then the offsets are utilized to accommodate the positions and shapes of convolution units respectively. The offsets can be learned from the input feature maps via a convolutional block. To further enhance the capability of ACM, MFAF is employed to obtain the offsets.

Recently, the success of most state-of-the-art (SOTA) modern detectors [24,25,26] derived from the use of anchors. However, these anchor-based detectors suffer from some drawbacks. For instance, a set of anchor boxes are uniformly predefined with fixed scales and shapes, which increases the difficulties of bounding box regression. Inspired by Guided Anchoring (GA) [25], a more efficient adaptive anchor module (AAM) is proposed in this paper. Specifically, AAM first identifies the locations that may contain surface defects and then determines the anchor shapes at different locations. In addition, a location-guided shape adaptation component is designed, which can utilize the location information to yield more suitable anchors. Therefore, AAM is able to yield more accurate preset anchors for different surface defects adaptively, rather than using hand-designed preset anchor boxes blindly. In addition, only one anchor is predicted by AAM at each location of the feature map, which decreases computational cost effectively.

In summary, the principal contributions of this work are as follows:

(1) A multi-scale feature adaptive fusion (MFAF) is proposed. It can adaptively extract and integrate the features from different layers and scales, and meanwhile take both channel and spatial attention into account.

(2) An adaptive convolution module (ACM) is proposed in this work, which can adjust the location and shape of the convolution unit according to surface defects with different shapes and locations.

(3) An adaptive anchor module (AAM) is proposed, which is capable of adjusting the shapes and scales of anchor for different surface defects instead of using fixed preset anchors blindly.

2. Related Work

2.1. Traditional Detection Approaches

Generally, the traditional surface defect detection mainly consists of two parts: feature extractor and classifier. This method uses hand-crafted features such as scale-invariant feature transform (SIFT) [27] and histogram of oriented gradient (HOG) [28]. Then these features are always input into a classification model, e.g., support vector machines (SVM) [29]. For instance, Ghorai et al. [30] evaluate the performance of a number of wavelet features and used SVM to localize defects. To against noise, Song et al. [31] improve the local binary pattern (LBP) and then adopt SVM to classify hot-rolled steel strip surface defects. Chu et al. [32] propose four types of statistical features, and then apply an enhanced twin support vector machine classifier to recognize steel surface defects. Wang et al. [33] extract HOG feature and then introduce a random forest algorithm to perform defect classification. These traditional surface defect detection methods have achieved noticeable improvement. However, this type of extracted features suffers from drawbacks brought by hand-crafted features. For instance, these features are heavily dependent on expertise. Moreover, these methods still cannot achieve satisfactory detection accuracy and robustness.

2.2. Deep-Learning-Based Detection Approaches

From the perspective of computer vision, the surface defect detection task is essentially object detection for defect images. With the rapid development of CNN [11], generic object detection technology has achieved outstanding success [6]. Generally, an object detector mainly consists of three parts: a backbone, a neck and a head [24]. The backbone is employed to extract features from an input image, such as VGG [19] and ResNet [21]. Then, the neck part is often utilized to combine feature maps from different stages of the backbone, such as feature pyramid network (FPN) [34]. Finally, the head part is used for predicting classes and bounding boxes of objects. The head of object detection can be categorized into two kinds, i.e., one-stage method and two-stage method. Generally, two-stage detectors are more accurate, but have lower inference speeds. While, one-stage detectors are much faster, but have a relatively poor performance [6]. The R-CNN series method is the most representative two-stage object detection method, it includes Faster R-CNN [35], GA-Faster R-CNN [25], and TridentNet [26]. As for the one-stage object detection method, the most representative models mainly include you only look once (YOLO) [36], Single Shot MultiBox Detector (SSD) [37], and RetinaNet [38].

With the development of computer vision, many works [12,13] demonstrate that generic object detection methods based in deep learning could be transferred as defect detection methods. Recently, most of these state-of-the-art surface defect detectors rely on an anchoring scheme. For instance, Li et al. [39] adapt SSD as the meta structure to achieve real-time and accurate detection of surface defects. A cascaded autoencoder (CASAE) architecture is designed in [40] to detect metallic surface defects. Faster R-CNN [35] is improved in [13] to detect workpiece surface defects. Wei et al. [41] propose a defect detectors-based YOLOv3 model for detecting identification of the railway. Cheng et al. [12] improve RetinaNet with difference channel attention and spatial feature fusion for steel surface defect detection. Despite the outstanding success, there are many challenges in the surface defect detection field. For instance, the receptive field of a convolution unit cannot be adjusted according to different defects. In addition, a set of anchor boxes are uniformly predefined with fixed scales and shapes, which increases the difficulties of bounding box regression.

2.3. Feature Fusion

Different from the neck (e.g., FPN [34]) of detector, feature fusion in this paper is utilized to fuse the features from different layers, scale, or branches. Many recent works demonstrate that the performance of detectors can be effectively improved by using a feature fusion module [14,15,16]. Recently, many excellent feature fusion modules are proposed, such as SPP [14], Receptive Fields Block (RFB) [15] and Hybrid Pooling-Atrous (PHA) [16]. Although many outstanding breakthroughs have been made, the feature fusion operation in these methods mentioned above are implemented via simple concatenation or addition. Nevertheless, this is not the best choice, because not all features from different branches contribute equally to its response. Therefore, Zhang et al. [17] design Pyramid Split Attention (PSA) to extract the multi-scale spatial information, and then capture recalibrated channel attention weights by Softmax operation to establish a long-range channel dependency. However, The PSA module misses the spatial attention which is important to determine where an object is.

2.4. Convolutional Neural Networks

In recent years, CNN has achieved outstanding breakthroughs, such as VGG [19], ResNet [21] and HRNet [22]. Benefiting from the rapid development of CNN, surface defect detection technology has achieved outstanding success [41]. However, there are many challenges in CNN. For instance, the receptive field, shape, and location of the convolution unit cannot be adjusted according to objects with different shapes. To address these problems, Dai et al. [23] propose DCN which adds 2D offsets to the sampling locations in standard convolution. DCN could adjust freely deformation of the sampling grid according to different objects. However, many samples may extend well beyond the region of the object, which causes the features to be influenced by irrelevant area in the image. Furthermore, this method leads to significant computation cost. For instance, the 18-channel offsets will be needed for a $3\times 3$ convolution unit.

2.5. Anchor Boxes

Recently, the success of most state-of-the-art surface defect detection relies on anchoring schemes. As for two-stage detectors, anchors are utilized to predict proposals, such as Faster R-CNN [35]. Different from two-stage detectors, anchors are employed to predict directly final bounding boxes in one-stage detectors, such as YOLOv3 [42], SSD [37] and RetinaNet [38]. However, the aspect ratios and scales of predefined anchors are fixed uniformly, which cannot adjust adaptively according different objects. To better leverage the anchor boxes, some excellent semi-anchor-free methods are proposed. For instance, rather than using all of the preset fixed anchors, Guided Anchoring (GA) [25] could predict only one variable anchor at each location that may contain objects. However, GA only utilizes the anchor shape information to refine features, the anchor location information is not efficiently utilized.

3. Materials and Methods

3.1. Overall Network Architecture

An adaptive convolution and anchor network ACA-Net is designed in this paper. Specifically, an adaptive convolution and anchor (ACA) module is proposed, which consists of ACM and AAM. Meanwhile, an MFAF is designed and used in ACM. Firstly, the overall network architecture of the proposed ACA-Net is described in this section. Then, the proposed MFAF is introduced. Next, we introduce the proposed adaptive convolution module in detail. Finally, the details of the proposed AAM are described.

The overview architecture of the proposed ACA-Net is illustrated in Figure 2. ACA-Net mainly consists of three parts: (a) a backbone to extract features from the input image; (b) a neck to combine feature maps from different stages of the backbone; (c) a head to predict classification and bounding boxes (BBox) of defect. Specially, ResNet-50 [21] pretrained on ImageNet [43] dataset is used as the backbone. It is worth noting that the ResNet-50 is modified for detecting surface defects, which are the same as RetinaNet. ACM improved by MFAF is employed at stages conv3–conv5 of ResNet-50 to enhance the modeling capability. Secondly, FPN [34] is adopted as the neck on top of the backbone to integrate multi-level features. Finally, we adopt the head which is similar to RetinaNet [38]. The box subnet (BSubnet) and the classification subnet (CSubnet) is respectively utilized to yield regression features (RF) and classification features (CF). Note that the RF and $CF$ are operated by AAM to yield more suitable anchors and make new features $RF\prime$ and $CF\prime$. Afterwards, the $RF\prime$ and $CF\prime$ is used for regressing BBox and predicting classes of surface defects respectively.

3.2. Multi-Scale Feature Adaptive Fusion

As shown in Figure 3, the proposed multi-scale feature adaptive fusion (MFAF) mainly consists of two parts: multi-scale feature extract (MFE) and feature adaptive fusion (FAF). ⊗ denotes the element-wise multiplication, and © denotes the concatenate operator in the channel dimension. in_ch and mid_ch denote the number of channel of feature maps. To capture multi-scale contextual information, the MFE makes use of a cascade of different kernels and dilated convolution layers. Specifically, MFE consists of two ordinary convolution layers with $1\times 1$ and $3\times 3$ kernels respectively, and a $3\times 3$ atrous convolution layer with the dilation rate of 3. The ordinary $3\times 3$ convolution layer is added before the atrous convolution layer. Then, the three kinds of features, from the ordinary convolution and atrous convolution, are concatenated in the channel dimension. Mathematically, the multi-scale feature map can be obtained by a concatenation way as

$$F=Cat\left(F_1,F_2,F_3\right)$$

(1)

where $F_1$, $F_2$ and $F_3$ are the three kinds of feature maps with different receptive fields, and F is a multi-scale feature map.

After obtaining the multi-scale features, FAF is used for fusing features adaptively. Firstly, the channel attention weights of the three kinds of features are extracted by the SEWeight [17] respectively. As illustrated in Figure 3b, it uses a global average pooling and two fully-connected (FC) layers. The vector of channel attention weights can be represented as

$$Z_i=SEWeight\left(F_i\right),i=1,2,3$$

(2)

where $Z_i$ is the channel attention weights of the feature map with different scales. The multi-scale attention weight vectors can be obtained in a concatenation way as

$$Z=Z_1\mathsf{\copyright }{Z}_2\mathsf{\copyright }{Z}_3$$

(3)

where © is the concatenation operator, Z is the multi-scale attention weight vectors. Then, these channel weights are re-calibrated by using the Softmax operation to establish long-range channel dependency. A soft assignment weight can be obtained as

$$at{t}_i=Softmax\left(Z_i\right)=\frac{exp\left(Z_i\right)}{{\sum }_{i=1}^{3}exp\left(Z_i\right)}$$

(4)

where $at{t}_i$ is the re-calibrated weight of the multi-scale channel. Analogously, the whole channel attention weight can be given by

$$att=at{t}_1\mathsf{\copyright }at{t}_2\mathsf{\copyright }at{t}_3$$

(5)

where $att$ denotes the final multi-scale channel weights after Softmax operation. Thus, the channel-wise features can be obtained by multiplying the re-calibrated weight attention $att$ with the feature map F. It could be defined as

$$Y=F\otimes att$$

(6)

where ⊗ denotes the element-wise multiplication, and Y is channel-wise multi-scale feature map.

Finally, these channel-wise features are operated by improved SAM. As illustrated in Figure 3b, we employ an extra added $1\times 1$ convolution layer compared to original SAM [18]. The spatial information from average-pooling, max-pooling and $1\times 1$ convolution layers is operated by $7\times 7$ convolution layer with the Sigmoid function to obtain spatial attention weights. Mathematically, the final feature can be given by

$$Out=SAM\left(Y\right)$$

(7)

where $Out$ is the channel-wise and spatial-wise feature map with multi receptive fields, and $SAM$ denotes the improved SAM operation.

3.3. Adaptive Convolution Module

The traditional 2D convolution uses a regular grid over the input feature map, which means that the sampling locations and receptive field sizes of regular convolution are fixed. For example, in regular convolution, a $3\times 3$ kernel could be defined as

$$p_k\in \left\{(-1,-1),(-1,0),\cdots ,(1,1)\right\}.$$

(8)

In this example, we let $K=3\times 3$, which denotes the 9 sampling locations. Let $x\left(p\right)$ denotes the features at location p from the input feature map. As for the output feature map, the features at p can be expressed as

$$y\left(p\right)=\sum _{k=1}^K{w}_k·x\left(p+p_k\right)$$

(9)

where $y\left(p\right)$ are the features at location p from the output feature map, and $w_k$ is the learnable weight. Due to the fixed sampling locations and receptive fields of regular convolution, these convolution filters cannot appropriately cover the surface defects with various shapes.

To solve the issues above, we propose an adaptive convolution module (ACM). For simplicity, the ACM is described in 2 dimensions. As illustrated in Figure 4, the regular convolution kernels in ACM could be adjusted with learnable offsets containing 4 elements $\left\{\mathsf{\Delta }x,\mathsf{\Delta }y,\mathsf{\Delta }w,\mathsf{\Delta }h\right\}$. $\mathsf{\Delta }x,\mathsf{\Delta }y\in \left(-0.5,0.5\right)$ and $\mathsf{\Delta }w,\mathsf{\Delta }h\in \left(-1,1\right)$ are respectively used for adjusting the sampling locations and shapes (i.e., weight and height) of regular convolution kernels. The sampling location offsets can be expressed as

$$\mathsf{\Delta }xy=\left(\mathsf{\Delta }x,\mathsf{\Delta }y\right)$$

(10)

The sampling shape offsets can be expressed as

$$\mathsf{\Delta }w{h}_k\in \left\{(-\mathsf{\Delta }w,-\mathsf{\Delta }h),(-\mathsf{\Delta }w,0),(-\mathsf{\Delta }w,\mathsf{\Delta }h),\cdots ,(\mathsf{\Delta }w,\mathsf{\Delta }h)\right\}$$

(11)

Hence, in ACM, the features at location p from the output feature maps becomes

$$y\left(p\right)=\sum _{k=1}^K{w}_k·x\left(p+p_k+\mathsf{\Delta }xy+\mathsf{\Delta }w{h}_k\right)$$

(12)

As $\mathsf{\Delta }xy$ and $\mathsf{\Delta }wh$ are typically fractional, the value of $y\left(p\right)$ is calculated via bilinear interpolation, just as in [23]. In this way, for $3\times 3$ kernel, ACM can adjust the regular convolution kernels with only 4-dimensional offsets. Therefore, no heavy computation is introduced by ACM. To further enhance ACM, the offsets are obtained via MFAF block with $mid\_ch=8$. The spatial resolution of the offset field is the same as the input feature map. In the training, both the convolutional kernels and the offsets could be learned simultaneously. In this paper, ACM is utilized in all the $3\times 3$ convolutional layers in conv3–conv5 stages in ResNet-50 [21].

3.4. Adaptive Anchor Module

Inspired by GA [25], an adaptive anchor module (AAM) is proposed in this work which could yield more suitable anchor boxes for different surface defects. Given an image I, the probability of a surface defect can be expressed as

$$p\left(x,y,w,h|I\right)=p\left(x,y|I\right)p\left(w,h|x,y,I\right)$$

(13)

where $(x,y)$ denotes the center of surface defect, and $(w,h)$ is the width an height respectively. This equation indicates that the shape of a surface defect is closely related to its location. Following this formulation, AAM first identifies locations where the centers of surface defects are likely to exist and then predicts the shapes at different locations. Furthermore, we devise a location-guided shape adaptation component in this work, which can utilize the location information to yield more suitable anchors.

As demonstrated in Figure 5. Firstly, given a feature map from the neck of detector, the subnet CSubnet and BSubnet is utilized to yield class features (RF) and box features (CF), respectively. Next, a $1\times 1$ convolution with Sigmoid function is employed to yield a probability map which indicates the locations where the centers of surface defects are likely to exist. The probability map is the same size as CF. The value of each entry of probability map denotes the probability that the center of a surface defect may exist at that location. Based on the probability map, the regions where surface defects may exist could be selected. Therefore, there is no need to take those excluded regions into account.

Then, the shapes of the anchors where surface defects may exist at each location is determined. As shown in Equation (13), the shape of an anchor closely relates to its location. Therefore, a location-guided shape adaptation module is designed in this work to yield more suitable anchors. Specifically, the shape adaptation module consists of two parts: (1) A probability map is input two $3\times 3$ convolutions to yield shape offsets; (2) A $3\times 3$ deformable convolution (DC) is utilized to improve regression features by using the shape offsets. By these operations, the shapes of anchors can be adapted via location information. Afterwards, the shapes of anchors can be predicted via a $1\times 1$ convolutional layer. It is also worth noting that just one anchor box is placed at each available location.

Finally, feature adaptation is adopted to transform the original feature map (i.e., CF and BF) according to the anchor shape information. It utilizes a $1\times 1$ convolutional layer to obtain location offsets from the shape map, and then applies a $3\times 3$ deformable convolution to the original features, similar to the shape adaptation module.

A multi-task loss function as in GA [25] is utilized to optimize the framework in an end-to-end way. The multi-task loss function can be defined as

$$L={\lambda }_1{L}_{loc}+{\lambda }_2{L}_{shape}+L_{cls}+L_{reg}$$

(14)

where $L_{loc}$ is a focal loss [38] to optimize the locations. As for $L_{shape}$, we use a variant bounded iou loss as in [25] to only optimize the shape of anchor. $L_{cls}$ is a focal loss [38] and $L_{reg}$ is $L_1$ loss [44]. In our experiments, ${\lambda }_1=1$ and ${\lambda }_1=0.1$.

3.5. Dataset

Surface defect images are affected by a variety of factors, such as environment, light, noise, and camera. Furthermore, image quality seriously affects the of surface defect detectors. However, we focus on how to improve the detection accuracy, not how to get good images. Therefore, we only used the images with good quality. In this work, the NEU-DET [45] dataset is used. NEU-DET consists of six kinds of surface defects from hot-rolled steel plates, which includes patches, rolled-in scale, crazing, inclusion, pitted surface and scratches. Each kind of defect includes 300 images, and an image may consists many defects. The annotations are provided in NEU-DET, which marks the class and location of each defect in an image. Some examples of defect images are shown in Figure 6. Each yellow box is the bounding box which denotes the groundtruth box of a defect. In the NEU-DET dataset, there are about 5000 groundtruth boxes. Conventionally, the NEU-DET is divided into a training set containing 1260 images and a testing set containing 540 images.

4. Results and Discussions

4.1. Implementation Details

The ResNet-50 [21] with FPN [34] is used as the backbone network of ACA-Net. The NEU-DET dataset is a small dataset which is not enough to support the training of complex neural networks. Therefore, the ResNet-50 is pretrained on ImageNet [46] dataset. All of the surface defect detectors are trained on a Tesla V100 32G GPU with 2 images. Stochastic gradient descent (SGD) is used to train with a learning rate of 0.002, momentum as 0.9 and weight decay of 0.0001. We train 36 epochs in the end-to-end manner, and decrease the learning rate at the 28th and 34th epochs. The models are trained by the MMDetection toolkit [47].

The accuracies of all detectors are evaluated by the COCO [48] standard metric average precision (AP). The speed of each detector is evaluated on frame per second (FPS), i.e., the number of images that a detector can process per second. The IoU strategy in [25] is employed to determine the positive and negative samples from the anchors. The $AP$ is evaluated at a different IoU threshold ($IoU\in \left[0.5:0.95\right]$) over all kinds of surface defects. It can be defined as

$$AP=\frac{1}{K}\sum _{n=1}^K\int \left(P_n\left(R_n\right)\right)d{R}_n$$

(15)

where $R_n$ denotes the recall for class n, $P_n$ is the corresponding precision, and K is the number of classes. The precision (P) and recall (R) are defined as follows

$$P=\frac{TP}{TP+FP}$$

(16)

$$R=\frac{TP}{TP+FN}$$

(17)

where $TP$, $FP$, and $FN$ denotes the number of true positive samples, false positive samples, and false negative samples, respectively.

Table 1. Evaluation metrics.

Alias	IoU	Area	Meaning
$AP$	0.50–0.95	all	$AP$ at IoU thresholds from 0.50 to 0.95
$A{P}_{50}$	0.50	all	$AP$ at 0.50 IoU threshold
$A{P}_{75}$	0.75	all	$AP$ at 0.75 IoU threshold
$A{P}_S$	0.50–0.95	small	$AP$ for small objects: $area<{32}^2$
$A{P}_M$	0.50–0.95	medium	$AP$ for medium objects: ${32}^2<area<{96}^2$
$A{P}_L$	0.50–0.95	large	$AP$ for large objects: $area>{96}^2$

shows all of the evaluation metrics in out experiment.

4.2. Ablation Studies

In order to verify the effectiveness of the proposed modules, ablation experiments are conducted. The results are shown in

Table 2. The effects of proposed module.

ACM	MFAF	AAM	$AP$	$A{P}_{50}$	$A{P}_{75}$	$A{P}_S$	$A{P}_M$	$A{P}_L$
			0.355	0.728	0.290	0.231	0.283	0.437
✓			0.361	0.736	0.311	0.272	0.293	0.431
✓	✓		0.366	0.732	0.324	0.291	0.295	0.452
✓	✓	✓	0.373	0.746	0.332	0.306	0.302	0.440

. We take the GA-RetinaNet [25] with ResNet-50 [21] and FPN [34] as our baseline.

Firstly, ACM (without MFAF) is applied to GA-RetinaNet. We could see that the AP increases from 35.5% to 36.1%. Meanwhile, the $A{P}_S$ and $A{P}_M$ is improved by 4.1% and 1.0% respectively. This result indicates that the modeling capability of CNN can be enhanced by adjusting the location and shape of the convolution unit, especially for small defects. However, we also find that $A{P}_L$ drops by 0.6% since a large receptive field is important for detecting large defects. However, the receptive field of the shape offsets in ACM is too small for large defects. We believe that this is the main reason for the dropping of $A{P}_L$.

Then, MFAF is used in ACM to adaptively integrate multi-scale features. With MFAF, the AP is improved by 0.5%. Meanwhile, we could also find that the $A{P}_S$, $A{P}_M$ and $A{P}_L$ is 1.9%, 0.2% and 2.1% better than before respectively. The result demonstrates that MFAF contributes to improving the surface defect detection performance for all surface defects with different sizes.

Finally, AAM is leveraged to replace original GA [25]. As illustrated in Table 2, the use of AAM shows a significant increase from 36.6% to 37.3%. This result better proves our previous views: The shape of an anchor closely relates to its location, and the location information could help AAM yield more suitable anchors. In addition, the $A{P}_{75}$ is improved by 0.8%. Theses improvements further demonstrate that AAM could yield more suitable anchor boxes for defects. We could also find that the $A{P}_S$ and $A{P}_M$ are improved by 1.5% and 0.7%. However, it also worth noting that $A{P}_L$ drops by 1.2%. We argue that the offsets in AAM have too-small receptive fields, which compromises our ability to locate large objects.

4.3. Comparsion

To further confirm the effectiveness of the proposed method, we apply the proposed ACA module to a two-stage detector GA-Faster R-CNN [25] and compare our methods with most of the SOTA one-stage and two-stage detectors. We could see that two-stage detectors commonly achieve better detection accuracy than one-stage detectors [6], but have lower inference speeds. And

Table 3. Comparison results.

	Method	Backbone	$\mathrm{AP}$	${\mathrm{AP}}_{50}$	${\mathrm{AP}}_{75}$	${\mathrm{AP}}_S$	${\mathrm{AP}}_M$	${\mathrm{AP}}_L$	$\mathrm{FPS}$
one- stage	YOLOv3 [42]	Darknet-53	0.284	0.661	0.209	0.337	0.238	0.270	18.8
	YOLOF [49]	ResNet-50	0.330	0.675	0.265	0.303	0.277	0.372	18.6
	YOLOX-l [50]	CSPDarknet	0.373	0.726	0.326	0.330	0.309	0.456	16.6
	SSD [37]	VGG-16	0.319	0.692	0.237	0.200	0.266	0.395	18.1
	GA-RetinaNet [25]	ResNet-50	0.355	0.728	0.290	0.231	0.283	0.437	8.6
	ACA-Net (ours)	ResNet-50	0.373	0.746	0.332	0.306	0.302	0.440	7.6
two- stage	TridentNet [26]	ResNet-50	0.401	0.753	0.390	0.371	0.337	0.474	4.9
	Faster R-CNN [35]	ResNet-50	0.383	0.731	0.370	0.289	0.320	0.459	10.6
	GA-Faster R-CNN [25]	ResNet-50	0.391	0.756	0.370	0.343	0.325	0.465	7.5
	ACA-Faster R-CNN (ours)	ResNet-50	0.403	0.764	0.380	0.440	0.336	0.461	6.0

shows that ACA-Net achieves an AP of 37.3%. Meanwhile, we could also see that ACA-Faster R-CNN achieves an AP of 40.3%, 1.2 points higher than the baseline GA-Faster R-CNN. These results indicate that both our one-stage and two-stage methods are comparable to most of the SOTA methods in accuracy. The results better confirm the effectiveness of the proposed ACA module. However, ACA-Net achieves an FPS of 7.6, 1.0 FPS lower than the baseline GA-RetinaNet. And compared with GA-Faster R-CNN, the FPS of ACA-Faster R-CNN decreases from 7.5 to 6.0. Nonetheless, it is worth noting that our aim is to verify the effectiveness of the proposed ACA module, instead of achieving SOTA performance in both accuracy and speed with many tricks. Besides, both ACA-Net and ACA-Faster R-CNN are obviously far faster than artificial visual inspection.

For a more intuitive comparison, some detection examples on NEU-DET dataset are shown in Figure 7. We could find that both ACA-Net and ACA-Faster R-CNN detect all surface defects with high accuracies, which further confirms the effectiveness of our methods.

4.4. Additional Experiments

To further demonstrate the effectiveness of the proposed methods, we carry out experiments on GC10 dataset from AI Studio [51]. The GC10 dataset is a small dataset which includes 2294 images from weld assemblies. It consists of nine kinds of surface defects and a welding line, which includes Crease, Crescent gap, Inclusion, Oil spot, Punching, Rolled pit, Silk spot, Waist folding, Water spot and Welding line. The annotations are provided in GC10, which marks the class and location of each defect in images. Conventionally, the GC10 is divided into a training set containing 1836 images and a testing set containing 458 images.

Table 4. Comparison results.

Method	Backbone	$\mathrm{AP}$	${\mathrm{AP}}_{50}$	${\mathrm{AP}}_{75}$	${\mathrm{AP}}_S$	${\mathrm{AP}}_M$	${\mathrm{AP}}_L$	$\mathrm{FPS}$
GA-RetinaNet [25]	ResNet-50	0.281	0.589	0.257	0.050	0.172	0.267	6.1
ACA-Net (ours)	ResNet-50	0.293	0.605	0.268	0.100	0.190	0.272	5.9

shows that all the metrics of accuracy are improved compared with the baseline GA-RetinaNet. However, we also find that ACA-Net achieves a FPS of 5.9 on GC10 dataset, just 0.2 FPS lower than the baseline GA-RetinaNet. Nonetheless, ACA-Net is obviously far faster than artificial visual inspection on the GC10 dataset. The results further confirm the effectiveness of our methods.

5. Conclusions

In this work, we propose an adaptive convolution and anchor network (ACA-Net) for metallic surface defect detection, which can produce more robust features and yield high-quality anchor boxes. Specifically, an adaptive convolution module (ACM) is designed, which can adaptively adjust the location and shape of each convolution unit to enhance the modeling capability of CNN. In addition, a multi-scale feature adaptive fusion (MFAF) is proposed to extract and integrate multi-scale features adaptively. Then, we design an adaptive anchor module (AAM) which can generate more accurate anchor boxes for different surface defects. Extensive experiment results on the NEU-DET dataset and GC10 dataset demonstrate the effectiveness of our approach.