Aircraft Detection in SAR Images Based on Peak Feature Fusion and Adaptive Deformable Network

Abstract

Due to the unique imaging mechanism of synthetic aperture radar (SAR), targets in SAR images often shows complex scattering characteristics, including unclear contours, incomplete scattering spots, attitude sensitivity, etc. Automatic aircraft detection is still a great challenge in SAR images. To cope with these problems, a novel approach called adaptive deformable network (ADN) combined with peak feature fusion (PFF) is proposed for aircraft detection. The PFF is designed for taking full advantage of the strong scattering features of aircraft, which consists of peak feature extraction and fusion. To fully exploit the strong scattering features of the aircraft in SAR images, peak features are extracted via the Harris detector and the eight-domain pixel detection of local maxima. Then, the saliency of aircraft under multiple imaging conditions is enhanced by multi-channel blending. All the PFF-preprocessed images are fed into the ADN for training and testing. The core components of ADN contain an adaptive spatial feature fusion (ASFF) module and a deformable convolution module (DCM). ASFF is utilized to reconcile the inconsistency across different feature scales, raising the characterization capabilities of the feature pyramid and improving the detection performance of multi-scale aircraft further. DCM is introduced to determine the 2-D offsets of feature maps adaptively, improving the geometric modeling abilities of aircraft in various shapes. The well-designed ADN is established by combining the two modules to alleviate the problems of the multi-scale targets and attitude sensitivity. Extensive experiments are conducted on the GaoFen-3 (GF3) dataset to demonstrate the effectiveness of the PFF-ADN with an average precision of 89.34%, as well as an F1-score of 91.11%. Compared with other mainstream algorithms, the proposed approach achieves state-of-the-art performance.

1. Introduction

Because of its capacity to obtain high-resolution SAR images regardless of time and weather, synthetic aperture radar has been widely applied in many fields, such as ocean observations, natural disaster prediction, and battlefield surveillance [1,2]. Automatic target detection in SAR images has been a hot research topic for many years, attracting extensive attention from scholars at home and abroad. The essence of object detection in SAR images is to extract the target from the background via the difference in scattering characteristics, figuring out the location of potential objects. However, it is difficult to interpret SAR images using coherent scattering and imaging mechanisms. Aircraft detection in complicated conditions of SAR images is always a challenging task.

Essentially, traditional SAR target detection algorithms are designed for target extraction based on the differences in the electromagnetic backscattering properties of targets and the clutter background, and the same goes for aircraft detection. The traditional methods applied in SAR images can be grouped into the following four categories: contrast-based, visual attention-based, complex data-based, and multi-feature-based. The methods based on contrast include CFAR [3], various CFAR-derived methods (CA-CFAR [4], SOCA-CFAR [5], GOCA-CFAR [6], OS-CFAR [7], VI-CFAR [8]), GLRT [9] and PR [10]. The advantage of methods based on contrast are easy to implement and can accomplish good performance in simple scenarios. However, it is difficult to select an appropriate statistical clutter model and tackle heterogeneous strong clutter in the practical application. In addition, there are many false alarms and missing targets in object detection under a complex background. The visual attention-based method [11] suppresses the strong hetero enhancement target by introducing prior information, improving the SCR (signal to clutter ratio). Yet, the prior information introduced needs to be concretely analyzed for specific conditions, and the algorithm may be redesigned for different detection tasks. For complex data-based methods [12], the scattering characteristics of the target and clutter, as well as the imaging mechanism, are utilized for detecting the target. It can reflect the difference between man-made targets and natural clutter from the physical mechanism. However, high acquisition costs are required to obtain the raw complex SAR image data, and the capacity to distinguish between artificial clutter interference and artificial targets of interest is yet to be verified. The principal features used in the multi-feature-based approach mainly comprise structural features [13], peak features [4], variance features [14], and extended fractal features [15]. The merit of this solution is that it can detect targets from multi-dimensional features. Nevertheless, due to the poor generalization of this method, it can only achieve good performance for specific data. Among the above detection methods, the CFAR detector is one of the most widely studied and intensively applied. The main components of aircraft are composed of metallic materials with high backscattering coefficients. The airport area has a low backscattering coefficient as it is more capable of absorbing electromagnetic waves. The contrast-based detection algorithm can efficiently detect aircraft in the ROI area of the airport under such conditions. However, in general, traditional methods are ineffective in the face of complex background scattering, owing to the poor robustness and the limited capacity of feature representation.

With the enrichment of SAR images, the traditional algorithm, represented by CFAR, no longer meets the requirements. In recent years, with the development of DL and the improvement of the related hardware device, convolutional neural networks have shown amazing performance in the field of computer vision; and yet these methods are mainly applied to optical images. Detection algorithms based on DL can be divided into the two-stage algorithm and the single-stage algorithm in terms of detection steps. The Faster-RCNN [16] is a typical two-stage algorithm that adopts a trainable region proposal network (RPN) to improve detection efficiency while maintaining high accuracy. However, only high-level features are employed to predict objects, and it is difficult to detect small targets because it loses the low-level features. The single-stage representative algorithm is YOLO [17], which transforms the detection into a regression problem. The algorithm achieves a significant improvement in the detection rate, but the precision becomes poor. An FPN structure [18] is proposed in the literature, which integrates high-resolution information about shallow features with high semantic information on deep features. The detection performance is raised by predicting the different feature layers separately. All the above methods are based on anchor frames. Firstly, the location of potential targets is inferred from the generated anchors in the RPN stage, and the final results are filtered by NMS (Non-Maximum Suppression). After 2019, algorithms based on DL without anchor frames have been developed, such as CornetNet [19], ExtremeNet [20], FCOS [21], and CenerNet [22]. CornerNet is a new one-stage detection method that treats the target as a set of key points. ExtremeNet evolves based on CornerNet, which detects four extremes and one centroid via standard key point estimation. The central heat map of each class is proposed to predict the target center. CenterNet derives the centroid via key point estimation and additional information about the target by regression. Many other methods have been proposed for object detection, e.g., [23,24].

With deep learning (DL) achieving amazing performance in computer vision, DL has also gradually become a standard paradigm in SAR image analysis, including aircraft detection [25], which is detailed in Section 2.1.1. Although many approaches have achieved a certain effect in aircraft detection of SAR images, they still leave much room for improvement.

The imaging mechanisms of SAR and optical satellites display great differences, resulting in an entirely different appearance of the target. The represented features of aircraft in SAR images can be summarized via the following three aspects:

(1)

Discrete points. Due to the mechanism of SAR imaging, aircraft appearing in SAR images consist of a few strong scattering points, which form the faint outline of the aircraft. Coupled with the interference of background clutter, it is easy to miss detection;

(2)

Multi-scale targets. In this study, the size of most aircraft in the GF3 dataset ranges from 20 to 110 pixels. However, with the deepening of the CNN, the spatial information of small-size aircraft with fewer pixels is easy to lose, which poses a challenge to multi-scale aircraft detection;

(3)

Attitude sensitivity. With changes in azimuth angle, the appearance of the same aircraft in different SAR images is not quite identical. It is difficult to acquire the modeling geometric transformation of changeable-appearance aircraft in limited and available training samples.

As is well-known, there is no doubt that abstract features extracted by DL have better representation capacities than traditional manual features. However, should the traditionally established hand-made features be completely abandoned in favor of total reliance on abstract features? This is a meaningful issue. Regretfully, most detection algorithms of SAR based on DL are established by fine-tuning the classical optical network, and do not consider the differences in the imaging mechanism between SAR and optical satellites. Otherwise, compared with optical data, SAR image data are relatively scarce. A large amount of image data is acquired to train the network model, so transfer learning from optical networks alone does not work well. Besides this, many publications are only tested on specific circumstances, e.g., aircraft with clear contours and a single background. Fewer detection networks take the multi-scale and attitude sensitivity of targets in the various and complicated conditions of SAR images into consideration.

Intending to address the above-mentioned problems, an adaptive pyramid deformable network combined with the peak feature fusion (PFF-ADN) is proposed to ameliorate the performance of aircraft detection under complex backgrounds and target dense arrangement conditions. The strong scattering points represent the maximum pixel value of the local scattering region in SAR images, which have superior significance and stability [26]. The distribution of scattering centers can be responded to the peak features, which also can describe the geometric shape of the target in a complex environment. Therefore, combining the peak features is a feasible way to enhance the performance of aircraft detection. (1) In the peak feature extraction stage, because the strong scattering points tend to be distributed on the common components such as nose, fuselage, tail, wings, and engines, the corner points extracted by the Harris detector [27] with rotational invariance, are selected as the basis for modeling the scattering information of aircraft. Then the peak features of images are acquired based on the 8-domain pixel detection of local maxima, which will be sent to the next stage for feature fusion. (2) In the peak feature fusion stage, the image data processed by PFE will be integrated into the image as the G-channel data. This operation can fuse the peak features of aircraft into the image, enhancing the saliency of aircraft under various imaging conditions. Meanwhile, it facilitates the mitigation of variability and provides more effective information to the detection network. (3) In the deep feature extraction stage, an adaptive deformable network for aircraft detection is proposed by incorporating the ASFF and DCM. The ASFF can cope with the inconsistency of multi-scale features, which is beneficial to fully exploit the representational power of the feature pyramid network, improving the ability to detect multi-scale aircraft, especially for small-size aircraft. The DCM is adopted to cope with the attitude sensitivity of aircraft in SAR imaging and various shapes of aircraft, making the detection network accommodate the geometric variations. Finally, experiments are conducted on the GF3 dataset to validate the effectiveness of each presented module and to demonstrate the superiority of the proposed method.

The main contributions of our work can be summarized as follows:

(1)

A novel integrated framework named PFF-ADN is proposed for SAR aircraft detection, which enhances the scattering features of the target by fusing the peak features of images and improving the network structure. This presented method achieves state-of-the-art performance on the GF3 dataset.

(2)

A peak feature fusion strategy is designed for enhancing the brightness information of aircraft in the SAR images by extracting and fusing the peak feature information, which has stronger robustness to deal with the variability and obscureness caused by the scattering mechanism of SAR. Compared to the raw images, the aircraft characteristics are highlighted in the enhanced images, providing effective information on aircraft for the subsequent network.

(3)

An adaptive deformable network for aircraft detection is designed, which is composed of a Feature Pyramid Network with ASFF structure and deformable convolution module (DCM). The ASSF is introduced to solve the inconsistency of multi-scale features and retain more discrete information about small-size aircraft, which enhances the detectability, especially for small-size aircraft. The DCM is adopted to cope with the attitude sensitivity of aircraft in SAR imaging and various shapes of aircraft, making the detection network accommodate the geometric variations.

The remainder of this paper is organized as follows. Section 2.1 shows a review of the related work, which contains DL methods and traditional methods of aircraft detection in the SAR domain. Section 2.2 describes the proposed PFF-ADN algorithm in detail. In Section 3, the parameter configuration, assessment criteria, experimental results as well as performance evaluation are shown in detail. Finally, we summarize this article and provide the prospect for further research in Section 4.

2. Materials and Methods

2.1. Related Work

2.1.1. Aircraft Detection Based on DL in SAR Images

Recently, with deep learning (DL) achieving amazing performance in computer vision, DL has also gradually become a standard paradigm in SAR image analysis. Guo et al. [25] designed a novel method that combines scattering information enhancement and an attention pyramid network for aircraft detection. The candidate areas containing aircraft are enhanced by scattering information, improving the average precision of aircraft detection. Therefore, inspired by this work, we propose the peak feature can be fused into the image to enhance the information of the target, thus improving the detection performance. Wang et al. [28] proposed a method based on a convolutional neural network (CNN) and data enhancement for aircraft detection. The regions of interest are acquired by the saliency pre-detection approach and then fed into CNN to get the final detection results. Zhang et al. [29] showed a cascaded three-look network to detect aircraft in SAR images. It firstly narrows the inspected image down to the airport area by pixel-based comparison, and it obtains the detection results through Faster-RCNN. Zhao et al. [30] adopted an integration network that contains the dilated attention block and convolution block attention module for aircraft detection, and they confirmed the effectiveness of this method on a mixed SAR aircraft dataset. He et al. [31] exploited the components of aircraft to improve detection performance. To find out the inner link between the aircraft and the components, they established a multi-scale detector including a root filter and part filters. The experimental results indicate this method has higher precision and is able to detect the components of aircraft. Dou et al. [32] came up with an optimized target attitude estimation method wherein prior information about the target shape is acquired by the generative adversarial network. Detection probability is improved with valid prior knowledge of aircraft. Julie et al. [33] suggested a hybrid clustering active learning method to improve the performance of aircraft detection. Han et al. [34] utilized this method, which combines region segmentation and multi-feature decision for aircraft detection in POL-SAR images, and the detection results showed a lower false positive rate. Jia et al. [35] designed a component discrimination network for aircraft detection, which utilizes the dominant component features of aircraft to improve detection and recognition performance. Considering the SAR imaging mechanism, Kang et al. [36] proposed an innovative scattering feature relation network to improve detection performance. It guarantees the completeness of detection results by analyzing and enhancing their correlation. Zhao [37] presented a detector called Attentional Feature Refinement and Alignment Network for detecting aircraft in SAR images, achieving competitive accuracy and speed. Bi et al. [38] present a novel POL-SAR image classification method, achieving promising classification performance and preferable consistency.

2.1.2. Traditional Candidate Feature Methods in SAR Images

Manually designed features based on target structural and scattering features are used for feature representation in the conventional detection approach in the SAR domain. Target structural features include target contour, component shape, and structural distribution. The structural characteristics of the target are mainly manifested by the length and width of the target, the contour, the shape of components, and the structural configuration. Margarit et al. [39], Margarit and Mallorqui [40], and Margarit and Tabasco [41] conducted some work to improve the accuracy of ship classification by modeling the geometric features of the bow, midship and stern sections. Xu [42] et al. proposed a novel structured low-rank and sparse algorithm for high-resolution ISAR imaging, which can effectively deal with various types of sparse sampling and improve the image formation quality greatly. Gao [43] proposed a geometric feature-based aircraft interpretation method. The key geometric parameters of aircraft are obtained for interpretation by extracting the components of the aircraft, such as the engine and nose. In addition, as the SAR image resolution develops, the structural information of objects is presented in more detail, so the detection performance of SAR objects will be facilitated by making full use of radiometric and geometric scattering information. The aircraft in SAR images are usually composed of a series of strong scattered bright spots due to the imaging mechanism of SAR and the material construction of aircraft. Therefore, some scholars have studied how to effectively exploit strong scattering points for aircraft interpretation in SAR. The backscattering characteristics of aircraft are researched to acquire salient point vectors, which are integrated into the template-matching process for aircraft identification in TerraSAR-X images [44]. Zhu et al. [45] put forward a method in which the accuracy of SAR ship classification is improved by integrating the geometric topology of strong spots with amplitude intensity. In conclusion, the scattering features used for aircraft detection in SAR images can be divided into three main categories: peak features, statistical features, and texture features. The distribution of scattering centers can respond to the peak features, which can also describe the geometric shape of the target in a complex environment. Importantly, within a certain range of viewing angles, the position and intensity of strong scattering centers are statistically quasi-invariant [46]. Therefore, the extraction and application of peak features are critical to the orientation of aircraft in SAR images. Here, we focus on the peak features for aircraft detection, which is illustrated in Section 2.2.2.

2.2. The Proposed Method

2.2.1. Overview of the Proposed Method

In this work, the framework of PFF-ADN is illustrated in Figure 1, which mainly comprises three parts: peak feature extraction, peak feature fusion for enhancing the features of aircraft, and the ADN structure for detecting the image processed by PFF. In Section 2.2.2, we first introduce the extraction process of the peak feature, because it is the basis for the next step. Next, the peak feature fusion in Section 2.2.3 is introduced in detail. Finally, The ADN, which incorporates the ASFF and DCM, is described in detail. It is worth mentioning that the DCM and ASFF are introduced to the proposed approach. The two modules can greatly enhance the detector’s capacity for modeling geometric transformations and extracting abstract features, respectively. In Section 2.2.4, the principles, expression forms, and realization course of the two modules will be discussed in detail.

2.2.2. PFE

Target scattering features are the key information used in understanding SAR images deeply. The peak features of SAR images essentially reflect the scattering center of the target. Meanwhile, peak features can not only depict the geometric features and shape characteristics of the target, but also be very stable in the complex radar imaging system. Therefore, they are one of the most widely used features in SAR image target detection. The structure of an aircraft is relatively complex, the dimensions and details of which can be observed in single-polarization high-resolution SAR images. The strong scattering of aircraft mainly comes from the engine, nose, fuselage, wing, and tail edge. The different parts have different scattering mechanisms, including tip diffraction, cavity scattering, mirror reflection, edge diffraction, multiple reflections and so on [47], as illustrated in Figure 2.

Based on the aforementioned analyses, the scattering from aircraft is relatively strong, as exhibited in the highlighted regions in the SAR images. According to experiments, the extraction of peak features is generally stable, regardless of prior knowledge. The specific operation procedure of PFE is shown in Algorithm 1. Therefore, a peak feature enhancement and fusion approach based on the extraction of peak points and multi-channel fusion is proposed to enhance the scattering features of aircraft in SAR images, which facilitates the subsequent detection network.

Algorithm 1: Peak Feature Extraction
Input:	$\mathrm{The} \mathrm{image} I(x,y)$
	the mean $\mu$ and variance $\sigma$ of the background region
	Hyper-parameter: $k$ = 0.4
Output:	$\mathrm{Generated} \mathrm{peak} \mathrm{feature} \mathrm{points}: P=(p_1,p_2^{},\dots ,p_k)$
Main loop:
1:	Compute the gradient in the direction of $x$ and $y$:
2:	$I_x=\frac{\partial I}{\partial x}=I\otimes (-1,0,1),I_y=\frac{\partial I}{\partial x}=I\otimes {(-1,0,1)}^T$
3:	Calculate the product of the gradients of the two directions:
4:	$I_x^2=I_x^{}\cdot I_x^{}$$, I_y^2=I_y^{}\cdot I_y^{}$$, I_{xy}^{}=I_x^{}\cdot I_y^{}$
5:	A Gaussian weight is assigned to each gradient acquired:
6:	$A=g(I_x^2)=I_x^2\otimes \omega$$, C=g(I_y^2)=I_y^2\otimes \omega$,
7:	$B=g(I_{xy}^{})=I_{xy}^{}\otimes \omega$$, M=\left[\begin{array}{l}A C\\ C B\end{array}\right]$
8:	for $x$ $\in$ corner-point do
9:	$\hspace{1em}\mathrm{if} R=\left\{R:\det M-\alpha {(traceM)}^2<threshol{d}_H\right\}$
10:	add $x$$\mathrm{into} \mathrm{the} \mathrm{corner} \mathrm{points} \mathrm{set} C=C\cup \left\{x\right\}$
11:	else
12:	pass
13:	for $x$ $\in$ $C$ do
14:	if   $x>\mu +k\sigma { \mathrm{and} \min (\mathrm{a}}_{ij}-a_{N(i,j)})>\sigma$
15:	add $x$$\mathrm{into} \mathrm{the} \mathrm{peak} \mathrm{points} \mathrm{set} P=P\cup \left\{x\right\}$
16:	else
17:	pass

The peak feature points have strong gradient information, which is stable with changes in the external environment. Thus, it is of great importance to obtain the peak feature points for the subsequent feature enhancement and fusion. Given that the corner points have the advantage of rotation invariance, and insensitivity to noise and gradient features, the Harris detector is utilized to extract corner points first. Then, by comparing the corner points value and threshold, the peak points are automatically searched for. The mathematical expression of Harris is given below (1), where $w(x,y)$ denotes the window function and $w$ is an adjustable parameter, and $(x,y)$ is the pixel position in the image $I$. $E(\Delta x,\Delta y)$ represents the variation caused by window movement and $(\Delta x,\Delta y)$ is the shift pixel size of window.

$$E(\Delta x,\Delta y)={\displaystyle \sum _{x,y}w(x,y)[I(x+\Delta x,y+\Delta y)-I(x,y)]}{ }^2$$

(1)

Referring to Taylor series expansion, Equation (2) is derived by using Equation (1). $g(\sigma )$ stands for the Gaussian convolution kernel with scale $\sigma$, which convolves with a matrix, improving the anti-noise ability of the algorithm. $I_x$ is the first derivative of the position $x$. $\otimes$ represents the convolution operation.

$$M=g(\sigma )\otimes \left[\begin{array}{l}{I}_x{}^2\hspace{1em}{I}_x{I}_y\\ I_x{I}_y\hspace{1em}{I}_y{}^2\end{array}\right]$$

(2)

The scattering centers are defined as two kinds of peak feature points, which are comprised of a two-dimensional peak point (vertex) and a one-dimensional peak point (vertex of row and vertex of column). The row (column) vertex is the row (column) local maximum of the pixel within the target region of the SAR image and the vertex is the two-dimensional local maximum of the pixel within the target region of the SAR image. To weaken the influence of noise, when we acquire the peak points, the mean $\mu$ and variance $\sigma$ of the background region need to be estimated first. The peak features of a target $(i,j)$ can be defined by pixels within their domain:

$$p_{ij}=\{\begin{array}{l}0, else\\ 1, if{ \mathrm{a}}_{ij}>\mu +k\sigma { \mathrm{and} \min (\mathrm{a}}_{ij}-a_{N(i,j)})>\sigma \end{array}$$

(3)

If $p_{ij}=1$, it denotes that the current pixel is the peak point. If $p_{ij}=0$, it represents that the current pixel is not the peak point. ${\mathrm{a}}_{ij}$ is the pixel value of the current point. $N_{(i,j)}$ is the local area of ${\mathrm{a}}_{ij}$ The row (column) vertex $N_{(i,j)}$ is the two nearest neighbor pixels in $i$ row ($j$ column) of ${\mathrm{a}}_{ij}$. $k$ is an empirical parameter determined by conducting multiple experiments. In this paper, we adopt a peak extraction method based on the eight-domain pixel detection of local maxima, which is a classical method with pixel-level accuracy.

For the peak feature extraction of aircraft, three different types of aircraft are selected for a detailed analysis. As shown in Figure 3, Figure 3a is the original image of an aircraft slice; Figure 3b shows the three-dimensional maps of aircraft; Figure 3c shows the extraction results of peak points, which are marked in red to offer more intuitive information. It demonstrates the strong scattering points of aircraft can be more effectively indicated by the peak feature points.

The peak features of aircraft are scattered all over, mainly in the cavity structure of the engine, the complex electromagnetic structure of the cockpit, the dihedral angle formed by the combined components, the three-sided angle structure, and other strong scattering regions. A fine description of the aircraft structure can be obtained by the extraction of the peak features. The peak points of Figure 3a are mainly concentrated in the engine, tail, and wings and fewer peak points are in the fuselage. In Figure 3b, more peak points are distributed in the junction of fuselage and wing, which is more intuitive to visualize in three-dimensional maps. As shown in Figure 3c, the peak feature distribution is presented as a cross shape, which cannot be used to clearly distinguish the characteristics of each component. It can be inferred that the numbers and locations of peak points vary between types of aircraft. Based on this, peak features are critical in the application of target detection and recognition in SAR images.

It is necessary to handle the SAR images via the PFE method, owing to the aim, which is to detect aircraft in SAR images. By comparing Figure 4a–d, the intensity of the aircraft’s position in SAR images is significantly enhanced. The changes are intuitively observed in 2-D and 3-D illustrations of SAR images. As shown in the 3-D maps in Figure 4a–d, the differences in the scattering strength of aircraft in SAR images can be observed more clearly.

2.2.3. PFF

In this study, a very effective and simple method is applied to SAR data preprocessing. k times the average of non-zero data is computed to truncate and normalize the original SAR data, which is abbreviated to KTN, as detailed in Formula (4). For the experimental dataset, the single-channel SAR data is processed by the KTN method.

$$I=unit8\left[\frac{I(I\ge k\cdot mean(I)=k\cdot mean(I)}{\max (I)}\times 255\right]\in [0,255]$$

(4)

In Formula (4), $I$ stands for the data value of each channel. $k$ is a manually tunable parameter. $mean(I)$ and $\max (I)$ denote the average value of all non-zero data for each channel and the max value of data for each channel, respectively. The KTN method is utilized to handle the SAR data and convert them to a uint8 data format. Because of the network pre-trained by the optical samples of COCO [48], the images of COCO present in the RGB format with three-channel. Thus, the SAR data is necessary to be transformed into a three-channel format for detection network training better. In most pre-processing methods, single-channel eight-bit data is often transformed into three-channel data by replication. In this study, one channel of data is replaced with peak feature data $I_{PFE}$ which is obtained in Section 2.2.2, and the other channels adopt different multiples of the mean. By fusing these enhancements in target scattering, the strong scattering characteristics of targets can be learned by the network more effectively. In the process of synthesizing the training image, R and B channels are assigned the data processed by KTN. The data of the G channel is substituted for the data processed by PFE.

$$R=unit8\left[\frac{I(I\ge k\cdot mean(I)=k\cdot mean(I)}{\max (I)}\times 255\right]\in [0,255]$$

(5)

$$G=I_{PFE}$$

(6)

$$B=unit8\left[\frac{I(I\ge k\cdot mean(I)=k\cdot mean(I)}{\max (I)}\times 255\right]\in [0,255]$$

(7)

Data visualization based on KNT is shown in Figure 5. Three-times and six-times are set in the KNT, as illustrated in Figure 5a,c. Compared with Figure 5c, Figure 5a shows higher visual lightness. Figure 5b is processed by PFE in Section 2.2.2 and the peak features of aircraft in the image are extracted and enhanced, leading to the information of the target being more distinct.

2.2.4. ADN

(1)

DCM: The input feature map is sampled at fixed positions in the standard convolution operation, and the sampled pixels are mostly rectangles. This gives rise to obvious issues, such as that the receptive fields are of the same size in the same layer. So, the standard convolution has no ability to handle geometric transformations. Nevertheless, because of the SAR scattering mechanisms, the same target in SAR images appears in various shapes with changes in azimuth angle. In this paper, deformable convolution [49] is adopted to accommodate geometric variations or attitude sensitivity in aircraft viewpoint and the part deformation of aircraft in SAR images.

As shown in Figure 1, ResNet50 [50] is used in this experiment and a deformable convolution kernel is applied in stages 3,4,5 of ResNet50, which is inspired by the work of Zhu [51]. As mentioned below, due to the adaptive receptive field sizes of deformable convolution, the sampling positions of feature maps in stage Conv2D can be adaptively adjusted, which allows the subsequent ASFF module to give more robust representations of geometric transformations. The standard convolution includes two steps. Firstly, the input feature graph $X$ is sampled with a grid $R$ of a specific size. Secondly, the sampling points are dotted with the weight coefficient $W$ and these products are simply summed. As shown in Figure 6a, taking a 3 × 3 convolution kernel, for example, nine positions $p_0$ are taken as samples from $X$ to get the output feature map $y$. For grid $R$, (−1, 1), and (1, 1) represent the upper left corner and bottom right corner respectively, and define a 3 × 3 kernel with dilation 1.

$$\mathrm{R}=\left\{\right(-1,-1),(-1,0),\dots ,(0,1),(1,1)\}$$

(8)

The convolution output for each point $p_0$ is $y$

$$y(p_0)={\displaystyle \sum _{p_n\in R}w(}{p}_n)\times x(p_0+p_n)$$

(9)

In deformable convolution, the offsets $\Delta p_n(n=1,2,\dots ,N)$ is added to the grid $R$, So the output $y$ becomes

$$y(p_0)={\displaystyle \sum _{p_n\in R}w(}{p}_n)\times x(p_0+p_n+\Delta p_n)$$

(10)

As illustrated in Figure 6a, the offsets can be obtained by a parallel convolution unit and the sampling points of the input feature map are no longer regular rectangles. The nine cyan boxes indicate the sampling position in 3 × 3 standard convolution, while the corresponding blue boxes denote the sampling position in 3 × 3 deformable convolution. Figure 6b shows the receptive field of standard convolution (right) and deformable convolution (left). The sampling positions of standard convolution are fixed and rectangular, while the sampling positions are adaptively adjusted according to the shape and scale of instances. In conclusion, deformable convolution has the ability to learn receptive fields adaptively and enables adaptive part localization for aircraft with different shapes. For target location in a visual field, the deformable convolution proposed solves the problem of geometric transformations and it is conducive to feature extraction and fine localization.

(2)

ASFF: To fully exploit the semantic information of deep features and the high-resolution information of shallow features, the structure of the feature pyramid network is often used for feature fusion. However, the representational ability of the feature pyramid is constrained because of the inconsistency between multi-scale features. Inconsistency is reflected in detecting multi-scale aircraft in the same SAR image. The high-resolution information in the shallow layer is beneficial to the detection of small-size aircraft, while large-size aircraft can be detected with semantic information in the deep layer. When large-size aircraft are recognized as true positives, small-size aircraft are easily mistaken as false negatives, resulting in leak detection. Considering that that aircraft in SAR images consist of several discrete points and problems do occur with the detection of multi-scale aircraft, ASFF is introduced to filter conflicting information, suppressing the inconsistency and improving the scale-invariance of features [52]. The essence of ASFF is to adaptively learn the spatial weight of fusion for feature maps at each scale. Firstly, for the features of a given layer, features from other layers are adjusted to the same scale for fusion. Secondly, the best spatial weight for fusion is acquired by subsequent training. Finally, the features of all levels are adaptively aggregated at each level. In other words, some features carrying contradictory information may be filtered out, while other features with cataloged clues are retained.

ASFF-5 is shown in the ADN stage of Figure 1, as an example, the necessary measures of feature fusion can be divided into two steps. Firstly, to derive ASFF-5, the resolution of the feature at the level $l(l=3,4,5)$ is resized to $l(l=5)$. Secondly, the feature in each layer $(X3,X4,X5)$ is multiplied by the corresponding weight $(\alpha ,\beta ,\gamma )$ and sum the results. The equation is as follows:

$${\mathrm{y}}_{ij}^l={\alpha }_{ij}^l·x_{ij}^{3\to l}+{\beta }_{ij}^l·x_{ij}^{4\to l}+{\gamma }_{ij}^l·x_{ij}^{5\to l}$$

(11)

where $x_{ij}^{n\to l}$ represents the feature vector of the position $(i,j)$ on the feature maps adjusted from level $n$ to level $l$. ${\mathrm{y}}_{ij}^l$ denotes the output features, and ${\alpha }_{ij}^l$, ${\beta }_{ij}^l$, ${\gamma }_{ij}^l$ are respectively the weight parameters of different layers from $n(n=3,4,5)$ to $l$.

$${\alpha }_{ij}^l=\frac{{\mathrm{e}}^{{\lambda }_{{\alpha }_{ij}}^l}}{{\mathrm{e}}^{{\lambda }_{{\alpha }_{ij}}^l}+{\mathrm{e}}^{{\lambda }_{{\beta }_{ij}}^l}+{\mathrm{e}}^{{\lambda }_{{\gamma }_{ij}}^l}}$$

(12)

Here ${\lambda }_{\alpha ij}^l,{\lambda }_{\beta ij}^l,{\lambda }_{\gamma ij}^l$ are taken as the control parameters. The 1 × 1 convolution is utilized to compute the weight scalar maps $({\lambda }_{\alpha ij}^l,{\lambda }_{\beta ij}^l,{\lambda }_{\gamma ij}^l)$.

3. Experiments and Analyses

In short, experiments on the PFF-ADN we have proposed are conducted for illustration and comparison with GF3 datasets. To confirm the effectiveness of each module, the detection results of ablation experiments are presented and the contributions of different modules in this method are discussed.

3.1. Data Set Description and Parameter Setting

The built datasets contain 69 GF3 SAR images with sufficient desired variations. The GF3 images show C-band HH polarization with 1 × 1 m resolution, and contain 1903 SAR aircraft. These magnitude images with expert annotation cover a variety of complex scenes in which it is difficult to distinguish targets from the background noise, such as scenes where small aircraft are densely arranged, and scenes where both large and small aircraft exist together. In addition, aircraft adjacent to buildings are common in this dataset. Therefore, GF3 is established to verify the performance of the proposed algorithm in a complex background. We randomly chose 49 images as the training set and the remaining 20 images as the test set. The large-scale images in GF3 are about 20,000 × 20,000 pixels, which need to be cut into slices of 1000 × 1000 pixels with an overlap of 200 pixels. The reason for the 200-pixel overlap is to avoid cutting off the aircraft. Detailed information on the experimental dataset is given in

Table 1. Detailed information on the experimental dataset.

Dataset	Training	Test	Total
GF-3	495	165	660

. Besides this, the size distribution of aircraft in this dataset also varies considerably, as shown in Figure 7.

Parameters in the PFF stage: In the PFF step, for the image containing varied brightness information, the mean values of R and B channels are set to three and six empirically when the image data are processed by KTN.

Parameters in the ADN stage: the Resnet50 with DCM is taken as the backbone of the ADN, which is pre-trained on the COCO. The input size for ADN is 1000 × 1000 pixels and the basic anchor sizes for P3, P4, and P5 are set to 32, 64, and 128, respectively. The stochastic gradient descent (SGD) is applied to train the model with a batch size of eight images. The momentum and weight decay of the optimizer are given fixed values: 0.9, 0.0001, and the training steps are set to 60,000. The initial learning rate is set to 0.001 and decays to 0.0001 after 60,000 steps for the convergence of loss.

In addition, the experiment is performed on an Nvidia Titan 2080 Ti, and the configuration of the operating system environment includes Ubuntu16.04, CUDA10.1, Pytorch1.17, and Tensorflow 2.1.0.

3.2. Evaluation Metrics

To evaluate the performance of the detection task, a variety of evaluation metrics are employed in the experiment, namely, Precision ($P$), Recall ($R$), False Alarm Rate ($FA$), $F_1$-score, Average Precision ($AP$), and Running Time. These metrics are computed from four well-established values measured in the experiments: True Positive ($TP$), False Positive ($FP$), False Negative ($FN$), and running time for each scene. TP represents a correctly detected target.

$$P=\frac{TP}{TP+FP}$$

(13)

$$R=\frac{TP}{TP+FN}$$

(14)

$$FA=\frac{FP}{TP+FP}$$

(15)

$$F_1=\frac{2\times P\times R}{P+R}$$

(16)

$$AP={\displaystyle {\int }_0^{1}P(R)dR}$$

(17)

$TP$ and $FP$ represent the correctly detected objects and false alarms, respectively. $FN$ denotes a missing target. The intersection over union (IoU) indicates the extent of overlap between the predicted bounding box and the true bounding box in the image. The value of IoU is in the range of 0 to 1. When the two bounding boxes coincide completely, the value of IoU is 1. In this experiment, if the IoU is greater than 0.5, the detected bounding box can be considered as $TP$.

In addition, the $F_1$-score is utilized to evaluate the comprehensive performance of the proposed approach, which is defined in Formula (16). Since different precision and recall values arise at different confidence thresholds, the Precision–Recall (PR) curve is introduced to balance the two metrics, where the recall rate is the horizontal coordinate and the precision rate is the vertical coordinate. The value of $AP$ is the area under the PR curve, which can be used for evaluating the overall effectiveness of the algorithm. The method for calculating $AP$ is shown in the Formula (17).

3.3. Ablation Experiments

3.3.1. Effect of PFF

The effective pre-processing algorithm is beneficial to improving the performance of detection in SAR images, especially for the higher recall rate. The effectiveness of the PFF is given in

Table 2. Effectiveness of different components.

Algorithm	$\mathit{A}\mathit{P}$ (%)	$\mathit{P}$ (%)	$\mathit{R}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)
Baseline (*)	77.08	88.08	77.81	82.63
* + PFF	82.72	86.34	85.71	86.02

. As can be seen from the table, the recall rate decreases obviously, but the precision reduces only a little bit. Fortunately, when the PFF is added, the test AP and F1 increase by 5.64% and 3.39%. The two lines in Figure 8 show the implementation results of adding PFF on the number of false alarms and detected targets through a before-and-after comparison. In the first (from left to right) column, two false alarms are suppressed. In the second column, not only are two small false alarm targets suppressed, but a small target in a densely arranged SAR scene is also detected. However, there are still two large-size false alarms that are mistaken for aircraft, even with the added PFF. The reason why the false alarms are not eliminated may be that the features of the background are enhanced along with the target. Another reason may be that the characterization ability of the detection network is insufficient. In the third column, the aircraft at the bottom of the image is also detected. In the fourth column, although the results show three more false alarms of small size, three more aircraft can be assumed to be detected. Since the bounding boxes of the detected aircraft are too large, the IoU between the detected bounding box and the true bounding box is less than 0.5, so these are considered false alarms. It can be inferred that the features from PFF are rich but redundant, leading to serious interference. In a word, the scale-invariance features of aircraft in SAR images are enhanced by adding the PFF, improving the detection performance in SAR images to a certain extent. As shown in Figure 9, the PR curve indicates the superiority of the proposed model.

3.3.2. Effect of ADN

Experiments are designed to verify the effectiveness of ADN, which contains two aspects: the ASFF and the DCM. The quantitative evaluation results of ASFF are displayed in

Table 3. Effectiveness of different components.

Algorithm	$\mathit{A}\mathit{P}$ (%)	$\mathit{P}$ (%)	$\mathit{R}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)
Baseline (*)	77.08	88.08	77.81	82.63
* + ASFF	84.65	88.60	88.34	88.46
* + ASFF + DCM	86.15	87.36	90.97	89.12

. The precision rate and recall rate of the baseline algorithm are all increased up to 88.60% and 88.34%, respectively, by adding ASFF. The first two lines in Figure 10 show the results of adding ASFF on the number of false alarms and detected targets through a before-and-after comparison. In the first (from left to right) column, two false alarms are suppressed. In the second column, one more aircraft is detected in dense arrangements of SAR scenes. In the third column, two more aircraft are detected by adding the ASFF. However, one more false alarm appears, because the IoU is less than 0.5. The reason may be that the detection network cannot accurately extract the features of small targets. So, the DCM will be utilized to improve this condition in the next step.

In the fourth column, one more small-size aircraft and two large-size aircraft are detected. It might be inferred that the features of ASFF are rich, leading to more aircraft being detected. In a word, the detection algorithm gives full play to the characterization capabilities of the feature pyramid by adding ASFF, further improving the detection performance of multi-scale aircraft, especially for the small-size aircraft in SAR images.

Furthermore, another experiment is performed to reveal the effectiveness of DCM. As shown in Table 3, the algorithm’s performance is greatly enhanced by adding DCM. Compared to the methods without DCM, namely, * + ASFF, the method (* + ASFF + DCM) acquires a competitive advantage in terms of quantitative indicators. The recall rate is increased up to 88.34% from 90.97%. As displayed in the last two lines of Figure 10, more aircraft against a complicated background are detected in the fourth column of SAR scenes. This confirms that the deformable convolution introduced is beneficial to the fine localization of SAR aircraft. However, in the fourth column, there is still a situation in which noise is recognized as a false alarm. Encouragingly, when the DCM is added, the $F_1$ value increased to 88.46%, up from 89.12%. The PR curve representing the AP value is shown in Figure 9. It can be concluded from the PR curve that a great improvement in the AP value is achieved by adding the ASFF and the DCM models, respectively, demonstrating the effectiveness of each proposed model.

3.3.3. Performance of PFF-ADN

To validate the superiority of the proposed PFF-ADN further, the results are compared with those of other detectors.

Table 4. Effectiveness of different components.

Algorithm	$\mathit{A}\mathit{P}$ (%)	$\mathit{P}$ (%)	$\mathit{R}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)	FPS (Slice)
RFB	42.24	94.21	42.85	58.91	85
Yolov4-tiny	58.28	81.46	61.27	69.93	341
Mobile-yolov4	68.58	79.47	72.92	76.05	325
FasterRcnn	49.87	65.76	53.75	58.84	13
RetinaNet	78.36	80.04	81.95	80.98	268
ADN	86.15	87.36	90.97	89.12	11
PFF-ADN	89.34	89.44	92.85	91.11	11

lists the quantitative assessment results of the algorithms. All the metrics of the proposed approach have been improved greatly. Meanwhile, to better illustrate the performance of each approach, four representative images are selected to visually show the detection results of aircraft in different conditions. This is exemplified in Figure 11 A high false alarm rate and low recall rate appear in the results of lightweight algorithms, such as Yolov4-tiny and Mobile-yolov4. Besides this, in the case of small-size aircraft, the detection results of Yolov4-tiny, RFB, and FastRcnn-resnet50 are not very good. The reason why small-size aircraft are not detected is that these algorithms only utilize the high-semantic information of deeper layers to predict objects. RetinaNet adopts a feature pyramid structure that fuses the semantic information of deep features and the high-resolution information of shallow features, and then makes predictions in different feature layers, significantly improving the recall rate of small-size aircraft. However, as can be seen in the results of other detectors, there are still some aircraft missing. As such, in this paper, the PFF is proposed based on SAR image pre-processing. The PFF is employed to enhance the scattering features of aircraft in SAR images and thus catch the non-obvious features of aircraft. The ADN structure that incorporates the ASFF and the DCM is proposed as the network structure to improve accuracy. The ASFF is introduced to solve inconsistency between multi-scale features, boosting the representational capacity of the feature pyramid structure. The DCM is exploited to cope with the geometric variations or attitude sensitivity of aircraft in SAR images. The last three rows of Figure 11 show that the detection results of PFF-ADN are better than those of RetinaNet in four typical SAR scenes. In the first column, all the small-size aircraft are detected, while small-size aircraft appearing in dense arrangements are not detected in the RetinaNet. In the third column, the small-size aircraft in a dense arrangement are all detected by PFF-ADN, while two aircraft are missed by RetinaNet. In the fourth column, all aircraft are detected by PFF-ADN under complex conditions wherein both large and small aircraft exist.

To sum up, the validity of the two modules in PFF-ADN has been demonstrated by ablation experiments. All the results are presented in Table 2, Table 3 and Table 4. The algorithm we propose can not only achieve a higher recall rate, but also suppress false alarms in multiple complex conditions.

The Precision–Recall curve, which measures the detection performance of the network structure, is shown in Figure 12. The larger the area covered by the curve, the better the performance of the network structure. Therefore, in comparison with other detectors, the approach we propose achieves better detection performance in SAR images.

4. Discussion

The experimental results in Section 3.3.1 show the designed PFF is beneficial to enhance the detection performance in SAR images. To better assess the impacts, the quantitative results and visualizations are presented, demonstrating the effectiveness of the PFF. In a word, the pre-processing for SAR images is of great importance. Therefore, image pre-processing methods should be considered in practice. In the future, we can try to introduce some classical SAR image processing methods to explore their effects on SAR object interpretation based on DL.

The experimental results in Section 3.3.2 verified the effectiveness of the DCM and ASFF by ablation experiment. The precision rate and recall rate of the baseline algorithm are all increased up to 88.60% and 88.34%, respectively, by adding ASFF. It confirms that the characterization capability of the detection network is improved greatly by reconciling the inconsistency across different feature scales. The recall rate is increased up to 88.34% from 90.97% by adding the DCM. It demonstrates the DCM is conducive to fine localization by enhancing the geometric modeling abilities of aircraft in various shapes. However, ASFF and DCM requires more computation, which affects the detection efficiency. Therefore, in the future, we can explore how to achieve the balance between precision and speed in specific conditions.

To further verify the superiority of PFF-ADN, the contrast experiment with other classical detectors is conducted. The detailed results are shown in Section 3.3.3. The quantitative results and visualizations all demonstrate that the proposed method can not only achieve a higher recall rate, but also suppress false alarms in multiple complex conditions. In the future, we consider applying the proposed method to remote sensing large-scale images, paving the way for practical application.

5. Conclusions

A novel approach called PFF-ADN for aircraft detection under different arrangements of SAR images is proposed in this paper. This method improves detection performance by combining the classical feature extraction methods with DL approaches. Firstly, an effective image preprocessing approach (PFF) is outlined, in which the features of the target in the image are enhanced by the extraction of peak features and multi-channel data fusion. It is beneficial for handling incomplete scattering and ambiguous characteristics in SAR images. Secondly, in the ADN phase, a new topology of the network composed of ASFF and DCM is proposed to tackle the multiple scales and attitude sensitivity. The ASFF optimizes the characterization capabilities of the network by reconciling the inconsistency across different feature scales, which facilitates the detection of multi-scale aircraft, especially small-size aircraft. The DCM refines the geometric variation capacity of the detection network, which contributes to the handling of the attitude sensitivity and the various shapes of aircraft in SAR images. The satisfactory results of the ablation experiments on the GF3 datasets demonstrate the effectiveness of each proposed module. Meanwhile, detailed comparisons with various methods have also verified the superiority of the proposed algorithm in various circumstances.

In the future, a tight combination between network design and intrinsic properties of aircraft in SAR images need to be considered. In addition, in order to promote the application as soon as possible, it is also very important to carry out SAR object detection under the SAR large-scale scenarios.