FFA: Foreground Feature Approximation Digitally against Remote Sensing Object Detection

Abstract

In recent years, research on adversarial attack techniques for remote sensing object detection (RSOD) has made great progress. Still, most of the research nowadays is on end-to-end attacks, which mainly design adversarial perturbations based on the prediction information of the object detectors (ODs) to achieve the attack. These methods do not discover the common vulnerabilities of the ODs and, thus, the transferability is weak. Based on this, this paper proposes a foreground feature approximation (FFA) method to generate adversarial examples (AEs) that discover the common vulnerabilities of the ODs by changing the feature information carried by the image itself to implement the attack. Specifically, firstly, the high-quality predictions are filtered as attacked objects using the detector, after which a hybrid image without any target is made, and the hybrid foreground is created based on the attacked targets. The images’ shallow features are extracted using the backbone network, and the features of the input foreground are approximated towards the hybrid foreground to implement the attack. In contrast, the model predictions are used to assist in realizing the attack. In addition, we have found the effectiveness of FFA for targeted attacks, and replacing the hybrid foreground with the targeted foreground can realize targeted attacks. Extensive experiments are conducted on the remote sensing target detection datasets DOTA and UCAS-AOD with seven rotating target detectors. The results show that the mAP of FFA under the IoU threshold of 0.5 untargeted attack is 3.4% lower than that of the advanced method, and the mAP of FFA under targeted attack is 1.9% lower than that of the advanced process.

1. Introduction

The development of remote sensing (RS) imaging techniques has led to the advancement of many important applications in RS, including image classification [1,2,3], image segmentation [4,5], object detection [6,7,8], and object tracking [9,10]. Recent studies have found that image-processing techniques based on deep learning exhibit greater potential than traditional image-processing techniques, which makes them a hot research direction and leads to the current implementation of these tasks using deep neural networks (DNNs). Xie et al. [11] introduced a new landslide extraction framework using context correlation features, effectively utilizing contextual information and improving feature extraction capabilities. A deeply supervised classifier was also designed to enhance the network’s ability to recognize information in the prediction stage. Zhu et al. [12] proposed a cross-view intelligent human search method based on multi-feature constraints, which designed a variety of feature modules to fully extract the context information, feature information, and confidence information and improve the searchability of people in complex scenarios. Xu et al. [13] proposed a method to extract building height from high-resolution, single-view remote sensing images using shadow and side information. First, multi-scale features were used to extract shadow and side information. Then, the accuracy of measuring the building side and shadow length was improved by analyzing the geometric relationship between buildings.

Recent research has demonstrated that DNNs are susceptible to certain perturbations, and the results of detectors can be altered by adding a perturbation to an image that is unrecognizable to the human eye, despite the fact that deep learning has achieved increased success [14,15]. This phenomenon reveals the great security risks of DNNs and also inspires people to research their security. Adversarial examples (AEs), which can be created by subtly altering the source image to increase the success rate of attacks, were initially identified by Szegedy et al. [16]. Since then, more and more people have been devoted to studying the vulnerabilities of DNNs and improving their security and robustness.

Object detection [17], as an important research area in image processing, has likewise given birth to some important applications, such as autonomous driving [18,19] and intelligent recognition systems [20,21]; naturally, it has yet to escape the attack of AEs. Compared with attacking image classification, attacking object detection is more difficult [22]. In image classification, each image contains only one category, which is easier to realize. In object detection, in which each image contains multiple objects, it is necessary to find out the object as well as determine its location and lock its size, which leads to object detection being more practical but more difficult to attack. Therefore, it is more meaningful to attack the object detection system and study its vulnerabilities.

Adversarial attacks can be classified into digital and physical attacks, depending on the domains in which they are applied [22]. Digital attacks mainly make the detector detect errors by changing the image pixels, which is applicable to all images, such as the DFool attack proposed by Lu et al. [23] and the DAG attack proposed by Xie et al. [24]. Physical attacks mainly make the detector misclassify by adding patches to the objects, mainly for images with a large percentage of objects. Kurakin et al. [25] first experimentally verified that AEs are also present in the actual world, and Thys et al. [26] successfully spoofed a pedestrian monitor by generating adversarial patches.

Remote sensing images (RSIs), with their high resolution and wide spatial distribution, add some difficulties for object detection and likewise lead to their difficulty in realizing physical attacks [27,28]. Therefore, the majority of the current attacks on RSIs are digital attacks, and a few physical attacks are limited to RSIs with a large percentage of targets. Czaja et al. [27] first investigated the AEs in RSIs and successfully attacked the image classification models. Lu et al. [28] successfully designed an adaptive patch to attack the object detectors (ODs) for RSIs. Du et al. [29] and Lian et al. [30] successfully applied adversarial patches against RSIs to the physical world. More and more people have been devoted to the work of RS adversarial attacks.

The work of Xu et al. [3] and Li et al. [22] inspires us. Xu et al. [3] investigated the generic AEs for RSI classification models, and Li et al. [22] devised a method for adaptively controlling the perturbation size and location based on the behavior of the ODs. It is well known that object detection has three main tasks: determining the object location, determining the object size, and determining the object category [17]. These are indispensable, and the attack can be realized by successfully attacking any one of the three tasks. Determining the object category is the image classification task; so, we choose to attack the classifier in the ODs. We found that for object detection, using global adversarial perturbation will result in significant resource waste since the objects in the RSIs only represent a minor fraction of the entire image, and the background accounts for a much larger portion than the foreground; thus, we use local adversarial perturbation, which only changes the pixels in the foreground portion.

We suggest a digital attack method in this paper, which we call foreground feature approximation (FFA), which mainly utilizes the backbone network of the ODs (feature extraction network) to attack the classifier, which is an important part of the detection task and is most effective in utilizing it for the attack. The attack effect of FFA is shown in Figure 1.

Figure 1. Illustration of the FFA realizing the targetless attack and the targeted attack. The purple box indicates the part that joins the perturbation and succeeds in the attack, and the green box indicates the part that does not join the perturbation and is recognized correctly. The orange dashed box part indicates the targetless attack, and the blue dashed box part indicates the targeted attack.

The contributions of this paper are as follows:

• We propose a universal digital attack framework for foreground feature approximation (FFA) using the output information of multiple modules of target detectors, aiming to find common vulnerabilities between different detectors and improve the aggression and migration of adversarial examples;
• We used a relatively complete evaluation system, including the attack efficiency, attack speed, quality of the confrontation sample, and other relevant parameters and made a detailed quantitative evaluation of the algorithm in this paper, which improved the persuasiveness of the results;
• The results of attacking seven rotating target detectors on the remote sensing target detection datasets DOTA and UCAS-COD show that our method can produce confrontation samples with strong attack ability, high mobility, and strong non-sensing type at a fast speed.

2. Related Work

We touch on the theory of adversarial attacks in this section, including those used in image classification, object detection, and RS.

2.1. Adversarial Attacks in Image Classification

Image classification is a critical task in machine learning, the goal of which is to enable the computer to assign input images to the real categories [31]. Given an input image $\mathit{x}$, whose true label is y, the model’s prediction function is $f(·)$. The task of correctly classifying the image is even if the model’s prediction is equal to the true label of the target—that is, $f\left(\mathit{x}\right)=y$. Generating AEs ${\mathit{x}}^{\prime }$ for an image classification model requires generating an adversarial perturbation $\delta$; then, ${\mathit{x}}^{\prime }=\mathit{x}+\delta$. By limiting the size of the perturbation, it is possible to generate AEs that appear unchanged to the human eyes but that the computer cannot correctly classify, even if $f\left({\mathit{x}}^{\prime }\right)\ne y$. The above method is a targetless attack, which only requires that the computer be rendered incapable of correct classification. Adversarial examples also enable targeted attacks, also known as directed attacks, where the goal is to make the computer assign our carefully designed AEs to a particular class, assuming that the label of the particular class is p and $f\left({\mathit{x}}^{\prime }\right)=p\ne y$.

In the beginning, researchers studied white-box attacks [32,33,34]. This method assumes that the structure and parameters of the models we want to attack are known, and when we attack according to this known information, the attack success rate of these methods is higher. For example, the optimization-based method L-BFGS [16] and gradient-based method FGSM [35] are white-box attack methods. However, in practice, it is difficult to obtain the framework and settings for the target models, so black-box attacks arise [36]. A black-box attack assumes that the framework and settings for the target models are unknown, and we need to discover the vulnerability of the target models through the known models and other information to realize the attack. The attack success rate of this method is generally low but has a high value of use. For example, transfer-based attacks (Curls&Whey [36], FA [2], etc.), query-based attacks (ZOO [37], MCG [38], etc.), and decision boundary-based attacks (BoundaryAttack [39], CGBA [40], etc.) are black box attack methods.

2.2. Adversarial Attacks in Object Detection

Object detection, an advanced task of image classification and segmentation, occupies an increasing share in computer vision and is also applied in task scenarios such as face recognition [41], unmanned vehicles [42], aircraft recognition [43], terrain detection [44], etc. The goal is for the computer to find the part of interest in an image and give it the correct classification, location, and size. The emergence of AEs also puts higher demands on the security of object detection.

Xie et al. [24] first discovered the existence of AEs in object detection and semantic segmentation tasks. They proposed the DAG attack with good transferability between multiple datasets and different network structures. The target of the attack is placed in the region of interest (RoI), and the attack expression is

$${\mathit{\delta }}_m=\sum _{t_n\in T_m}[{\nabla }_{X_m}{f}_{l_n^{\prime }}(X_m,t_n)-{\nabla }_{X_m}{f}_{l_n}({\mathrm{X}}_m,t_n)],$$

(1)

$${\mathit{X}}_{m+1}={\mathit{X}}_m+{\displaystyle \frac{\alpha }{{∥{\mathit{\delta }}_m∥}_{\infty }}}{\mathit{\delta }}_m,$$

(2)

where ${\mathit{\delta }}_m$ denotes the perturbation generated by the mth iteration and ${\mathit{X}}_m$ denotes the AE generated by the mth iteration. $t_n$ denotes a certain object among all objects $T_m$ in the image, $f(·)$ denotes the result function of object detection, $l_n$ refers to the true category label of the attacked object, and $l_n^{\prime }$ refers to the category label specified by the attack. The perturbation is also normalized in order to limit the range of the perturbation. $\alpha$ denotes the iterative learning rate, which controls the range of the perturbation and is a fixed hyperparameter value.

Lu et al. [23] proposed the DFool attack to construct AEs against intersection stop signs in videos to successfully attack the then more advanced Faster RCNN  [45] model and implement the attack in the physical world as well. It generates AEs by minimizing the average score produced by Faster RCNN for all stop signs in the training set. The average score of stop signs is

$$\Phi \left(\mathcal{T}\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}b\in B_s\left({\mathcal{I}}^{mean}({\mathcal{M}}_i,\mathcal{T})\right){\Phi }_s\left(b\right),$$

(3)

where $\mathcal{T}$ denotes the texture of the stop sign in the root coordinate system and $\mathcal{I}({\mathcal{M}}_i,\mathcal{T})$ denotes the image $\mathcal{I}$ formed using the mapping ${\mathcal{M}}_i$ superimposed on the $\mathcal{T}$. ${\mathcal{M}}_i$ is the mapping formed by the correspondence between the eight vertices of the stop sign and the vertices of the $\mathcal{T}$. $B_s\left(\mathcal{I}\right)$ is the set of stop signs detected by the image $\mathcal{I}$ through the Faster RCNN, b is a certain detection box, and ${\Phi }_s\left(b\right)$ denotes the score predicted by the Faster RCNN for the detection box.

Afterwards, a small gradient descent direction is used for optimization:

$${\delta }^{\left(n\right)}=\mathrm{sign}({\nabla }_T\Phi \left({\mathcal{T}}^{\left(n\right)}\right)),$$

(4)

$${\mathcal{T}}^{(n+1)}={\mathcal{T}}^{\left(n\right)}+\epsilon {\mathit{\delta }}^{\left(n\right)},$$

(5)

where sign(·) denotes the sign function, which is used to control the size of the perturbation, and ${\mathcal{T}}^{(n+1)}$ denotes the AE generated by the n + 1st iteration. $\epsilon$ denotes the iteration step size and ${\mathit{\delta }}^{\left(n\right)}$ denotes the adversarial perturbation generated by the nth iteration.

Both of the above methods are global perturbation attacks [3,46], i.e., each pixel in the image is changed to generate the AE. The defects of these methods are very obvious. They change the foreground and background at the same time, resulting in a waste of attack resources and prolonging the attack time. Based on this, researchers have discovered local perturbation attacks [22,47] and gradually developed them as patch attacks [26,30] in the physical world.

The Dpatch attack was put forward by Liu et al. [48], which centers on generating a patch with the ground truth detection box property and optimizing it into a real object through backpropagation. In addition, the OD is made to ignore other possible objects. For two-stage ODs, such as Faster RCNN [45], the idea of the attack is to make the RPN network unable to generate the correct candidate network so that the generated patch becomes the only object. For single-stage ODs, such as YOLO [49], the core of which is the bounding box prediction and confidence scores, the attack is to make the grid where the patch is located be regarded as an object and ignore other grids. The objective function is optimized by training the patches, and the resulting patches are

$$\widehat{p}=arg\underset{p}{max}{E}_{x,t,l}[log\mathcal{P}\left(\widehat{y}\right|\mathcal{A}(\mathit{p},\mathit{x},l,t))],$$

(6)

where $\mathcal{A}(\mathit{p},\mathit{x},l,t)$ represents that the input image $\mathit{x}$ places the patch $\mathit{p}$ on l through a transformation t—the transformation t includes basic transformations such as rotation scaling. $\mathcal{P}\left(\widehat{y}\right|\mathcal{A})$ represents the likelihood that the input $\mathcal{A}$ is associated with the label $\widehat{y}$. The principle of Dpatch is to maximize the loss of the ODs to the true label and the bounding box label.

Thys et al. [26] designed the first printable adversarial patch for pedestrians, which achieved better attack results against the YOLO V2 OD in the physical world. Their optimization objective consists of three parts.

One is the non-printable score $L_{nps}$, which donates whether the patch can be printed using a printer:

$$L_{nps}=\sum _{p_{\mathit{patch}}\in P}\underset{c_{\mathit{print}}\in C}{min}\left|p_{\mathit{patch}}-c_{\mathit{print}}\right|L_{nps},$$

(7)

where $p_{\mathit{patch}}$ represents a pixel in P, and $c_{\mathit{print}}$ represents a color in a set of printable colors in C. This loss benefits colors in the image that closely resemble the colors in the set of printable colors.

The second is the total cost $L_{tv}$ in the image, which is used to ensure a smoother transition between the patch and the image:

$$L_{tv}=\sum _{m,n}\sqrt{{(p_{m,n}-p_{m+1,n})}^2+{(p_{m,n}-p_{m,n+1})}^2},$$

(8)

The third is the maximum objective score $L_{obj}$ of the image, which represents the score assigned to the object detected by the detector. The three are combined according to reasonable weights, and adversarial patches are generated iteratively.

2.3. Adversarial Attacks in Remote Sensing

Adversarial attacks have been investigated in ground imagery but they also have numerous uses in RS. Li et al. [22] designed an adaptive local perturbation attack method aiming at generalizing the attack to all ODs, which establishes an object constraint to adaptively utilize foreground–background separation to induce perturbations as a way of controlling the magnitude and distribution of perturbations so that perturbations are attached to the object region of the image. It employs three loss functions—classification loss, position loss, and shape loss—to generate the AE, and the three parts of the loss are rationally combined using a weighted approach.

$$L=\sum _{n=1}^N\alpha {\mathcal{L}}_{shape}(b_n,b_n^{\prime })/N+\sum _{n=1}^N\beta {\mathcal{L}}_{loc}(b_n,b_n^{gt})/N+\sum _{n=1}^N\tau {\mathcal{L}}_{cls}(p_n,{\ell }_b),$$

(9)

where N denotes objects’ number in the image, ${\mathcal{L}}_{shape}$ denotes the shape constraint, ${\mathcal{L}}_{loc}$ denotes the location constraint, ${\mathcal{L}}_{cls}$ denotes the classification constraint, and $\alpha ,\beta ,\tau$ refers to the weights of the three constraints. Each object is assigned a random bounding box $b_n^{\prime }$ for a shape attack, a real border $b_n^{gt}$ for a localization attack, and a background category ${\ell }_b$ for a classification attack. In addition, the distribution of perturbations is adaptively controlled using the foreground separation method so that the perturbations are attached to the foreground as much as possible.

Lian et al. [30] investigated physical attacks in RS. They proposed the patch-based Contextual Background Attack (CBA), which aims to achieve steganography of the object in the detector without contaminating the object in the image. It places the patch outside the object instead of on the object, which is more conducive to the realization of physical attacks. An AE with an adversarial patch can be precisely described as

$${\mathbf{x}}_{adv}=(1-{\mathcal{M}}_{{\mathbf{p}}^{*}})\odot x+{\mathcal{M}}_{{\mathbf{p}}^{*}}\odot {\mathbf{P}}^{*},$$

(10)

where ${\mathbf{P}}^{*}$ denotes the antipatch, ${\mathcal{M}}_{{\mathbf{p}}^{*}}$ denotes the mask of the antipatch, foreground pixels are 1 and background is 0, and ⊙ denotes the Hadamard product.

Objectivity loss and smoothness shrinkage are also used to update the adversarial patches. Objectivity loss refers to updating the AEs by detecting differences between objects in the inner and outer parts of the patch, and smoothing contraction reduces the distance between the patch and the target to make the transition smoother.

3. Methodology

3.1. Problem Analysis

To design a compliant AE, two aspects need to be considered: (1) where the perturbation is placed; (2) how the perturbation should be designed.

3.1.1. Location of the Perturbations

To enhance the effectiveness of the perturbation, we need to choose a suitable location to add the perturbation. Compared with the global adversarial perturbation for the image classification task, it is not very suitable to design a global adversarial perturbation for the object detection task. Especially for RSI object detection, the number of objects is large, and the area occupied is small. In this case, we need to design a more useful perturbation generation strategy to limit the perturbation to a suitable range.

3.1.2. Magnitude of the Perturbations

After determining the location of the perturbation, the next step is to determine its design scheme. The principle of perturbation generation is to successfully deceive the ODs and ensure that human eyes cannot perceive them. Therefore, the design of the perturbation can be seen as a joint optimization problem by considering multiple factors and balancing multiple losses so that the final generated AEs are globally optimal rather than locally optimal.

The input picture is expected to be $\mathit{x}$, and there are n objects in the image. ${B^i}_n=(w_n,h_n,x_n,y_n,l_n,s_n)$ is the parameter of the ith object, which represents the width, height, centroid horizontal coordinate, centroid vertical coordinate, object label, and object score of the object box, respectively. When detecting rotated objects, the object box’s rotation angle parameter must be added. The goal of AE design is to identify an appropriate perturbation $\delta$ that satisfies   

$$arg\underset{\delta }{min}{P}^i(\mathit{x}+\delta )\ne {B^i}_n,$$

(11)

where P denotes the detector, $P^i(\mathit{x}+\delta )$ represents the image’s ith detection result, and the generated AE is $\mathit{x}+\delta$. It can be seen from this that there are three manifestations of a successful attack: first, the size of the detection box is incorrect; second, the location of the detection box is incorrect (including no detection box, i.e., the object is “invisible” in the detector); third, the category of the detected object is incorrect. These three manifestations can be superimposed.

3.2. Overview of the Methodology

The topic of attacks against ODs has been extensively studied. Since ODs contain several important components, attacks against these specific components are effective. Still, they pose a huge problem at the same time: specific ODs contain specific frameworks and components, which prevent the generalization of the attacks [22]. In order to improve the transferability of attacks, we need to discover the similarities between different ODs. All ODs are inseparable from a common backbone network. The most commonly used backbone network is the ResNet network [8]. So, we can change the features of the image for the attack and form an AE with high transferability. The process of extracting features is visually represented in Figure 2.

Figure 2. Visualization of the feature extraction process. The left column shows the input image, the middle column shows the foreground portion of various images, and the right column represents the corresponding shallow features of the foreground image. Blue arrows indicate the foreground extraction process, and purple arrows indicate the feature extraction process. The orange dashed box portion indicates an targetless attack, and the blue dashed box portion indicates a targeted attack.

We first use an OD to detect the image; separate the high-quality foreground and the background; make a record of the foreground data of the image, including the object category as well as the position and the object box size; and cut the image of all the object boxes as a backup. After that, we need to design a hybrid image with the principle that it cannot contain objects of either category and its dimensions are identical to those of the original image. Utilizing the backbone network for obtaining the features of hybrid foreground and high-quality foreground, we design the body of the loss function by minimizing the KL divergence between the latter and the former and use the prediction result of the OD to perform the assisted attack. To accomplish the attack, the original image’s pixels should finally be altered so that the hybrid image’s features roughly match those of the original image. The flowchart of FFA attack is shown in Figure 3.

Figure 3. Flowchart of FFA for generating AEs. The bottommost arrow in the figure represents the adversarial example generation process. First, we need to generate a hybrid image $\tilde{\mathit{x}}$. Input image $\mathit{x}$ after the detector to obtain the predicted foreground box $B_{pre}$ and predicted labels $l_{pre}$. Use $B_{pre}$ to intercept the input image $\mathit{x}$ and hybrid image $\tilde{\mathit{x}}$ to obtain the input foreground ${\mathit{x}}_{\mathit{f}}$ and hybrid foreground $\widetilde{{\mathit{x}}_{\mathit{f}}}$. After extracting the features through the backbone network, the KL divergence of the two is calculated as the feature loss. The Smooth L1 loss between $B_{pre}$ and the real object box $B_{truth}$ is also calculated as the object box loss, and the cross-entropy loss between $l_{pre}$ and the background label $l_{bg}$ is calculated as the classification loss, which together constitute the detector prediction loss. The sum of the L2 distance between the i-1st generated AE ${\mathit{x}}_{adv}^{i-1}$ and $\mathit{x}$ is calculated as the perception loss. The weighted combination of feature loss, prediction loss, and perception loss yields the total loss, which is iterated to generate the adversarial perturbation ${\delta }_i$, which is summed with $\mathit{x}$ to obtain the adversarial example ${\mathit{x}}_{adv}^i$ after iteration.

3.3. Foreground Feature Approximation (FFA) Attack

3.3.1. Targetless Attack

Location of the Perturbation

For the object detection task, only the foreground section of the image, which is a small part, is relevant, while the background portion is irrelevant. Therefore, to attack the object detection, we only need to add perturbations to the portion of the ODs that is of interest. We use the attacked ODs to predict the image, obtain the prediction boxes, and perform further filtering based on the prediction boxes scores:

$${\mathit{T}}_{bboxes}=\left\{\begin{array}{cc}{P}_{bboxes}\left(\mathit{x}\right),\hfill & score{s}_{max}>\alpha \hfill \\ 0,\hfill & score{s}_{max}<\alpha \hfill \end{array}\right.(0<\alpha <1),$$

(12)

where ${\mathit{T}}_{bboxes}$ is the object to be attacked; ${\mathit{P}}_{bboxes}$ is the prediction boxes of the ODs; $score{s}_{max}$ is the maximum value of the prediction scores for each category; $\alpha$ is the threshold of the prediction scores; the prediction boxes with prediction scores greater than $\alpha$ are selected as the object to be attacked; and the prediction boxes with prediction scores less than $\alpha$ are considered to have no value to be attacked, so we do not attack them.

The Size of the Perturbation

Most of the previous methods design perturbations based on the predictive information of ODs and construct a variety of loss functions based on the object boxes detected by the ODs as well as the object categories, including position loss, shape loss, category loss, and so on. These methods depend on the accuracy of the ODs and the fact that the generated AEs do not have high transferability. Based on this, we propose a feature approximation strategy to change the features of the image in the view of the backbone network to mislead the detectors and achieve the attack effect. The loss function is designed as follows.

Feature Loss: After selecting the foreground portion to attack, we need to design a suitable loss function to update the perturbation iteratively. Before starting, we need to generate a hybrid image $\tilde{\mathit{x}}$ to determine the direction of each iterative update. We choose ten images from the training set of the dataset to be superimposed, and the final image formed by the superimposition is a hybrid image that does not contain any object, which we call the full background image.

Using the prediction boxes obtained from the ODs, the input image ${\mathit{x}}_{adv}^i$ and the hybrid image $\tilde{\mathit{x}}$ of the ith iteration are cropped separately to obtain the input foreground ${\mathit{x}}_{adv\_f}^i$ and the corresponding hybrid image ${\tilde{\mathit{x}}}_f$. We define the feature loss function as the KL divergence between the two images’ pixel distributions:

$${\mathcal{L}}_{feat}(\mathit{\theta },{\mathit{x}}_{adv\_f}^i,{\tilde{\mathit{x}}}_f)=-\sum _{R=1}^{N_R}\sum _{C=1}^{N_C}\sum _{K=1}^{N_K}{f}_{\mathit{\theta }}{\left({\mathit{x}}_{adv\_f}^i\right)}^{(R,C,K)}log{\displaystyle \frac{f_{\mathit{\theta }}{\left({\mathit{x}}_{adv\_f}^i\right)}^{(R,C,K)}}{f_{\mathit{\theta }}{\left({\tilde{\mathit{x}}}_f\right)}^{(R,C,K)}}},$$

(13)

where ${\mathcal{L}}_{feat}$ denotes the feature loss; $\mathit{\theta }$ denotes the backbone network parameters; and $N_R,N_C,N_K$ denote the number of rows, columns, and channels of the feature map, respectively. $f_{\mathit{\theta }}(·)$ denotes the mapping function for extracting shallow features using the backbone network. Note that the formula is preceded by a negative sign in order to minimize the KL divergence of the clean and mixed foregrounds.

To improve the effectiveness of the attack, we also have to design the prediction loss to assist the attack based on the ODs’ prediction results. The classification loss and the object box loss are the two components of this loss.

Classification Loss: Let the AEs carry incorrect labeling information to mislead the classifier. Since the probability that the predicted object’s background is calculated during the detector prediction process, it has a background label, which we assume to be $l_{bg}$. Assume that the prediction results in m objects ${\mathit{x}}_0,{\mathit{x}}_1,\cdots {\mathit{x}}_{m-1}$. Their labels are finding the true label of a prediction box by calculating the cross-entropy loss of the true box to the prediction box. By reducing the cross-entropy loss between the background category and the prediction box category, the attack is accomplished. The classification loss for each object is

$${{\mathcal{L}}^j}_{cls}({\mathit{x}}_j,l_{bg})=\sum _{C=1}^{N_L}{l_{bg}}^{\left(C\right)}logP{\left({\mathit{x}}_j\right)}^{\left(C\right)},$$

(14)

where $N_L$ indicates the number of categories in all images, $P(·)$ is the mapping function of the ODs’ detection results, ${\mathit{x}}_j$ is the jth object, and the total classification loss for this image is

$${\mathcal{L}}_{cls}(\mathit{x},l_{bg})=\sum _{j=1}^m{\mathcal{L}}_{cls}^j({\mathit{x}}_j,l_{bg}),$$

(15)

Object box loss: This loss is used to control the offset between the real object box and the prediction box. By increasing this offset, the gap between the detection result and the real result becomes bigger and bigger to carry out the attack. Assuming that the centroid of the jth prediction box has the coordinates $O_j=(h_j,v_j)$, and the centroid of the real box corresponding to this prediction box has the coordinates $O=(h,v)$, the Euclidean distance $D_j$ between the two is calculated and the object box loss is defined in terms of the Smooth L1 loss at this distance:   

$${{\mathcal{L}}^j}_{box}=SL1\left(D_j\right)=\left\{\begin{array}{cc}0.5D_j^2,\hfill & D_j<1.0\hfill \\ D_j-0.5,\hfill & otherwise\hfill \end{array}\right.,$$

(16)

The object box loss for the entire image is then the sum of the object box losses for each prediction box:   

$${\mathcal{L}}_{box}=\sum _{j=1}^m{{\mathcal{L}}^j}_{box}\left(D_j\right),$$

(17)

The prediction loss is the sum of the categorization loss and the object box loss, both of which are equally weighted:   

$${\mathcal{L}}_{pre}={\mathcal{L}}_{cls}+{\mathcal{L}}_{box},$$

(18)

Perceptual Loss: In order to make the perturbation look more natural, we also need to control the range of the perturbation. The perceptibility of the AEs is reduced by minimizing the L2 distance between the AEs and the original images. Assuming that the ith iteration generates an adversarial example ${\mathit{x}}_{adv}^i$, we define the perceptual loss as

$${{\mathcal{L}}^i}_{per}={∥{\mathit{x}}_{adv}^i-\mathit{x}∥}_2,$$

(19)

The ultimate loss function is a weighted combination of the three losses:   

$$\mathcal{L}={\mathcal{L}}_{feat}+\beta {\mathcal{L}}_{pre}+\gamma {\mathcal{L}}_{per},$$

(20)

The AE generated in the ith iteration is

$${\mathit{x}}_{adv}^i={\mathit{x}}_{adv}^{i-1}+{\nabla }_{\mathbf{x}}{\mathcal{L}}^i,$$

(21)

where ${\nabla }_{\mathit{x}}{\mathcal{L}}^i$ denotes the gradient of the computational loss ${\mathcal{L}}^i$ with respect to the original image $\mathit{x}$.

Algorithm 1 gives the implementation details of the FFA to achieve the targetless attack.

Algorithm 1: Foreground feature approximation (FFA).

Input:       - Input image $\mathit{x}$ and its ground truth $B_n$;       - A object detector D with the prediction function $P(·)$ and the feature extraction function $f(·)$;       - Hybrid image $\tilde{\mathit{x}}$.   Initialization:       1. ${\mathit{x}}_{adv}^0\leftarrow \mathit{x},I\leftarrow 50,\alpha \leftarrow 0.75,\beta \leftarrow 0.1\gamma \leftarrow 0.1.$       2. $T_{bboxes}\leftarrow P^D{\left(\mathit{x}\right)|}_{score{s}_{max}>a}$   Iteration:        for t in $range(0,I)$ do            1. ${\mathit{x}}_{adv\_f}^i\leftarrow T_{bboxes}\left({\mathit{x}}_{adv}^i\right),{\tilde{\mathit{x}}}_f\leftarrow T_{bboxes}\left(\tilde{\mathit{x}}\right)$            2. ${\mathcal{L}}^i\leftarrow \mathrm{Combine}\{{\mathcal{L}}_{feat}^i,{\mathcal{L}}_{pre}^i,{\mathcal{L}}_{per}^i\}$            3. ${\mathit{x}}_{adv}^i\leftarrow Update\{{\mathit{x}}_{adv}^{i-1},{\mathcal{L}}^i\}$   end for   return ${\mathit{x}}_{adv}^I$

3.3.2. Targeted Attack

Location of the Perturbation

As with the targetless attack, we use Equation (12) to obtain the objects $T_{bboxes}$ of the attack.

The Size of the Perturbation

The targeted attack is executed by deceiving the classifier in the detector, causing it to classify all of the objects in the image as the specific target. The loss function is designed as follows.

Feature Loss: To realize the targeted attack, we use an image of the target category instead of the original hybrid image and extract a target in the target image as a benchmark. Due to the large number of prediction boxes and their different sizes, we have to redesign the target foreground before performing feature approximation. First, create a blank image that is the same shape as the input image, identify the location and size of each prediction box, and resize the target image so that the two are of the same size; following this, the target image is transferred into the corresponding position in the blank image to create a new target foreground ${\tilde{\mathit{x}}}_{f\_t}$. After that, the feature loss is computed:

$${\mathcal{L}}_{feat\_t}={\mathcal{L}}_{feat}(\mathit{\theta },{\mathit{x}}_{adv\_f}^i,{\tilde{\mathit{x}}}_{f\_t}),$$

(22)

Classification Loss: The design principle of the targeted attack is to move the classification result of the classifier continuously closer to the target label, assuming that the target label is $l_t$. The classification loss is calculated as follows:

$${\mathcal{L}}_{cls\_t}={\mathcal{L}}_{cls}(\mathit{x},l_t),$$

(23)

Object box loss: In contrast to targetless attacks, the object box loss in targeted attacks is designed to allow the detection box to coincide with the real box gradually; thus, the loss is designed to be

$${\mathcal{L}}_{box\_t}=-\sum _{j=1}^m{{\mathcal{L}}^j}_{box}\left(D_j\right),$$

(24)

As with the targetless attack, the total of the object box loss and the classification loss is the prediction loss.

Perceptual Loss: As with the targetless attack, we construct the perceptual loss by minimizing the L2 distance between the clean image and the AE.

The ultimate loss function is a weighted amalgamation of the three loss functions. The adversarial perturbation is determined by calculating the gradient of the loss function in relation to the input image. Adding the perturbation to the input image creates the AE.

4. Experiments

4.1. Experimental Preparation

4.1.1. Datasets

We examine our method on two frequently used RSOD datasets, DOTA [50] and UCAS-AOD [51].

DOTA [50]: Since the original images of this dataset are too large, we cropped the original dataset to facilitate the processing. The whole dataset has more than 5000 PNG images with 1024 × 1024 pixels after cropping. After our filtering, we obtain 1500 images covering 15 categories. This research utilizes the DOTA dataset to validate the efficacy of FFA in targetless attacks.

UCAS-AOD [51]: This dataset contains a total of 1510 images and 14596 image instances of only two categories, aircraft and vehicles. The counterexample images in the dataset are of less value for the study of adversarial attacks; thus, we do not consider them. As with the DOTA dataset, we cut the images and processed them to contain a total of 3000 plane images and 1500 vehicle images, each with pixels of 680 × 680. We filter the dataset finely and remove similar images to obtain 1000 aeroplane images and 1000 car images, which constitute our experimental dataset. This dataset is used to validate the performance of FFA with targeted attack.

4.1.2. Detectors

In total, we selected seven rotated ODs as both training and attacked detectors; four for two-stage detectors, namely, Oriented R-CNN (OR) [52], Gliding Vertex (GV) [53], RoI Transformer (RT) [54], and ReDet (RD) [55]; and three in total for single-stage detectors, namely, Rotated Retinanet (RR) [56], Rotated FCOS (RF) [57], and S2A-Net (S2A) [58]. For the feature extraction network, we use ResNet50 [8], hereafter referred to as R50. All the above models are implemented using the open-source MMrotate [59] toolbox.

4.1.3. Evaluation Metrics

For targetless attacks, we use the mean accuracy (mAP) to evaluate the attack effect of the FFA attack. As long as the detected object class is not a real class, the attack is judged to be successful. The smaller the mAP, the better the attack effect is.

The mAP is the average of AP from multiple categories, and the AP value of each category refers to the area under the precision–recall curve of that category, which is a function of precision and recall. Values range between 0 and 1, and a larger value indicates a better detection effect. The calculation formula of AP is

$$\mathrm{AP}=\sum _{i=1}^{n-1}(R_{i+1}-R_i)·P_i,$$

(25)

where $R_{i+1}$ and $R_i$ are the values of two adjacent recall points and $P_i$ is the precision corresponding to $R_i$.

For targetless attacks, we use mAP and the number of detection boxes $N_t$ of the target class detected by the detector to validate the effect of the attack. The smaller the mAP, the bigger the $N_t$, indicating that the effect of the attack is better. The IoU threshold is set to 0.5. For image perceptibility, we use Information content weighted Structural SIMilarity (IW-SSIM) [60] to evaluate the AEs’ perceptibility. The smaller the value, the lower the perceptibility of the image perturbation and the better the effect.

The calculation formula for the IW-SSIM is

$$\mathrm{IW}\text{-}\mathrm{SSIM}=\prod _{j=1}^M{({\displaystyle \frac{1}{N_j}}\sum _{i}l(x_{j,i},y_{j,i})c(x_{j,i},y_{j,i})s(x_{j,i},y_{j,i}))}^{{\beta }_j},$$

(26)

where M is the number of scales and $N_j$ is the number of evaluation windows under scale j. $l,c,s$ are the calculated brightness, contrast, and structural similarity. $x_{j,i}$ and $y_{j,i}$ are the i-th window sections under two images on scale j and ${\beta }_j$ is the scale weight.

Brightness, contrast, and structural similarity are calculated as follows:

$$l(x,y)={\displaystyle \frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1}},c(x,y)={\displaystyle \frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2}},s(x,y)={\displaystyle \frac{{\sigma }_{xy}+C_3}{{\sigma }_x{\sigma }_y+C_3}},$$

(27)

where ${\mu }_x,{\sigma }_x,{\sigma }_{xy}$ represent the mean, standard pool, and cross correlation evaluation, respectively, and $C_1={\left(K_{1}L\right)}^2,C_2={\left(K_{2}L\right)}^2,C_3=C_2/2$ are small constants.

For observational comparison, we multiply both mAP50 and IW-SSIM data by 100. In addition, we assess the attack speed of various algorithms by calculating the average attack time of an image computed using an NVIDIA GeForce RTX 3060 (12 GB) graphics card.

4.1.4. Parameter Setting

The iterations number I is set to 50, the learning rate is assigned a value of 0.1, and the threshold for high-quality prediction box filtering $\alpha$ is set to 0.75. For the targetless attack, the weight of feature loss is set to 1, and the weights $\beta$ and $\gamma$ of prediction loss and perceptual loss are set to 0.1. For the targetless attack, the weight of prediction loss $\beta$ is set to 1 and that of perceptual loss $\gamma$ is set to 0.1.

This study was implemented on the Pytorch platform (https://pytorch.org/).

4.2. Targetless Attack

The test of targetless attack is performed on the DOTA [50] dataset using seven rotating ODs, whose backbone networks are all ResNet50 [8]. We test the attack capability of the attacks using the mAP with an IoU threshold of 0.5 (hereafter referred to as mAP₅₀), with smaller values indicating stronger attack capability. We test the imperceptibility of the perturbation using IW-SSIM [60], with smaller values indicating that the perturbation is imperceptible. We use the average attack time of each image to test the attack speed of all attacks; the smaller the value, the faster the attack speed. Table 1 illustrates the experimental outcomes. In order to facilitate the observation, we multiply the value of mAP₅₀ with the value of IW-SSIM by 100 to obtain the data in the table.

Table 1. Targetless attack effectiveness of different adversarial attack methods on DOTA dataset.

Trained ODs/Backbone	Attack Method	mAP50 ↓ Attacked ODs							IW-SSIM ↓	Time ↓ (s/Image)
		OR	GV	RT	RD	S2A	RR	RF
	Clean	84.1	80.9	87.4	83.5	81.0	78.9	77.6	-	-
OR [52] R50	TOG [61]	11.8	26.8	30.7	40.9	26.7	28.5	26.3	0.49	4.05
	CWA [62]	9.2	25.7	28.5	38.5	25.6	26.3	25.5	1.31	4.23
	LGP [22]	4.1	19.3	21.6	35.3	22.0	20.4	20.7	0.22	6.12
	FFA (ours)	3.3	14.2	19.0	30.2	20.5	19.1	19.8	0.85	6.68
GV [53] R50	TOG [61]	40.5	29.4	28.1	60.7	34.7	36.7	37.1	0.74	6.73
	CWA [62]	38.1	27.6	35.9	59.4	35.4	34.2	35.3	0.86	5.81
	LGP [22]	32.7	23.1	33.7	56.3	32.0	30.6	32.2	0.38	7.85
	FFA (ours)	27.2	21.6	30.5	52.8	29.5	25.7	30.6	0.61	9.03
RT [54] R50	TOG [61]	40.8	36.3	30.4	62.7	37.1	35.4	37.2	0.66	7.71
	CWA [62]	39.3	35.1	28.6	63.1	35.8	33.6	36.4	1.07	8.34
	LGP [22]	35.4	29.8	20.8	60.2	32.3	30.1	32.4	0.52	10.37
	FFA (ours)	31.9	26.2	18.0	55.0	28.6	27.5	29.6	0.62	8.47
RD [55] R50	TOG [61]	58.3	61.2	68.4	27.9	62.8	64.7	60.3	0.51	6.74
	CWA [62]	55.7	60.1	65.5	25.1	60.6	65.4	59.5	0.87	8.80
	LGP [22]	53.6	55.8	60.3	22.0	58.2	61.1	57.9	0.18	9.65
	FFA (ours)	50.2	50.6	53.0	19.6	53.4	59.7	56.2	0.36	10.35
S2A [58] R50	TOG [61]	47.5	49.8	54.1	62.7	20.6	53.1	57.4	0.98	14.26
	CWA [62]	45.1	45.6	52.4	60.3	11.9	50.4	55.3	1.25	19.54
	LGP [22]	42.8	43.3	49.6	57.0	5.2	47.4	51.7	0.64	25.32
	FFA (ours)	39.3	38.9	47.8	55.4	4.5	43.6	48.2	0.74	26.47
RR [56] R50	TOG [61]	55.7	56.4	52.1	52.8	60.3	17.3	50.1	0.83	13.67
	CWA [62]	56.9	55.3	50.7	50.9	58.4	14.6	48.6	1.07	15.82
	LGP [22]	52.6	50.8	47.2	48.0	54.6	10.2	46.4	0.67	27.62
	FFA (ours)	49.3	47.1	45.8	44.9	51.0	8.5	41.5	0.75	28.46
RF [57] R50	TOG [61]	46.5	47.9	47.4	50.5	50.4	55.7	18.3	0.71	15.65
	CWA [62]	45.8	45.3	45.6	47.3	47.6	53.4	15.7	0.93	19.54
	LGP [22]	40.6	43.7	42.1	44.7	42.1	49.8	10.3	0.55	28.31
	FFA (ours)	35.1	40.5	38.6	41.7	38.2	46.3	9.2	0.69	30.66

↓ means smaller data is better. The most optimal outcomes in the table are emphasized in bold. Green font indicates white-box attacks. For ease of observation, the values of mAP₅₀ and IW-SSIM in the table are the results after expanding the original values by a factor of 100.

From the data in the table, we can see that the attack capability of FFA is significantly stronger than the current state-of-the-art methods, especially since the attack transferability achieves a large improvement. The reason is that we designed a generic attack for the current commonly used backbone network by changing the characteristics of the image in the view of the backbone network, attacking the model’s attentional mechanism, thus generating the adversarial perturbation with a more powerful attack. The imperceptibility of the perturbation generated by FFA is slightly weaker than the LGP attack due to the fact that LGP designs an adaptive perturbation region updating strategy to select more cost-effective regions to attack. On the other hand, FFA selects the attacked regions based on the prediction of the ODs and the real labels of the clean images, which implies that the ODs’ detection accuracy and the attacking effect of FFA are somewhat correlated, and it will inevitably select some cost-ineffective regions for attacking during the attacking process, which attenuates the imperceptibility of the perturbation. In terms of attack speed, FFA is slightly slower than LGP because FFA uses the feature approximation strategy, and the feature distance has to be calculated during the attack process, which increases the amount of computation. FFA sacrifices part of the imperceptibility and attack speed for the improvement of attack effectiveness.

The visualization results of the targetless attack are displayed in Figure 4. Based on the figure, it is evident that the FFA produces better attack results not only for large and sparse objects but also for small and dense objects.

Figure 4. Visualization of FFA targetless attack detection results. There are four layers in total. The top two layers represent the original images and AEs detection results for small objects, and the bottom two layers represent the original images and AEs detection results for large objects. The orange dashed section represents the detection outcomes of the two-stage ODs, while the blue dashed section represents the detection outcomes of the single-stage ODs.

4.3. Targeted Attack

4.3.1. White Box Attack Performance

Targeted attacks were tested on the UCAS-AOD [51] dataset using seven rotating ODs whose backbone networks are all ResNet50. We tested the method’s attack capability using mAP₅₀ and the number of targets $N_t$ with the IoU threshold of 0.75 (hereafter referred to as $N_{t0.75}$), where a smaller value of mAP₅₀ and a larger value of mAP₅₀ indicate a stronger attack capability. The test results are shown in Table 2. From the table, it can be seen that FFA’s attack capability is significantly stronger than the existing advanced attack methods. This shows that FFA can realize a high success rate by changing the object features in the image.

Table 2. White-box targeted attack effectiveness of different adversarial attack methods on UCAS-AOD dataset.

Origin	Target	Attack Method	mAP50 ↓							${\mathit{N}}_{\mathit{t}0.75}$ ↑
			OR	GV	RT	RD	S2A	RF	RR	OR	GV	RT	RD	S2A	RF	RR
Plane	Clean		90.1	90.5	89.7	91	89.3	90.2	89.1	3505	3241	3359	3418	3525	3684	3457
	Ground track field	TOG [61]	25.7	22.3	21.2	25.5	17.1	15.7	10.3	2215	2339	2237	2189	2681	2706	2892
		CWA [62]	23.4	19.6	20.1	25.2	15.3	14.2	9.8	2367	2551	2342	2135	2764	2833	3085
		LGP [22]	20.2	16.7	16.1	23.1	11.9	10.2	3.7	2553	2780	2718	2554	2937	2823	3142
		FFA (ours)	18.7	13.1	12.5	19.0	7.8	6.3	1.2	2876	3108	3005	2735	3121	3027	3331
	Basket- ball court	TOG [61]	24.7	19.4	22.3	25.7	16.7	17.3	8.4	2147	2371	2287	2108	2576	2478	3059
		CWA [62]	21.3	17.3	20.5	24.4	15.2	15.2	8.9	2213	2564	2349	2235	2635	2593	2974
		LGP [22]	19.2	14.6	17.8	21.2	10.4	10.7	4.1	2632	2744	2605	2573	2854	2849	3213
		FFA(ours)	17.5	12.1	15.6	17.3	6.9	7.3	2.2	2719	2893	2819	2746	3122	3085	3397
	Round- about	TOG [61]	20.6	20.1	22.3	25.3	12.4	13.5	8.5	2368	2271	2263	2182	2403	2513	2889
		CWA [62]	19.4	18.7	20.5	22.7	10.9	11.6	5.2	2507	2584	2416	2237	2648	2678	3011
		LGP [22]	17.3	16.1	17.1	20.6	9.6	9.2	2.6	2640	2747	2661	2519	2931	2832	3234
		FFA (ours)	15.8	12.7	17.8	18.1	7.7	6.1	0.9	2841	2908	2773	2795	3086	3225	3411
Vehicle	Clean		81.5	82.3	79.8	85.2	83.1	81.6	82.7	3565	3483	3217	3238	3804	3561	3615
	Ground track field	TOG [61]	27.1	30.3	24.6	25.8	14.2	9.6	15.2	2306	2241	2048	2246	2763	3047	2971
		CWA [62]	25.3	26.7	22.3	22.6	12.4	8.3	12.5	2253	2437	2369	2517	2845	3112	3102
		LGP [22]	22.7	21.6	19.1	20.3	9.2	4.9	6.7	2537	2614	2715	2658	3183	3474	3368
		FFA (ours)	20.2	18.3	16.5	19.7	8.6	3.7	5.1	2618	2736	2823	2709	3341	3507	3472
	Basket- ball court	TOG [61]	25.5	27.6	27.0	28.9	12.4	9.8	10.7	2201	2356	2207	2087	2655	2736	2867
		CWA [62]	22.4	25.4	25.8	26.3	10.7	10.5	9.8	2351	2409	2365	2139	2813	2912	2983
		LGP [22]	19.8	20.5	19.3	22.5	8.4	7.6	5.8	2689	2654	2677	2483	3017	3202	3391
		FFA(ours)	19.6	18.7	17.9	20.7	8.0	7.6	4.4	2605	2736	2782	2625	3224	3285	3457
	Round- about	TOG [61]	28.9	29.1	22.2	26.7	16.4	12.6	11.3	2168	2145	2516	2253	2806	3013	2975
		CWA [62]	25.7	27.7	20.3	25.4	15.7	10.4	9.2	2253	2208	2453	2310	2849	3121	3010
		LGP [22]	22.4	22.3	17.8	21.6	9.6	5.8	7.1	2537	2580	2769	2544	3142	3348	3249
		FFA(ours)	21.3	19.6	16.1	21.3	10.1	5.1	6.5	2591	2674	2848	2569	3077	3403	3386

↓ means smaller data is better. ↑ means bigger data is better. The most optimal outcomes in the table are emphasized in bold. The green font refers to the detection result of the clean image.

4.3.2. Transferability Experiments

To verify the targeted attack transferability of FFA, we designed the following experiment. A two-stage OD OR [52] and a single-stage OR S2A [58] are selected as trained detectors to attack the remaining five detectors, with the original categories of planes and vehicles and the corresponding target categories of basketball courts and ground track fields, respectively. The evaluation metrics are mAP₅₀ and $N_{t0.75}$. The empirical findings are displayed in Table 3.

Table 3. Mobility testing of targeted adversarial attack methods.

Origin	Target	Attack Method	Trained OD	mAP50 ↓					${\mathit{N}}_{\mathit{t}0.75}$ ↑
				Attacked ODs
				GV	RT	RD	RF	RR	GV	RT	RD	RF	RR
Plane	Basketball court	TOG [61]	OR [52]	58.6	54.5	53.5	49.1	48.5	953	1024	986	1055	1284
		CWA [62]		55.1	52.7	47.9	45.4	42.3	899	980	1443	1274	1596
		LGP [22]		48.7	46.8	44.6	39.6	35.9	1258	1386	1549	1595	1876
		FFA (ours)		45.8	43.2	42.6	37.1	34.3	1431	1503	1661	1752	1919
		TOG [61]	S2A [58]	73.8	70.6	61.5	62.2	50.5	536	683	974	961	1017
		CWA [62]		69.1	63.2	58.7	60.1	49.8	848	871	1037	1083	1385
		LGP [22]		58.5	62.7	58.3	54.4	46.1	1032	896	1096	1264	1568
		FFA (ours)		59.1	60.1	57.6	53.8	44.7	1075	935	1209	1348	1792
Vehicle	Ground track field	TOG [61]	OR [52]	70.2	69.8	69.5	65.5	68.1	753	726	774	873	754
		CWA [62]		68.5	70.4	68.7	65.2	66.3	796	698	886	912	817
		LGP [22]		65.7	67.5	65.8	63.1	64.0	905	903	1027	1108	1283
		FFA (ours)		65.1	66.3	64.4	60.9	61.7	937	971	1136	1395	1407
		TOG [61]	S2A [58]	73.2	70.7	72.3	67.3	65.1	685	751	634	890	1018
		CWA [62]		70.3	68.8	70.9	65.5	63.4	758	891	776	1068	1039
		LGP [22]		67.7	63.4	68.2	60.3	59.6	979	1008	872	1321	1237
		FFA (ours)		67.1	62.8	65.4	59.1	57.8	1021	1085	958	1377	1414

↓ means smaller data is better. ↑ means bigger data is better. The most optimal outcomes in the table are emphasized in bold. The two leftmost columns are the original and target categories.

The data presented in the table demonstrate that FFA has better transferability compared to other methods. We also find that FFA has a stronger attack effect on simple models than on complex models. mAP₅₀ of FFA for single-stage detectors is, on average, 2 percentage points lower than that of dual-stage detectors, while $N_{t0.75}$ is, on average, more than 200. In addition, the accuracy of the trained OD also affects the attack effect. Since the accuracy of the two-stage OD is slightly higher than that of the single-stage OD, the attack effect of the attack using the two-stage OD is stronger than that using the single-stage OD. At the same time, the success rate of targeted attacks on planes is significantly higher than that on vehicles because the ODs’ detection accuracy for planes is higher than that for vehicles.

4.3.3. Imperceptibility and Attack Speed Test

Similar to the targetless attack, we also tested the perturbation’s imperceptibility and the algorithm’s attack speed. The empirical findings are displayed in Table 4.

Table 4. Imperceptibility and attack speed tests for targeted attacks.

Attack Method	IW-SSIM ↓
	OR	GV	RT	RD	S2A	RF	RR
TOG [61]	1.96	2.09	1.77	2.41	2.56	1.94	2.09
CWA [62]	2.53	2.74	1.98	2.37	2.91	2.04	3.62
LGP [22]	1.56	1.47	1.53	1.94	2.13	1.06	2.76
FFA (ours)	1.78	1.95	1.72	2.01	2.35	1.25	3.12
Attack Method	Time (s/Image) ↓
	OR	GV	RT	RD	S2A	RF	RR
TOG [61]	4.49	5.58	6.53	7.43	6.14	7.07	5.86
CWA [62]	5.36	4.07	7.74	8.65	13.55	11.68	11.23
LGP [22]	6.83	6.15	10.62	10.31	17.54	15.47	16.21
FFA (ours)	6.95	7.81	11.83	10.47	18.75	16.29	18.58

↓ means smaller data is better. The most optimal outcomes in the table are emphasized in bold.

The data presented in the table demonstrate that the perturbation imperceptibility type of FFA is slightly weaker and the attack speed is slower than LGP. The reason is that we will sacrifice some of the perturbation imperceptibility with the attack speed to obtain a high-transferability attack. In addition, we also find that the success rate of an attack against single-stage ODs is higher than that against two-stage ODs due to the simpler model structure and longer attack time of single-stage ODs, leading to better attack performance.

The visualization results of targeted attacks are displayed in Figure 5 and Figure 6. We can see that FFA has better attack results on different targets in different detectors.

Figure 5. Visualization of object detection results for targeted attacks against planes. The part indicated by the red arrow is the target category, the orange color is the detection result of the two-stage OD, and the blue color is the detection result of the single-stage OD.

Figure 6. Visualization of object detection results for targeted attacks against vehicles. The part indicated by the red arrow is the target category, the orange color is the detection result of the two-stage OD, and the blue color is the detection result of the single-stage OD.

4.4. Effect of Iteration Number

The number of iterations is a direct parameter that impacts the efficiency of the attack; so, we assessed the influence of the iteration number using multiple evaluation metrics. The results are displayed in Table 5 and Figure 7.

Table 5. Effect of iteration number on attack results.

Attack Method	OD/Backbone	I	1	10	20	50	100
Targetless   FFA	OR/R50	mAP50 ↓	34.2	6.7	4.1	3.3	6.8
		IW-SSIM ↓	0.21	0.20	0.32	0.85	0.12
		Time ↓	0.85	2.41	4.36	6.68	13.75
Targeted   FFA	OR/R50	mAP50 ↓	55.6	39.6	25.4	18.7	22.1
		$N_{t0.75}$ ↑	1283	1583	2283	2876	2679
		IW-SSIM ↓	0.51	0.67	1.05	1.78	1.03
		Time ↓	1.03	2.29	4.71	6.95	15.33

↓ means smaller data is better. ↑ means bigger data is better. The most optimal outcomes in the table are emphasized in bold.

Figure 7. The impact of the iteration number on the attack. The trained and attacked ODs for both targetless attack and targeted attack are OR, and the backbone network is ReSNet50.

From the results, we can see that only the attack time is positively correlated with the iteration number, and the more iteration numbers, the longer the attack time. The relationship between the attack effect and imperceptibility and the iteration number is more complicated: the attack effect is first enhanced and then weakened with the increase in iteration number, while the imperceptibility of the perturbation is exactly the opposite, first weakened and then enhanced. This shows that there is a contradiction between the attack effect and imperceptibility, and the stronger the attack effect is, the weaker the imperceptibility of the perturbation is. Since the results in the table show that the imperceptibility of the perturbation is still at a low level, taking this fact into account, we set the iteration number of FFA to 50.

4.5. Ablation Experiments

In this section, we verify the effect of different losses used in the method on the attack efficacy. The quantitative results are displayed in Table 6, and the visualization results are displayed in Figure 8.

Table 6. The impact of different components of the loss function on the attack effectiveness.

Targetless   FFA	OR	${\mathcal{L}}_{feat}$	${\mathcal{L}}_{pre}$	${\mathcal{L}}_{per}$	IW-SSIM ↓	mAP50 ↓	Time ↓
	Clean	-	-	-	-	83.3	-
	1	√			3.81	11.2	5.38
	2	√	√		2.51	8.9	7.45
	3	√	√	√	0.85	3.3	9.51
Targeted   FFA	OR/Plane	${\mathcal{L}}_{feat}$	${\mathcal{L}}_{pre}$	${\mathcal{L}}_{per}$	IW-SSIM ↓	mAP50 ↓	Time ↓
	Clean	-	-	-	-	90.1	-
	1	√			5.85	25.7	2.32
	2	√	√		3.64	23.9	2.96
	3	√	√	√	1.50	18.7	3.55

↓ means smaller data is better. The most optimal outcomes in the table are emphasized in bold. We used OR as the trained and attacked detector, and for targeted attacks, we selected plane as the original category and ground track field as the target category.

Figure 8. Visualization results of perturbations in ablation experiments. The upper layer is the targetless attack, and the lower layer is the targeted attack. (a,e) are the original images; (b,f) are the perturbations with feature loss only; (c,g) are the perturbations with prediction loss and feature loss; and (d,h) are the perturbations with feature loss, prediction loss, and perception loss.

5. Conclusions

This work explores a strategy for generating AEs for RSOD. Most existing methods directly attack the predictive information of the image to achieve the effect of fooling the ODs. Unlike these methods, we propose an attack method that utilizes the information of the image itself by changing the feature of the image in view of the backbone. Specifically, we first use the OD to filter out high-quality predictions as the object to implement the attack, create a hybrid foreground without any target, and use KL divergence to minimize the shallow features of the attack object and the shallow features of the hybrid foreground to achieve a targetless attack. We replace the hybrid foreground with the target foreground and repeat the above process to realize the targeted attack. Although this method is relatively simple, the results of attacking seven rotating ODs on two datasets show that this method can generate AEs with high transferability.

There are still some directions that can be improved in this study: (1) When selecting the object to implement the attack, we use the prediction scores and IoU for threshold screening, which can result in poor-quality attack objects, leading to the weakening of the effect of the attack; this can be improved by adding a better screening strategy. (2) The imperceptibility of the perturbation is weak. More effective perturbation control methods can be set to increase the imperceptibility of the perturbation. (3) The attack speed of the algorithm is slow, and there are still some redundant calculations in the algorithm, which increases the amount of calculation; a more efficient iteration method can be designed to reduce the amount of calculation.