An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks

Abstract

Deep neural networks (DNNs) are sensitive to adversarial data in a variety of scenarios, including the black-box scenario, where the attacker is only allowed to query the trained model and receive an output. Existing black-box methods for creating adversarial instances are costly, often using gradient estimation or training a replacement network. This paper introduces Query-Efficient Evolutionary Attack—QuEry Attack—an untargeted, score-based, black-box attack. QuEry Attack is based on a novel objective function that can be used in gradient-free optimization problems. The attack only requires access to the output logits of the classifier and is thus not affected by gradient masking. No additional information is needed, rendering our method more suitable to real-life situations. We test its performance with three different, commonly used, pretrained image-classifications models—Inception-v3, ResNet-50, and VGG-16-BN—against three benchmark datasets: MNIST, CIFAR10 and ImageNet. Furthermore, we evaluate QuEry Attack’s performance on non-differential transformation defenses and robust models. Our results demonstrate the superior performance of QuEry Attack, both in terms of accuracy score and query efficiency.

1. Introduction

Deep neural networks (DNNs) have become the central approach in modern-day artificial intelligence (AI) research. They have attained superb performance in multifarious complex tasks and are behind fundamental breakthroughs in a variety of machine-learning tasks that were previously thought to be too difficult. Image classification, object detection, machine translation, and sentiment analysis are just a few examples of domains revolutionized by DNNs.

Despite their success, recent studies have shown that DNNs are vulnerable to adversarial attacks. A barely detectable change in an image, for example, can cause a misclassification in a well-trained DNN. Targeted adversarial examples can even evoke a misclassification of a specific class (e.g., misclassify a car as a cat). Researchers have demonstrated that adversarial attacks are successful in the real world and may be produced for data modalities beyond imaging, e.g., natural language and voice recognition [1,2,3,4]. DNNs’ vulnerability to adversarial attacks has raised concerns about applying these techniques to safety-critical applications.

To discover effective adversarial instances, most past work on adversarial attacks has employed gradient-based optimization [5,6,7,8,9]. Gradient computation can only be executed if the attacker is fully aware of the model architecture and weights. Thus, these approaches are only useful in a white-box scenario, where an attacker has complete access and control over a targeted DNN. Attacking real-world AI systems, however, might be far more arduous. The attacker must consider the difficulty of implementing adversarial instances in a black-box setting, in which no information about the network design, parameters, or training data is provided. In this situation, the attacker is exposed only to the classifier’s input-output pairs. In this context, a typical strategy has been to attack trained replacement networks and hope that the generated examples transfer to the target model [10]. The substantial mismatch of the model between the alternative model and the target model, as well as the significant computational cost of alternative network training, often renders this technique ineffective.

In our work we assume a real-world, black-box attack scenario, wherein a DNN’s input and output may be accessed but not its internal configuration. We focus on a scenario in which a specific DNN is an image classifier, specifically, a convolutional neural network (CNN), which accepts an image as input and outputs a probability score for each class.

Herein, we present an evolutionary, gradient-free optimization approach for generating adversarial instances, more suitable for real-life scenarios, because usually there is no access to a model’s internals, including the gradients; thus, it is important to craft attacks that do not use gradients. Our proposed attack can deal with either constrained ($ϵ$ value that constrains the norm of the allowed perturbation) or unconstrained (no constraint on the norm of the perturbation) problems, and focuses on constrained, untargeted attacks. We believe that our framework can be easily adapted to the targeted setting.

In the next section we review the literature on adversarial attacks. Section 3 summarizes the threat model we assume for our proposed evolutionary attack algorithm. The algorithm itself—QuEry Attack (for Query-Efficient Evolutionary Attack)—is delineated in Section 4. The experiments conducted to test the method, along with results, are described in Section 5. We discuss our findings and present concluding remarks in Section 6.

Figure 1 shows examples of successful and unsuccessful instances of images generated by QuEry Attack, evaluated against ImageNet, CIFAR10, and MNIST.

2. Related Work

Adversarial attacks against DNNs have become an important research field in the last few years. For a comprehensive survey, we refer the reader to [11].

An important distinction is between ‘white box’ and ‘black box’ attacks. In white-box attacks, the attacker has knowledge of the attacked model’s internal structure and parameters, and exploits that knowledge. It is commonly performed by using the model’s gradients [5,7,12].

Although white-box methods achieved good results they usually do not represent a real-world scenario. More realistic is a black-box attack, where the attacker has no knowledge of the model’s structure and parameters. The attacker can only query the model—and act upon its outputs. We can further distinguish between the more common ‘light black box’ attacks, where the model gives the prediction’s confidence (which can be exploited), and ‘dark black box’ attacks where the model only gives the final class prediction (e.g., [10]).

The first effective black-box attack traded a priori information of the model with extensive runtime querying [10]. Using a large number of queries it builds a substitute model, and attacks it with traditional white-box methods. It uses the transferability property, namely, an attack that succeeds on one model will likely succeed on a similar—though not identical—model.

Other works used the more permissive ‘light black-box’ scenario, which can use the prediction’s confidence value. Some works estimate the gradient with this information and then use traditional gradient-based attacks [13].

All these black-box methods rely on gradients, and, thus, are sensitive to many defense methods that obscure gradients [14,15,16]. This has given rise to methods that do not rely on gradients at all, e.g., [17], which uses random search and is also query-efficient.

Instead of randomness, one can use evolutionary methods. In Evolutionary Algorithms (EAs), core concepts from evolutionary biology—inheritance, random variation, and selection—are harnessed in algorithms that are applied to complex computational problems. EA techniques have been shown to solve numerous difficult problems from widely diverse domains, and also to produce human-competitive machine intelligence [18]. EAs also present many important benefits over popular Machine Learning methods, including [19]: less reliance on the existence of a known or discoverable gradient within the search space; ability to handle design problems, where the objective is to design new entities from scratch; fewer required a priori assumptions about the problem at hand; seamless integration of human expert knowledge; ability to solve problems where human expertise is very limited; support of interpretable solution representations; and support of multiple objectives.

The evolutionary method GenAttack is a targeted attack (thus not directly comparable to ours, which is untargeted) that used a fitness function that tries to increase the probability of the target class and decrease the probability of the other classes [20]. Its fitness function ignored the distance between the images. Interestingly, GenAttack uses fitness-proportionate selection, which is employed less often nowadays due to known problems. It uses an adaptive mutation rate to balance between exploration in early phases and exploitation in later phases.

Ref. [21] treated the adversarial problem as one of multi-objective optimization: minimize the class prediction’s score on one hand, and the distance between the original image and a perturbed one on the other hand.

Another attack method changes a single pixel [22]. This method uses differential evolution (DE) [23] without crossover. However, it sometimes required thousands of queries.

Ref. [24] also uses DE, but unlike the other evolutionary computation (EC) methods reviewed, it uses EC to approximate the gradients.

Unlike the above methods, which tried to minimize the perturbation as much as possible and make it as unnoticeable to the human eye as possible, [25] makes a small but noticeable change, which looks like a regular scratch (a similar approach in the domain of Natural Language Processing creates sentences that do not make sense to a human reader [26]). The approach uses DE as well, and also Covariance-Matrix Adaptation Evolution Strategies (CMA-ES) [27]. Unlike most attacks, which use the $l_{\infty }$ or $l_2$ norms, this one is based on the $l_0$ norm.

In the EC methods seen so far, evolution is run against a single image, and each individual is a perturbation added to that image. Ref. [28], on the other hand, used the transferability property mentioned earlier to evolve a universal perturbation. An individual is an image mask that can be applied as a perturbation to any image.

Our extensive scrutiny of the literature and software repositories revealed that many authors compare their work to prior works that do not use the same threat model: there might be a mismatch in norms (e.g., $l_2$ vs. $l_{\infty }$), white box vs. black box, or other subtle differences. Moreover, having investigated numerous software repositories, we found that running the code of many papers is far from straightforward.

3. Threat Model

In the black-box attack setting, queries to the network are permitted but access to internal states is prohibited (e.g., executing backpropagation). Hence, our threat model, or scenario, is as follows:

• The attacker is unaware of the network’s design, settings, or training data.
• The attacker has no access to intermediate values in the target model.
• The attacker can only use a black-box function to query the target model.

Note that the above threat model determines the comparisons we perform, which focus on attacks that are:

1.

Black-box,

2.

Untargeted,

3.

$l_{\infty }$-norm bounded.

We can consider a network model to be a function:

$$f:{[0,1]}^d\to {\mathbb{R}}^C,$$

(1)

where d is the number of input features and C is the number of classes. The c-th value $f_c\left(x\right)\in \mathbb{R}$ specifies the predicted score of classifying input image x as class c. The classifier assigns class $y={argmax}_{c=1,\cdots ,C}{f}_c\left(x\right)$ to the input x.

A targeted attack aims to create an image that will be incorrectly classified into a given (incorrect) class. An untargeted attack aims to create an image that will be incorrectly classified into any class except the correct one. An image $\widehat{x}$ is termed an adversarial example, with an $l_p$-norm bound of $ϵ$ for x, if:

$$\begin{array}{c}\underset{c=1,\cdots ,C}{argmax}{f}_c\left(\widehat{x}\right)\ne y,\\ s.t.\parallel \widehat{x}{-x\parallel }_p\le ϵ\mathrm{and}\widehat{x}\in {[0,1]}^d.\end{array}$$

(2)

To wit, the model should classify $\widehat{x}$ incorrectly, while preserving similarity between x and $\widehat{x}$ under an $l_p$ distance metric.

We focus on a black-box, score-based attack, wherein the only information of the threat model is the raw output (logits).

Our suggested black-box approach may theoretically be used in conjunction with classic machine-learning models, with the same input–output relationship. Because DNNs have reached state-of-the-art performance in a variety of image tasks, we focus on them in this paper.

4. QuEry Attack

QuEry Attack is an evolutionary algorithm (EA) that explores a space of images, defined by a given input image and a given input model, in search of adversarial instances. It ultimately generates an attacking image for the given input image. Unlike white-box approaches, we make no assumptions about the targeted model, its architecture, dataset, or training procedure. We assume that we have an image x, which a black-box neural network, f, classifies by outputting a probability distribution over the set of classes, as stated in Section 3. The actual label y is computed as $y=argmaxf\left(x\right)$.

Our objective is to find a perturbed image, $\widehat{x}$, of image x, such that, $\parallel \widehat{x}{-x\parallel }_{\infty }\le ϵ$, which causes the network to predict $y^{\prime }=argmaxf\left(\widehat{x}\right)$, such that $y^{\prime }\ne y$. Finding $\widehat{x}$ may be cast as a constrained optimization problem:

$$\underset{\widehat{x}\in {[0,1]}^d}{min}\mathcal{L}(f\left(\widehat{x}\right),y),s.t.{\parallel \widehat{x}-x\parallel }_{\infty }\le ϵ,$$

(3)

for a given loss function $\mathcal{L}$.

We use loss $\mathcal{L}$ as the fitness function, defined in our case as:

$$\mathit{fitness}\left(\widehat{x}\right)=f_y\left(\widehat{x}\right)-\underset{c\ne y}{max}{f}_c\left(\widehat{x}\right)+\lambda {\parallel \widehat{x}-x\parallel }_2,$$

(4)

where $\widehat{x}$ is a perturbed image, $f_y$ is the predicted score that $\widehat{x}$ belongs to class y, $f_c$ is the predicted score that $\widehat{x}$ belongs to class $c\ne y$. In order to guarantee that the adversarial perturbation is as imperceptible as possible we penalize the $l_2$ distortion of the perturbation by including a regularization component in the fitness function. We use $l_2$ regularization because we noticed that most of the evolved adversarial examples were on the edges of the $ϵ$-ball, and we wanted to give precedence to examples which were closer to the original input. This penalization is determined by the $\lambda$ value, which is the regularization strength. In our experiments we used $\lambda =1$.

The ultimate goal is to minimize the fitness value: Essentially, the lower the logit of the correct class, and the higher the maximum logit of the other classes, the better the value.

Algorithm 1 provides the pseudo-code of QuEry Attack. The original image x, along with a number of hyperparameters, are given as input to the algorithm. QuEry Attack generates an adversarial image $\widehat{x}$, with the model classifying $\widehat{x}$ as $y^{\prime }$, such that $y^{\prime }\ne y$ and $\parallel \widehat{x}{-x\parallel }_{\infty }\le ϵ$.

The main goal of QuEry Attack is to produce a successful attack, using as few queries to the model as possible. The maximal number of queries equals generation count $\left(G\right)$× population size $\left(N\right)$.

4.1. Initialization

Initialization is crucial for optimization problems, e.g., in deep-learning training, gradient descent reaches a local minimum that is significantly determined by the initialization technique [29,30]. QuEry Attack generates an initial population of perturbed images by randomly selecting images from the edges of the sphere centered on the original image x with radius = $ϵ$. This is accomplished by adding vertical stripes of width 1 along the image, with the color of each stripe sampled uniformly at random from $\{-ϵ,ϵ\}$ per channel (i.e., the pixels of each stripe can be either $-ϵ$ or $ϵ$); in [17], they discovered that convolutional neural networks (CNNs) are especially vulnerable to such perturbations.

4.2. Mutation

Considering the use of (square-shaped) convolutional filters by convolutional neural networks, we used square-shaped perturbations. Specifically, we employed [17]’s technique for determining square size. Let $p\in [0,1]$ be the proportion of elements to be perturbed for an image of shape $h\times w$. The nearest positive integer to $\sqrt{p·h·w}$ determines the length of the square’s edge, with p being a hyperparameter. We set it initially to $p=0.1$, then halved it after {40, 200, 800, 4000, 8000, 16,000, 24,000, 32,000} queries, respectively (similar to [17]).

4.3. Crossover

We experimented both with single-point and two-point crossover, eventually settling on the latter as it performed better. The operator works by flattening both (two-dimensional image) parents, randomly picking two indices, then swapping the pixels between the chosen pixels.

The EA then proceeds by evaluating the fitness of each individual, selecting parents, and performing crossover and mutation to generate the next generation. This latter is obtained by adding one elite individual from the current generation, with all other next-generation individuals derived through crossover and mutation. The process is repeated until a successful perturbation is found or until the termination condition is met.

A major advantage of QuEry Attack is its amenability to parallelization—due to being evolutionary—in contrast to most other adversarial, iterative (non-evolutionary) attacks in this field.

Algorithm 1 QuEry Attack

Input:    x ← original image y ← original label $ϵ$ ← maximum $l_{\infty }$ distance p ← proportion of elements in x to be perturbed N ← population size G ← maximum number of generations T ← tournament size Output:    $\widehat{x}$ ← adversarial image #  Main loop 1: $gen\leftarrow 0$ 2: $pop\leftarrow$ INIT() 3: while not TERMINATION_CONDITION($pop,gen$) do 4:     for $\widehat{x}\in pop$ do 5:         compute fitness of $\widehat{x}$ using Equation (4) 6:     $new\_pop\leftarrow \varnothing$ 7:     $elite\leftarrow$ ELITISM($pop$) 8:     add $elite$ to $new\_pop$ 9:     for $i\leftarrow 1$ to $\frac{P-1}{3}$ do 10:         $paren{t}_1\leftarrow$ SELECTION($pop$) 11:         $paren{t}_2\leftarrow$ SELECTION($pop$) 12:         ${\mathit{offspring}}_1,{\mathit{offspring}}_2\leftarrow$ CROSSOVER($pop$) 13:         ${\mathit{mut}}_1\leftarrow$ SQUARE_MUTATION(${\mathit{offspring}}_1$) 14:         ${\mathit{mut}}_2\leftarrow$ SQUARE_MUTATION(${\mathit{offspring}}_2$) 15:         add ${\mathit{offspring}}_1,{\mathit{mut}}_1,{\mathit{mut}}_2$ to $new\_pop$ 16:     $pop\leftarrow new\_pop$ 17:     $gen\leftarrow gen+1$ 18:   19: return best $\widehat{x}$ from $pop$ # QuEry Attack’s final output 20: function INIT( ) 21:     $pop\leftarrow \varnothing$ 22:     for i← 1 to N do 23:         $\widehat{x}\leftarrow$ STRIPES_INIT(x) 24:         add $\widehat{x}$ to $pop$ 25:     return $pop$ 26: function ELITISM(pop) 27:     return best $\widehat{x}$ from $pop$ 28: function TERMINATION_CONDITION(pop, gen) 29:     if $gen=G$ then 30:         return true 31:     for $\widehat{x}\in pop$ do 32:         $\widehat{y}\leftarrow$ predicted label of $\widehat{x}$ 33:         if $\widehat{y}\ne y$ then 34:            return true 35:     return false 36: function SELECTION(pop) 37:     $tournament\leftarrow$ randomly and uniformly pick T individuals from $pop$ 38:     return best $\widehat{x}$ from $tournament$ 39: function STRIPES_INIT($\widehat{x}$) 40:     for i← 1 to c do # c is the image’s number of channels 41:         stripe ← create a vertical stripe of width 1, randomly positioned, with random values $\in \{-ϵ,ϵ\}$ 42:         $\widehat{x}$←$\widehat{x}$ + stripe 43:     $\widehat{x}$$\leftarrow {\Pi }_{ϵ}\left(\widehat{x}\right)$ # ${\Pi }_{ϵ}$: clipping operator to ensure pixel values are within $ϵ$-ball 44:     return $\widehat{x}$ 45: function SQUARE_MUTATION($\widehat{x}$) 46:     $c\leftarrow$ number of channels 47:     $f\leftarrow$ number of features ($h\times w)$ # h: height, w: width 48:     $k\leftarrow \lceil \sqrt{p·f}\rceil$ 49:     $\delta \leftarrow$ array of ones of size $k\times k\times c$. 50:     $row,col\leftarrow \mathcal{U}\left(\right\{0,\cdots ,w-h\left\}\right)$ # $\mathcal{U}$ randomly and uniformly selects from given set 51:     for i← 1 to c do 52:         $\tau$←$\mathcal{U}\left(\right\{-2ϵ,2ϵ\left\}\right)$ 53:         ${\delta }_{row+1:row+h,col+1:col+h,i}\leftarrow \tau ·\delta$ 54:         $\widehat{x}\leftarrow \widehat{x}+\delta$ 55:     $\widehat{x}\leftarrow {\Pi }_{ϵ}\left(\widehat{x}\right)$ 56:     return $\widehat{x}$ 57: function CROSSOVER($paren{t}_1$, $paren{t}_2$) 58:     Flatten $paren{t}_1$ and $paren{t}_2$ 59:     Perform standard two-point crossover (as explained in text), creating ${\mathit{offspring}}_1,{\mathit{offspring}}_2$ 60:     ${\mathit{offspring}}_1,{\mathit{offspring}}_2$ $\leftarrow {\Pi }_{ϵ}\left({\mathit{offspring}}_1\right),{\Pi }_{ϵ}\left({\mathit{offspring}}_2\right)$ 61:     return ${\mathit{offspring}}_1,{\mathit{offspring}}_2$

5. Experiments and Results

To evaluate QuEry Attack we set out to collect commonly used algorithms for comparative purposes. A somewhat disconcerting reality we then encountered involved our struggle to find good benchmarks and software for comparison purposes. Sadly, we found ourselves wasting many a day (which, alas, turned into weeks) trying to run buggy software, chasing down broken links, issuing GitHub issues, and so forth. Perhaps this is due in part to the field of adversarial attacks being young.

Our experimental results are summarized in

Table 1. Experimental results. ASR: Attack Success Rate. Queries: Number of model queries. Each value represents the median of 200 runs (images). Epsilon: $\mathcal{E}\in \{0\cdots 255\}$ (8-bits pixel values). Top results are marked in boldface.

ImageNet
Model	$\mathcal{E}$	QuEry Attack		Square		AdversarialPSO
		ASR	Queries	ASR	Queries	ASR	Queries
Inception-v3	24	100%	1	100%	5	98.5%	51
	18	100%	2	100%	8	98.5%	69
	12	100%	22	95.5%	32	97%	102
	6	99.5%	276	99.0%	263	95%	285
ResNet-50	24	100%	1	99.5%	5	98.5%	51
	18	100%	1	100%	5	98.5%	69
	12	100%	13	100%	26	97%	102
	6	100%	211	99.0%	248	95%	285
VGG-16-BN	24	100%	1	100%	5	98.5%	51
	18	100%	1	100%	5	98.5%	69
	12	100%	1	100%	5	97%	102
	6	100%	77	100%	86	95%	285
CIFAR10
Model	$\mathcal{E}$	QuEry Attack		Square		AdversarialPSO
		ASR	Queries	ASR	Queries	ASR	Queries
Inception-v3	24	100%	1	89.5%	5	97.5%	31
	18	100%	2	92.5%	17	96.0%	41
	12	97.5%	20	94.5%	77	95.0%	54
	6	91.0%	428	95.5%	776	94.5%	223
ResNet-50	24	100%	2	92.0%	8	97.5%	31
	18	100%	8	91.0%	23	96.0%	41
	12	100%	96	87.0%	110	95.0%	54
	6	99.0%	565	87.0%	449	94.5%	223
VGG-16-BN	24	100%	1	89.5%	5	97.5%	31
	18	99.0%	2	87.0%	14	96.0%	41
	12	98.0%	60	87.0%	86	95.0%	54
	6	95.5%	741	88.5%	890	94.5%	223
MNIST
Model	$\mathcal{E}$	QuEry Attack		Square		AdversarialPSO
		ASR	Queries	ASR	Queries	ASR	Queries
Conv Net	80	100%	5	86.0%	14	76%	2675
	60	93.5%	72	93.0%	77	99%	292

. The code is available at https://github.com/razla/QuEry-Attack, accessed on 13 September 2022.

We evaluated QuEry Attack by executing experiments against three different pretrained image-classification models, taken from PyTorch [31]—Inception-v3 [32], ResNet-50 [33], and VGG-16-BN [34]—over three image datasets: ImageNet, CIFAR-10, and MNIST. We employed 200 randomly picked and correctly classified images from the test sets. For ImageNet, Inception-v3 has an accuracy of 78.8%, ResNet-50 has an accuracy of 76.1%, and VGG-16-BN has an accuracy of 73.3% (these are top-1 accuracy values; for ImageNet, top-5 accuracy values are also sometimes given, which in our case are: Inception-v3—94.4%, ResNet-50–92.8%, VGG-16-BN—91.5%). CIFAR10 accuracy values are: Inception-v3—93.7%, ResNet-50—93.6%, and VGG-16-BN—94.0%. For the MNIST dataset we trained a convolutional neural network (CNN), whose architecture is delineated in

Table 2. CNN used for MNIST.

Conv Block		Hyperparameters
Layer	Layer type	Hyperparameter		Value
1	Convolution	Epochs		300
2	BatchNorm2d	Batch size		64
3	ReLU	Optimizer		$Adam$
		Learning rate		$0.01$
		Weight decay		1 × 10−6
CNN Architecture
Layer	Layer type	No. channels	Filter size	Stride
1	Conv Block	32	$3\times 3$	1
2	Max Pooling	N/A	$2\times 2$	2
3	Conv Block	64	$3\times 3$	1
4	Max Pooling	N/A	$2\times 2$	2
5	Conv Block	128	$3\times 3$	1
6	Max Pooling	N/A	$2\times 2$	2
7	Dropout $(p=0.5)$	N/A	N/A	N/A
8	Fully Connected	128	N/A	N/A
9	Fully Connected	10	N/A	N/A

, which attained 98.9% accuracy.

We chose to use the above models since they are the most commonly used CNN architectures. QuEry Attack exploits the nature of convolution layers by using square mutations and stripes initialisation, which were shown to be very effective [17]. Further, these architectures serve as a backbone for many other downstream tasks, such as object detection [35], semantic segmentation [36], and image captioning [37]. Recently, vision transformers have beaten CNNs in image classification tasks [38], but they require huge amounts of data and resources that are not available too many (including us). Future work will be dedicated to expand the attack to vision transformers.

All accuracy values are over test images. We used the Adversarial Robustness Toolbox (ART) [39] to evaluate QuEry Attack against other attacks. We restricted all attacking algorithms to a maximum of 42K queries to the model ($N=70$, $G=600$) for MNIST and CIFAR10, and 84K queries ($N=70$, $G=1200$) for ImageNet. A query refers to a prediction supplied by the model for a given image. To make the most of the computational resources we had available we prioritized actual, experimental runs over hyperparameter runs, so hyperparameters were chosen through limited trial and error. In the future, we plan to perform a more thorough hyperparameter sweep using Optuna [40]. The only hyperparameters we set were the population size $N=70$, tournament size $T=25$, and $p=0.1$; these are used for all experiments reported herein. The number of generations (G) was derived from the query budget.

AdversarialPSO [41] results were obtained by running the code in the GitHub repository referred to in their paper. Due to technical difficulties it was run against the original models that this attack was planned to run against, namely, Inception-v3 for ImageNet, and their own trained networks for CIFAR-10 and MNIST. We duplicated these results in the table for all models.

5.1. Attacking Defenses

We show how QuEry Attack breaks a collection of defense strategies designed to boost the robustness of models against adversarial attacks.

5.1.1. Attacking Non-Differentiable Transformations

Gradient masking is achieved via non-differentiable input transformations, which rely on manipulating gradients to defeat gradient-based attackers [42,43]. Further, randomized transformations make it more difficult for the attacker to be certain of success. It is possible to foil such a defense by altering the defense module that performs gradient masking, but this is not an option within the black-box scenario. Herein, we investigated three non-differentiable transformations against QuEry Attack: JPEG compression, bit-depth reduction (also known as feature squeezing), and spatial smoothing. We show that QuEry Attack can defeat these input modifications, due to its gradient-free nature.

JPEG compression [44] tries to generate patterns in color values to minimize the amount of data that has to be captured, resulting in a smaller file size. Some color values are estimated to match those of surrounding pixels in order to produce these patterns. This compression means that slight imperfections in the quality of the image will not be as noticeable. The degree of compression may be tweaked, providing a customizable trade-off between image quality and storage space. An example of the different compression degrees is shown in Figure 2. The results in

Table 3. QuEry Attack’s resistance to non-differentiable transformation defenses. JPEG compression, bit-depth reduction, and spatial smoothing were tested on CIFAR10 and ImageNet.

Defense	$\mathcal{E}$	CIFAR10		ImageNet
		ASR	Queries	ASR	Queries
JPEG Compression $(q=70)$	18	100%	2	100%	2
	12	98.0%	13	97.5%	14
	6	93.5%	287	99.5%	311
Bit-Depth Reduction $(d=3)$	18	100%	2	100%	2
	12	99.5%	5	100%	27
	6	96.0%	72	99.5%	145
Spatial Smoothing $(w=5)$	18	100%	1	100%	1
	12	100%	1	100%	2
	6	99.0%	6	99.5%	144

were evaluated with image quality $q=70$.

Bit-depth reduction [45] can be done both by reducing the color depth of each pixel in an image and using spatial smoothing to smooth out individual pixel discrepancies. By merging samples that correspond to many different feature vectors in the original space into a single sample, bit-depth reduction decreases the search space accessible to an opponent. An example of different bit-depth values is shown in Figure 3. The results in Table 3 were evaluated with bit depth $d=3$.

The term “spatial smoothing“ refers to the averaging of data points with their neighbors [46]. This has the effect of a low-pass filter, with high frequencies of the signal being eliminated from the data while low frequencies are enhanced. As a result, an image’s crisp “edges” are softened, and spatial correlation within the data becomes more prominent, as shown in Figure 4. Data averaging is determined according to a given window size. The results in Table 3 were evaluated with window $w=5$.

Our results are delineated in Table 3. For this experiment we used a total budget of 82K queries to the model ($N=70$, $G=1200$)—which was Inception-v3. For each given image, we first checked that it was correctly classified after applying the defense on the image, then we applied QuEry Attack on it. The different input values for the transformations were chosen such that applying them would not be destructive.

For these experiments we used the same budget of queries as in the previous experiments. For both CIFAR-10 and ImageNet, QuEry Attack has a high success rate against all non-differentiable transformations.

5.1.2. Attacking Robust Models

A model is considered to be robust when some of the input variables are largely perturbed, but the model still makes correct predictions. Recently, several techniques have been proposed to render the models more robust to adversarial attacks. One commonly used technique to improve model robustness is adversarial training. Adversarial training integrates adversarial inputs—generated with other trained models—into the models’ training data. Adversarial training has been proved to be one of the most successful defense mechanisms for adversarial attacks [47,48,49,50].

We conducted an experiment to see how well QuEry Attack performs on robust models. For CIFAR10 we used the robust model, WideResNet-70-16 [51], wherein they used generative models trained only on the original training set in order to enhance adversarial robustness to $l_p$ norm-bounded perturbations. For ImageNet we used WideResNet-50-2 [52], which is a variant of ResNet wherein the depth of the network is decreased and the width of the network is increased. This is achieved through the use of wide residual blocks. Both of these top models were taken from the RobustBench repository [53]. We used the same 200 randomly selected images from our previous experiments, a budget of 84K queries ($N=70$, $G=1200$) for CIFAR10, and a budget of 126K queries ($N=70$, $G=1800$) for ImageNet. As seen in

Table 4. QuEry Attack’s performance on robust models over CIFAR10 and ImageNet.

Model	$\mathcal{E}$	ImageNet
		ASR	Queries
Wide ResNet-50-2	12	98.0%	98
	6	93.5%	1187
		CIFAR10
Wide ResNet-70-16	18	94.5%	156
	12	85.0%	487

, QuEry Attack succeeds at breaking those strongly defended models.

5.2. Transferability

An adversarial example for one model can often serve as an adversarial example for another model, even if the two models were trained on different datasets, using different techniques; this is known as transferability [54]. White-box attacks may overfit on the source model, as evidenced by the fact that black-box success rates for an attack are almost always lower than those of white-box attacks [7,13,55,56]. Herein, we checked transferability of our proposed black-box attack on 200 correctly classified ImageNet images by both the source model and the target model, using different $ϵ$ values. The results, summarized in

Table 5. QuEry Attack’s transferability on ImageNet models. TSR: Transferability Success Rate.

Source Model → Target Model	$\mathcal{E}$	TSR
Inception-v3 → ResNet-50	24	67.0%
	18	47.5%
	12	33.5%
	6	13.5%
ResNet-50 → VGG-16-BN	24	90.0%
	18	81.5%
	12	61.5%
	6	28.5%
VGG-16-BN → Inception-v3	24	59.0%
	18	46.5%
	12	30.0%
	6	12.5%

, show a positive correlation between the $ϵ$ values and the transferability success rate. We noted that attacks are better transferred between ResNet-50 to VGG-16-BN models and surmise this is due to the fact that ResNets models were mostly inspired by the philosophy of VGG models, wherein they use relatively small $3\times 3$ convolutional layers.

6. Discussion and Concluding Remarks

We presented an evolutionary, score-based, black-box attack, showing its superiority in terms of ASR (attack success rate) and number-of-queries over previously published work. QuEry Attack is a strong and fast attack that employs a gradient-free optimization strategy. We tested QuEry Attack against MNIST, CIFAR10, and ImageNet models, comparing it to other commonly used algorithms. We evaluated QuEry Attack’s performance against non-differential transformations and robust models, and it proved to succeed in both scenarios.

As noted, we discovered that the software scene in adversarial attacks is a tad bit muddy. We encourage researchers to place executable code on public repositories—code that can be used with ease. Furthermore, we feel that the field lacks standard means of measuring and comparing results. We encourage the community to establish common baselines for these purposes.

We came to realize the importance of a strong initialization procedure. Although this is true of many optimization algorithms, it seems doubly so where adversarial optimization is concerned. Table 1 shows that successful attacks are sometimes found during initialization—the vertical-stripes initialization in particular proved highly potent—and even if not, the number of queries (and generations) is significantly curtailed.

Figure 5 and Figure 6 show that adversarial examples are barely distinguishable to the human eye. Clearly, neural networks function quite differently than humans, capturing entirely different features. More work is needed to create networks that are robust in a human sense.

We think that evolutionary algorithms are well-suited for this kind of optimization problems and our findings imply that evolution is a potential research avenue for developing gradient-free black-box attacks. Furthermore, evolution needs to be evaluated against a fully black-box model.

Evolution may also be a solution for rendering models more robust. In [57] it was shown that combining different activation functions could be used to increase model accuracy; this approach might also be used for obtaining robustness.