Zero-Shot Proxy with Incorporated-Score for Lightweight Deep Neural Architecture Search

Abstract

Designing a high-performance neural network is a difficult task. Neural architecture search (NAS) methods aim to solve this process. However, the construction of a high-quality accuracy predictor, which is a key component of NAS, usually requires significant computation. Therefore, zero-shot proxy-based NAS methods have been actively and extensively investigated. In this work, we propose a new efficient zero-shot proxy, Incorporated-Score, to rank deep neural network architectures instead of using an accuracy predictor. The proposed Incorporated-Score proxy is generated by incorporating the zen-score and entropy information of the network, and it does not need to train any network. We then introduce an optimal NAS algorithm called Incorporated-NAS that targets the maximization of the Incorporated-Score of the neural network within the specified inference budgets. The experiments show that the network designed by Incorporated-NAS with Incorporated-Score outperforms the previously proposed Zen-NAS and achieves a new SOTAaccuracy on the CIFAR-10, CIFAR-100, and ImageNet datasets with a lightweight scale.

1. Introduction

Neural architecture search (NAS) is a well-known automated, high-performance deep neural network architecture designed for various deep learning tasks, such as image recognition and language modeling. Most NAS approaches comprise two main components, namely a network generator and a performance predictor. The network generator suggests prospective high-quality models, while the accuracy predictor estimates the accuracy of these proposed networks.

There are three commonly used generators, namely uniform sampling [1], reinforcement learning [2], and the evolutionary algorithm [3]. Three main approaches to build performance predictors are brute-force [3,4,5,6], predictor-based [2,7,8], and one-shot methods [5,9,10,11,12,13,14,15,16,17].

Due to the huge computational overhead, the construction of an accuracy predictor that provides good suggestions at the optimal cost is a significant challenge. For instance, a considerable number of networks must be trained using both brute-force and predictor-based approaches. To alleviate these issues, one-shot methods propose parameter-sharing strategies such as DARTS [9], SNAS [13], PC-DARTS [10], ProxylessNAS [14], GDAS [15], FBNetV2 [16], DNANet [18], and Single-Path One-Shot NAS [1]. Although these one-shot methods are more efficient than previous methods, there is still a drawback that requires the training of a large supernet with high computational costs. Another disadvantage is the degradation of accuracy and predictor quality owing to model interference [17,19,20] in almost all supernet-based methods. Moreover, searching for large target networks in the context of resource limitations is challenging because the supernet needs to be significantly larger than the target network. Therefore, designing high-performance networks using one-shot methods is difficult.

To address these issues, in recent works, zero-cost predictors have been proposed instead of expensive predictors for the search process. The key idea behind these methods is to leverage the expressiveness of the deep neural network as a proxy that is positively correlated with the network’s performance [21,22,23,24,25,26].

In this paper, we propose a new optimal zero-shot proxy as the network’s expressiveness for lightweight NAS called Incorporated-Score. The proposed method is inspired by two theories in deep learning research. First, recent breakthroughs in deep learning research [27,28,29,30,31,32,33,34,35,36,37] show that the advantages of deep models compared to shallow models come from the greater expressiveness of deep models, even when the number of neurons is the same in deep models and shallow models. The number of linear regions is known as a good metric to estimate the performance of a neural network. However, Lin et al. [25] showed that directly counting the linear regions is infeasible in a large network. By using Gaussian complexity, they proposed zero-proxy zen-score, which takes advantage of the power of both the distribution of linear regions and the linear classifier’s Gaussian complexity. Second, based on information theory in deep learning information research [26,38,39,40,41,42,43], the differential entropy of the network is an expressiveness of the deep neural network. The deep neural network’s entropy depends on several factors, such as the number of channels, kernel size, and group. These methods are two different approaches to generate a high-quality neural network. Inspired by the two theories mentioned above, we use the zen score [25] and entropy of the network [26] as inputs to compute the network’s expressiveness. This approach enables our proxy to account for the power of distribution of linear regions and the coefficient matrix under entropy optimization. However, the conventional approach with a simple sum of the zen score and entropy of the network [26] leads to dominant issues. Therefore, we propose an efficient method to incorporate two factors called Incorporated-Score. The proposed approach employs neural architecture search (NAS) to discover high-quality networks across diverse model inference measures, encompassing floating-point operations per second (FLOPs) and the number of model parameters. The achieved performance sets a new benchmark, outperforming Zen-NAS on CIFAR-10/CIFAR-100/ImageNet and establishing state-of-the-art (SOTA) results on a lightweight scale. In particular, our work achieves top-1 accuracy of $96.86\%$ and $81.1\%$ on the CIFAR-10 and CIFAR-100 datasets, respectively, with less than 1 M params and outperforms ZenNet-cifar-1M ($96.2\%$/$80.1\%$). Within 5 h of GPU, the lightweight EZenNet designed by Incorporated-NAS peaks at $80.1\%$ top-1 accuracy on ImageNet-1K, which only has 800 M FLOPs.

The contributions of this study are summarized as follows:

• We propose a zero-cost proxy for NAS called Incorporated-Score. It leverages zen-score and entropy factors to make an efficient proxy to rank networks. The entropy of the network plays a role as the auxiliary optimization factor to zen score in the proposed method.
• EZenNet, which is designed by Incorporated-Score, outperforms the baseline SOTA ZenNet, which is designed by Zen-NAS, and it achieves a new SOTA on the CIFAR-10/CIFAR-100/ImageNet datasets at a lightweight scale.

2. Related Works

2.1. Information Theory in Deep Learning

Information theory is known as a robust method for investigating complicated systems such as machine learning and deep learning networks. The maximum entropy principle [38,39] is one of the most popular applied principles in this field of research. Numerous studies [41,43,44,45,46] have made an effort to seek the relation between the entropy factor and neural network architecture. For instance, Chan et al. [44] attempted to clarify the learning capabilities of deep neural networks by reducing the subspace entropy. Saxe et al. [46] tried to understand the power of bottlenecks in deep neural network architectures by investigating the entropy distribution and the outflow of information. Yu et al. [43] introduced the principle of maximizing the decrease in coding rate to achieve optimization, while Sun et al. [41] proposed high-performance object detection networks by using the maximal entropy of a multi-scale feature map. Additionally, a monograph [45] examined the mutual information among different neurons in the multilayer perceptron model.

In the context of DeepMAD [26], rather than focusing on coding rate reduction as in [44], the entropy of the model was considered in this study. The effectiveness of this approach was also experimentally verified to demonstrate that solely maximizing entropy, as in [41], was insufficient.

2.2. Neural Architecture Search (NAS)

The target of NAS is to automatically generate a neural architecture that delivers optimal performance using limited computing resources and requiring minimal human intervention. In early methods, brute force was used to find the most accurate network structure by training the neural network models for accuracy. The evolutionary algorithm (EA) and reinforced learning (RL) are popular generators or samplers in NAS. AmoebaNet [3] was used to conduct a neural network search on the CIFAR-10 dataset applying EA, then transfer the architecture to the ImageNet dataset. However, it took approximately 3150 GPU days to search and reach more than 80% top-1 accuracy on ImageNet dataset. Subsequently, several EA-based NAS algorithms have been proposed to enhance search capacity by using downsampled images or by decreasing the query counts, such as CARS [12], EcoNAS [11], PNAS [47], and GeNet [5]. Other researchers used RL as a generator in NAS, including Mnasnet [48], NASNet [49], and MetaQNN [6]. However, both RL and EA methods consume hundreds of GPU days or even more computing resources, which can be highly detrimental to researchers due to the extensive computational cost. Several one-shot methods have been proposed to address these problems. Several architectures have been trained to predict the accuracy of networks [2,8]. Thanks to the training of a large supernet, the one-shot method decreased the training cost of NAS. Therefore, it is commonly utilized in efficient NAS works, such as SNAS [13], DARTS [9], ProxylessNAS [14], PC-DARTS [10], single-path one-shot NAS [1], GDAS [15], FBNetV2 [16], and DNANet [18]. OFANet [17] showed that weight sharing is hindered by model interference, which leads to a significant drop in the accuracy of the target model. Therefore, a progressive shrinkage strategy was proposed to address this issue. It took approximately 52 GPU days to structure OFANet, achieving 80% top-1 accuracy on ImageNet. Another high-quality network constructed by NAS is EfficientNet [50], which achieved 84.4% accuracy on ImageNet after searching for 3800 GPU days. Although the above efforts have significantly decreased search costs, many days have been spent searching for neural networks using one-shot methods.

Recently, several works have been actively proposed to explore zero-shot proxies for efficient NAS, including [21,23,24,25]. Abdelfattah et al. [21] conducted experiments to evaluate the effectiveness of several zero-shot pruning expressiveness methods, among which synflow [22] achieves the highest performance. Pruning proxy methods conduct parameter pruning using a saliency metric that estimates the change in loss or gradient norm when a parameter of the network is pruned. The loss is the predicted by-product of all parameters in the networks in synflow, while the previous proxies need to use a mini batch of training data and cross-entropy loss. Although these proxies achieve moderate accuracy on the CIFAR-10 dataset, the top-1 accuracy needs more improvement on the CIFAR-100 dataset. Another work called training-free neural architecture search (TE-NAS) [23] ranks architectures by considering two factors of the deep neural networks. The first factor is the spectrum of the neural tangent kernel (NTK), and the second is the count of linear regions in the input space. However, a limitation of this method is that directly calculating the count of linear regions becomes impractical when the neural network is too large. This work achieved 74.1% top-1 accuracy on ImageNet, which is still far behind the current state-of-the-art baselines. NASWOT [24] is another approach that achieves similar network performance to TE-NAS on the CIFAR-10 and CIFAR-100 datasets with greater time efficiency by utilizing the kernel matrix of binary activation patterns. Zen-NAS [25] proposes a zero-proxy zen score that takes a few forward inferences on randomly initialized networks using random Gaussian input. Solving the limitation of directly counting the number of linear regions in a large deep network, it takes advantage of both the distribution of linear regions and the power of the coefficient matrix in each region by using the complexity of Gaussian input. In comparison, Zen-NAS achieves more robust performance than TE-NAS and NASWOT in terms of both searching time and network performance. This is a study that reports NAS using a zero-cost proxy, surpassing the performance reported in studies involving training on ImageNet.

3. Preliminary

In this section, we briefly introduce the notations used in this paper, including the L-layer neural network and vanilla convolutional neural network (VCNN). Moreover, we present the entropy of the MLP model and control of an over-deep network issue in MLP.

3.1. L-Layer Neural Network Notation

A neural network with a depth of M layers is defined as a function ($f:{\mathcal{R}}^{d_0}\to {\mathcal{R}}^{d_M}$, where $d_0$ and $d_M$ are the input and output dimensions, respectively). The input image and the t-th layer’s feature map are represented by ${\mathit{x}}_0\in {\mathcal{R}}^{d_0}$ and ${\mathit{x}}_t$, respectively. For each t-th layer, the number of input channels is denoted by $d_{t-1}$, while the number of output channels is represented as $d_t$. ${\mathit{\theta }}_t\in {\mathcal{R}}^{d_t\times d_{t-1}\times k\times k}$ refers to the convolutional kernel. The image resolution and the mini-batch size are denoted by $H\times W$ and B, respectively.

3.2. Vanilla Convolutional Neural Network

The VCNN is a commonly applied prototype in theoretical studies [30,35,36]. Multiple convolutional layers, followed by the RELU activation stack, comprise the main body of a vanilla network. All additional components, such as residual links and batch normalization layers, are cut off from the main network. The resolution of the feature map is decreased to 1 × 1 by a global average pooling (GAP) layer following the backbone. After that, a fully connected layer is attached. Finally, the features of the neural network are transferred to the distribution target by applying the softmax function. The feature map is $f\left(\mathit{x}\right|\mathit{\theta })$, which is considered the main body output of the network with input of $\mathit{x}$ and network parameters of $\mathit{\theta }$ before the GAP layer. Network expressivity is measured concerning the pre-GAP because of most of the necessary information is remaining. Recently, some theoretical studies have been conducted on the expressivity of deep networks [35,37]. An important insight gleaned from these studies is that any standard network can be viewed as an ensemble of piecewise linear convex polytopes ($\mathcal{U}=\{{\mathcal{U}}_1,{\mathcal{U}}_2,\dots ,{\mathcal{U}}_{\left|\mathcal{U}\right|}\}$, where $\left|\mathcal{U}\right|$ represents the number of linear regions) as follows:

Lemma 1

([27,33]). At the t-th layer, the activation pattern is denoted as ${\mathcal{G}}_t\left(\mathit{x}\right)$. Then, for any standard network, $f(·)$ is defined as follows:

$$f\left(\mathit{x}\right|\mathit{\theta })=\sum _{{\mathcal{U}}_i\in \mathcal{U}}{\mathcal{I}}_{\mathit{x}}\left({\mathcal{U}}_i\right){\mathbf{W}}_{{\mathcal{U}}_i}\mathit{x}$$

(1)

where a convex polytope is expressed as ${\mathcal{U}}_i$, relying on $\{{\mathcal{G}}_1\left(\mathit{x}\right),{\mathcal{G}}_2\left(\mathit{x}\right),\dots ,{\mathcal{G}}_M\left(\mathit{x}\right)\}$; $\mathcal{U}$ is a finite set of complex polytopes in ${\mathcal{R}}^{d_0}$; ${\mathcal{I}}_{\mathit{x}}\left({\mathcal{U}}_i\right)=1$ if $\mathit{x}\in {\mathcal{U}}_i$ and zero otherwise; and ${\mathbf{W}}_{{\mathcal{U}}_i}$ is a coefficient matrix of size $R^{d_M\times d_0}$.

In other words, a standard network can be construed as a set of piecewise linear functions that depend on activation patterns. Therefore, the abovementioned studies used the number of linear regions ($\left|\mathcal{U}\right|$) playing a role in expressivity as a proxy for the VCNN family. However, the computation of counting ($\left|\mathcal{U}\right|$) is infeasible for large networks. Moreover, directly using $\left|\mathcal{U}\right|$ does not take advantage of the representational power of each coefficient matrix (${\mathbf{W}}_{{\mathcal{U}}_i}$) in Lemma 1.

3.3. Entropy of MLP Models

Entropy is measured as the expressiveness of a deep network [41,44]. Suppose that in an L-layer MLP ($f(·)$), $w_i$ and $w_{i+1}$ are defined as the input and output channels of the i-th layer, respectively. Output ${\mathbf{x}}_{i+1}$ is defined by ${\mathbf{x}}_{i+1}={\mathbf{M}}_i{\mathbf{x}}_i$, where ${\mathbf{M}}_i\in {\mathbb{R}}^{w_{i+1}\times w_i}$ denotes a trainable weight. According to the entropy analysis reported in [44], the MLP model’s entropy ($f(·)$) is expressed as follows:

$$H_f=w_{L+1}\sum _{i+1}^{L}log\left(w_i\right),$$

(2)

where $H_f$ is the upper bound of the normalized Gaussian entropy of the MLP ($f(·)$). In [26], it was shown that simply maximizing the entropy ($H_f$) to determine the best expressiveness of a network results in an over-deep network problem. This is due to the fact that entropy increases exponentially faster with depth compared to width, as shown in Equation (2). An excessively deep network is challenging to train because effective information propagation is hindered [45]. This leads to the question of how to address this problem, which is discussed in Section 3.4.

3.4. Effectiveness of Controlling Extremely Deep Networks in MLP

A network that hinders effective information propagation can be considered chaotic. This phenomenon is known as an over-deep network [45]. Specifically, in an over-deep network, when the network weights are randomly initialized, a small perturbation in the first network layers can result in an exponential large perturbation in the output of the final network’s layers. During backpropagation, the gradient flow does not effectively propagate through the entire network. Hence, training a network becomes difficult when it is over-deep. In [45], the metric term network effectiveness was proposed as follows:

$$\rho =L/w,$$

(3)

where L is the number of MLPs and the width of each layer (w) is the same. The MLP behaves as a single-layer linear model if $\rho \to 0$. In contrast, it is a chaotic system if $\rho \to \infty$. We can easily see that Equation (3) assumes that the MLP has a uniform width, whereas in practice, the width ($w_i$) of each layer may be different. To solve this problem, Shen et al. [26] proposed using the average width of the MLP in Equation (3) as follows:

$$\overline{w}={\left(\prod _{i=1}^L{w}_i\right)}^{1/L}=exp({\displaystyle \frac{1}{L}}\sum _{i=1}^{L}log{w}_i).$$

(4)

4. Method

The overall workflow of the proposed Incorporated-NAS is represented in Figure 1. There are two main components, which are the architecture generator and the accuracy predictor. As shown in Figure 1, we apply an evolutionary algorithm (EA) in the architecture generator, which generates the architecture from the search space ($\mathcal{A}$). Then, each generated network is evaluated by the accuracy predictor based on the proposed zero-cost proxy, which is Incorporated-Score. The proposed Incorporated-Score proxy is a key component of the accuracy predictor in scoring the networks without any training in NAS.

The problem of measuring the expressivity of a deep neural network and its positive correlation with model accuracy plays an important role in NAS. In this section, we briefly describe the expressivity of the VCNN using zen-score. The original paper [25] that proposed zen-score, contains many more details and in-depth specifications. Then, we briefly introduce the efficient maximization of entropy as expressivity of the CNN. Finally, we present our methods, consisting of the effectiveness evaluation of Incorporated-Score as a new zero-cost proxy and Incorporated-NAS to optimize the proposed score in the search step.

4.1. Zen-Score

Owing to the Gaussian complexity [51] of a linear classifier, Lin et al. [25] proposed a new expressivity for a network that can be efficiently measured by its expected Gaussian complexity, represented by $\mathrm{\Psi }$ score as follows:

$$\mathrm{\Psi }\left(f\right)=log{\mathbb{E}}_{\mathit{x},\mathit{\theta }}\left\{\sum _{{\mathcal{U}}_i\in \mathcal{U}}{\mathcal{I}}_{\mathit{x}}\left({\mathcal{U}}_i\right){\parallel {\mathbf{W}}_{{\mathcal{U}}_i}\parallel }_F\right\}$$

(5)

$$=log{\mathbb{E}}_{\mathit{x},\mathit{\theta }}{\parallel {▿}_{\mathit{x}}f\left(\mathit{x}\right|\mathit{\theta })\parallel }_F$$

(6)

Specifically, $\mathit{x}$ and $\mathit{\theta }$ are randomly sampled from the prior distributions, then the average $\parallel {\mathbf{W}}_{{\mathcal{U}}_i}{\parallel }_F$. Following [25], we use the Gaussian distribution represented by $\mathcal{N}(0,1)$ to sample the input ($\mathit{x}$) and $\mathit{\theta }$ parameter. This corresponds to computing the expected gradient norm of f with respect to input $\mathit{x}$. It is crucial to highlight that in the $\mathrm{\Psi }$ score, only the gradient of $\mathit{x}$ is considered, not the gradient with respect to $\mathit{\theta }$.

However, Lin et al. [25] showed that a numerical overflow is incurred when directly computing the $\mathrm{\Psi }$ score for a very deep network due to gradient explosion without a BN layer. This problem could be resolved by adding a BN layer, but it leads to a scale-sensitive issue that is introduced in deep learning complexity analysis [52,53]. To address these issues, the $\mathrm{\Psi }$ score is rescaled using the product of BN layers’ variance statistics. To differentiate from the $\mathrm{\Psi }$ score, the new score is defined as the zen-score.

4.2. Maximizing Entropy as Expressivity of Network

In this subsection, we briefly describe the definition of entropy and how its effectiveness is generalized from an MLP to a CNN as in [26].

4.2.1. Entropy of CNN

Suppose that the i-th CNN layer is defined by the input channel number ($c_i$), the output channel number ($c_{i+1}$), the kernel size ($k_i$), and the group number ($g_i$). The operator of this CNN corresponds to matrix multiplication ($W_i\in {\mathbb{R}}^{c_{i+1}\times c_i\times k_i^2/g_i}$). A new dimension of CNN feature maps is the resolution ($r_i\times r_i$) in the i-th layer. The definition of the CNN’s entropy is proposed as follows [26]:

$$H_L\triangleq log\left(r_{L+1}^2{c}_{L+1}\right)\sum _{i=1}^{L}log(c_i{k}_i^2/g_i).$$

(7)

In Equation (7), inspired by [54], using logarithms provides a more effective formulation of the ground-truth entropy for natural images.

4.2.2. Effectiveness Entropy for Scoring Networks

Some studies have used entropy as expressivity to find high-performance CNN models. However, the model performance does not always increase with entropy (see Figure 2 in [26]). Specifically, a positive relationship no longer exists in an over-deep network. Therefore, Shen et al. [26] proposed using the effectiveness ($\rho$) to control this issue, as described in Section 3.4. To find the best network by maximizing the entropy of the network, they achieved optimization by using a mathematical programming problem. The final score is defined as

$$E_{score}=\sum _{i=1}^M{\alpha }_i{H}_i-\beta Q,$$

(8)

where the L-layer CNN model ($f(·)$) has M stages; $H_i$ denotes the entropy of the i-th stage, which is defined in Equation (7); and ${\alpha }_i$ and $\beta$ are hyperparameters. The weight of the entropy at different scales is denoted as $\left\{{\alpha }_i\right\}$. $E_{score}$ is maximized by finding $w_i^{*}$ and $L_i^{*}$ under given constraints, such as budget (FLOPs, params, ${\rho }_n\le {\rho }_0$). According to Equations (3) and (4), the effectiveness of CNN ${\rho }_n$ is expressed as follows:

$${\rho }_n=L·{\left(\prod _{i=1}^L{w}_i\right)}^{-1/L}$$

(9)

where each CNN layer’s width is denoted as $w_i=c_i{k}_i^2/g_i$ and $L_i$ is each stage’s depth for $i=\{1,2,3,\dots ,M\}$. The effectiveness of the network is controlled by ${\rho }_0$, and its value is typically adjusted within the range of $[0.1,2.0]$. Q, which penalizes the objective function if the depth distribution is nonuniform across stages as follows:

$$Q\triangleq exp\left[Var(L_1,L_2,\dots ,L_M)\right].$$

(10)

4.3. Effectiveness of the Proposed Incorporated-Score as a Proxy for NAS

We observed that using $E_{score}$ as the expressivity of the network is based on information theory in deep learning to optimize high-performance CNNs on the same scale as ResNet. The key idea is to maximize the network entropy factor while maintaining the network effectiveness to avoid extremely deep network generation, controlled by a small constant. However, the method proposed in [26] has several limitations as well, such as the fact that the three empirical guidelines are not based on a strong theoretical foundation.

Zen-NAS [25] uses the gradient norm of an input image as the network’s expressiveness score and proposes the use of two feed-forward inferences to approximate this metric for classification. In particular, it works efficiently to find lightweight networks for the classification task. Moreover, this metric takes the power of the weight of the network and the number of linear regions. Entropy is a different approach in information theory based on the channel number, kernel size, and group number. We take advantage of entropy as an additional optimal factor in the zen score to rank neural networks for image classification tasks.

First, the simplest idea is to take the sum of the zen score and the effectiveness entropy ($E_{score}$) with balance weight. The generated network performance is dramatically decreased compared to the original Zen-NAS. We investigated this issue by analyzing the value of the component score in our proxy during searching. Interestingly, we observed that the value of $E_{score}$ is much higher than the zen-score value, as shown in Figure 2, which leads to a bias in searching for a high-performance network. In particular, the $E_{score}$ is in the range of $[0,8000]$, whereas the maximum value of the zen score is approximately 120. Secondly, we drastically reduce the entropy weight to ${10}^{-4}$ and increase the zen-score weight to 0.9999. The generated network performance is at the same level as the original Zen-NAS, proving that the $E_{score}$ needs to be normalized at the same level as zen score. Therefore, we propose an efficient normalization of $E_{score}$ to address the issue of biased values. The effective Incorporated-Score is proposed as follows:

$$\text{Incorporated-Score}=w_z{Z}_{score}+w_{d}F\left(E_{score}\right)$$

(11)

where $Z_{score}$ denotes the zen score; $F(·)$ is the normalization function, for which we choose log or square-root functions to normalize the value of $E_{score}$; and $w_z$ and $w_d$ are hyperparameters used to balance the weights between zen score and normalized $E_{score}$, respectively, with constraints of $0<w_z<1$, $0<w_d<1$, and $w_z+w_d=1$. We consider two functions, namely log and square-root functions, as the $E_{score}$ normalization function because both of them are monotonic functions and yield values greater than zero for positive inputs.

In our experiments, we set the weights of the entropies to ${\alpha }_i=1$ on each scale for comparison with the original Zen-NAS. This is because the entropy works as an additional optimal factor in the zen score, and our architectural design generator is the same as that reported in [25]. We set $\beta =10$, as suggested in [26].

4.4. Incorporated-NAS with Optimized Incorporated-Score

We propose the Incorporated-NAS algorithm, which targets the maximization of the Incorporated-Score of the neural network. Incorporated-NAS is based on Zen-NAS and further computes the efficiency scores (${\rho }_n$ and $E_{score}$) in the searching stage. The evolutionary algorithm is used as an architecture generator in our Incorporated-NAS. Other generators, such as greedy selection or reinforcement learning, can also be chosen. The sequential process of Incorporated-NAS is provided in Algorithm 1.

First, N structures are randomly generated as a population ($\mathcal{P}$). The structure with the highest incorporated score is the desired output of Incorporated-NAS after T evolutionary iterations. Specifically, a network in the population ($\mathcal{P}$) is mutated to generate a new architecture for each iteration step (t), as shown in Algorithm 2. In the mutation stage, the depth and width of the chosen layer are transformed within the mutation ratio range. The range of the mutation ratio is $[0.5,2.0]$, as in [25]. This means that the new value is generated by half or two times the current value in our work. A new child structure (${\widehat{A}}_t$) is added to the population if its inference expense is within the given budget, as shown in row 5 of Algorithm 1. In addition, aiming to avoid generating overly deep structures, the depth of the network is regulated by the maximal value (L) and effectiveness (${\rho }_0$). Finally, we remove the network with the lowest score if the population size is more than N.

Algorithm 1 Incorporated-NAS

Input: Given search space S, evolutionary population size N, constraints budget O, total number of iterations T, maximal depth L, initial structure $A_0$, effectiveness score ${\rho }_0$ Ensure: NAS for optimal EZenNet $A_{ezennet}^{*}$     Initialization of the population $\mathcal{P}=\left\{A_0\right\}$.     for $t=1,2,\dots ,T$do      Select randomly $A_t\in \mathcal{P}$.      Mutation stage: ${\widehat{A}}_t=\mathrm{MUTATION}(A_t,\mathcal{S})$      if ${\widehat{A}}_t$ has more than L layers or overcome budget O or ${\rho }_n\ge {\rho }_0$ then       continue      else       Compute zen-score $z=\text{zen-calculation}\left({\widehat{A}}_t\right)$       Compute $E_{score}$       Compute Incorporated-Score $\mathrm{y}=w_{z}z+w_{d}F\left(E_{score}\right)$       Append ${\widehat{A}}_t$ to $\mathcal{P}$      end if      Cut off architecture from the population list which has the lowest Incorporated-Score if the size of $\mathcal{P}$ overcomes population size N.     end for     Return $A_{ezennet}^{*}$, the network which has max value of incorporate-score in $\mathcal{P}$

Algorithm 2 Mutation

Input: Structure $A_t$, Search space $\mathcal{S}$ Ensure: Mutation random structure ${\widehat{A}}_t$    Uniformly choose a block b in $A_t$    The block type, kernel size, width, and depth of b are alternated uniformly in the mutation ratio.    Get the child structure ${\widehat{A}}_t$ from structure $A_t$

5. Experiments

To validate the effectiveness of the proposed Incorporated-Score, we conduct experiments on three datasets, namely CIFAR-10, CIFAR-100, and ImageNet-1k. First, we compare the proposed method to the baseline SOTA proxy, zen-score, on the CIFAR-10 and CIFAR-100 datasets using the same settings for search policy, search space, and training. Next, we compare the search cost of Incorporated-NAS with that of the other methods. Finally, we compare our proposed method to zen-score on ImageNet in lightweight instances.

5.1. Experimental Setting

5.1.1. NAS Settings

We conduct the experiment in the following search spaces:

• Search Space A: comprises residual and bottleneck blocks in ResNet following [55,56].
• Search Space B: comprises of MobileNet blocks following [57,58]. The depth-wise expansion ratio is in the range of $\{1,2,4,6\}$ in the search process.

To satisfy the inference budget, we set the initial architecture as a small, randomly selected network, as in [25], for fair comparison. The set of kernel sizes is $\{3,5,7\}$. In terms of the count of stages, we set three for CIFAR-10 and CIFAR-100 and five for ImageNet. The size of the evolutionary population (N) is 256, and the number of evolutionary iterations is $T=$ 96,000. We set the maximal depth value of the network to $L=18$ and effectiveness to ${\rho }_0=0.5$ following previous works [25,26]. The resolutions are $32\times 32$ for the CIFAR-10 and CIFAR-100 datasets and $224\times 224$ for the ImageNet dataset. We find the optimal pair of $(w_z,w_d)$ on the CIFA-10 and CIFAR-100 datasets, which we then use for NAS on ImageNet. The search for optimal hyperparameters $(w_z,w_d)$ is described in Section 5.2. The following experiments are conducted.

• Incorporated-Score-l: Incorporated-Score is generated by using the logarithm function as the normalization function.
• Incorporated-Score-s: Incorporated-Score is generated by using the square-root function as the normalization function.
• Incorporated-Score-w/o-ls: To explain the effect of our normalization approach, we experiment with Incorporated-Score without any normalization and balanced weights.

5.1.2. Training Setting

• Dataset: CIFAR-10 and CIFAR-100 are two benchmark datasets for image classification. CIFAR-10 consists of 50,000 images for training and 10,000 images for testing in 10 classes. Each image has a resolution of $32\times 32$. CIFAR-100 has a similar number of samples for training and testing but is divided into 100 classes. ImageNet-1k is a large dataset that includes over 1.2 million images for training and 50,000 test images divided into 1000 classes. We experiment with the official training and validation dataset.
• Augmentation: Augmentations including label smoothing, AutoAugment [59,60], mix-up [61], random erasing [62], and random crop/resize/flip/light are used.
• Optimizer: The optimizer is SGD with a momentum of 0.9 for all experiments. The weight decay is $5\times {10}^{-4}$ and $4\times {10}^{-5}$ for CIFAR-10/100 and ImageNet, respectively. The batch size is 256, with an initial learning rate of 0.1 and cosine learning rate decay [63]. For CIFAR-10 and CIFAR-100, we train the models up to 1440 epochs. For ImageNet, the number of train epochs is 480. Following previous research [17,18,64], we use EfficientNet-B3 as a teacher network when training EZenNets.

We compare Incorporated-Score with the baseline SOTA proxy, zen-score. Moreover, we provide the results for three other proxies, namely gradient-norm (grad), synflow [22], and TE-Score [23], to provide an overview of zero-shot methods. For each proxy, the search step operates as in Algorithm 1 with $T=$ 96,000. The best networks on CIFAR-10 and CIFAR-100 are searched for within fewer than 1 M network parameters in the NAS step. After the NAS step, the best-scoring network is trained in the same training setting.

5.2. Search for Hyperparameters ($w_z$,$w_d$) for Each Normalization Proxy

In order to find the best pair $(w_z,w_d)$ in Equation (11) to achieve optimal performance, we set the value of two parameters in the range of $(0,1)$ and $w_z+w_d=1$. We find the best pair on the CIFAR-10 and CIFAR-100 datasets with fewer than 1 M network parameters. Then, we apply these pairs to NAS on the ImageNet-1k dataset. All experimental results are provided in

Table 1. The results of top-1 accuracy on CIFAR-10 and CIFAR-100 with a difference pair of $(w_z,w_d)$ of Incorporated-Score and a model size of $N\le 1 M$. The bold represents the selected pairs of $(w_z,w_d)$.

Proxy	$({\mathit{w}}_{\mathit{z}},{\mathit{w}}_{\mathit{d}})$	CIFAR-10	CIFAR-100	FLOPs
Incorporated-Score-s	(0.9; 0.1)	96.10%	79.00%	280 M
	(0.8; 0.2)	96.86%	81.10%	309 M
	(0.7; 0.3)	96.86%	80.50%	472 M
	(0.6; 0.4)	97.21%	79.85%	592 M
	(0.5; 0.5)	96.63%	79.57%	315 M
	(0.4; 0.6)	96.74%	81.00%	572 M
	(0.3; 0.7)	96.80%	79.61%	692 M
	(0.2; 0.8)	97.08%	80.10%	630 M
	(0.1; 0.9)	96.71%	80.40%	634 M
Incorporated-Score-l	(0.9; 0.1)	96.66%	80.67%	112 M
	(0.8; 0.2)	96.50%	80.18%	411 M
	(0.7; 0.3)	96.09%	79.32%	472 M
	(0.6; 0.4)	96.56%	78.03%	592 M
	(0.5; 0.5)	96.29%	79.19%	315 M
	(0.4; 0.6)	96.21%	79.17%	572 M
	(0.3; 0.7)	96.53%	79.49%	692 M
	(0.2; 0.8)	96.39%	79.40%	630 M
	(0.1; 0.9)	96.33%	79.66%	634 M
Incorporated-Score-w/o-ls	(0.5; 0.5)	96.66%	75.97%	560 M
	(0.9999; ${10}^{-4}$)	96.33%	80.10%	411 M

. For Incorporated-Score-s, which uses the square-root function for normalization, we observe that the computational cost of network FLOPs tends to rise as $w_d$ increases. Our goal is to design a high-performance, lightweight network, which is why we choose $(w_z,w_d)=(0.8,0.2)$ for the Incorporated-Score-s proxy. For Incorporate-Score-l, $(w_z,w_d)=(0.9,0.1)$ is the best pair to achieve optimal results in terms of both performance and the network’s computational cost. In terms of Incorporated-Score-w/o-ls, it can be seen that the top-1 accuracy improves significantly on the CIFAR-100 dataset when the weight of $E_{score}$ is extremely small ($w_d={10}^{-4}$) relative to the weight of zen-score. This further proves that we need to normalize the $E_{score}$.

5.3. Comparison of Results between Incorporated-Score and Zen-Score on CIFAR-10 and CIFAR-100 Datasets

Table 2. Top-1 accuracy on CIFAR-10 and CIFAR-100 to compare the proposed Incorporated-Score on the best pair of weights $(w_z,w_d)$ and four other zero-shot proxies, including the SOTA zen-score. The bold represents the best accuracy.

Proxy	CIFAR-10	CIFAR-100
zen-score	96.20%	80.10%
grad	92.80%	65.40%
synflow	95.10%	75.90%
TE-Score	96.10%	77.20%
Incorporated-Score-l	96.66%	80.67%
Incorporated-Score-s	96.86%	81.10%
Incorporated-Score-w/o-ls	96.59%	75.97%

presents the detailed experimental results for the CIFAR-10 and CIFAR-100 datasets with a model size of $N\le$ 1 M. The proposed Incorporated-Score outperforms the original zen-score. In particular, the top-1 accuracy reaches $96.66\%$ and $96.86\%$ on the CIFAR-10 dataset when the Incorporated-Score-l and Incorporated-Score-s proxies are used, respectively. For the CIFAR-100 dataset, the top-1 accuracy exceeds that of zen-score by $80.67\%$ and $81.10\%$ when using the Incorporated-Score-l and Incorporated-Score-s proxies, respectively. The results demonstrate that the entropy of the network works effectively when incorporated into zen-score.

5.4. The Effectiveness of $E_{score}$ Normalization

To explain the effectiveness of normalizing $E_{score}$ when computing Incorporated-Score, we provide the model performance generated without any normalized function in Table 2. The experimental results show that normalization of the entropy affects the NAS performance because using Incorporated-Score with entropy normalization results in better performance than using Incorporated-Score without entropy normalization. As mentioned, the main reason is that the value of $E_{score}$ is much larger than that of zen-score, which leads to $E_{score}$ being dominant in Incorporated-Score compared to zen-score. Figure 3a,c show the maximum value of Incorporated-Score with normalization using logarithmic and square-root functions through the NAS stage. The normalized $E_{score}$ values are shown in Figure 3b,d. The figures show that the value of $E_{score}$ after normalization is at the same level as zen-score. Specifically, the value distribution is spread in the range of $[0,100]$. This avoids the dominant problem that occurs when we incorporate zen-score and $E_{score}$. Finally, Figure 3e shows the correlation between zen-score and Incorporated-Score, with Incorporated-Score exibiting a smoother curve than zen-score. For a fair comparison with zen-score, we also compare the computational efficiency of Incorporated-Score and zen-score in

Table 3. Time cost of Incorporated-Score and zen-score for the ZenNet-400M-imagenet model at a resolution of $224\times 224$ and ZenNet-1M-cifar at a resolution of $32\times 32$. The computational times of Incorporated-Score and zen-score for N images are reported in seconds, with the results averaged across 100 trials.

Proxy	Model	N	Time
Incorporated-Score	ZenNet-400M-imagenet	16	0.0345
	ZenNet-1M-cifar	16	0.0348
zen-score	ZenNet-400M-imagenet	16	0.0337
	ZenNet-1M-cifar	16	0.0346

. It can be easily seen that the computational time of Incorporated-Score is almost equal to that of zen-score. The main point is that we can find more robust, high-performance networks with insignificant increases in computational cost compared with zen-score.

5.5. Incorporated-Score for Lightweight Model on ImageNet Dataset

Table 4. Top- 1 accuracies of Incorporated-Score and zen-score proxies on the on ImageNet database in lightweight cases. The architectures that are searched for using the Incorporated-Score and zen-score proxies are denoted as EZenNet and ZenNet, respectively. The bold represents the best accuracy.

Model	Top1-Acc (%)	FLOPs
EZenNet-400M-SE-l	78.30	405 M
EZenNet-400M-SE-s	78.29	418 M
ZenNet-400M-SE [25]	78.00	410 M
EZenNet-600M-SE-l	79.64	610 M
EZenNet-600M-SE-s	79.73	609 M
ZenNet-600M-SE [25]	79.10	611 M

shows a comparison between the proposed Incorporated-Score and zen-score on the ImageNet dataset. As shown in Table 4, the models that are searched for using the proposed Incorporated-Score outperform the models that are searched for using zen-score with both 400 M and 600 M FLOPs.

To demonstrate the effectiveness of the proposed Incorporated-Score, we provide additional experimental results using other NAS approaches for lightweight cases, as shown in

Table 5. Top- 1 accuracies of lightweight models on ImageNet with a computational cost budge of FLOPs ≤ 1G. ‘*’: OFANet is trained from scratch. ‘+’: OFANet is trained using supernet parameters as an initialization.

Model	Resolution	Params	FLOPs	Top1-Acc
MobileNetV2-0.25	224	1.5 M	44 M	51.80%
MobileNetV2-0.5	224	2.0 M	108 M	64.40%
MobileNetV2-0.75	224	2.6 M	226 M	69.40%
MobileNetV2-1.0	224	3.5 M	320 M	74.70%
MobileNetV2-1.4	224	6.1 M	610 M	74.70%
DFNet-1	224	8.5 M	746 M	69.80%
RegNetY-200 MF	224	3.2 M	200 M	70.40%
RegNetY-400 MF	224	4.3 M	400 M	74.10%
RegNetY-600 MF	224	6.1 M	600 M	75.50%
RegNetY-800 MF	224	6.3 M	800 M	76.30%
OFANet-9 ms (+)	224	5.2 M	313 M	75.30%
OFANet-11 ms (+)	224	6.2 M	352 M	76.10%
OFANet-389 M (*)	224	8.4 M	389 M	76.30%
OFANet-482 M (*)	224	9.1 M	482 M	78.80%
OFANet-595 M (*)	224	9.1 M	595 M	79.80%
EfficientNet-B0	224	5.3 M	390 M	76.30%
EfficientNet-B1	240	7.8 M	700 M	78.80%
EfficientNet-B2	260	9.2 M	1.0G	79.80%
DNANet-a	224	4.2 M	348 M	77.10%
DNANet-b	224	4.9 M	406 M	77.50%
DNANet-c	224	5.3 M	466 M	77.80%
DNANet-d	224	6.4 M	611 M	78.40%
MnasNet-1.0	224	4.4 M	330 M	74.20%
Deep MAD-B0	224	5.3 M	390 M	76.10%
ZenNet-400 M-SE	224	5.7 M	410 M	78.00%
ZenNet-600 M-SE	224	7.1 M	611 M	79.10%
ZenNet-900 M-SE	224	13.3 M	926 M	80.80%
EZenNet-400 M-SE-l	224	7.1 M	405 M	78.30%
EZenNet-400 M-SE-s	224	7.2 M	418 M	78.29%
EZenNet-600 M-SE-l	224	9.1 M	610 M	79.64%
EZenNet-600 M-SE-s	224	9.6 M	609 M	79.73%
EZenNet-800 M-SE-s	224	12.9 M	801 M	80.10%

. The experiments were carried out with the following popular networks as baselines: (a) manually designed networks such as MobileNet-V2 [57]; (b) NAS-designed networks for fast inference on GPUs, i.e., DFNet [65] and RegNet [56], based on the search space optimization approach; (c) NAS-designed networks optimized for FLOPs, including one-shot methods such as OFANet [17], DNANet [18], and EfficientNet [50]; and (d) NAS-designed networks optimized for FLOPs, including RL methods such as MnasNet [48]. As shown in Table 5, EZenNet models searched for using the proposed Incorporated-Score outperform other models with similar FLOPs.

Table 6. Comparison of of NAS searching time. ’Top1-Acc’: The top-1 accuracy on the ImageNet-1k dataset. ’Method’: evolutionary algorithm (EA), gradient descent (GD), reinforcement learning (RL), zero-shot proxy (ZS), and progressive shrinkage (PS). The bold represents the best peformance.

Model	Method	Top1-Acc (%)	GPU Day
CARS-I [12]	EA	75.20	0.40
PC-DARTS [10]	GD	75.80	3.80
FBNetV2 [16]	GD	77.20	25.00
MetaQNN [6]	RL	77.40	96.00
TE-NAS [23]	ZS	74.10	0.20
OFANet [17]	PS	80.10	51.60
EZenNet-400M-SE-l	ZS	78.30	0.13
EZenNet-400M-SE-s	ZS	78.29	0.09
EZenNet-600M-SE-l	ZS	79.64	0.13
EZenNet-600M-SE-s	ZS	79.73	0.11
EZenNet-800M-SE-s	ZS	80.10	0.20

provides the NAS searching time of the proposed approach on ImageNet-1k compared to logarithms proposed in previous research, including CARS-I [12], PC-DARTS [10], FBNetV2 [16], MetaQNN [6], TE-NAS [23], and OFANet [17]. As shown in Table 6, the proposed method outperforms the methods proposed in previous research, with an efficient search time. In comparison, the proposed Incorporated-NAS reaches 80.1% accuracy in approximately 0.2 GPU days, while OFANet spends 51.6 GPU days to achieve the same accuracy.

5.6. Architecture Comparison

For further information, we compare architectures generated using the proposed Incorporated-Score proxy and the zen-score proxy, as shown in

Table 7. Architecture comparison of EZenNet-1.0M-l (orange) vs. ZenNet-1.0M (black) for the CIFAR-10 and CIFAR-100 datasets.

Block		Kernel		Input		Output		Stride		Bottlenecks		# Layers
Conv	Conv	3	3	3	3	88	24	1	1	-	-	1	1
Btn	Res	7	5	88	24	120	88	1	1	16	8	1	1
Btn	Btn	7	7	120	88	192	304	2	2	16	16	3	5
Btn	Res	5	5	192	304	224	48	1	1	24	8	4	3
Btn	Btn	5	7	224	48	96	304	2	1	24	16	2	4
Btn	Btn	3	5	96	304	168	80	2	2	40	32	3	2
Btn	Btn	3	5	168	80	112	256	1	2	48	40	3	1
Conv	Conv	1	1	112	256	512	232	1	1	-	-	1	1

,

Table 8. Architecture comparison of EZenNet-1.0M-s (blue) vs. ZenNet-1.0M (black) for the CIFAR-10 and CIFAR-100 datasets.

Block		Kernel		Input		Output		Stride		Bottlenecks		# Layers
Conv	Conv	3	3	3	3	88	56	1	1	-	-	1	1
Btn	Res	7	7	88	56	120	48	1	1	16	8	1	5
Btn	Btn	7	3	120	48	192	200	2	2	16	32	3	3
Btn	Btn	5	5	192	200	224	160	1	1	24	24	4	3
Btn	Btn	5	7	224	160	96	624	2	2	24	16	2	4
Btn	Btn	3	3	96	624	168	48	2	2	40	40	3	1
Btn	-	3	-	168	-	112	-	1	-	48	-	3	-
Conv	Conv	1	1	112	48	512	304	1	1	-	-	1	1

,

Table 9. Architecture comparison of EZenNet-400M-SE-l (orange) vs. ZenNet-400M-SE (black) for the ImageNet dataset.

Block		Kernel		Input		Output		Stride		Bottleneck		Expansion		# Layers
Conv	Conv	3	3	3	3	16	32	2	2	-	-	-	-	1	1
MB	MB	7	7	16	32	40	40	2	2	40	40	1	1	1	1
MB	MB	7	7	40	40	64	64	2	2	64	40	1	2	1	1
MB	MB	7	7	64	64	96	128	2	2	96	176	4	2	5	1
MB	MB	7	7	96	128	224	256	2	2	224	152	2	4	5	3
-	MB	-	7	-	256	-	104	-	1	-	152	-	6	-	4
Conv	Conv	1	1	224	104	2048	2048	1	1	-	-	-	-	1	1

,

Table 10. Architecture comparison of EZenNet-400M-SE-s (blue) vs. ZenNet-400M-SE (black) for the ImageNet dataset.

Block		Kernel		Input		Output		Stride		Bottleneck		Expansion		# Layers
Conv	Conv	3	3	3	3	16	40	2	2	-	-	-	-	1	1
MB	MB	7	7	16	40	40	80	2	2	40	16	1	1	1	1
MB	MB	7	7	40	80	64	80	2	2	64	48	1	1	1	1
MB	MB	7	7	64	80	96	336	2	2	96	112	4	1	5	1
MB	MB	7	7	96	336	224	360	2	2	224	360	2	1	5	3
-	MB	-	7	-	360	-	360	-	1	-	360	-	1	-	4
Conv	Conv	1	1	224	360	2048	2048	1	1	-	-	-	-	1	1

,

Table 11. Architecture comparison of EZenNet-600M-SE-l (orange) vs. ZenNet-600M-SE(black) for the ImageNet dataset.

Block		Kernel		Input		Output		Stride		Bottleneck		Expansion		# Layers
Conv	Conv	3	3	3	3	24	16	2	2	-	-	-	-	1	1
MB	MB	7	7	24	16	48	32	2	2	16	32	1	4	1	1
MB	MB	7	7	48	32	72	72	2	2	16	40	2	4	1	1
MB	MB	7	7	72	72	96	304	2	2	24	96	6	2	5	1
MB	MB	7	7	96	304	192	360	2	2	24	360	4	1	5	3
-	MB	-	7	-	360	-	176	-	1	40	240	-	4	-	5
Conv	Conv	1	1	192	176	2048	2048	1	1	-	-	-	-	1	1

and

Table 12. Architecture comparison of EZenNet-600M-SE-s (blue) vs. ZenNet-600M-SE (black) for the ImageNet dataset.

Block		Kernel		Input		Output		Stride		Bottleneck		Expansion		# Layers
Conv	Conv	3	3	3	3	24	16	2	2	-	-	-	-	1	1
MB	MB	7	7	24	16	48	24	2	2	16	64	1	2	1	1
MB	MB	7	7	48	24	72	120	2	2	16	120	2	1	1	1
MB	MB	7	7	72	120	96	192	2	2	24	144	6	2	5	1
MB	MB	7	7	96	192	192	320	2	2	24	224	4	2	5	4
-	MB	-	7	-	320	-	384	-	1	40	384	-	1	-	5
Conv	Conv	1	1	192	384	2048	2048	1	1	-	-	-	-	1	1

, on the CIFAR-10, CIFAR-100, and ImageNet datasets in ResNet and MobileNet search spaces.

In the tables, ‘Conv’ represents the standard convolution layer followed by Batch Normalization (BN) and a RELU; ‘Res’ is the residual block designed in ResNet-18; ‘Btn’ is the residual bottleneck block designed in ResNet-50; ‘MB’ is the MobileBlock applied in MobileNet and EfficientNet; ‘Input’ and ‘Output’ are short for the count of input and output channels, respectively; and ‘Bottleneck’ is the number of bottleneck channels. It can be seen that EZenNet models, which are generated using the proposed Incorporated-Score, tend to expand horizontally and are less deep than ZenNet models. For example, when the computation cost is 400 M FLOPs, the number of layers of EZenNet models is 12, whereas that of the ZenNet model is 14. This proves that the entropy value component helps control over-deep network generation and optimizes the networks for the width dimension.

6. Conclusions

In this paper, we propose Incorporated-Score, which is an efficient proxy for zero-shot neural architecture search in the design of high-performance deep image recognition networks based on zen-score and entropy. We conducted experiments to verify that entropy plays a role as the optimal factor in zen-score, and EZenNet models automatically designed by Incorporated-NAS outperformed the state-of-the-art NAS models in top-1 accuracy on the CIFAR-10 and CIFAR-100 datasets with the same budget. Moreover, EZenNet outperforms ZenNet on the the ImageNet dataset with lightweight instances. In the future, we will further investigate this proxy for other deep learning problems such as object detection and natural language processing.