Smish: A Novel Activation Function for Deep Learning Methods

Abstract

Activation functions are crucial in deep learning networks, given that the nonlinear ability of activation functions endows deep neural networks with real artificial intelligence. Nonlinear nonmonotonic activation functions, such as rectified linear units, Tan hyperbolic (tanh), Sigmoid, Swish, Mish, and Logish, perform well in deep learning models; however, only a few of them are widely used in mostly all applications due to their existing inconsistencies. Inspired by the MB-C-BSIF method, this study proposes Smish, a novel nonlinear activation function, expressed as $f\left(x\right)=x·tanh\left[\ln \left(1+sigmoid\left(x\right)\right)\right]$, which could overcome other activation functions with good properties. Logarithmic operations are first used to reduce the range of sigmoid(x). The value is then calculated using the tanh operator. Inputs are ultimately used to multiply the previous value, thus exhibiting negative output regularization. Experiments show that Smish tends to operate more efficiently than Logish, Mish, and other activation functions on EfficientNet models with open datasets. Moreover, we evaluated the performance of Smish in various deep learning models and the parameters of its function $f\left(x\right)=\alpha x·tanh\left[\ln \left(1+sigmoid\left(\beta x\right)\right)\right]$, and where α = 1 and β = 1, Smish was found to exhibit the highest accuracy. The experimental results show that with Smish, the EfficientNetB3 network exhibits a Top-1 accuracy of 84.1% on the CIFAR-10 dataset; the EfficientNetB5 network has a Top-1 accuracy of 99.89% on the MNIST dataset; and the EfficientnetB7 network has a Top-1 accuracy of 91.14% on the SVHN dataset. These values are superior to those obtained using other state-of-the-art activation functions, which shows that Smish is more suitable for complex deep learning models.

1. Introduction

The principle of deep learning networks is that input is passed from one neuron to the next via an activation function, and the process is repeated until the output layer is reached. By repeating the aforementioned process, a linear relationship is transformed into a nonlinear relationship by using the activation function. Thus, the activation function provides deep learning networks with sufficient power to derive complex calculations and obtain state-of-the-art results in various fields. An activation function is used to improve representation ability via nonlinear operations in convolutional neural networks (CNNs). If an inappropriate activation function is selected, the residual is considerably smaller after multilayer convolutions in CNNs, requiring the termination of the training process [1]. Currently, activation functions are mainly divided into linear activation functions and nonlinear activation functions. Nonlinear activation functions can be further subdivided into the nonlinear monotonic and nonlinear nonmonotonic types [2]. The former mainly consists of simple linear and threshold functions. These functions include the step, sign, and identity functions, which are the common early linear activation functions [3,4,5,6]. The most typical and simplest linear function is the activation function [7]. However, if the output of the node is equal to the input, the activation function has no classification ability. Linear activation functions are not widely applied in CNNs because of the discontinuous characteristics of their derivatives. In deep learning networks, nonlinear activation functions—Sigmoid, ReLU, Swish, Mish, and Logish—are frequently used [8,9]. Sigmoid maps all values to (0, 1), which is associated with the vanishing gradient problem. To address this concern, the tanh activation function is proposed [10], although it does not eliminate the aforementioned problem in deep neural networks. The activation function SoftPlus provides greater enhancement, allowing the ReLU function to further boost deep learning networks, hence its wide application used in different fields [11,12,13,14]. ReLU exhibits reduced saturation, sparsity, efficiency, and ease of use and thus can alleviate the vanishing gradient problem. However, its negative value directly returns to zero; consequently, the network loses a certain amount of valid information. Such a disadvantage has prompted the need to develop new methods, including the Hard Sigmoid and Hard Tanh activation functions, which can easily perform calculation tasks and effectively realize network learning. Meanwhile, Leaky ReLU assigns a small slope of 0.01 to negative input, and ELU contains a negative exponential term. With these approaches, the presence of silent neurons is reduced, and negative gradient learning is enabled [15,16].

In addition to nonlinear monotonic activation functions, nonlinear nonmonotonic activation functions also perform efficiently in deep learning networks. Nonlinear nonmonotonic functions, including Swish and Mish, are broadly used. Swish has an advantage in big data and deeper complex networks and thus performs more efficiently than others [17]. Proposed by Misra et al., Mish is also a nonmonotonic differentiable function, which has a lower bound but no upper bound [18]. Mish is almost smooth at any point on the curve, allowing the transformation of more valid information into the model to improve accuracy and generalization performance. In 2021, Hegui Zhu et al. proposed Logish activation, which is similar to Swish but exhibits better performance and Top-1 accuracy [2]. Proposed by Jaafari et al., PRenu activation function uses hidden layer incorporating with a chosen activation such as Relu [19]. AIS activation proposed by Wang Z et al. disposes positive and negative values separately, and its curve distribution is not very different from Mish [20]. In the data processing of the MB-C-BSIF method proposed by Adjabi et al., the input is normalized firstly and then nonlinear filtered, which makes it ideal for deep learning methods [21].

Although great success has been made by the above activation functions in some deep learning networks, the classification results are unsatisfactory in more complex networks in our study. In order to improve the accuracy rate of classification results, we designed a new activation function named Smish to solve the previously mentioned problems in deep learning networks, the aforementioned characteristics being present not only to ensure negative activation and derivative values but also to maintain partial sparsity and a regularization effect for negative inputs. Similar to Mish and Logish, Smish is a differentiable function with a lower bound and no upper bound, expressed as

$$f\left(x\right)=x·tanh\left[\ln \left(1+sigmoid\left(x\right)\right)\right],$$

(1)

Its variant is given by

$$f\left(x\right)=\alpha x·tanh\left[\ln \left(1+sigmoid\left(\beta x\right)\right)\right],$$

(2)

where the logarithmic operation is used to reduce the range of the sigmoid(x) function value and is calculated with the tanh function, ensuring the smoothness and stability of the normalized curve. Meanwhile, inspired by Mish, x is multiplied with the value that renders the negative output regularized, and the positive output is translated into simple linearization. The results also confirm that Smish is more suitable than Mish and Logish for learning complex networks. The innovations provided by this study are as follows:

1. A new monotonic nonlinear activation function named Smish is proposed.

2. Smish provides a higher learning accuracy, compared with Logish, Mish, Swish, and ReLu, all of which are used in several EfficientNet models.

3. Smish performs better than other functions in classification in several open datasets.

Other parts of the paper are organized as follows: Section 2 briefly introduces Mish and Logish. Section 3 presents the Smish activation function and illustrates its major properties. In Section 4, we analyze the performance of the Smish nonmonotonic functions with parameter settings. In Section 5, the Smish algorithm is used in image classification, and other common activation functions on EfficientnetBx are employed to evaluate performance. Section 6 summarizes the main results.

2. Related Work

2.1. Mish

A new activation function referred to as Mish has recently been widely used in deep-learning methods and is described as

$$f\left(x\right)=x·tanh\left[\ln \left(1+e^x\right)\right],$$

(3)

The derivative formula of Mish is represented as

$$f^{\prime }\left(x\right)=\frac{2}{\frac{1}{{(e^x+1)}^2}+1}+\frac{4x{e}^x}{{\left(e^x+1\right)}^3{\left(\frac{1}{{\left(e^x+1\right)}^2}+1\right)}^2}-1,$$

(4)

As shown in Figure 1, the Mish curve has no upper bound. Thus, positive values for Mish functions can reach any height similar to that of Swish. Theoretically, in ReLU, since the gradient of this function is zero when it is negative, “necrosis” may occur during training, and the derivative with a small negative value produces a better gradient flow than that of the derivative without response. If the input is reduced, information may be lost, rendering the activation value randomly dependent on the input. From 2013 to 2021, studies were conducted to improve an activation function for the occurrence of “necrosis” caused by ReLu. Lu et al. proposed the use of randomized asymmetric weight initialization, leaky ReLus, ELU, and so on [22,23], which allow learning the negative slope, in contrast to leaky ReLU and tanh-ReLu [24,25].

Dan Hendrycks et al. introduced the concept of random regularity into the Gaussian error linear unit (GELU). It is similar to Dropout, which generates output via random multiplication by 0 or 1 [26]. Mish exhibits a random pattern similar to that of GeLU and has the advantages of being unsaturated and nonmonotonic and can easily replace ReLU. All the characteristics provide the Swish activation function with excellent response normalization. On FastAI open datasets, Mish combined with a Ranger optimizer achieves enhanced accuracy [27]. In Mish, the derivative of a continuous order is infinite, which is an advantage over ReLU. The continuous order of ReLU is 0, indicating that it is not continuously differentiable, which can potentially present problems in gradient-based optimization.

2.2. Logish

Logish activation function is described by

$$f\left(x\right)=x·\ln \left(1+sigmoid\left(x\right)\right),$$

(5)

$$f^{\prime }\left(x\right)=\ln \left(\frac{1}{e^{-x}+1}+1\right)+\frac{x·e^{-x}}{{\left(e^{-x}+1\right)}^2·\left(\frac{1}{e^{-x}+1}+1\right)},$$

(6)

As shown in Figure 1, the curve of Logish closely resembles that of Mish; no upper limit to positive values exists. Theoretically, a negative value near 0 produces a continuous differentiable derivative flow that can maintain partial sparsity. A logarithmic operation is then used in the sigmoid function to reduce the range of values and generate smooth and stable curves. The negative values are normalized, and the positive values tend to be linear. As X decreases, the input is more likely to be abandoned. Thus, in Logish, the uncertainty is preserved by deleting parts of the input value, thus increasing its dependence on other inputs [2]. Misra concluded that the smoother the activation function, the greater the ability to efficiently acquire information, the higher the network accuracy, and the better the generalization ability [11].

Additionally, for the purpose of improving the accuracy and deep learning convergence speed, the PRenu activation function uses hidden-layer incorporating with a chosen activation, which is nearly similar to Relu. However, the difference between them was that PRenu multiplies a value with a received gradient for all positive values in its back-propagation [19]. In the field of industry engineering, in the task of that the fitted model can be applied to, AIS activation consists of positive and negative regions. In the former region, the exponential and quadratic function are combined to alleviate the problem of deviation, and the linear function is added to make the positive region smoother and overcomes the problem of gradient vanishing. In the negative region, the cubic function structure is applied to solve the negative region problem and accelerate the convergence of the deep learning networks [20].

However, in complex deep learning networks, the above activation functions could not overcome the problems of gradient vanishing in negative regions, and make the deviation of networks increased, slow down the network fitting and make the feature extraction ability of the model decreased. Inspired by the MB-C-BSIF method, in the data processing stage of which the data is normalized firstly and then filtered by coefficients, Smish, which is proposed, could basically solve the above problems, and the characteristics of Smish could meet the required features [21]. The conclusions drawn from the aforementioned studies are presented as follows:

1. Mish and Logish are close to the linear function $f\left(x\right)=\alpha x$ when $x>0$ but close to 0 when $x<0$. These characteristics render the network more stable and directional propagation calculation easier.

2. The derivatives when $x<0$ are not always 0, which effectively avoids gradient death.

3. When $x>0$, the structure is similar to that of Mish; thus, the neural network exhibits sparsity and reduces computational complexity.

3. Proposed Smish

3.1. Construction of Smish

This study proposes a novel activation function called Smish, which inherits the nonmonotonic properties of the Logish function. The function is particularly suitable for deep learning networks.

$$f\left(x\right)=x·tanh\left[\ln \left(1+sigmoid\left(x\right)\right)\right],$$

(7)

$$f^{\prime \left(x\right)}=\frac{e^x·\left(15·e^{3x}+\left(8x+28\right)e^{2x}+\left(12x+18\right)e^x+4x+4\right)}{{\left(5e^{2x}+6e^x+2\right)}^2},$$

(8)

The Smish function is consistent with the Logish function in the early stages: $sigmoid\left(x\right)$ is used to reduce the range of values, and the logarithm operation is applied to obtain a smooth curve and a flat trend [9,12]. Smish then multiplies its $tanh$ operation by $x$ simultaneously, thus exhibiting the regularization ability for negative inputs; by contrast, the positive values tend to become a simpler linear expression. The Smish curve itself and its derivatives are shown in Figure 2.

Similar to Mish and Logish, Smish has a lower bound and no upper bound, and its range is $\left[-0.25,+\infty \right)$. The minimum value of Smish is approximately $-0.25$, and the input value is $-1.3945$. It is prompted by self-gating in the long short-term memory of deep recurrent convolution, the advantage of this pattern is its convenience of replacing activation functions such as ReLU because of the transformation of input values [28]. When $x<-12.427,$ for a negative value of $x$, the value is as same as ReLU. Smish activation functions can be easily implemented using some frameworks by customizing an activation layer. For example, in the framework of TensorFlow, it is Keras $x\ast K.Tanh \left(K.log\left(1+K.sigmoid\left(x\right)\right)\right)$ while in Torch it is $x \ast torch.tanh\left(torch.log\left(1+torch.sigmoid\left(x\right)\right)\right)$; it is more suitable to set the learning rate as le-4 in experiments. Formulas (7) and (8) and Figure 1 and Figure 2 indicate that the simplified formula for Smish and Mish is exponential. As x approaches infinity, Mish is closer to $x$; by comparison, Smish is closer to $\frac{2^2-1}{2^2+1}x$.

3.2. Approximate Linear Transformation

As the depth of the network model increases, the value of the output does not change significantly with that of the nearly linear activation function. Owing to the nearly linear transformation characteristics of the network, it is stable and convenient for gradient backpropagation. Logish is then proved to be approximately 0.6$x$.

$$f\left(x\right)=x·\left(\frac{2}{1+e^{-2·\ln \left(1+\frac{1}{1+e^{-x}}\right)}}-1\right)=x·\frac{{\left(1+\frac{1}{1+e^{-x}}\right)}^2-1}{{\left(1+\frac{1}{1+e^{-x}}\right)}^2+1}=\frac{3}{5}x=0.6x,$$

(9)

Smish reduces computational complexity by using the aforementioned calculation. The derivative of Smish for a positive $x$ is equal to $0.6x$. Whereas the derivatives of Mish and ReLU tend to be equal to 1, Smish is lower than Logish and thus exhibits a more gradual growth trend, compared with other functions. Thus, Smish performs better in deep learning fields.

3.3. Nonmonotonicity

A good activation function should not induce the disappearance of the gradient, and a small number of negative values are allowed to exert regularization effects [29]. In Figure 2, Smish satisfies all properties of good activation functions. Therefore, Smish is a good choice for deep learning networks. Moreover, the nonmonotonicity of Smish ensures the stability of negative training and improves the performance for expression. On the basis of Cauchy’s theorem, given that $f \left(-2.5\right)=f \left(-0.63\right)$, there must be $x,$ $x\in \left[-2.5,-0.63\right], f\prime \left(x\right)=0, x=-1.3945$. Smish decreases monotonically in $\left(-\infty ,-1.3945\right)$ and increases monotonically in $\left(-1.3945,-\infty \right)$. Therefore, Smish is nonmonotonic in$\left(-\infty ,+\infty \right)$, which can improve the ability of network learning and gradient transformation.

4. Analysis of Hyperparameter Tuning for Smish

This section focuses on the superiority of the Smish activation function by adjusting several classical super parameters. These parameters include batch sizes, learning rates, dropout, optimizers, and regularization methods. To better reflect the performance of the activation function, the model adopts 20 convolution layers and 2 pooling layers. Moreover, EfficientNet is employed as our network model for fine-tuning parameters in the experiments. All experiments are trained and evaluated on the CIFAR10 datasets [30,31].

4.1. Analysis of the Number of Layers

To observe the effect of network layers with the Smish activation function on EfficientNet, we trained fully connected networks at different depths on MNIST datasets with 500 neurons per layer under similar conditions. Deep residual models were not used because depth only slightly affects the training result [32]. In this section, BatchNorm was employed to reduce reliance on initialization, and the dropout rate was set to 25% [33]. The network was optimized by stochastic gradient descent (SGD), and the batch size was 128, which achieved the highest accuracy, as proved in a subsequent section [34]. For a fair comparison, the same learning rate was used to train each model with ‘le-4’. In the experiment, all other activation functions maintained nearly identical levels of test accuracy for the 15-layer network. With a gradual increase in the number of layers, the testing accuracy of ReLU sharply declined from 15, and that of Swish markedly decreased from 29. Regardless, Smish outperformed the other four activation functions in simple and complex networks (Figure 3).

4.2. Analysis of Batch Sizes

The batch size parameter refers to the number of training samples selected in every training epoch, which largely influences the results of model training. Generally, as batch_size increases, the gradient descent changes more accurately; however, gradients can easily disappear. By contrast, as the situation is reversed, the training time becomes longer. In our study, the conditions were set as follows: epoch, 100 on CIFAR10; learning rate, 0.0001; dropout rate, 0.8. All network parameters were kept constant, and the batch size was set to 16, 32, 64, 128, 256, 512, and 1024. In Figure 4, the highest accuracy of all five activation functions occurs when the batch size is set to 128, where Smish obtained the best performance on CIFAR10. Moreover, Smish is higher than the other activation functions during the entire training process for batch size.

In addition, as shown in Figure 5, time is also an important factor for deep learning samples. With Smish as the activation function, when the batch size was set to 128, the least time and highest accuracy were attained; when the batch size was set to other values, more time was consumed. Thus, in subsequent experiments, batch_size was set to 128.

4.3. Analysis of learning rates

The learning rate plays an important role in the optimization of deep learning networks, and the value for the learning rate is always inconclusive. The learning rate of the training sample was selected from $\mathrm{le}$-1 to $\mathrm{le}$-7 for testing, and the CIFAR10 dataset was trained. As shown in Figure 6, ReLU initially exhibits the highest accuracy, but overall, Smish shows the highest accuracy (0.91). Although Logish performs best when the learning rate was $\mathrm{le}$-3, its performance remained lower than that of Smish. Subsequently, as the learning rate decreased, the testing accuracy levels of all functions decreased simultaneously, with ReLU exhibiting the lowest accuracy. These results indicate that Smish significantly outperformed other functions at higher learning rates. In subsequent experiments, the learning rate was set to le-4.

4.4. Analysis of Different Dropout Rates

The dropout layer processes some features of the hidden layer to regularize and suppress overfitting. Nine values ranging from 0.1 to 0.9 were selected. The test accuracy levels of the five activation functions are shown in Figure 7. Overall, dropout only slightly affected the accuracy of the model. Smish performed more efficiently than other activation functions, achieving the highest accuracy at a dropout of 0.2. The synergistic effect between different features of the effective link exhibited high robustness.

4.5. Analysis of Different Optimizers

In deep learning, an appropriate optimizer is selected, and the super parameters are adjusted to extract information from the input to allow it to adapt to the output value. The aim is to identify the optimal solution. Thus, choosing the appropriate optimizer is important in network training. In the current study, we selected Adadelta [35], Adagrad [36], Adam [37], RMSprop [38], and SGD [39] optimizers. As shown in Figure 8, Smish achieves an accuracy of 0.9 with Adam and 0.83 with RMSprop, which are significantly superior to the accuracy rates of the other activation functions. Logish exhibited an accuracy of 0.84 with Adam, and several other activation functions showed accuracy rates higher than 0.8 with Adam. Thus, with any optimizer, Smish performed better than any other activation function.

4.6. Analysis of Different Parameters

The evaluation of the performances of Smish and other compared functions in a deep learning model is discussed in previous sections. This section verifies the influence of $\alpha$ and $\beta$ parameters in $f\left(x\right)=\alpha x·tanh\left[\ln \left(1+sigmoid\left(\beta x\right)\right)\right]$ on the experimental model.

Table 1. Training time, accuracy, and loss with the Smish activation function at different values of $\alpha$ and $\beta$.

$\mathit{\alpha }$	$\mathit{\beta }$	Train Time (ms)	Loss	Val_Accuracy
20	1	1664.87985	2.093894768	0.317399998
15	1	1657.337511	2.084924269	0.326699999
10	1	1688.491895	2.054765987	0.356000003
5	1	1683.588445	1.366328526	0.626699996
0.1	1	1697.313161	1.278249788	0.790999961
0.01	1	1719.354016	1.111771631	0.818499994
0.005	1	1708.812285	2.102614927	0.299999994
0.001	1	1718.249439	2.102625656	0.299999994
1	20	1722.055284	1.059448051	0.717399988
1	15	1723.015788	1.032156754	0.723999621
1	10	1717.969835	1.057105231	0.755299997
1	5	1727.269962	1.081895041	0.851999986
1	1	1724.575549	0.870445440	0.909814782
1	0.5	1717.444711	1.033060956	0.833099973
1	0.1	1725.162607	0.92743938	0.815499973
1	0.05	1730.499959	1.000445652	0.791300011
1	0.01	1731.330632	1.108899283	0.755499971

shows the training time of the selected model and the loss value of the test samples and verification accuracy of different values of $\alpha$ and $\beta$.

As shown in Table 1, when $\beta =1$ and $\alpha >1$, the smaller the value of $\alpha$, the higher the accuracy. When $0<\alpha <1$, the accuracy decreases with a decrease in $\alpha$. When $\alpha =1$ and $\beta >1$, the accuracy was inversely proportional with the increase in $\beta$. When $0<\beta <1$, the smaller the value of $\beta$, the lower the accuracy. The highest accuracy was achieved when both $\alpha$ and $\beta$ were set to one; thus, in subsequent experiments, both $\alpha$ and $\beta$ were assigned to one.

5. Experimental Results of Open Datasets

In this chapter, the performance of Smish relative to other activation functions is evaluated with three open datasets and several EfficientNet models. Notably, our evaluation is based on test samples rather than training samples.

5.1. Datasets and Experimental Settings

All models were trained and tested on MNIST, CIFAR10, and SVHN datasets. MNIST datasets contained 60,000 training samples and 10,000 test samples, which were sorted into 10 categories—that is, “0–9”, ten Arabic numerals. The CIFAR10 dataset contained 50,000 training samples and 10,000 test samples, and these photos were RGB data images with three $32\ast 32$ channels. The SVHN datasets were derived from the Street View House Number (SVHN) dataset, and each image contained a set of Arabic numerals ‘0–9’. As shown in the image below, the training samples contain 73,257 digits, 26,032 test digits, and 531,131 additional digits. All experiments were performed on TensorFlow GPU using Python 3.7 with the following parameters: batch_size, 128; learning_rate, 0.0001; epoch, 100; and optimizer, Adam. The training process is as follows:

First, we determined from the experiments that data enhancement exerted no considerable effect on the training results of several models but increased computational cost. Thus, in the experiment, we did not enhance data for initialization. Second, we chose four EfficientNet models, considering that EfficientNet has been proved in numerous literature reviews to have a greater advantage in image deep learning, compared with DensenNet, ResNet [18], EfficientNetB3, EfficientNetB5, EfficientNetB6, and EfficientNetB7 [40]. EfficientNetB0 to EfficientNetB7 included seven blocks. The most critical aspect of any network was its stem, which was determined before subsequent experiments were conducted. This structure was common in all eight models and the last layer. The blocks also varied in the number of subblocks, which could increase as we moved from EfficientNetB0 to EfficientNetB7. The differences between all models gradually increased the number of subblocks.

5.2. Classification Result on CIFAR10

The classification results on CIFAR10, obtained via the activation functions ReLU, Swish, Mish, Logish, and Smish, are listed in

Table 2. Accuracy of EfficientNet models achieved via the five activation functions on CIFAR10.

Model	EfficientNetB3	EfficientNetB5	EfficientNetB6	EfficientNetB7
ReLU	0.812400	0.818400	0.838900	0.837200
Swish	0.820700	0.840400	0.846900	0.856600
Mish	0.823200	0.837800	0.842700	0.853000
Logish	0.833500	0.843500	0.850700	0.858000
Smish	0.841000	0.854900	0.851000	0.859400

. The first four functions were trained on EfficientNetB3, B5, B6, and B7 networks, respectively. We could conclude that Smish consistently outperformed the other four activation functions in any model on CIFAR10. In EfficientNetB3, Smish achieved an improvement of 0.01 relative to Logish and 0.018 relative to Mish. The improvement in EfficientNetB5 and EfficientNetB7 was not significant, but Smish outperformed the others. The Smish classification accuracy of 0.85 in EfficientNetb5–7 proved that it could be widely used in complex models, compared with other activation functions. As indicated in Figure 9 and Figure 10, Smish had the lowest value, compared with the other activation functions, and consistently attained the highest accuracy in any epoch.

5.3. Classification Results on MNIST

We also conducted experiments on MNIST datasets to evaluate the performance of several activation functions. As shown in

Table 3. Accuracy of EfficientNet models achieved via the five activation functions on Mnist.

Model	EfficientNetB3	EfficientNetB5	EfficientNetB6	EfficientNetB7
ReLu	0.564600	0.128999	0.986900	0.990800
Swish	0.912500	0.752599	0.987700	0.991900
Mish	0.397300	0.189899	0.987800	0.992600
Logish	0.925700	0.784300	0.990100	0.992800
Smish	0.977800	0.966500	0.998900	0.993500

, Smish performs better than the other four activation functions. In EfficientNetB3, Smish improved by 0.05 relative to Logish and by 0.5 relative to Mish. The EfficientNetB6 and EfficientNetB7 results on CIFAR10 were similar. Thus, for more complex models, an increase in parameters was not markedly effective; however, the effect was better on Smish than on the four other activation functions, as determined from its accuracy of 0.99. Smish also consistently outperformed the other four activation functions in loss and accuracy epochs, respectively (Figure 11 and Figure 12).

5.4. Classification Results on SVHN

The experimental accuracy levels of the five functions on the SVHN dataset were ultimately compared. Among the four EfficientNet models, Smish achieved the largest improvement in classification (

Table 4. Accuracy of EfficientNet models achieved via the five activation functions on SVHN.

Model	EfficientNetB3	EfficientNetB5	EfficientNetB6	EfficientNetB7
ReLu	0.829095	0.854179	0.851605	0.195874
Swish	0.860787	0.882683	0.844729	0.195874
Mish	0.864897	0.889405	0.830670	0.775738
Logish	0.863745	0.890174	0.899739	0.848494
Smish	0.874232	0.893977	0.901467	0.911417

). In EfficientNetB3, Smish attained an improvement of 0.05 relative to ReLU and 0.05 relative to Logish. In EfficientNetB7, Smish improved in accuracy by 0.06 relative to Logish, 0.7 relative to ReLU and Swish, and 0.2 relative to Mish. Smish outperformed the other activation functions (Figure 13 and Figure 14).

6. Conclusions

In our study, we proposed Smish, a deep learning activation function and its variation to improve the classification performance of deep learning models. Smish is expressed as $f\left(x\right)=x·tanh\left[\ln \left(1+sigmoid\left(x\right)\right)\right]$, which is a smooth nonmonotonic function with a lower bound but without an upper bound. These properties of Smish enable deep learning networks to benefit negative representation, which achieves better performance. Simulations and experiments confirmed the effectiveness of Smish, which outperformed ReLu, Swish, Mish, and Logish in EfficientNetBx. Smish also performed more efficiently when learning rates were set to higher values. Therefore, Smish is a better alternative to ReLu than Mish and Logish.

Obviously, the complexity of Smish is higher than other compared activation functions, so there are some limitations in the lightweight models, which will be solved in the future research. Probably the most improvement of this study would be the consideration of restricting calculation by subdividing the interval in different conditions so as to expand its scope of applications [41].