Frac-Vector: Better Category Representation

Abstract

For this paper, we proposed the fractional category representation vector (FV) based on fractional calculus (FC), of which one-hot label is only the special case when the derivative order is 0. FV can be considered as a distributional representation when negative probability is considered. FVs can be used either as a regularization method or as a distributed category representation. They gain significantly in the generalization of classification models and representability in generative adversarial networks with conditions (C-GANs). In image classification, the linear combinations of FVs correspond to the mixture of images and can be used as an independent variable of the loss function. Our experiments showed that FVs can also be used as space sampling, with fewer dimensions and less computational overhead than normal distributions.

1. Introduction

Fractional calculus (FC) means that the order of derivatives or integrals in a calculation can be real or complex and not just integers. The theory of FC goes back to Leibniz’s note to L’Hospital in which the derivative order of one half was discussed [1,2]. In recent decades, applications involving FC have already become an emerging and promising area in many branches of science and engineering because of its higher accuracy and more realistic description of the world than integer-order calculus [3]. In image processing, two main styles of algorithms are represented by methods using FC, namely, traditional algorithms and artificial neural networks’ (ANNs) tuning parameters. Traditional algorithms include fractional masks, fractional filtering, and fractional transformations. The mask is an important tool for simplifying the convolution of differential or integral operations and plays an important role in many types of image processing such as image blurring, edge detection, and image enhancement. The Laplace operator is a prime example of a 3 × 3 mask approximation of the second derivative used to detect edges [4]. Fractional-order masks offer greater expressiveness and accuracy than integer-order masks. Pu et al. [5] designed a class of fractional differential masks for texture enhancement, obtaining the performance of preserving continuous contour features and improving texture details [6,7]. For image filtering, setting the filter as a fractional-order filter gives more freedom. Zhang et al. proposed an adaptive fractional differential filter based on a rough set and particle swarm optimization (PSO) algorithm [8], enhanced by which images retain a clear edge and rich texture details. Fractional-order derivatives can also be applied to multi-focus image fusion to reflect the clarity of the image; this method has been experimentally proven to outperform the state-of-the-art methods for multi-focus image fusion [9]. In image fusion, the fractional gradient can also be used to construct image decomposition filters that not only preserve edges but also reduce the influence of the infrared background so that the fused image better matches human visual perception even in dim light [10]. Yan et al. [11] proposed an adaptive fractional multi-scale edge-preserving decomposition to fuse infrared (IR) and visible (VIS) images, extract the target, and preserve background information. New filters based on fractional derivatives have also been utilized for edge detection with much more accurate and efficient edge presentation [12,13]. It is well known that the Fourier transform is commonly used in image processing. Namias applied the idea of FC to the Fourier transform and introduced the fractional transform [14]. Since then, the fractional transform has been a hot research topic in many fields because of the benefit that it is easy to control the transformation function by varying the parameter of fractional order [14,15]. Fractional transform is widely used in image registration [16,17], image encryption [18,19,20], image compression, and other image processing applications. Since 2006, artificial neural networks have become a popular research topic; since then, ANNs involving FC have become an important force. Khan et al. [21] proposed the fractional back-propagation through time (FBPTT) for ANNs, showing that this algorithm outperformed the conventional back-propagation. Other fractional gradient descent methods for training ANNs demonstrated the competitive performance of the fractional models [22,23,24]. ANNs involving FC have been applied to image encryption, resisting common attack methods and improving safety performance [25,26].

In image processing and ANNs, category representation is the most fundamental concept. However, are there methods based on FC that represent different classes? For this paper, we proposed a category representation vector, called the fractional category representation vector(abbreviated as frac-vector(FV)), which extends the category representation from integer vectors (such as the famous one-hot label, only 1 for target category and 0 for non-category) to discretized fractional vectors. What is more, FV can not only be used to represent different categories but can also be seen as a strong regularization tool, which means to reduce the test errors, alleviating the overfitting problems of models caused by one-hot labels due to overconfidence. In Section 3, we give the definition of FV and explain rationality, the relationship with the one-hot label. In Section 4, we review experiments on CIFAR-10, CIFAR-100, and MNIST by applying FV to ResNet-18, ResNet121, DenseNet, and DenseNet121. The results showed that Deep Neural Networks (DNNs) with FV as the category representation take no more time than the one-hot label and achieve state-of-the-art performance over traditional methods. In addition, FV labels improve the robustness and categorical representability of the generative adversarial works with additional conditions (C-GANs). We also showed that FV can be used as a space sampling, with fewer dimensions and less computational overhead than normal distributions. In Section 2, we review the relevant work; in Section 5, we conclude with some points for further discussion.

2. Related Work

In machine learning, the one-hot vector is the most popular and most practical among category representation vectors because of its simplicity and ease of use, especially when the cross-entropy loss function is adopted. However, its shortcomings are as plain as the nose on your face. One-hot vectors tend to be overfitting models. The fatal flaw is that dimensional disasters become worse as the number of categories increases. This is particularly unfeasible for representing thousands of words. Therefore, neural language models are proposed in which word vectors are trained with a given length [27]. Such distributed representations [28] of words have better generalization than one-hot representations [29]. However, this algorithm cannot be used in image classification, perhaps because images already lie in a rich vector space where distance is a poor quality similarity metric [29]. Another non-negligible flaw is that the Euclidean distance between any two vectors is equal in one-hot vectors. Sometimes we need category representation vectors for stronger category representation ability, and the distance between any two vectors is unequal. In this paper, we extend one-hot to fractional vectors, which not only represent classes but can also be used for strong regularization.

Additionally well known is the label smoothing mechanism proposed by Christian Szegedy et al. [30], which is an effective regularization tool for DNNs. Label smoothing can reduce to some extent the inference caused by overfitting. A training sample with label y can be expressed by the following:

$$\mathrm{v}\left(\mathrm{k},\mathrm{y}\right)=\left(1-\epsilon \right){\delta }_{k,y}+ \mathsf{\epsilon }\mathrm{u}\left(\mathrm{k}\right)$$

$$\mathrm{where} {\delta }_{k,y}=\{\begin{array}{c}1 if k=y\\ 0 if k\ne y\end{array}$$

$$\mathrm{u}\left(\mathrm{k}\right)=\frac{\epsilon }{K}(\mathrm{K} \mathrm{is} \mathrm{is} \mathrm{the} \mathrm{number} \mathrm{of} \mathrm{categories})$$

When $\epsilon$ is zero, the smoothing label becomes a one-hot label. In essence, the smoothing label is the mixture of the one-hot label and the uniform distribution u(k) with weights (1 − $\epsilon$) and $\epsilon$, respectively. Another advantage of label smoothing is that it is easy to evaluate the cross-entropy loss. However, the label smoothing assumption that each non-target category has the same probability is not realistic. In the experimental part, we compared the differences in the performance of generalization between FV and label smoothing on the test sets.

Motivated by the issue of label smoothing and knowledge distillation [31], Chang-Bin Zhang et al. presented an online label smoothing (OLS) strategy [32] that dynamically updates the probability distribution between target categories and non-target categories during the training process. OLS provides a new idea for training the distributed representation of image categories.

Other similar studies also have been carried out on how to generate soft labels. For example, Li et al. trained two networks in which images and labels were embedded to capture the relationships between image features and labels [33]. Instead, our approaches sought meaningful distributed representations of categories that were simple, easy to use, and also based on FC.

3. Method

3.1. About Grünwald–Letnikov Fractional-Order Derivative

Since the introduction of non-integer-order differentiation, many scientists have made important contributions to the theory of FC. The most famous are Grünwald, Letnikov, Riemann, Liouville, Caputo, Riesz, and others [1,2]. The n-order (where $n$ is an integer) derivative of the function $f\left(x\right)$ can be written as the following form:

$$f^{\left(n\right)}\left(x\right)=\underset{h\to 0}{\lim }\frac{1}{h^n}{\displaystyle \sum }_{r=0}^n{\left(-1\right)}^r\left(\begin{array}{c}n\\ r\end{array}\right)f\left(x-rh\right)$$

(1)

where

$$\left(\begin{array}{c}n\\ r\end{array}\right)=\frac{n\left(n-1\right)\left(n-2\right)\cdots \left(n-r+1\right)}{r!}$$

are the binomial coefficients. When $n$ is negative, (1) represents n-fold integral of $f\left(x\right)$.

Similar to the definition of integer derivative, we obtain the definition of the left and right Grünwald–Letnikov fractional derivative by replacing integer n with real number $\alpha$ [34]:

$${}_a{}^{G}D{}_x{}^{\alpha }f\left(x\right)= \underset{h\to 0^{+}}{\lim }{h}^{-\alpha }{\displaystyle \sum }_{j=0}^n{\left(-1\right)}^j\left(\begin{array}{c}\alpha \\ j\end{array}\right)f\left(x-jh\right)$$

$${}_x{}^{G}D{}_b{}^{\alpha }f\left(x\right)= \underset{h\to 0^{+}}{\lim }{h}^{-\alpha }{\displaystyle \sum }_{j=0}^n{\left(-1\right)}^j\left(\begin{array}{c}\alpha \\ j\end{array}\right)f\left(x+jh\right)$$

where n denotes that there are n discretized numbers from a to x or from x to b.

Based on the expressions for the left and right Grünwald–Letnikov fractional derivatives, we can construct a discretized vector consisting of the coefficients of the fractional-order derivatives:

$$\mathrm{FV}\left(\mathsf{\alpha },i\right)={\left(-1\right)}^i\left(\begin{array}{c}\alpha \\ i\end{array}\right), i=0,1,2,3,$$

$$\mathrm{stipulation}:\{\begin{array}{c}\mathrm{FV}\left(\alpha ,0\right)=1 \\ \mathrm{FV}\left(\alpha ,i\right)=FV\left(\alpha ,-i\right) \end{array}$$

We can calculate $\mathrm{FV}\left(\alpha ,i\right)$ iteratively in the way:

$$\mathrm{FV}\left(\alpha ,0\right)=1, \mathrm{FV}\left(\alpha ,i\right)=\left(1-\frac{\alpha +1}{i}\right)\mathrm{FV}\left(\alpha ,i-1\right)$$

(2)

From the above formula of iteration, we may obtain the following properties of FV:

(1) $\mathrm{if} \alpha =0, \mathrm{FV}\left(\alpha ,0\right)=1, \mathrm{FV}\left(\alpha ,i\right)=0$

(2) $\mathrm{if} 0<\alpha <1, \mathrm{FV}\left(\alpha ,0\right)=0, \mathrm{FV}\left(\alpha ,1\right)=-\alpha , \mathrm{FV}\left(\alpha ,i\right)<0, i=2,3,\cdots$

(3) $\mathrm{if} 1<\alpha <2, \mathrm{FV}\left(\alpha ,0\right)=0, \mathrm{FV}\left(\alpha ,1\right)=-\alpha , \mathrm{FV}\left(\alpha ,i\right)>0, i=2,3,\cdots$

(4) $\mathrm{if} \alpha \ne 0, \sum _{i=-\infty }^{\infty }\mathrm{FV}\left(\alpha ,i\right)=-1$

Mathematically, the Riesz fractional derivative in a boundary interval is defined as [35]:

$$\frac{{\partial }^{\alpha }}{\partial {\left|x\right|}^{\alpha }}=-\frac{1}{2\cos \left(\frac{\alpha \pi }{2}\right)}\left[{}_a{}^{R}D{}_x^{\alpha }f\left(x\right)+{}_x{}^{R}D{}_b^{\alpha }f\left(x\right)\right]$$

where ${}_a{}^{R}D{}_x^{\alpha }f\left(x\right)$ and ${}_x{}^{R}D{}_b^{\alpha }f\left(x\right)$ are the left and right Riemann–Liouville fractional derivative, respectively. In numerical calculations, the Riemann–Liouville fractional derivative is calculated in the same way as the Grünwald–Letnikov fractional derivative. The following central difference methods are usually used to calculate the Riesz fractional derivative:

$$\frac{{\partial }^{\alpha }}{\partial {\left|x\right|}^{\alpha }}f\left(x\right)=-\frac{1}{h^{\alpha }}{\displaystyle \sum }_{i=-\infty }^{\infty }\frac{{\left(-1\right)}^i\mathsf{\Gamma }\left(\alpha +1\right)}{\mathsf{\Gamma }\left(\frac{\alpha }{2}-i+1\right)\mathsf{\Gamma }\left(\frac{\alpha }{2}+i+1\right)}f\left(x-ih\right)+O\left(h^2\right)$$

We use FR to mark the above coefficients:

$$\mathrm{FR}\left(\mathsf{\alpha },i\right)=\frac{{\left(-1\right)}^i\mathsf{\Gamma }\left(\alpha +1\right)}{\mathsf{\Gamma }\left(\frac{\alpha }{2}-i+1\right)\mathsf{\Gamma }\left(\frac{\alpha }{2}+i+1\right)}, i=0,{}_{+}{}^{-}1, {}_{+}{}^{-}2,\cdots$$

A visual representation of the vectors constructed from the coefficients of the Grünwald–Letnikov fractional derivative is shown in Figure 1, where a = 0, b = 9, and $x=5$. Figure 2 shows the vectors constructed from the coefficients of the Riesz fractional derivatives.

Comparing Figure 1 with Figure 2, we find that the shape of the two vectors is generally consistent. This means that the hypothesis of the Grünwald–Letnikov fractional derivative coefficient vectors is well founded and reasonable. However, the computational complexity of gamma functions hinders the further use of the Riesz fractional operation in deep neural networks; so, we adopt the Grünwald–Letnikov method.

3.2. From Coefficient Vectors to Category Representation Vectors

In terms of the total amount of information contained, FV contains more information than the one-hot and label smoothing. The fractional vector with n elements can represent $n$ different characteristics, each of which can be different as the order changes. When $0<\mathsf{\alpha }<1$, the value of the non-target element is $\mathrm{between} -1 \mathrm{and} 0$, which does not need to be positive, whereas negative probabilities exist in the physical world and are necessary in some cases [36,37,38]. In our experiments, models with FV representing categories converged more easily in some cases, perhaps providing the evidence for the existence of negative distribution.

The basic assumption for the classifier is that the greater the difference between different categories, the better. Correspondingly, the similarity of different category representation vectors must be more heterogeneous. The similarity between any two one-hot vectors is equal to 0, while the similarity between any two vectors represented by label smoothing is also equal, though not equal to 0. For a classifier with 10 classes, the cosine similarity between any two label smoothing vectors is 0.0229 if $\mathsf{\epsilon } \mathrm{is} \mathrm{taken} \mathrm{as} \mathrm{the} \mathrm{fault} \mathrm{value} 0.1$. However, the similarity between any two adjacent FVs is negative, and the cosine similarity between any two FVs is different. The details of the data analysis can be seen in

Table 1. Cosine similarity between FVs, supposing there are 10 classes and $\mathsf{\alpha }$ = 0.5.

	0	1	2	3	4	5	6	7	8	9
0	1	−0.6644	0.0283	0.0182	0.0129	0.0096	0.0074	0.0057	0.0039	0.0000
1		1	−0.5636	0.0461	0.0292	0.0205	0.0152	0.0116	0.0086	0.0039
2			1	−0.5552	0.0488	0.0310	0.0217	0.0160	0.0116	0.0057
3				1	−0.5530	0.0496	0.0314	0.0217	0.0152	0.0074
4					1	−0.5525	0.0496	0.0310	0.0205	0.0096
5						1	−0.5530	0.0488	0.0292	0.0129
6							1	−0.5552	0.0461	0.0182
7								1	−0.5636	0.0283
8									1	−0.6644
9										1

. According to the table, even without the use of AI, the average person can correctly identify the target category based on the difference in cosine similarity.

When FV is applied to DNNs, the speed of operation is intolerable. Iterative computation according to Equation (2) means to count each element repeatedly. If the number of elements counted with Pythonis greater, the computation crashes. Therefore, the iterative calculation must be changed to the sequential method. The running speed of the sequential calculation method is more than 1000 times faster than that of the iterative method. Algorithm 1 shows the few lines of codes needed to quickly construct an FV. Compared to one-hot vectors, the linear operation of FVs makes sense in the training of DNNs, as shown in the next section, where the convex combination of data [39] directly corresponds to the weighted combination of the corresponding FVs. The one-hot representation category, on the other hand, cannot be used directly for arithmetic in this way. Using the mix-up arithmetic trick described in [39], the loss function must be weighted after the individual categories have been operated on using cross-entropy loss function. Note that FVs cannot be used to calculate the loss function using cross-entropy because some of the elements of the FV are negative and cannot be logged. However, FVs can use cosine similarity to estimate the loss function for training; the effect is barely distinguishable from the cross-entropy loss function.

Algorithm 1: Codes of the sequential calculation method for the design of FVs

def lateral_index(index, alpha):   x = []   for i in range(index):     if i == 0:       tmp = 1     else:       tmp = (1 − (alpha + 1)/i) * tmp     x.append(tmp)   return x

From Figure 1 and Figure 2, we can see that FVs have the property of being non-local, which is related to the universal property of fractional calculus of being non-local and memorable. This can be used in deep neural networks and plays an important role in the generalization performance of networks, as will be shown in the next section.

4. Experiments

4.1. CIFAR-10

We conducted image classification experiments on the CIFAR-10 data sets to evaluate the generalization performance of FVs, shown in

Table 2. Accuracy of CIFAR-10 data set.

	Plus	Train_acc	Test_acc	Best_acc	Top5_acc
ResNet18		1.0	0.1439	0.1500	0.5359
	Label Smoothing	1.0	0.4030	0.4100	0.8282
	FVs	0.9990	0.8648	0.9050	0.9451
	mix-up	0.7577	0.5194	0.5600	0.9256
	FVs+mix-up	0.8137	0.9520	0.9559	0.9888
ResNet101		0.9983	0.9441	0.9557	0.9986
	FVs	0.9236	0.8838	0.8883	0.9697
	FVs+mix-up	0.9987	0.9450	0.9450	0.9871

. We compared FVs with label smoothing regularization or mix-up methods [39].The data set consisted of 60,000 color images of three channels, each of which had a size of 32 × 32. There were 50,000 images in the training set and 10,000 images in the test set. We started the learning rate at 0.1 and divided it by 10 after 70 epochs or 140 epochs. ResNet18 was trained for 90 epochs, and ResNet101 was trained for 200 epochs. All the models were trained on Kaggle with a Tesla P100-GPU using pyTorch1.10. We set weight decay with ${10}^{-4}$. All the methods were evaluated on the test set after training.

From Table 2, we can conclude that the generalization of FVs performed better than other methods. The small-capacity model ResNet18 with FVs plus mix-up was equivalent in performance to the high-capacity model ResNet101. For high-capacity models, FVs had no advantages, perhaps because high-capacity models already had good generalization performance. The training accuracy of the mix-up method was low because the combination of data could not directly correspond to the combination of labels in the training process. Finally, the convex combination of one-hot vectors was meaningless. However, FVs can be directly operated linearly. Therefore, the training accuracy of the FVs+mix-up method was much higher than that of the mix-up method.

4.2. CIFAR-100

We also evaluated FVs on the CIFAR-100 dataset with the result shown in

Table 3. Test accuracy on CIFAR-100 data set.

	Plus	Train_acc	Top1_acc	Best_acc	Top5_acc
DenseNet		0.9711	0.5769	0.6328	0.8290
	Label Smoothing	0.9777	0.5786	0.6172	0.8255
	FVs	0.7777	0.5833	0.6719	0.7058
	mix-up	0.5717	0.5543	0.6442	0.6833
DenseNet121		0.7124	0.5764	0.6328	0.6847
	FVs	0.9930	0.6283	0.7344	0.7530
	mix-up	0.6247	0.5897	0.6014	0.6731
	FVs+mix-up	0.6912	0.6628	0.7412	0.8216

. This data set contained 100 classes. Each class contained 600 32 × 32 color images, of which 500 were used as training images and 100 were used in the test set. During training, data augmentation such as random horizontal flip and color jitter were used and the learning rate was divided by 10 when the epoch was increased to 90 or 140. All the models were training on Kaggle with a Tesla P100-GPU for running 150 epochs with a mini-batch size of 128. The experiments showed that $\mathsf{\alpha }\in \left(0, 1\right)$ made FVs appropriate.

For FVs, we found that FVs plus the mix-up method led to improved performance in generalization, almost more than 10 percentage points over DenseNet121 alone. We also found that there was little difference between using any regularization method alone and training DenseNet series directly.

4.3. MNIST for InfoGAN

The emergence of generative adversarial nets (GANs) [40] is an innovative milestone in deep learning; but the images generated by GANs are random and unpredictable. Conditional GANs (C-GANs) make GANs controllable with additional conditions [41]. The objective function of C-GANs can be denoted as follows [41]:

$$\underset{G}{\min } \underset{D}{\max }V\left(D,G\right)=E_{x~P_{data\left(x\right)}}[\log \left(D\left(x|y\right)\right]+E_{z~P_{z\left(z\right)}}\left[\log \left(1-D\left(G\left(z|y\right)\right)\right)\right]$$

where y is the condition; these can be category conditions.

Due to the simplicity and applicability, we found that both FVs and one-hot used in C-GANs can achieve a completely correct classification effect. We could generate the images according to the categorical conditions. However, the images generated by C-GANs were blurred. Motivated by the knowledge of C-GANs, many promising methods have been proposed, such as InfoGAN [42], pix2pixGAN [43], and ACGAN [44]. FVs can provide these models with more category representation information and more degrees of freedom than one-hot or label smoothing vectors. Here, we evaluated FVs using the InfoGAN model on a MNIST data set. During the training process, we set $\mathsf{\alpha }\in \left(1.0, 1.5\right)$ while the dimension of the FVs was 32 and the dimension of the noise vector was 16. The model was trained over 30 epochs. The fractional orders $\mathsf{\alpha }$ that were used to generate each row of images were random numbers obeying a uniform distribution between 1 and 1.5. Experiments showed that $\mathsf{\alpha } \mathrm{in}$ this interval produced the best quality images. From Figure 3, we can see that as the fractional order $\mathsf{\alpha }$ changed, the resulting images changed consistently from line to line, such as line thickness and rotation angle, and that the image quality was better with less interference. However, adding disentangled latent codes to vectors during training to represent different meanings worked better, as explained in the paper [42].

In the previous one-hot category representation algorithm, we needed a 100-dimensional vector to represent the categories and noise samples for training the GAN model on the MNIST data set. With FVs, we only needed 48 dimensions to obtain the same or even better image quality. Similar experiments confirmed that FVs can be used for noise sampling. The results were consistent with standard normal distribution sampling.

5. Discussion and Future Directions

In this paper, we propose FVs based on the left and right Grünwald–Letnikov fractional derivatives. We showed that FVs have the performance of generalization and, as category representation vectors, FVs can be computed directly according to the combination of data. Experiments showed that FVs plus the mix-up method can produce a stronger regularization of the classification during DNNs’ training. The generic one-hot is the peculiarity of frac-vectors when the fractional order is 0. When utilized in GAN models, frac-vectors provide more information and higher degrees of freedom about the category without more computational overhead.

FVs also open up avenues for further exploration of the category representation based on the fractional calculus theory, which is still mysterious in its application to ANNs, especially in deep neural networks. There are still too many puzzles to be solved, such as how to make the non-target elements of the FVs change as training progresses and be used for image processing, just as word vectors can be trained in DNNs, and how to improve the regularization of FVs and generalization performance. Although our discussion involves the simple extension of the one-hot category representation, we are excited about the implicit layers of the fractional category representation and hope that our discussion will prove useful for further development.