Granular Elastic Network Regression with Stochastic Gradient Descent

Abstract

Linear regression is the use of linear functions to model the relationship between a dependent variable and one or more independent variables. Linear regression models have been widely used in various fields such as finance, industry, and medicine. To address the problem that the traditional linear regression model is difficult to handle uncertain data, we propose a granule-based elastic network regression model. First we construct granules and granular vectors by granulation methods. Then, we define multiple granular operation rules so that the model can effectively handle uncertain data. Further, the granular norm and the granular vector norm are defined to design the granular loss function and construct the granular elastic network regression model. After that, we conduct the derivative of the granular loss function and design the granular elastic network gradient descent optimization algorithm. Finally, we performed experiments on the UCI datasets to verify the validity of the granular elasticity network. We found that the granular elasticity network has the advantage of good fit compared with the traditional linear regression model.

1. Introduction

In the 1970s, Zadeh, an American automatic control expert, introduced the concept of fuzzy sets [1]. Fuzzy sets and fuzzy logic have attracted a large number of scholars worldwide to research on the topic. Hobbs proposed the concept of granularity [2], and which is consistent with human cognition, memory, reasoning, and logic. After this, Zadeh introduced the concept of fuzzy information granulation [3]. He stated that information granulation is the division of a whole into many information granules, after which those granules are studied. In 1999, T.Y. Lin proposed granular computing [4] on this base and applied it to the field of data mining [5]. Skowron proposed a granular calculation method based on rough set theory [6]. Liu [7] constructed an approach to information granule reasoning from the perspective of fuzzy logic. Then, in 2001, Yao defined the neighborhood system [8] and studied the connection between hierarchical granulation and rough set approximation. In 2002, Miao proposed a knowledge-based approach to granular computing [9]. Then Miao [10] reconstructed the granular computing structure from the perspective of set theory. Wang et al. applied granular computing to big data information processing [11]. Chen [12] et al. presented a PS-tree based rough set feature selection method. In 2014, Qian reduced the computation time of the attribute approximate reduction problem by using parallel granular computation [13]. After that the neighborhood system granular structure, distance, and metric were proposed by Chen [14]. Chen further introduced the rough granular KNN algorithm. Li [15] combined the boosting algorithm with the granular KNN algorithm based on Chen’s work [14] to further improve the performance of the granular KNN algorithm. Although different scholars have different understandings of the definition of granules, the structural idea of granules has attracted more and more scholars to conduct research. Thus granular computing has been applied to a variety of fields, including information systems [16], image processing [17], cognitive systems [18], attribute selection [19,20], multi-attribute decision making [21], and other fields [22,23,24,25,26].

Machine learning is a general data processing technique that is divided into unsupervised learning and supervised learning. Unsupervised learning includes PCA, local linear embedding, K-means clustering, and other algorithms. Supervised learning mainly includes K-nearest neighbor, decision tree, linear regression, logistic regression, support vector machine, etc. Regression models are supervised learning modalities including linear regression, ridge regression, lasso regression, and elastic network regression. Linear regression [27] simulates the relationship between a dependent variable and one or more independent variables using the linear function. By averaging the weights by adding L2 penalty, ridge regression [28] can improve the fitting ability of regression models in pathological data. The lasso [29] regression can suppress the overfitting of linear regression to some extent, and the lasso regression obtains a sparse model by adding L1 regularization to make some weight coefficients zero. In contrast, elastic networks combine ridge regression and lasso regression to both sparse the weights and shrink the weight coefficients [30]; however, these traditional regression models struggle to deal with uncertainty data and set data. Granular computing is an effective tool to deal with uncertain data; therefore, we construct a granular-based regression model of elastic networks. The contributions of this paper are as follows.

First, we propose a granulation approach based on feature distance and define various granulation operations and granularity measure approaches. Secondly, we construct the granular elastic network regression system and give the loss function of granular elastic network. Similarly, we optimize the granular elastic network regression equation by stochastic gradient descent. Finally, we conducted experiments on the UCI datasets to verify the effectiveness of the granular elastic network and found that the granular elastic network has a better fitting performance than the traditional regression models.

This article consists of the following parts. The first chapter briefly introduces the granular computations and regression algorithms. Then, we define the granularity approach in Section 2. In Section 3, we propose the granular elastic network regression and further propose the granular elastic network optimizer and its optimization algorithm. After that, we show the experimental results in Section 4. Finally, we conclude the whole paper in Section 5.

2. Granulation

Fuzzy sets are an effectual tool for dealing with uncertain information. For the information system $U=(X,C,D)$, $X=\{x_1,x_2,x_3,\dots ,x_n\}$ is the sample set, $C=\{c_1,c_2,c_3,\dots ,c_m\}$ is the feature set corresponding to the sample, and D is the decision set corresponding to the sample. For the given sample $x\in X$, where the single feature $c\in C$, $v(x,c)\in [0,1]$ denotes the value of the sample x normalized on the feature c.

Definition 1.

In the data set $U=(X,C,D)$, samples $x_1,x_2\in X$, and a single feature $c\in C$. The distance between $x_1$ and $x_2$ on the feature c is:

$$s_c\left(x_1,x_2\right)=|v\left(x_1,c\right)-v(x_2,c)|.$$

(1)

Definition 2.

For the data set $U=(X,C,D)$, we granulate the sample $x\in X$ on the feature $c\in C$. The sample x is granulated on feature c and forms a granule, which is defined as feature c to form a fuzzy granule, which is defined as:

$$g_c\left(x\right)={\left\{g_c{\left(x\right)}_j\right\}}_{j=1}^k={\left\{r_j\right\}}_{j=1}^k=\{r_1,r_2,...,r_k\},$$

(2)

$$where{r}_j=S_c(x,x_j)=|v(x,c)-v(x_j,c)|.$$

(3)

$r_j$ is the distance between sample x and sample $x_j$ on feature c. It is easy to know from Definition 1 that $r_j=s_c\left(x,x_j\right)\in [0,1]$. We define $g_c\left(x\right)$ as the granule and $g_c{\left(x\right)}_j$ as the $jth$ granule kernels of the granule $g_c\left(x\right)$, and the granule consists of the granule kernels.

Definition 3.

For the data set $U=(X,C,D)$, there is any $x\subseteq X$ and any feature subset $B=\{b_1,b_2,b_3,...,b_m\}\subseteq X$. The granular vector of x on the feature subset B is:

$$G_B\left(x\right)={(g_{b1}\left(x\right),g_{b2}\left(x\right),g_{b3}\left(x\right),\dots ,g_{bm}\left(x\right))}^T.$$

(4)

$g_{bm}\left(x\right)$ is the granule of x on the feature $b_m$. The granular vector is made up of granules, and the granules are consisted of granular kernels. So the granular vector can also be represented by the granular kernel matrix as follows.

$$G\left(x\right)=\left[\begin{array}{ccc}{g}_1{\left(x\right)}_1& g_1{\left(x\right)}_2& \begin{array}{cc}\cdots & g_1{\left(x\right)}_k\end{array}\\ g_2{\left(x\right)}_1& g_2{\left(x\right)}_2& \begin{array}{cc}\cdots & g_2{\left(x\right)}_k\end{array}\\ \begin{array}{c}\cdots \\ g_m{\left(x\right)}_1\end{array}& \begin{array}{c}\cdots \\ g_m{\left(x\right)}_2\end{array}& \begin{array}{cc}\begin{array}{c}\cdots \\ \cdots \end{array}& \begin{array}{c}\cdots \\ g_m{\left(x\right)}_k\end{array}\end{array}\end{array}\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& \begin{array}{cc}\cdots & r_{1k}\end{array}\\ r_{21}& r_{22}& \begin{array}{cc}\cdots & r_{2k}\end{array}\\ \begin{array}{c}\cdots \\ r_{m1}\end{array}& \begin{array}{c}\cdots \\ r_{m2}\end{array}& \begin{array}{cc}\begin{array}{c}\cdots \\ \cdots \end{array}& \begin{array}{c}\cdots \\ r_{mk}\end{array}\end{array}\end{array}\right]$$

(5)

The granular vector $g\left(x\right)$ is a set consisting of granules, while the elements of a conventional vector are real numbers.

Definition 4.

For the data set $U=(X,C,D)$, there is any $x\subseteq X$ and the decision set $d_x\subseteq D$. The extended decision granule of x is:

$$Y\left(x\right)=\{d_x,d_x,...,d_x\},$$

(6)

where the number of the elements is the size of X.

Example 1.

The following is an example of granulation.

As shown in

Table 1. A data set.

X	a	b	c	D
$x_1$	1	0.5	0.6	22.5
$x_2$	0.35	0.4	0.5	33
$x_3$	0.4	1	0.9	45
$x_4$	0.7	0.7	0.1	10.05

, the sample set $X=\{x_1,x_2,x_3,x_4\}$ is granulated on feature a as:

$$g_a\left(x_1\right)=\{0,0.65,0.6,0.3\},g_a\left(x_2\right)=\{0.65,0,0.05,0.35\},g_a\left(x_3\right)=\{0.6,0.05,0,0.3\},g_a\left(x_4\right)=\{0.3,0.35,0.3,0\}.$$

(7)

The sample set $X=\{x_1,x_2,x_3,x_4\}$ is granulated on feature b as:

$$g_b\left(x_1\right)=\{0,0.1,0.5,0.2\},g_b\left(x_2\right)=\{0.1,0,0.6,0.3\},g_b\left(x_3\right)=\{0.5,0.6,0,0.3\},g_b\left(x_4\right)=\{0.2,0.3,0.3,0\}.$$

(8)

The sample set $X=\{x_1,x_2,x_3,x_4\}$ is granulated on feature c as:

$$g_c\left(x_1\right)=\{0,0.1,0.3,0.5\},g_c\left(x_2\right)=\{0.1,0,0.4,0.4\},g_c\left(x_3\right)=\{0.3,0.4,0,0.8\},g_c\left(x_4\right)=\{0.5,0.4,0.8,0\}.$$

(9)

The feature granular vectors of the sample $X=\{x_1,x_2,x_3,x_4\}$ are:

$$G_{\{a,b,c\}}\left(x_1\right)={(g_a\left(x_1\right),g_b\left(x_1\right),g_c\left(x_1\right))}^T={\left(\right\{0,0.65,0.6,0.3\},\{0,0.1,0.5,0.2\},\{0,0.1,0.3,0.5\left\}\right)}^T,$$

(10)

$$G_{\{a,b,c\}}\left(x_2\right)={(g_a\left(x_2\right),g_b\left(x_2\right),g_c\left(x_2\right))}^T={\left(\right\{0.65,0,0.05,0.35\},\{0.1,0,0.6,0.3\},\{0.1,0,0.4,0.4\left\}\right)}^T,$$

(11)

$$G_{\{a,b,c\}}\left(x_3\right)={(g_a\left(x_3\right),g_b\left(x_3\right),g_c\left(x_3\right))}^T={\left(\right\{0.5,0.6,0,0.3\},\{0.5,0.6,0,0.3\},\{0.3,0.4,0,0.8\left\}\right)}^T,$$

(12)

$$G_{\{a,b,c\}}\left(x_4\right)={(g_a\left(x_4\right),g_b\left(x_4\right),g_c\left(x_4\right))}^T={\left(\right\{0.3,0.35,0.3,0\},\{0.2,0.3,0.3,0\},\{0.5,0.4,0.8,0\left\}\right)}^T.$$

(13)

The decision set $Y=\{x_1,x_2,x_3,x_4\}$ is extended and granularized as

$$Y\left\{x_1\right\}=\{22.5,22.5,22.5,22.5\},$$

(14)

$$Y\left\{x_2\right\}=\{33,33,33,33\},$$

(15)

$$Y\left\{x_3\right\}=\{45,45,45,45\},$$

(16)

$$Y\left\{x_4\right\}=\{10.05,10.05,10.05,10.05\}.$$

(17)

3. Granular Elastic Network Regression

3.1. Granular Operations and Measures

To construct the granular regression system, in this subsection, we give the granular operators and measures.

Definition 5.

Let $g_a\left(x\right)$, $g_b\left(x\right)$ be the two granules of sample x on features $a,b$, respectively. The addition, subtraction, multiplication, and division operations for the granules are:

$$g_a\left(x\right)+g_b\left(x\right)={\left\{g_a{\left(x\right)}_j+g_b{\left(x\right)}_j\right\}}_{j=1}^k,$$

(18)

$$g_a\left(x\right)-g_b\left(x\right)={\left\{g_a{\left(x\right)}_j-g_b{\left(x\right)}_j\right\}}_{j=1}^k,$$

(19)

$$g_a\left(x\right)*g_b\left(x\right)={\left\{g_a{\left(x\right)}_j*g_b{\left(x\right)}_j\right\}}_{j=1}^k,$$

(20)

$$g_a\left(x\right)/g_b\left(x\right)={\left\{g_a{\left(x\right)}_j/g_b{\left(x\right)}_j\right\}}_{j=1}^k.$$

(21)

Definition 6.

Let $g_a\left(x\right)$, $g_a\left(y\right)$ be the two granules of samples x and y on features a. The addition, subtraction, multiplication, and division operations for the granules are:

$$g_a\left(x\right)+g_a\left(y\right)={\left\{g_a{\left(x\right)}_j+g_a{\left(y\right)}_j\right\}}_{j=1}^k,$$

(22)

$$g_a\left(x\right)-g_a\left(y\right)={\left\{g_a{\left(x\right)}_j-g_a{\left(y\right)}_j\right\}}_{j=1}^k,$$

(23)

$$g_a\left(x\right)*g_a\left(y\right)={\left\{g_a{\left(x\right)}_j*g_a{\left(y\right)}_j\right\}}_{j=1}^k,$$

(24)

$$g_a\left(x\right)/g_a\left(y\right)={\left\{g_a{\left(x\right)}_j/g_a{\left(y\right)}_j\right\}}_{j=1}^k.$$

(25)

The result of four operations with two granules is also a granule. The operations in Definition 5 are granular operations for granules of the same sample on different features, and the operations in Definition 6 are granular operations for granules of different samples on the same feature.

Definition 7.

Let the granular vector be $G(x_j)={(g_1{\left(x\right)}_j,g_2{\left(x\right)}_j,g_3{\left(x\right)}_j,\dots ,g_m{\left(x\right)}_j)}^T,G(x_i)={(g_1{\left(x\right)}_i,g_2{\left(x\right)}_i,g_3{\left(x\right)}_i,\dots ,g_m{\left(x\right)}_i)}^T$, the dot product of these two granular vectors is defined as:

$$G\left(x_i\right)·G\left(x_j\right)=g_1\left(x_i\right)*g_1\left(x_j\right)+g_2\left(x_i\right)*g_2\left(x_j\right)+g_3\left(x_i\right)*g_3\left(x_j\right)+\dots +g_m\left(x_i\right)*g_m\left(x_j\right).$$

(26)

The dot product of two granular vectors outcomes to a granule.

Definition 8.

Let the granule be $g\left(x\right)={\left\{r_j\right\}}_{j=1}^k$, whose size is:

$$size\left(g\left(x\right)\right)=\left|g\left(x\right)\right|=\sum _{j=1}^k{r}_j$$

(27)

Definition 9.

Let the granule be $g\left(x\right)={\left\{r_j\right\}}_{j=1}^k$, and its granular norms as follows:

(a). The 1-norm of the granule:

$$Norm-1\left(g\left(x\right)\right)=|{|g\left(x\right)\left|\right|}_1=\sum _{j=1}^k|r_j|,$$

(28)

(b). The 2-norm of the granule:

$$Norm-2\left(g\left(x\right)\right)=|{|g\left(x\right)\left|\right|}_2=\sqrt{{\sum }_{j=1}^k{r_j}^2},$$

(29)

(c). The p-norm of the granule:

$$Norm-p\left(g\left(x\right)\right)={\left|\right|g\left(x\right)\left|\right|}_p={(\sum _{j=1}^k{r_j}^p)}^{\frac{1}{p}}.$$

(30)

Definition 10.

Let the granular vector be $g\left(x\right)={\left\{r_j\right\}}_{j=1}^k$, and its granular vector norms as follows:

(a). The 11-norm of the granular vector:

$$Norm-11\left(G{(x}_i\right))={\left|\right|G\left(x_i\right)\left|\right|}_{11}=|\sum _{j=1}^i\sum _{j=1}^k|r_j\left|\right|,$$

(31)

(b). The 12-norm of the granular vector:

$$Norm-12\left(G{(x}_i\right))={\left|\right|G\left(x_i\right)\left|\right|}_{12}=\sum _{j=1}^i\sqrt{\sum _{j=1}^k{r_j}^2},$$

(32)

(c). The 21-norm of the granular vector:

$$Norm-21\left(G{(x}_i\right))={\left|\right|G\left(x_i\right)\left|\right|}_{21}=\sqrt{\sum _{j=1}^i{(\sum _{j=1}^k{r}_j)}^2},$$

(33)

(d). The 22-norm of the granular vector:

$$Norm-22\left(G{(x}_i\right))={\left|\right|G\left(x_i\right)\left|\right|}_{22}=\sqrt{\sum _{j=1}^i\sqrt{\sum _{j=1}^k{r_j}^2}}.$$

(34)

3.2. Granular Elasticity Regression Model

In the previous subsection, we defined several granular operations and granular norms. In the following, we construct our granular elastic network regression model with the defined granular operations.

Definition 11.

In the regression data set $U=(X,C,D)$, $x\in X$ is granularized and expanded according to Definition 1 as $G\left(x\right)=(g_1\left(x\right),g_2\left(x\right),g_3\left(x\right),\dots ,g_m\left(x\right),1)$, and the weight granular vector is ${W=(w}_1,w_2,w_3,\dots ,w_m,b)$. The granular regression equation is defined as:

$$Reg\left(x\right)=W·G\left(x\right)=w_1*g_1\left(x\right)+w_2*g_2\left(x\right)+\dots +w_m*g_m\left(x\right)+1*b.$$

(35)

Similar to the traditional linear regression weight randomization, the granular regression network also uses the normalized distribution to randomly initialize the granular weights.

In the traditional regression model, lasso regression can make the weights sparse, while ridge regression can shrink the weight coefficients to make the weights average. By combining the L1 penalty with the L2 penalty, the elastic network is able to both sparse the weights and shrink the weight coefficients. Similarly, in the granular regression system the granular elastic network uses a granular vector 11-norm penalty and a granular vector 22-norm penalty. The loss function of granular elastic network regression is defined as follows.

Definition 12.

In the regression data set U = (X,C,D), the weight granular vector is ${W=(w}_1,w_2,\dots ,w_m,b)$, the granular vector is $G\left(x\right)=(g_1\left(x\right),g_2\left(x\right),g_3\left(x\right),\dots ,g_m\left(x\right),1)$ and the decision granular vector is $G\left(y\right)=(g_1\left(y\right),g_2\left(y\right),g_3\left(y\right),\dots ,g_m\left(y\right),0)$. The granular loss function is defined as

$$G\left(e\right)=\frac{1}{2}{\left|\right|W·G\left(x\right)-\frac{1}{m}G\left(y\right)\left|\right|}_2^2+\alpha {\left|\right|W\left|\right|}_{11}+\frac{1}{2}\beta {\left|\right|W\left|\right|}_{22}^2,$$

(36)

where $W·G\left(x\right)-\frac{1}{m}G\left(y\right)$ is a granule.

We have the derivative operation for $G\left(e\right)$:

$$\frac{\partial G\left(e\right)}{\partial w}=-1*G\left(x\right)*(\frac{1}{m}G\left(y\right)-W·G\left(x\right))+\alpha *sign\left(W\right)+\beta *W.$$

(37)

3.3. Granular Elastic Network Optimization Algorithm

In order to optimize the granular regression elastic network to attain better regression results, we propose the granular elastic regression optimizer in the following:

$$W_c=\underset{W_c}{argmin}G\left(e\right)=\underset{W_c}{argmin}\frac{1}{2}{\left|\right|W·G\left(x\right)-\frac{1}{m}G\left(y\right)\left|\right|}_2^2+\alpha {\left|\right|W\left|\right|}_{11}+\frac{1}{2}\beta {\left|\right|W\left|\right|}_{22}^2,$$

(38)

We use stochastic gradient descent to solution the granular elasticity network parameters. To obtain the granular regression error, we define the root mean square error $RMSE$ of the granules expressed as follows:

$$GRMSE=\frac{1}{n}\sqrt{\sum _{i=1}^n{\left|\left|W_c•G\left(x_i\right)-\frac{1}{m}G\left(y_i\right)\right|\right|}_2^2}$$

(39)

The details of the stochastic gradient descent optimized granular elastic network regression algorithm are shown in Algorithm 1. In the training algorithm, m represents the maximum number of training iterations. $\eta$ is the learning rate.

Algorithm 1 Granular elastic network optimization algorithm

Input: The training set is $U=(X,Y)$, where is the feature vector, $x_i\in X$, is the regression decision vector $y_i\in Y,i=1,2,...,n$; the learning rate is $\eta$, and the number of iterations m Output: Granular weight matrix W. (1) The sample set X is granularized over the feature set $(X,C)$ to obtain $G\left(X\right)$ (2) The decision set Y is extended granularly as $G\left(Y\right)$ (3) Construct a granular elastic network and randomly initialize the granular weights in the network (4) i = 0 (5) for i to m (6)      ${GRMSE}^i=\frac{1}{n}\sqrt{{\sum }_{i=1}^n{\left|\left|W_c•G\left(x_i\right)-\frac{1}{m}G\left(y_i\right)\right|\right|}_2^2}$ (7)      ${W_c}^{i+1}={W_c}^i-\eta \frac{\partial G\left(e\right)}{\partial w}$ (8)      $G{RMSE}^{i+1}=\frac{1}{n}\sqrt{{\sum }_{i=1}^n{\left|\left|{W_c}^{i+1}•G\left(x_i\right)-\frac{1}{m}G\left(y_i\right)\right|\right|}_2^2}$ (9)      $\Delta RMSE=G{RMSE}^i-{GRMSE}^{i+1}$ (10)      if $\Delta RMSE>\xi$ (11)            break

4. Experimental Analysis

Multiple UCI datasets including Estates, Concrete, QSAR fish, Airfoil, Energy, Yacht, etc., are used for comparison experiments. Further, in order to comprehensively analyze the proposed method, we select the Estates, Concrete, and QSAR fish datasets for hyperparameter analysis and fitting analysis. The Concrete dataset consists of 8 conditional attributes and 1 decision attribute, the Estates dataset consists of 9 conditional attributes and 1 decision attribute, and the QSAR fish dataset consists of 6 conditional attributes and 1 decision attribute. We randomly select 80% of the samples from the dataset as the training set and the remaining 20% as the test set. The random number seed is fixed to ensure that the results of each selection are the same.

We measure the model by mean absolute error $\left(MAE\right)$ and the coefficient of determination $\left(R^2\right)$. $MAE$ is expressed as follows:

$$MAE=\frac{1}{n}\sum _{i=1}^n\left|\left|y_i^{\prime }-y_i\right|\right|,$$

(40)

where $y_i^{\prime }$ denotes the predicted value and $y_i$ denotes the true value. The smaller the $MAE$, indicates the smaller the average prediction error of the model. In order to represent $R^2$, we need to represent the concepts of sum of squares for regression $SSR$, sum of squares for error $SSE$, and sum of squares for total $SST$. $SSR$ is expressed as follows:

$$SSR=\sum _{i=1}^n{(y_i^{\prime }-\overline{y_i})}^2,where\overline{y_i}=\frac{{\sum }_{i=1}^n{y}_i}{n}.$$

(41)

$SSE$ is represented as:

$$SSE=\sum _{i=1}^n{(y_i^{\prime }-y_i)}^2.$$

(42)

$SST$ is defined as:

$$SST=SSR+SSE=\sum _{i=1}^n{(y_i-\overline{y_i})}^2.$$

(43)

$R^2$ is calculated by

$$R^2=1-\frac{SSE}{SST}=1-\frac{{\sum }_{i=1}^n{(y_i^{\prime }-y_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y_i})}^2}.$$

(44)

For the test set, the larger $R^2$ model indicates the better fit, the greater interpretation of the dependent variable by the variables and the high percentage of the total variation caused by the independent variables. The maximum value of $R^2$ is 1.

4.1. Convergence Analysis

We select two kinds of learning rate for convergence analysis. First of all, we set the learning rate of the algorithm to 0.001 and the loss function to MSE. Figure 1, Figure 2 and Figure 3 show the training processes of the granular elastic network on the Concrete, Estate, and QSAR datasets when learning the rate is 0.001, respectively. The granular elastic network converges on all three datasets. When the number of iteration rounds is 800, the granular elastic network reaches convergence on the Estate and QSAR datasets with smooth loss curves. For the Concrete data set, the granular elastic network also converges, but the loss curve is more volatile.

Then, the learning rate is adjusted to 0.005. Figure 4, Figure 5 and Figure 6 show the training processes of granular elastic network when the learning rate is 0.005. For Concrete dataset, it can be seen from Figure 1 and Figure 4 that the convergence curve when the learning rate is 0.005 is smoother than that when the learning rate is 0.001. However, for QSAR data set and Estate data set, the training curves are tortuous when the learning rate is 0.005, but they are able to convergence.

4.2. α and β Penalty Coefficient Proportional Impacts

In this subsection we show the effect of the $\alpha$ and $\beta$ ratios on the different test sets. To facilitate the analysis, we set $\alpha +\beta =1$, and the L11 ratio $=\frac{\alpha }{\alpha +\beta }$. L11 ratio indicates the ratio of L11 penalty to L11 and L22 combined penalty. Smaller L11 ratio means smaller L11 penalty to L11 and L22 combined penalty. Larger L11 ratio means smaller L22 penalty to L11 and L22 combined penalty.

For the Concrete test set, the effect of L11 ratio on the model is shown in Figure 7. It can be seen from Figure 7 that the MAE is minimum when the L11 ratio = 0.5. When L11 ratio = 0.8, the $MAE$ is maximum and reaches about 4.894. From Figure 8, it can be seen that when L11 ratio = 0.9, the $MAE$ of the estate test set is minimum. When L11 ratio = 0.6, the $MAE$ of the estate test set is the smallest. For the QSAR test set, as shown in Figure 9, the $MAE$ reaches a minimum of about 0.799 at L11 ratio = 0.8, while the $MAE$ of the QSAR test set reaches a maximum of about 0.831 at L11 ratio = 0.2.

4.3. Fitting Analysis

To better show the regression results, we selected 50 samples from the test set to plot the regression fitting curves, as shown in Figure 10, Figure 11 and Figure 12. Figure 10 represents the fitting curves of the granular elastic network in the Concrete test set. From Figure 10, we can see that the granular elastic network has good regression accuracy in the first 15 samples, but the regression error is larger in the last 10 samples. For the Estate test set, the first 5 samples are fitted very accurately, but samples 9 to 14 are fitted poorly. Sample 35 to final sample 50 are also fitted with less error. Comparing Figure 10 and Figure 12, it can be seen that the granular elastic network does not fit as well on the QSAR dataset as it does on the Concrete dataset.

4.4. Comparison of Granular Elastic Network Regression and Classical Regression Algorithms

In order to illustrate the effectiveness of the proposed method, the comparative experiments of granular elastic network and several classical regression algorithms in several UCI data sets are shown below. We use mean absolute error $\left(MAE\right)$ and the coefficient of determination $\left(R^2\right)$ to measure model fitting performance. For ridge regression, lasso regression, elastic network regression, and granular elastic network regression all 5-fold cross-validation are performed on the training set to extract regular coefficients with the best results, and then substituted into the test set for testing. We bold the best results in

Table 2. The $MAE$ results for granular elastic network regression and classical regression algorithms.

Datasets	Linear Regression	Ridge	Lasso	Elastic Network	Granular Elastic Network
Concrete	8.1142	8.0997	8.0997	8.0978	4.3481
Estate	6.2929	6.2205	6.2284	6.2346	5.1689
QSAR	0.777	0.769	0.7765	0.7727	0.7911
Airfoil	3.5452	3.5472	3.5477	3.5621	3.2883
Ale	0.223	0.2217	0.2179	0.2169	0.1471
Slump	2.7413	2.7503	2.7543	2.7615	1.9253
Daliy demand	71.8545	80.9481	71.8734	77.0038	85.7092
Boston	3.9358	3.926	3.917	3.8874	3.0508
Energy	2.2216	2.2439	2.2582	2.3377	0.9997
Yacht	8.3883	7.581	8.3399	7.462	1.9873

and

Table 3. The coefficient of determination results for granular elastic network regression and classical regression algorithms.

Datasets	Linear Regression	Ridge	Lasso	Elastic Network	Granular Elastic Network
Concrete	0.3937	0.3904	0.3929	0.3906	0.8842
Estate	0.5289	0.4995	0.5128	0.5059	0.6303
QSAR	0.3637	0.331	0.3626	0.3499	0.4113
Airfoil	0.1676	0.1621	0.1598	0.1257	0.3566
Ale	0.1009	0.0937	0.1011	0.0829	0.7212
Slump	0.791	0.7865	0.7926	0.7816	0.9125
Daliy demand	0.5698	0.4637	0.5692	0.4466	0.3353
Boston	0.3531	0.3455	0.3437	0.3029	0.5521
Energy	0.893	0.8917	0.8906	0.8749	0.9844
Yacht	0.2779	−0.1795	0.2705	−1.3873	0.9299

.

Table 2 shows the $MAE$ results on the test sets of granular elastic network and traditional regression networks. Smaller MAE means smaller regression error. Except for the Daily demand dataset and the QSAR dataset, the granular elastic network has a smaller mean absolute error $\left(MAE\right)$ than the classical regression algorithm. Especially in the Concrete, Energy and Yacht datasets, the MAE of the granular elastic network is particularly small compared to the classical regression algorithms.

The coefficient of determination $\left(R^2\right)$ of the granular elastic network and the traditional regression algorithms on the test sets are shown in Table 3. A higher coefficient of determination $\left(R^2\right)$ means better regression fitting performance. The maximum value of $R^2$ is 1. In the vast majority of datasets, the $R^2$ of the proposed algorithm is better than the classical regression algorithms. In the Yacht dataset, even though the $R^2$ of some traditional algorithms is negative, the $R^2$ of the granular elastic network is as high as about 0.93.

In general, the granular elastic network regression algorithm proposed in this paper has better regression performance than the classical regression algorithm.

5. Conclusions and Discussion

Traditional regression algorithms can only be mostly applied only to real-valued data, but not to fuzzy and aggregated domains. We design relevant granulation methods and granulation algorithms for fuzzy and ensemble data. Then, a novel granular regression system is proposed in this paper and two regularization penalties are added to it. Similarly, the proposed granular elastic network optimization algorithm is applied to granular elastic network learning. Finally, we conducted experiments on the UCI datasets and compared it with several traditional regression algorithms. The experimental results show that the granular elastic network proposed in this paper has the advantage of smaller regression error and better fit compared to traditional regression algorithms.

As the granulation method in this paper is global granulation, it is difficult to process big data in this granular regression method. We will propose a new granulation approach to apply granular regression to big data. In future work, we will apply the granular regression system to the nonlinear regression system. We will also propose new granulation methods to further improve the regression performance.