Kibria–Lukman-Type Estimator for Regularization and Variable Selection with Application to Cancer Data

Abstract

Following the idea presented with regard to the elastic-net and Liu-LASSO estimators, we proposed a new penalized estimator based on the Kibria–Lukman estimator with L1-norms to perform both regularization and variable selection. We defined the coordinate descent algorithm for the new estimator and compared its performance with those of some existing machine learning techniques, such as the least absolute shrinkage and selection operator (LASSO), the elastic-net, Liu-LASSO, the GO estimator and the ridge estimator, through simulation studies and real-life applications in terms of test mean squared error (TMSE), coefficient mean squared error (βMSE), false-positive (FP) coefficients and false-negative (FN) coefficients. Our results revealed that the new penalized estimator performs well for both the simulated low- and high-dimensional data in simulations. Also, the two real-life results show that the new method predicts the target variable better than the existing ones using the test RMSE metric.

1. Introduction

The linear regression model describes the response variable as a linear combination of one or more predictors, and it is widely applicable in fields such as medicine, biological science, economics, etc. The model is defined as follows:

$$y_i=x_i^T\beta +{\epsilon }_i$$

(1)

where $y={\left(y_1,\dots ,y_n\right)}^T$ is a $n\times 1$ vector of the dependent (response) variable; $X={\left(x_1,\dots ,x_n\right)}^T$, $x_i\in {\mathbb{R}}^{n\times p}$ and $i=1,\dots ,n$ are the known $n\times p$ matrix of predictors; $\beta ={\left({\beta }_1,\dots ,{\beta }_p\right)}^T$ is a $p\times 1$ vector of regression coefficients; and $\epsilon ={\left({\epsilon }_1,\dots ,{\epsilon }_n\right)}^T$ is the random error vector such that $\mathrm{E}\left({\epsilon }_i\right)=0$,.$\mathrm{V}\mathrm{a}\mathrm{r}\left({\epsilon }_i\right)={\sigma }^2\in {\mathbb{R}}^{+},i=1,\dots ,n.$

The model parameters are usually estimated using the least squares (LS) estimator. The LS estimator minimized the residual sum of squares (RSS) of model (1) and produced the following expression:

$${\widehat{\mathit{\beta }}}_n^{LS}=\arg \underset{\beta \in {\mathbb{R}}^p}{\min }\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2$$

(2)

where ${‖\cdot ‖}_2$ denotes the Euclidean norm. The high-dimensional quantitative structure–activity relationship (QSAR) modeling method is a popular example of the linear regression models. A high-dimensional problem arises when there are more features (variables) than the sample size (n) in a dataset. The popular OLSE does not give a unique solution via high-dimensional QSAR modeling since the number of molecular descriptors exceeds the compound number [1,2]. Likewise, the performance of the OLSE suffers a setback when there is linear dependency among the features (predictors), a condition called multicollinearity [3,4,5,6]. The setback includes a high standard error, insignificant t-tests, wrong conclusions, etc. [6,7,8].

The shrinkage estimators were mainly adopted to account for the linear dependency among the predictors. The shrinkage estimators include the ridge estimator [9], the Liu estimator [10], the Liu-type estimator [11], the two-parameter estimator [12], the modified ridge-type estimator [6], the modified Liu estimator [13], the Kibria–Lukman (KL) estimator [14], the modified KL estimator [15], and the Dawoud–Kibria (DK) estimator [16]. These shrinkage estimators account for more multicollinearity than the OLSE. The ridge estimator minimizes the residual sum of the squares (RSS) subject to an L2-norm of the coefficients, and it is defined as follows:

$${\widehat{\mathit{\beta }}}_n^{\lambda }=\arg \underset{\beta \in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+\frac{\lambda }{2}{‖\mathit{\beta }‖}_2^2\right\},$$

(3)

where $\lambda$ > 0 represents the penalized parameter. The estimator produces a more stable estimate and a better prediction than the LS estimator. The ridge estimator will only perform regularization in high-dimensional settings and will not perform variable selection. It fails to reduce any of its coefficients to zero. Hence, it becomes difficult to interpret the regression models in high-dimensional settings.

Variable selection forms a crucial aspect of modeling high-dimensional problems. It is worthwhile to state that removing the important features (predictors) produces more biased regression estimates and predictions. Likewise, including irrelevant predictors degrades the estimation efficiency and leads to imprecise predictions [17]. Thus, variable selection seeks to select the most relevant variables from a large dataset and remove irrelevant and redundant variables. Moreover, it improves the performance of the model prediction and reduces the computational cost. The problem of variable selection has been studied extensively in the literature [17,18,19,20,21,22].

The least absolute penalty and selection operator (LASSO) [18] is a well-known variable selection technique obtained by subjecting the RSS to an L1-norm penalty, and it is defined as follows:

$${\widehat{\mathit{\beta }}}_n^L=\arg \underset{\beta \in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+\lambda {‖\mathit{\beta }‖}_1\right\},$$

(4)

where $\lambda >0$ is the penalized parameter. LASSO retains the regularization property of the ridge and performs variable selection due to the introduction of the L1-norm penalty. It also improves the model prediction by eliminating irrelevant variables. However, the estimator often chooses only one variable from the group of pair-wise correlated predictors, and it is dominated by the ridge estimator when the multicollinearity level is high [1,8,18,22].

The elastic-net (E-net) approach [1] is another popular variable selection technique obtained by subjecting the RSS to the L1 and L2 norms penalty simultaneously. This technique retains the features of the ridge and the LASSO estimators. It is defined as follows:

$${\widehat{\mathit{\beta }}}_n^E=\arg \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+{\frac{{\lambda }_2}{2}‖\mathit{\beta }‖}_2^2+{{\lambda }_1‖\mathit{\beta }‖}_1\right\},$$

(5)

where ${\lambda }_1,{\lambda }_2>0$, ${‖\cdot ‖}_2$ is the L2-norm, and ${‖\cdot ‖}_1$ is L1-norm. The drawback of this method is that the L2-penalty term introduced extra bias that increases its variance [23].

Recently, the GO estimator [23] was developed as an improvement on the elastic net model. The estimator combines the features of the two-parameter estimator [24] and the LASSO. It is defined as follows:

$${\widehat{\mathit{\beta }}}_n^{GO}=\arg \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+\frac{{\lambda }_2}{2}{‖\mathit{\beta }-\mathit{d}{\widehat{\mathit{\beta }}}_n^{LS}‖}_2^2+{\lambda }_1{‖\mathit{\beta }‖}_1\right\},$$

(6)

where ${\lambda }_1,{\lambda }_2,d>0$.

The Liu-LASSO [21] is another type of E-net that is defined as follows:

$${\widehat{\mathit{\beta }}}_n^{LL}=\arg \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+\frac{1}{2}{‖\mathit{\beta }-\mathit{d}{\widehat{\mathit{\beta }}}_n^{LS}‖}_2^2+{\lambda }_1{‖\mathit{\beta }‖}_1\right\},$$

(7)

where ${\lambda }_1,d>0$. Both the GO estimator and the Liu-LASSO adopted two other shrinkage estimators (the two-parameter and the Liu estimators) other than the ridge estimator. Research shown that these shrinkage estimators compete well with the ridge estimator. Recently, the Kibria–Lukman (KL) estimator [14] shows competitive advantage over the ridge and the Liu estimator. Thus, the KL estimator will be adopted in a manner similar to the Liu-LASSO to form a new variable selection technique. The goal is to produce a new estimator in the class of the previously mentioned ones and assess its ability to deal with multicollinearity and high-dimensional problems.

The outline of the article is as follows: We define the new estimator in Section 2 and derive its properties. We analyze low-dimensional data and high-dimensional QSAR data to compare the proposed estimator to the existing ones, as shown in Section 4. The simulation study and conclusion are given in Section 3 and Section 5, respectively.

2. Proposed Estimator (KL1)

The Kibria–Lukman estimator is an optimization problem given as follows:

$${\widehat{\mathit{\beta }}}_n^{KL}=\arg \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+{\frac{{\lambda }_2}{2}‖\mathit{\beta }+{\widehat{\mathit{\beta }}}_n^{LS}‖}_2^2\right\},$$

(8)

where ${\lambda }_2>0$. The KL estimator possesses smaller bias compared to the ridge and the Liu estimator. Therefore, in accordance with the Liu-LASSO estimator, we proposed the KL1 estimator, and it is defined as follows:

$${\widehat{\mathit{\beta }}}_n^{KL1}=\arg \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}{‖\mathit{y}-\mathit{X}\mathit{\beta }‖}_2^2+{\frac{1}{2}‖\mathit{\beta }+{\widehat{\mathit{\beta }}}_n^{LS}‖}_2^2+{{\lambda }_1‖\mathit{\beta }‖}_1\right\},$$

(9)

where ${{\lambda }_1,\lambda }_2>0$. The estimator automatically retains the regularization and the variable selection property due to the L1-norm penalty.

Coordinate Descent for KL1

We assume that the predictor variables are standardized and the response variable is centered. We attempt to find ${\widehat{\beta }}_j$, which minimizes the following objective function:

$$\begin{gathered} \left({\widehat{\beta }}_j\right)=\frac{1}{2n}\sum _{i=1}^n{\left(y_i-x_i\beta \right)}^2+\frac{1}{2}\sum _{j=1}^p{\left({\beta }_j+{\widehat{\mathit{\beta }}}_j^{LS}\right)}^2+{\lambda }_1\sum _j^p\left|{\beta }_j\right| \\ =\frac{1}{2n}\sum _{i=1}^n{[y_i-x_{ij}{\beta }_j-\sum _{k\ne j}{x}_{ik}{\beta }_k]}^2+\frac{1}{2}\sum _{j=1}^p{\left({\beta }_j+{\widehat{\mathit{\beta }}}_j^{LS}\right)}^2+{\lambda }_1\left|{\beta }_j\right|+{\lambda }_1\sum _{k\ne j}\left|{\beta }_k\right| \\ =\frac{1}{2n}\sum _{i=1}^n{[y_i-{\overline{y}}_i^{\left(j\right)}{-x}_{ij}{\beta }_j]}^2+\frac{1}{2}\sum _{j=1}^p{\left({\beta }_j+{\widehat{\mathit{\beta }}}_j^{LS}\right)}^2+{\lambda }_1\left|{\beta }_j\right|+{\lambda }_1\sum _{k\ne j}\left|{\beta }_k\right| \end{gathered}$$

(10)

where ${\overline{y}}_i^{\left(j\right)}=\sum _{k\ne j}{x}_{ik}{\beta }_k.$ The objective function (10) is differentiable everywhere except for $\left|{\beta }_j\right|.$ The first derivative for $f\left({\widehat{\beta }}_j\right)$ when ${\beta }_j\ne 0$ is given as follows:

$$\frac{\partial f\left({\widehat{\beta }}_j\right)}{\partial {\beta }_j}=\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}\left(y_i-{\overline{y}}_i^{\left(j\right)}\right)+{\widehat{\mathit{\beta }}}_j^{LS}+\frac{1}{n}{\sum }_{i=1}^n{x_{ij}}^2{\beta }_j+{\beta }_j+{\lambda }_1\mathrm{s}\mathrm{g}\mathrm{n}\left({\beta }_j\right)$$

(11)

where $\mathrm{s}\mathrm{g}\mathrm{n}\left({\beta }_j\right)=\left\{\begin{array}{c}1 {\beta }_j>0\\ 0 {\beta }_j=0\\ -1{ \beta }_j<0\end{array}\right.$. The soft threshold estimate of ${\beta }_j$ was obtained by setting Equation (11) to zero. Hence,

$${\beta }_j\leftarrow \frac{S\left(\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}\left(y_i-{\overline{y}}_i^{\left(j\right)}\right)-{\widehat{\mathit{\beta }}}_j^{LS},{\lambda }_1\right)}{\frac{1}{n}{\sum }_{i=1}^n{x_{ij}}^2+1}$$

(12)

where $y_i-{\overline{y}}_i^{\left(j\right)}$ is the partial residual for fitting ${\beta }_j$, and S(z,${\lambda }_1$) is the soft-thresholding operator defined by [22] as follows:

$$\mathrm{S}(z, {\lambda }_1)=\left\{\begin{array}{c}z+{\lambda }_1, z<{\lambda }_1\\ 0, \left|z\right|\le {\lambda }_1\\ z-{\lambda }_1, z>{\lambda }_1\end{array}\right.$$

(13)

3. Simulation Studies

In this section, we designed three different cases to explore the performances of the proposed method (KL1) and other estimators. The simulation model is defined as follows:

$$y=X\beta +\sigma ϵ$$

(14)

where the random error is $\mathit{ϵ}~N\left(0,I\right).$ The predictor $X$ is generated such that $X~N_p(0, \sum )$ where ${\sum }_{kj}={\rho }^{|k-j|}$. $\rho$ is the correlation between the predictors. Each independent simulated dataset is divided into three parts: a training set, a validation set and a test set. The division is performed at a ratio of 60%:20%:20%. The training data were used to train the model, while the validation data were used to predict the response while tuning the model’s hyperparameters. The tuning parameters used in the coordinate descent algorithm (Algorithm 1) specify a grid in the interval [0, 1]. The test data are used to provide an unbiased evaluation of a final model fit on the training dataset. The simulation is conducted in two senses of dimensionality—high-dimensional data and low-dimensional data.

Algorithm 1 Pseudo-code for the coordinate descent algorithm

Input: maximum iteration (max_iter),    tolerance (tol),hyperparameters (${\lambda }_1$),    predictors (matrix X), response (vector Y) Initialize:  ${\tilde{\beta }}_j, j=1,\dots ,p$ While iter < max_iter do:    ${\beta }_j\leftarrow {\tilde{\beta }}_j$    While $j<p$ do:      $u_j\leftarrow \frac{1}{n}{\sum }_{i=1}^n{x}_{ij}\left(y_i-{\overline{y}}_i^{\left(j\right)}\right)-{\widehat{\mathit{\beta }}}_j^{LS}$      $v_j\leftarrow \frac{1}{n}{\sum }_{i=1}^n{x_{ij}}^2+1$      ${\beta }_j\leftarrow S\left(\frac{u_j}{v_j},\frac{{\lambda }_1}{v_j}\right)$    End While    If ${\left(\tilde{\beta }-\beta \right)}^{\prime }\left(\tilde{\beta }-\beta \right)<tol$:     Break    Else:      ${\tilde{\beta }}_j\leftarrow {\beta }_j$ End While Output: ${\beta }_j, j=1,\dots ,p$

The low-dimensional data simulation was carried out for sample sizes of $n=100, 200, \mathrm{and} 400$. The simulation was also computed to determine the various correlations between the predictors and variances such that $\rho =0.5, 0.8, \mathrm{and} 0.99$ and $\sigma =5, 10$. We simulated 8 true $\beta$s with 50 datasets for different cases as follows:

Case 1: $\beta ={\left(3,1.5,0,0,2,0,0,0\right)}^{\prime }$

Case 2: $\beta ={\left(3,0,0,0,0,0,0,0\right)}^{\prime }$

Case 3: $\beta ={\left(3,1.5,2,1.6,2,1.5,1.5,1.5\right)}^{\prime }$

The high-dimensional data simulation was carried out for sample sizes of $n=150 \mathrm{a}\mathrm{n}\mathrm{d} 200$. The simulation was also computed to determine the various correlations between the predictors and variances such that $\rho =0.5, 0.8, \mathrm{a}\mathrm{n}\mathrm{d} 0.99$ and $\sigma =5, 10$. We simulated 100 true $\beta$s with 50 datasets for different cases as follows:

Case 1: $\beta ={\left(\underset{15}{\underbrace{0.5,\dots ,0.5}},\underset{25}{\underbrace{0,\dots ,0}},\underset{15}{\underbrace{0.7,\dots ,0.7}},\underset{30}{\underbrace{0,\dots ,0}},\underset{15}{\underbrace{0.7,\dots ,0.7}}\right)}^{\prime }$

Case 2: $\beta ={\left(\underset{15}{\underbrace{0.5,\dots ,0.5}},\underset{25}{\underbrace{0,\dots ,0}},\underset{15}{\underbrace{0,\dots ,0}},\underset{30}{\underbrace{0,\dots ,0}},\underset{15}{\underbrace{0,\dots ,0}}\right)}^{\prime }$

Case 3: $\beta ={\left(\underset{15}{\underbrace{0.5,\dots ,0.5}},\underset{25}{\underbrace{0.2,\dots ,0.2}},\underset{15}{\underbrace{0.7,\dots ,0.7}},\underset{30}{\underbrace{0.4,\dots ,0.4}},\underset{15}{\underbrace{0.9,\dots ,0.9}}\right)}^{\prime }$

The model performance was measured based on the mean square error. The mean square errors of the predictions from the test data were estimated as given in (15). The mean square error for estimating true coefficients β-MSE through the training model was also observed in (16). The mean and standard errors of these MSEs were recorded and reported. We also reported the median of the false positives (FP) and false negatives (FN) involved in estimating the true coefficients of the model. By false positive, we mean the coefficients that were initially zero from the simulation model specification but returned a positive real number as the estimated coefficient. False-negative coefficients return negative estimated coefficients, though the true coefficients are positive. This allows us to see how well the estimator reports the significant coefficients when it performs dimension reduction.

$$TMSE=\frac{1}{n_{test}}{\left({\widehat{y}}_{test}-y_{test}\right)}^{\prime }\left({\widehat{y}}_{test}-y_{test}\right)$$

(15)

$$\beta MSE=\frac{1}{R}{\left(\beta -\widehat{\beta }\right)}^{\prime }\left(\beta -\widehat{\beta }\right)$$

(16)

The simulation results of the low-dimensional and high-dimensional simulations are reported in Supplementary Materials (Sections S1 and S2). We also report the spread of the TMSEs for the test prediction and βMSE for true coefficients for the high-dimensional cases in Supplementary Materials (Section S3).

A general observation of the results of the simulated data (Supplementary Materials, Sections S1 and S2) indicates that the values of the MSEs of the estimators increase as the standard deviation ($\sigma$) and the degree of multicollinearity ($\rho$) increase. There are also decreases in the values of the MSEs of the estimators as the sample size increases. Although our main focus is feature reduction in high-dimensional data, we observe a prediction for our proposed estimator better than those of other estimators.

For the low-dimensional data result discussed in case 1 (Supplementary Materials, Section S1, Tables S1–S3), we observed that the KL1 and GO estimators performed better than the other estimators. The KL1 estimator performed better at predicting the test sample based on the MSE criterion, while the GO estimator had the minimum MSE for estimating the beta coefficients. In case 2 (Supplementary Materials, Section S1, Tables S4–S6), the KL1 estimator performed better both at model prediction and beta coefficient estimation. The performance of the KL1 estimator in case 3 (Supplementary Materials, Section S1, Tables S7–S9) was the same as that in case 1. In the high-dimensional simulation result, the KL1 estimator performed better in model prediction, beta coefficient estimation and sensitivity with regard to detecting the right coefficients using FN and FP (Supplementary Materials, Section S2, Tables S10–S15).

Furthermore, though the GO estimator may compete with the KL1 estimator in the MSEs in terms of estimating the beta coefficients for the low-dimensional data, it is more appropriate to look at the FP and the FN. The FP and the FN proves that the KL1 estimator is preferable to the GO estimator where feature reduction is needed, that is, in cases 1 and 2.

4. Real-Life Application

In this section, we analyzed two datasets to evaluate the performances of the following estimators: the ridge, LASSO (L), E-net (E), GO, Liu-LASSO (LL) and the proposed (KL1) estimators. We used two datasets: the prostate cancer dataset and the asphalt binder dataset. Each dataset was analyzed by splitting them into two—training (80%) and test (20%) data. Model fitting and parameter tunning were carried out via k-fold cross classification, with $k=10$. The tuning parameters were selected from a sequence 0 to 1. We computed the average errors over the folds for each of the tuning parameters. We chose the tuning parameters that minimized the mean square error. The coefficients from the better model were used to predict the response in the test sample, and the model performance was judged via the test mean squared error (TMSE), which is defined as follows:

$$\mathrm{T}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{n_{test}}{\sum }_{i=1}^{n_{test}}{\left({\mathit{y}}_{test}-{\mathit{x}}_i^T\widehat{\mathit{\beta }}\right)}^2$$

(17)

Figure 1 below gives the general framework of the analytical process employed in the analysis of the real-life dataset. This process is followed through for implementing all the estimation techniques outlined in Section 2.

4.1. Dataset I (Prostate Cancer Data)

The prostate data are employed to model the relationship between the level of prostate-specific antigen (lpsa) and a number of clinical measures derived from men who were about to receive a radial prostatectomy [1,21,22]. The model is defined as follows:

$${\mathit{y}}_i={\mathit{\beta }}_0+{\mathit{\beta }}_1{\mathit{x}}_{i1}+{\mathit{\beta }}_2{\mathit{x}}_{i2}+\dots +{\mathit{\beta }}_8{\mathit{x}}_{i8}+{\mathit{ϵ}}_i i=1,\dots 97,$$

(18)

where response ${\mathit{y}}_i$ is the level of prostate-specific antigen, ${\mathit{x}}_{i1}$ is the logarithm of cancer volume, ${\mathit{x}}_{i2}$ is the logarithm of prostate weight, ${\mathit{x}}_{i3}$ is age in years, ${\mathit{x}}_{i4}$ is the natural logarithm of the amount of benign prostatic hyperplasia, ${\mathit{x}}_{i5}$ is seminal vesicle invasion, ${\mathit{x}}_{i6}$ is the logarithm of capsule penetration, ${\mathit{x}}_{i7}$ is the Gleason score, and ${\mathit{x}}_{i8}$ is a percentage of Gleason score 4 or 5.

Table 1. Summary statistics of the prostate cancer dataset.

Feature	Mean	SD	Kurtosis	Skewness
y	2.478387	1.154329	0.588369	−0.00043
lcavol	1.35001	1.178625	−0.51681	−0.2503
lweight	3.652686	0.496631	5.528852	1.216012
age	63.86598	7.445117	1.162491	−0.82848
lbph	0.100356	1.450807	−1.75144	0.133813
svi	0.216495	0.413995	−0.04574	1.398441
lcp	−0.17937	1.39825	−0.95909	0.728634
gleason	6.752577	0.722134	2.670319	1.26048
pgg45	24.38144	28.20403	−0.26962	0.968105

gives the statistical summary of the dataset and the correlation plot in Figure 2. The response variable was centered, while the predictors were standardized. The following estimators were compared: the ridge estimator, the LASSO estimator, the Liu-LASSO estimator, the GO estimator, the elastic-net (E-net), and the KL1 estimator. The estimated TMSE and the regression coefficients are provided in

Table 2. Regression coefficients and $\mathbf{T}\mathbf{M}\mathbf{S}\mathbf{E}$.

Coef.	LASSO	Liu-LASSO	Goest	E-Net	KL1	Ridge
lcavol	0.0139	0.0139	0.0156	0.0141	0.0139	0.0157
lweight	0.9721	0.9685	0.9737	0.9687	0.9641	0.9699
age	−0.0024	−0.0011	−0.0049	−0.0012	0.0000	−0.0037
lbph	0.0006	0.0006	0.0007	0.0006	0.0006	0.0007
svi	0.0000	0.0000	−0.0015	0.0000	0.0000	−0.0021
lcp	0.0000	0.0000	−0.0021	0.0000	0.0000	−0.0036
gleason	0.0046	0.0066	0.0096	0.0071	0.0091	0.0127
pgg45	−0.0035	−0.0035	−0.0036	−0.0034	−0.0034	−0.0035
TMSE	2.6167	2.6109	2.6329	2.6125	2.6038	2.6284

.

The results in Table 1 show that the LASSO estimator, Liu-LASSO estimator and E-Net selected six active sets. The GO estimator failed to shrink any of the variables to zero for this particular data, and the ridge estimator, as expected, only performed regularization. The proposed estimator, namely KL1, selected five active sets and possessed the lowest test mean squared error. This makes the KL1 estimator a useful fit for this dataset.

4.2. Dataset 2 (Asphalt Binder Data)

The data were employed to model the effects of some chemical compositions on surface free energy [8]. The model was defined as follows:

$${\mathit{y}}_i={\mathit{\beta }}_0+{\mathit{\beta }}_1{\mathit{x}}_{i1}+{\beta }_2{\mathit{x}}_{i2}+\dots +{\beta }_{12}{\mathit{x}}_{i12}{+\mathit{ϵ}}_i i=1,\dots 23,$$

(19)

where ${\mathit{y}}_{\mathit{i}}$ denotes surface free energy, ${\mathit{x}}_{i1}$ denotes saturates, ${\mathit{x}}_{i2}$ denotes aromatics, ${\mathit{x}}_{i3}$ denotes resins, ${\mathit{x}}_{i4}$ denotes asphaltenes, ${\mathit{x}}_{i5}$ denotes wax, ${\mathit{x}}_{i6}$ denotes carbon, ${\mathit{x}}_{i7}$ denotes hydrogen, ${\mathit{x}}_{i8}$ denotes oxygen, ${\mathit{x}}_{i9}$ denotes nitrogen, ${\mathit{x}}_{\mathit{i}10}$ denotes sulfur, ${\mathit{x}}_{i11}$ denotes nickel and ${\mathit{x}}_{i12}$ denotes vanadium. The summary statistics are presented in

Table 3. Summary statistics for asphalt dataset.

Feature	Mean	SD	Kurtosis	Skewness
y	18.4213	4.104044	−0.11539	1.157349
x1	8.817391	2.863509	0.354511	−0.56121
x2	34.62174	5.639951	−0.58327	0.298716
x3	40.40435	7.272831	0.424932	0.025278
x4	14.67391	5.876549	−0.59483	−0.44598
x5	2.766087	1.417686	−1.54606	0.032829
x6	84.48261	1.745293	−1.21593	−0.1494
x7	10.36478	0.332181	0.463533	0.679561
x8	0.966522	0.297911	4.368568	1.163691
x9	0.699565	0.252991	−0.24003	0.514575
x10	4.371739	2.15229	−1.17692	−0.07012
x11	71.34783	41.79562	−1.14499	0.434363
x12	243.1739	354.6355	8.152615	2.915543

and the correlation plot in Figure 3. The response variable is centered, and the predictors are standardized to have zero mean and unit standard deviation before model fitting. The following estimators were compared: the ridge estimator, the LASSO estimator, the Liu-LASSO estimator, the GO estimator, the elastic-net (E-net), and the KL1 estimator. The estimated TMSE and the regression coefficients are provided in

Table 4. Regression coefficients and ${\mathbf{M}\mathbf{S}\mathbf{E}}_{\mathbf{y}}$.

Coef.	LASSO	Liu-LASSO	Goest	E-Net	KL1	Ridge
saturates	0.0010	0.0010	0.0012	0.0010	0.0010	0.0012
aromatics	0.9323	0.9317	0.9326	0.9317	0.9310	0.9320
resins	−0.0524	−0.0523	−0.0526	−0.0523	−0.0521	−0.0525
asphaltenes	−0.0600	−0.0600	−0.0600	−0.0600	−0.0600	−0.0600
wax	−0.0337	−0.0337	−0.0338	−0.0337	−0.0336	−0.0338
carbon	−0.0070	−0.0070	−0.0109	−0.0066	−0.0071	−0.0113
hydrogen	0.0621	0.0621	0.0623	0.0621	0.0621	0.0623
oxygen	0.0134	0.0133	0.0138	0.0134	0.0133	0.0138
nitrogen	0.0000	0.0000	0.0000	0.0000	0.0000	0.0041
sulfur	0.0000	0.0000	0.0098	0.0000	0.0000	0.0123
nickel	−0.0027	−0.0027	−0.0070	−0.0025	−0.0026	−0.0084
vanadium	−0.0004	−0.0004	−0.0003	−0.0004	−0.0004	−0.0003
TMSE	9.1905	9.1849	9.2127	9.1862	9.1781	9.2165

. The ridge estimator did not shrink any of its coefficients, as expected. LASSO estimator, the Liu-LASSO estimator, the E-net estimator and the KL1 estimator selected ten active sets for the regression coefficients. Of all the estimators, the KL1 estimator possessed the smallest test mean squared error.

5. Conclusions

In this study, we proposed a new penalized operator (KL1). It is a shrinkage and feature selection method that adopts the coordinate descent algorithm. The KL1 is an equivalent form of the Elastic net and the Liu-LASSO, which produces a sparse model with high prediction accuracy. The new estimator inherits the selection features of estimators such as Lasso and Elastic-net and equally possesses the stability of the ridge estimator. We applied the new estimator and the existing ones to two dataset (health data and chemical data). The application and simulation results supported the performance of KL1 and its dominance over the ridge estimator, elastic-net, GO, Liu-LASSO and LASSO. Although, the existing methods compete favorably.