Relaxed Adaptive Lasso and Its Asymptotic Results

Abstract

This article introduces a novel two-stage variable selection method to solve the common asymmetry problem between the response variable and its influencing factors. In practical applications, we cannot correctly extract important factors from a large amount of complex and redundant data. However, the proposed method based on the relaxed lasso and the adaptive lasso, namely, the relaxed adaptive lasso, can achieve information symmetry because the variables it selects contain all the important information about the response variables. The goal of this paper is to preserve the relaxed lasso’s superior variable selection speed while imposing varying penalties on different coefficients. Additionally, the proposed method enjoys favorable asymptotic properties, that is, consistency with a fast rate of convergence with $O_p\left(n^{-1}\right)$. The simulation demonstrates that the proper variable recovery, i.e., the number of significant variables selected, and prediction accuracy of the relaxed adaptive lasso in a limited sample is superior to the regular lasso, relaxed lasso and adaptive lasso estimators.

1. Introduction

Rapid advancements in research and technology have resulted in enormous data in a variety of scientific domains. How to efficiently extract information from complex data and develop an ideal model that relates critical features to response variables has become a challenge for researchers in the data explosion era. Over the past two decades, statisticians have conducted substantial research on the subject of feature selection.

Tibshirani [1] first proposed lasso, a technique for screening high-dimensional variables that improves least squares estimation by including an $L_1$ penalty component. The penalty parameter of the lasso set some of the coefficients to zero, thus achieving the proposal of coefficient shrinkage and model selection. Lasso sacrifices unbiasedness for minimizing variance and solves the convex optimization problem to find the globally optimal solution. In the rare signal scenario, when the signal strength exceeds a certain level, lasso shows good performance, far outperforming other variable selection methods [2]. However, Meinshausen and Bühlmann [3] discovered a conflict in the lasso model between optimal prediction and consistent variable selection, which is one of the lasso’s downsides. Due to the strong sensitivity of the lasso to the presence of correlation and multicollinearity in real data, insignificant noisy variables may be selected for the model. As a result, the noise variables in the model exacerbate the model fitting effect. Fan and Li [4] presented a more adaptive novel approach for maximizing the likelihood penalty function that applies to generalized linear models and other types of models. Moreover, Fan and Li [5] enhanced the preceding approach and stated that as long as the dimensionality of the model is not too large, the penalized likelihood technique can be used to estimate the model’s parameters via the penalty function. To address the issue of inconsistent lasso selection, Zou [6] reported an adaptive lasso estimator,

$${\widehat{\beta }}^{Alasso}=\underset{\beta }{argmin}{∥Y-\sum _{j=1}^p{X}_j^T{\beta }_j∥}_2^2+\lambda \sum _{j=1}^p{\widehat{\omega }}_j\left|{\beta }_j\right|,$$

(1)

where $\widehat{\omega }=1/{\left|\widehat{\beta }\right|}^{\gamma },\gamma >0$. The primary reason why adaptive lasso is superior to lasso is that it has an oracle quality that depends on the weight vector value $\widehat{\omega }$. Without this quality, adaptive lasso’s oracle property would be suboptimal. Fan and Peng [7] claimed that when the dimension p is less than the sample size n, the lasso and adaptive lasso can both be used to accelerate and optimize variable selection. The theory of Donoho and Johnstone can also be used to demonstrate the adaptive lasso’s near-minimax optimality [8]. The non-negative garotte [9] is another regularization method. It can be considered as a special case of the adaptive lasso and was proved to have the property of consistent variable screening [10].

Meinshausen [11] defined the relaxed lasso estimator on the set $M\subseteq \left\{1,\dots ,p\right\}$, where p is the number of nonzero variables selected into the true model,

$${\widehat{\beta }}^{Rlasso}=\underset{\beta }{argmin}{∥Y-\sum _{j=1}^p{X}_j^T\left\{{\beta }_j·1_M\right\}∥}_2^2+\varphi \lambda \sum _{j=1}^p\left|{\beta }_j\right|,$$

(2)

where $\lambda \in \left[0,\infty \right],\varphi \in \left(0,1\right]$, $1_M$ is an indicator function, $1_M=\left\{\begin{array}{cc}0,\hfill & k\in M\hfill \\ 1,\hfill & k\notin M\hfill \end{array}\right.$, for all $k\in \left\{1,\dots ,p\right\}$. Hastie et al. [12] compared the performance of lasso and forward stepwise regression across a range of signal-to-noise ratios (SNRs) and showed that it is extremely competitive in any environment. The relaxation parameter $\varphi$ contributes to relaxed lasso’s superior performance. By adjusting the control parameter $\varphi$ appropriately, it can ensure that the sparse solution on the path does not experience excessive shrinkage. This is the primary reason we chose to expand the model using a relaxed lasso. In recent works, numerous studies have demonstrated that the relaxed lasso has excellent performances compared to other methods. Mentch and Zhou [13] showed that in high-dimensional settings, lasso, forward selection and randomized forward selection perform similarly at low SNRs, but for larger SNRs, relaxed lasso performs much better in terms of the relative test error. Bloise et al. [14] suggested that relaxed lasso is able to avoid overfitting by using two separate tuning parameters so as to obtain a more accurate model. Comparing relaxed Lasso to least squares and stepwise regression, He [15] came to the conclusion that relaxed Lasso improves the accuracy of the model by deleting insignificant variables. Kang et al. [16] proposed a new method that combines the relaxed lasso and a generalized multiclass support vector machine to obtain fewer feature variables and higher classification accuracy. Tay et al. [17] combined elastic net regularized regression with a simplified relaxed lasso model and built a prediction matrix to measure model performance, which speeds up the computational efficiency of the model.

We discuss the properties of different variable selection methods in the case of large samples. Consistency and asymptotic normality are two large sample properties of OLS; for consistency, we can assume a weaker overall zero-correlation assumption $Cov\left(x,\epsilon \right)=0$ and a zero-mean assumption of error $E\left(\epsilon \right)=0$. Fu and Knight [18] examined the consistency and asymptotic features of bridge estimation in convex and nonconvex scenarios and established that lasso is consistent when certain conditions are met. Thus, another disadvantage of lasso is that variable selection is conditional, which means that it does not work similarly to an oracle estimator. Zhao and Yu [19] claimed that the irrepresentable condition is a necessary and sufficient criterion for lasso to satisfy consistency. However, both adaptive lasso and relaxed lasso have been proved to be consistent without satisfying the strict condition. Zou [6] demonstrated that even with huge quantities of data, adaptive lasso can efficiently filter out the model’s sparse solution while retaining oracle features. According to Meinshausen [11], relaxed lasso can still retain a high rate of convergence with $O_p\left(n^{-1}\right)$ and can lead to consistent variable selection no matter what the asymptotic result is. To combine the advantages of the preceding two models, we propose a relaxed adaptive lasso and demonstrate that it has the same asymptotic properties and excellent convergence rate as the relaxed lasso. The Lars algorithm [20] and an improved algorithm have been shown to solve the relaxed adaptive lasso.

In this paper, we propose a novel variable filtering method named relaxed adaptive lasso, which can effectively address the model selection issue, and we demonstrate the method’s asymptotic properties. We prove that the relaxed adaptive lasso estimator can achieve the same rate of convergence as the relaxed lasso, indicating that it can obtain the sparse solution at the optimal rate. The simulation of this study demonstrates that the relaxed adaptive lasso performs well in variable recovery and predictin accuracy. In particular, when sample size n and the number of variables p are varied, the performance of the relaxed adaptive lasso to filter true nonzero variables is superior to that of the lasso, relaxed lasso, and adaptive lasso. As the sample size n increases, the mean square error (MSE) of the model remains the best.

The rest of this article is structured as follows: Section 2 defines relaxed adaptive lasso, describes its computational algorithms, and establishes its asymptotic properties. Then, in Section 3, we compare the performance of the relaxed adaptive lasso to that of the lasso, adaptive lasso, and relaxed lasso using a simulation experiment. Section 4 discusses the application of real-world data. Section 5 makes a conclusion of the proposed method. The Appendix A, Appendix B, Appendix C, Appendix D, Appendix E, Appendix F, Appendix G contains additional information about the proof.

2. Relaxed Adaptive Lasso and Asymptotic Results

2.1. Definition

Recall that adaptive lasso estimation improves the shrinkage force to equalize the coefficients in lasso by applying a weight vector. The adaptive lasso estimator’s set of predictor variables ${\widehat{\beta }}^{\lambda ,\omega }$ is denoted by $S^{\lambda ,\omega }$,

$$S^{\lambda ,\omega }=\left\{1\le k\le p\mid {\widehat{\beta }}_k^{\lambda ,\omega }\ne 0\right\}.$$

(3)

The solution of the relaxed adaptive lasso is obtained via the adaptive lasso estimator in $S^{\lambda ,\omega }$ if and only if in the low-dimension case.

We now consider the linear regression model

$$Y=X^T{\beta }^{*}+\epsilon ,$$

(4)

where $\epsilon ={\left({\epsilon }_1,\dots ,{\epsilon }_n\right)}^T$ is a vector composed of i.i.d. random variables with mean 0 and variance ${\sigma }^2$. $X=\left(X_1,\dots ,X_p\right)$ is an $n\times p$ matrix with a normally distribution $X\sim N\left(0,\mathrm{\Sigma }\right)$, where $X_i$ is the ith column and Y is an $n\times 1$ vector of response variables. Now, we define relaxed adaptive lasso estimation. The variable selection and shrinkage are controlled by adding two constraints, $\lambda$ and $\varphi$, and one weight vector, $\omega$, to the $L_1$ penalty term. According to the setup of Zou [6], suppose that $\widehat{\beta }$ is an $\sqrt{n}$-consistent estimator of ${\beta }^{*}$.

Definition 1.

Define the relaxed adaptive lasso estimator as ${\widehat{\beta }}^{\lambda ,\omega }$ denoted by $S^{\lambda ,\omega }$,

$${\widehat{\beta }}^{*}=\underset{\beta }{argmin}{∥Y-\sum _{j=1}^p{X}_j^T\left\{{\beta }_j·1_{S^{\lambda ,\omega }}\right\}∥}_2^2+\varphi \lambda \sum _{j=1}^p{\widehat{\omega }}_j\left|{\beta }_j\right|,$$

(5)

where $1_{S^{\lambda ,\omega }}$ is an indicator function ${\left\{1_{S^{\lambda ,\omega }}\right\}}_k=\left\{\begin{array}{cc}1,\hfill & k\in S^{\lambda ,\omega }\hfill \\ 0,\hfill & k\notin S^{\lambda ,\omega }\hfill \end{array}\right.$, for all $k\in \left\{1,\dots ,p\right\}$; $\varphi \in$ [0,1]; given a $\gamma >0$, define the weight vector $\widehat{\omega }=1/{\left|\widehat{\beta }\right|}^{\gamma }$.

Notably, only predictor variables in the set $S^{\lambda ,\omega }\subseteq \left\{1,\dots ,p\right\}$ can be chosen as the relaxed adaptive lasso solution. In the following, we discuss different functions and value ranges of parameters under the set $S^{\lambda ,\omega }$. The parameter $\lambda \ge 0$ determines the number of variables retained in the model. For $\lambda =0$ or $\varphi =0$, the problem of solving the estimators in Equation (5) is transformed into an ordinary least squares problem where $S_0^{\lambda ,\omega }=\left\{1,\dots ,p\right\}$ so that the purpose of variable selection cannot be achieved. As $\lambda$ increases, all coefficients of the variables selected by adaptive lasso are compressed towards 0, and some finally become exactly 0. However, for a large $\lambda \to \infty$, all estimators are shrunk to 0, where $S^{\lambda ,\omega }=\varnothing$, leading to a null model. In addition, the relaxation parameter $\varphi$ controls the amount of shrinkage applied to the coefficients in estimation. When $\varphi =1$, the adaptive lasso and relaxed adaptive lasso estimators are the same. When $\varphi <1$, the shrinkage force on the estimators is weaker than that of the adaptive lasso. The optimal tuning parameters $\lambda$ and $\varphi$ are chosen by cross-validation. The vector $\widehat{\omega }=1/{\left|\widehat{\beta }\right|}^{\gamma }$assigns different weights to the coefficients; hence, the relaxed adaptive lasso has consistency when the weight vector is correctly chosen.

2.2. Algorithm

We will discuss the algorithm for computing the estimator of the relaxed adaptive lasso in this section. Note that (5) is a convex optimization problem, which means that we can obtain the global optimal solution effectively. Unlike concave penalties, however, multiple minimal penalties, such as SCAD, suffer from the multiple minimal problem. In the following, we discuss a simplified version of the relaxed adaptive lasso estimator algorithm. An improved algorithm is then proposed based on the process of computation for the relaxed lasso estimator [11].

The simple algorithm for relaxed adaptive lasso

Step (1).

For a given $\gamma >0$, we use ${\widehat{\beta }}^{OLS}$ to construct the weight in an adaptive lasso based on the definition from Zou [6]. We can also replace ${\widehat{\beta }}^{OLS}$ with other consistent estimators, e.g., ${\widehat{\beta }}^{Ridge}$.

Step (2).

Define $X_j^{*}=X_j/{\widehat{\omega }}_j,j=1,\dots ,p$, where ${\widehat{\omega }}_j=1/{\left|{\widehat{\beta }}^{OLS}\right|}^{\gamma }$

Step (3).

Then, the process of computing relaxed adaptive lasso solutions is identical to that of solving the relaxed lasso solutions in Meinshausen [11]. The relaxed lasso estimator is defined as

$${\widehat{\beta }}^{**}=argmin{∥Y-\sum _{j=1}^p{\left(X_j^{*}\right)}^T\left\{{\beta }_j·1_{S^{\lambda ,\omega }}\right\}∥}_2^2+\varphi \lambda \sum _{j=1}^p\left|{\beta }_j\right|.$$

(6)

The Lars algorithm is first used to compute all the adaptive lasso solutions. Select a total of h resulting models $S_1,\dots ,S_h$ attained with the sorted penalty parameters ${\lambda }_1>{\lambda }_2>\dots >{\lambda }_h=0$. When ${\lambda }_h=0$, for example, all variables with nonzero coefficients are selected, which is identical to the OLS function. On the other hand, ${\lambda }_0=\infty$ completely shrinks the estimators to zero, thus leading to a null model. Therefore, a moderate ${\lambda }_k,k=1,\dots ,h$ in the sequence of $\left\{{\lambda }_1,\dots ,{\lambda }_h\right\}$ is chosen such that $S_k=S^{\lambda ,\omega }$. Then, define the OLS estimator $\tilde{\beta }={\widehat{\beta }}^{{\lambda }_k}+{\lambda }_k\delta \left(k\right)$, where $\delta \left(k\right)=$$\left({\widehat{\beta }}^{{\lambda }_k}-{\widehat{\beta }}^{{\lambda }_{k-1}}\right)/\left({\lambda }_{k-1}-{\lambda }_k\right)$ is the direction of adaptive lasso solutions, which can be obtained from the last step. If there exists at least one component j such that $\mathrm{sgn}\left({\tilde{\beta }}_j\right)\ne \mathrm{sgn}\left({\widehat{\beta }}_j^{{\lambda }_k}\right)$, then all the adaptive lasso solutions on the set $S_k$ of variables are identical to the set of relaxed lasso estimators ${\widehat{\beta }}^{**}$ for $\lambda \in {\mathcal{L}}_{\mathit{k}}$. Otherwise, ${\widehat{\beta }}^{**}$ for ${\lambda }_k\in {\mathcal{L}}_{\mathit{k}}$ are computed by linear interpolation between ${\widehat{\beta }}_j^{{\lambda }_k}$ and ${\tilde{\beta }}_j$.

Step (4).

Output the relaxed adaptive lasso solutions: ${\widehat{\beta }}_j^{*}={\widehat{\beta }}_j^{**}/{\widehat{\omega }}_j,j=1,\dots ,p$.

Simple algorithms have the same computational complexity as Lars-OLS hybrid algorithms. However, due to the high computing complexity, this approach is frequently not ideal. Then, we consider an improved algorithm introduced by Hastie et al. [12], which uses the definition of the relaxed adaptive lasso estimators to solve a problem of high computational complexity.

The improved algorithm for relaxed adaptive lasso

Step (1).

As before, $S^{\lambda ,\omega }$ denotes the active set of the adaptive lasso. Let ${\widehat{\beta }}^{Alasso}$ denote the adaptive lasso estimator. The relaxed adaptive lasso solution can be defined as

$${\widehat{\beta }}^{*}=\varphi {\widehat{\beta }}^{Alasso}+\left(1-\varphi \right){\widehat{\beta }}^{OLS},$$

(7)

where $\varphi$ is a constant with a value between 0 and 1.

Step (2).

The submatrix $X_{S^{\lambda ,\omega }}$ of active predictors is a reversible matrix; thus, ${\widehat{\beta }}^{OLS}={\left(X_{S^{\lambda ,\omega }}{X}_{S^{\lambda ,\omega }}^T\right)}^{-1}{X}_{S^{\lambda ,\omega }}Y$.

Step (3).

Define $X^{\#}=X_{S^{\lambda ,\omega }}/\widehat{\omega }$, where $\widehat{\omega }=1/{\left|{\widehat{\beta }}^{OLS}\right|}^{\gamma }$; then, the adaptive lasso solution ${\widehat{\beta }}^{ALasso}$ is identical to solving the lasso problem

$${\widehat{\beta }}^{Lasso}=\underset{\beta }{argmin}{∥Y-\sum _{j=1}^p{\left(X_j^{\#}\right)}^T\beta ∥}_2^2+\lambda \sum _{j=1}^p\left|{\beta }_j\right|.$$

(8)

By means of the Karush–Kuhn–Tucker (KKT) optimality condition, the lasso solution over its active set can be written as

$${\widehat{\beta }}^{Lasso}={\left(X_{S^{\lambda ,\omega }}^{\#}{\left(X_{S^{\lambda ,\omega }}^{\#}\right)}^T\right)}^{-1}\left(X_{S^{\lambda ,\omega }}^{\#}Y-\lambda \mathrm{sgn}\left({\widehat{\beta }}^{Lasso}\right)\right).$$

(9)

From the transformation of the predictor matrix in Step (2), it follows that the adaptive lasso estimator is ${\widehat{\beta }}^{Alasso}={\widehat{\beta }}^{Lasso}/\widehat{\omega }$.

Step (4).

Thus the improved solution of the relaxed adaptive lasso can be written as

$${\widehat{\beta }}_j^{*}=\left\{\begin{array}{cc}\frac{\varphi }{\widehat{\omega }}{\left(X_{S^{\lambda ,\omega }}^{\#}{\left(X_{S^{\lambda ,\omega }}^{\#}\right)}^T\right)}^{-1}\left(X_{S^{\lambda ,\omega }}^{{}^{\#}}Y-\lambda \mathrm{sgn}\left({\widehat{\beta }}^{Lasso}\right)\right)+\left(1-\varphi \right){\left(X_{S^{\lambda ,\omega }}{X}_{S^{\lambda ,\omega }}^T\right)}^{-1}{X}_{S^{\lambda ,\omega }}Y,\hfill & j\in S^{\lambda ,\omega },\hfill \\ 0,\hfill & j\notin S^{\lambda ,\omega }.\hfill \end{array}\right.$$

(10)

The computational complexity of Algorithm 1 in the best case is equivalent to the ordinary lasso. Specifically, in Step (3) of the simple algorithm, the relaxed adaptive lasso estimator can be solved in the same way as the relaxed lasso. The improved algorithm is computed from the adaptive lasso and lasso estimators. Given the weight vector, the computational cost of the relaxed adaptive lasso is the same as that of the lasso [21]. Therefore, the computational complexity of Algorithm 2 is equivalent to that of the lasso.

Now we compare the computational cost of the two algorithms. The relaxed lasso’s computational cost in the worst scenario is $O\left(n^{3}p\right)$, which is slightly more expensive than the cost of the regular lasso with $O\left(n^{2}p\right)$ Meinshausen [11]. For this reason, we compute the relaxed adaptive lasso estimator using the improved algorithm.

Algorithm 1. The simple algorithm for relaxed adaptive lasso.

Input: a given constant $\gamma >0$, the weight vector $\widehat{\omega }=1/{\left|{\widehat{\beta }}^{OLS}\right|}^{\gamma }$,

Precompute:$X^{*}=X/\widehat{\omega }$

Initialization: Let ${\lambda }_1>{\lambda }_2>\dots >{\lambda }_h$ to be the optimal parameter

corresponding to the modified models $S_1,\dots ,S_h$.

Set $k=1$ to an initial order number of ${\lambda }_k$

Define $Q\left(\beta \right)={∥Y-{\displaystyle \sum _{j=1}^p}{\left(X_j^{*}\right)}^T\left\{{\beta }_j·1_{S^{\lambda ,\omega }}\right\}∥}_2^2+\varphi \lambda {\displaystyle \sum _{j=1}^p}\left|{\beta }_j\right|$,

$\tilde{\beta }={\widehat{\beta }}^{{\lambda }_k}+{\lambda }_k\delta \left(k\right)$, where $\delta \left(k\right)=$$\left({\widehat{\beta }}^{{\lambda }_k}-{\widehat{\beta }}^{{\lambda }_{k-1}}\right)/\left({\lambda }_{k-1}-{\lambda }_k\right)$

for$j=1,\dots ,p$do

if $sgn\left({\tilde{\beta }}_j\right)\ne sgn\left({\widehat{\beta }}_j^{{\lambda }_k}\right)$ then

${\widehat{\beta }}^{**}\leftarrow {\widehat{\beta }}^{Alasso}$

else

${\widehat{\beta }}^{**}\leftarrow Q\left(\tilde{\beta }\right)+\frac{Q\left(\tilde{\beta }\right)-Q\left({\widehat{\beta }}^{{\lambda }_{k-1}}\right)}{\tilde{\beta }-{\widehat{\beta }}^{{\lambda }_{k-1}}}\left(\tilde{\beta }-{\widehat{\beta }}^{{\lambda }_{k-1}}\right)$

Set $k=k+1$

until$k=h$

Output:${\widehat{\beta }}_j^{*}={\widehat{\beta }}_j^{**}/{\widehat{\omega }}_j$

Algorithm 2. The improved algorithm for the relaxed adaptive lasso.

Input: Adaptive lasso estimator ${\widehat{\beta }}^{Alasso}$, OLS estimator ${\widehat{\beta }}^{OLS}$,

weight vector $\widehat{\omega }=1/{\left|{\widehat{\beta }}^{OLS}\right|}^{\gamma }$

Precompute:$X^{\#}=X_{S^{\lambda ,\omega }}/\widehat{\omega }$, Let $S^{\lambda ,\omega }$ be the active set of the adaptive lasso

Initialization: Define ${\widehat{\beta }}^{Lasso}=\underset{\beta }{argmin}{∥Y-{\displaystyle \sum _{j=1}^p}{\left(X_j^{\#}\right)}^T\beta ∥}_2^2+\lambda {\displaystyle \sum _{j=1}^p}\left|{\beta }_j\right|$

for$j=1,\dots ,p$do

if $j\in S^{\lambda ,\omega }$ then

compute ${\widehat{\beta }}^{OLS}={\left(X_{S^{\lambda ,\omega }}{X}_{S^{\lambda ,\omega }}^T\right)}^{-1}{X}_{S^{\lambda ,\omega }}Y$,

${\widehat{\beta }}^{Lasso}={\left(X_{S^{\lambda ,\omega }}^{\#}{\left(X_{S^{\lambda ,\omega }}^{\#}\right)}^T\right)}^{-1}\left(X_{S^{\lambda ,\omega }}^{{}^{\#}}Y-\lambda \mathrm{sgn}\left({\widehat{\beta }}^{Lasso}\right)\right)$

else

Stop iterations

until$j=p$

Output:${\widehat{\beta }}^{Alasso}={\widehat{\beta }}^{Lasso}/\widehat{\omega }$, ${\widehat{\beta }}^{*}=\varphi {\widehat{\beta }}^{Alasso}+\left(1-\varphi \right){\widehat{\beta }}^{OLS}$

2.3. Asymptotic Results

To investigate the asymptotic property, we make the following two assumptions about the architecture used in the setup of Fu and Knight [18]:

$$\frac{1}{n}\sum _{i=1}^n{x}_i{x}_i^T\to \mathrm{\Sigma },$$

(11)

where $\mathrm{\Sigma }$ is a positive definite matrix. Furthermore,

$$\frac{1}{n}\underset{1\le i\le n}{max}{x}_i^T{x}_i\to 0.$$

(12)

Without loss of generality, the sparse constant vector $\beta$ is defined as the true coefficient of the model. We assume that the number of nonzero estimators selected into the real model is q, that is $\beta =\left({\beta }_1,\dots ,{\beta }_q,0,0,\dots \right)$, where ${\beta }_j\ne 0$ only for $j=1,\dots ,q$ and ${\beta }_j=0$ for $j=q+1,\dots ,p$. The true model is, hence, $S_{*}=\left\{1,\dots ,q\right\}$. The covariance matrix $\mathrm{\Sigma }=\frac{1}{n}X{X}^T$ can be written in block-wise form, i.e., $\mathrm{\Sigma }=\left(\begin{array}{cc}{\mathrm{\Sigma }}_{11}& {\mathrm{\Sigma }}_{12}\\ {\mathrm{\Sigma }}_{21}& {\mathrm{\Sigma }}_{22}\end{array}\right)$, where ${\mathrm{\Sigma }}_{11}$ is a $q\times q$ matrix. The random loss $L\left(\lambda ,\omega \right)$ of the adaptive lasso is defined as

$$L\left(\lambda ,\omega \right)=E{\left(Y-X^T{\widehat{\beta }}^{Alasso}\right)}^2-{\sigma }^2.$$

(13)

The loss $L\left(\lambda ,\varphi ,\omega \right)$ of the relaxed adaptive lasso is analogously defined as

$$L\left(\lambda ,\varphi ,\omega \right)=E{\left(Y-X^T{\widehat{\beta }}^{*}\right)}^2-{\sigma }^2.$$

(14)

We discover that the relaxed adaptive lasso estimator has the same rapid convergence rate as the relaxed lasso estimator when the exponential growth rate of the size p is ignored. Additionally, the adaptive lasso has a slower pace than both of them but is slightly faster than the lasso estimator. We make the following assumptions concerning asymptotic results for low-dimensional sparse solutions to demonstrate the above conclusion.

Assumption 1.

The number of predictors $p=p_n$ increases exponentially with the number of observations n, that is, there exist some $c>0$, $0<s<1$ such that $p_n\sim s{e}^{cn}$.

We cannot rule out the possibility that the remaining $p_n-q$ noise factors are linked with the response. A square matrix is said to be diagonally dominant if the magnitude of the diagonal entry in each row of the matrix is greater than or equal to the sum of the magnitudes of all the other (nondiagonal) entries in that row.

Assumption 2.

Σ and ${\mathrm{\Sigma }}^{-1}$ are diagonally dominant at some constant $c<0$, for all $n\in \mathbb{N}$.

Notably, when the diagonal is positive, the diagonally dominating symmetric matrix is positive definite. Based on this premise, the inverse matrix of $\mathrm{\Sigma }$ can guarantee its existence.

Assumption 3.

We limit the penalty parameter λ to the range $\mathcal{L}$,

$$\mathcal{L}=\left\{\lambda \ge 0:c{e}^{p_n}\le n\right\},$$

(15)

if and only if there exists an arbitrarily large $c>0$.

Assumption 3 holds true if the exponent of the number of variables in the selected model is less than the sample size n. Using $\lambda$ values in the range $\mathcal{L}$, relaxed lasso, adaptive lasso, and relaxed adaptive lasso can obtain consistent variable selection and a specified number of nonzero coefficients.

Lemma 1.

Assume that predictor variables are independent of each other, ${\lambda }_n$, $n\in \mathbb{N}$ is the penalty parameter of the adaptive lasso, and its order is ${\lambda }_n=O\left(n^{\frac{s-1-2\gamma }{2}}\right)$ for $n\to \infty$. Under Assumptions 1–3,

$$P\left(\exists k>q:k\in S_{{\lambda }_n}\right)\to 1,n\to \infty .$$

(16)

As a result of Lemma 1, the chance of at least one noise variable being evaluated as nonzero is close to one. We prove Theorem 1 by utilizing the conclusion of Lemma 1 on the order of the penalty parameter.

Lemma 2.

Let $lim\underset{n\to \infty }{inf}\frac{n^{*}}{n}\to \frac{1}{A}$ with $A\ge 2$, $n^{*}$ being the number of observations. Then, under Assumptions 1–3,

$$\underset{\lambda \in \mathcal{L},\gamma >0}{sup}\left|L\left(\lambda ,\varphi ,\omega \right)-L_{n^{*}}\left(\lambda ,\varphi ,\omega \right)\right|=O_p\left(n^{-1}{log}^{2}n\right),n\to \infty .$$

(17)

We want to investigate the computational cost of the specified parameters by examining the order of the relaxed adaptive lasso loss function. Lemma 2 is a technique that will assist us in proving Theorem 3.

Lemma 3.

Assume that predictor variables are independent of each other, ${\lambda }_n$, $n\in \mathbb{N}$ is the penalty parameter of the relaxed adaptive lasso, and $n^{s+1}{\lambda }_n^3\to \infty$ for $n\to \infty$. Under Assumptions 1–3,

$$P\left(\exists k>q:k\in S_{{\lambda }_n}\right)\to 0.$$

(18)

As a result of Lemma 3, the noise variable can be predicted to be 0. If the penalty parameter ensures that ${\lambda }_n^3$ converges to 0 at a slower rate than $n^{s+1}$, the noise variable can be precisely evaluated as nonzero with a probability approaching 0. In addition, Lemma 3 helps to prove Theorem 3 by describing the order of the penalty parameter of the relaxed adaptive lasso.

Theorem 1 addresses the question of whether the adaptive lasso can sustain a faster convergence rate as the number of noise variables increases rapidly and the convergence speed exceeds that of the lasso. The addition of the weight parameter enables the adaptive lasso to gain oracle qualities while also increasing the algorithm’s rate of convergence.

Theorem 1.

Assume that predictor variables are independent of each other. Under Assumptions 1–3, $\mathrm{\Sigma }=1$ for any $t>0$ and $n\to \infty$. The convergence rate of the adaptive lasso is as follows:

$$P\left(\underset{\lambda \in \mathcal{L}}{inf}L\left(\lambda ,\omega \right)>t{n}^{-r}\right)\to 1,\forall r>1+2\gamma -s.$$

(19)

On the other hand, Theorem 2 establishes that the convergence rate of the relaxed adaptive lasso is equivalent to that of the relaxed lasso. Theorem 2 resolves the question of whether the convergence rate of the relaxed adaptive lasso is consistent with that of the relaxed lasso by establishing that the convergence rate of the relaxed adaptive lasso is not related to the noise variable’s growth rate r or the parameter s that determines the growth rate.

Theorem 2.

Assume that predictor variables are independent of each other. Under Assumptions 1–3, for $n\to \infty$, the convergence rate of the relaxed adaptive lasso is as follows:

$$\underset{\lambda \in \mathcal{L},\varphi \in \left[0,1\right],\gamma >0}{inf}L\left(\lambda ,\varphi ,\omega \right)=O_p\left(n^{-1}\right).$$

(20)

The shade in Figure 1 represents the rate at which various models converge. The rate of the relaxed adaptive lasso is the same as that of the relaxed lasso; this indicates that the convergence rate of the relaxed adaptive lasso is unaffected by the rapid increase in the noise variable, and it can still retain a high rate. Although the adaptive lasso’s convergence rate is suboptimal, it is faster than the lasso’s due to the presence of the weight vector. The addition of an excessive number of noise variables slows the Lasso estimator, regardless of how the penalty parameter is chosen [11].

The convergence rate of the relaxed adaptive lasso is as robust as the rate of the relaxed lasso, i.e., it is unaffected by noise factors. Theorem 3 demonstrates that cross-validation selection of the parameters $\lambda$, $\varphi$ can still maintain a rapid rate.

Franklin [22] indicated that K-fold cross-validation includes K partitions and each partition consists of $\tilde{n}$ observation data, where $\frac{\tilde{n}}{n}\to \frac{1}{K}$ for $n\to \infty$. When building an estimator on a different set of observations than $\mathit{R}$, define the empirical loss of observations as $L_{R,\tilde{n}}\left(\lambda ,\varphi ,\omega \right)$ for $R=1,\dots ,K$. Let $L_{cv}\left(\lambda ,\varphi ,\omega \right)$ be the empirical loss function,

$$L_{cv}\left(\lambda ,\varphi ,\omega \right)=K^{-1}\sum _{R=1}^K{L}_{R,\tilde{n}}\left(\lambda ,\varphi ,\omega \right).$$

(21)

The selection of $\widehat{\lambda }$, $\widehat{\varphi }$ and $\widehat{\omega }$ is performed by minimizing the loss function $L_{cv}\left(\lambda ,\varphi ,\omega \right)$, that is,

$$\left(\widehat{\lambda },\widehat{\varphi },\widehat{\omega }\right)=argmin{L}_{cv}\left(\lambda ,\varphi ,\omega \right).$$

(22)

This article uses five-fold cross-validation in the numerical study.

Theorem 3.

Under Assumptions 1–3, the convergence rate of K-fold cross-validation with $2\le K<\infty$ holds that

$$L\left(\widehat{\lambda },\widehat{\varphi },\widehat{\omega }\right)=O_p\left(n^{-1}{log}^{2}n\right).$$

(23)

Therefore, when K-fold cross-validation is used to determine the relaxed adaptive lasso’s penalty parameters $\lambda$, $\varphi$, the convergence speed may maintain a relatively ideal outcome. As a result, if using cross-validation to select the penalty parameters, the optimal rate and consistent variable selection under the oracle selection of penalty parameters may be nearly achieved.

Theorem 4.

If $\frac{{\lambda }_n}{n}\to {\lambda }_0\ge 0$, then $\beta ·1_S\stackrel{p}{\to }\beta$ in the relaxed adaptive lasso estimator; moreover, if $\varphi {\lambda }_n=o\left(n\right)$, ${\widehat{\beta }}^{*}$ is consistent.

Theorem 4 indicates that the relaxed adaptive lasso estimator is consistent under the condition $\varphi {\lambda }_n=o\left(n\right)$. ${\widehat{\beta }}^{*}$ does not have to be root-n consistent; nonetheless, the consistency of the relaxed adaptive lasso is determined by the conclusion drawn from probability convergence.

3. Simulation

3.1. Setup

We present a numerical study in this section to compare the performance of the relaxed adaptive lasso to that of the lasso, relaxed lasso, and adaptive lasso. Based on the simulation setup of Meinshausen [11], we considered the linear model $y=x^T\beta +\epsilon$, where $x=\left(x_1,\dots ,x_p\right)$ is the predictor vector and random error $\epsilon$ is an independent and identically distributed random variable with mean 0 and variance ${\sigma }^2$. The remaining parameter settings and procedures are as follows.

i.

Given sample size $n=100,500,1000$ and data dimension $p=20,50$.

ii.

The true regression coefficient $\beta \in {\mathbb{R}}^{\mathit{p}}$ has its first $q=10\left(q\le p\right)$ signal variables taking nonzero coefficients equally spaced from 0.5 to 10 in the sense that ${\beta }_j\ne 0$ for all $j\le q$ and the remaining $p-q$ coefficients are zero.

iii.

The design matrix $X\in {\mathbb{R}}^{\mathit{n}\times \mathit{p}}$ is generated from a normal distribution $N\left(0,\mathrm{\Sigma }\right)$, where covariance matrix $\mathrm{\Sigma }=\mathrm{cov}\left(x\right)={\left(c_{ij}\right)}_{p\times p}$ has entries $c_{ij}=1,i=j=1,\dots ,p$ and $c_{ij}={\rho }^{|i-j|},i\ne j$. The correlation between predictor variables is set to $\rho =0$.5.

iv.

The theoretical signal-to-noise ratio in this simulation is defined as $\mathrm{SNR}=\mathrm{Var}\left(x^T\beta \right)/$${\sigma }^2$. We discuss either SNR = 0.2 for low or SNR = 0.8 for high to calculate the variance of $\epsilon$ so that the response variable Y generated from the linear regression model follows $N_n\left(x^T\beta ,{\sigma }^{2}I\right)$.

v.

We compute the weight of the adaptive lasso via the ridge regression estimator with $\gamma =1$. For each method, five-fold cross-validation is used to select the penalty parameters, and the loss function to apply for cross-validation is chosen by minimizing the prediction error on the test set. Furthermore, we pick the least complex model that is comparable in accuracy to the best model under the “one-standard-error” criterion Franklin [22].

For each of the settings above, this process is repeated a total of 100 times to compute the following evaluation metrics, and the average results are recorded.

3.2. Evaluation Metrics

The data are split randomly into a training set and a test set. Suppose that $x_0\in {\mathbb{R}}^{\mathit{p}}$ is drawn from the row of the testing design matrix X, and ${\widehat{y}}_0$ denotes its connected response value by fitting the model. Additionally, let ${\widehat{\beta }}_0$ denote the corresponding estimated coefficient of the predictor variable $x_0$.

Mean-square error:

$$\mathrm{MSE}=E{\left(y_{test}-{\widehat{y}}_0\right)}^2=E{\left(y_{test}-x_0^T{\widehat{\beta }}_0\right)}^2.$$

(24)

This value assesses the accuracy of the model prediction. A good model has the highest prediction accuracy in the sense that its prediction error, MSE, is minimized. The following metrics were developed by Hastie et al. [12].

Relative accuracy:

$$\mathrm{RA}\left(\widehat{\beta }\right)=\frac{E{\left(x_0^T{\widehat{\beta }}_0-x_0^T\beta \right)}^2}{E{\left(x_0^T\beta \right)}^2}=\frac{{\left({\widehat{\beta }}_0-\beta \right)}^T\mathrm{\Sigma }\left({\widehat{\beta }}_0-\beta \right)}{{\beta }^T\mathrm{\Sigma }\beta }.$$

(25)

Relative test error:

$$\mathrm{RTE}\left(\widehat{\beta }\right)=\frac{E{\left(y_{test}-x_0^T{\widehat{\beta }}_0\right)}^2}{{\sigma }^2}=\frac{{\left({\widehat{\beta }}_0-\beta \right)}^T\mathrm{\Sigma }\left({\widehat{\beta }}_0-\beta \right)+{\sigma }^2}{{\sigma }^2}.$$

(26)

Proportion of variance explained:

$$\mathrm{PVE}\left(\widehat{\beta }\right)=1-\frac{E{\left(y_{test}-x_0^T{\widehat{\beta }}_0\right)}^2}{Var\left(y_{test}\right)}=1-\frac{{\left({\widehat{\beta }}_0-\beta \right)}^T\mathrm{\Sigma }\left({\widehat{\beta }}_0-\beta \right)+{\sigma }^2}{{\beta }^T\mathrm{\Sigma }\beta +{\sigma }^2}.$$

(27)

Number of nonzeros: The average number of nonzero estimated coefficients,

$${∥{\widehat{\beta }}_{nonzero}∥}_0=\sum _{j=1}^p{1}_{\left\{{\widehat{\beta }}_j\ne 0\right\}}.$$

(28)

where $1_{\left\{{\widehat{\beta }}_j\ne 0\right\}}=\left\{\begin{array}{cc}1,\hfill & {\widehat{\beta }}_j\ne 0\hfill \\ 0,\hfill & {\widehat{\beta }}_j=0\hfill \end{array}\right.$. An ideal score should be close to the number of true nonzero coefficients q.

Furthermore, in addition to the assessment of prediction accuracy, we explore the last metric to measure the right variable recovery. This metric quantifies the degree to which the valid solution $\widehat{\beta }$ to the convex optimization problem in Equation (5) matches the true coefficient $\beta$.

3.3. Summary of Results

Table 1. Simulation results for SNR = 0.2.

p	n	Method	RR	RTE	PVE	MSE	Number of Nonzeros
20	100	Lasso	0.997	1.206	0.4	96.2	1
		Rlasso	0.997	1.205	0.6	95.4	1
		Alasso	0.995	1.205	0.8	94.5	1
		Radlasso	0.986	1.203	2.4	100.1	2
	500	Lasso	0.990	1.199	1.6	91.4	4
		Rlasso	0.987	1.198	2.1	90.3	2
		Alasso	0.989	1.198	1.9	90.5	3
		Radlasso	0.974	1.196	4.3	86.5	6
	1000	Lasso	0.987	1.197	2.1	90.2	5
		Rlasso	0.983	1.196	2.9	89.6	3
		Alasso	0.985	1.197	2.4	90.1	4
		Radlasso	0.974	1.195	4.4	86.8	7
50	100	Lasso	0.998	1.197	0.4	99.7	1
		Rlasso	0.997	1.197	0.5	99.9	1
		Alasso	0.993	1.196	1.2	98.8	2
		Radlasso	0.985	1.195	2.3	106.8	2
	500	Lasso	0.992	1.200	1.4	93.4	4
		Rlasso	0.986	1.199	2.3	92.5	2
		Alasso	0.988	1.199	1.9	91.6	3
		Radlasso	0.976	1.197	4.0	90.6	5
	1000	Lasso	0.987	1.195	2.1	88.8	5
		Rlasso	0.982	1.195	2.9	88.0	3
		Alasso	0.985	1.195	2.5	88.4	4
		Radlasso	0.974	1.193	4.3	86.5	6

summarizes the average results of simulation for lasso, relaxed lasso, adaptive lasso and relaxed adaptive lasso with SNR = 0.2. We find that the relaxed adaptive lasso has the best RR, RTE, PVE and MSE scores on average. In other words, the proposed method achieves the maximum prediction accuracy in the majority of cases, despite occasions where the adaptive lasso’s MSE is somewhat better than that of the relaxed adaptive lasso. Specifically, the adaptive lasso yields a much smaller MSE due to the small sample size (e.g., $n=100$). However, when the sample size is increased to $n=1000$, the relaxed adaptive lasso outperforms all other methods owing to the feature of large samples in which parametric estimators converge in probability to true parameters.

In

Table 2. Simulation results for SNR = 0.8.

p	n	Method	RR	RTE	PVE	MSE	Number of Nonzeros
20	100	Lasso	0.980	1.789	8.8	75.1	5
		Rlasso	0.972	1.783	12.1	73.8	3
		Alasso	0.975	1.785	11.1	72.8	4
		Radlasso	0.960	1.773	17.8	75.2	5
	500	Lasso	0.969	1.781	13.8	61.5	7
		Rlasso	0.962	1.775	17.1	60.7	5
		Alasso	0.967	1.780	14.7	61.9	6
		Radlasso	0.956	1.771	19.7	58.8	9
	1000	Lasso	0.966	1.762	14.8	59.3	8
		Rlasso	0.959	1.756	17.8	58.5	6
		Alasso	0.964	1.760	15.9	59.3	7
		Radlasso	0.956	1.753	19.4	57.1	9
50	100	Lasso	0.985	1.784	6.7	75.5	4
		Rlasso	0.978	1.779	9.4	73.4	3
		Alasso	0.974	1.775	11.4	69.7	6
		Radlasso	0.963	1.766	16.2	83.3	4
	500	Lasso	0.970	1.773	13.1	62.9	7
		Rlasso	0.963	1.767	16.6	61.5	5
		Alasso	0.967	1.770	14.6	61.9	6
		Radlasso	0.958	1.763	18.7	60.4	7
	1000	Lasso	0.967	1.774	14.6	59.7	8
		Rlasso	0.960	1.768	17.9	58.7	6
		Alasso	0.964	1.772	15.8	59.4	7
		Radlasso	0.957	1.765	19.3	57.6	8

NOTE: The MSE and PVE values in the table are 100 and 1000 times larger to emphasize the distinction between these methods.

, excellent performance is observed for all methods when the SNR is increased to 0.8. As expected, relaxed adaptive lasso maintains its competitive edge and achieves overall good performance. In particular, it roughly maintains the correct number of nonzero variables as the number of observations n increases. For $\left(n,p\right)=\left(100,20\right)$ and $\left(n,p\right)=\left(100,50\right)$, it holds up to five and four variables, respectively. For $\left(n,p\right)=\left(1000,20\right)$ and $\left(n,p\right)=\left(1000,50\right)$, up to nine and eight, respectively, approach the number of truly valid features $q=10$. This illustrates that the sparsity pattern of estimators in the relaxed adaptive lasso achieves the proper variable recovery when n is quite large. In contrast, the relaxed lasso and adaptive lasso shrink too many coefficients toward zero; as a result, fewer variables remain in the resulting model. Therefore, we conclude that as the number of observations n grows rapidly, the number of variables preserved in the model grows as well, and it is possible to select the important variables approximately correctly, i.e., having proper variable recovery.

4. Application to Real Data

4.1. Dataset

The real dataset used in this study is from the CSMAR Database, which contains 11 research series on stocks, companies, funds, the economy, industries, etc. It is widely recognized as one of the most professional and accurate databases available for research purposes. Our data include a total of 2137 records, with each record corresponding to the financial data of one listed company in 2021. The training set is made up of the first 1496 observations, and the test set is made up of the rest. The response variable is the R&D investment of the company, and the predictor variables include 86 factors that may have an effect on the firm’s R&D investment, such as fixed-assets depreciation, accounts receivable and payroll payable. To compare the model selection performance of the method proposed in this paper to that of the other three methods, the aforementioned methods are used to fit the model on the training set, and the prediction accuracy of these models is measured in terms of the MSE on the test set. It is shown in the following that the relaxed adaptive lasso has the highest prediction accuracy with the smallest MSE value.

4.2. Analysis Results

As can be seen in

Table 3. Prediction accuracy for R&D investment study.

Method	Lasso	Rlasso	Alasso	Radlasso
MSE	0.521	0.485	0.575	0.429

, the MSE values of the lasso and adaptive lasso are, respectively, as large as 0.521 and 0.575, indicating that they have the worst prediction accuracy. The relaxed lasso performs somewhat better than the lasso and the adaptive lasso in terms of MSE. As expected, the relaxed adaptive lasso estimator’s prediction accuracy remains satisfactory, with the smallest MSE of 0.429. In

Table 4. Variables selected by Radlasso.

Order Number	Explanatory Variable	Coefficient
$x_{10}$	Cash Flow from Operations	0.008
$x_{13}$	Net Increase in Cash and Cash Equivalents	0.048
$x_{15}$	Net Accounts Receivable	0.208
$x_{26}$	Non-Current Assets	−0.214
$x_{48}$	Business Taxes and Surcharges	−0.265
$x_{67}$	Interest Income	0.130
$x_{70}$	Profit and Loss from Asset Disposal	0.154
$x_{73}$	Cash Paid for Commodities or Labor	0.386
$x_{74}$	Cash Paid to and for Employees	0.569
$x_{83}$	Cash Flow from Financing Activities Net Amount	−0.080

, a total of 10 variables are selected by the relaxed adaptive lasso. It has shown that Cash Paid to and for Employees, Cash Paid for Commodities or Labor, Business Taxes and Surcharges are identified as the three most influential factors on R&D investment. As a result, we can conclude that relaxed adaptive lasso leads to the simplest model with the highest prediction accuracy among the four foregoing methods.

Among the most important explanatory variables affecting R&D investment, Cash Paid to and for Employees measures the company’s actual benefits and rewards; Cash Paid for Commodities or Labor measures the overall payment ability of the company; and Business Taxes and Surcharges measure the tax burden of the company’s operation. According to the estimator coefficients estimated by the simplified model, firms with high Cash Paid to and for Employees and Cash Paid for Commodities or Labor tend to spend more on R&D (the positive coefficient on the response variable), whereas Business Taxes and Surcharges have a negative influence on the company’s R&D investment. From the results of the analysis, it is not surprising that companies focused on welfare take more advantage of innovative technology because generous compensation not only improves employees’ work motivation but also helps to retain and recruit talent. Furthermore, strong payment ability implies the high profitability of companies with successful operations, allowing them to spend massive amounts of money on R&D. Note that a heavy tax burden may result in a lower investment cost for a company. In general, increasing R&D input is highly influenced by a few selected variables, the three most important of which are the company’s welfare, payment ability and tax burden.

5. Conclusions

In this article, we have proposed a two-stage variable selection method called relaxed adaptive lasso as a combination of relaxed lasso and adaptive lasso estimation. From the proof of the theorem, we conclude that the relaxed adaptive lasso has the same convergence rate as the relaxed lasso with $O_p\left(n^{-1}\right)$ and that both are faster than adaptive lasso and ordinary lasso in the low-dimensional setting. Furthermore, the relaxed adaptive lasso has the property of consistency, which means that the probability of selecting the true model approaches one under the condition of $\varphi {\lambda }_n=o\left(n\right)$. The simulation study has shown that the proposed method has comparable prediction accuracy and accurate variable recovery as the number of observations increases. In practical applications, the conclusion has been confirmed by the analysis of the financial data of the listed company.

We have shown the asymptotic property of the relaxed adaptive lasso in the linear model. For further research, it is suggested to extend the theory and methodology to the generalized linear model [23]. In addition, the model does not handle the high-dimensional case well, where the variable dimension is much larger than the sample size. We propose to combine the existing idea with two-stage variable selection methods such as Sure Independence Screening (SIS) [24] and Distance Correlation Based SIS (DC-SIS) [25] to overcome this challenge.