Robust Variable Selection Based on Penalized Composite Quantile Regression for High-Dimensional Single-Index Models

Abstract

The single-index model is an intuitive extension of the linear regression model. It has been increasingly popular due to its flexibility in modeling. In this work, we focus on the estimators of the parameters and the unknown link function for the single-index model in a high-dimensional situation. The SCAD and Laplace error penalty (LEP)-based penalized composite quantile regression estimators, which could realize variable selection and estimation simultaneously, are proposed; a practical iterative algorithm is introduced to obtain the efficient and robust estimators. The choices of the tuning parameters, the bandwidth, and the initial values are also discussed. Furthermore, under some mild conditions, we show the large sample properties and oracle property of the SCAD and Laplace penalized composite quantile regression estimators. Finally, we evaluated the performances of the proposed estimators by two numerical simulations and a real data application.

1. Introduction

As a generalized regression model, the single-index regression model has a wide range of applications in the fields of finance, economics, biomedicine, etc. The single-index regression model not only avoids the so-called “curse of dimensionality” in the nonparametric models, but also significantly improves the efficiency of model estimation and reveals the relationship between the response variables and the high-dimensional covariates, to keep good interpretability of the parametric models and flexibility of the nonparametric models simultaneously [1,2,3,4,5]. However, the single-index regression model also inherits the shortcomings of the classical regression models. For example, in practical applications, especially in heavy-tailed error distribution scenarios, it is difficult to satisfy the bounded error variance, which a single-index regression model requires. Moreover, in the mean regression scenario, the estimation results of a single-index regression model are very sensitive to extreme values. To overcome these drawbacks, robust regression methods are necessary for the single-index regression model when fitting the real data.

Compared with the mean regression, the quantile regression proposed by [6] can measure the effect of the explanatory variable, not only on the distribution center but also on the upper and lower tails of the response variable. The quantile regression (QR) estimation, which is not restricted by the error distribution and can effectively avoid the impact of outliers, is more robust than the least squares estimation. Furthermore, in order to utilize the information on different quantiles adequately, composite quantile regression (CQR) was proposed by [7]. [8] added the SCAD-L2 penalty to the loss function and proposed a robust variable selection method based on the weighted composite quantile regression (WCQR), which made variable selection insensitive to high-leverage points and outliers. In this article, we studied the estimation and variable selection of the single-index quantile regression model. The single-index quantile regression model is specified in the following form

$$Y=g\left({\mathit{X}}^{\top }\gamma \right)+\epsilon ,$$

(1)

where Y is the response variable, $\mathit{X}$ is a d-dimensional covariate vector, $\gamma$ is an unknown parameter vector, $g(·)$ is an unknown link function, $\epsilon$ is the random error, and the $\tau$th conditional quantile is zero, i.e., $P(\epsilon \le 0|\mathit{X})=\tau$. In order to identify it more easily, we assume that $\Vert \gamma \Vert =1$ and the first component of $\gamma$ is positive, where $\Vert ·\Vert$ denotes the Euclidean norm.

There are two estimation problems for the single-index quantile regression model. One is the estimation of parameters and the other is the estimation of the link function. The study of estimation for single-index quantile regression models began with [9], which generalized the average derivative method. Meanwhile, [10] proposed a simple algorithm to achieve the quantile regression for single-index models and proved the asymptotic properties of estimators. [3] proposed D-Vine Copula-based quantile regression, which is a new algorithm that does not require accurately assuming the shape of conditional quantiles and avoids the typical defects of linear models, such as multicollinearity. [11] proposed a non-iterative coincidence quantile regression (NICQR) estimation algorithm for the single-index quantile regression model, which has high computational efficiency and is suitable for analyzing massive data sets.

In real data, the model is often sparse. The variables inevitably contain a few irrelevant and unnecessary variables while modeling the real data, which can degrade the efficiency of the resulting estimation procedure and increase the complexity of models. In the case of linear models, many authors have considered variable selection via penalized least squares, which allows for a simultaneous selection of variables and estimation of regression parameters. Several penalty functions, including the SCAD [12], the adaptive LASSO [13], and the adaptive elastic net [14] have been shown to possess favorable theoretical properties: unbiased, sparsity, and continuity; it is regarded as the basic properties that a good estimator should enjoy [15]. It enjoys the oracle property. [5] combined the SCAD penalty variable selection method with LM-ANN for modeling, making good use of the advantages of SCAD in dimension reduction and the efficiency of LM-ANN in nonlinear relationship modeling.

Similar to the linear regression model, the set of predictors for the single-index quantile regression model can contain a large number of irrelevant variables. Therefore, it is important to select the relevant variables when fitting the single-index quantile regression model. However, the problem of variable selection for the high-dimensional single-index quantile regression model is not well settled in the literature. In recent years, many significant research results have emerged on the variable selection problem of the single-index quantile regression model. [16] proposed a non-iterative estimation and variable selection method for the single-index quantile regression model. The initial value and the weight of the penalty function were obtained via the inverse regression technique, which is the key to this method. [17] combined least absolute deviations (LAD) and SCAD for single-index models. However, we note that SCAD is a piecewise continuous spline function. Because of this structure, different splines need different derivative formulas; it is necessary to select different derivative formulas to match different splines when we carry out penalty optimization. This certainly adds to the programming complexity. So, [18] proposed a continuous bounded smooth penalty function–Laplace error penalty (LEP) that does not have a piecewise spline structure and proved its oracle property. LEP is infinitely differentiable everywhere except at the origin and, therefore, is much smoother than SCAD. Furthermore, LEP can approach the $L_0$ penalty as close as possible, which is viewed as the optimal penalty. Moreover, LEP can yield a convex objective function for optimization under mild conditions, such that it is easier to calculate and obtain the only optimal solution with desired properties.

In this paper, we combined (composite) the quantile regression method with the SCAD penalty and Laplace error penalty to construct two sparse estimators for the single-index quantile regression model. Our method realizes variable selection and parameter estimation simultaneously. In addition, we prove that the proposed estimator has large sample properties, including N-consistency and oracle properties. A simulation study showed that our method has some resistance to heavy tail errors and outliers, and the accuracy of parameter estimation is higher.

The rest of this paper is organized as follows. In Section 2, the SCAD penalized composite quantile regression and the Laplace penalized composite quantile regression for single-index models are introduced. Furthermore, an iterative algorithm for the single-index model is analyzed and the selections of bandwidth, tuning parameters, and initial values are discussed. In Section 3, we state the large sample properties of SCAD and Laplace penalized composite quantile estimators for single-index models. In Section 4, we show our method and algorithm by two numerical simulations and real data. Section 5 includes some concluding remarks. Technical proofs and the algorithm based on LEP are relegated to Appendix A and Appendix B, respectively.

2. Problem Setting and Methodology

2.1. Composite Quantile-SCAD Method for Single-Index Models

We assume $\{{\mathit{X}}_i,Y_i,i=1,2,\dots ,n\}$ are n independent samples from the single-index model (1). Note that there are two estimation problems, which are the estimation of the parameter vector $\gamma$ and the estimation of the link function $g(·)$. Given an accurate estimate of $\gamma$, the link function $g(·)$ can be locally approximated by a linear function

$$g\left({\mathit{X}}^{\top }\gamma \right)\approx g\left(u\right)+g^{\prime }\left(u\right)({\mathit{X}}^{\top }\gamma -u)=a+b({\mathit{X}}^{\top }\gamma -u),$$

(2)

for ${\mathit{X}}^{\top }\gamma$ in the neighborhood of u, where $a=g\left(u\right)$ and $b=g^{\prime }\left(u\right)$ are local constants. Namely, we can obtain a good local linear estimation of $g\left(u\right)$ and $g^{\prime }\left(u\right)$, which are $\widehat{g}\left(u\right)=\widehat{a}$ and $\widehat{g^{\prime }}\left(u\right)=\widehat{b}$, respectively, based on an accurate estimate of $\gamma$. So our main interest is to estimate the parameter vector. Following [19], we adopt the MAVE estimate of $\gamma$, which is obtained by solving the minimization problem

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{i=1}^n{[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]}^2{w}_{ij},$$

(3)

where $w_{ij}=k_h({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )/{\sum }_{l=1}^n{k}_h({\mathit{X}}_l^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )$, $\mathit{a}={(a_1,a_2,\dots ,a_n)}^{\top }$, $\mathit{b}={(b_1,b_2,\dots ,b_n)}^{\top }$, $k_h(·)=k(·/h)/h$ and $k(·)$ is a symmetric kernel function, h is the bandwidth. [20] combined the MAVE and LASSO to obtain the sparse estimate (sim-lasso) of $\gamma$ by solving the following minimization problem

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{i=1}^n{[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]}^2{w}_{ij}+\lambda \sum _{j=1}^n|b_j|\sum _{k=1}^d|{\gamma }_k|,$$

(4)

where $\lambda$ is a nonnegative penalty parameter. Note that the above target loss function is the square loss function based on the least squares method and, naturally, the LAD is extended to a single-index model as an alternative to the LS method. [17] combined LAD with SCAD to construct a sparse estimator of $\gamma$ by solving the following minimization problem

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{i=1}^n|Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )|w_{ij}+\sum _{j=1}^n|b_j|\sum _{k=1}^d{p}_{\lambda }(|{\gamma }_k|),$$

(5)

where $p_{\lambda }(·)$ is the SCAD penalty function proposed by [3]; it is defined in terms of its first order derivative. For $\theta >0$

$$p_{\lambda }^{\prime }\left(\theta \right)=\lambda \{I(\theta \le \lambda )+\frac{{(a\lambda -\theta )}_{+}}{(a-1)\lambda }I(\theta >\lambda )\},$$

(6)

where $a>2$ and $\lambda$ is a nonnegative penalty parameter, the notation $Z_{+}$ stands for the positive part of Z. The LAD is a special case of the quantile regression, which only indicates the case when the quantile is 1/2. Thus, this motives us to generalize composite quantile regression for single-index models. Combining the SCAD penalty function with the compound quantile regression, we can obtain a sparse estimation ${\widehat{\gamma }}^{qr.sim.scad}$ of the parameter $\gamma$, which is the solution to the following minimization problem

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{q=1}^Q\sum _{i=1}^n{\rho }_{{\tau }_q}[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]w_{ij}+\sum _{j=1}^n|b_j|\sum _{k=1}^d{p}_{\lambda }(|{\gamma }_k|)$$

(7)

where ${\tau }_q=\frac{q}{q+1}\in (0,1)$ stands for ${\tau }_q$-quantile and $q=1,2,\dots ,Q$ with the number of quantile Q, and ${\rho }_{{\tau }_q}\left(z\right)={\tau }_{q}z·I_{[0,\infty ]}\left(z\right)-(1-{\tau }_q)z·I_{(-\infty ,0)}\left(z\right)$ is the ${\tau }_q$-quantile loss function. In addition, we assume that the ${\tau }_q$-quantile of the random error $\epsilon$ is 0. Thus, $g\left({\mathit{X}}^{\top }\gamma \right)$ is the conditional ${\tau }_q$-quantile of the response variable Y. We denote the target function in (7) by $Q_{\lambda }^S(\mathit{a},\mathit{b},\gamma )$.

2.2. Composite Quantile–LEP Method for Single-Index Models

The Laplace error penalty function with two tuning parameters is proposed by [18]. Unlike other penalty functions, this new penalty function is naturally constructed as a bounded smooth function rather than a piecewise spline. The figure of LEP is similar to SCAD, but is much smoother than SCAD, which prompts us to apply it to the composite quantile regression for single-index models. Combining LEP with composite quantile regression, we can obtain a sparse estimation ${\widehat{\gamma }}^{qr.sim.lep}$ of the parameter $\gamma$, which is the solution to the following minimization problem

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{q=1}^Q\sum _{i=1}^n{\rho }_{{\tau }_q}[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]w_{ij}+\sum _{j=1}^n|b_j|\sum _{k=1}^d{p}_{\lambda ,\kappa }(|{\gamma }_k|)$$

(8)

where $P_{\lambda ,\kappa }(·)$ is LEP. For $\theta >0$,

$$P_{\lambda ,\kappa }\left(\theta \right)=\lambda (1-e^{-\frac{\theta }{\kappa }}),$$

(9)

where $\lambda$ and $\kappa$ are two nonnegative tuning parameters regularizing the magnitude of penalty and controlling the degree of approximation to the $L_0$ penalty, respectively. This penalty function is called Laplace penalty function because function $e^{-\frac{\theta }{\kappa }}$ has the form of the Laplace density. We denote the target function in (8) by $Q_{\lambda ,\kappa }^S(\mathit{a},\mathit{b},\gamma )$.

2.3. Computation

Given the initial estimate $\widehat{\gamma }$, the SCAD penalty function can be locally linear, approximated as follows [21]. For ${\widehat{\gamma }}_j\ne 0$,

$$p_{\lambda }(|{\gamma }_j|)\approx p_{\lambda }(|{\widehat{\gamma }}_j|)+p_{\lambda }^{\prime }(|{\widehat{\gamma }}_j|)(|{\gamma }_j|-|{\widehat{\gamma }}_j|),$$

(10)

where ${\gamma }_j\approx {\widehat{\gamma }}_j$. Remove a few irrelevant terms, (7) can be rewritten as

$$\underset{\mathit{a},\mathit{b},\Vert \gamma \Vert =1}{min}\sum _{j=1}^n\sum _{q=1}^Q\sum _{i=1}^n{\rho }_{{\tau }_q}[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]w_{ij}+\sum _{j=1}^n|b_j|\sum _{k=1}^d{p}_{\lambda }^{\prime }(|{\widehat{\gamma }}_k|)|{\gamma }_k|.$$

(11)

We denote the target function in (11) by $Q_{\lambda }^{S*}(\mathit{a},\mathit{b},\gamma )$. We can easily discover that $Q_{\lambda }^{S*}$ is invariant, so when minimizing $Q_{\lambda }^{S*}$, we restrict $\Vert \gamma \Vert =1$ to be the unit length $\Vert \gamma \Vert =1$ through normalization $\gamma$.

In order to obtain an accurate estimate of $\gamma$ and $g(·)$, we introduce a new iterative algorithm. Then our estimation procedure is described in detail as follows:

Step 0.

Obtain an initial estimate of $\gamma$. Standardize the initial estimate $\widehat{\gamma }$, such that $\Vert \gamma \Vert =1$ and ${\widehat{\gamma }}_1>0$.

Step 1.

Given an estimate $\widehat{\gamma }$, we obtain $\{{\widehat{a}}_j,{\widehat{b}}_j,j=1,2,\dots ,n\}$ by solving

$$\begin{array}{cc}\hfill & \underset{(a_j,b_j)}{min}\sum _{i=1}^n\sum _{q=1}^Q{\rho }_{{\tau }_q}[Y_i-a_j-b_j({\mathit{X}}_i^{\top }\widehat{\gamma }-{\mathit{X}}_j^{\top }\widehat{\gamma })]w_{ij}+|b_j|\sum _{k=1}^d{p}_{\lambda }^{\prime }(|{\widehat{\gamma }}_k|)|{\widehat{\gamma }}_k\mid \hfill \\ \hfill =& \underset{(a_j,b_j)}{min}\sum _{i=1}^{n+1}\sum _{q=1}^Q\rho [Y_i^{*}-(A,B)\left(\begin{array}{c}{a}_j\\ b_j\end{array}\right)]w_{ij}^{*},\hfill \end{array}$$

(12)

where h is the optimal bandwidth, $(\rho ,Y_i^{*},A,B,w_{ij}^{*})=({\rho }_{{\tau }_q},Y_i,1,{\mathit{X}}_{\mathit{i}}^{\top }\widehat{\gamma }-{\mathit{X}}_{\mathit{j}}^{\top }\widehat{\gamma },w_{ij})$ for $i=1,2,\dots ,n$, and $(\rho ,Y_i^{*},A,B,w_{ij}^{*})=(1/Q,0,0,{\sum }_{k=1}^d{p}_{\lambda }^{\prime }(|{\widehat{\gamma }}_k|)|{\widehat{\gamma }}_k|,1)$ for $i=n+1$. The $rq(·)$ function in R package “quantreg” is helpful to obtain $\{{\widehat{a}}_j,{\widehat{b}}_j,j=1,2,\dots ,n\}$. Moreover, for the SCAD penalty, we can apply a difference-of-convex algorithm [22] to the simple computation.

Step 2.

Given $\{{\widehat{a}}_j,{\widehat{b}}_j,j=1,2,\dots ,n\}$, update $\widehat{\gamma }$ by solving

$$\underset{\gamma }{min}\sum _{j=1}^n\sum _{q=1}^Q\sum _{i=1}^n{\rho }_{{\tau }_q}[Y_i-{\widehat{a}}_j-{\widehat{b}}_j({\mathit{X}}_i^{\top }\gamma -{\mathit{X}}_j^{\top }\gamma )]w_{ij}+\sum _{j=1}^n|{\widehat{b}}_j|\sum _{k=1}^d{p}_{\lambda }^{\prime }(|{\widehat{\gamma }}_k|)|{\gamma }_k|.$$

(13)

We can apply a fast and efficient coordinate descent algorithm [23] if d is very large, or combine the MM algorithm [24] to reduce the calculation.

Step 3.

Scale $\widehat{\mathit{b}}\leftarrow$ sgn$\left({\widehat{\gamma }}_1\right)·\Vert \widehat{\gamma }\Vert \widehat{\mathit{b}}$, and $\widehat{\gamma }\leftarrow$ sgn$\left({\widehat{\gamma }}_1\right)·\widehat{\gamma }/\Vert \widehat{\gamma }\Vert$.

Step 4.

Continue Step 1–Step 3 until convergence.

Step 5.

Given the final estimate $\widehat{\gamma }$ from Step 4, we estimate $g(·)$ at any u by $\widehat{g}(·,h,\widehat{\gamma })=\widehat{a}$, where

$$(\widehat{a},\widehat{b})=\underset{(a,b)}{min}\sum _{q=1}^Q\sum _{i=1}^n{\rho }_{{\tau }_q}[Y_i-a-b({\mathit{X}}_i^{\top }\widehat{\gamma }-u)]k_h({\mathit{X}}_i^{\top }\widehat{\gamma }-u).$$

(14)

Remark 1.

The above algorithm is aimed at the SCAD penalty function. Moreover, similarly, we can obtain the other algorithm for the Laplace penalty function, replacing SCAD with LEP.

2.4. The Selections of Bandwidth, Tuning Parameters, and Initial Value

The selection of the bandwidth plays a crucially important role in local polynomial smoothing because it controls the curvature of the fitted function. The cross-validation (CV) and the generalized cross-validation (GCV) can be utilized to choose a proper bandwidth, but these methods are not computationally practical due to the large calculation amounts. For the local linear quantile regression, [25] obtained an approximate optimal bandwidth under some suitable assumptions and found the rule-of-thumb bandwidth: $h_{\tau }=h_m{\{\tau (1-\tau )/{\psi }^2\left({\mathsf{\Phi }}^{-1}\left(\tau \right)\right)\}}^{1/5}$, where $\psi (·)$ and $\mathsf{\Phi }(·)$ are the probability density function and the cumulative distribution function of the normal distribution, respectively; $h_m$ is the optimal bandwidth of the least squares regression. There are many algorithms for the selection of $h_m$. [26] found that the rule-of-thumb bandwidth: $h_m={\{4/(d+2)\}}^{1/(4+d)}{n}^{-1/(4+d)}$ in the single-index models acts fairly well, where d is the dimension of the kernel function. We combined a multiplier only consisting of $\tau$ with the optimal bandwidth $h_m$ of the LS regression to obtain a $h_{\tau }$ with good characters.

There are several kinds of selection methods for SCAD’s nonnegative tuning parameter $\lambda$, such as CV, GCV, AIC, BIC, and so on. Following [27], we utilized the BIC criterion to choose a proper tuning parameter of SCAD in this paper

$$\mathrm{BIC}\left(\lambda \right)=\frac{1}{\tilde{\sigma }}\sum _{i=1}^n\sum _{q=1}^Q{\rho }_{{\tau }_q}[Y_i-g\left({\mathit{X}}_i^{\top }\widehat{\gamma }\left(\lambda \right)\right)]+log\left(n\right)·df/2,$$

(15)

where $\tilde{\sigma }=(1/n){\sum }_{i=1}^n{\sum }_{q=1}^Q{\rho }_{{\tau }_q}[Y_i-g\left({\mathit{X}}_i^{\top }\tilde{\gamma }\right)]$ with $\tilde{\gamma }$ being the composite quantile estimator without penalty and $df$ is the number of non-zero coefficients of $\widehat{\gamma }\left(\lambda \right)$. Then, we chose the optimal tuning parameter by minimizing the above criteria. Moreover, for LEP, we utilized the extended Bayesian information criterion (EBIC) [28] to choose proper tuning parameters $\lambda$ and $\kappa$.

$$\mathrm{EBIC}(\lambda ,\kappa )=log\left({\widehat{\sigma }}^2\right)+\frac{log\left(n\right)+loglog\left(d\right)}{n}df$$

(16)

where ${\widehat{\sigma }}^2=1/(n-1){\sum }_{i=1}^n{\sum }_{q=1}^Q{\rho }_{{\tau }_q}[Y_i-g\left({\mathit{X}}_i^{\top }\widehat{\gamma }\right)]$ and $df$ is the number of non-zero coefficients of $\widehat{\gamma }(\lambda ,\kappa )$. Similarly, in order to select the best tuning parameters, we minimized the above criteria in the arrangement of $\lambda$ values.

The initial value of the unknown parameter is required at the beginning of the iteration of our algorithm. A convenient choice is $\widehat{\gamma }/\Vert \widehat{\gamma }\Vert$ where $\widehat{\gamma }$ is the composite quantile estimator without penalty.

3. Theoretical Properties

A good estimate is supposed to satisfy unbiasedness, continuity, and the so-called oracle property. Thus, in this section, we discuss the large sample properties of the SCAD penalized composite quantile regression and the Laplace penalized composite quantile regression for single-index models. We consider the data $\{({\mathit{X}}_i,Y_i),i=1,2,\dots ,n\}$ including n observations from model (1), such as Section 2. Moreover, let ${\mathit{X}}_i={({\mathit{X}}_{i1}^{\top },{\mathit{X}}_{i2}^{\top })}^{\top }$, $\gamma ={({\gamma }_1^{\top },{\gamma }_2^{\top })}^{\top }$, ${\mathit{X}}_{i1}\in {\Re }^s$, ${\mathit{X}}_{i2}\in {\Re }^{d-s}$. In addition, ${\gamma }_0={({\gamma }_{10}^{\top },{\gamma }_{20}^{\top })}^{\top }$ stands for the true regression parameters of model (1) and $\Vert {\gamma }_0\Vert =1$, where the s components in ${\gamma }_{10}$ are not zero. We suppose the following regularity conditions to hold:

(i)

The density function of ${\mathit{X}}^{\top }\gamma$ is positive and uniformly continuous for $\gamma$ in a neighborhood of ${\gamma }_0$. Further, the density of ${\mathit{X}}^{\top }{\gamma }_0$ is continuous and bounded away from 0 and ∞ on its support D.

(ii)

The function $g(·)$ has a continuous and bounded second derivative in D.

(iii)

The kernel function $k(·)$ is a symmetric density function with bounded support and a bounded first derivative.

(iv)

The density function $f_Y(·)$ of Y is continuous and has a bounded derivative; it is bounded away from 0 and ∞ on compact support.

(v)

The following expectations exist:

$$\begin{array}{cc}& C_0=E\left\{g^{\prime }{\left({\mathit{X}}^{\top }{\gamma }_0\right)}^2[\mathit{X}-E\left(\mathit{X}\right|{\mathit{X}}^{\top }{\gamma }_0)]{[\mathit{X}-E\left(\mathit{X}\right|{\mathit{X}}^{\top }{\gamma }_0)]}^{\top }\right\}\hfill \\ & C_1=E\left\{f_Y\left(g\left({\mathit{X}}^{\top }{\gamma }_0\right)\right)g^{\prime }{\left({\mathit{X}}^{\top }{\gamma }_0\right)}^2[\mathit{X}-E\left(\mathit{X}\right|{\mathit{X}}^{\top }{\gamma }_0)]{[\mathit{X}-E\left(\mathit{X}\right|{\mathit{X}}^{\top }{\gamma }_0)]}^{\top }\right\}\hfill \end{array}$$

(vi)

$h\to 0$ and $nh\to \infty$.

Given $({\widehat{a}}_j,{\widehat{b}}_j)$, let $H={\sum }_{j=1}^n|{\widehat{b}}_j|$, $a_n=max\{P_{\lambda }^{\prime }\left({\gamma }_{0j}\right):{\gamma }_{0j}\ne 0\}$ and ${\widehat{\gamma }}^{qr.sim.scad}$ be the solution of (7). We should note that under condition (ii), the first derivative is bounded. Thus, $H=O\left(n\right)$.

Theorem 1.

Under the conditions (i)–(v). If $max\{P_{\lambda }^{″}(|{\gamma }_{0k}|):{\gamma }_{0k}\ne 0\}\to 0$ and $a_n=O\left(n^{-1/2}\right)$, then there exists a local minimizer in (7) such that $\Vert {\widehat{\gamma }}^{qr.sim.scad}-{\gamma }_0\Vert =O_P(n^{-1/2}+a_n)$ with $\Vert {\widehat{\gamma }}^{qr.sim.scad}\Vert =\Vert {\gamma }_0\Vert =1$.

According to Theorem 1, we show that there exists a $\sqrt{n}$-consistent SCAD penalized composite quantile regression estimate for $\gamma$ if a proper tuning parameter $\lambda$ is selected. Let $c_n=\{p_{\lambda }^{\prime }(|{\gamma }_{01}|)$ sgn$\left({\gamma }_{01}\right),\dots ,p_{\lambda }^{\prime }(|{\gamma }_{0s}|)$ sgn$\left({\gamma }_{0s}\right){\}}^{\top }$, and ${\sum }_{\lambda }=\mathrm{diag}(p_{\lambda }^{″}(|{\gamma }_{01}|),\dots ,p_{\lambda }^{″}(|{\gamma }_{0s}|)$.

Lemma 1.

Under the same conditions as in Theorem 1. Assume that

$$\underset{n\to +\infty }{lim\; inf}\underset{\theta \to 0^{+}}{lim\; inf}{p}_{{\lambda }_n}^{\prime }(\theta /{\lambda }_n)>0.$$

(17)

If $\lambda \to 0$ and $\sqrt{H}\lambda \to \infty$ as $n\to \partial$, then with probability tending to 1, for any given ${\widehat{\gamma }}_1$ satisfying $\Vert {\widehat{\gamma }}_1-{\gamma }_{10}\Vert =O_P\left(n^{-1/2}\right)$, and any constant C, we have

$$Q_{\lambda }^S\left({({\gamma }_1^{\top },{\mathbf{0}}^{\top })}^{\top }\right)=\underset{\Vert {\gamma }_2\Vert \le C{n}^{-1/2}}{min}{Q}_{\lambda }^S\left({({\gamma }_1^{\top },{\gamma }_2^{\top })}^{\top }\right).$$

(18)

Theorem 2.

Under the same conditions as in Theorem 1—assume that the penalty function $p_{\lambda }(|\theta |)$ satisfies condition (17). If $\lambda \to 0$ and $\sqrt{H}\lambda \to \infty$, then with the probability tending to 1, the $\sqrt{n}$-consistent local minimizer ${\widehat{\gamma }}^{qr.sim.scad}={({\left({\widehat{\gamma }}_1^{qr.sim.scad}\right)}^{\top },{\left({\widehat{\gamma }}_2^{qr.sim.scad}\right)}^{\top })}^{\top }$ in Theorem 1 must satisfy:

(i) Sparsity: ${\widehat{\gamma }}_2^{qr.sim.scad}=0$.

(ii) Asymptotic normality:

$$\sqrt{n}\{(Q{C}_{11}+H{\sum }_{\lambda }/n)({\widehat{\gamma }}_1^{qr.sim.scad}-{\gamma }_{10})+H{c}_n/n\}\stackrel{D}{\to }N(0,0.2C_{01}),$$

where $C_{11}$ is the top-left s-by-s sub-matrix of $C_1$ and $C_{01}$ is the top-left s-by-s sub-matrix of $C_0$.

Theorem 2 shows that the SCAD penalized composite quantile regression estimator has the so-called oracle property when $\lambda \to 0$ and $\sqrt{H}\lambda \to \infty$.

Remark 2.

In this section, we discuss the large sample properties of the SCAD penalized composite quantile estimator (${\widehat{\gamma }}^{qr.sim.scad}$) in detail. Similarly, we can also show the large sample properties of the Laplace penalized composite quantile estimator (${\widehat{\gamma }}^{qr.sim.lep}$).

4. Numerical Studies

4.1. Simulation Studies

In this section, we evaluate the proposed estimator by simulation studies. We compare the performances of different penalized estimators with the oracle estimator. Specially, in order to reduce the burden of computation and simplify the calculation, we take the Gaussian kernel as the kernel function in our simulations. Moreover, we do not tune the value of the parameter a and set $a=3.7$, which is suggested by [12] for the SCAD penalty. Moreover, we set the quantile number: $Q=5$. Next, we compare the performances of the following four estimates for the single-index model:

• lad.sim.scad: the LAD estimators with the SCAD penalty;
• cqr.sim.scad: the composite quantile estimators with the SCAD penalty;
• cqr.sim.lep: the composite quantile estimators with the SCAD penalty;
• Oracle: the oracle estimators (composite quantile regression without penalty under the true model).

In order to evaluate the performances of the above estimators, we consider the following criteria:

• MAD (the mean absolute deviation) of $\widehat{\gamma }$: $\mathrm{MAD}=\frac{1}{n}{\sum }_{i=1}^n|X_i^{\top }\widehat{\gamma }-X_i^{\top }{\gamma }_0|$.
• NC: the average number of non-zero coefficients that are correctly estimated to be non-zero.
• NIC: the average number of zero coefficients that are incorrectly estimated to be non-zero, respectively.

Additionally, an estimated coefficient is viewed as 0 if its absolute value is smaller than ${10}^{-6}$.

Scenario 1. We assume that the single-index model has the following form:

$$Y=2{\mathit{X}}^{\top }{\gamma }_0+10exp(-{\left({\mathit{X}}^{\top }{\gamma }_0\right)}^2/5)+ϵ,$$

where ${\gamma }_{\mathbf{0}}=\gamma /\Vert \gamma \Vert$ with $\gamma ={(1,-1,2,-0.5,0,\dots ,0)}^{\top }$ being a 15-dimensional vector with only four non-zero values (the true coefficients). The X-variables are generated from the multivariate normal distribution and set the correlation between $X_i$ and $X_j$ to be ${0.5}^{|i-j|}$ and the Y-variable is generated from the above model. Then, to eliminate the impacts of different error distributions, we consider the following five error distributions:

• $N(0,1)$: the standard normal distribution (N);
• $t\left(3\right)$: the t-distribution with 3 degrees of freedom;
• $DE$: the double exponential distribution with media 0 and scale parameter $1/\sqrt{2}$;
• $CN$: the polluted normal distribution $0.9N(0,1)+0.1N(0,25)$ (CN);
• $Outlier$: an outlier case is considered, in which $10\%$ of the responses are shifted with a constant $c=5$.

In order to perform the simulations, we generated 200 replicates with moderate sample sizes, $n=100$, 200. Then, the corresponding results are reported in

Table 1. Simulation results for Scenario 1 based on 200 replications.

Error Distribution	Method	$\mathit{n}=100$			$\mathit{n}=200$
		MAD (%)	NC	NIC	MAD (%)	NC	NIC
$N(0,1)$	lad.sim.scad	11.65	3.96	3.53	7.63	4.00	1.57
	cqr.sim.scad	11.61	3.93	3.45	7.60	4.00	1.54
	cqr.sim.lep	11.59	3.91	3.44	7.58	4.00	1.53
	Oracle	11.57	3.90	3.42	7.56	4.00	1.51
$t\left(3\right)$	lad.sim.scad	13.72	3.90	3.73	9.25	3.99	1.99
	cqr.sim.scad	13.70	3.95	3.70	9.21	4.00	1.94
	cqr.sim.lep	13.68	3.96	3.68	9.18	4.00	1.93
	Oracle	13.67	3.98	3.67	9.15	4.00	1.91
$DE$	lad.sim.scad	8.79	3.97	3.20	5.69	4.00	1.76
	cqr.sim.scad	8.76	3.97	3.08	5.65	4.00	1.76
	cqr.sim.lep	8.74	3.98	3.05	5.63	4.00	1.76
	Oracle	8.72	3.99	3.00	5.61	4.00	1.76
$CN$	lad.sim.scad	16.65	3.82	2.83	10.78	3.94	1.55
	cqr.sim.scad	16.63	3.85	2.80	10.75	3.95	1.53
	cqr.sim.lep	16.62	3.87	2.78	10.74	3.97	1.52
	Oracle	16.61	3.89	2.77	10.71	3.98	1.50
$Outlier$	lad.sim.scad	13.76	3.95	2.84	10.24	3.97	1.84
	cqr.sim.scad	13.74	3.96	2.83	10.23	3.97	1.81
	cqr.sim.lep	13.73	3.97	2.82	10.22	3.98	1.80
	Oracle	13.72	3.98	2.81	10.20	3.99	1.78

MAD (the mean absolute deviation) of $\widehat{\gamma }$: $\mathrm{MAD}=\frac{1}{n}{\sum }_{i=1}^n|X_i^{\top }\widehat{\gamma }-X_i^{\top }{\gamma }_0|$; NC: the average number of non-zero coefficients that are correctly estimated to be non-zero; NIC: the average number of zero coefficients that are incorrectly estimated to be non-zero, respectively.

.

Scenario 2. The model set-up is similar to Example 1, except the link function is ${\mathit{X}}^{\top }{\gamma }_0+4\sqrt{|{\mathit{X}}^{\top }{\gamma }_0+1|}$. These link functions are also analyzed by [20]. Then,

Table 2. Simulation results for Scenario 2 based on 200 replications.

Error Distribution	Method	$\mathit{n}=100$			$\mathit{n}=200$
		MAD (%)	NC	NIC	MAD (%)	NC	NIC
$N(0,1)$	lad.sim.scad	9.90	3.98	3.27	8.26	4.00	1.41
	cqr.sim.scad	9.84	3.98	3.24	8.19	4.00	1.35
	cqr.sim.lep	9.81	3.99	3.23	8.16	4.00	1.34
	Oracle	9.79	3.90	3.21	8.14	4.00	1.32
$t\left(3\right)$	lad.sim.scad	12.15	3.97	3.41	8.57	4.00	1.72
	cqr.sim.scad	12.08	3.98	3.36	8.51	4.00	1.68
	cqr.sim.lep	12.05	3.99	3.34	8.50	4.00	1.67
	Oracle	12.03	3.88	3.32	8.47	4.00	1.64
$DE$	lad.sim.scad	7.63	4.00	3.16	4.97	4.00	2.18
	cqr.sim.scad	7.56	4.00	3.13	4.94	4.00	2.11
	cqr.sim.lep	7.54	4.00	3.11	4.92	4.00	2.08
	Oracle	7.51	4.00	3.09	4.89	4.00	2.06
$CN$	lad.sim.scad	12.02	3.95	3.06	11.31	3.97	1.24
	cqr.sim.scad	11.97	3.96	3.03	11.26	3.97	1.21
	cqr.sim.lep	11.96	3.98	3.02	11.25	3.98	1.18
	Oracle	11.93	3.86	2.97	11.23	3.88	1.15
$Outlier$	lad.sim.scad	11.95	3.95	3.18	9.47	4.00	1.62
	cqr.sim.scad	11.92	3.96	3.15	9.42	4.00	1.59
	cqr.sim.lep	11.89	3.98	3.13	9.41	4.00	1.58
	Oracle	11.86	3.87	3.10	9.39	4.00	1.55

MAD (the mean absolute deviation) of $\widehat{\gamma }$: $\mathrm{MAD}=\frac{1}{n}{\sum }_{i=1}^n|X_i^{\top }\widehat{\gamma }-X_i^{\top }{\gamma }_0|$; NC: the average number of non-zero coefficients that are correctly estimated to be non-zero; NIC: the average number of zero coefficients that are incorrectly estimated to be non-zero, respectively.

summarizes the corresponding results.

From Table 1 and Table 2, we can note that the performance of cqr.sim.lep is best, cqr.sim.scad is second, and lad.sim.scad is the worst for five different error distributions. This is consistent with our theoretical findings. Furthermore, with the sample size increasing, all estimators become better.

4.2. Real Data Application: Boston Housing Data

In this section, the methods are illustrated through an analysis of Boston housing data. The data (506 observations and 14 variables) are available in the package (‘MASS’) in R, and the definitions of the dependent variable (MEDV) and explanatory variables are described in

Table 3. Description of variables for Boston housing data.

Variables	Description
MEDV	Median value of owner-occupied homes in USD thousands
CRIM	Per capita crime rate by town
ZN	Proportion of residential land zoned for lots over 25,000 sq.ft.
INDUS	Proportion of non-retail business acres per town
CHAS	Charles River dummy variable (=1 if tract bounds river, 0 otherwise)
NOX	Nitric oxide concentrations (parts per 10 million)
RM	Average number of rooms per dwelling
AGE	Proportion of owner-occupied units built prior to 1940
DIS	Weighted distances to five Boston employment centers
RAD	Index of accessibility to radial highways
TAX	Full-value property-tax rate per USD 10,000
PTRATIO	Pupil–teacher ratio by town
B	$1000{(\mathrm{Bk}-0.63)}^{\wedge }2$ which $\mathrm{Bk}$ is the black proportion of the population
LSTAT	% lower status of the population

. We checked whether there were missing values in the data through the violin diagram of each variable. Figure 1 and Figure 2 show the violins between the first and last seven variables, respectively. It is obvious from Figure 1 and Figure 2 that there are obvious outliers in CRIM and Black columns. In order to test the linear relationship among variables, the heat map between the variables is given in Figure 3. It can be seen from the heat map that variables RM, Ptratio, Istat, and MEDV have certain correlations. The correlation coefficients between Indus and nox, CRIM and RAD, RAD and tax, and Indus and tax were 0.7, 0.8, 0.9, and 0.7, respectively. Therefore, there was a high correlation between variables, so the single-index regression model could be considered.

Boston housing data have been utilized by many regression studies; the potential relationship between MEDV and X-variables was also founded [10,17]. For the single-index quantile regression, [10] introduced a practical algorithm where the unknown link function $g(·)$ is estimated by local linear quantile regression and the parametric index was estimated through linear quantile regression. However, the authors did not consider the variable selection. For the single-index regression models, [17] considered the penalized LAD regression, which dealt with variable selection and estimation simultaneously. However, the LAD estimator is only the special case of the quantile estimator in which the quantile $\tau$ is equal to 0.5. Moreover, the two literature studies mentioned above only used the case of a single quantile. In this article, we constructed new sparse estimators for single-index quantile regression models based on the composite quantile regression method combined with the SCAD penalty and Laplace error penalty.

Due to the sparsity of data in the region concerned, possible quantile curves cross at both tails similar to [14]. The results of the real data example and simulation studies confirm the reasonableness and effectiveness of our method in practice.

In order to better numerical performance, we need to standardize the response variable MEDV and the predictor variables except CHAS before applying our method. The estimated coefficient is treated as 0 if its absolute value is smaller than ${10}^{-12}$. Then, the corresponding results are reported in

Table 4. Coefficient estimates for Boston housing data.

Variables	Methods
	cqr.sim.scad		cqr.sim.lep		lad.sim.scad	ls.sim.lasso	cqr.sim
	$\mathsf{\tau }$ = 0.25	$\mathsf{\tau }$ = 0.75	$\mathsf{\tau }$ = 0.25	$\mathsf{\tau }$ = 0.75	$\mathsf{\tau }$ = 0.5		$\mathsf{\tau }$ = 0.25	$\mathsf{\tau }$ = 0.75
CRIM	0.3092	0.3089	0.3082	0.3081	0.3083	0	0.3076	0.3075
ZN	0	0	0	0	0	−0.069	0	0
INDUS	0	0	0	0	0	0	0	0
CHAS	0	0	0	0	0	0	0	0
NOX	0	0	0	0	0	0	0	0
RM	−0.1884	−0.1883	−0.1871	−0.1870	−0.1872	−0.5300	−0.1866	−0.1864
AGE	0	0	0	0	0	0	0	0
DIS	0.1453	0.1451	0.1443	0.1442	0.1444	0.1163	0.1439	0.1437
RAD	0	0	0	0	0	−0.0460	0	0
TAX	0	0	0	0	0	0	0	0
PTRATIO	0.1877	0.1876	0.1870	0.1868	0.1871	0.1069	0.1865	0.1863
B	0	0	0	0	0	0	0	0
LSTAT	0.9042	0.9040	0.9031	0.9029	0.9032	0.8319	0.9024	0.9023

.

From Table 4, we see that all methods can achieve the variable selection and parameter estimation simultaneously in the real problem. Moreover, all methods choose the sparse model including RM, DIS, PTRATIO, LSTAT, the same as the one using all predictors without penalty (cqr.sim). Moreover, the estimation of cqr.sim.lep is the closest to cqr.sim. These results indicate that only four explanatory variables are significant and the rest are irrelevant.

5. Conclusions

In this article, we propose SCAD and Laplace penalized composite quantile regression estimators for single-index models in a high-dimensional case. Compared with the least squares method, composite quantile regression can obtain the robust estimator with respect to heavy-tailed error distributions and outliers.Then, a practical iterative algorithm was introduced. It is based on composite quantile regression and uses local linear smoothing to estimate the unknown link function. This method realizes parameter selection and estimation simultaneously by combining two kinds of penalty functions with the composite quantile regression. In addition, we proved that the proposed estimator has large sample properties, including $\sqrt{n}$-consistence and the oracle property. Furthermore, the estimator was evaluated and illustrated by numerical studies. Moreover, we can draw the conclusion from Boston housing data: the sparse model with the same significant variables was selected by all three estimation methods; however, the estimators of cqr.sim.lep were the closest to that of cqr.sim. This reveals that using—or not using—the LEP penalty essentially acts the same when we estimate the link function.