Robust Variable Selection with Exponential Squared Loss for the Spatial Error Model

Abstract

With the widespread application of spatial data in fields like econometrics and geographic information science, the methods to enhance the robustness of spatial econometric model estimation and variable selection have become a central focus of research. In the context of the spatial error model (SEM), this paper introduces a variable selection method based on exponential square loss and the adaptive lasso penalty. Due to the non-convex and non-differentiable nature of this proposed method, convex programming is not applicable for its solution. We develop a block coordinate descent algorithm, decompose the exponential square component into the difference of two convex functions, and utilize the CCCP algorithm in combination with parabolic interpolation for optimizing problem-solving. Numerical simulations demonstrate that neglecting the spatial effects of error terms can lead to reduced accuracy in selecting zero coefficients in SEM. The proposed method demonstrates robustness even when noise is present in the observed values and when the spatial weights matrix is inaccurate. Finally, we apply the model to the Boston housing dataset.

1. Introduction

With the widespread application of spatial effects data in various fields such as spatial econometrics, geography, and epidemiology, spatial regression models used for handling such data have been extensively studied by many scholars. Spatial effects include spatial correlation (dependence) and spatial heterogeneity. To reflect different forms of spatial correlation, Anselin (1988) [1] categorized spatial econometric models into spatial autoregressive models (SARs), spatial Durbin models (SDMs), and spatial error models (SEMs). Among them, the spatial error model is represented as $Y=\rho WY+X\beta +U,U=\omega MU+V$. The spatial error model (SEM) considers the mutual influence between spatially adjacent regions. The fundamental idea is to incorporate spatial autocorrelation as an error term into the regression model, addressing the limitation of traditional regression models in handling spatial autocorrelation issues. In the SEM model, the error terms are no longer independently and identically distributed but exhibit spatial autocorrelation with the error terms of neighboring regions.

With the rapid increase in data dimensionality, the issue of variable selection in spatial error models has attracted widespread attention. In the classic field of linear regression, there have been numerous studies on variable selection. Penalty methods are commonly used for model variable selection, and  several penalty functions have been proposed, such as the least absolute shrinkage and selection operator (LASSO, Tibshirani, 1996 [2]), smoothed trimmed absolute deviation (SCAD, Fan and Li, 2001 [3]) and adaptive lasso (Zou, 2006 [4]). Due to the spatial dependence inherent in SEM, the aforementioned penalty methods can be directly applied to variable selection in the spatial error model.

Due to observation noise and inaccurate spatial weight matrices, traditional variable selection methods can become unstable and inaccurate in parameter estimation. This can also lead to misguidance in variable selection, resulting in the erroneous choice of variables that do not reflect the actual situation.Therefore, efforts have been made to adopt more robust approaches. Many studies employ the Huber loss function (Huber, 2004 [5]); however, the Huber loss function may not be sufficiently robust in the presence of extreme outliers, and it often leads to high computational complexity when dealing with high-dimensional data. Wang et al. (2013) [6] introduced a class of robust estimates with exponential square loss functions ${\varphi }_{\gamma }=1-exp(-t^2/\gamma )$. For large values of $\gamma$, the exponential square estimates are similar to the least squares estimates; when $\gamma$ is very small, larger absolute values of $\left|t_i\right|$ observations will result in an empirical loss close to 1.0. Therefore, outliers have little impact on the estimates when $\gamma$ is small, limiting the influence of outliers on the estimates. In contrast, this method exhibits stronger robustness compared to other robust methods, including Huber estimates (Huber, 2004 [5]), quantile regression estimates (Koenker and Bassett, 1978 [7]), and composite quantile regression estimates (Zou and Yuan, 2008 [8]). Wang et al. (2013) [6] also introduced a method for selecting the parameter $\gamma$.

We focus on the variable selection problem of the spatial error model (SEM). However, currently, there is not much exploration regarding robust variable selection methods for spatial error models. Liu et al. (2020) [9] employed penalized pseudo-likelihood and SCAD penalties for variable selection in the spatial error model. Through Monte Carlo simulations, they demonstrated that this method exhibits good variable selection performance based on normal data. Dogan (2023) [10] used modified harmonic mean estimates to estimate the marginal likelihood function of the cross-sectional spatial error model, thereby improving the estimation performance of spatial error model parameters. However, their methods are still susceptible to the influence of outliers. Therefore, exploring more robust variable selection methods is imperative. Considering the robustness of the exponential square loss function, we employ an adaptive lasso penalty method in conjunction with the exponential square loss to penalize all unknown parameters in the loss function except for the variance of the random disturbance term. Song et al. (2021) [11] applied this method to the SAR model and achieved satisfactory results. In this paper, we apply the same method to the SEM model. We construct the following optimization model:

$$\begin{array}{c}\hfill {\displaystyle \underset{\beta \in R^p,\rho \in [0,1],\omega \in [0,1]}{min}{\displaystyle L(\beta ,\rho ,\omega )=\frac{1}{n}\sum _{i=1}^n{\varphi }_{\gamma }(Y_i-\rho \widetilde{Y_i}}-X_i\beta -\omega \widetilde{U_i})+\lambda \sum _{j=1}^{p}P(\left|{\beta }_j\right|),}\end{array}$$

(1)

where $\rho$, $\beta$, and $\omega$ are all parameters to be estimated, and $\tilde{Y}=WY$, $\tilde{U}=MU$. $\lambda$ is a non-negative regularization term weight coefficient. ${\displaystyle \sum _{j=1}^{p}P(\left|{\beta }_j\right|)}$ is a penalty term, and ${\varphi }_{\gamma }(·)$ is the exponential square loss mentioned earlier.

Due to the non-convex nature of the exponential square loss function, the empirical loss term is inherently a structured non-convex function with respect to three variable blocks, namely $\beta \in R^p,\rho \in [0,1],\omega \in [0,1]$. Furthermore, since many penalty methods (lasso or adaptive lasso) are non-differentiable, (1) represents a non-convex, non-differentiable optimization problem. In this paper, we propose a robust variable selection method for the spatial error model based on the exponential squared loss function and adaptive lasso penalty. This method allows for the selection of variables while simultaneously estimating the regression coefficients. The main contributions of this paper are as follows.

(1)

We established a robust variable selection method for SEM, which employs the exponential squared loss and demonstrates good robustness in the presence of outliers in the observations and inaccurate estimation of the spatial weight matrix.

(2)

We designed a BCD algorithm to solve the optimization problem in SEM. We decomposed the exponential square loss component into the difference of convex functions and built a CCCP program for solving the BCD algorithm subproblems. We utilized an accelerated FISTA algorithm to solve the optimization problem with the adaptive lasso penalty term. We also provide the computational complexity of the BCD algorithm.

(3)

We verified the robustness and effectiveness of this method through numerical simulation experiments. The simulation results indicate that neglecting the spatial effects of error terms leads to a decrease in variable selection accuracy. When there is noise in the observations and inaccuracies in the spatial weight matrix, the method proposed in this paper outperforms the comparison methods in correctly identifying zero coefficients, non-zero coefficients, and median squared error (MedSE).

The organizational structure of this paper is as follows: In Section 2, we introduce the SEM model, construct the loss function with exponential squared loss and adaptive lasso regularization, and provide methods for selecting model hyperparameters and estimating variance. In Section 3, we design an efficient algorithm to optimize the loss function, accomplishing variable selection and parameter estimation. Section 4 includes numerical simulation experiments to assess the variable selection and estimation performance of the proposed method. In Section 5, we apply this method to the Boston Housing dataset. Finally, in Section 6, we summarize the entire paper.

2. Model Estimation and Variable Selection

2.1. Spatial Error Model

We consider the following SEM model with covariates

$$\begin{array}{c}\hfill Y=\rho WY+X\beta +U,U=\omega MU+V,\end{array}$$

(2)

where $Y_i\in R$,$Y={(Y_1,...,Y_n)}^T$ is an n-dimensional response vector, $X={(X_1,...,X_n)}^T\in R^{n\times p}$ is the design matrix, and $U={(U_1,...,U_n)}^T\in R^{n\times 1}$ is the error vector. W and M are $n\times n$ spatial weight matrix. In practical applications, M is often constructed using a spatial rook matrix. The spatial autocorrelation coefficients $\rho$ and $\omega$ are scalars, while $\omega$ measures the impact of errors in the dependent variable in neighboring areas on the local observations. A larger $\omega$ indicates a stronger spatial dependence effect in the sample observations, while a smaller $\omega$ implies a weaker effect. A high-order spatial error model with p lag terms and q disturbance terms is typically represented as follows:

$$\begin{array}{c}\hfill {\displaystyle Y=\sum _{r=1}^p{\rho }_r{W}_{r}Y+X\beta +U,U=\sum _{l=1}^q{\omega }_l{M}_{l}U+V.}\end{array}$$

(3)

This study considers a first-order spatial error model. $I_n\in R_{n\times n}$ is an n-dimensional identity matrix. It is typically assumed that $V_i$ is independently and identically distributed, i.e., $V_i\backsim N(0,{\sigma }^2)$. Therefore, U and Y can be represented as follows:

$$\begin{array}{c}\hfill Y={(I_n-\rho W)}^{-1}(X\beta +U),\end{array}$$

(4)

$$\begin{array}{c}\hfill U={(I_n-\omega M)}^{-1}V,\end{array}$$

(5)

where it is necessary to ensure that $I_n-\rho W$ and $I_n-\omega M$ are invertible. According to Banerjee et al. (2014) [12], under certain normalization operations, the maximum singular value of the matrix W is 1. Therefore, condition $\left|\rho \right|<1$ ensures the invertibility of ($I_n-\rho W$). We impose constraints on $\rho$ and $\omega$, i.e., $\left|\rho \right|<1$ and $\left|\omega \right|<1$. Furthermore, we neglect the endogeneity induced by spatial dependence (Ma et al., 2020) [13].

2.2. Variable Selection Methods

This section considers variable selection for model (2). Following Fan and Li (2001) [3], it is typically assumed that the regression coefficient vector $\beta$ is sparse. A natural approach is to use penalty methods for model handling, which can simultaneously select important variables and estimate unknown parameters. The constructed penalty method can be represented as follows:

$$\begin{array}{c}\hfill {\displaystyle \underset{\beta \in R^p,\rho \in [0,1],\omega \in [0,1]}{min}{\displaystyle L(\beta ,\rho ,\omega )=\frac{1}{n}\sum _{i=1}^n{\varphi }_{\gamma }(Y_i-\rho \widetilde{Y_i}}-X_i\beta -\omega \widetilde{U_i})+\lambda \sum _{j=1}^{p}P(\left|{\beta }_j\right|).}\end{array}$$

(6)

When considering the choice of penalty methods, this paper opts for the adaptive lasso penalty. Let $\widehat{\beta }$ be a consistent estimator of $\beta$. Following the recommendation of Zou (2006) [4], we set $r=1$ and define the weight vector $\eta \in R^p,{\eta }_j=\frac{1}{\left|{\widehat{{\beta }_j}}^r\right|},j=1,\cdots ,p$. The adaptive lasso is redefined as

$$\begin{array}{c}\hfill {\displaystyle \sum _{j=1}^{p}P(\left|{\beta }_j\right|)=\sum _{j=1}^p{\eta }_j\left|{\beta }_j\right|.}\end{array}$$

(7)

The penalty-robust regression formula with an exponential square loss and adaptive lasso penalty is redefined as

$$\begin{array}{c}\hfill {\displaystyle \underset{\beta \in R^p,\rho \in [0,1],\omega \in [0,1]}{min}{\displaystyle L(\beta ,\rho ,\omega )=\frac{1}{n}\sum _{i=1}^n{\varphi }_{\gamma }(Y_i-\rho \widetilde{Y_i}}-X_i\beta -\omega \widetilde{U_i})+\lambda \sum _{j=1}^p{\eta }_j\left|{\beta }_j\right|,}\end{array}$$

(8)

where $\tilde{Y}=WY$, $\tilde{U}=MU$, and ${\varphi }_{\gamma }(·)$ is the exponential square loss mentioned earlier ${\varphi }_{\gamma }(t)=1-exp(-t^2/\gamma )$. $\lambda$ is a positive regularization coefficient. The selection of the tuning parameter $\gamma$ in the exponential square function will be discussed in the next section.

2.3. The Selection of $\gamma$

The parameter $\gamma$ controls the robustness and estimation efficiency of the proposed variable selection method. Wang et al. (2013) [6] introduced a method for selecting $\gamma$. It first identifies a set of tuning parameters such that the proposed penalty estimates have an asymptotic break-even point at 0.5, and then selects the parameter based on the principle of maximum efficiency. The process is as follows:

Step 1. Initialize a set of parameters with $\widehat{\beta }={\beta }^{(0)}$, $\widehat{\rho }={\rho }^{(0)}$, $\widehat{\omega }={\omega }^{(0)}$, where ${\rho }^{(0)}={\omega }^{(0)}=0.5$. Rewrite the model as $Y^{*}=X\beta +U$, where $U^{*}=\omega MU$ and $Y^{*}=Y-\rho WY-U^{*}$.

Step 2. Let $D_n=(X_1,Y_1^{*}),...,(X_n,Y_n^{*})$ be a sample set that consists of a collection of samples, and calculate $r_i(\widehat{\beta })=Y_i^{*}-X_i\widehat{\beta },$ $i=1,\cdots ,n$ and $S_n=1.4826\times media{n}_i\left|r_i(\widehat{\beta })-media{n}_j(r_j(\widehat{\beta }))\right|$. Then, there exists a set of pseudo-outliers

$D_m=\left\{(X_i,Y_i):\left|r_i(\widehat{\beta })⩾2.5S_n\right|\right\}$, $m=♯\left\{1\le i\le n:\left|r_i(\widehat{\beta })\right|⩾2.5S_n\right\}$, and $D_{n-m}=D_n\backslash D_m$.

Step 3. Construct $\widehat{V}(\gamma )={\left\{\widehat{I}(\widehat{\beta })\right\}}^{-1}{\tilde{\Sigma }}_2{\left\{\widehat{I}(\widehat{\beta })\right\}}^{-1}$, where

$$\begin{array}{cc}\hfill & \widehat{I}(\widehat{\beta })=\frac{2}{\gamma }\left\{\frac{1}{n}\sum _{i=1}^{n}exp\left(-r_i^2(\widehat{\beta })/\gamma \right)\left(\frac{2r_i(\widehat{\beta })}{\gamma }-1\right)\right\}·\left(\frac{1}{n}\sum _{i=1}^n{X}_i{X}_i^T\right),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & {\tilde{\Sigma }}_2=Cov\left\{exp\left(-r_1^2(\widehat{\beta })/\gamma \right)\frac{2r_1(\widehat{\beta })}{\gamma }{X}_1,\dots ,exp\left(-r_n^2(\widehat{\beta })/\gamma \right)\frac{2r_n(\widehat{\beta })}{\gamma }{X}_n\right\}.\hfill \end{array}$$

(10)

Next, let ${\gamma }_n$ be the minimum value of $det(\widehat{V}(\gamma ))$ in the set $G=\{\gamma :\zeta (\gamma )\in (0,1\left]\right\}$, where $\zeta (·)$ is defined the same way as in Wang et al. (2013) [6].

Step 4. The optimal solution from the optimization problem ${min}_{\beta \in R^p,\rho \in [0,1],\omega \in [0,1]}$$L(\beta ,\rho ,\omega )=\frac{1}{n}{\sum }_{i=1}^n{\varphi }_{\gamma }(Y_i-\rho \widetilde{Y_i}-X_i\beta -\omega \widetilde{U_i})$ updates $\widehat{\rho }$, $\widehat{\beta }$, and $\widehat{\omega }$, where $\tilde{Y}=WY$ and $\tilde{U}=MU$. Terminate if the result converges; otherwise, return to Step 2.

Considering the potential difficulties arising from the significant computational burden, we do not use cross validation to select $\gamma$ and the penalty coefficient $\lambda$. In practical applications, we can find a threshold ${\gamma }_1$ for which $\zeta ({\gamma }_1)=1$. The values of $\gamma$ typically fall within the interval $[5{\gamma }_1,30{\gamma }_1]$. In the initial steps mentioned above, an initial estimate ${\beta }^{(0)}$ is required. In numerical simulations, we use an estimate with the LAD loss as the initial estimate.

2.4. The Selection of $\lambda$ and ${\eta }_j$

When considering the selection of $\lambda$ and ${\eta }_j$, it is common to unify the two parameters: ${\lambda }_j=\lambda ·{\eta }_j$. Various methods such as AIC (Akaike Information Criterion), BIC (Bayesian Information Criterion), and cross validation can be used to choose these parameters.

To ensure the consistency of the variable selection and reduce computational complexity, we follow the method described in Wang et al. (2007) [14] by selecting the regularization parameter through the minimization of the BIC objective function. The objective function is as follows:

$$\begin{array}{c}\hfill {\displaystyle \sum _{i=1}^n\left[1-exp\left\{-{(Y_i-\rho \widetilde{Y_i}-X_i\beta -\omega \widetilde{U_i})}^2/{\gamma }_n\right\}\right]+n\sum _{j=1}^p{\lambda }_j\left|{\beta }_j\right|-\sum _{j=1}^{p}log(0.5n{\lambda }_j)log(n).}\end{array}$$

(11)

From this, we can deduce that ${\lambda }_j=\frac{log(n)}{n\left|{\beta }_j\right|}$. In practical applications, we may not know the values of ${\beta }_j$, but we can easily obtain ${\tilde{\beta }}_j$ through the exponential square loss. In Section 2.3, we obtained the estimate for $\gamma$. It is important to note that when making this simple selection, the following conditions should be satisfied: $\sqrt{n}{\widehat{\lambda }}_j\stackrel{}{\to }0,j\le p_0$ and $\sqrt{n}{\widehat{\lambda }}_j\stackrel{}{\to }\infty ,j>p_0$, where $p_0$ is the number of non-zero values in $\beta$. This ensures the consistency of the variable selection in the final estimate.

2.5. Estimation of Noise Variance

We estimate the variance of the random disturbance term; let $G={(I_n-\rho W)}^{-1}$, $F={(I_n-\omega M)}^{-1}$, then the estimated variance of the random disturbance term V is given by

$$\begin{array}{c}\hfill {\widehat{\sigma }}^2=\frac{1}{n}(Y-GX\beta ){\left[(GF){(GF)}^T\right]}^{-1}(Y-GX\beta ),\end{array}$$

(12)

where $\beta$, $\rho$, and $\omega$ can be obtained by solving Equation (8). It is known from the constraints on $\rho$ and $\omega$ that both G and F are non-singular matrices. Let $u=GX\beta$ and $v=GF$. Then, the estimated ${\widehat{\sigma }}^2$ defined in Equation (10) can be calculated using the following equation:

$$\begin{array}{c}\hfill {\widehat{\sigma }}^2=\frac{1}{n}{∥v·(y-u)∥}_2^2.\end{array}$$

(13)

3. Model Solving Algorithm

This section outlines the algorithm for solving Equation (8). In this optimization problem, the objective function can be divided into three variable blocks: $\beta \in R^p$, $\rho \in [0,1]$, and $\omega \in [0,1]$. We can use a block coordinate descent algorithm to solve it. However, the objective function in Equation (8) is non-convex and non-smooth. Using block coordinate descent alone can be challenging to ensure convergence. Therefore, we perform a convex–concave procedure (CCCP) by decomposing the exponential square loss part and utilize the CCCP algorithm to solve the subproblem for $\beta$. For the lasso and adaptive lasso penalty terms, we use the ISTA (Iterative Soft Thresholding) algorithm, and this paper employs an accelerated FISTA algorithm.

3.1. Block Coordinate Descent Algorithm

In Algorithm 1, we provide the framework of a block coordinate descent algorithm for the alternating iterative solution of $\beta ,\omega ,\rho$.

The next tasks are to solve subproblems (14)–(16). Subproblems (14) and (15) can be transformed into single-variable function optimization problems with the other two parameters fixed. The range for achieving the optimal solution of the objective function is [0, 1]. Therefore, the parabolic interpolation-based golden section method can be employed for solving them. For specific algorithm details, please refer to Forsythe et al. (1977) [15]. In the next section, we will discuss the solution to (16) in detail.

Algorithm 1 BCD algorithm framework.

1. Set the initial iteration point ${\beta }^0\in R^p$, ${\rho }^0\in [0,1]$, ${\omega }^0\in [0,1]$; 2. Repeat $k=\left\{k=0,1,2,...\right\}$; 3. Solve the subproblem about $\rho$ with initial point ${\rho }^k$; $${\rho }^{k+1}\leftarrow \underset{\rho \in [0,1]}{min}L\left({\beta }^k,\rho ,{\omega }^k\right)$$ (14) 4. Solve the subproblem about $\omega$ with initial point ${\omega }^k$; $$\underset{\omega \in [0,1]}{min}L\left({\beta }^k,{\rho }^{k+1},\omega \right)\stackrel{}{\to }{\omega }^{k+1}$$ (15) 5. Solve the subproblem with initial value ${\beta }^k$, $${\beta }^{k+1}\leftarrow \underset{\beta \in R^p}{min}L\left(\beta ,{\rho }^{k+1},{\omega }^{k+1}\right)$$ (16) to obtain a solution ${\beta }^{k+1}$, ensuring that $L({\beta }^k,{\rho }^{k+1},{\omega }^{k+1})-L({\beta }^{k+1},{\rho }^{k+1},{\omega }^{k+1})\le 0$, and that ${\beta }^{k+1}$ is a stationary point of $L({\beta }^k,{\rho }^{k+1},{\omega }^{k+1})$. 6. Iterate until convergence.

3.2. Solving Subproblem (16)

Given that the optimization problem is composed of the exponential square loss term and the lasso or adaptive lasso penalty term, it can be observed that when $\rho$ and $\omega$ are fixed, the lasso or adaptive lasso is convex. The exponential square loss can be decomposed into the difference of two convex functions. Therefore, subproblem (16) is a DC (difference of convex) program, and it can be solved using the appropriate algorithms. When decomposing the exponential square loss function into a DC form, we made the following attempts.

Proposition 1. 

${\varphi }_{\gamma }(t)=1-exp(-t^2/\gamma )$, We assume that ${\varphi }_{\gamma }(t)=f(t)-g(t)$, where both $f(t)$ and $g(t)$ are convex functions. We can assume that $g^{″}(t)=\frac{4}{{\gamma }^2}{t}^2$ based on the non-negativity of the second derivative. Then, we set:

$$\begin{array}{c}\hfill g(t)=\frac{1}{3{\gamma }^2}{t}^4,\end{array}$$

(17)

$$\begin{array}{c}\hfill f(t)={\varphi }_{\gamma }(t)+g(t)=1-exp(-t^2/\gamma )+\frac{1}{3{\gamma }^2}{t}^4.\end{array}$$

(18)

It can be proven that both $f(t)$ and $g(t)$ are convex functions, completing the DC decomposition of the exponential square part ${\varphi }_{\gamma }(t)=f(t)-g(t)$. We can then define the following two functions:

$$\begin{array}{c}\hfill {\displaystyle J_{vex}(\beta )=\frac{1}{n}\sum _{i=1}^{n}f(Y_i-{\rho }^{k+1}{W}_{i}Y-X_i\beta -{\omega }^{k+1}{M}_{i}U)+\lambda \sum _{j=1}^p{\eta }_j\left|{\beta }_j\right|,}\end{array}$$

(19)

$$\begin{array}{c}\hfill {\displaystyle J_{cav}(\beta )=\frac{1}{n}\sum _{i=1}^{n}g(Y_i-{\rho }^{k+1}{W}_{i}Y-X_i\beta -{\omega }^{k+1}{M}_{i}U),}\end{array}$$

(20)

where $f(·)$ and $g(·)$ are defined in (17) and (18), $W_i$ and $M_i$ represent the i-th row of the spatial weight matrices W and M, $\lambda {\sum }_{j=1}^p{\eta }_j\left|{\beta }_j\right|$ is the convex penalty with respect to $\beta$. Then, $J_{vex}(·)$ and $J_{cav}(·)$ are convex and concave functions, respectively. Subproblem (16) can be rewritten as

$$\begin{array}{c}\hfill \underset{\beta \in R^p}{min}L\left(\beta ,{\rho }^{k+1},{\omega }^{k+1}\right)=J_{vex}(\beta )+J_{cav}(\beta ).\end{array}$$

(21)

For this convex–concave procedure, we can use Algorithm 2 for solving it as described in Yuille et al. (2003) [16].

Algorithm 2 Convex–concave procedure algorithm.

1. Set an initial point ${\beta }^0$ and initialize $k=0$. 2. Repeat 3. $$\begin{array}{c}\hfill {\beta }^{k+1}=argmi{n}_{\beta }{J}_{vex}(\beta )+J_{cav}^{\prime }({\beta }^k)·\beta \end{array}$$ (22) 4. Update the iteration counter: $k=k+1$ 5. Until convergence of ${\beta }^k$

The CCCP algorithm is a monotonically decreasing global optimization method that approximates the global optimal solution by alternating between optimizing the convex and concave parts. Therefore, it is possible to use iterative solutions to minimize the objective function in (21) by solving (22). We are considering which algorithm to use for solving (22) to improve the efficiency of Algorithm 2.

We notice that the first term in (22) consists of a convex function ${\displaystyle \frac{1}{n}\sum _{i=1}^{n}f(·)}$ and a convex penalty term on $\beta$ given by ${\displaystyle \lambda \sum _{j=1}^p{\eta }_j\left|{\beta }_j\right|}$, while the second term is a linear function of $\beta$. We can rewrite (22) as follows:

$$\begin{array}{c}\hfill \underset{\beta \in R^p}{min}{\displaystyle \psi (\beta )+\lambda \sum _{i=1}^p{\eta }_i\left|{\beta }_j\right|},\end{array}$$

(23)

where $\psi (\beta )$ is a continuously differentiable convex function, defined as ${\displaystyle \psi (\beta )=\frac{1}{n}\sum _{i=1}^{n}f(Y_i-{\rho }^{k+1}{W}_{i}Y-X_i\beta -{\omega }^{k+1}{M}_{i}U)+J_{cav}^{\prime }({\beta }^k)·\beta }$. Beck and Teboulle (2009) [17] introduced an algorithm called ISTA (Iterative Soft Thresholding Algorithm) to solve optimization problems with lasso penalties. Song et al. (2021) [11] proved that this algorithm can be applied to models with adaptive lasso penalties. Therefore, ISTA can be used to solve problems structured like (23). This paper chooses the accelerated version of ISTA, known as the FISTA algorithm, for optimization. The convergence properties of FISTA have been established by Beck and Teboulle (2009) [17]. Algorithm 3 provides an overview of the FISTA algorithm.

With this, we complete the algorithm design for solving (16). In addition, we provide the computational complexity of the BCD program and an introduction to the machine specifications used in subsequent experiments in Appendix B.

Algorithm 3 FISTA algorithm with backtracking step for solving (22).

Require: $A,\xi ,w\lambda >0$ Ensure: solution $\beta$ 1: Step 0. Select $L^0>0,\eta >1,{\beta }^0\in R^{2p}$. Let ${\xi }^1={\beta }^0,t^1=1$ 2: Step $k(k\ge 1)$. 3:  Determine the smallest non-negative integer $i^k$ which makes $\overline{L}={\eta }^{i^k}{L}^{k-1}$ satisfy 4: $$F\left({\Theta }_{\overline{L}}\left({\xi }^k\right)\right)\le Q_{\overline{L}}\left({\Theta }_{\overline{L}}\left({\xi }^k\right),{\xi }^k\right).$$ 5: Let $L^k={\eta }^{i^k}{L}^{k-1}$ and calculate: 6: ${\beta }^k={\Theta }_{L^k}\left({\xi }^k\right)$ 7: $t^{k+1}=\frac{1}{2}\left[1+\sqrt{1+4{\left(t^k\right)}^2}\right]$ 8: ${\xi }^{k+1}={\beta }^k+\frac{t^k-1}{t^{k+1}}\left({\beta }^k-{\beta }^{k-1}\right)$ 9: Output $\beta :={\beta }^k$.

4. Numerical Simulations

In this section, we conducted several sets of numerical simulations to validate the impact of spatial errors on the estimation and variable selection of the SEM model. We also compared the performance of the proposed variable selection method with other methods in various scenarios, including small and large sample sizes, the presence of many non-significant covariates, noisy response variable observations, and inaccurate spatial weight matrix.

4.1. Generation of Simulated Data

The data generation process is based on model (2). We consider covariates to follow a normal distribution with a mean of zero and a covariance matrix of $({\sigma }_{ij})$, where ${\sigma }_{ij}={\left|0.5\right|}^{i-j}$. In other words, the design matrix $X_n$ is an $n\times (q+3)$ matrix. Let the sample size $n\in \left\{30,150,300\right\}$ and the number of non-significant covariates $q\in \left\{5,20,100,200\right\}$.

The spatial autoregressive coefficient $\rho$ follows a uniform distribution on the interval $[{\rho }_1-0.1,{\rho }_1+0.1]$, where ${\rho }_1\in \left\{0.8,0.5,0.2\right\}$. To verify whether spatial dependence among response variables affects the model estimation and variable selection, an additional experiment with $\rho =0$ is included. In this case, the SEM model reduces to a linear error model. The spatial dependence coefficient in the error term $\omega \in [0,0.6]$ is used to examine the influence of spatial effects in the error terms on SEM model estimation and variable selection. The coefficients for covariates are set as a sparse vector $\beta =({\beta }_1,{\beta }_2,{\beta }_3,0_q)$, where q is the number of non-significant covariates. The values of $({\beta }_1,{\beta }_2,{\beta }_3)$ are sampled from a normal distribution with means of $(3,2,1.6)$ and a variance of $0.01\times I_3$, where $I_3$ is the $3\times 3$ identity matrix in $R^{3\times 3}$.

Let the spatial weight matrix $W_n=I_R\otimes B_m$, where $B_m=\frac{1}{m-1}(1_m·1_m^T-I_m)$, ⊗ denotes the Kronecker product, and $I_m$ is an m-dimensional column vector consisting of ones. In this study, let $m=3$. The error term’s spatial weight matrix M is constructed as a spatial rook matrix, considering only adjacent elements in the horizontal and vertical directions. In this matrix, adjacent element positions are set to 1, while all other positions are set to 0. The response variable Y and the error term U are generated according to the (4) and (5) provided in the context.

Independent random disturbance terms $V\backsim N(0,{\sigma }^2{I}_n)$. The parameter ${\sigma }^2$ is uniformly distributed in the interval [${\sigma }_1$− 0.1, ${\sigma }_1$ + 0.1], where ${\sigma }_1=1$. When considering noise in the response variables, the interference term follows a mixture of normal distributions. Specifically, V is distributed as a mixture of two normal distributions $V\sim (1-{\delta }_1)·N(0,1)+{\delta }_1·N(10,6^2)$, where ${\delta }_1=0.01$.

We also designed inaccurate spatial weight matrix W by randomly removing $30\%$ of the non-zero values in each row of W and adding $50\%$ non-zero elements randomly in each row of W. Then, the constructed inaccurate W was normalized and used in the SEM model.

Each scenario was based on 100 simulations. To gauge the model’s superiority, we compared it with square loss and absolute loss. For accurate performance evaluation, we employed the median squared error (MedSE, Liang and Li, 2009) [18] as the metric for accuracy comparison.

4.2. Unregularized Estimation on Gaussian Noise Data

In this section, the unregularized estimation of independent variable coefficients, spatial weight coefficients, and noise variance is performed using the SEM model on Gaussian noise data for both $q=5$ and $q>5$.

When considering the spatial effect of the error term ($\omega =0.6$), the model’s estimation results are shown in Figure 1. (1) All three loss functions yield estimates of ${\beta }_1,{\beta }_2,{\beta }_3$, and ${\sigma }^2$ that closely align with their true values (${\beta }_1,{\beta }_2,{\beta }_3$ have means of 3, 2, 1.6, and the mean of ${\sigma }^2$ is 1) as seen in

Table A1. Unregularized estimation on normal data ($q=5$).

	n = 30, q = 5			n = 150, q = 5			n = 300, q = 5
	Exp	Square	LAD	Exp	Square	LAD	Exp	Square	LAD
$\rho =0.8$, $\omega =0.6$
${\beta }_1$	4.171	3.072	3.098	3.254	3.123	3.084	3.078	3.146	3.074
${\beta }_2$	2.259	2.012	2.312	1.974	2.109	2.171	2.138	2.086	2.061
${\beta }_3$	1.118	1.659	1.834	1.672	1.678	1.702	1.579	1.617	1.708
$\widehat{\rho }$	0.742	0.786	0.734	0.797	0.796	0.788	0.792	0.798	0.795
${\widehat{\sigma }}^2$	1.300	0.804	1.808	1.164	1.040	1.105	1.121	1.081	1.094
MedSE	1.703	0.764	1.695	0.224	0.371	0.458	0.236	0.245	0.297
$\rho =0.5$, $\omega =0.6$
${\beta }_1$	3.114	2.939	2.898	3.094	2.987	2.911	2.935	3.042	3.024
${\beta }_2$	2.011	2.157	1.964	1.891	2.020	2.059	2.052	2.003	2.022
${\beta }_3$	1.646	1.622	1.739	1.607	1.578	1.580	1.542	1.565	1.629
$\widehat{\rho }$	0.507	0.502	0.500	0.510	0.510	0.500	0.509	0.509	0.506
${\widehat{\sigma }}^2$	0.557	0.735	0.947	1.098	0.981	0.966	1.066	1.030	1.045
MedSE	0.398	0.703	0.964	0.176	0.290	0.357	0.140	0.188	0.244
$\rho =0$, $\omega =0.6$
${\beta }_1$	2.910	2.957	2.922	3.082	3.000	2.912	2.942	3.029	3.027
${\beta }_2$	2.004	2.156	1.931	1.891	2.050	2.063	2.056	2.012	2.047
${\beta }_3$	1.885	1.632	1.711	1.609	1.594	1.620	1.544	1.572	1.612
$\widehat{\rho }$	0.058	0.016	0.054	0.006	0.002	0.007	0.000	0.000	0.007
${\widehat{\sigma }}^2$	0.581	0.740	0.938	1.107	0.979	0.974	1.070	1.031	1.047
MedSE	0.517	0.648	0.962	0.185	0.293	0.372	0.139	0.185	0.242
$\rho =0.8$, $\omega =0$
${\beta }_1$	4.294	2.237	2.721	2.528	2.567	1.751	1.212	1.820	1.942
${\beta }_2$	0.096	2.414	1.975	1.233	1.381	1.673	−0.259	1.257	1.873
${\beta }_3$	2.146	1.226	1.898	0.244	1.457	1.447	1.089	1.216	0.794
$\widehat{\rho }$	0.911	0.964	0.929	0.987	0.991	0.999	1.000	0.995	1.000
${\widehat{\sigma }}^2$	13.450	7.520	5.206	55.573	15.911	31.349	32.997	67.428	15.859
MedSE	3.484	4.041	8.717	2.598	3.896	5.733	4.231	5.101	6.523
$\rho =0.5$, $\omega =0$
${\beta }_1$	3.314	2.475	2.978	2.623	2.747	2.168	1.433	2.119	1.942
${\beta }_2$	0.282	2.319	1.865	1.418	1.429	1.963	−0.407	1.558	1.298
${\beta }_3$	2.693	1.259	1.840	0.275	1.595	1.636	1.209	1.394	1.018
$\widehat{\rho }$	0.860	0.921	0.832	0.955	0.953	0.996	0.984	0.964	1.000
${\widehat{\sigma }}^2$	12.508	7.199	4.851	6.237	20.207	3.899	18.063	40.884	25.945
MedSE	3.163	3.674	7.766	2.449	4.086	5.785	4.624	5.503	6.422
$\rho =0$, $\omega =0$
${\beta }_1$	3.943	2.692	3.217	3.176	3.330	2.444	1.827	2.874	1.896
${\beta }_2$	0.362	2.673	1.921	1.623	1.718	2.273	−0.573	1.892	1.911
${\beta }_3$	2.994	1.392	1.646	0.314	1.997	1.623	1.401	1.618	1.010
$\widehat{\rho }$	0.595	0.695	0.500	0.785	0.795	0.883	0.866	0.812	0.931
${\widehat{\sigma }}^2$	17.035	37.543	42.054	28.235	29.141	58.824	13.277	12.483	13.397
MedSE	3.593	4.421	7.071	2.809	4.954	6.589	5.417	6.489	9.667

. As the sample size increases, the estimates gradually converge to the true values. Notably, the square loss function provides the most accurate estimates. (2) When not considering the spatial effect of the response variable ($\rho =0$), the SEM model also produces reasonably accurate estimates. However, ignoring the spatial effect of the error term ($\omega =0$) significantly reduces the accuracy of the SEM estimates. This is especially apparent in the estimation of the noise variance, and the estimates for $\beta$ and $\rho$ also become less accurate, resulting in an increase in MedSE. (3) According to Figure 2, by comparing the median squared error (MedSE), it is evident that the SEM model estimates based on the squared loss function perform the best in terms of estimation performance.

The results in Figure 3 illustrate the estimation performance of the Structural Equation Modeling (SEM)model on normal data when there are numerous non-significant covariates. It is evident that the accuracy of the parameter estimates for the three loss functions is much lower than the results shown in Figure 1. Specifically, there are changes in the median values of parameter estimates and an increase in the number of outliers, with the exponential square loss function yielding the fewest outlier estimates. Examining the specific data in

Table A2. Unregularized estimation on high-dimensional data ($q>5$).

	n = 30, q = 20			n = 150, q = 100			n = 300, q = 200
	Exp	Square	LAD	Exp	Square	LAD	Exp	Square	LAD
$\rho =0.8$, $\omega =0.6$
${\beta }_1$	1.099	3.608	4.362	3.558	3.666	4.079	4.763	3.673	4.484
${\beta }_2$	1.237	2.772	2.246	4.340	2.275	2.726	1.996	2.440	2.662
${\beta }_3$	−0.108	1.765	2.791	1.618	1.829	2.114	1.889	1.959	2.162
$\widehat{\rho }$	0.510	0.598	0.500	0.532	0.679	0.500	0.545	0.674	0.500
${\widehat{\sigma }}^2$	45.621	1.288	6.188	8.776	1.492	11.372	7.246	1.937	10.486
MedSE	11.084	7.086	12.913	9.651	4.764	9.521	9.006	4.919	9.523
$\rho =0.5$, $\omega =0.6$
${\beta }_1$	1.088	2.958	3.044	3.127	3.033	3.005	3.086	2.962	3.037
${\beta }_2$	1.500	1.958	1.849	2.023	1.999	1.959	2.173	1.984	1.976
${\beta }_3$	1.112	1.753	1.695	1.312	1.618	1.658	1.225	1.615	1.596
$\widehat{\rho }$	0.505	0.502	0.500	0.502	0.505	0.500	0.509	0.504	0.500
${\widehat{\sigma }}^2$	19.757	0.183	0.557	0.587	0.335	0.723	0.410	0.353	0.704
MedSE	6.721	2.618	2.710	2.043	1.985	2.540	2.176	1.930	2.357
$\rho =0.2$, $\omega =0.6$
${\beta }_1$	2.530	2.940	3.104	3.187	2.946	2.956	2.851	2.925	2.858
${\beta }_2$	1.599	1.928	1.729	1.665	2.029	1.839	2.298	1.928	1.960
${\beta }_3$	1.964	1.804	1.844	1.250	1.546	1.485	1.246	1.568	1.517
$\widehat{\rho }$	0.492	0.401	0.500	0.430	0.321	0.500	0.408	0.321	0.500
${\widehat{\sigma }}^2$	0.229	0.239	1.066	0.986	0.383	1.295	0.507	0.404	1.469
MedSE	5.446	3.522	4.221	2.856	2.264	3.505	2.477	2.228	3.432
$\rho =0$, $\omega =0.6$
${\beta }_1$	0.731	2.994	2.911	3.556	3.025	3.083	2.995	2.977	2.896
${\beta }_2$	1.535	1.908	1.939	1.641	2.073	1.861	2.320	1.990	2.067
${\beta }_3$	2.216	1.834	2.009	1.102	1.575	1.596	1.257	1.607	1.486
$\widehat{\rho }$	0.476	0.164	0.500	0.336	0.109	0.500	0.148	0.083	0.500
${\widehat{\sigma }}^2$	3.716	0.259	1.777	2.420	0.371	2.517	0.471	0.389	2.854
MedSE	9.766	3.359	5.954	4.412	2.107	4.720	2.360	2.082	4.605
$\rho =0.8$, $\omega =0$
${\beta }_1$	−4.722	2.551	6.874	−2.831	3.651	7.391	−0.018	2.533	4.660
${\beta }_2$	8.293	2.333	4.512	−2.847	2.325	2.775	0.048	2.883	0.733
${\beta }_3$	3.117	1.760	2.022	4.418	0.425	6.648	−0.110	1.959	5.006
$\widehat{\rho }$	0.508	0.821	0.500	0.717	0.898	0.500	0.833	0.903	0.500
${\widehat{\sigma }}^2$	27.293	20.282	24.512	17.575	19.964	22.157	9.306	13.666	16.381
MedSE	39.686	27.898	67.610	35.056	35.448	31.470	14.067	13.665	25.497
$\rho =0.5$, $\omega =0$
${\beta }_1$	−0.634	2.694	3.102	−2.223	3.472	4.273	−0.018	2.608	2.457
${\beta }_2$	3.128	1.900	3.563	−3.418	1.873	0.452	0.048	2.129	1.651
${\beta }_3$	1.783	2.053	2.287	7.501	0.888	4.250	−0.110	1.498	3.339
$\widehat{\rho }$	0.506	0.767	0.500	0.674	0.858	0.500	0.604	0.868	0.500
${\widehat{\sigma }}^2$	32.226	29.173	27.807	12.572	14.280	15.795	9.519	14.606	13.914
MedSE	15.091	19.440	28.257	23.328	24.803	57.961	4.067	19.192	10.382
$\rho =0.2$, $\omega =0$
${\beta }_1$	0.333	2.403	3.076	−1.754	3.674	3.566	−0.018	2.733	3.423
${\beta }_2$	2.118	1.904	3.280	−3.263	2.101	0.280	0.048	2.335	1.651
${\beta }_3$	1.454	2.556	2.786	8.025	1.190	3.249	−0.110	1.360	3.278
$\widehat{\rho }$	0.499	0.655	0.500	0.603	0.801	0.500	0.390	0.805	0.500
${\widehat{\sigma }}^2$	20.537	17.192	33.633	21.775	23.257	26.938	25.876	15.694	84.560
MedSE	10.570	18.943	18.371	24.953	26.615	14.682	14.067	14.201	18.278
$\rho =0$, $\omega =0$
${\beta }_1$	0.271	2.498	3.162	−1.888	3.922	3.214	−0.018	2.932	4.047
${\beta }_2$	1.934	1.877	1.642	−2.505	2.193	0.938	0.048	2.427	1.655
${\beta }_3$	1.054	2.561	1.787	3.895	1.224	3.179	−0.110	1.403	1.953
$\widehat{\rho }$	0.497	0.546	0.500	0.545	0.690	0.500	0.382	0.714	0.500
${\widehat{\sigma }}^2$	25.288	21.138	30.855	25.724	62.474	21.793	52.331	16.704	7.310
MedSE	10.854	19.911	17.329	14.779	18.196	22.821	14.067	11.641	18.816

, it is observed that as the sample size increases, the estimates for parameters gradually approach the true values, and MedSE shows a decreasing trend. However, as anticipated, the results are still unsatisfactory.

4.3. Unregularized Estimation When the Observed Values of Y Have Outliers

In this section, we use the SEM model to estimate the data with outliers in the response variable without regularization. (1) Similar to Table A1, all three loss functions provide reasonably accurate estimates for $\beta$ and ${\sigma }^2$, and as the sample size increases, the estimates get closer to the true values. Furthermore, the exponential loss, in particular, provides better estimates for $\beta$ and ${\sigma }^2$ than the other two methods. (2) When ignoring the spatial effect of the model’s error term, the SEM estimates for the coefficients and noise variance are not accurate, and MedSE increases significantly compared to when $\omega \ne 0$. (3) By comparing Figure 1a with Figure 4a, it can be observed that the exponential square loss demonstrates better resistance to the influence of noise in the observed values. Additionally, through Figure 4b, similar results to Figure 2 can be observed, further highlighting the robustness advantage of the exponential square loss.

4.4. Unregularized Estimation with Noisy Spatial Weight Matrix

In this section, we designed an inaccurate spatial weight matrix and used the SEM model to estimate the coefficients and noise variance. “Remove 30% ” and “Add 50%” refer to randomly removing $30\%$ of non-zero values in each row of W and adding $50\%$ non-zero elements in each row, respectively.

Table A4. Unregularized estimation with noisy W.

	n = 30, q = 5			n = 150, q = 5			n = 300, q = 5
	Exp	Square	LAD	Exp	Square	LAD	Exp	Square	LAD
Remove 30%
$\rho =0.5$, $\omega =0.6$
${\beta }_1$	2.354	2.881	3.018	3.018	3.122	3.127	3.037	3.090	3.127
${\beta }_2$	2.987	2.073	1.746	1.912	2.077	1.971	2.202	2.009	2.021
${\beta }_3$	1.239	1.620	1.547	1.791	1.698	1.552	1.529	1.693	1.656
$\widehat{\rho }$	0.461	0.436	0.499	0.417	0.387	0.410	0.385	0.399	0.396
${\widehat{\sigma }}^2$	1.457	1.266	1.811	1.340	1.313	1.362	1.367	1.377	1.332
MedSE	1.473	0.984	1.323	0.417	0.447	0.575	0.272	0.353	0.386
Remove 30%
$\rho =0.5$, $\omega =0$
${\beta }_1$	2.950	2.223	2.784	2.058	2.485	2.845	1.364	2.763	2.652
${\beta }_2$	1.032	2.676	2.846	2.162	1.694	1.471	0.513	1.640	1.773
${\beta }_3$	1.381	1.138	0.034	−0.095	1.204	2.104	1.350	1.173	1.009
$\widehat{\rho }$	0.880	0.860	0.744	0.893	0.927	0.907	0.938	0.933	0.933
${\widehat{\sigma }}^2$	16.184	21.415	28.473	19.800	33.276	36.677	15.618	23.684	29.837
MedSE	3.155	4.693	9.478	2.925	5.896	8.407	3.934	7.624	9.886
Add 50%
$\rho =0.5$, $\omega =0.6$
${\beta }_1$	4.936	3.286	3.396	3.649	3.266	3.365	3.334	3.379	3.279
${\beta }_2$	1.816	2.287	2.220	2.120	2.220	2.092	2.103	2.167	2.210
${\beta }_3$	0.613	1.907	1.653	1.990	1.756	1.779	1.729	1.772	1.896
$\widehat{\rho }$	0.381	0.425	0.500	0.364	0.415	0.443	0.408	0.419	0.430
${\widehat{\sigma }}^2$	2.532	1.972	2.801	1.963	1.540	1.640	1.436	1.511	1.589
MedSE	2.799	1.662	2.170	0.824	0.852	0.927	0.535	0.637	0.673

presents the estimation results. All simulated data are generated based on $\rho =0.5$ and ${\sigma }_1=1$. Compared to the estimation results on normal data in Table A1, the inaccurate W leads to increased MedSE values and a decrease in the estimation performance for all three loss functions. To observe the variations in MedSE for each loss function, we plotted Figure 5. (1) When some non-zero weights in the matrix W are removed, the MedSE for all three loss functions decreases with an increase in the sample size. Among the three loss functions, the MedSE for the exponential square loss shows the most significant decrease, regardless of whether weights are added or removed. (2) When half of the non-zero weights in each row of W are added, all three methods have higher MedSE values compared to removing some non-zero weights from W. (3) Through the comparison of the estimated values and MedSE, we find that the exponential square loss shows more robustness when combating the impact of inaccurate spatial weight matrices.

4.5. Variable Selection with Regularizer on Gaussian Noise Data

In this section, we perform variable selection on the generated Gaussian noise data (including $q=5$ and $q>5$) using the SEM model with three different loss functions and the penalty method (lasso or adaptive lasso). “Correct” represents the average number of correctly selected zero coefficients, and “InCorrect” represents the average number of non-zero coefficients incorrectly identified as zero. “Exp + l1 ”, “S + l1” and “Lad + l1” represent SEM models with exponential loss, square loss, and absolute loss, respectively, using the lasso penalty. “Exp + ${\tilde{l}}_1$”, “S + ${\tilde{l}}_1$” and “Lad + ${\tilde{l}}_1$” represent SEM models with exponential loss, square loss, and absolute loss, respectively, using the adaptive lasso penalty. The sample size for the variable selection simulation experiments is $n\in \left\{30,300\right\}$.

We visualized the results of variable selection for the model when considering the spatial effect of the error term and created Figure 6. (1) By observing Figure 6a, it is evident that the correctness of selecting non-zero regression coefficients for the three loss functions is relatively low when the sample size is small. Additionally, the loss functions incorporating the adaptive lasso penalty achieve higher accuracy in variable selection compared to those with the lasso penalty alone. As the sample size increases, all methods can almost correctly select all variables. A comparison reveals that the exponential square loss, combined with the adaptive lasso penalty, achieves higher accuracy. (2) The results in Figure 6b indicate that ignoring the spatial effect of the error term introduces significant disturbance to variable selection. The correctness of variable selection for all methods drops below half, and even with an increase in sample size, the improvement in variable selection is limited. However, through comparison, it is found that the exponential square loss exhibits the best robustness. (3) Combining with

Table A5. Variable selection with regularizer on normal data ($q=5$).

	n = 30, q = 5						n = 300, q = 5
	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$
$\rho =0.8,\omega =0.6$
Correct	1.80	2.17	2.20	2.10	1.57	1.77	5.00	5.00	4.90	4.87	4.87	4.57
Incorrect	0.00	0.00	0.00	0.00	0.03	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	1.50	1.48	1.01	0.85	1.41	1.43	0.23	0.22	0.27	0.25	0.30	0.33
$\rho =0.5,\omega =0.6$
Correct	4.17	4.87	2.90	2.40	2.57	2.70	5.00	5.00	5.00	4.90	4.80	4.83
Incorrect	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	0.38	0.25	0.60	0.71	0.82	0.74	0.13	0.11	0.24	0.22	0.24	0.28
$\rho =0,\omega =0.6$
Correct	4.03	5.00	3.10	2.37	1.83	2.17	5.00	5.00	5.00	4.90	4.80	4.83
Incorrect	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	0.44	0.16	0.60	0.77	1.00	0.98	0.13	0.11	0.23	0.20	0.28	0.26
$\rho =0.8,\omega =0$
Correct	1.57	3.03	0.60	0.40	0.40	0.67	3.07	3.00	0.47	0.33	0.47	0.47
Incorrect	0.67	1.00	0.10	0.13	0.07	0.27	1.73	1.67	0.07	0.13	0.27	0.27
MedSE	3.39	3.72	3.65	3.92	6.13	5.26	3.89	3.90	5.36	5.92	5.54	6.81
$\rho =0.5,\omega =0$
Correct	0.90	3.00	0.50	0.20	0.37	0.77	2.10	2.00	0.43	0.20	0.57	0.67
Incorrect	0.00	0.00	0.10	0.10	0.10	0.17	1.07	1.00	0.03	0.00	0.17	0.27
MedSE	2.44	2.71	3.87	4.13	4.31	4.51	4.19	4.34	5.77	6.32	5.44	6.21
$\rho =0,\omega =0$
Correct	1.00	2.97	0.50	0.33	0.63	0.87	1.97	2.00	0.40	0.33	0.30	0.40
Incorrect	0.00	0.00	0.13	0.17	0.10	0.17	0.97	1.00	0.10	0.03	0.13	0.17
MedSE	2.63	3.21	4.36	4.50	4.36	4.67	4.65	4.94	6.77	7.52	7.04	7.67

, it is observed that among the three loss functions, the SEM model with the exponential loss function has lower MedSE compared to the other two loss functions.

Based on the variable selection structure when the number of non-significant covariates approaches the sample size, we plotted Figure 7. Due to the significant differences in variable selection results for different values of q (30 and 300), it is challenging to display them well in the graph. Therefore, we compared the variable selection differences for each method with or without considering the spatial effect of the error term. From Figure 7, it is evident that not considering the spatial effect of the error term significantly reduces the correct selection of variables.

Table A6. Variable selection with regularizer on high-dimensional data ($q>5$).

	n = 30, q = 20						n = 300, q = 200
	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$
$\rho =0.8,\omega =0.6$
Correct	3.4	4.9	1.2	2.7	4.5	5.8	94.5	135.0	89.1	89.2	96.6	103.0
Incorrect	0.2	0.5	0.0	0.1	0.0	0.1	0.0	0.0	0.0	0.0	0.0	0.0
MedSE	7.4	7.2	9.0	7.0	3.4	3.8	4.3	2.8	4.9	5.1	4.2	4.3
$\rho =0.5,\omega =0.6$
Correct	5.2	8.3	4.8	5.3	10.3	11.6	178.0	198.0	170.0	166.0	188.0	191.0
Incorrect	0.2	0.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
MedSE	4.8	2.2	3.1	2.5	1.5	1.4	1.7	0.9	2.0	2.1	1.5	1.4
$\rho =0,\omega =0.6$
Correct	4.8	6.1	4.2	4.3	6.9	8.2	180.2	200.0	165.3	158.4	153.8	156.6
Incorrect	0.1	0.1	0.1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
MedSE	5.6	4.3	3.6	3.5	2.1	2.1	1.8	0.9	2.1	2.2	2.4	2.3
$\rho =0.8,\omega =0$
Correct	5.5	8.0	0.7	0.6	1.1	2.4	20.0	20.0	7.7	6.6	4.1	10.1
Incorrect	1.1	1.6	0.0	0.0	0.1	0.2	3.0	3.0	0.1	0.0	0.1	0.0
MedSE	18.9	20.4	23.7	25.9	18.4	13.7	4.1	4.1	94.6	80.2	145.0	155.2
$\rho =0.5,\omega =0$
Correct	3.6	4.7	0.8	0.9	2.1	3.4	19.6	20.0	9.8	10.5	9.6	13.7
Incorrect	0.2	0.8	0.1	0.0	0.1	0.2	2.9	3.0	0.1	0.1	0.0	0.1
MedSE	11.3	12.3	14.8	18.9	8.3	7.8	4.1	4.1	6.3	5.3	6.2	5.6
$\rho =0,\omega =0$
Correct	3.8	4.8	0.9	1.5	2.6	3.8	19.6	20.0	9.2	8.7	12.6	14.8
Incorrect	0.1	0.5	0.3	0.1	0.1	0.2	2.9	3.0	0.1	0.0	0.1	0.1
MedSE	8.6	8.9	14.8	18.9	6.4	6.8	4.1	4.1	6.4	5.9	4.3	3.6

presents the detailed results. Comparing with the results in Table A5, it is observed that when the sample size is small, using both lasso and adaptive lasso penalties with all three loss functions may lead to a slightly higher number of incorrect identifications of zero coefficients. However, as the sample size increases, the error rate decreases to zero. Importantly, the combination of the exponential square loss function with lasso and adaptive lasso penalties performs better in variable selection, ensuring smaller MedSE even when identifying more zero coefficients correctly. Similarly, ignoring the spatial effect of the error term leads to an increase in the error rate in variable selection for the SEM model, accompanied by an increase in MedSE.

4.6. Variable Selection with Regularizer on Outlier Data

In order to compare the estimation and variable selection performance of various variable selection methods when data contain outliers, in this section, we conducted numerical simulations with response variable observations containing outliers and an inaccurate spatial weight matrix.

Table 1. Variable selection with regularizer when the observed values of y have outliers.

	n = 30, q = 5						n = 300, q = 5
$\delta =0.01$	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$
$\rho =0.8,\omega =0$
Correct	1.80	2.03	1.70	2.37	1.43	2.13	5.00	5.00	4.90	4.93	4.53	4.67
Incorrect	0.00	0.00	0.00	0.00	0.03	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	1.39	1.47	0.91	0.93	1.59	1.08	0.22	0.21	0.27	0.31	0.35	0.34
$\rho =0.5,\omega =0$
Correct	0.33	0.23	0.50	0.53	0.43	0.57	3.27	4.07	0.67	0.47	0.37	0.83
Incorrect	0.00	0.20	0.13	0.07	0.07	0.23	0.10	0.10	0.17	0.13	0.03	0.13
MedSE	4.85	5.50	4.56	4.51	7.06	5.11	1.40	1.32	6.35	6.80	9.09	5.16
$\rho =0.8,\omega =0.6$
Correct	3.67	4.03	2.40	2.53	2.77	2.93	4.97	5.00	4.97	4.93	4.63	4.60
Incorrect	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	0.44	0.33	0.76	0.73	0.83	0.82	0.16	0.14	0.20	0.23	0.30	0.28

presents the variable selection results of the SEM model when response variable observations contain outliers. We included data for sample sizes of 30 and 300, with $\delta =0.01$. It can be observed that in almost all test cases, the SEM model with the exponential square loss and lasso or adaptive lasso penalties identifies more true zero coefficients and has a lower MedSE. Compared to the variable selection results for normal data in Table A5, the superiority of the exponential square loss with lasso and adaptive lasso penalties is more pronounced in the presence of outliers. It achieves higher accuracy in identifying zero coefficients and has lower MedSE. This indicates that the exponential square loss exhibits stronger robustness when outliers are present in Y. By observing the results for $\omega =0$ and $\omega =0.6$, we can see that, similar to the results in Table A5 and Table A6, ignoring the spatial dependency in the error term often leads to a reduction in the number of correctly identified zero coefficients in the SEM model, decreased accuracy in identification, and an increase in MedSE.

Table 2. Variable selection with regularizer with noisy W.

	n = 30, q = 5						n = 300, q = 5
	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$	Exp + l1	Exp + ${\tilde{l}}_1$	S + l1	S + ${\tilde{l}}_1$	Lad + l1	Lad + ${\tilde{l}}_1$
Remove 30%
$\omega =0.6$
Correct	1.50	2.77	1.63	1.63	1.77	1.87	4.93	5.00	4.60	4.20	3.87	4.20
Incorrect	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	1.39	1.27	1.20	1.07	1.17	1.19	0.27	0.24	0.29	0.37	0.43	0.40
$\omega =0$
Correct	0.40	2.20	0.77	0.53	0.47	0.67	1.13	3.00	0.43	0.53	0.30	0.40
Incorrect	0.00	0.00	0.03	0.17	0.07	0.17	0.10	0.77	0.07	0.07	0.27	0.13
MedSE	2.48	2.62	6.32	5.06	5.23	4.81	3.40	3.53	6.89	4.11	8.48	6.47
Add 50%
$\omega =0.6$
Correct	0.40	0.73	1.37	1.20	1.03	1.87	4.63	5.00	3.47	3.17	3.20	3.53
Incorrect	0.00	0.03	0.00	0.03	0.00	0.03	0.00	0.00	0.00	0.00	0.00	0.00
MedSE	2.72	2.59	1.76	1.61	1.58	1.39	0.52	0.49	0.68	0.64	0.71	0.66

presents the variable selection results in the case of inaccurate W. Because in practical applications it can be challenging to estimate the spatial weight matrix, this simulation provides valuable insights. In all test scenarios, the SEM model with the exponential square loss and lasso or adaptive lasso penalties achieves higher accuracy in identifying zero coefficients. In cases where $30\%$ of the non-zero values in W are removed and $50\%$ of non-zero values are added to W, the superiority of the proposed method becomes more apparent. It can correctly identify more zero coefficients while keeping MedSE smaller. In the presence of an inaccurate weight matrix, the impact of the spatial effects on the variable selection is similar to the results in Table A5 and Table A6, as well as Table 1. When $\omega =0$, the SEM model incorrectly identifies a larger number of zero coefficients, but when $\omega =0.6$, the number of incorrectly identified zero coefficients is reduced to zero. These results indicate that the proposed SEM variable selection method with the exponential square loss and lasso or adaptive lasso penalties is robust and suitable for variable selection in spatially dependent data, even when the weight matrix W cannot be accurately estimated.

5. Empirical Data Verification

In this section, we apply the SEM variable selection method proposed in this paper to empirical dataset to verify the performance of parameter estimation and variable selection. The purpose of this experiment is to validate whether the proposed method can correctly select covariates that are useful for the response variable, according to Burnham and Andersen (2002) [19]. We choose BIC (Bayesian Information Criterion) as the criterion to assess the performance of the method. This paper uses the Boston Housing dataset, which was initially created by Harrison and Rubinfeld (1978 [20]). The data come from the real estate market in the Boston area in the 1970s and include information about housing prices in Massachusetts, USA. This dataset can be obtained from the “spdeep” library in R.

The Boston Housing dataset consists of 506 samples, each with 13 features and one response variable, making a total of 14 columns,

Table 3. Variable description.

Variable	Description
DIS	Weighted distance to five job centers in Boston
RAD	Highway convenience index
LSTAT	Proportion of lower income groups
RM	Average number of rooms
NOX	Nitric oxide concentration
CRIM	Per capita crime rate
ZN	The proportion of residential land
INDUS	The proportion of non-commercial land
CHAS	1 if it is a river Otherwise 0
AGE	Proportion of owner-occupied units built before 1940
GAR	number of car spaces in garage (0 = no garage)
AGE	age of dwelling in years
TAX	Full property tax rate per 10,000 dollars
PTRATIO	Ratio of teachers to students in town
B-1000	Percentage of blacks in the town
MEDV	Median owner-occupied home price

describes the meanings of various features in this dataset. The main objective is to study the relationship between housing prices and other variables and select the most important ones. The response variable is the logarithm of MEDV, and the predictor variables include the logarithm of DIS, RAD, LSTAT, as well as the square of RM and NOX. Other variables considered are CRIM, ZN, INDUS, CHAS, AGE, TAX, PTRATIO, and B-1000.

The spatial weight matrix W can be calculated based on the Euclidean distance using latitude and longitude coordinates. We set a distance threshold $d_0$, and when the distance $d_{ij}$ between two regions is greater than $d_0$, $w_{ij}=0$. $w_{ij}$ can be represented as:

$$\begin{array}{c}\hfill d_{ij}=\sqrt{{(LAT{T}_i-LAT{T}_j)}^2+{(LON{G}_i-LON{G}_j)}^2},\end{array}$$

(24)

$$\begin{array}{c}\hfill D_{ij}=max(1-\frac{d_{ij}}{d_0},0),i\ne j,\end{array}$$

(25)

$$\begin{array}{c}\hfill w_{ij}=\left\{\begin{array}{cc}0& i=j\\ {\displaystyle D_{ij}/(\sum _{j=1,i\ne j}^n{D}_{ij})}& i\ne j\end{array}\right.\end{array}$$

(26)

Then, normalize each row of the spatial weight matrix W and incorporate it into the SEM model.

Table 4. Variable selection for the Boston dataset.

	EXP			Square			LAD
	E + l1	E + $\widetilde{l_1}$	E + null	S + l1	S + $\widetilde{l_1}$	S + null	L + l1	L + $\widetilde{l_1}$	L + null
CRIM	−0.0066	−7.8600 ×${10}^{-4}$	−0.0066	−0.0059	−0.0059	−0.0070	−0.0035	−0.0034	−0.0063
ZN	3.1200 × ${10}^{-4}$	0	3.3400 × ${10}^{-4}$	−4.3300 ×${10}^{-4}$	−4.0800 ×${10}^{-4}$	3.8900 × ${10}^{-4}$	−8.0800 ×${10}^{-4}$	−8.7400 ×${10}^{-4}$	1.5200 × ${10}^{-4}$
INDUS	0.0015	0	0.0015	0.0017	0.0017	0.0013	0.0010	0.0010	0.0018
CHAS	0	0	−0.0017	0.0025	0.0025	0.0048	1.0700 × ${10}^{-4}$	1.1700 × ${10}^{-4}$	−0.0182
NOX	−0.1908	0	−0.1745	−2.5800 ×${10}^{-4}$	−2.9800 ×${10}^{-4}$	−0.2578	4.5000 × ${10}^{-5}$	5.2000 × ${10}^{-5}$	−0.1514
RM	0.0079	0.0076	0.0079	0.0146	0.0145	0.0067	0.0200	0.0199	0.0123
AGE	−4.5400 ×${10}^{-4}$	0	−4.3100 ×${10}^{-4}$	−0.0012	−0.0012	−2.8700 ×${10}^{-4}$	−0.0014	−0.0014	−0.0012
DIS	−0.1411	−0.0105	−0.1396	−0.0107	−0.0107	−0.1571	−5.8600 ×${10}^{-4}$	−6.3800 ×${10}^{-4}$	−0.1419
RAD	0.0635	0	0.0639	0.0103	0.0103	0.0701	3.6200 × ${10}^{-4}$	3.9800 × ${10}^{-4}$	0.0484
TAX	−3.5600 ×${10}^{-4}$	−1.1700 ×${10}^{-4}$	−3.6200 ×${10}^{-4}$	−3.8000 ×${10}^{-5}$	−4.1000 ×${10}^{-5}$	−3.6400 ×${10}^{-4}$	−1.0600 ×${10}^{-4}$	−1.3500 ×${10}^{-4}$	−2.8700 ×${10}^{-4}$
PTRATIO	−0.0108	0	−0.0106	−0.0171	−0.0166	−0.0115	−0.0032	−0.0032	−0.0081
B-1000	3.2800 × ${10}^{-4}$	2.1900 × ${10}^{-4}$	3.2900 × ${10}^{-4}$	3.9800 × ${10}^{-4}$	4.0200 × ${10}^{-4}$	2.8200 × ${10}^{-4}$	6.0200 × ${10}^{-4}$	5.6500 × ${10}^{-4}$	4.3600 × ${10}^{-4}$
LSTAT	−0.2090	−0.1818	−0.2110	−0.0249	−0.0249	−0.2279	−0.0014	−0.0014	−0.1503
$\widehat{\rho }$	0.5173	0.6172	0.5174	0.6177	0.6190	0.5190	0.5164	0.5139	0.5000
$\widehat{{\sigma }^2}$	0.0192	0.0223	0.0192	0.0247	0.0247	0.0192	0.0294	0.0294	0.0206
BIC	−567.8321	−491.5053	−567.6874	−446.1253	−446.9024	−564.2289	−355.5156	−356.8915	−540.9918

presents the estimates and variable selection results for the SEM with three loss functions, lasso or adaptive lasso penalties, and no penalty. The results indicate that the SEM with all three loss functions provide estimates for $\rho$ and ${\sigma }^2$ around 0.5 and 0.2, respectively. For variable selection, we consider a variable important when the absolute value of its coefficient estimate is greater than 0.05, while a variable is deemed unimportant when the absolute value of its coefficient estimate is less than 0.005. It can be observed that, in the absence of regularization terms, all three loss functions estimate the coefficients of NOX, DIS, RAD, and LSTAT to be greater than 0.05, classifying them as important variables. Moreover, all models suggest a negative correlation between NOX, DIS, LSTAT, and MEDV, indicating that higher nitrogen oxide concentrations, greater distances to employment centers, and a higher percentage of lower-income population are associated with lower housing prices. On the other hand, RAD is considered positively correlated with MEDV, implying that a higher highway accessibility index leads to higher housing prices in the area. However, INDUS, CHAS, and AGE are deemed unimportant variables, as their coefficient estimates have absolute values less than 0.005. ZN, TAX, and B-1000 have coefficient estimates close to zero. These findings suggest that the proposed method with the exponential square loss is effective in variable selection.

To visually compare the variable selection performance with penalty terms for the three loss functions, we marked the variable selection results of the SEM model with “ + ” and “−” symbols. We set a threshold of 0.005, and variables with coefficients greater than 0.005 were marked with “ + ”, while those with coefficients less than −0.005 were marked with “−”. As shown in

Table 5. Variable selection for the Boston dataset after marking.

	EXP			Square			LAD
	E + l1	E + $\widetilde{l_1}$	E + null	S + l1	S + $\widetilde{l_1}$	S + null	L + l1	L + $\widetilde{l_1}$	L + null
CRIM	−	−	−	−	−	−	−	−	−
ZN							−	−
INDUS	+		+	+	+	+	+	+	+
CHAS			−	+	+	+			−
NOX	−		−			−			−
RM	+	+	+	+	+	+	+	+	+
AGE				−	−		−	−	−
DIS	−	−	−	−	−	−		−	−
RAD	+		+	+	+	+			+
TAX
PTRATIO	−		−	−	−	−	−	−	−
B-1000							+
LSTAT	−	−	−	−	−	−	−	−	−
count +	3	1	3	4	4	4	3	2	3
count−	5	3	6	5	5	5	5	6	7
count	8	4	9	9	9	9	8	8	10
BIC	−567.83205	−491.50529	−567.68740	−446.12530	−446.90237	−564.22889	−355.5156	−356.89146	−540.99178

, the SEM models with the exponential square loss and adaptive lasso selected fewer variables. Furthermore, compared to the other two loss functions (with and without lasso penalties), the SEM model using adaptive lasso and exponential square loss has the lowest BIC index. This clearly demonstrates the superiority of the variable selection method proposed in this paper.

6. Conclusions and Discussions

The paper proposes a robust variable selection method for the spatial autoregressive model with exponential squared loss and the adaptive lasso penalty. This method can simultaneously select variables and estimate unknown coefficients. The main conclusions of this study are as follows:

• The penalized exponential squared loss effectively selects non-zero coefficients of covariates. When there is noise in the observations and an inaccurate spatial weight matrix, the proposed method shows significant resistance to their impact, demonstrating good robustness.
• The block coordinate descent (BCD) algorithm proposed in this work is effective in optimizing the penalized exponential squared loss function.
• In numerical simulation experiments and empirical applications, variable selection results with lasso and adaptive lasso penalties were compared, and adaptive lasso consistently outperformed in various scenarios.
• It is noteworthy that ignoring the spatial effects of error terms (i.e., $\omega =0$) severely reduces the accuracy of this variable selection method. However, for general error models (when $\rho =0$, $\omega \ne 0$), the proposed method remains applicable.

In the field of spatial econometrics, there are many other spatial regression models, as well as numerous robust loss functions and penalties. Building on the foundation of this study, we plan to explore more related issues in the future.