Change Point Detection Using Penalized Multidegree Splines

Abstract

We consider a function estimation method with change point detection using truncated power spline basis and elastic-net-type $L^1$-norm penalty. The $L^1$-norm penalty controls the jump detection and smoothness depending on the value of the parameter. In terms of the proposed estimators, we introduce two computational algorithms for the Lagrangian dual problem (coordinate descent algorithm) and constrained convex optimization problem (an algorithm based on quadratic programming). Subsequently, we investigate the relationship between the two algorithms and compare them. Using both simulation and real data analysis, numerical studies are conducted to validate the performance of the proposed method.

1. Introduction

A nonparametric function estimation is useful for estimating the actual relationship of data exhibiting nonlinear relationships. It is a statistical method for estimating the relationship between variables based on observed data, assuming that the actual function representing the relationships between input variables and response variables belongs to an infinite-dimensional parametric space. Representative examples, including kernel density estimation (chapter 1 of [1]), local polynomials [2], splines (pp. 118–180 of [3]) and curve estimation (chapter 2 of [4]), have been investigated previously.

A representative method used for function estimation is the basis function method. Among the basis function methods, splines are the most frequently used. In function estimation, an infinite number of parameters cannot be estimated using a finite amount of data. Therefore, function estimation is performed by introducing a function space and a basis function. If the function space and basis function are defined, then the shape of the function to be estimated using the basis function as a predictor variable can be expressed as a linear combination of the basis functions spanning the appropriate function space. Therefore, the function estimation problem can be regarded as estimating the regression coefficients.

Among the many basis functions used to interpolate data or fit smooth curves, the spline basis is the most typically used. The spline basis function is defined as a piecewise polynomial that is differentiable for every knot spacing. Its main underlying technology is truncated power splines (see chapter 3 of [5]). Additionally, it offers the advantages of simple construction and easy interpretation of the model parameters. However, in model with many overlapping intervals, basis functions are highly correlated. Therefore, several computational problems may occur when the number of knots increases significantly because the predictor matrix for the truncated power spline basis becomes dense. Consequently, incorrect fitting may occur (chapter 7 of [6]).

In the basis function methodology, the objective function to be optimized is a convex function. The coordinate descent algorithm (CDA) [7] is simple, efficient and useful for optimizing these objective functions. The concept of the algorithm is to optimize the solution of a problem involving the minimization (maximization) of a convex (concave) function for a multidimensional vector. Coefficient updates keep the remaining coefficients constant. The objective function is regarded as a one-dimensional function. Tseng [8] proved that the coordinate descent converges when a convex penalty term is non-differentiable but separable in each coordinate. This result implies that the coordinate-wise algorithms for the least absolute shrinkage and selection operator (lasso) [9], group lasso [10] and elastic net [11] converge to their optimal solutions. Another simpler method to obtain the optimal solution is to use quadratic programming (QP) [12]. QP is a type of nonlinear programming that obtains a mathematical optimization problem involving quadratic functions.

The selection of knots and the detection of change points in regression splines significantly affect the performance of the model. Knot selection and change point detection methods have been studied extensively. Osbone et al. [13] proposed an algorithm that allows an efficient computation of the lasso estimator for knot selection to be performed. Leitenstorfer and Tutz [14] considered the boosting technique to select variables in knot selection. Garton [15] proposed a method for selecting the number and position of knots using Gaussian and non-Gaussian data. Aminikhanhahi and Cook [16] proposed a change point detection method for time-series data. Meanwhile, Tibshirani et al. [17] proposed sparsity and smoothness using a fused lasso.

The main contributions of this study are as follows.

First, a new type of function estimator is established for change points in the nonparametric regression function estimation model. The proposed function estimator is defined by a linear combination of multidegree-based splines to simultaneously provide a change point detection and smoothing.

Second, two computational algorithms are introduced to address the constrained convex optimization problem (algorithm based on QP) and Lagrangian dual problem (CDA); moreover, the relationship between them is investigated. Numerical studies using both simulated and real datasets are provided to demonstrate the performance of the proposed method.

Herein, we present a new statistical learning methodology for modeling and analyzing data to address the change point detection problem. The proposed method involves two tuning parameters that control the penalty terms. We express the estimator as a linear combination of a polynomial and a truncated power-spline basis for zero and positive integer degrees, respectively. The coefficients are estimated using a CDA to minimize the residual sum of squares and QP. We consider three sets of simulation data to measure the performance of the proposed method and conduct an analysis on real data with changes or jump point(s).

This paper is organized as follows: In Section 2, we present a nonparametric regression model containing observed data and the penalized regression spline estimator. The process of updating the coefficients based on the CDA and the estimated coefficients obtained using QP is presented in Section 3. Additionally, the estimators obtained using the two methods are compared and their characteristics are introduced briefly. In Section 4, using the simulation and real data, we validate the performance of our proposed model. In Section 5, we summarize the conclusions of this study. All codes implemented using the software program R for numerical analysis are available as Supplementary Materials.

2. Model and Estimator

Consider a nonparametric regression model

$$y_i=f\left(x_i\right)+{\epsilon }_i\mathrm{for}i=1,\dots ,n,$$

(1)

where $\left\{x_i\right\}$’s $\in \mathcal{I}\subset \mathbb{R}$ are the predictors and $\left\{y_i\right\}$’s $\in \mathbb{R}$ are responses. The model is nonparametric because it does not contain any parametric assumptions regarding f. Model (1) is expressed similarly as the conditional expectation $E\left(Y\right|X=x)$. In this case, ${\epsilon }_1,\dots ,{\epsilon }_n$ are only required to satisfy the property that the expected value is zero with positive variance ${\sigma }^2$ and no distribution assumption is required. Without loss of generality, we write $\mathcal{I}=[0,1]$ hereinafter for notational simplicity.

Because it is impossible to estimate an infinite number of parameters using a finite amount of data, a specific function space for estimation should be considered. In nonparametric estimation, it is typically assumed that f belongs to a massive class of functions [1]. In this study, because the goal is to achieve a smooth function estimation that can detect change point(s), the function space to which f belongs can be specified as follows.

Let ${\mathcal{F}}_K$ be the space of spline functions of mixed degrees 0 and positive integer $d>0$ with K interior knots $0=t_0<t_1<\cdots <t_K<t_{K+1}=1$. Therefore, any function $\mathsf{f}\in {\mathcal{F}}_K$ can be expressed as

$$\mathsf{f}(x;\theta )={\alpha }_0+\sum _{j=1}^d{\alpha }_j{x}^j+\sum _{k=1}^K\left\{{\beta }_k{P}_{0,k}\left(x\right)+{\gamma }_k{P}_{d,k}\left(x\right)\right\},$$

where $\theta =(\alpha ,\beta ,\gamma )$ with $\alpha =({\alpha }_0,{\alpha }_1,\dots ,{\alpha }_d)$, $\beta =({\beta }_1,\dots ,{\beta }_K)$, $\gamma =({\gamma }_1,\dots ,{\gamma }_K)$,

$$P_{0,k}\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x>t_k\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\mathrm{and}{P}_{d,k}\left(x\right)={(x-t_k)}_{+}^d\mathrm{for}k=1,\dots ,K.$$

It is noteworthy that ${\left(a\right)}_{+}$ is a function that assumes a real value of a if $a>0$ and 0 otherwise. We first select the number of the initial knots K for the splines and then place them at the same interval within $(0,1)$. Generally, one may select a sufficiently large value of K—n, or $n/2$, for example—to select data-adaptive knots.

The empirical risk of $\theta$ is defined as

$$R\left(\theta \right)=\frac{1}{2n}\sum _{i=1}^n{\left(y_i-\mathsf{f}(x_i;\theta )\right)}^2.$$

The penalized objective function to be minimized is expressed as

$$R^{\lambda ,\tau }\left(\theta \right)=R\left(\theta \right)+\lambda \left(\tau {\parallel \beta \parallel }_1+(1-\tau ){\parallel \gamma \parallel }_1\right)$$

(2)

for $\lambda >0$, $\tau \in [0,1]$ and ${\parallel ·\parallel }_1$, the $L^1$-norm. The penalty term is the concept of elastic-net penalty [11]. Similar to the lasso, $\lambda$ reduces only $\beta$ and $\gamma$. Furthermore, similar to an elastic net, $\tau$ is a tuning parameter that controls the relative penalty. If $\tau$ is of a higher value, then a greater weight is assigned to the $L^1$-norm of $\beta$, where the jump detection penalty is. By contrast, if $\tau$ is of a lower value, then a greater weight is assigned to the $L^1$-norm of $\gamma$, which is a penalty indicating the function smoothness.

Let

$${\widehat{\theta }}^{\lambda ,\tau }=\underset{\theta }{argmin}{R}^{\lambda ,\tau }\left(\theta \right)$$

and define the multidegree spline estimator (MDSE) as

$$\widehat{f}=\mathsf{f}\left(·;\widehat{\theta }\right).$$

3. Implementation

Specifically in mathematical statistics fields, penalized regression problems can seek to optimize a multivariate quadratic function subject to linear constraints on the regression coefficients. To realize the proposed estimator, two computational methods are considered, i.e., the CDA and the algorithm for QP, using the quadprog package [18] in the R program. We describe the abovementioned two methods comprehensively and present their advantages and disadvantages. Both methods are implemented using the software program R.

3.1. CDA

The CDA optimizes the objective function with respect to a single coefficient at a time, iteratively cycling through all coefficients until convergence is reached. We adopt the CDA introduced by Jhong et al. (2017) to perform univariate smoothing for each coefficient. Because the objective function (2) is convex for $\theta$, one can adopt the CDA to obtain the MDSE. This algorithm is inspired by the studies of Friedman et al. [19] and Jhong et al. [20]. However, it is a new attempt, in that the proposed algorithm is applied to the elastic net penalty considering two hyper parameters, $\lambda$ and $\tau$.

For $j=0,1,2,\dots ,d$ and $k=1,2,\dots ,K$, let ${\tilde{\alpha }}_j$, ${\tilde{\beta }}_k$ and ${\tilde{\gamma }}_k$ denote the initial (or updated) values. Specifically,

$$\tilde{\alpha }=\left({\tilde{\alpha }}_0,{\tilde{\alpha }}_1,\dots ,{\tilde{\alpha }}_d\right)\in {\mathbb{R}}^{1+d},\hspace{1em}\tilde{\beta }=\left({\tilde{\beta }}_1,\dots ,{\tilde{\beta }}_K\right)\in {\mathbb{R}}^K,$$

and $\tilde{\gamma }=\left({\tilde{\gamma }}_1,\dots ,{\tilde{\gamma }}_K\right)\in {\mathbb{R}}^K$. For notational convenience, we use

$$\tilde{\theta }=(\tilde{\alpha },\tilde{\beta },\tilde{\gamma })=({\tilde{\theta }}_1,\dots ,{\tilde{\theta }}_J)\mathrm{for}J=1+d+2K.$$

Therefore, the coordinate descent update is written in the following forms:

$${\tilde{\theta }}_j\leftarrow \underset{{\theta }_j\in \mathbb{R}}{argmin}{R}^{\lambda ,\tau }\left({\tilde{\theta }}_1,\dots ,{\tilde{\theta }}_{j-1},{\theta }_j,{\tilde{\theta }}_{j+1},\dots ,{\tilde{\theta }}_J\right)\mathrm{for}j=1,\dots ,J.$$

3.1.1. Updating $\alpha$

The optimization problem for ${\alpha }_j$, $j=0,1,\dots ,d$ has a quadratic form. Hence, we can solve this problem by transforming the objective function for ${\alpha }_j$ into a perfect square form. It is noteworthy that $x_i^0=1$.

$$\begin{array}{cc}\hfill & R^{\lambda ,\tau }\left({\tilde{\alpha }}_0,\dots ,{\tilde{\alpha }}_{j-1},{\alpha }_j,{\tilde{\alpha }}_{j+1},\dots ,{\tilde{\alpha }}_d,\tilde{\beta },\tilde{\gamma }\right)\hfill \\ \hfill & =\frac{1}{2n}\sum _{i=1}^n{\left\{y_i-\sum _{\ell \ne j}{\tilde{\alpha }}_{\ell }{x}_i^{\ell }-{\alpha }_j{x}_i^j-\sum _{k=1}^K\left({\tilde{\beta }}_k{P}_{0,k}\left(x_i\right)+{\tilde{\gamma }}_k{P}_{d,k}\left(x_i\right)\right)\right\}}^2\hfill \\ \hfill & +\lambda \tau \sum _{k=1}^K\left|{\tilde{\beta }}_k\right|+\lambda \left(1-\tau \right)\sum _{k=1}^K|{\tilde{\gamma }}_k|\hfill \\ \hfill & =\frac{1}{2n}\sum _{i=1}^n{\left(r_{i,\alpha }^{\left(j\right)}-{\alpha }_j{x}_i^j\right)}^2+\left(\mathrm{terms} \mathrm{independent} \mathrm{for} {\alpha }_j\right)\hfill \\ \hfill & =\frac{{\sum }_{i=1}^n{\left(x_i^j\right)}^2}{2n}{\left\{{\alpha }_j-\frac{{\sum }_{i=1}^n{r}_{i,\alpha }^{\left(j\right)}{x}_i^j}{{\sum }_{i=1}^n{\left(x_i^j\right)}^2}\right\}}^2+\left(\mathrm{terms} \mathrm{independent} \mathrm{for} {\alpha }_j\right),\hfill \end{array}$$

where

$${\displaystyle r_{i,\alpha }^{\left(j\right)}=y_i-\sum _{\ell \ne j}{\tilde{\alpha }}_{\ell }{x}_i^{\ell }-\sum _{k=1}^K{\tilde{\beta }}_k{P}_{0,k}\left(x_i\right)-\sum _{k=1}^K{\tilde{\gamma }}_k{P}_{d,k}\left(x_i\right)}$$

is a partial residual. Because ${\alpha }_j$ is not subject to penalization, one can update each coefficient systematically using the partial residual. Therefore, we update

$${\displaystyle {\tilde{\alpha }}_j\leftarrow \frac{{\sum }_{i=1}^n{r}_{i,\alpha }^{\left(j\right)}{x}_i^j}{{\sum }_{i=1}^n{\left(x_i^j\right)}^2}\mathrm{for}j=0,1,\dots ,d.}$$

(3)

3.1.2. Updating $\beta$ and $\gamma$

Unlike updating $\alpha$, one cannot use only the partial residual to update $\beta$ and $\gamma$, because the objective function contains the penalty terms. Hence, we used the soft-thresholding operator [19] to solve the lasso solutions. The soft-thresholding operator is defined as

$$S\left(y,\lambda \right)=\left\{\begin{array}{cc}y-\lambda \hfill & \mathrm{if}y>\lambda \hfill \\ y+\lambda \hfill & \mathrm{if}y<-\lambda \hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\mathrm{for}y\in \mathbb{R}\mathrm{and}\lambda >0.$$

We select $k\in \{1,\dots ,K\}$ and transform the objective function for ${\beta }_k$ into a quadratic form. Subsequently,

$$\begin{array}{cc}\hfill & R^{\lambda ,\tau }\left(\tilde{\alpha },{\tilde{\beta }}_1,\dots ,{\tilde{\beta }}_{k-1},{\beta }_k,{\tilde{\beta }}_{k+1},\dots ,{\tilde{\beta }}_K,\tilde{\gamma }\right)\hfill \\ \hfill & =\frac{1}{2n}\sum _{i=1}^n{\left\{y_i-\sum _{j=0}^d{\tilde{\alpha }}_j{x}_i^j-\sum _{\ell \ne k}{\tilde{\beta }}_{\ell }{P}_{0,\ell }\left(x_i\right)-{\beta }_k{P}_{0,k}\left(x_i\right)-\sum _{\ell =1}^K{\tilde{\gamma }}_{\ell }{P}_{d,\ell }\left(x_i\right)\right\}}^2\hfill \\ \hfill & +\lambda \tau \sum _{\ell \ne k}|{\tilde{\beta }}_{\ell }|+\lambda \tau |{\beta }_k|+\lambda \left(1-\tau \right)\sum _{\ell =1}^K|{\tilde{\gamma }}_{\ell }|\hfill \\ \hfill & =\frac{1}{2n}\sum _{i=1}^n{\left(r_{i,\beta }^{\left(k\right)}-{\beta }_k{P}_{0,k}\left(x_i\right)\right)}^2+\lambda \tau \left|{\beta }_k\right|+\left(\mathrm{terms} \mathrm{independent} \mathrm{for} {\beta }_k\right),\hfill \end{array}$$

where

$${\displaystyle r_{i,\beta }^{\left(k\right)}=y_i-\sum _{j=0}^d{\tilde{\alpha }}_j{x}_i^j-\sum _{\ell \ne k}\widetilde{{\beta }_{\ell }}{P}_{0,\ell }\left(x_i\right)-\sum _{\ell =1}^K{\tilde{\gamma }}_{\ell }{P}_{d,\ell }\left(x_i\right).}$$

Finally, using the soft-thresholding operator, we can update ${\tilde{\beta }}_k$ as follows:

$${\displaystyle {\tilde{\beta }}_k\leftarrow \frac{S\left(\frac{1}{n}{\sum }_{i=1}^n{r}_{i,\beta }^{\left(k\right)}{P}_{0,k}\left(x_i\right),\lambda \tau \right)}{\frac{1}{n}{\sum }_{i=1}^n{\left[P_{0,k}\left(x_i\right)\right]}^2}.}$$

(4)

Similarly, ${\tilde{\gamma }}_k$ is expressed as

$${\displaystyle {\tilde{\gamma }}_k\leftarrow \frac{S\left(\frac{1}{n}{\sum }_{i=1}^n{r}_{i,\gamma }^{\left(k\right)}{P}_{d,k}\left(x_i\right),\lambda \left(1-\tau \right)\right)}{\frac{1}{n}{\sum }_{i=1}^n{\left[P_{d,k}\left(x_i\right)\right]}^2}\mathrm{for}k=1,\dots ,K.}$$

(5)

3.1.3. Algorithm Details

Algorithm 1 shows the process of computing the proposed estimator using the CDA. We implement the code using the software program R. For the observed data, we create a basis matrix B to fit to our model.

$$B=\left[\begin{array}{cccccc}\mathbf{1}& X^1& \cdots & X^d& P_0& P_d\end{array}\right]\in {\mathbb{R}}^{n\times \left(1+d+2K\right)},$$

where $\mathbf{1}=\left[\begin{array}{ccc}1& \cdots & 1\end{array}\right]\in {\mathbb{R}}^n$, $X^j=\left[\begin{array}{ccc}{x}_1^j& \cdots & x_n^j\end{array}\right]\in {\mathbb{R}}^n$ for $j=1,\cdots d$.

$P_0=\left[\begin{array}{ccc}{P}_{0,1}& \cdots & P_{0,K}\end{array}\right]\in {\mathbb{R}}^{n\times K}$, and $P_d=\left[\begin{array}{ccc}{P}_{d,1}& \cdots & P_{d,K}\end{array}\right]\in {\mathbb{R}}^{n\times K}$. After B is created, we select the tuning parameters. We compute ${\lambda }_{max}$, which is the smallest value that causes all coefficients to be zero, as follows:

$${\lambda }_{max}=\underset{k}{max}\left(\frac{1}{n}\left|\langle P_{0,k}\left(x\right),y\rangle \right|,\frac{1}{n}\left|\langle P_{d,k}\left(x\right),y\rangle \right|\right),$$

where $\langle x,y\rangle ={\sum }_{i=1}^n{x}_i{y}_i$ for n-dimensional vectors x and y, respectively. Thereafter, we generate candidates for the two tuning parameters $\lambda$ and $\tau$, respectively.

For the case involving $\lambda$, the equally spaced candidates decrease from $log\left({\lambda }_{max}\right)$ to $log({\lambda }_{max}\times {\lambda }_{min\_max\_ratio})$ for a maximum of $n_{\lambda }$ times, which is the number of $\lambda$ candidates. ${\lambda }_{min\_max\_ratio}$ is the ratio of the maximum and minimum values of $\lambda$, which is a sufficiently low positive value. Subsequently, it is reconverted to the exponential scale and the last candidate value is replaced with a zero. Next, in the case of $\tau$, we generate $n_{\tau }$ values from 0 to 1 at equal intervals.

Subsequently, we compute the value of the initial objective function as ${\sum }_{i=1}^n{y}_i^2/2n$. Thereafter, we begin updating all the coefficients until that is achieved. Finally, we select the best model among $n_{\lambda }\times n_{\tau }$ models. In statistical data analysis, some representative information criteria exist, such as the Akaike information criterion (AIC; [21]), the Bayesian information criterion (BIC) and cross validation (CV). In general, the AIC and CV involve the selection of a model with a relatively large variance and small bias compared with the BIC. Because the purpose of this study is to accurately select significant change points, we use the BIC [22], which involves the selection of a model with a relatively small variance, defined as follows:

$$BIC=nlog\left(2\times R^{\lambda ,\tau }(\widehat{\theta })\right)+plog\left(n\right),$$

where p is the number of non-zero coefficients. We select the best model that minimized the BIC. For a detailed calculation of the degrees of freedom p in penalized regression model, see [9].

Algorithm 1: Coordinate descent algorithm (CDA).

3.2. Quadratic Programming

The disadvantage of the CDA is that the computational cost increases with the higher number of iterations. Therefore, we use a new algorithm, i.e., QP, to compute the proposed estimator. In QP, certain mathematical optimization problems involving quadratic functions are solved. In this regard, a multivariate quadratic function subject to linear constraints on the coefficients can be minimized. Hence, because the solution can be obtained without iterations, QP demonstrates computational advantage over the CDA.

The Lagrangian dual problem (2) is converted into the constraint optimization problem, such that

$$\widehat{\theta }=\left(\widehat{\alpha },\widehat{\beta },\widehat{\gamma }\right)=\underset{\theta }{argmin}R\left(\theta \right){\mathrm{subject} \mathrm{to}\tau \left|\right|\beta \left|\right|}_1+\left(1-\tau \right){\left|\right|\gamma \left|\right|}_1\le t,$$

where $t\ge 0$. For $u\in \mathbb{R}$, let

$$u_{+}=\left\{\begin{array}{cc}u\hfill & \mathrm{if}u\ge 0\hfill \\ 0\hfill & \mathrm{if}u<0\hfill \end{array}\right.\mathrm{and}{u}_{-}=\left\{\begin{array}{cc}-u\hfill & \mathrm{if}u\le 0\hfill \\ 0\hfill & \mathrm{if}u>0\hfill \end{array}\right..$$

Therefore, $u=u_{+}-u_{-}$ and $\left|u\right|=u_{+}+u_{-}$. By reparameterizating ${\beta }_k$ and ${\gamma }_k$,

$${\beta }_k={\left({\beta }_k\right)}_{+}-{\left({\beta }_k\right)}_{-},|{\beta }_k|={\left({\beta }_k\right)}_{+}+{\left({\beta }_k\right)}_{-},$$

$${\gamma }_k={\left({\gamma }_k\right)}_{+}-{\left({\gamma }_k\right)}_{-},|{\gamma }_k|={\left({\gamma }_k\right)}_{+}+{\left({\gamma }_k\right)}_{-}$$

Therefore, the optimization problem is equivalent to minimizing

$$\frac{1}{2n}\sum _{i=1}^n{\left(y_i-\sum _{j=0}^d{\alpha }_j{x}_i^j-\sum _{k=1}^K\left[\left\{{\left({\beta }_k\right)}_{+}-{\left({\beta }_k\right)}_{-}\right\}{P}_{0,k}\left(x_i\right)+\left\{{\left({\gamma }_k\right)}_{+}-{\left({\gamma }_k\right)}_{-}\right\}{P}_{d,k}\left(x_i\right)\right]\right)}^2$$

subject to

$$\tau \left\{\sum _{k=1}^K\left({\left({\beta }_k\right)}_{+}+{\left({\beta }_k\right)}_{-}\right)\right\}+\left(1-\tau \right)\left\{\sum _{k=1}^K\left({\left({\gamma }_k\right)}_{+}+{\left({\gamma }_k\right)}_{-}\right)\right\}\le t,$$

in addition to $4\times K$ non-negativity constraints

$${\left({\beta }_k\right)}_{+}\ge 0,{\left({\beta }_k\right)}_{-}\ge 0,{\left({\gamma }_k\right)}_{+}\ge 0,{\left({\gamma }_k\right)}_{-}\ge 0fork=1,\dots ,K.$$

Next, we define

$$Z=\left[\begin{array}{cccccccc}1& X^1& \cdots & X^d& P_0& -P_0& P_d& -P_d\end{array}\right]\in {\mathbb{R}}^{n\times (1+d+4K)}$$

and

$$b=\left[\begin{array}{ccccc}\alpha & {\beta }_{+}& {\beta }_{-}& {\gamma }_{+}& {\gamma }_{-}\end{array}\right]\in {\mathbb{R}}^{1+d+4K},$$

where ${\beta }_{+}=\left\{{\left({\beta }_k\right)}_{+}\right\}$, ${\beta }_{-}=\left\{{\left({\beta }_k\right)}_{-}\right\}$, ${\gamma }_{+}=\left\{{\left({\gamma }_k\right)}_{+}\right\}$, and ${\gamma }_{-}=\left\{{\left({\gamma }_k\right)}_{-}\right\}$. Subsequently,

$$R\left(\theta \right)=\frac{1}{2n}{(y-Zb)}^{\top }(y-Zb)=\frac{1}{2n}{y}^{\top }y-\frac{1}{n}{y}^{\top }Zb+\frac{1}{2n}{b}^{\top }\left(Z^{\top }Z\right)b.$$

Hence, the minimization problem is equivalent to

$$\underset{b}{min}\left(-\frac{1}{n}{y}^{\top }Zb+\frac{1}{2n}{b}^{\top }\left(Z^{\top }Z\right)b\right)$$

subject to

$${\left[\begin{array}{ccccc}{\mathbf{0}}_{(1+d)\times 1}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}\\ -{\mathbf{\tau }}_{K\times 1}& I_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{\mathbf{\tau }}_{K\times 1}& {\mathbf{0}}_{K\times K}& I_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{(\mathbf{1}-\mathbf{\tau })}_{K\times 1}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& I_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{(\mathbf{1}-\mathbf{\tau })}_{K\times 1}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& I_{K\times K}\end{array}\right]}^{\top }\left[\begin{array}{c}\alpha \\ {\beta }_{+}\\ {\beta }_{-}\\ {\gamma }_{+}\\ {\gamma }_{-}\end{array}\right]\ge \left[\begin{array}{c}-t\\ {\mathbf{0}}_{K\times 1}\\ {\mathbf{0}}_{K\times 1}\\ {\mathbf{0}}_{K\times 1}\\ {\mathbf{0}}_{K\times 1}\end{array}\right],$$

where ${\mathbf{\tau }}_{K\times 1}=(\tau ,\cdots ,\tau )\in {\mathbb{R}}^K$ and ${(\mathbf{1}-\mathbf{\tau })}_{K\times 1}=((1-\tau ),\cdots ,(1-\tau ))\in {\mathbb{R}}^K$. Hence, the problem can be computed via QP.

In R, the solve.QP function in the quadprog package [18] provides the solution for QP problems that involve minimizing $(-{\mathtt{dvec}}^{\top }x+1/2x^{\top }\mathtt{Dmat}x)$ with respect to x with constraints ${\mathtt{Amat}}^{\top }x\ge \mathtt{bvec}$. Dmat is the matrix that appears in the quadratic function to be minimized. dvec is the vector that appears in the quadratic function to be minimized. bvec is the vector that has the values of the constraints. Amat is the matrix that defines the constraints for minimizing the quadratic function. Therefore, in our case, the matrices and vectors defined above correspond to

$$\mathtt{Dmat}\leftarrow \frac{1}{n}\left(Z^{\top }Z\right),\mathtt{dvec}\leftarrow \frac{1}{n}{y}^{\top }Z,\mathtt{bvec}\leftarrow \left[\begin{array}{c}-t\\ {\mathbf{0}}_{4K\times 1}\end{array}\right]$$

and

$$\mathtt{Amat}\leftarrow \left[\begin{array}{ccccc}{\mathbf{0}}_{(1+d)\times 1}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}& {\mathbf{0}}_{(1+d)\times K}\\ -{\mathbf{\tau }}_{K\times 1}& I_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{\mathbf{\tau }}_{K\times 1}& {\mathbf{0}}_{K\times K}& I_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{(\mathbf{1}-\mathbf{\tau })}_{K\times 1}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& I_{K\times K}& {\mathbf{0}}_{K\times K}\\ -{(\mathbf{1}-\mathbf{\tau })}_{K\times 1}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& {\mathbf{0}}_{K\times K}& I_{K\times K}\end{array}\right].$$

It is noteworthy that a small positive vector is added to the diagonal entries of $\frac{1}{n}\left(Z^{\top }Z\right)$ to guarantee a positive definite matrix and prevent numerical computational issues.

Finally, the solution of b obtained via QP is defined as

$$\widehat{b}=\left[\begin{array}{ccccc}\widehat{\alpha }& {\widehat{\beta }}_{+}& {\widehat{\beta }}_{-}& {\widehat{\gamma }}_{+}& {\widehat{\gamma }}_{-}\end{array}\right],$$

and the MDSE is expressed as

$${\widehat{\beta }}_k={\left({\widehat{\beta }}_k\right)}_{+}-{\left({\widehat{\beta }}_k\right)}_{-},\mathrm{and}{\widehat{\gamma }}_k={\left({\widehat{\gamma }}_k\right)}_{+}-{\left({\widehat{\gamma }}_k\right)}_{-}\mathrm{for}k=1,\dots ,K.$$

3.3. Comparison between CDA and QP

We introduce two computational algorithms to solve the constrained convex optimization problem (algorithm based on QP) and the Lagrangian dual problem (CDA); subsequently, we investigate the relationship between them.

First, we verify that the CDA estimator and the estimator obtained via QP are identical. Subsequently, a simulation is performed using the sample size $n=300$ with one true knot of 0.5, as shown in Figure 1.

In this simulation, the settings used for the CDA yield zero for all the initial coefficients, with $n_{\lambda }=50$ (the number of $\lambda$), $n_{\tau }=10$ (the number of $\tau$), $d=1$ (the non-zero degree), $ϵ=1\times {10}^{-10}$ (stopping criterion) and $R=5000$ (maximum iteration). Using $\lambda$ and $\tau$ obtained from the best CDA model, the corresponding tuning parameter t required for QP is obtained as follows:

$$t=\lambda \tau \sum _{k=1}^K|{\widehat{\beta }}_k|+\lambda \left(1-\tau \right)\sum _{k=1}^K\left|{\widehat{\gamma }}_k\right|.$$

The above relation can be derived using the Karush–Kuhn–Tucker (KKT) optimality condition [23] based on the sub-gradient [24], since the $L^1$-norm penalty term cannot be differentiated with respect to the coefficients.

The remaining options are set to the same value and then are simulated. As shown in Figure 1, we confirm that both algorithms yield the same results. However, when the number of knots (K) increases, the CDA does not converge appropriately to the minimum under a limited iteration number (M) and the stopping criterion ($ϵ$), unlike QP. Hence, we performe the next simulation using only QP. It is noteworthy that a higher number K of interior knots indicates a higher probability of overfitting.

4. Numerical Analysis

4.1. Simulation

In this section, we analyze the performance of the proposed method based on simulated examples. From model (1), we generate the predictor as n sequences in the range $[0,1]$. ${\epsilon }_i$ is generated from $N(0,{\sigma }^2)$ for $i=1,\dots ,n$ and $\sigma$ is set 0.25 for all examples.

Next, we present three examples, as follows.

Example 1.

Piecewise linear function with a single change point

$$\begin{array}{cc}\hfill f\left(x\right)& =\left\{\begin{array}{cc}\phantom{-}13x\hfill & if\phantom{.333}0\le x<0.333\hfill \\ -10x+3\hfill & if0.333\le x\le 1\hfill \end{array}\right.\hfill \end{array}$$

Example 2.

Piecewise linear function with two change points

$$\begin{array}{cc}\hfill f\left(x\right)& =\left\{\begin{array}{cc}\phantom{-}8x\hfill & if\phantom{.333}0\le x<0.333\hfill \\ \phantom{-}8x-2\hfill & if0.333\le x<0.570\hfill \\ \phantom{-}2.56\hfill & if0.570\le x<0.666\hfill \\ -10x+11.27\hfill & if0.666\le x\le 1\hfill \end{array}\right.\hfill \end{array}$$

Example 3.

Piecewise cubic function with a single change point

$$\begin{array}{cc}\hfill f\left(x\right)& =\left\{\begin{array}{cc}\phantom{-}20x^3+50x^2\hfill & if\phantom{.333}0\le x<0.333\hfill \\ -12x^2+15\hfill & if0.333\le x\le 1\hfill \end{array}\right.\hfill \end{array}$$

In Examples 1–2, a linear model with the jump(s) is considered. In Example 1, the true knot is 0.333, whereas they are 0.333 and 0.666 in Example 2. Example 3 is designed as a nonlinear function, where the true function is of quadratic and cubic form. Figure 2 shows the true functions of Examples 1–3.

We compare the proposed method with the fused lasso (FL) of Tibshirani et al. [17], the trend filtering (TF) of Kim et al. [25], the smoothing spline (SS) of Kim and Gu [26] and a fitting method for the structural model for a time series (ST) by Harvey [27].

The FL is a method that yields a solution path for the general fused lasso problem. The TF is the method of computing the solution path for the trend filtering problem of an arbitrary polynomial order. The SS is a method of fitting smoothing spline ANOVA models in Gaussian regression. The ST fits a structural model for a time series using the maximum likelihood. It is noteworthy that the FL, TF and SS estimators use nonparametric regression function estimation models. In particular, the TF and SS are function estimators based on linear combinations of spline basis functions, similar to ours. In addition, the FL is an estimator that specializes in change point detection as a zero-degree piecewise constant function.

The FL and TF estimators are provided in the genlasso package in R [28]. They require the use of cross-validation for parameter selection. In addition, the SS and ST can be used in R via the gss [29] and stats packages [30], respectively. To reduce the computational burden, we compute all the estimators using the default settings for each package.

We consider the mean squared error (MSE), mean absolute error (MAE) and maximum deviation (MXDV) criterion as loss functions that measure the discrepancy between the true function f and each function estimator $\widehat{f}$. They are expressed as follows:

$$\begin{array}{c}\hfill \mathrm{MSE}\left(\widehat{f}\right)=\frac{1}{n}\sum _{i=1}^n{\left(f\left(x_i\right)-\widehat{f}\left(x_i\right)\right)}^2,\mathrm{MAE}\left(\widehat{f}\right)=\frac{1}{n}\sum _{i=1}^n\left|f\left(x_i\right)-\widehat{f}\left(x_i\right)\right|\end{array}$$

$$\begin{array}{c}\hfill \mathrm{and}\mathrm{MXDV}\left(\widehat{f}\right)=\underset{1\le i\le n}{max}\left|f\left(x_i\right)-\widehat{f}\left(x_i\right)\right|.\end{array}$$

Examples 1–2 are simulated based on $d=1$, whereas Example 3 is simulated based on $d=3$, with $K=20$ and $K=50$, respectively. In addition, a simulation is performed by setting TF with orders 1 and 3 for linear and cubic functions, respectively. The simulation is repeated 100 times for each example based on sample sizes of 200, 300 and 500.

Table 1. Average ($\times 100$) of each criterion over 100 trials of QP, FL, TF, SS and ST for Example 1 (for sample size n = 200, 300 and 500; standard error in parentheses). Bold text represents smallest criterion for each scenario.

n		QP (K = 20)	QP (K = 50)	FL	TF (ord = 1)	TF (ord = 3)	SS	ST
200	MSE	0.52	0.80	1.72	1.96	3.54	9.47	4.55
		($3\times {10}^{-3}$)	($6\times {10}^{-3}$)	($3\times {10}^{-3}$)	($7\times {10}^{-3}$)	($8\times {10}^{-3}$)	($3\times {10}^{-2}$)	($9\times {10}^{-3}$)
	MAE	4.27	5.35	10.41	8.45	11.77	13.16	16.18
		($1\times {10}^{-2}$)	($2\times {10}^{-2}$)	($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($2\times {10}^{-2}$)	($2\times {10}^{-2}$)	($4\times {10}^{-3}$)
	MXDV	37.70	49.67	39.71	102.70	133.93	217.82	117.69
		($2\times {10}^{-1}$)	($3\times {10}^{-1}$)	($7\times {10}^{-2}$)	($4\times {10}^{-1}$)	($3\times {10}^{-1}$)	($2\times {10}^{-1}$)	($2\times {10}^{-2}$)
300	MSE	0.22	0.40	0.94	1.11	2.11	7.85	2.91
		($1\times {10}^{-3}$)	($4\times {10}^{-3}$)	($2\times {10}^{-3}$)	($4\times {10}^{-3}$)	($4\times {10}^{-3}$)	($3\times {10}^{-2}$)	($7\times {10}^{-3}$)
	MAE	2.74	3.78	7.72	6.15	8.66	10.70	12.76
		($8\times {10}^{-3}$)	($1\times {10}^{-2}$)	($7\times {10}^{-3}$)	($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($2\times {10}^{-2}$	($3\times {10}^{-3}$)
	MXDV	25.84	37.02	29.90	95.21	133.34	222.44	120.09
		($1\times {10}^{-1}$)	($2\times {10}^{-1}$)	($6\times {10}^{-2}$)	($3\times {10}^{-2}$)	($3\times {10}^{-2}$)	($1\times {10}^{-2}$)	($2\times {10}^{-2}$)
500	MSE	0.13	0.22	0.65	0.92	4.57	7.07	2.55
		($9\times {10}^{-4}$)	($2\times {10}^{-3}$)	($1\times {10}^{-3}$)	($2\times {10}^{-3}$)	($7\times {10}^{-3}$)	($2\times {10}^{-2}$)	($5\times {10}^{-3}$)
	MAE	2.01	2.70	6.44	5.43	6.91	9.43	11.62
		($5\times {10}^{-3}$)	($1\times {10}^{-2}$)	($4\times {10}^{-3}$)	($8\times {10}^{-3}$)	($5\times {10}^{-3}$)	($2\times {10}^{-2}$)	($2\times {10}^{-3}$)
	MXDV	19.84	28.95	26.03	113.64	218.35	232.12	159.22
		($1\times {10}^{-1}$)	($2\times {10}^{-1}$)	($5\times {10}^{-2}$)	($3\times {10}^{-1}$)	($6\times {10}^{-2}$)	($1\times {10}^{-1}$)	($2\times {10}^{-1}$)

,

Table 2. Average ($\times 100$) of each criterion over 100 trials of QP, FL, TF, SS and ST for Example 2 (for sample sizes n = 200, 300 and 500; standard error in parentheses). The bold text represents the smallest criterion for each scenario.

n		QP (K = 20)	QP (K = 50)	FL	TF (ord = 1)	TF (ord = 3)	SS	ST
200	MSE	0.62	1.39	1.51	2.10	2.89	4.44	3.51
		($3\times {10}^{-3}$)	($1\times {10}^{-2}$)	($3\times {10}^{-3}$)	($4\times {10}^{-3}$)	($8\times {10}^{-3}$)	($9\times {10}^{-3}$)	($8\times {10}^{-3}$)
	MAE	5.65	8.32	9.65	9.20	11.46	11.93	13.96
		($1\times {10}^{-2}$)	($3\times {10}^{-2}$)	($1\times {10}^{-2}$)	($2\times {10}^{-2}$)	($2\times {10}^{-2}$)	($1\times {10}^{-2}$)	($4\times {10}^{-3}$)
	MXDV	28.50	39.88	38.92	83.52	86.71	100.90	93.29
		($1\times {10}^{-1}$)	($2\times {10}^{-1}$)	($8\times {10}^{-2}$)	($1\times {10}^{-1}$)	($1\times {10}^{-1}$)	($6\times {10}^{-2}$)	($1\times {10}^{-1}$)
300	MSE	0.43	0.58	1.13	1.64	2.19	3.82	3.05
		($2\times {10}^{-3}$)	($4\times {10}^{-3}$)	($2\times {10}^{-3}$)	($3\times {10}^{-3}$)	($3\times {10}^{-3}$)	($6\times {10}^{-3}$)	($9\times {10}^{-3}$)
	MAE	4.72	5.50	8.37	7.87	9.72	10.53	12.67
		($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($8\times {10}^{-3}$)	($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($4\times {10}^{-3}$)
	MXDV	25.33	29.66	36.30	86.71	89.47	103.02	107.01
		($9\times {10}^{-2}$)	($1\times {10}^{-1}$)	($1\times {10}^{-1}$)	($1\times {10}^{-1}$)	($1\times {10}^{-1}$)	($5\times {10}^{-2}$)	($1\times {10}^{-2}$)
500	MSE	0.25	0.63	0.76	1.20	1.66	3.21	2.49
		($1\times {10}^{-3}$)	($5\times {10}^{-3}$)	($1\times {10}^{-3}$)	($1\times {10}^{-3}$)	($1\times {10}^{-3}$)	($6\times {10}^{-3}$)	($6\times {10}^{-3}$)
	MAE	3.52	5.21	6.87	6.54	7.76	9.09	11.23
		($8\times {10}^{-3}$)	($2\times {10}^{-2}$)	($6\times {10}^{-3}$)	($1\times {10}^{-2}$)	($9\times {10}^{-3}$)	($1\times {10}^{-2}$)	($2\times {10}^{-3}$)
	MXDV	20.33	32.89	32.42	91.12	96.79	105.51	122.67
		($9\times {10}^{-2}$)	($1\times {10}^{-1}$)	($1\times {10}^{-1}$)	($8\times {10}^{-2}$)	($5\times {10}^{-2}$)	($6\times {10}^{-2}$)	($1\times {10}^{-1}$)

and

Table 3. Average ($\times 100$) of each criterion over 100 trials of QP, FL, TF, SS and ST for Example 3 (for sample sizes n = 200, 300 and 500; standard error in parentheses). Bold text represents smallest criterion for each scenario.

n		QP (K = 20)	QP (K = 50)	FL	TF (ord = 1)	TF (ord = 3)	SS	ST
200	MSE	0.95	1.97	2.30	1.77	4.81	26.96	5.53
		($6\times {10}^{-3}$)	($3\times {10}^{-3}$)	($6\times {10}^{-3}$)	($5\times {10}^{-3}$)	($1\times {10}^{-2}$)	($1\times {10}^{-1}$)	($1\times {10}^{-2}$)
	MAE	6.73	11.11	11.90	8.62	12.59	18.83	18.58
		($2\times {10}^{-2}$)	($9\times {10}^{-3}$)	($1\times {10}^{-2}$)	($2\times {10}^{-2}$)	($2\times {10}^{-2}$)	($4\times {10}^{-2}$)	($6\times {10}^{-3}$)
	MXDV	50.06	41.95	46.85	91.63	165.54	382.14	81.19
		($1\times {10}^{-1}$)	($6\times {10}^{-2}$)	($9\times {10}^{-2}$)	($3\times {10}^{-1}$)	($6\times {10}^{-1}$)	($3\times {10}^{-1}$)	($2\times {10}^{-1}$)
300	MSE	0.63	1.46	1.65	1.57	3.97	23.48	5.20
		($2\times {10}^{-3}$)	($2\times {10}^{-3}$)	($2\times {10}^{-3}$)	($3\times {10}^{-3}$)	($6\times {10}^{-3}$)	($8\times {10}^{-2}$)	($9\times {10}^{-3}$)
	MAE	5.46	9.52	10.10	7.69	11.20	16.70	17.69
		($9\times {10}^{-3}$)	($8\times {10}^{-3}$)	($7\times {10}^{-3}$)	($1\times {10}^{-2}$)	($1\times {10}^{-2}$)	($4\times {10}^{-2}$)	($5\times {10}^{-3}$)
	MXDV	47.32	38.72	42.29	113.08	191.36	395.16	119.95
		($1\times {10}^{-1}$)	($6\times {10}^{-2}$)	($8\times {10}^{-2}$)	($3\times {10}^{-1}$)	($4\times {10}^{-1}$)	($2\times {10}^{-1}$)	($3\times {10}^{-1}$)
500	MSE	0.50	1.02	1.17	1.50	15.90	21.60	4.79
		($1\times {10}^{-3}$)	($2\times {10}^{-3}$)	($1\times {10}^{-3}$)	($7\times {10}^{-3}$)	($2\times {10}^{-2}$)	($8\times {10}^{-2}$)	($7\times {10}^{-3}$)
	MAE	4.56	7.79	8.50	6.96	11.23	15.17	16.37
		($7\times {10}^{-3}$)	($9\times {10}^{-3}$)	($5\times {10}^{-3}$)	($9\times {10}^{-3}$)	($9\times {10}^{-3}$)	($4\times {10}^{-2}$)	($4\times {10}^{-3}$)
	MXDV	47.87	38.85	39.65	146.05	390.08	407.96	193.31
		($8\times {10}^{-2}$)	($6\times {10}^{-2}$)	($9\times {10}^{-2}$)	($5\times {10}^{-1}$)	($8\times {10}^{-2}$)	($2\times {10}^{-1}$)	($3\times {10}^{-1}$)

show the experimental results of various scenarios for each of the examples.

As shown in Table 1, because QP not only can detect change points (or jumps), but can also fit piecewise polynomial regression, it demonstrates the best performance compared with the other methods. Comparing the QP results based on K, it is discovered that the best performance is obtained when $K=20$. Figure 3 shows the plot for the results of Example 1 when $n=500$. QP detects the change point and fits the linear regression accurately, particularly when $K=20$. Although the performances of FL and TF with order 1 improve in general as the sample size n increases, it cannot detect the change point. Herein, “cannot detect the change point” means that, around the true knot, the plot has point(s) alone or successive.

Table 2 lists the results of Example 2, which is involved the same slope, two jumps and no jump at $x=0.57$. It is observed that QP demonstrates the best performance when $K=20$, not $K=50$. Figure 4 shows the plot for Example 2 when $n=500$. QP detects the change point and fits the linear regression accurately, particularly when $K=20$. Although most performances of FL and TF with order 1 improve in general, as the sample size n increases, they cannot detect the change point. Herein, “cannot detect the change point” means that, around the true knot, the plot has point(s) alone or successive.

Table 3 lists the results of Example 3, which is involved one jump as well as quadratic and cubic forms. Similarly, the proposed QP estimator yieldes the best results. $K=20$ yieldes the best results in terms of the MSE and MAE, whereas $K=50$ in terms of the MXDV. Figure 5 shows the results of Example 3 for all methods when $n=500$. As shown, even when it do not fit the polynomial regression line well at $K=50$, it detects the point of change accurately at the true knot.

QP detects the change point and fits the linear regression accurately, particularly when $K=20$. Although the performances of the FL and TF with order 1 improve as the sample size n increases, they cannot detect the change point. Herein, “cannot detect the change point” means that, around the true knot, the plot has point(s) alone or successive.

4.2. Real Data Analysis

The data obtained from Investing.com (last accessed date: 1 October 2021) the exchange rate change of the Icelandic króna (ISK) per US dollar (USD) from January 2004 to December 2015, measured as the monthly average.

The data contained 144 sets of observations associated with the subprime mortgage crisis, which occurred between 2007 and 2008. This crisis refers to a series of economic crises that began with the bankruptcy of the largest mortgage lenders in the United States and caused a credit crunch not only in the United States, but also in the international financial market. One of affected countries was Iceland, whose three largest banks were affected by the crisis in September 2008. To revive the economy, the Icelandic government has implemented various policies, including the nationalization of banks. Consequently, the ISK exchange rate per USD increased. In Figure 6, the top panel shows the starting point for the data. In January 2004, the observed value is 69.345; in March 2008, the observed value increases gradually to 75.805 and then increases significantly to 105.965 in September. In the last observation, performed in December 2016, a continuous increase of the value is observed (129.960).

Because the input variable is a date and not numeric data, 144 samples are equally spaced in the interval [0, 1] for the proposed QP implementation. A degree exceeding four can result in a more complicated interpretation and overfitting. Therefore, we apply $d=1,2$ and 3.

Figure 6 shows the fitted results. The top panel of Figure 6 shows a scatterplot of the data. The remaining panels show the results of the QP model fitting when $d=1,2$ and 3. It appears that the proposed MDSE based on the BIC detects a change point in September 2008 and fits the flow of the exchange rate well. Prior to 2008, no significant change is detected, but from January 2008, the exchange rate of ISK per USD increases gradually. Subsequently, beginning from September 2008, the exchange rate of ISK per USD increases significantly, as indicated by the vertical black dashed line in the figure (representing September 2008).

5. Conclusions

In this study, we develop a nonparametric regression function estimation method with change point detection. In order to provide a data-driven knot selection method, we consider elastic-net-type $L^1$-norm penalty for the estimating regression function. Once we expressed the estimator with a linear combination of truncated power splines, the coefficients were estimated by minimizing a penalized residual sum of squares. A new coordinate descent algorithm and an algorithm based on quadratic programming are introduced to handle penalty terms determined by the regression coefficients. In the numerical analysis, we only use an algorithm based on quadratic programming because of the high computational cost of the coordinate descent algorithm when the number of knots increases. To verify the performance of the proposed estimator (MDSE), in the three simulations, we compared MDSE with four methods; the fused lasso (FL), trend filtering (TF), smoothing spline (SS) and structural model for a times series (ST). As a result, in all simulations, the MDSE detected the change point(s) and fitted linear and cubic trends of the piecewise polynomial well. Furthermore, we performed an actual data analysis about the sharp change in exchange rate of the Icelandic króna per US dollar in September 2008 caused by the subprime mortgage crisis in the United States between 2007 and 2008. In conclusion, MDSE can accurately detect September 2008, the time when a sharp increase in the exchange rate occurred, and can also represent the exchange rate trend before and after September 2008 well.

To improve the proposed coordinate descent algorithm, one may apply the B-spline basis and total variation penalty terms. The B-spline basis function is a numerically superior alternative to the truncated power basis. Because it has small supports, the predictor matrix is sparse and the information matrix is banded (chapter 1 and 8 of [31]). The B-spline-based total variation penalty term is a generalized lasso problem [32,33] that is difficult to implement, but affords high computational power. These approaches will be investigated in future studies.