Maximum Entropy Expectation-Maximization Algorithm for Fitting Latent-Variable Graphical Models to Multivariate Time Series

Abstract

This work is focused on latent-variable graphical models for multivariate time series. We show how an algorithm which was originally used for finding zeros in the inverse of the covariance matrix can be generalized such that to identify the sparsity pattern of the inverse of spectral density matrix. When applied to a given time series, the algorithm produces a set of candidate models. Various information theoretic (IT) criteria are employed for deciding the winner. A novel IT criterion, which is tailored to our model selection problem, is introduced. Some options for reducing the computational burden are proposed and tested via numerical examples. We conduct an empirical study in which the algorithm is compared with the state-of-the-art. The results are good, and the major advantage is that the subjective choices made by the user are less important than in the case of other methods.

1. Introduction

Graphical models are instrumental in the analysis of multivariate data. Originally, these models have been employed for independently sampled data, but their use has been extended to multivariate, stationary time series [1,2], which triggered their popularity in statistics, machine learning, signal processing and neuroinformatics.

For better understanding the significance of graphical models, let $\mathbf{x}$ be a random vector having a Gaussian distribution with zero-mean and positive definite covariance matrix $\mathsf{\Gamma }$. A graph $G=(V,E)$ can be assigned to $\mathbf{x}$ in order to visualize the conditional independence between its components. The symbol V denotes the vertices of G, while E is the set of its edges. There are no loops from a vertex to itself, nor multiple edges between two vertices. Hence, E is a subset of $\left\{(a,b)\in V\times V:a\ne b\right\}$. Each vertex of the graph is assigned to an entry of $\mathbf{x}$. We conventionally draw an edge between two vertices a and b if the random variables ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$ are not conditionally independent, given all other components of $\mathbf{x}$. The description above follows the main definitions from [3] and assumes that the graph G is undirected. Proposition 1 from the same reference provides a set of equivalent conditions for conditional independence. The most interesting one claims that ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$ are conditionally independent if and only if the entry $(a,b)$ of ${\mathsf{\Gamma }}^{-1}$ is zero. This shows that the missing edges of G correspond to zero-entries in the inverse of the covariance matrix, which is called the concentration matrix.

There is an impressive amount of literature on graphical models. In this work, we focus on a generalization of this problem to time series. The main difference between the static case and the dynamic case is that the former relies on the sparsity pattern of the concentration matrix, whereas the latter is looking for zeros in the inverse of the spectral density matrix. One of the main difficulties stems from the fact that the methods developed in the static case cannot be applied straightforwardly to time series.

Parametric as well as non-parametric methods have been already proposed in the previous literature dedicated to graphical models for time series. Some of the recently introduced estimation methods are based on convex optimization. We briefly discuss below the most important algorithms which belong to this class.

• Reference [4] extends the static case by allowing the presence of latent variables. The key point of their approach is to express the manifest concentration matrix as the sum of a sparse matrix and a low-rank matrix. Additionally, they provide conditions for the decomposition to be unique, in order to guarantee the identifiability. The two matrices are estimated by minimizing a penalized likelihood function, where the penalty involves both the ${\ell }_1$-norm and the nuclear norm. Interestingly enough, the authors of the discussion paper [5] pointed out that an alternative solution, which relies on the Expectation-Maximization algorithm, can be easily obtained.
• In the dynamic case, reference [6] has an important contribution which consists in showing that the graphical models for multivariate autoregressive processes can be estimated by solving a convex optimization problem which follows from the application of the Maximum Entropy principle. This paved the way for the development of efficient algorithms dedicated to topology selection in graphical models of autoregressive processes [7,8] and autoregressive moving average processes [9].
• A happy marriage between the approach from [4] and the use of Maximum Entropy led to the solution proposed in [10] for the identification of graphical models of autoregressive processes with latent variables. Similar to [4], the estimation is done by minimizing a cost function whose penalty term is given by a linear combination of the ${\ell }_1$-norm and the nuclear norm. The two coefficients of this linear combination are chosen by the user and they have a strong influence on the estimated model. The method introduced in [10] performs the estimation for various pairs of coefficients which yield a set of candidate models; the winner is decided by using a score function.

According to the best of our knowledge, there is no other work that extends the estimation method from [5] to the case of latent-variable autoregressive models. The main contribution of this paper is to propose an algorithm of this type, which combines the strengths of Expectation-Maximization and convex optimization. The key point for achieving this goal is to apply the Maximum Entropy principle.

The rest of the paper is organized as follows. In the next section, we introduce the notation and present the method from [10]. Section 3 outlines the newly proposed algorithm. The outcome of the algorithm is a set of models from which we choose the best one by employing information theoretic (IT) criteria. Section 4 is focused on the description of these criteria: We discuss the selection rules from the previous literature and propose a novel criterion. The experimental results are reported in Section 5. Section 6 concludes the paper.

2. Preliminaries and Previous Work

Let ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_T$ be a $\kappa$-dimensional ($\kappa >1$) time series generated by a stationary and stable $\mathrm{VAR}$ process of order p. We assume that the spacing of observation times is constant and ${\mathbf{x}}_t={\left[{\mathbf{x}}_{1t},\dots ,{\mathbf{x}}_{\kappa t}\right]}^{\top }$. The symbol ${\left(·\right)}^{\top }$ denotes transposition. The difference equation of the process is

$${\mathbf{x}}_t={\mathbf{A}}_1{\mathbf{x}}_{t-1}+\dots +{\mathbf{A}}_p{\mathbf{x}}_{t-p}+{\mathbf{ϵ}}_t,t=\overline{1,T},$$

(1)

where ${\mathbf{A}}_1,\dots ,{\mathbf{A}}_p$ are matrix coefficients of size $\kappa \times \kappa$ and ${\mathbf{ϵ}}_t$ is a sequence of independently and identically distributed random $\kappa$-vectors. We assume that the vectors ${\left\{{\mathbf{ϵ}}_t\right\}}_{t=1}^T$ are drawn from a $\kappa$-variate Gaussian distribution with zero mean vector and covariance matrix $\mathsf{\Sigma }\succ 0$. Additionally, the vectors ${\left\{{\mathbf{x}}_t\right\}}_{t=1-p}^0$ are assumed to be constant.

The conditional independence relations between the variables in ${\mathbf{x}}_t$ are provided by the inverse of the spectral density matrix (ISDM) of the $\mathrm{VAR}$-process $\left\{{\mathbf{x}}_t\right\}$. The ISDM has the expression

$${\mathsf{\Phi }}^{-1}\left(\omega \right)={\mathbf{A}}^H\left(\omega \right){\mathsf{\Sigma }}^{-1}\mathbf{A}\left(\omega \right)=\sum _{i=-p}^p{\mathbf{Q}}_i{e}^{-j\omega i},$$

(2)

where $j=\sqrt{-1}$ and ${\left(·\right)}^H$ is the operator for conjugate transpose. We define ${\mathbf{A}}_0=-\mathbf{I}$, where $\mathbf{I}$ stands for the identity matrix of appropriate size, and $\mathbf{A}\left(\omega \right)=-{\sum }_{i=0}^p{\mathbf{A}}_i{e}^{-j\omega i}$. For $i\ge 0$, we have that ${\mathbf{Q}}_i={\sum }_{k=0}^{p-i}{\mathbf{A}}_k^{\top }{\mathsf{\Sigma }}^{-1}{\mathbf{A}}_{k+i}$ and ${\mathbf{Q}}_{-i}={\mathbf{Q}}_i^{\top }$. The sparse structure of the ISDM contains conditional dependence relations between the variables of ${\mathbf{x}}_t$, i.e., two variables ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$ are independent, conditional on the other variables, if and only if [1,11]

$${\left[{\mathsf{\Phi }}^{-1}\left(\omega \right)\right]}_{ab}=0,\forall \omega \in \left(-\pi ,\pi \right].$$

(3)

In the graph corresponding to the ISDM ${\mathsf{\Phi }}^{-1}\left(\omega \right)$, the nodes stand for the variables of the model, and the edges stand for conditional dependence, i.e., there is no edge between conditionally independent variables.

In a latent-variable graphical model it is assumed that $\kappa =\mathrm{K}+\mathrm{r}$, where $\mathrm{K}$ variables are accessible to observation (they are called manifest variables) and $\mathrm{r}$ variables are latent, i.e., not accessible to observation, but playing a significant role in the conditional independence pattern of the overall model. The existence of latent variables in a model can be described in terms of the ISDM by the block decomposition

$$\mathsf{\Phi }\left(\omega \right)=\left[\begin{array}{cc}{\mathsf{\Phi }}_m\left(\omega \right)& {\mathsf{\Phi }}_{\ell m}^{\top }(-\omega )\\ {\mathsf{\Phi }}_{\ell m}\left(\omega \right)& {\mathsf{\Phi }}_{\ell }\left(\omega \right)\end{array}\right],{\mathsf{\Phi }}^{-1}\left(\omega \right)=\left[\begin{array}{cc}{\mathbf{{\rm Y}}}_m\left(\omega \right)& {\mathbf{{\rm Y}}}_{\ell m}^{\top }(-\omega )\\ {\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)& {\mathbf{{\rm Y}}}_{\ell }\left(\omega \right)\end{array}\right],$$

(4)

where ${\mathsf{\Phi }}_m\left(\omega \right)$ and ${\mathsf{\Phi }}_{\ell }\left(\omega \right)$ are the manifest and latent components of the spectral density matrix, respectively. Using the Schur complement, the ISDM of the manifest component has the form [10] (Equation (21)):

$${\mathsf{\Phi }}_m^{-1}\left(\omega \right)={\mathbf{{\rm Y}}}_m\left(\omega \right)-{\mathbf{{\rm Y}}}_{\ell m}^{\top }\left(-\omega \right){\mathbf{{\rm Y}}}_{\ell }^{-1}\left(\omega \right){\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right).$$

(5)

When building latent variable graphical models, we assume that $\mathrm{r}\ll \mathrm{K}$, i.e., few latent variables are sufficient to characterize the conditional dependence structure of the model. The previous formula can therefore be written

$${\mathsf{\Phi }}_m^{-1}\left(\omega \right)=\mathbf{S}\left(\omega \right)-\mathsf{\Lambda }\left(\omega \right),$$

(6)

where $\mathbf{S}\left(\omega \right)$ is sparse, and $\mathsf{\Lambda }\left(\omega \right)$ has (constant) low-rank almost everywhere in $(-\pi ,\pi ]$. Furthermore, we can write [12] (Equation (4)):

$$\begin{array}{c}\hfill \mathbf{S}\left(\omega \right)-\mathsf{\Lambda }\left(\omega \right)=\mathbf{\Delta }\left(\omega \right)\mathbf{X}\mathbf{\Delta }{\left(\omega \right)}^H,\end{array}$$

(7)

$$\begin{array}{c}\hfill \mathsf{\Lambda }\left(\omega \right)=\mathbf{\Delta }\left(\omega \right)\mathbf{L}\mathbf{\Delta }{\left(\omega \right)}^H,\end{array}$$

(8)

where $\mathbf{\Delta }\left(\omega \right)=\left[\mathbf{I},e^{j\omega }\mathbf{I},\dots ,e^{j\omega p}\mathbf{I}\right]$ is a shift matrix, and $\mathbf{X}$ and $\mathbf{L}$ are $\mathrm{K}\left(p+1\right)\times \mathrm{K}\left(p+1\right)$ positive semidefinite matrices. We split all such matrices in $\mathrm{K}\times \mathrm{K}$ blocks, e.g.,

$$\mathbf{L}=\left[\begin{array}{ccc}{\mathbf{L}}_{00}& \dots & {\mathbf{L}}_{0p}\\ ⋮& \ddots & ⋮\\ {\mathbf{L}}_{0p}^{\top }& \dots & {\mathbf{L}}_{pp}\end{array}\right].$$

(9)

The block trace operator for such a matrix is $\mathcal{D}(·)$, defined by

$${\mathcal{D}}_i\left(\mathbf{L}\right)=\sum _{h=0}^{p-i}{\mathbf{L}}_{h,h+i},i=\overline{0,p}.$$

(10)

For negative indices, the relation ${\mathcal{D}}_{-i}\left(\mathbf{L}\right)={\mathcal{D}}_i{\left(\mathbf{L}\right)}^{\top }$ holds. Note that (8) can be rewritten as

$$\mathsf{\Lambda }\left(\omega \right)=\sum _{i=-p}^p{\mathcal{D}}_i\left(\mathbf{L}\right)e^{j\omega i}.$$

The first $p+1$ sample covariances of the $\mathrm{VAR}$ process are [13]:

$${\widehat{\mathbf{C}}}_i=\frac{1}{T}\sum _{t=1}^{T-i}{\mathbf{x}}_{t+i}{\mathbf{x}}_t^{\top },i=\overline{0,p}.$$

(11)

However, only the upper left $\mathrm{K}\times \mathrm{K}$ blocks corresponding to the manifest variables can be computed from data; they are denoted ${\widehat{\mathbf{R}}}_i$. With $\widehat{\mathbf{R}}=[{\widehat{\mathbf{R}}}_0\dots {\widehat{\mathbf{R}}}_p]$, we build the block Toeplitz matrix

$$\mathcal{T}\left(\widehat{\mathbf{R}}\right)=\left[\begin{array}{ccc}{\widehat{\mathbf{R}}}_0& \dots & {\widehat{\mathbf{R}}}_p\\ ⋮& \ddots & ⋮\\ {\widehat{\mathbf{R}}}_p^{\top }& \dots & {\widehat{\mathbf{R}}}_0\end{array}\right].$$

(12)

It was proposed in [10] to estimate the matrices $\mathbf{X}$ and $\mathbf{L}$ by solving the optimization problem

$$\begin{array}{c}\hfill \begin{array}{cc}\underset{\mathbf{X},\mathbf{L}}{min}& \mathrm{tr}\left(\mathcal{T}\left(\widehat{\mathbf{R}}\right)\mathbf{X}\right)-\log det{\mathbf{X}}_{00}+\lambda \gamma f\left(\mathbf{X}+\mathbf{L}\right)+\lambda \mathrm{tr}\left(\mathbf{L}\right)\hfill \\ \mathrm{s}.\mathrm{t}.& \mathbf{X}⪰0\mathbf{L}⪰0,\hfill \end{array}\end{array}$$

(13)

where $\mathrm{tr}(·)$ is the trace operator, $\log (·)$ denotes the natural logarithm and $det(·)$ stands for the determinant. Minimizing $\mathrm{tr}\left(\mathbf{L}\right)$ induces low rank in $\mathbf{L}$ and $\lambda ,\gamma >0$ are trade-off constants. The function $f(·)$ is a group sparsity promoter whose expression is given by

$$f\left(\mathbf{Z}\right)=\sum _{a=1}^{\mathrm{K}}\sum _{b=1}^{a-1}\underset{i=\overline{0,p}}{max}\left|{\mathcal{D}}_i\left(\mathbf{Z}\right)(a,b)\right|.$$

(14)

Note that ${\mathcal{D}}_i(\mathbf{X}+\mathbf{L})(a,b)$ is the i-th degree coefficient of the polynomial that occupies the $(a,b)$ position in the matrix polynomial ${\mathsf{\Phi }}_m^{-1}\left(\omega \right)$. Sparsity is encouraged by minimizing the ${\ell }_1$-norm of the vector formed by the coefficients that are maximum for each position $(a,b)$.

3. New Algorithm

The obvious advantage of the optimization problem (13) is its convexity, which allows the safe computation of the solution. However, a possible drawback is the presence of two parameters, $\lambda$ and $\gamma$, whose values should be chosen. A way to eliminate one of the parameters is to assume that the number $\mathrm{r}$ of latent variables is known. At least for parsimony reasons, it is natural to suppose that $\mathrm{r}$ is very small. Since a latent variable influences all manifest variables in the ISDM (5), there cannot be too many independent latent variables. Therefore, giving $\mathrm{r}$ a fixed small value is likely to be not restrictive.

In this section, we describe an estimation method which is clearly different from the one in [10]. More precisely, we generalize the Expectation-Maximization algorithm from [5], developed there for independent and identically distributed random variables, to a $\mathrm{VAR}$ process. For this purpose, we work with the full model (4) that includes the ISDM part pertaining to the $\mathrm{r}$ latent variables. Without loss of generality, we assume that ${\mathbf{{\rm Y}}}_{\ell }\left(\omega \right)$ equals the identity matrix $\mathbf{I}$; the effect of the latent variables on the manifest ones in (5) can be modeled by ${\mathbf{{\rm Y}}}_{\ell m}$ alone. Combining with (2), the model is

$${\mathsf{\Phi }}^{-1}\left(\omega \right)=\left[\begin{array}{cc}{\mathbf{{\rm Y}}}_m\left(\omega \right)& {\mathbf{{\rm Y}}}_{\ell m}^{\top }(-\omega )\\ {\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)& \mathbf{I}\end{array}\right]=\sum _{i=-p}^p{\mathbf{Q}}_i{e}^{-j\omega i},$$

(15)

where the matrices ${\mathbf{Q}}_i$ have to be found.

The main difficulty of this approach is the unavailability of the latent part of the matrices (11). Were such matrices available, we could work with SDM $\mathsf{\Phi }\left(\omega \right)$ estimators (confined to order p) of the form

$$\tilde{\mathsf{\Phi }}\left(\omega \right)=\sum _{i=-p}^p{\mathbf{C}}_i{e}^{-j\omega i},$$

(16)

where ${\mathbf{C}}_i$ denotes the i-th covariance lag for the VAR process $\left\{{\mathbf{x}}_t\right\}$ (see also (1) and (11)). We split the matrix coefficients from (15) and (16) according to the size of manifest and latent variables, e.g.,

$${\mathbf{C}}_i=\left[\begin{array}{cc}{\mathbf{C}}_{m,i}& {\mathbf{C}}_{\ell m,-i}^{\top }\\ {\mathbf{C}}_{\ell m,i}& {\mathbf{C}}_{\ell ,i}\end{array}\right].$$

(17)

To overcome the difficulty, the Expectation-Maximization algorithm alternatively keeps fixed either the model parameters ${\mathbf{Q}}_i$ or the matrices ${\mathbf{C}}_i$, estimating or optimizing the remaining unknowns. The expectation step of Expectation-Maximization assumes that the ISDM ${\mathsf{\Phi }}^{-1}\left(\omega \right)$ from (15) is completely known. Standard matrix identities [5] can be easily extended to matrix trigonometric polynomials for writing down the formula

$$\begin{array}{c}\hfill \mathsf{\Phi }\left(\omega \right)=\left[\begin{array}{cc}{\mathsf{\Phi }}_m\left(\omega \right)& -{\mathsf{\Phi }}_m\left(\omega \right){\mathbf{{\rm Y}}}_{\ell m}^{\top }(-\omega )\\ -{\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right){\mathsf{\Phi }}_m\left(\omega \right)& \mathbf{I}+{\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right){\mathsf{\Phi }}_m\left(\omega \right){\mathbf{{\rm Y}}}_{\ell m}^{\top }(-\omega )\end{array}\right].\end{array}$$

(18)

Identifying (16) with (18) gives expressions for estimating the matrices ${\mathbf{C}}_i$, depending on the matrices ${\mathbf{Q}}_i$ from (15). The upper left corner of (18) needs no special computation, since the natural estimator is

$${\mathsf{\Phi }}_m\left(\omega \right)=\sum _{i=-p}^p{\widehat{\mathbf{R}}}_i{e}^{-j\omega i},$$

where the sample covariances ${\widehat{\mathbf{R}}}_i$ are directly computable from the time series. It results that

$${\mathbf{C}}_{m,i}={\widehat{\mathbf{R}}}_i,i=\overline{-p,p}.$$

(19)

The other blocks from (17) result from convolution expressions associated with the polynomial multiplications from (18). The lower left block of the coefficients is

$${\mathbf{C}}_{\ell m,i}=-\sum _{k+s=i}{\mathbf{Q}}_{\ell m,k}{\widehat{\mathbf{R}}}_s=-\sum _{k=max(-p,i-p)}^{min(p,i+p)}{\mathbf{Q}}_{\ell m,k}{\widehat{\mathbf{R}}}_{i-k},i=\overline{-2p,2p}.$$

(20)

Note that the trigonometric polynomial ${\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right){\mathsf{\Phi }}_m\left(\omega \right)$ has degree $2p$, since its factors have degree p. With (20) available, we can compute

$${\mathbf{C}}_{\ell ,i}={\delta }_i\mathbf{I}+\sum _{k-s=i}{\mathbf{C}}_{\ell m,k}{\mathbf{Q}}_{\ell m,-s}^{\top }={\delta }_i\mathbf{I}+\sum _{k=max(-2p,i-p)}^{min(2p,i+p)}{\mathbf{C}}_{\ell m,k}{\mathbf{Q}}_{\ell m,i-k}^{\top },i=\overline{-p,p},$$

(21)

where ${\delta }_i=1$ if $i=0$ and ${\delta }_i=0$ otherwise. Although the degree of the polynomial from the lower right block of (18) is $3p$, we need to truncate it to degree p, since this is the degree of the ISDM ${\mathsf{\Phi }}^{-1}\left(\omega \right)$ from (15). This is the reason for computing only the coefficients $i=\overline{-p,p}$ in (21). The same truncation is applied on (20); note that there we cannot compute only the coefficients that are finally needed, since all of them are required in (21).

In the maximization step of Expectation-Maximization, the covariance matrices ${\mathbf{C}}_i$ are assumed to be known and are fixed; the ISDM can be estimated by solving an optimization problem that will be detailed below. The overall solution we propose is outlined in Algorithm 1, explained in what follows.

Algorithm 1 Algorithm for Identifying $\mathrm{SP}$ of ISDM (AlgoEM)

Input: $\mathrm{Data}\left({\mathbf{x}}_{1:K,1},\dots ,{\mathbf{x}}_{1:K,T}\right)$, $\mathrm{VAR}$-order p, number of latent variables $\mathrm{r}$, an information theoretic criterion (ITC). Initialization: Evaluate ${\widehat{\mathbf{R}}}_i$ for $i=\overline{0,p}$; (see (11) and the discussion below it) $\widehat{\mathbf{R}}\leftarrow [{\widehat{\mathbf{R}}}_0\dots {\widehat{\mathbf{R}}}_p]$; ${\displaystyle {\widehat{\mathsf{\Phi }}}_m(\omega )\leftarrow \sum _{i=-p}^p{\widehat{\mathbf{R}}}_i{e}^{-j\omega i}}$; ${\displaystyle {\left\{{\check{\mathbf{Q}}}_i^{(0)}(1:\mathrm{K},1:\mathrm{K})\right\}}_{i=0}^p\leftarrow {\mathrm{ME}}_{\mathrm{I}}(\widehat{\mathbf{R}})}$ (see (22)); Compute ${\check{\mathbf{{\rm Y}}}}_{\ell m}^{(0)}(\omega )$ from EIG of ${\check{\mathbf{Q}}}_0^{(0)}$; for all $\lambda \in \{{\lambda }_1,\dots ,{\lambda }_L\}$ do  Maximum Entropy Expectation-Maximization (penalized setting):  for $\mathrm{it}=1,\dots ,N_{\mathrm{it}}$ do   Use ${\widehat{\mathsf{\Phi }}}_m(\omega )$ and ${\check{\mathbf{{\rm Y}}}}_{\ell m}^{(\mathrm{it}-1)}(\omega )$ to compute ${\check{\mathbf{C}}}^{(\mathrm{it})}$ (see (16)–(18));   ${\displaystyle {\left\{{\check{\mathbf{Q}}}_i^{(\mathrm{it})}\right\}}_{i=0}^p\leftarrow {\mathrm{ME}}_{\mathrm{II}}({\check{\mathbf{C}}}^{(\mathrm{it})},\lambda )}$ (see (23));   Get ${\check{\mathbf{{\rm Y}}}}_{\ell m}^{(\mathrm{it})}(\omega )$ from ${\displaystyle {\left\{{\check{\mathbf{Q}}}_i^{(\mathrm{it})}\right\}}_{i=0}^p}$ (see (15));  end for  Use ${\displaystyle {\left\{{\check{\mathbf{Q}}}_i^{(N_{\mathrm{it}})}\right\}}_{i=0}^p}$ to compute ${\check{\mathsf{\Phi }}}_{\lambda }^{-1}(\omega )$;  Determine ${\mathrm{SP}}_{\lambda }$ (see (24));  if ADAPTIVE then   ${\check{\mathbf{{\rm Y}}}}_{\ell m}^{(0)}(\omega )\leftarrow {\check{\mathbf{{\rm Y}}}}_{\ell m}^{(N_{\mathrm{it}})}(\omega )$  end if  ${\widehat{\mathbf{{\rm Y}}}}_{\ell m}^{(0)}(\omega )\leftarrow {\check{\mathbf{{\rm Y}}}}_{\ell m}^{(N_{\mathrm{it}})}(\omega )$;  Maximum Entropy Expectation-Maximization (constrained setting):  for $\mathrm{it}=1,\dots ,N_{\mathrm{it}}$ do   Use ${\widehat{\mathsf{\Phi }}}_m(\omega )$ and ${\widehat{\mathbf{{\rm Y}}}}_{\ell m}^{(\mathrm{it}-1)}(\omega )$ to compute ${\widehat{\mathbf{C}}}^{(\mathrm{it})}$ (see (16)–(18));   ${\displaystyle {\left\{{\widehat{\mathbf{Q}}}_i^{(\mathrm{it})}\right\}}_{i=0}^p\leftarrow {\mathrm{ME}}_{\mathrm{III}}({\widehat{\mathbf{C}}}^{(\mathrm{it})},{\mathrm{SP}}_{\lambda })}$ (see (25));   Get ${\widehat{\mathbf{{\rm Y}}}}_{\ell m}^{(\mathrm{it})}(\omega )$ from ${\displaystyle {\left\{{\widehat{\mathbf{Q}}}_i^{(\mathrm{it})}\right\}}_{i=0}^p}$ (see (15));  end for  Use ${\displaystyle {\left\{{\widehat{\mathbf{Q}}}_i^{(N_{\mathrm{it}})}\right\}}_{i=0}^p}$ to compute ${\widehat{\mathsf{\Phi }}}_{\lambda }^{-1}(\omega )$;  Find the matrix coefficients of the $\mathrm{VAR}$-model by spectral factorization of ${\widehat{\mathsf{\Phi }}}_{\lambda }^{-1}(\omega )$ and compute $\mathrm{ITC}(\mathrm{Data};{\mathrm{SP}}_{\lambda })$. end for ${\displaystyle \widehat{\mathrm{SP}}\leftarrow \arg \underset{\lambda }{min}\mathrm{ITC}(\mathrm{Data};{\mathrm{SP}}_{\lambda })}$;

The initialization stage provides a first estimate for the ISDM, from which the Expectation-Maximization alternations can begin. An estimate for the left upper corner of ${\mathsf{\Phi }}^{-1}\left(\omega \right)$ is obtained by solving the classical Maximum Entropy problem for a $\mathrm{VAR}\left(p\right)$-model, using the sample covariances of the manifest variables. We present below the matrix formulation of this problem, which allows an easy implementation in CVX (Matlab-based modeling system for convex optimization) [14]. The mathematical derivation of the matrix formulation from the information theoretic formulation can be found in [6,9].

First Maximum Entropy Problem [${\mathrm{ME}}_{\mathrm{I}}\left(\widehat{\mathbf{R}}\right)$]:

$$\begin{array}{c}\hfill \begin{array}{cc}\underset{\mathbf{X}}{min}& \mathrm{tr}\left(\mathcal{T}\left(\widehat{\mathbf{R}}\right)\mathbf{X}\right)-\log det{\mathbf{X}}_{00}\hfill \\ \mathrm{s}.\mathrm{t}.& \mathbf{X}⪰0\hfill \end{array}\end{array}$$

(22)

The block Toeplitz operator $\mathcal{T}$ is defined in (12). The size of the positive semidefinite matrix variable $\mathbf{X}$ is $\mathrm{K}(p+1)\times \mathrm{K}(p+1)$. For all $i=\overline{0,p}$, the estimate ${\check{\mathbf{Q}}}_i^{\left(0\right)}(1:\mathrm{K},1:\mathrm{K})$ of the ISDM (15) is given by ${\mathcal{D}}_i\left(\mathbf{X}\right)$.

In order to compute an initial value for ${\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)$, we resort to the eigenvalue decomposition (EIG) of ${\check{\mathbf{Q}}}_0^{\left(0\right)}(1:\mathrm{K},1:\mathrm{K})$. More precisely, after arranging the eigenvalues of ${\check{\mathbf{Q}}}_0^{\left(0\right)}(1:\mathrm{K},1:\mathrm{K})$ in the decreasing order of their magnitudes, we have ${\check{\mathbf{Q}}}_0^{\left(0\right)}(1:\mathrm{K},1:\mathrm{K})=\mathbf{U}\mathbf{D}{\mathbf{U}}^{\top }$. Then, we set ${\check{\mathbf{Q}}}_0^{\left(0\right)}(\mathrm{K}+1:\mathrm{K}+r,1:\mathrm{K})={\mathbf{D}}^{1/2}(1:\mathrm{r},1:\mathrm{r}){\mathbf{U}}^{\top }(1:\mathrm{K},1:\mathrm{r})$ and ${\check{\mathbf{Q}}}_i^{\left(0\right)}(\mathrm{K}+1:\mathrm{K}+\mathrm{r},1:\mathrm{K})=0$ for $i=\overline{1,p}$.

When the covariances ${\mathbf{C}}_i$ are fixed in the maximization step of the Expectation-Maximization algorithm, the coefficients of the matrix polynomial that is the ISDM (15) are estimated from the solution of the following optimization problem:

Second Maximum Entropy Problem [${\mathrm{ME}}_{\mathrm{II}}(\mathbf{C},\lambda )$]:

$$\begin{array}{c}\hfill \begin{array}{cc}\underset{\mathbf{X}}{min}& \mathrm{tr}\left(\mathcal{T}\left(\mathbf{C}\right)\mathbf{X}\right)-\log det{\mathbf{X}}_{00}+\lambda f\left(\mathbf{X}\right)\hfill \\ \mathrm{s}.\mathrm{t}.& \mathbf{X}⪰0\hfill \\ & {\mathcal{D}}_0\left(\mathbf{X}\right)(\mathrm{K}+1:\mathrm{K}+\mathrm{r},\mathrm{K}+1:\mathrm{K}+\mathrm{r})=\mathbf{I}\hfill \\ & {\mathcal{D}}_i\left(\mathbf{X}\right)(\mathrm{K}+1:\mathrm{K}+\mathrm{r},\mathrm{K}+1:\mathrm{K}+\mathrm{r})=\mathbf{0},i=\overline{1,p}\hfill \end{array}\end{array}$$

(23)

Since now we work with the full model, the size of $\mathbf{X}$ is $(\mathrm{K}+\mathrm{r})(p+1)\times (\mathrm{K}+\mathrm{r})(p+1)$. The function $f(·)$ is the sparsity promoter defined in (14) and depends only on the entries of the block corresponding to the manifest variables. The equality constraints in (23) guarantee that the latent variables have variance one and they are independent, given the manifest variables, corresponding to the lower right block of (15).

The estimates ${\displaystyle {\left\{{\check{\mathbf{Q}}}_i^{\left(N_{\mathrm{it}}\right)}\right\}}_{i=0}^p}$ obtained after these iterations are further employed to compute ${\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)$ by using (15). If $\lambda$ is large enough, then ${\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)$ is expected to have a certain sparsity pattern, ${\mathrm{SP}}_{\lambda }$. Since the objective of (23) does not ensure exact sparsification and also because of the numerical calculations, the entries of ${\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)$ that belong to ${\mathrm{SP}}_{\lambda }$ are small, but not exactly zero. In order to turn them to zero, we apply a method similar to the one from [6] (Section 4.1.3). We firstly compute the maximum of partial spectral coherence (PSC),

$$\begin{array}{c}\hfill \underset{\omega \in (-\pi ,\pi ]}{max}\frac{\left|{\left[{\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)\right]}_{ab}\right|}{\sqrt{{\left[{\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)\right]}_{aa}{\left[{\check{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)\right]}_{bb}}},\end{array}$$

(24)

for all $a\ne b$ with $1\le a,b\le \mathrm{K}$. Then ${\mathrm{SP}}_{\lambda }$ comprises all the pairs $(a,b)$ for which the maximum PSC is not larger than a threshold $\mathrm{Th}$. The discussion on the selection of parameters $N_{\mathrm{it}}$ and $\mathrm{Th}$ is deferred to Section 5.

The regularized estimate of ISDM is further improved by solving a problem similar to (23), but with the additional constraint that the sparsity pattern of ISDM is ${\mathrm{SP}}_{\lambda }$, more precisely:

Third Maximum Entropy Problem [${\mathrm{ME}}_{\mathrm{III}}(\mathbf{C},\mathrm{SP})$]:

$$\begin{array}{c}\hfill \begin{array}{cc}\underset{\mathbf{X}}{min}& \mathrm{tr}\left(\mathcal{T}\left(\mathbf{C}\right)\mathbf{X}\right)-\log det{\mathbf{X}}_{00}\hfill \\ \mathrm{s}.\mathrm{t}.& \mathbf{X}⪰0\hfill \\ & {\mathcal{D}}_0\left(\mathbf{X}\right)(\mathrm{K}+1:\mathrm{K}+\mathrm{r},\mathrm{K}+1:\mathrm{K}+\mathrm{r})=\mathbf{I}\hfill \\ & {\mathcal{D}}_i\left(\mathbf{X}\right)(\mathrm{K}+1:\mathrm{K}+\mathrm{r},\mathrm{K}+1:\mathrm{K}+\mathrm{r})=\mathbf{0},i=\overline{1,p}\hfill \\ & {\mathcal{D}}_i\left(\mathbf{X}\right)(a,b)=0,i=\overline{0,p},\mathrm{if}\mathrm{SP}(a,b)=0\hfill \end{array}\end{array}$$

(25)

This step of the algorithm has a strong theoretical justification which stems from the fact that ${\widehat{\mathsf{\Phi }}}^{-1}\left(\omega \right)$ is the Maximum Entropy solution for a covariance extension problem (see [10] (Remark 2.1)). The number of iterations, $N_{\mathrm{it}}$, is the same as in the case of the first loop.

The spectral factorization of the positive matrix trigonometric polynomial ${\widehat{\mathsf{\Phi }}}_{\lambda }^{-1}\left(\omega \right)$ is computed by solving a semidefinite programming problem. The implementation is the same as in [8], except that in our case the model contains latent variables. Therefore, the matrix coefficients produced by spectral factorization are altered to keep only those entries that correspond to manifest variables. The resulting $\mathrm{VAR}$ model is fitted to the data and then various IT criteria are evaluated. The accuracy of the selected model depends on the criterion that is employed as well as on the strategy used for generating the $\lambda$-values that yield the competing models. In the next section, we list the model selection rules that we apply; the problem of generating the $\lambda$-values is treated in Section 5.

As already mentioned, the estimation problem is solved for several values of $\lambda$: ${\lambda }_1<{\lambda }_2<\cdots <{\lambda }_L$. From the description above we know that, for each value of the parameter $\lambda$, ${\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)$ gets the same initialization, which is based on (22). It is likely that this initialization is poor. A better approach is an ADAPTIVE algorithm which takes into consideration the fact that the difference ${\lambda }_i-{\lambda }_{i-1}$ is small for all $i=\overline{2,L}$. This algorithm initializes ${\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)$ as explained above only when $\lambda ={\lambda }_1$. When $\lambda ={\lambda }_i$ for $i=\overline{2,L}$, the initial value of ${\mathbf{{\rm Y}}}_{\ell m}\left(\omega \right)$ is taken to be the estimate of this quantity that was previously obtained by solving the optimization problem in (23) for $\lambda ={\lambda }_{i-1}$. The effect of the ADAPTIVE procedure will be investigated empirically in Section 5.

The newly proposed estimation method outlined in Algorithm 1 is dubbed AlgoEM. The Matlab code for AlgoEM can be downloaded from https://www.stat.auckland.ac.nz/~cgiu216/PUBLICATIONS.htm.

4. Model Selection

4.1. IT Criteria

It is well-known that the IT criteria can be expressed as the sum of a goodness-of-fit (GOF) term and a penalty. They are derived on various grounds, but their expressions cannot be obtained easily for the problem we investigate. Due to this reason, we resort to the methodology applied previously to VAR without latent variables, where the criteria originally proposed for model order selection have been modified such that to be employed for finding the best sparsity pattern. As the GOF term is obtained straightforwardly by fitting the model to the data, the difficult part is the alteration of the penalty term. Based on the observation that all the penalty terms of the criteria for model order selection involve the number of parameters of the model, reference [6] proposed to replace it with the effective number of parameters:

$$\begin{array}{c}\hfill N_{\mathrm{ef}}=\frac{\mathrm{K}(\mathrm{K}+1)}{2}-N_0+p({\mathrm{K}}^2-2N_0).\end{array}$$

(26)

Note that $N_0$ is the number of zeros in the lower triangular part of ${\mathrm{SP}}_{\lambda }$. The expression above can be obtained straightforwardly by counting the number of non-zero entries in the $\mathrm{K}\times \mathrm{K}$ upper-left block of the matrices ${\displaystyle {\left\{{\widehat{\mathbf{Q}}}_i^{\left(N_{\mathrm{it}}\right)}\right\}}_{i=0}^p}$ produced by Algorithm 1; counting takes into consideration all the existing symmetries.

The formula in (26) was used in [6] in order to modify three celebrated criteria: Schwarz Bayesian Criterion—$\mathrm{SBC}$ [15], Akaike Information Criterion—$\mathrm{AIC}$ [16] and its corrected version—${\mathrm{AIC}}_{\mathrm{c}}$ [13] (pp. 432) and [17]. We name $\mathrm{SBC}$ the criterion from [15] because this is the term used in time series literature; the same selection rule is called Bayesian Information Criterion ($\mathrm{BIC}$) in other works. Based on the empirical evidence from [6], $\mathrm{SBC}$ is ranked best when the sample size is large, whereas ${\mathrm{AIC}}_{\mathrm{c}}$ works better for small sample sizes. This makes us to employ these two criteria in our experiments. Their expressions are [6]:

$$\begin{array}{ccc}\hfill \mathrm{SBC}(\mathrm{Data};\mathrm{SP})& =& T\log det\widehat{\mathsf{\Sigma }}+N_{\mathrm{ef}}\log T,\hfill \\ \hfill {\mathrm{AIC}}_{\mathrm{c}}(\mathrm{Data};\mathrm{SP})& =& T\log det\widehat{\mathsf{\Sigma }}+\frac{2N_{\mathrm{ef}}T}{T-N_{\mathrm{ef}}-1},\hfill \end{array}$$

(27)

where $\widehat{\mathsf{\Sigma }}$ is the error covariance matrix. For simplicity, we write $\mathrm{SP}$ instead of ${\mathrm{SP}}_{\lambda }$. Even if our notation does not emphasize on this fact, it is clear that both $N_{\mathrm{ef}}$ and $\widehat{\mathsf{\Sigma }}$ depend on $\lambda$.

Another criterion employed in our tests is a variant of the Final Prediction Error—FPE [18]. The formula we use was obtained in [8] by relying on the asymptotic equivalence between $\log \mathrm{FPE}$ and $\mathrm{AIC}$. For ease of comparison with other criteria, we do not give the expression of $\mathrm{FPE}$, but that of $\log \mathrm{FPE}$:

$$\begin{array}{ccc}\hfill \log \mathrm{FPE}(\mathrm{Data};\mathrm{SP})& =& \log det\widehat{\mathsf{\Sigma }}+\mathrm{K}\log \frac{T+\eta }{T-\eta },\hfill \end{array}$$

where $\eta =N_{\mathrm{ef}}/\mathrm{K}$.

We also apply the Renormalized Maximum Likelihood ($\mathrm{RNML}$) criterion. Its derivation is related to the problem of encoding losslessly measurements assumed to be a sequence of outcomes from an unknown distribution. It has been proven in [19] that there is a unique distribution which, if it is used to encode these measurements, then it leads to the minimum code length in the worst case scenario. This particular distribution was further utilized in [20,21] in order to introduce the $\mathrm{RNML}$ criterion for model selection. The major problem comes from the fact that, for most of the family of models, its closed-form expression is hard to be obtained. The expression of $\mathrm{RNML}$ that can be employed for choosing the order of $\mathrm{VAR}$-models was firstly derived in [22]. The properties of this criterion have been investigated in [8], where the criterion was also altered such that to be applied in the selection of the sparsity pattern for ISDM of $\mathrm{VAR}$-models. With our notation, the $\mathrm{RNML}$ criterion can be written as follows:

$$\mathrm{RNML}(\mathrm{Data};\mathrm{SP})=\frac{T-\eta -\mathrm{K}+1}{2}\log det\widehat{\mathsf{\Sigma }}+\frac{N_{\mathrm{ef}}}{2}\log \mathrm{tr}\left({\widehat{\mathbf{R}}}_0-\widehat{\mathsf{\Sigma }}\right)-\log {\mathsf{\Gamma }}_{\mathrm{K}}\left(\frac{T-\eta }{2}\right)-\log \mathsf{\Gamma }\left(\frac{N_{\mathrm{ef}}}{2}\right),$$

(28)

where ${\mathsf{\Gamma }}_{\mathrm{K}}(·)$ is the multivariate Gamma function and $\mathsf{\Gamma }(·)$ is the Gamma function. The matrix ${\widehat{\mathbf{R}}}_0$ has the same significance as in (12). It is remarkable that the penalty term of $\mathrm{RNML}$ depends on the actual measurements, and not only on the triple $(T,\mathrm{K},N_{\mathrm{ef}})$.

A common feature of all the criteria presented so far is that they do not take into consideration how big is the family of the competing candidates. This might be problematic because the total number of possible sparsity patterns is as large as $2^{\overline{K}}$, where $\overline{K}=\mathrm{K}(\mathrm{K}-1)/2$. The solutions proposed in the previous literature for circumventing this difficulty are called extended IT criteria. We show below how these criteria can be applied for selecting the sparsity pattern. In this context, we also propose a novel variant of $\mathrm{RNML}$.

4.2. Extended IT Criteria

First we write down the expression of the extended $\mathrm{SBC}$ proposed in [23]. In the statistical literature, this criterion is named $\mathrm{EBIC}$ (Extended Bayesian Information Criterion):

$$\begin{array}{c}\hfill \mathrm{EBIC}(\mathrm{Data};\mathrm{SP})=\mathrm{SBC}(\mathrm{Data};\mathrm{SP})+2\gamma \log \left(\genfrac{}{}{0pt}{}{\overline{K}}{N_0}\right),\end{array}$$

(29)

where $\gamma \in [0,1]$. It is evident that $\mathrm{EBIC}$ is equivalent to $\mathrm{SBC}$ when $\gamma =0$. More interestingly, reference [24] uses arguments from information theory for justifying the use of a penalty term which counts the number of models that have the same number of parameters. For our problem, this is equivalent to choosing $\gamma =1$ in (29). Because this is also the value of $\gamma$ that we use in this work for evaluating $\mathrm{EBIC}$, we explain briefly the significance of the supplementary penalty term. The key point is to consider a scenario in which $\mathrm{Data}$ should be transmitted losslessly from an encoder to a decoder by employing the model given by $\mathrm{SP}$. According to [24], the first step is to transmit the value of $N_0$. The assumption that all possible values of $N_0$ are equally probable leads to the conclusion that the code length for $N_0$ is $-\log \left(1/(\overline{K}+1)\right)=\log (\overline{K}+1)$. As this quantity is the same for all models, it can be neglected. Then the decoder should be informed about the actual locations of the zeros in the sparsity pattern $\mathrm{SP}$. Since the list of all sparsity patterns for a given $N_0$ is known by both the encoder and the decoder, all that remains is to send to the decoder the index of $\mathrm{SP}$ in this list. Under the hypothesis that all the sparsity patterns in the list are equally probable, the code length for the index is ${\displaystyle \log \left(\genfrac{}{}{0pt}{}{\overline{K}}{N_0}\right)}$. In (29), this quantity is multiplied by two because of the scaling factor used in the definition of $\mathrm{SBC}(\mathrm{Data};\mathrm{SP})$.

Another formulation of $\mathrm{EBIC}$ was introduced in [25] for finding the graphical structure of a Gaussian model (static case), in the situation when the number of variables and the number of observations grow simultaneously. After modifying the criterion from [25] by replacing the number of edges of the graph with $N_{\mathrm{ef}}$, we get the following formula:

$$\begin{array}{c}\hfill {\mathrm{EBIC}}_{\mathrm{FD}}(\mathrm{Data};\mathrm{SP})=\mathrm{SBC}(\mathrm{Data};\mathrm{SP})+4\gamma N_{\mathrm{ef}}\log \mathrm{K},\end{array}$$

(30)

where $\gamma$ has the same significance as in (29), and again we take $\gamma =1$. Remark that the term $4\gamma N_{\mathrm{ef}}\log \mathrm{K}$ grows when $N_0$ decreases; the term in (29), ${\displaystyle 2\gamma \log \left(\genfrac{}{}{0pt}{}{\overline{K}}{N_0}\right)=2\gamma \log \left(\genfrac{}{}{0pt}{}{\overline{K}}{\overline{K}-N_0}\right)}$, does not have the same property.

Relying on the asymptotic equivalence between $\mathrm{RNML}$ and $\mathrm{SBC}$ [8], we alter $\mathrm{RNML}$ by adding half of the extra penalty from (30); the scaling factor is needed because the ratio between the GOF term in (28) and the GOF term in (27) tends to $1/2$ when $T\to \infty$. The new criterion, which is dubbed ${\mathrm{RNML}}_{\mathrm{FD}}$, has the following expression:

$$\begin{array}{c}\hfill {\mathrm{RNML}}_{\mathrm{FD}}(\mathrm{Data};\mathrm{SP})=\mathrm{RNML}(\mathrm{Data};\mathrm{SP})+2N_{\mathrm{ef}}\log \mathrm{K}.\end{array}$$

(31)

For all the selection rules listed above, the best model is the one which minimizes the value of the criterion. For evaluating the performance of IT criteria, we conduct an empirical study. The main results of this study are reported in the next section.

5. Experimental Results

5.1. Artificial Data

In our simulations, the order of the VAR model in (1) is taken to be one, the number of manifest variables is $\mathrm{K}=15$, and there is one single latent variable ($\mathrm{r}=1$). Let $\mathrm{KS}$ denote the number of non-zero entries in the lower triangular part of the sparsity pattern of size $\mathrm{K}\times \mathrm{K}$. We consider three different values for $\mathrm{KS}$: 2, 3 and $\mathrm{K}$. Remark that the larger is $\mathrm{KS}$, less sparse is the ISDM. The locations of the non-zero entries for each value of $\mathrm{KS}$ are graphically represented in Figure 1.

When generating the ISDM, all the matrices ${\displaystyle {\left\{{\mathbf{Q}}_i\right\}}_{i=0}^p}$ in (2) have only ones on their main diagonals. The entries of the $\mathrm{K}\times \mathrm{K}$ upper-left block of the matrix ${\mathbf{Q}}_i$, which should be non-zero according to Figure 1, are equal to $0.5/(i+1)$. Additionally, the entries on the last row and on the last column of ${\mathbf{Q}}_i$, except the one on the main diagonal, are equal to $0.3/(i+1)$. Integer multiples of the $\kappa \times \kappa$ identity matrix are added to ${\mathbf{Q}}_0$ until the resulting ISDM is positive definite. Furthermore, the spectral factorization is applied in order to obtain the matrix coefficients of the $\mathrm{VAR}$-model from the ISDM (see [8] (Section 4.2) for more details). Hence, for each value of $\mathrm{KS}$, one single $\mathrm{VAR}$-model is produced and this is used for generating $N_{\mathrm{tr}}=10$ $\kappa$-variate time series of length $T=$ 50,000. To this end, we utilize Matlab R2014b functions from the package available at the address http://climate-dynamics.org/software/#arfit. After discarding from each time series the component corresponding to the latent variable, the simulated data are used for evaluating the performance of AlgoEM.

5.2. Settings for AlgoEM

The order of $\mathrm{VAR}$, as well as the number of latent variables, is assumed to be known. The parameter $\lambda$ takes values on a regular grid defined on the interval $[{10}^{-3},{10}^{-1}]$, for which the grid step is ${10}^{-3}$. It follows that the total number of values for $\lambda$ is $L=100$. The threshold $\mathrm{Th}$, which is used in conjunction with (24) in order to get the estimated sparsity pattern, equals ${10}^{-3}$.

We are interested to evaluate the impact of the adaptive initialization procedure that was introduced in Section 3. This is why we run AlgoEM with and without this procedure, for all the time series we have generated. For each time series and for each value of $\lambda$ on the grid, the estimated ${\mathrm{SP}}_{\lambda }$ is compared to the true sparsity pattern. The comparison reduces to computing the distance between the two sparsity patterns, which is given by the number of positions below the main diagonal where the patterns differ. For the case $\mathrm{KS}=15$, statistics related to this distance are presented in Figure 2. Remark that values of $\lambda$ close to zero lead to estimated patterns which are not sparse. As expected, this happens disregarding if the adaptive procedure is applied or not. The use of the procedure has the positive effect that, for a large range of $\lambda$-values, the estimated patterns are close to the true one. The same is true for both $\mathrm{KS}=2$ and $\mathrm{KS}=3$, which makes us apply the adaptive procedure in all the experiments outlined below.

The results reported in Figure 2 are obtained by taking the number of iterations to be $N_{\mathrm{it}}=4$. As the computational burden of the algorithm depends strongly on $N_{\mathrm{it}}$, we investigate the effect of reducing the number of iterations to $N_{\mathrm{it}}=3$ and $N_{\mathrm{it}}=2$, respectively. In each case, the evaluation of performance is done by an oracle having complete knowledge about the true sparsity pattern. From the set of sparsity patterns produced when applying AlgoEM to a particular time series, the oracle selects the one which is closest to the true sparsity pattern. The closeness is measured by the distance defined above. The average distances computed from $N_{\mathrm{tr}}=10$ trials are plotted in Figure 3. We can see in the figure that, in the case when $\mathrm{KS}=2$ and AlgoEM performs only two iterations, the true sparsity pattern is always in the set of the candidates produced by the algorithm and this makes the average distance to be zero. In general, all the results shown in the figure are good as the average distance is smaller than one in all cases. Since the increase of $N_{\mathrm{it}}$ does not guarantee the improvement in performance, we take $N_{\mathrm{it}}=2$ for reducing the complexity of the algorithm.

It is clear from the description of Algorithm 1 that $N_{\mathrm{it}}$ is the same for the two major loops of AlgoEM. We name the first loop MEEM(Pen) and the second one MEEM(Con). Obviously, MEEM is the acronym for Maximum Entropy Expectation-Maximization, Pen stands for penalized settings, and Con means constrained settings. Because we want to quantify the influence of MEEM(Con) on the accuracy of the estimation, we show in Figure 4 the average distances to the true pattern, when the estimates are produced by MEEM(Pen) and by MEEM(Con), respectively. This time we do not report results obtained only when an oracle is used for selecting the sparsity pattern, but also for the case when the selection is done with the IT criteria defined in Section 4. We can observe that ${\mathrm{AIC}}_{\mathrm{c}}$ and $\mathrm{FPE}$ perform very poorly, whereas both ${\mathrm{EBIC}}_{\mathrm{FD}}$ and ${\mathrm{RNML}}_{\mathrm{FD}}$ are very good. The fact that ${\mathrm{RNML}}_{\mathrm{FD}}$ is superior to $\mathrm{RNML}$ demonstrates the importance of the extra-term in (31). Remark also that ${\mathrm{EBIC}}_{\mathrm{FD}}$ is more accurate than $\mathrm{SBC}$, while $\mathrm{EBIC}$ has the same level of performance as $\mathrm{SBC}$. The most important conclusion is that removing MEEM(Con) from AlgoEM does not deteriorate the final outcome.

The next step is to compare AlgoEM with the algorithm that solves the optimization problem in (13). We use the implementation from [12], which is publicly available at the address https://drive.google.com/file/d/0BykD2O6uX6KjSGlkQTRYWFVBZGM/view, and we call it AlgoSL. The name comes from the fact that the method in (13) is sparse plus low-rank.

5.3. Comparison of AlgoEM and AlgoSL

In [12], the sparsity pattern for a given pair of parameters $(\lambda ,\gamma )$ is found by solving (13). Reference [10] uses further this sparsity pattern as an initialization for a constrained optimization problem (see also the discussion below Equation (25)). In both cases, a set of sparsity patterns is generated by choosing various values for the parameters $\lambda$ and $\gamma$. For selecting the best one, they consider an alternative to IT criteria, which is dubbed score function ($\mathrm{SF}$). The key point is that, when using $\mathrm{SF}$, it is not needed to compute explicitly the matrix coefficients of the VAR-model. More details are provided below.

Let ${\widehat{\mathsf{\Phi }}}_m$ be an estimate of ${\mathsf{\Phi }}_m$ in (4), which is constrained to have a certain sparsity pattern, $\mathrm{SP}$. Using an idea from [9], reference [10] suggests to employ $\mathrm{Data}$ for computing the correlogram ${\widehat{\mathsf{\Phi }}}_m^c$ (with Bartlett window) [26], and then to evaluate the relative entropy rate:

$$\begin{array}{c}\hfill \mathbb{D}({\widehat{\mathsf{\Phi }}}_m^c\left|\right|{\widehat{\mathsf{\Phi }}}_m)=-\frac{1}{4\pi }{\int }_{-\pi }^{\pi }\log det\mathsf{\Omega }\left(\omega \right)+\mathrm{tr}(\mathbf{I}-\mathsf{\Omega }\left(\omega \right))\mathrm{d}\omega ,\end{array}$$

(32)

where $\mathsf{\Omega }\left(\omega \right)={\widehat{\mathsf{\Phi }}}_m^c\left(\omega \right){\widehat{\mathsf{\Phi }}}_m^{-1}\left(\omega \right)$ for all $\omega \in (-\pi ,\pi ]$. This formula can be regarded as a generalization of the I-divergence of two positive definite matrices, which was originally introduced in connection with graphical models (static case) [3]. The expression in (32) was also employed by [27] in the inference of graphical models for time series. Some of the properties of the relative entropy rate are discussed in [28]. It is worth noting that this index also belongs to the Tau divergence family [29] and to the Beta divergence family [30]. It has been proven in these references that the use of (32) in the formulation of the Maximum Entropy problem leads to the simplest solution, in the sense of the minimum McMillan degree.

The most important is that, in [10], $\mathbb{D}\left({\widehat{\mathsf{\Phi }}}_m^c\right|\left|{\widehat{\mathsf{\Phi }}}_m\right)$ is utilized for quantifying the adherence of the model to the data. The complexity of the model is determined by $N_{\mathrm{e}}$, the total number of edges in the graph (including the edges that connect the latent variables to the manifest variables). For instance, if the model has a single latent variable, then $N_{\mathrm{e}}=\mathrm{K}(\mathrm{K}+1)/2-N_0$. Note that $N_0$ has the same significance as in (26). It follows that $N_{\mathrm{ef}}=N_{\mathrm{e}}+p({\mathrm{K}}^2-2N_0)$, which leads to the conclusion that $N_{\mathrm{ef}}>N_{\mathrm{e}}$, and the difference between the two quantities increases when the order of the model raises. If $p=1$ and the number of latent variables is at least three ($r\ge 3$), then $N_{\mathrm{ef}}<N_{\mathrm{e}}$ when $N_0$ is large.

The score functions given in [10] are:

$$\begin{array}{ccc}\hfill \log {\mathrm{SF}}_1(\mathrm{Data};\mathrm{SP})& =& \log \mathbb{D}({\widehat{\mathsf{\Phi }}}_m^c\left|\right|{\widehat{\mathsf{\Phi }}}_m)+\log N_{\mathrm{e}},\hfill \end{array}$$

(33)

$$\begin{array}{ccc}\hfill {\mathrm{SF}}_2(\mathrm{Data};\mathrm{SP})& =& \mathbb{D}({\widehat{\mathsf{\Phi }}}_m^c\left|\right|{\widehat{\mathsf{\Phi }}}_m)+\frac{N_{\mathrm{e}}}{T},\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill {\mathrm{SF}}_3(\mathrm{Data};\mathrm{SP})& =& \mathbb{D}({\widehat{\mathsf{\Phi }}}_m^c\left|\right|{\widehat{\mathsf{\Phi }}}_m)+\frac{N_{\mathrm{e}}\log T}{T},\hfill \end{array}$$

(35)

where $\mathbb{D}(·||·)$ is defined in (32). The formula of ${\mathrm{SF}}_1$ is logged for ease of reading.

We apply AlgoSL to all the time series we have simulated (see again Section 5.1). In our experiments, we consider the pairs $(\lambda ,\gamma )$ for which $\lambda \in \left\{0.1,0.2,\dots ,0.6\right\}$ and $\gamma \in \left\{0.01,0.02,\dots ,0.5\right\}$. For the selection of the sparsity pattern, we do not use only the score functions in (33)–(35), but we also employ the IT criteria from Section 4. The results are shown in Figure 5. The best criterion is ${\mathrm{RNML}}_{\mathrm{FD}}$, which is able to find the true sparsity pattern in all the experiments; the second best is ${\mathrm{EBIC}}_{\mathrm{FD}}$. The comparison of the plots in Figure 5 to those in Figure 4 leads to the conclusion that AlgoSL works better than AlgoEM when ${\mathrm{RNML}}_{\mathrm{FD}}$ is employed for selecting the model. For understanding these results, we should take into consideration two important aspects: (i) Oracle gives perfect results for all $\mathrm{KS}$ in the case of AlgoSL, but not in the case of AlgoEM; to some extent this is due to the fact that 300 $(\lambda ,\gamma )$-pairs allow to produce a better set of candidates for AlgoSL than the one generated by 100 $\lambda$-values for AlgoEM; (ii) The score functions have been used in [10] for time series of hundreds of samples, whereas the size of the time series we simulated is much larger ($T=$ 50,000).

In order to clarify the second aspect, we conduct an experiment with simulated time series for which $T=500$. The simulation procedure is the same as in Section 5.1 and all the settings are the same, except that the non-zero entries of the matrices ${\displaystyle {\left\{{\mathbf{Q}}_i\right\}}_{i=0}^p}$ (which are not located on the main diagonal) have values that are fifty times larger. For AlgoEM, the uniform grid for $\lambda$ takes values on the interval $[{10}^{-3},{10}^{-1}]$; the grid step is ${10}^{-3}$ (see also [31]). Based on some empirical evidence, we use the parameters $\lambda \in \left\{0.02,0.03,\dots ,0.3\right\}$ and $\lambda \gamma \in \left\{0.01,0.015,\dots ,0.2\right\}$ for AlgoSL. All IT criteria and all the score functions are used for both AlgoEM and AlgoSL, and the estimation results are reported in Figure 6. In general, they are worse than the results for large sample size ($T=$ 50,000), and the difference is more evident when $\mathrm{KS}=15$. This shows that, for small T, it might be difficult to recover the true structure of ISDM when it is not sparse enough. It is encouraging that, even for $\mathrm{KS}=15$, oracle used in conjunction with AlgoSL finds the true pattern in all trials.

To gain more insight, we give in Figure 7 a graphical representation of all the distances which are computed by oracle for a time series whose ISDM has $\mathrm{KS}=15$. It follows from the way in which we have chosen the experimental parameters that the total number of such distances calculated for AlgoSL is more than eleven times larger than the number of distances for AlgoEM. We should keep in mind that we need to compute the distance from the pattern estimated by each candidate in the list to the true sparsity pattern. After a laborious process of selecting the values of $\lambda$ and $\lambda \gamma$, we ended-up with a two-dimensional grid which yields a good number of estimated patterns that are identical to the true pattern (see the white area inside the square shown in Figure 7b). The behavior of AlgoEM is different: For a large range of $\lambda$-values, the $\mathrm{K}\times \mathrm{K}$ upper-left block of the estimated pattern is equal to the identity matrix, which makes the distance to the true pattern to be equal to $\mathrm{KS}$. This happens when the number of latent variables used in AlgoEM equals the true number ($r=1$) as well as when $r=2$ is utilized in estimation. However, it is evident in Figure 7a that the estimation results are better when $r=1$. If the number of latent variables is not known a priori, one possibility might be to use an $\mathrm{ITC}$ for selecting it. Note that the definition of $N_{\mathrm{ef}}$ in (26) should be modified. The score functions given in (33)–(35) do not need to be altered in order to be employed in selection of the number of latent variables.

5.4. Real-World Data

In addition to the experiments with simulated data, we test the capabilities of AlgoEM on multivariate time series of daily stock markets indices at closing time, which can be downloaded from the following address: http://au.mathworks.com/matlabcentral/fileexchange/48611-international-daily-stock-return-data-for-system-identification. This dataset was produced by pre-procesing the original data from http://finance.yahoo.com, which have been measured from 4 January 2012 to 31 December 2013. We refer to [10] for details regarding the pre-processing. The time series is $\mathrm{K}$-variate with $\mathrm{K}=22$, and the sample size is $T=518$.

For each component of the time series, we give the name of the country where was measured and we write in parentheses the acronym used in this analysis: Australia (AU), New Zealand (NZ), Singapore (SG), Hong Kong (HK), China (CH), Japan (JA), Korea (KO), Taiwan (TA), Brazil (BR), Mexico (ME), Argentina (AR), Switzerland (SW), Greece (GR), Belgium (BE), Austria (AS), Germany (GE), France (FR), Netherlands (NL), United Kingdom (UK), United States (US), Canada (CA), Malaysia (MA). The official names of the price indices can be found in [10].

Similar to [10], we take the order of $\mathrm{VAR}$-model to be $p=1$, and we assume that there is a single latent variable ($\mathrm{r}=1$). For AlgoEM, we have $N_{\mathrm{it}}=4$ and $\lambda$ takes values on a uniform grid on the interval $[2\times {10}^{-3},2\times {10}^{-1}]$, for which the grid step is $2\times {10}^{-3}$. When ${\mathrm{AIC}}_{\mathrm{c}}$ is used for choosing the model, the results are disappointing because the selected sparsity pattern contains very few zeros. The same type of outcome is also obtained when applying ${\mathrm{SF}}_2$ or ${\mathrm{SF}}_3$. It is interesting that ${\mathrm{SF}}_1$ as well as six different IT criteria ($\mathrm{SBC}$, $\mathrm{EBIC}$, ${\mathrm{EBIC}}_{\mathrm{FD}}$, $\mathrm{FPE}$, $\mathrm{RNML}$, ${\mathrm{RNML}}_{\mathrm{FD}}$) select exactly the same sparsity pattern. This one is compared in Figure 8 to the sparsity pattern given in [10], for the same data set. Observe that the conditional independence graph yielded by AlgoEM has only four edges which connect manifest variables: Three of them connect vertices corresponding to Asian markets, $\left(\mathrm{HK},\mathrm{CH}\right),\left(\mathrm{HK},\mathrm{JA}\right),\left(\mathrm{JA},\mathrm{KO}\right)$, and the fourth one connects two European markets, $\left(\mathrm{GR},\mathrm{AT}\right)$. This is different from the conditional graph in [10], where all the edges between manifest variables connect only the European markets (with the exception of Greece). An in-depth analysis of the two graphs is beyond the interest of this paper.

6. Final Remarks

The main motivation for this study is the solution proposed in [5] for the static case, as an alternative to the method from [4]. We have shown how the estimation method from [5] can be generalized for the dynamic case. The resulting algorithm is dubbed AlgoEM. We have conducted an empirical study in which we have investigated the capabilities of AlgoEM and we have also compared it with the generalization of the method from [4], which was proposed in [10]. It is important to emphasize that the two methods for latent-variable autoregressive models, which we have compared, have some common features: apply the Maximum Entropy principle, use convex optimization, and generate a set of candidate models from which the best one is selected by a certain rule. In the case of AlgoEM, the set of the candidates depends strongly on the parameter $\lambda$, which is chosen by the user. For the method in [10], the user should choose two parameters, $\lambda$ and $\gamma$. Based on our experience, the selection of the two parameters is much more difficult than the selection of the single parameter for AlgoEM. Another important aspect is how to pick-up the winner from the competing models. In [10], this was restricted to the use of score functions. We have demonstrated empirically that the IT criteria might be an option to consider. Especially when the sample size is large, it is recommended to employ the criterion ${\mathrm{RNML}}_{\mathrm{FD}}$ which we have introduced in this work.