Estimating the Variance of Estimator of the Latent Factor Linear Mixed Model Using Supplemented Expectation-Maximization Algorithm

Abstract

This paper deals with symmetrical data that can be modelled based on Gaussian distribution, such as linear mixed models for longitudinal data. The latent factor linear mixed model (LFLMM) is a method generally used for analysing changes in high-dimensional longitudinal data. It is usual that the model estimates are based on the expectation-maximization (EM) algorithm, but unfortunately, the algorithm does not produce the standard errors of the regression coefficients, which then hampers testing procedures. To fill in the gap, the Supplemented EM (SEM) algorithm for the case of fixed variables is proposed in this paper. The computational aspects of the SEM algorithm have been investigated by means of simulation. We also calculate the variance matrix of beta using the second moment as a benchmark to compare with the asymptotic variance matrix of beta of SEM. Both the second moment and SEM produce symmetrical results, the variance estimates of beta are getting smaller when number of subjects in the simulation increases. In addition, the practical usefulness of this work was illustrated using real data on political attitudes and behaviour in Flanders-Belgium.

1. Introduction

The latent factor multivariate linear mixed model (LFLMM) is a combination between the Factor Analysis (FA) and the Linear Mixed Model (LMM), as proposed by [1]. The model aims to analyze longitudinal data sets with large numbers of multivariate responses, i.e., high-dimensional longitudinal data. The authors proposed estimation of the LFLMM by means of the EM algorithm, which is a closed-form solution. They showed by way of simulation that EM estimation of the LFLMM provides accurate parameter estimates and is more efficient in terms of adding variables other than time variables in the model than alternatives like the structural equation model. As shown by [2,3], the combination of fixed and random effects and the interaction of covariates with time can be straightforwardly handled by the LFLMM estimated by EM.

The LFLMM assumes that the responses are continuous and that the number of latent variables is known [1]. Moreover, convergence of the EM algorithm is sometimes slow. The main disadvantage, however, is that the EM algorithm does not produce standard errors of the estimator of the regression coefficients because it does not calculate the derivatives of the likelihood function, which are often complicated and tedious to derive [4,5]. Thus, it is difficult to study the effects of different covariates or fixed variables for different latent factors simultaneously.

The general Supplemented EM algorithm was proposed by [6] to obtain the standard errors by calculating the complete information matrix as the base of the variance-covariance matrix of the estimator. The Supplemented EM algorithm has been applied to various kinds of models, notably item response models [6,7,8,9]. However, its suitability and features in the case of application to the LFLMM have not been investigated yet. In this study, we extend the work of [1] by employing the Supplemented EM algorithm as a by-product of the EM estimator for the case of fixed variables. We used simulation studies to investigate the computational aspects of the Supplemented EM algorithm and used a real data example to illustrate the practical usefulness of this work.

The remainder of this study is organized as follows. In Section 2, we specify the LFLMM and summarize the EM algorithm to estimate it. Section 3 presents the Supplemented EM algorithm. Section 4 and Section 5 present the results of the simulation and real data example. Conclusions follow in Section 6.

2. The LFLMM and the EM Algorithm

Following [1], an LFLMM can be composed of two parts. The first is the factor analysis model, which represents the relationships between the observed and latent variables. This is similar to the structural equation model, which explains the relationship of the latent variables and the measurement indicators carried out through factor analysis [10]. This part can be written as:

$${\mathit{Y}}_{it}=\Lambda {\mathit{\eta }}_{it}+{\mathit{ϵ}}_{it}$$

(1)

Specifically, for the i-th of $N \mathrm{individuals}$, we observe j = 1, …, $J$ responses which characterize $d$ latent factors $\left({\mathit{\eta }}_{it}=\left({\mathit{\eta }}_{it}^1,\dots ,{\mathit{\eta }}_{it}^d\right), d<J\right)$ at time $t$, $t=1,\dots ,T_i$, where $T_i$ is the number of time periods for subject $i$. $\Lambda$ is the matrix of factor loadings and ${\mathit{ϵ}}_{it}=\left({\mathit{ϵ}}_{it1},\dots , {\mathit{ϵ}}_{itJ}\right)$ the vector of measurement errors for subject $i$ at time $t$. It is assumed that ${\mathit{ϵ}}_{itJ}~N\left(0,{\mathit{\tau }}_j^2\right)$ and ${\mathit{ϵ}}_{itj}\perp {\mathit{ϵ}}_{ith}$, $j\ne h$. In matrix notation, Equation (1) reads:

$${\mathit{Y}}_i=\left({\mathit{I}}_{T_i}⨂\Lambda \right){\mathit{\eta }}_i+{\mathit{ϵ}}_i$$

(2)

where

$$\begin{gathered} {\mathit{Y}}_i=\left(y_{i1}^{\prime }, \dots , y_{i{T}_i}^{\prime }\right){\prime }_{\left[J\times T_i,1\right]} \\ {\mathit{\eta }}_i=\left({\mathit{\eta }}_{i1}^{\prime }, \dots , {\mathit{\eta }}_{i{T}_i}^{\prime }\right){\prime }_{\left[d\times T_i,1\right]} \\ {\Lambda }_{\left[J\times d\right]}=\left(\begin{array}{c}{\lambda }_1^{\prime }\\ ⋮\\ {\lambda }_J^{\prime }\end{array}\right) \end{gathered}$$

The second part of the LFLMM is a multivariate linear mixed model containing the fixed and random effects for each latent variable (${\mathit{\eta }}_{it})$. For individual $i$, $i=1,2, \dots N,$ at time $t$, $t=1, 2, \dots , T_i$, and latent variable $l, l$ $=1, 2, \dots ,d$, we thus have:

$${\mathit{\eta }}_{it}^l=x_{it}^l{\mathit{\beta }}^l+z_{it}^l{\mathit{a}}_i^l+{\mathit{\epsilon }}_{it}^l$$

(3)

where $x_{it}^l$ and $z_{it}^l$ are the elements of design matrices of the $p \mathrm{fixed}$variables and $q$ random effects, respectively. ${\mathit{\beta }}^l$ is an $\mathrm{unknown} \mathrm{coefficient}$, ${\mathit{a}}_i^l=\left({\mathit{a}}_{i1}^l,\dots , {\mathit{a}}_{iq}^l\right)$ and ${\mathit{\epsilon }}_i^l=\left({\mathit{\epsilon }}_{i1}^l,\dots , {\mathit{\epsilon }}_{iTi}^l\right)$, $l=1,2, \dots d,$ are the random effects and errors for subject $i$ and factor $l, \mathrm{respectively}$. The random effects are assumed to be normally distributed with mean 0 and variance-covariance matrix $\mathit{V}\left(\mathit{a}\right)={\mathit{\Sigma }}_{\mathit{a}}$. It is assumed that ${\mathit{\Sigma }}_{\mathit{a}}$ captures the changes among the latent variables [1]. For example, a positive covariance between the random effects for the latent variables 1 and 2 means that if for a given individual $i$ the latent variable 1 increases over time, the latent variable 2 also increases for that individual. Note that in this setting, the covariates are included in the multivariate linear mixed model (MLMM) of Equation (3) but not in the factor analysis model of Equation (1).

In matrix notation, Equation (3) reads:

$${\mathit{\eta }}_i={\mathit{X}}_i\mathit{\beta }+{\mathit{Z}}_i{\mathit{a}}_i+{\mathit{\epsilon }}_i$$

(4)

where

$$\begin{gathered} {\mathit{X}}_i={\left(\begin{array}{c}{\mathit{x}}_{i1} \\ ⋮\\ {\mathit{x}}_{i{T}_i}\end{array}\right)}_{\left[d\times T_i,p\times d\right]}, {\mathit{x}}_{it}={\left(\begin{array}{cc}\begin{array}{cc}{x}_{it}^1& 0\\ 0& x_{it}^2\end{array}& \begin{array}{cc}\cdots & 0\\ \cdots & 0\end{array}\\ \begin{array}{cc}\cdots & \cdots \\ 0& 0\end{array}& \begin{array}{cc}\ddots & ⋮\\ \cdots & x_{it}^d\end{array}\end{array}\right)}_{\left[d, p\times d\right]} \\ {\mathit{Z}}_i={\left(\begin{array}{c}{\mathit{z}}_{i1} \\ ⋮\\ {\mathit{z}}_{i{T}_i}\end{array}\right)}_{\left[d\times T_i,q\times d\right]}, {\mathit{z}}_{it}={\left(\begin{array}{cc}\begin{array}{cc}{z}_{it}^1& 0\\ 0& z_{it}^2\end{array}& \begin{array}{cc}\cdots & 0\\ \cdots & 0\end{array}\\ \begin{array}{cc}\cdots & \cdots \\ 0& 0\end{array}& \begin{array}{cc}\ddots & ⋮\\ \cdots & z_{it}^d\end{array}\end{array}\right)}_{\left[d, q\times d\right]} \\ \mathit{\beta }=\left({\mathit{\beta }}^1{}^{\prime },\dots , {\mathit{\beta }}^d{}^{\prime }\right){\prime }_{\left[p\times d,1\right]} \\ {\mathit{a}}_i=\left({\mathit{a}}_i^{1\prime }, \dots , {\mathit{a}}_i^{d\prime }\right){\prime }_{\left[q\times d,1\right]}\sim N\left(\mathbf{0},{\Sigma }_a\right) \\ {\mathit{\epsilon }}_i=\left({\mathit{\epsilon }}_{i1}^{\prime }, \dots , {\mathit{\epsilon }}_{i{T}_i}^{\prime }\right){\prime }_{\left[d\times T_i,1\right]}, {\mathit{\epsilon }}_{it}\sim N\left(\mathbf{0}, {\Sigma }_{\mathit{\epsilon }}\right) \end{gathered}$$

The marginal distribution of ${\mathit{Y}}_i$ is assumed multivariate normal with mean:

$$\mathit{E} \left({\mathit{Y}}_i\right)=\left({\mathit{I}}_{T_i}⨂\Lambda \right){\mathit{X}}_i\mathit{\beta }$$

and variance-covariance matrix

$$\mathit{V}\left({\mathit{Y}}_i\right)=\left({\mathit{I}}_{T_i}\otimes \Lambda \right)\mathit{V}\left({\mathit{\eta }}_i\right){\left({\mathit{I}}_{T_i}\otimes \Lambda \right)}^{\prime }+{\mathit{I}}_{T_i}\otimes diag\left({\mathit{\tau }}_1^2, \dots , {\mathit{\tau }}_J^2\right)$$

The first term in $\mathit{V}\left({\mathit{Y}}_i\right)$ denotes the variances and covariance of the latent factors and the last term the variances of the error term, ${\mathit{ϵ}}_{it}$. The mean and variance-covariance matrix of ${\mathit{\eta }}_i$ are $\mathit{E}\left({\mathit{\eta }}_i\right)={\mathit{X}}_i\mathit{\beta }$ and $\mathit{V}\left({\mathit{\eta }}_i\right)={\mathit{Z}}_i{\mathit{\Sigma }}_{\mathit{a}}{\mathit{Z}}_i^{\prime }+{\mathit{I}}_{T_i}\otimes {\Sigma }_{\mathit{\epsilon }}$, respectively.

To estimate the LFLMM by EM, we summarize it below, as proposed by [1]. Before going into detail, we observe that $\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}$ is treated as missing data. Hence, the complete dataset is $\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\}$ whereas the observed data is $\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\}$. It follows that the complete data likelihood is:

$$L={\displaystyle \prod }_{i=1}^N\mathit{P}\left({\mathit{Y}}_i|{\mathit{\eta }}_i,\Lambda , {\mathit{\tau }}^2\right)\mathit{P}\left({\mathit{\eta }}_i|{\mathit{X}}_i,{\mathit{Z}}_i, {\mathit{a}}_i,\mathit{\beta },{\Sigma }_{\mathit{\epsilon }}\right)\mathit{P}\left({\mathit{a}}_i|{\Sigma }_{\mathit{a}}\right)$$

(5)

The corresponding complete data loglikelihood is:

$$\log L={\displaystyle \sum }_{i=1}^N\left[\log \mathit{P}\left({\mathit{Y}}_i|{\mathit{\eta }}_i,\Lambda , {\mathit{\tau }}^2\right)+\log \mathit{P}\left({\mathit{\eta }}_i|{\mathit{X}}_i,{\mathit{Z}}_i, {\mathit{a}}_i,\mathit{\beta },{\Sigma }_{\mathit{\epsilon }}\right)+\log \mathit{P}\left({\mathit{a}}_i|{\Sigma }_{\mathit{a}}\right)\right]$$

(6)

where

$$\begin{array}{ll}{\displaystyle {\displaystyle \sum }_{i=1}^N}\log \mathit{P}\left({\mathit{Y}}_i|{\mathit{\eta }}_i,\Lambda , {\mathit{\tau }}^2\right)& ={\displaystyle {\displaystyle \sum }_{i=1}^N}{\displaystyle {\displaystyle \sum }_{j=1}^J}\log \mathit{P}\left({\mathit{Y}}_{ij}|{\mathit{\eta }}_i,{\Lambda }_j, {\mathit{\tau }}_j^2\right)\\ & ={\displaystyle {\displaystyle \sum }_{i=1}^N}{\displaystyle {\displaystyle \sum }_{j=1}^J}\left[-\frac{n_i}{2}\log {\mathit{\tau }}_j^2-\frac{1}{2{\mathit{\tau }}_j^2}{\left({\mathit{Y}}_{ij}-{\mathit{\eta }}_i^{\prime }{\Lambda }_j\right)}^{\prime }\left({\mathit{Y}}_{ij}-{\mathit{\eta }}_i^{\prime }{\Lambda }_j\right)\right]\end{array}$$

(7)

$$\begin{array}{ll}{\displaystyle {\displaystyle \sum }_{i=1}^N}\log \mathit{P}\left({\mathit{\eta }}_i|{\mathit{X}}_i,{\mathit{Z}}_i, \mathit{\beta },{\mathit{a}}_i,{\mathit{\Sigma }}_{\mathit{\epsilon }}\right)& ={\displaystyle {\displaystyle \sum }_{i=1}^N}{\displaystyle {\displaystyle \sum }_{t=1}^{T_i}}\log \mathit{P}\left({\mathit{\eta }}_{it}|{\mathit{X}}_i,{\mathit{Z}}_i, {\mathit{a}}_i,\mathit{\beta }, {\mathit{\Sigma }}_{\mathit{\epsilon }}\right)\\ & \hfill ={\displaystyle {\displaystyle \sum }_{i=1}^N}{\displaystyle {\displaystyle \sum }_{t=1}^{T_i}}\left[-\frac{1}{2}\log \left|{\mathit{\Sigma }}_{\mathit{\epsilon }}\right|-\frac{1}{2}{\left({\mathit{\eta }}_{it}-{\mathit{X}}_{it}\mathit{\beta }-{\mathit{Z}}_i{\mathit{a}}_i\right)}^{\prime }{\mathit{\Sigma }}_{\mathit{\epsilon }}^{-1}\left({\mathit{\eta }}_{it}-{\mathit{X}}_{it}\mathit{\beta }-{\mathit{Z}}_i{\mathit{a}}_i\right) \right]\hfill \end{array}$$

(8)

$${\displaystyle \sum }_{i=1}^N\log \mathit{P}\left({\mathit{a}}_i|{\Sigma }_{\mathit{a}}\right)=-\frac{1}{2}{\displaystyle \sum }_{i=1}^N\left[\log \left|{\Sigma }_{\mathit{a}}\right|-\frac{1}{2}\mathit{a}{\prime }_i{\Sigma }_{\mathit{a}}^{-1}{\mathit{a}}_i\right]$$

(9)

Let the $\mathit{\theta }$ denote the parameter vector $\left(\Lambda , {\mathbf{\tau }}^2, \mathit{\beta }, \mathrm{and} {\Sigma }_{\mathsf{\epsilon }}\right)$, ${\mathit{\theta }}^{\left(w\right)}$ be the ML estimate of $\mathit{\theta }$ at the $w$th iteration for $w=0,1,\dots$., and $Q\left(\mathit{\theta }|{\mathit{\theta }}^{\left(w\right)}\right)$ the expectation of the joint loglikelihood for the complete data$\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\}$ conditional on the observed data $\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\}$:

$$Q\left(\mathit{\theta }|{\mathit{\theta }}^{\left(w\right)}\right)=\mathit{E}\left\{\log L\left(\mathit{\theta }|{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\right)|{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i,{\mathit{\theta }}^{\left(w\right)}\right\}$$

(10)

Then the $\left(w+1\right)$th iteration of the EM algorithm consists of (i) the E–step, which is the expectation of the joint loglikelihood computed according to (10) and (ii) the M-step, which maximizes $Q\left(\mathit{\theta }|{\mathit{\theta }}^{\left(w\right)}\right)$ to yield ${\mathit{\theta }}^{\left(w+1\right)}$. Further details on EM estimation of the LFLMM can be found in [1].

3. The Supplemented EM

Below we discuss the Supplemented EM algorithm, denoted as SEM. Before going into detail, we observe that the main purpose of this study is to estimate the standard errors of the fixed effects, $\mathit{\beta }$.

Consider the mapping M defined by iteration w of the EM algorithm:

$${\mathit{\beta }}^{\left(w+1\right)}=M\left({\mathit{\beta }}^{\left(w\right)}\right), \mathrm{for} w=\mathbf{0},\mathbf{1},\dots$$

when the parameter vector converges to ${\mathit{\beta }}^{\ast }, \mathrm{we} \mathrm{obtain} {\mathit{\beta }}^{\ast }=M\left({\mathit{\beta }}^{\ast }\right)$. For $M\left(\mathit{\beta }\right)$ continuous we have by Taylor expansion in the neighbourhood of ${\mathit{\beta }}^{\ast }$

$${\mathit{\beta }}^{\left(\mathit{w}+1\right)}=M\left({\mathit{\beta }}^{\left(\mathit{w}\right)}\right)\approx M\left({\mathit{\beta }}^{\ast }\right)+\mathit{D}\mathit{M}\left({\mathit{\beta }}^{\left(\mathit{w}\right)}-{\mathit{\beta }}^{\ast }\right)={\mathit{\beta }}^{\ast }+\mathit{D}\mathit{M}\left({\mathit{\beta }}^{\left(\mathit{w}\right)}-{\mathit{\beta }}^{\ast }\right)$$

(11)

where

$$\mathit{D}\mathit{M}={\left(\frac{\partial M_h\left(\mathit{\beta }\right)}{\partial {\mathit{\beta }}_g}\right)|}_{\mathit{\beta }={\mathit{\beta }}^{\ast }}$$

(12)

$g=1,2,\dots , k$ and $h=1,2,\dots ,k$ is the $k\times k$ Jacobian matrix of $M\left(\mathit{\beta }\right)=\left(M_1\left(\mathit{\beta }\right),\dots ,M_k\left(\mathit{\beta }\right)\right)$ evaluated at the ML estimate of $\mathit{\beta }$ with $k=p\times d.$DM is known as the rate matrix. To obtain the loglikelihood of $\mathit{\beta }$, we consider the complete data density of the LFLMM:

$$\begin{array}{l}f\left(\right\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\left\}\right|\mathit{\theta })\\ =f\left(\right\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\left\}\right|\mathit{\theta }\left)f\right(\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}\left|\right\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\},\mathit{\theta })\end{array}$$

where $f\left(\right\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\left\}\right|\mathit{\theta })$ is the density of the observed data and $f\left(\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}\right|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\},\mathit{\theta })$ the density of missing data, given the observed data. Thus, the loglikelihood of $\mathit{\beta }$ given the complete data is:

$$\log L(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\})=\log L(\mathit{\beta }\left|\right\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})+\log f\left(\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}\right|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\},\mathit{\beta })$$

(13)

where $\log L(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})$ is the observed-data loglikelihood and $\log L(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\})$ is the complete data loglikelihood.

The asymptotic variance-covariance matrix of $\mathit{\beta }$, $\mathit{V}\left(\mathit{\beta }\right)$, is the inverse of the observed information matrix $\left({\mathit{I}}_o\right)$. In the case of the LFLMM, the observed data is $\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\}$ so that $\mathit{V}\left(\mathit{\beta }\right)$ is:

$$\mathit{V}\left(\mathit{\beta }\right)={\mathit{I}}_o^{-1}(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})$$

(14)

where ${\mathit{I}}_o(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})$is the information matrix of the observed data loglikelihood (which is assumed to exist). That is [6,11]:

$${\mathit{I}}_o(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})=-E\left[\frac{{\partial }^2\log L(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\})}{\partial \mathit{\beta }·\partial \mathit{\beta }}\right]$$

(15)

Equation (15) is difficult to evaluate directly using the EM algorithm [6,11]. As a way out, [7] suggested to evaluate the complete data information matrix:

$${\mathit{I}}_o(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\})=-E\left[\frac{{\partial }^2\log L(\mathit{\beta }|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\})}{\partial \mathit{\beta }·\partial \mathit{\beta }}\right]$$

(16)

The conditional complete data information, given the observed data evaluated at $\mathit{\beta }={\mathit{\beta }}^{\ast }$, is:

$${\mathit{I}}_{oc}=E\left[{\mathit{I}}_o\left(\mathit{\beta }\right|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i, {\mathit{\eta }}_i, {\mathit{a}}_i\left\}\right|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\},{\mathit{\beta }}^{\ast } \right]$$

(17)

After taking second derivatives, averaging over $f\left(\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}\right|\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\},\mathit{\beta })$, and evaluating at $\mathit{\beta }={\mathit{\beta }}^{\ast }$, Equation (13) implies:

$${\mathit{I}}_o\left({\mathit{\beta }}^{\ast }\left|\right(\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\}\right)={\mathit{I}}_{oc}-{\mathit{I}}_{om}$$

(18)

where the missing information matrix $\left({\mathit{I}}_{om}\right)$ is

$${\mathit{I}}_{om}=E\left[-\frac{{\partial }^2\log f \left(\left\{{\mathit{\eta }}_i, {\mathit{a}}_i\right\}|\left\{{\mathit{Y}}_i,{\mathit{X}}_i,{\mathit{Z}}_i\right\},{\mathit{\beta }}^{\ast } \right)}{\partial \mathit{\beta }·\partial \mathit{\beta }}\right]$$

(19)

ref [12] interpreted Equation (18) as

$$observed inform\mathit{a}tion=complete inform\mathit{a}tion-missing inform\mathit{a}tion$$

and called it the “missing information principle”. Equation (18) can be written as:

$${\mathit{I}}_o\left({\mathit{\beta }}^{\ast }\left|\right(\{{\mathit{Y}}_i, {\mathit{X}}_i, {\mathit{Z}}_i\}\right)=\left(\mathit{I}-{\mathit{I}}_{om}{\mathit{I}}_{oc}^{-1}\right){\mathit{I}}_{oc},$$

(20)

where $\mathit{I}$ is the $k\times k$ identity matrix and ${\mathit{I}}_{om}{\mathit{I}}_{oc}^{-1}$ is the matrix of the fraction of missing information [7,11]. According to [13], the rate of convergence of the EM algorithm is determined by the fraction of missing information in the neighborhood of ${\mathit{\beta }}^{\ast }$:

$$\mathit{D}\mathit{M}={\mathit{I}}_{om}{\mathit{I}}_{oc}^{-1}$$

(21)

Substituting $\mathit{D}\mathit{M}={\mathit{I}}_{om}{\mathit{I}}_{oc}^{-1}$ into Equation (20) and inverting, the asymptotic variance-covariance matrix of ${\mathit{\beta }}^{\ast }$, $\mathit{V}\left({\mathit{\beta }}^{\ast }\right)$ is:

$$\mathit{V}\left({\mathit{\beta }}^{\ast }\right)={\mathit{I}}_{oc}^{-1}{\left(\mathit{I}-\mathit{D}\mathit{M}\right)}^{-1}$$

(22)

From the equality ${\left(\mathit{I}-\mathit{P}\right)}^{-1}=\left(\mathit{I}-\mathit{P}+\mathit{P}\right){\left(\mathit{I}-\mathit{P}\right)}^{-1}=\mathit{I}+\mathit{P}{\left(\mathit{I}-\mathit{P}\right)}^{-1}$ it follows that:

$$\mathit{V}\left({\mathit{\beta }}^{\ast }\right)={\mathit{I}}_{oc}^{-1}\left\{\mathit{I}+\mathit{D}\mathit{M}{\left(\mathit{I}-\mathit{D}\mathit{M}\right)}^{-1}\right\}={\mathit{I}}_{oc}^{-1}+{\mathit{I}}_{oc}^{-1}\mathit{D}\mathit{M}{\left(\mathit{I}-\mathit{D}\mathit{M}\right)}^{-1}$$

(23)

or

$$\mathit{V}\left({\mathit{\beta }}^{\ast }\right)={\mathit{I}}_{oc}^{-1}+\Delta \mathit{V}\left({\mathit{\beta }}^{\ast }\right)$$

(24)

where $\Delta \mathit{V}\left({\mathit{\beta }}^{\ast }\right)$ is the increase of the diagonal elements of $\mathit{V}\left({\mathit{\beta }}^{\ast }\right)$ related to missing information.

Calculation of the $\mathit{D}\mathit{M}$ matrix can be done using the code and output of the original EM algorithm as follows [6,7]. The $\mathit{D}\mathit{M}$ matrix represents the differential of the parameter mappings during the EM algorithm. Hence, each element of the $\mathit{D}\mathit{M}$ matrix represents a component-wise increase of the rate of convergence per iteration of the EM algorithm. Let $r_{gh}$ be the $\left(g,h\right)$th element of the $\mathit{D}\mathit{M}$ matrix. From Equation (13), we have:

$$\begin{array}{ll}{r}_{gh}=\frac{\partial M_h\left({\mathit{\beta }}^{\ast }\right)}{\partial {\mathit{\beta }}_g}& =\underset{{\beta }_g\to {\beta }_g^{\ast }}{\lim }\frac{M_h\left({\beta }_1^{\ast },\dots ,{\beta }_{g-1}^{\ast },{\beta }_g,{\beta }_{g+1}^{\ast }, \dots ,{\beta }_k^{\ast }\right)-M_h\left({\mathit{\beta }}^{\ast }\right)}{{\beta }_g-{\beta }_g^{\ast }}\\ & =\underset{w\to \infty }{\lim }\frac{M_h\left({\beta }^{\left(w\right)}\left(g\right)\right)-M_h\left({\beta }^{\ast }\right)}{{\beta }_g^{\left(w\right)}-{\beta }_g^{\ast }}\equiv \underset{w\to \infty }{lim}{r}_{gh}^{\left(w\right)}\end{array}$$

(25)

$g=1,2,\dots , k$ and $h=1,2,\dots ,k$

where ${\mathit{\beta }}^{\left(w\right)}\left(g\right)$ is called the semi-active parameter set

$${\mathit{\beta }}^{\left(w\right)}\left(g\right)=\left({\beta }_1^{\ast },\dots ,{\beta }_{g-1}^{\ast },{\beta }_g^{\left(w\right)},{\beta }_{g+1}^{\ast }, \dots ,{\beta }_k^{\ast }\right) , w=1, 2,\dots$$

(26)

which converges to ${\beta }_g^{\ast }$. Note that only the $g$th component in ${\mathit{\beta }}^{\left(w\right)}\left(g\right)$ takes a value different from its maximum likelihood estimate.

To calculate$r_{gh}$, the Supplemented EM algorithm requires ${\mathit{\theta }}^{\ast }=\left\{{\Lambda }^{\ast }, {\mathit{\tau }}^{2\ast }, {\mathit{\beta }}^{\ast }\mathrm{and} {\Sigma }_{\mathit{\epsilon }}{}^{\ast }\right\}$ and ${\mathit{\theta }}^{\left(\mathrm{w}\right)}=\left\{{\Lambda }^{\left(w\right)}, {\mathit{\tau }}^{2\left(w\right)}, {\mathit{\beta }}^{\left(w\right)}\mathrm{and} {\Sigma }_{\mathit{\epsilon }}{}^{\left(w\right)}\right\}$ for $w=1, 2,\dots$ as input$. {\mathit{\theta }}^{\ast }$ can be obtained by the EM algorithm using a set of arbitrarily chosen initial parameters ${\mathit{\theta }}^{init}$including ${\mathit{\theta }}^{\left(w\right)}$ for $w=1, \mathrm{i}.\mathrm{e}., {\mathit{\theta }}^{\left(1\right)}$. The starting point ${\mathit{\theta }}^{\left(1\right)}$ may, but need not, be close ${\mathit{\theta }}^{\ast }$. The algorithm below closely follows [14,15].

• Select input: ${\mathit{\theta }}^{\left(\mathrm{w}\right)}$ and ${\mathit{\theta }}^{\ast }$
• Set ${\mathit{\theta }}^{\left(\mathrm{w}\right)}$ for $w=1, 2,\dots$. Then take the E step and M step of the LFLMM EM algorithm to produce ${\mathit{\theta }}^{\left(w+1\right)}$.
• For rows $=1,2,\dots , k$:

  (i)

  Set ${\tilde{\mathit{\beta }}}^{\left(w\right)}\left(g\right)$ be equal to ${\mathit{\beta }}^{\ast }$, except for the $g$th element: $\left({\tilde{\mathit{\beta }}}^{\left(w\right)}\left(g\right)=({\beta }_1^{\ast },\dots ,{\beta }_{g-1}^{\ast },{\beta }_g^{\left(w\right)},{\beta }_{g+1}^{\ast }, \dots ,{\beta }_k^{\ast }\right))$

  (ii)

  Run the LFLMM EM algorithm with ${\tilde{\mathit{\beta }}}^{\left(w\right)}\left(g\right)$ as the current estimate of $\mathit{\beta }$ to obtain ${\tilde{\mathit{\beta }}}^{\left(w+1\right)}\left(g\right)$.

  (iii)

  Calculate the $g$th row of $r_{gh}^{\left(w\right)}$ as

  $$r_{gh}^{\left(w\right)}=\frac{{\tilde{\beta }}_h{}^{\left(w+1\right)}\left(g\right)-{\beta }_h^{\ast }}{{\beta }_g^{\left(w\right)}-{\beta }_g^{\ast }} , \mathrm{for} h=1,2,\dots ,k$$

The output after a single run of the Supplemented EM algorithm (Step 1 and 2) are ${\mathit{\beta }}^{w+1}$ and $r_{gh}^{\left(w\right)} g=1,2,\dots , k$ and $h=1,2,\dots ,k$. Based on the final estimates of DM, $\mathit{V}\left({\mathit{\beta }}^{\ast }\right)$ is calculated using (24). The diagonal elements of $\mathit{V}$ are the variance of ${\mathit{\beta }}^{\ast }$.

4. Simulation

To evaluate the statistical properties and computational aspects of the SEM, we set up a simulation study. The number of subjects ($N)$ is set at 500, 1000, and 1500 with six time periods. The number of simulations ($S$) is set at 50 and 250. The other set-up of the simulations is adopted from [1]. Particularly, we use the same initial values of the parameters of the LFLMM model (12 items, 2 latent factors, and a simple structure to model the relationship between the items and the latent factors). It is done to check if the bias resulting from the Supplemented EM algorithm on LFLMM is in line with the results presented in [1].

Table 1. The absolute difference between the true parameters and the SEM averages.

No	Para	True	SEM						No	Para	True	SEM
	$\mathit{N}$		500		1000		1500			$\mathit{N}$		500		1000		1500
	$\mathit{S}$		50	250	50	250	50	250		$\mathit{S}$		50	250	50	250	50	250
1	${\lambda }_{2,1}$	1	0.0013	0.0012	0.0010	0.0009	0.0009	0.0009	23	${\mathit{\beta }}_1^1$	1	0.0040	0.0036	0.0032	0.0004	0.0018	0.0007
2	${\lambda }_{3,1}$	1	0.0019	0.0015	0.0013	0.0009	0.0012	0.0009	24	${\mathit{\beta }}_1^2$	−1	0.0111	0.0079	0.0008	0.0008	0.0034	0.0017
3	${\lambda }_{4,1}$	1	0.0015	0.0009	0.0005	0.0009	0.0012	0.0010	25	${\mathit{\beta }}_2^1$	0	0.0038	0.0018	0.0012	0.0007	0.0016	0.0003
4	${\lambda }_{5,1}$	1	0.0014	0.0016	0.0007	0.0009	0.0007	0.0009	26	${\mathit{\beta }}_2^2$	0	0.0031	0.0039	0.0014	0.0021	0.0044	0.0006
5	${\lambda }_{6,1}$	1	0.0016	0.0014	0.0011	0.0009	0.0013	0.0010	27	${\mathit{\beta }}_3^1$	1	0.0012	0.0016	0.0008	0.0003	0.0007	0.0004
6	${\lambda }_{7,2}$	1	0.0021	0.0018	0.0013	0.0014	0.0016	0.0013	28	${\mathit{\beta }}_3^2$	1	0.0016	0.0010	0.0002	0.0004	0.0000	0.0005
7	${\lambda }_{8,2}$	1	0.0015	0.0016	0.0017	0.0014	0.0010	0.0013	29	${\mathit{\beta }}_4^1$	1	0.0025	0.0027	0.0002	0.0001	0.0007	0.0002
8	${\lambda }_{9,2}$	1	0.0020	0.0015	0.0015	0.0014	0.0019	0.0013	30	${\mathit{\beta }}_4^2$	−1	0.0018	0.0025	0.0004	0.0001	0.0003	0.0001
9	${\lambda }_{10,2}$	1	0.0015	0.0015	0.0015	0.0014	0.0022	0.0017	31	${\sigma }_{\mathit{a},11}$	3	0.0332	0.0444	0.0152	0.0154	0.0039	0.0160
10	${\lambda }_{11,2}$	1	0.0019	0.0012	0.0017	0.0017	0.0016	0.0016	32	${\sigma }_{\mathit{a},12}$	1	0.0410	0.0120	0.0024	0.0057	0.0265	0.0085
11	${\mathit{\tau }}_1^2$	0.5	0.0007	0.0001	0.0006	0.0004	0.0009	0.0004	33	${\sigma }_{\mathit{a},13}$	1.5	0.0289	0.0200	0.0218	0.0141	0.0229	0.0133
12	${\mathit{\tau }}_2^2$	0.5	0.0003	0.0006	0.0000	0.0011	0.0019	0.0006	34	${\sigma }_{\mathit{a},14}$	1	0.0286	0.0131	0.0077	0.0083	0.0169	0.0094
13	${\mathit{\tau }}_3^2$	0.5	0.0027	0.0001	0.0021	0.0001	0.0005	0.0007	35	${\sigma }_{\mathit{a},22}$	3	0.0233	0.0105	0.0247	0.0017	0.0276	0.0150
14	${\mathit{\tau }}_4^2$	0.5	0.0007	0.0009	0.0002	0.0002	0.0005	0.0008	36	${\sigma }_{\mathit{a},23}$	1	0.0003	0.0045	0.0158	0.0042	0.0124	0.0051
15	${\mathit{\tau }}_5^2$	0.5	0.0002	0.0006	0.0010	0.0013	0.0010	0.0006	37	${\sigma }_{\mathit{a},24}$	2	0.0073	0.0020	0.0057	0.0004	0.0126	0.0089
16	${\mathit{\tau }}_6^2$	0.5	0.0044	0.0010	0.0017	0.0005	0.0004	0.0004	38	${\sigma }_{\mathit{a},33}$	3	0.0014	0.0125	0.0383	0.0260	0.0387	0.0274
17	${\mathit{\tau }}_7^2$	0.5	0.0005	0.0008	0.0016	0.0021	0.0017	0.0004	39	${\sigma }_{\mathit{a},34}$	1	0.0080	0.0075	0.0236	0.0069	0.0043	0.0022
18	${\mathit{\tau }}_8^2$	0.5	0.0027	0.0012	0.0006	0.0006	0.0011	0.0009	40	${\sigma }_{\mathit{a},44}$	3	0.0021	0.0015	0.0215	0.0020	0.0148	0.0094
19	${\mathit{\tau }}_9^2$	0.5	0.0016	0.0010	0.0004	0.0009	0.0001	0.0004	41	${\sigma }_{\mathit{\epsilon },11}$	0.5	0.0020	0.0013	0.0028	0.0005	0.0015	0.0013
20	${\mathit{\tau }}_{10}^2$	0.5	0.0009	0.0012	0.0012	0.0005	0.0003	0.0008	42	${\sigma }_{\mathit{\epsilon },12}$	0.2	0.0017	0.0006	0.0006	0.0003	0.0006	0.0001
21	${\mathit{\tau }}_{11}^2$	0.5	0.0001	0.0001	0.0013	0.0000	0.0012	0.0001	43	${\sigma }_{\mathit{\epsilon },22}$	0.5	0.0021	0.0033	0.0027	0.0029	0.0028	0.0023
22	${\mathit{\tau }}_{12}^2$	0.5	0.0011	0.0012	0.0001	0.0002	0.0011	0.0002

presents the absolute difference for the true parameters, and the averages of the SEM estimates are calculated as a measure of performance.

Table 1 shows that the absolute difference of ${\sigma }_{\mathit{a},11}$ has a range from 0 to 0.0444 $(N=500$ and $S=250)$. The results are in line with [1] the parameters of the measurement model (factor loadings and error variances) are estimated more precisely than those of the latent mixed regression model. Overall, these results indicate that with the increasing number of subjects in the simulation, the absolute difference between the actual parameters and the SEM average is getting smaller. This means the SEM can estimate the model parameter very well.

Although the results in Table 1 indicate that the accuracy of the estimates in the latent mixed regression model (No. 23–43) is not as good as in the measurement model part (No. 1–22), through Figure 1a–c, it can be shown that the median of boxplots (which generally is close to the true parameters) are all at the same level. This means that the parameters of the latent mixed regression models $\left(\beta , {\sigma }_{\mathit{a}}, {\sigma }_{\mathit{\epsilon }}\right)$ are estimated more precisely, especially for the number of simulations $S=250$. Furthermore, all the boxplots are also shown to have different distributions of views with an increasing number of subjects in the simulation. This is indicated by the smaller size of the boxplot as the number of subjects increases for both numbers of simulations.

The results from the simulations of the Supplemented EM algorithm in estimating the asymptotic variance matrix of beta is summarized as a standard deviation of beta in

Table 2. The parameter estimates for $\sqrt{\mathit{V}\left(\widehat{\mathit{\beta }}\right)}$.

Number of Subjects	Parameter	The 2nd Moment		SEM
		50	250	50	250
500	${\mathit{\beta }}_1^1$	0.0512	0.0528	0.0249	0.0251
	${\mathit{\beta }}_1^2$	0.0587	0.0530	0.0187	0.0195
	${\mathit{\beta }}_2^1$	0.0378	0.0392	0.0110	0.0126
	${\mathit{\beta }}_2^2$	0.0391	0.0379	0.0164	0.0158
	${\mathit{\beta }}_3^1$	0.0184	0.0190	0.0245	0.0268
	${\mathit{\beta }}_3^2$	0.0184	0.0190	0.0212	0.0200
	${\mathit{\beta }}_4^1$	0.0283	0.0283	0.0105	0.0105
	${\mathit{\beta }}_4^2$	0.0288	0.0308	0.0145	0.0145
1000	${\mathit{\beta }}_1^1$	0.0338	0.0327	0.0179	0.0184
	${\mathit{\beta }}_1^2$	0.0342	0.0339	0.0138	0.0134
	${\mathit{\beta }}_2^1$	0.0232	0.0232	0.0071	0.0077
	${\mathit{\beta }}_2^2$	0.0253	0.0232	0.0105	0.0100
	${\mathit{\beta }}_3^1$	0.0095	0.0100	0.0176	0.0184
	${\mathit{\beta }}_3^2$	0.0089	0.0095	0.0130	0.0130
	${\mathit{\beta }}_4^1$	0.0141	0.0130	0.0077	0.0077
	${\mathit{\beta }}_4^2$	0.0130	0.0130	0.0110	0.0105
1500	${\mathit{\beta }}_1^1$	0.0276	0.0253	0.0152	0.0152
	${\mathit{\beta }}_1^2$	0.0270	0.0265	0.0105	0.0110
	${\mathit{\beta }}_2^1$	0.0187	0.0182	0.0063	0.0063
	${\mathit{\beta }}_2^2$	0.0164	0.0192	0.0084	0.0084
	${\mathit{\beta }}_3^1$	0.0071	0.0071	0.0145	0.0152
	${\mathit{\beta }}_3^2$	0.0063	0.0071	0.0105	0.0105
	${\mathit{\beta }}_4^1$	0.0110	0.0105	0.0063	0.0063
	${\mathit{\beta }}_4^2$	0.0110	0.0105	0.0084	0.0084

. We also calculate the standard deviation of beta using the 2nd moment, $\sqrt{\mathit{V}\left(\widehat{\mathit{\beta }}\right)}=\sqrt{\frac{1}{S-1}{\displaystyle \sum }_{s=1}^S{\left({\widehat{\mathit{\beta }}}_s-\overline{\widehat{\mathit{\beta }}}\right)}^2}$ as a benchmark to compare with the standard deviation of beta of SEM. Both the 2nd moment and SEM produce symmetrical results the parameter estimate for the standard deviation of beta is getting smaller with the increasing number of subjects in the simulation. Overall, it can be concluded that by using SEM, changes in the parameter estimate for the standard deviation of beta are not too different for all number of subjects (Figure 2). Therefore, the simulation results suggest that the asymptotic variance of beta from the Supplemented EM Algorithm can be used to estimate the asymptotic variance of beta in real data analysis.

5. Real Data Example

The real data-set that we used to illustrate the development of the Supplemented EM algorithm is the political attitudes and behavior data of Flemish. The data was designed to include a representative sample of the target population under the Belgian electorate. The Flemish data set (Flemish and Dutch speaking respondents from Brussels Capital Region) consists of 1274 respondents, who have been interviewed three times (1991, 1995, and 1999) [16,17,18]. There are four latent factors measured on political attitudes of Flemish used, i.e., Individualism, Nationalism, Ethnocentrism, and Authoritarianism. This data has been analyzed using various methods by several authors, including [19,20,21,22,23]. There are three interesting questions in this real data case, i.e., how Individualism, Nationalism, Ethnocentrism, and Authoritarianism of the Flemish develop over time; whether there is an association between these four developments, and whether the gender of the respondent affects the change patterns of latent developments.

I, N, E, and A in

Table 3. Correlation matrix of random effects.

Random Effects		I		N		E		A
		${\mathit{a}}_{11}$	${\mathit{a}}_{12}$	${\mathit{a}}_{21}$	${\mathit{a}}_{22}$	${\mathit{a}}_{31}$	${\mathit{a}}_{32}$	${\mathit{a}}_{41}$	${\mathit{a}}_{42}$
I	$a_{11}$	1	0.892	0.832	0.903	0.966	0.930	0.956	0.880
	$a_{12}$	0.892	1	0.854	0.878	0.937	0.931	0.913	0.895
N	$a_{21}$	0.832	0.854	1	0.311	0.864	0.883	0.856	0.835
	$a_{22}$	0.903	0.878	0.311	1	0.918	0.893	0.899	0.861
E	$a_{31}$	0.966	0.937	0.864	0.918	1	0.946	0.973	0.919
	$a_{32}$	0.930	0.931	0.883	0.893	0.946	1	0.946	0.918
A	$a_{41}$	0.956	0.913	0.856	0.899	0.973	0.946	1	0.873
	$a_{42}$	0.880	0.895	0.835	0.861	0.919	0.918	0.873	1

correspond to Individualism, Nationalism, Ethnocentrism, and Authoritarianism, respectively. $a_{11}$ and $a_{12}$ are the random intercept and random slope for Individualism. $a_{21}$ and $a_{22}$are the random intercept and random slope for Nationalism. $a_{31}$ and $a_{32}$are the random intercept and random slope for Ethnocentrism. $a_{41}$ and $a_{42}$are the random intercept and random slope for Authoritarianism. The positive correlation of random intercept between $a_{11}$ and $a_{21}$, $a_{11}$ and $a_{31}$, $a_{11}$ and $a_{41}$ suggests that the development of Individualism and other political attitudes is highly related, which highest correlated with Ethnocentrism. The results indicate that those who have a better sense of Individualism tend to have a better sense of Nationalism, Ethnocentrism, and Authoritarianism. The results find a positive correlation of random intercept between $a_{21}$ and $a_{31}$, $a_{21}$ and $a_{41}$. It suggests that those who have a better sense of Nationalism tend to have a better sense of Ethnocentrism and Authoritarianism, as well as those who have a better sense of Ethnocentrism tend to have a better sense of Authoritarianism. There is also a positive correlation of random slope between $a_{12}$ and $a_{22}$. It means that if one subject’s Individualism decreases over time, then it is reasonable to expect that his or her Nationalism will decrease over time and vice versa. This also holds between Individualism and Ethnocentrism and between Individualism and Authoritarianism. The positive correlation of random slope between $a_{22}$ and $a_{32}$, meaning that if one subject’s Nationalism decreases over time, then it is reasonable to expect that his or her Ethnocentrism will decrease over time. The correlation matrix of random effects confirms that all latent factors have a positive correlation over time.

The significance of parameter estimate of $\beta$ is analyzed via the $z–$values. By using the Supplemented EM algorithm, the standard errors of $\beta$ for all parameters can be calculated. The standard errors of $\beta$ are listed in

Table 4. Parameter estimates of $\beta$.

Parameter	Estimate	$\mathit{S}\mathit{E} (\widehat{\mathit{\beta }})$	Lower	Upper
Individualism	−0.195	0.004	−0.203	−0.187
Nationalism	−0.177	0.028	−0.231	−0.123
Ethnocentrism	−0.252	0.011	−0.273	−0.231
Authoritarianism	−0.186	0.002	−0.190	−0.182
Slope of Male on Individualism	0.110	0.010	0.091	0.129
Slope of Male on Nationalism	0.219	0.017	0.185	0.253
Slope of Male on Ethnocentrism	−0.038	0.003	−0.045	−0.031
Slope of Male on Authoritarianism	0.022	0.017	−0.011	0.055

. Using a 95 percent confidence interval of $\beta$, almost all confidence intervals do not include the null value, except the slope of Male on Authoritarianism. Hence there are statistically significant differences in the parameter estimate of $\beta$. In other words, all latent factors of Flemish people decrease over time, with Ethnocentrism having the highest rate of decline over time (−0.252) and Nationalism the lowest (−0.177). On average, the Individualism and Nationalism of the male respondent are higher than that of the female. However, Ethnocentrism of the male respondent is lower than that of the female.

6. Conclusions

This paper proposed the Supplemented EM algorithm for LFLMM in estimating the asymptotic variance-covariance matrix as a by-product of the EM estimator for the case of fixed variables in the model. Results from simulation studies suggest that the Supplemented EM algorithm can estimate the model very close to the initial parameters.

As a result of the development of EM algorithm of LFLMM, the Supplemented EM algorithm is very slow to converge, as stated by [1], especially when the number of simulations is 250 times with 1500 subjects. For this reason, further research is needed to find techniques that can be used to accelerate the speed of the algorithm. Several approaches to speed the EM algorithm have been proposed and can be found in [24,25,26] (the ECM algorithm), [27] (the ECME algorithm), and [28] (the Parameter-Expanded EM algorithm).