A Cautionary Note Regarding Multilevel Factor Score Estimates from Lavaan

Abstract

To compute factor score estimates, lavaan version 0.6–12 offers the function lavPredict( ) that can not only be applied in single-level modeling but also in multilevel modeling, where characteristics of higher-level units such as working environments or team leaders are often assessed by ratings of employees. Surprisingly, the function provides results that deviate from the expected ones. Specifically, whereas the function yields correct EAP estimates of higher-level factors, the ML estimates are counterintuitive and possibly incorrect. Moreover, the function does not provide the expected standard errors. I illustrate these issues using an example from organizational research where team leaders are evaluated by their employees, and I discuss these issues from a measurement perspective.

1. Introduction

It goes without saying that most concepts in psychology cannot directly be observed but can only be inferred from answered items of questionnaires or from other instruments. One way of obtaining individual scores on these dimensions is estimating them based on statistical models. This approach may thus be called model-based (for a discussion of this and other approaches that do not necessarily rely on statistical models, see [1,2,3]). There are different methods for obtaining factor score estimates from a model, with the two best known being the Bartlett or Maximum Likelihood (ML) methods [4] and the regression or Bayesian methods [5,6]. It is interesting to note that the obtained factor score estimates can differ greatly between these methods; the more they differ, the lower the reliability is.

A context in which these methods may be applied is diagnostics, where factor score estimates are used to assess an unobserved characteristic. Another context is secondary data analysis, where these estimates are used in further analyses (e.g., in regression analyses) to infer population characteristics such as the relationship between two variables of interest. However, it has been emphasized that in order to yield unbiased regression coefficients, corrections must be applied that take measurement error into account and correct the coefficients accordingly [7]. Various approaches for correcting the coefficients have been suggested in the literature (e.g., [8,9,10]) and are currently being further developed (e.g., [11,12,13,14,15,16,17]).

Types of Factor Score Estimate

In the following, it is assumed that the model is unidimensional (i.e., there is only one factor). By assuming unidimensionality, the factor score estimates can easily be defined, avoiding complicated notation. In the definitions of the factor score estimates, no specific measurement model is assumed, but later in the Motivating Example section, a measurement model will be used in order to illustrate my point.

The definitions follow from the general assumption that an unbiased estimate is equal to the population parameter plus an error and an additional assumption of normality. Specifically, the ML estimate is defined as a random variable that is centered around the true value T and that follows a normal distribution. Formally, this can be expressed as:

$$\mathrm{MLE}\sim \mathrm{N}\left(T,{\mathsf{\sigma }}^2\right)$$

(1)

The standard error of the ML estimate is:

$${\mathit{SE}}_{\mathrm{MLE}}=\mathsf{\sigma }$$

(2)

It can be argued that Equations (1) and (2) represent “a paradigm of the concept of measurement, namely: that a measurement equals the true quantity measured plus an orthogonal error of measurement” ([18], p. 513). Admittedly, normality is not necessarily assumed by this “paradigm”. However, this additional assumption is made in order to facilitate the computation of the Expected A Posteriori (EAP) estimate.

Further assuming that the true value is normally distributed around zero (or in Bayesian terminology, assuming a normal prior for the true value with zero mean), the EAP estimate is simply:

$$\mathrm{EAP}=\mathsf{\omega }·\mathrm{MLE}$$

(3)

where ω is the reliability of the ML estimate. The standard error of the EAP estimate, which is sometimes also referred to as the measurement error, can be expressed as a function of the standard error of the ML estimate:

$${\mathit{SE}}_{\mathrm{EAP}}=\mathsf{\omega }·{\mathit{SE}}_{\mathrm{MLE}}$$

(4)

See Appendix A for more details about how these equations were obtained (see also, [18,19,20,21,22]).

To easily compute the factor score estimates in application contexts, the software lavaan [23] could be used, which is a package for latent variable modeling in R [24]. For this purpose, lavaan version 0.6–12 offers the function lavPredict( ). However, during the course of my research, I noticed that in multilevel modeling, where characteristics of higher-level units are often assessed by ratings of lower-level units, the function’s results deviated from what had been expected. For example, working environments or team leaders may be assessed by the ratings of employees. Whereas the function yielded correct EAP estimates of higher-level factors, the ML estimates deviated from the expected ones. In addition, the function did not provide the expected standard errors. Because users should be aware of these issues, they will be briefly illustrated next.

2. Motivating Example

Suppose leadership behavior, which is a characteristic of a team leader, is assessed by their employees’ perceptions Y by first asking employees to rate their team leader and then averaging the ratings across the employees in the team (e.g., [25]). This mean is assumed to reflect the employees’ shared perception and is thus conceptualized as a latent rather than a manifest variable (see [26] for a detailed discussion). This latent mean varies between teams, whereas the employee-specific deviations from this mean vary within a team. In Mplus notation, the small letters b and w indicate between and within, respectively. Thus, Y can be decomposed into a latent between component $Y^b$ and a within component $Y^w$ that is centered around $Y^b$ (i.e., it has a mean of zero; [27]; see also [26]). Formally, for an employee i in a team j:

$$Y_{ij}=Y_j^b+Y_{ij}^w$$

(5)

Note that this equation is of the type “observed indicator = latent factor + measurement error”, and this is why it can also be considered a measurement model for the employees’ shared perception and thus for leadership behavior. More specifically, it can be considered a parallel model because the employees are exchangeable raters that act as parallel indicators of leadership behavior (see [28,29,30]); that is, they have equal loadings and equal error variances.

2.1. Maximum Likelihood Estimate

Given the parallel model, the ML estimate of a team leader j’s behavior is simply the average across their employees’ ratings:

$${\mathrm{MLE}}_j=\frac{{\sum }_{i=1}^n{Y}_{ij}}{n}$$

(6)

and the standard error of the ML estimate is obtained by analogy to the simple normal model as:

$${\mathit{SE}}_{\mathrm{MLE}}=\sqrt{\frac{\mathrm{var}\left(Y^w\right)}{n}}$$

(7)

where $\mathrm{var}\left(Y^w\right)$ is the variance of $Y^w$ (see [31]).

To obtain an expression for the reliability of the ML estimate, it is instructive to note that perceptions of the employees from the same team should be more similar than perceptions of employees from different teams. This similarity can be assessed by the Intraclass Correlation, or ICC:

$$\mathrm{ICC}=\frac{\mathrm{var}\left(Y^b\right)}{\mathrm{var}\left(Y^b\right)+\mathrm{var}\left(Y^w\right)}$$

(8)

Thus, it is very intuitive to consider the ICC as one determining factor of the reliability: the larger the ICC is (i.e., the more similar the perceptions are), the more reliable is the ML estimate. In addition, note that other employees would be equally suited as raters of the team leader if they were their employees. That is, the employees in a team are only a sample from a much larger population of raters. Therefore, the number of raters in a team n can be considered another determining factor of the reliability: the larger n, the higher the reliability tends to be (e.g., [26]). In the literature on multilevel modeling (e.g., [32,33,34]), the reliability is thus often expressed as:

$$\mathrm{rel}=\frac{n·\mathrm{ICC}}{1+\left(n-1\right)·\mathrm{ICC}}$$

(9)

It reflects the extent to which the differences in the ML estimates between team leaders can be explained by true differences in leadership behavior. For an equivalent expression obtained from generalizability theory [35], see the formula for Design B in [31].

2.2. Expected a Posteriori Estimate

Using the reliability, the EAP estimate is:

$${\mathrm{EAP}}_j=\frac{n·\mathrm{ICC}}{1+\left(n-1\right)·\mathrm{ICC}}·\frac{{\sum }_{i=1}^n{Y}_{ij}}{n}$$

(10)

and the standard error is:

$${\mathit{SE}}_{\mathrm{EAP}}=\frac{n·\mathrm{ICC}}{1+\left(n-1\right)·\mathrm{ICC}}·\sqrt{\frac{\mathrm{var}\left(Y^w\right)}{n}}$$

(11)

See Appendix B for the derivation.

In the next section, the factor score estimates obtained from lavaan will be investigated with the help of an artificial dataset and compared with “custom-built” estimates.

3. Factor Score Estimates from Lavaan

3.1. Data and Method

Suppose 1000 employees from 100 teams evaluated their team leader by indicating their level of agreement with whether their team leader is able to set goals and to support the team in achieving these goals on a five-point scale from $-2$ (disagree) to 2 (fully agree). Artificial data were generated according to the model in Equation (5), adapting the procedure of [36]. As the number of employees surveyed in a team was 10, and the ICC was assumed to be $0.10$, the reliability was $0.53$. Using these values, each team leader’s true score was simulated, as well as their employees’ ratings. More specifically, each team leader’s true score was drawn from a normal distribution with a mean of $0.0$ and a variance of $0.1$. Their 10 employees’ ratings were drawn from a normal distribution with a mean equal to the team leader’s true value and a variance of $0.9.$

The (grand-mean) centered data can be downloaded at https://figshare.com/articles/dataset/Example_data_of_A_Cautionary_Note_Regarding_Multilevel_Factor_Score_Estimates_from_Lavaan/21613872. Each line contains the rating of a single employee. Column g, the grouping variable, indicates which team leader is evaluated. After having downloaded them, they can be read into R for analysis.

The function lavPredict( ) in lavaan version 0.6–12 [23] is a general way for obtaining factor score estimates from (almost) arbitrary models with latent variables. However, before the factor score estimates can be computed, the model needs to be specified as:

• mlm <- ’
•       level: 1
•          Yw =~ a*Y
•       level: 2
•          Yb =~ a*Y
• ’

Lavaan applies the within–between framework of Mplus, which decomposes variables into within and between components. Accordingly, the syntax has two parts: the within part and the between part. The within part of the syntax contains the definition of a lower-level factor, which is indicated by the observed variable. Its loading is fixed at 1 to identify the metric of the latent variable, which is lavaan’s default strategy. As there is only one item, the error variance is defaulted to 0. In the between part, the higher-level factor, which represents leadership behavior, is defined analogously. However, the indicator’s loading was constrained to be equal to the corresponding loading at the lower level, which is often done in multilevel modeling (e.g., [37,38,39,40]). If not explicitly specified, lavaan would default the loading to 1, which—for this specific model—would lead to the same numerical results. A summary of the output from lavaan is shown in Appendix C.

3.2. Results

Factor score estimates are obtained from the model by the lavPredict( ) function, which allows users to choose between the two discussed types of factor score estimates. ML estimates are obtained by specifying the option method = “Bartlett”, whereas EAP estimates are obtained by method = “regression”. Note that the option level = 2 has to be specified because the factor score estimates of interest are those from the higher-level factor. Normally, applying the attributes( ) function to the output objects would yield their standard errors. However, here, both standard errors are NA. This issue will be elaborated further in the Discussion section.

The obtained factor score estimates look as follows:

•               MLE SE_MLE         EAP SE_EAP
•  [1,]  0.01350424     NA   0.01350424    NA
•  [2,]  0.32330489     NA   0.32330489    NA
•  [3,]  -0.17971389    NA  -0.17971389    NA
•  [4,]  0.03329527     NA   0.03329527    NA
•  [5,]  -0.13949739    NA  -0.13949739    NA
•  [6,]  -0.63425187    NA  -0.63425187    NA
•  [7,]  -0.21836302    NA  -0.21836302    NA
•  [8,]  0.51805248     NA   0.51805248    NA
•  [9,]  0.05040566     NA   0.05040566    NA
• [10,] -0.35429797     NA   -0.35429797   NA
•     .
•     .
•     .

From this table’s columns MLE and EAP, it can be seen that the ML and EAP estimates that the lavPredict( ) function provided are numerically equal, meaning that they cannot be distinguished from one another. This is surprising given that from the definition of the EAP estimate in Equation (3), it immediately follows that the ML estimate should generally be larger (in absolute value) than the EAP estimate by a factor of $1/\mathrm{reliability}$. One would expect the equivalence only if the reliability is perfect; that is, if it is equal to one. However, computing the reliability by Equation (9) using the model results from lavaan yielded a value of $0.60$, which is much smaller than $1.00$. Hence, one or both of the factor score estimates from lavaan must deviate from the expected ones. In order to determine which one deviates, “custom-built” estimates are computedaccording to Equations (6) and (10), using the model results from lavaan (i.e., the estimates of $\mathrm{var}\left(Y^w\right)$ and $\mathrm{var}\left(Y^b\right)$) as plug-in estimates:

•              MLE    SE_MLE          EAP     SE_EAP
•  [1,]  0.02267735  0.301777   0.01350424  0.1797065
•  [2,]  0.54291830  0.301777   0.32330489  0.1797065
•  [3,] -0.30178930  0.301777  -0.17971389  0.1797065
•  [4,]  0.05591196  0.301777   0.03329527  0.1797065
•  [5,] -0.23425468  0.301777  -0.13949739  0.1797065
•  [6,] -1.06508424  0.301777  -0.63425187  0.1797065
•  [7,] -0.36669188  0.301777  -0.21836302  0.1797065
•  [8,]  0.86995334  0.301777   0.51805248  0.1797065
•  [9,]  0.08464504  0.301777   0.05040566  0.1797065
• [10,] -0.59496425  0.301777  -0.35429797  0.1797065
•     .
•     .
•     .

As can be seen from this table’s columns MLE and EAP, the ML estimate is larger than the EAP estimate, which is in accordance with the expectation. Comparing the table’s column MLE with the corresponding column in the table with the results from lavaan, it becomes evident that the ML estimate provided by lavaan deviates from the “custom-built” one. However, whereas the ML estimates differ, the EAP estimate from lavaan is equal to the “custom-built” estimate, indicating that lavaan provides the correct EAP estimate. Thus, it is the ML estimate from lavaan that is possibly incorrect. The findings can be reproduced by the R code in Appendix D.

4. Discussion

Lavaan version 0.6–12 offers the lavPredict( ) function, which could be applied to obtain factor score estimates, even in multilevel modeling, where characteristics of higher-level units are often assessed by ratings of lower-level units. With the help of an example from organizational research, it has been shown that the function provides results that deviate from the expected ones, particularly the ML estimate.

One explanation for the finding that the ML estimate was numerically equal to the EAP estimate is that Yves Rosseel, the programmer and mastermind behind lavaan, has made a mistake. Although this scenario certainly lies within the realms of possibility, it appears not very likely. Another possible, more likely explanation is that the reliability is assumed to be perfect (i.e., no measurement error), and Yves Rosseel himself brought this up after he had read a preprint version of this article (Y. Rosseel, personal communication, 10 December 2022). Notice that the employees in a team were assumed to be exchangeable raters from a larger population of raters, resulting in an imperfect reliability. This assumption can, however, be debated, particularly when one does not want to generalize to a team leader’s true value that would be obtained if all raters from the population of raters evaluated the team leader. For example, rather than in the team leader’s behavior towards employees in general, one may be interested in how the team leader behaves in their specific team. In this case, the formula in Equation (9) does not apply, and the reliability is indeed perfect (i.e., it is equal to 1), yielding:

$${\mathrm{EAP}}_j=1·\frac{{\sum }_{i=1}^n{Y}_{ij}}{n}={\mathrm{MLE}}_j$$

(12)

It is interesting to askwhether this can also explain why lavPredict( ) does not provide standard errors, neither for the ML nor the EAP estimates?

By expressing the standard error of the ML estimate as a function of the reliability and assuming a perfect reliability, the standard errors of the ML and EAP estimates become:

$${\mathit{SE}}_{\mathrm{MLE}}=\sqrt{\frac{1-\frac{n·\mathrm{ICC}}{1+\left(n-1\right)·\mathrm{ICC}}}{\frac{n·\mathrm{ICC}}{1+\left(n-1\right)·\mathrm{ICC}}}·\mathrm{var}\left(Y^B\right)}=\sqrt{\frac{1-1}{1}·\mathrm{var}\left(Y^B\right)}=0$$

(13)

and

$${\mathit{SE}}_{\mathrm{EAP}}=1·0=0$$

(14)

Hence, even if the aim is not generalization across raters (i.e., a perfect reliability), standard errors will not be absent! Rather than NA, the result of the lavPredict( ) function should be zero for each of the standard errors in this case.

My example model is a simple two-level model with only one item. However, more realistic models have more than one item. Thus, it is interesting to ask whether findings would be similar in multilevel Confirmatory Factor Analysis (CFA) models. To address this question, artificial data were generated according to a simple extension of the example model, namely, a two-level CFA model with 12 items, assuming that the items are parallel indicators (i.e., they have equal loadings and equal error variances; see [17,41]). The findings from this model can be summarized as follows. First, as in the simple model, the lavPredict( ) function yielded the correct EAP estimate of the higher-level factor as verified by the custom-built EAP estimate. In the more complex model, the ML estimate differed from the EAP estimate, and it stood to reason that the ML estimate is correct. However, although in this model the reliability was smaller than one, it still deviated from the expected result, and so did the ML estimate. The deviation between the ML estimate and the custom-built ML estimate could again be explained by the fact that the function took unreliability due to surveying only a limited number of raters not into account, whereas it correctly took unreliability due to the items into account. Thus, only in the special case where one does not want to generalize across raters are the ML estimate and its standard error correct! Moreover, it was found that only in this case, the standard error of the EAP estimate was close to the expected one. However, it was slightly larger because the function provides the prediction error instead of the measurement error (for an in-depth discussion of these types of standard error, see [18]).

Conclusions

To conclude, the function lavPredict( ) yields correct EAP estimates of higher-level factors, but given my assumptions, the ML estimates are possibly incorrect. Fortunately, as EAP estimates are essentially Bayes estimates, they can be more accurate than ML estimates (i.e., they exhibit a smaller mean-squared error; e.g., [42,43,44]). Therefore, EAP estimates may be the better choice in diagnostics (e.g., [45,46]) and secondary data analysis (e.g., [9]) anyway. However, lavaan does not provide the expected standard errors, which users should, as a minimum, keep in mind when using them.