1. Introduction
It is usually challenging for researchers to know the exact number of questions that will be answered in a survey, which can lead to incomplete data. Such a challenge can be overcome by either imposing a limit to the number of questions in the survey or using a planned missing design. The latter allows the questions in a survey to be divided into groups, with each person only answering some of these (a subset of groups). This strategy allows collecting data from all questions in the survey, thus avoiding nonresponses and invasive questionnaires.
In a planned missing design, the nonresponses occur according to the will of the researcher. The goal of planned missing design is to ensure data quality with lower effort of the participants [
1,
2,
3,
4]. However, considering how missing data can negatively influence the results of the analysis [
5], some researchers avoid planned missing design.
According to Enders [
5], the use of contemporary methods for handling missing data, such as full information maximum likelihood (FIML) and multiple imputation (MI), allows researchers to analyze and fit models without excluding incomplete cases. These techniques incorporate the omissions in the study designs and thus allow for incomplete questionnaires. Such methods for handling missing data are theoretically appealing because they require weaker assumptions about the causes of missing data. Here, I use the FIML method on handling missing data.
Furthermore, the structural equation model (SEM) is a modeling technique very popular in different fields, particularly education, psychology, and marketing [
6,
7]. Fitting an SEM involves finding estimates for the model parameters that result in a variance–covariance matrix with the best fit to the theoretical model.
Many fit measures have been developed and implemented in different statistical software packages when using a SEM. Bandalous and Finney [
8], Hair and colleagues [
9] and Nye [
10] recommended that more than one fit index should be reported when studying the quality of model fit in applied research. Conversely, Schumacker and Lomax [
11] have shown that additional indices might depend on whether the researcher is interested in model fitting, model parsimony or model comparison. The fit indices usually used in this type of modeling are the root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI) and Tucker–Lewis index (TLI).
There are several studies that aim to evaluate the effect of the data non-normality, sample dimension, model type and factor loading values on the fit indices used in an SEM [
12,
13,
14]. Other studies were conducted using different estimation techniques in the adjustment of a structural equation models [
15]. The effect of model misspecification, is also discussed by some authors, including Fitzgerald and colleagues [
16], Maydeu-Olivares [
17] or Zhang and Savalei [
18]. Some other work on the behavior of fit measures under the existence of nonresponses in SEM with ordinal data has been published by Liu and colleagues [
19,
20], and some others have proposed correcting some fit indices [
16,
18,
21]. In contrast, Jia and colleagues [
6], Moore and colleagues [
7] and Schoemman and colleagues [
22] studied the effect of a planned missing design in the parameter estimate bias, standard error bias, model convergence and power analysis. The effect of omissions by design in SEM fit measures was not considered despite its importance in multiple fields.
Omissions by design consist of deliberately and randomly omitting some questions from a survey for some participants, with the goal of avoiding extensive questionnaires and ensuring participant engagement. Indeed, if the demands on participants are too high, both the quality and quantity of the responses will decrease. Examples of these kinds of omissions are the three-form design and the two-method presented by Graham and colleagues [
1].
As such, I aim to evaluate the effect of the existence of omissions by design on the adjustment of a structural equation model with distinct sizes. Also considered are different sample sizes, parameters values and some model misspecification.
I start by presenting the used planned missing design, followed by the model under analysis, the respective fit measures, and the simulation conditions of the performed work. The results of the Monte Carlo simulation study are then detailed. Finally, the results are discussed, and the conclusions are established. This simulation study was performed with the package simsem in R [
23].
3. Results
The obtained results of the proposed simulation work are presented in
Table 3,
Table 4,
Table 5 and
Table 6. All of them present the mean value of the indices of interest obtained from the 1000 replications, from which it is possible to conclude that, when the sample size increases, all the indices present better values: CFI and TLI increase and SRMR and RMSEA decrease. In general, the results are worst with nonresponses when considering a misspecified models versus correctly specified models.
In
Table 3, I present the obtained CFI values. With low factor loadings (0.4) and small sample size (200), it is possible to see that the CFI values are below the cutoff when there are missing data. In big models, even with complete data the obtained, the CFI value is under the acceptable (0.933). If the factor loadings are high (0.8), the CFI values obtained are acceptable despite the existence of nonresponses.
The CFI values for models with strong correlation between factors (0.9) show that nonresponses in correctly specified models causes values to be below the cutoff (0.942), in big models (36 indicators) with low factor loadings (0.4) and small samples. Independently of the factor loading values or sample size, omissions in models with misspecification causes unacceptable CFI values. The worst results are for models with low factor loadings and medium number of indicators, when this index takes values of 0.706, 0.724 and 0.726, according to the different samples sizes (200, 500 and 1000).
In
Table 4, I show the obtained results for the TLI index. When a weak correlation between factors is considered, the existence of nonresponses causes TLI values to be below the cutoff in large models (36 indicators) with factor loadings equal to 0.4 and small sample sizes. Even with complete data, the obtained value is under the acceptable (0.932). For models with strong correlation between factors, the existence of omissions in models with misspecification leads to bad TLI values (below the desirable), whatever the factor loading values, model size or sample size are. TLI lowers with smaller factor loading values. The worst situation happens when the factor loadings are low and the model is of medium size (18 indicators), taking the values 0.666, 0.687 and 0.690, according to sample dimension.
When analyzing
Table 5, in small samples with low factor loadings and weak correlation, the SRMR values are above 0.05 with both complete and missing data in correctly or misspecified models. With factor loading value of 0.8 and small and medium sample sizes, only when there are omissions with misspecified models, the SRMR index takes values above the acceptable, although the obtained values are worse in larger models. The results of SRMR index for models with strong correlation between factors show that the existence of omissions in correctly specified models only causes unacceptable values when the sample is small. However, if there are omissions when considering misspecified models, regardless of the sample size or the model size, the SRMR index values are far above the acceptable. If factor loadings are high, the existence of nonresponses with misspecified models causes the index’s values to be much higher than 0.05. The worst situation happens when a big model is considered, with the values lying around 0.392.
The obtained values for RMSEA index are presented in
Table 6. When in the presence of a medium sized model with low factor loadings and a strong correlation between factors, this index presents values above the cutoff if there are nonresponses with misspecified models, regardless the sample size. Also, with high factor loadings and models of medium size, RMSEA values are unacceptable, around 0.097. If the model is of big size and small sample size the obtained result is 0.056, slightly above the desirable.
4. Discussion
This study explores the effect of nonresponses in the most popular fit measures of a structural equation model. Missing data are generated according a three-form design, in which the nonresponses are due to the will of the researcher. The purpose of using such a design is to increase the quality of the data, avoiding the effort of inquiry and the consequent non-response. However, there is low usage of this type of design, according to Enders [
5], “Planned missing designs are highly useful and underutilized tools that will undoubtedly increase in popularity in the future”. Another procedure presented in psychology or marketing studies, to avoid nonresponses, is the forced answer options. However, the costs of the effect of using this method in terms of quality reduction and dropout rate may be high.
Also, in this study, I considered correctly specified and misspecified models with misspecification in the correlation between factors, as in the works of Shi and colleagues [
14], Fitzgerald and colleagues [
16], McNeisch and colleagues [
55].
The findings regarding the effects of this type of missing values on the RMSEA index, only show values above the cutoff, when the correlation between factors is strong in misspecified models with high factor loadings and medium sized models, or in small samples with large models. The RMSEA index did not show to be affected by nonresponses in correctly specified models, which has been noted by Davey [
57] and Hoyle [
58]. In their work, Zhang and Savalei [
59] explored the existence of a bias in the RMSEA values, concluding that factors such as the missing data mechanism and the type of misspecification are determinant for it. Recently, Fitzgeral and colleagues [
16] proposed a correction for the RMSEA index to overcome this issue. The work of Lai [
21] follows the same purpose.
The SRMR values showed this index to be affected by nonresponses, particularly with small samples, low factor loadings and with medium and large number of observed variables in the model. In these cases, it presents values above the cutoff, even for complete data. However, if a misspecified model with data from a three-form design in small samples is considered, the SRMR values are the worst. With high factor loadings, even with medium and large samples, the SRMR is above the acceptable. As stated by Jia and colleagues [
6], when using a planned missing design, it is not simply that a larger sample is better, it depends on the other conditions of the study.
The CFI index is affected by a three-form design, particularly, with small samples, low factor loadings and high number of observed variables (large models). In this case, it presents values under the cutoff, even for complete data. As presented by Shi and colleagues [
14], CFI showed to be affected by the size of the model when data from a three-form design are considered. If there is a misspecification in the correlation between factors, the CFI index presents unacceptable values, despite the sample and model size as well as the factor loadings assumed. Zhang and Savalei [
18,
59] point the use of the estimation method FIML for handling missing data as the cause of the distortion in CFI and RMSEA values.
For all the considered indices, a larger number of observed variables in the model causes worse values, lower for the CFI index and higher for the RMSEA and SRMR indices. Additionally, models with low factor loadings have worse values for the considered fit indices. These results for all the four indices are in line with the work of Shi and colleagues [
14]. According to these authors, small sample sizes, high number of observed variables in the model and lower factor loadings lead to a worse fit. In the same sense is the work of Moshagen [
60], which shows the effect of the sample size and model size on the fit indices.
When factor loading values are low, in small samples, the obtained SRMR and CFI values suggest a weak fit model-data. According to McNeish and colleagues [
55] and Hancock and Mueller [
61], there is another measure that is important to pay attention to when performing a study, measurement quality, considered as the strength of the standardized factor loadings by the authors. In McNeish and colleagues [
55] work, they show the large influence measurement quality (factor loadings values) has on fit indices values and how greatly the cutoffs would change if they were derived under an alternative level of measurement quality. They highlight the fact that the cutoffs established by Hu and Bentler [
45] are derived from models, most of them with factor loading values of 0.7 and a few with 0.75 or 0.80, considered high values for factor loadings. So, the findings of this study do not seem to be aligned with the reliability paradox presented by Hancock and Mueller [
61].
The indices CFI, TLI and RMSEA are affected in the same way by a three-form design, when the correlation between factors is weak, regardless of the model being or not correctly specified. However, for the SRMR index, this only happens with low factor loadings. According to Themessl-Huber [
62], all statistics have problems detecting misspecified models when factor loadings are low. These findings in models with low factor loadings are particularly important, as in psychology, where studies are performed with a mean value for factor loadings, according to Peterson [
63], of 0.32. As such, there is the need to look to all the assumptions of the study, as suggested by Jia and colleagues [
6]. When the correlation is strong, all the indices are affected by the model misspecification, regardless of sample size.
Considering large sample sizes, with data from a three-form design, for different models and parameter values, all the four fit measures did not appear to be affected as they all presented acceptable fit values (good fit) in correctly specified models. Furthermore, the RMSEA, TLI and CFI indices presented similar results with complete and missing data, whereas the same does not happen with the SRMR index. In the study of Cangur and Ercan [
15], with complete data, the SRMR show to be the index with worst performance. As in the work of Fan, Thompson and Wang [
64], all the fit indices showed to be affected by the sample size, particularly by small samples.
On the other hand, it was possible to see that the number of indicators in the model (model size) impacts the fit measures, particularly, CFI, TLI and SRMR, as stated by Shi and colleagues [
14], when considering misspecified models. For RMSEA the effect is the opposite to that stated in the work of these authors.
This work has some limitations that can be further improved. The findings are based on a particular three-form design, as presented by Schoemann and colleagues [
22]. This three-form design has a specific number of variables in each set X, A, B and C, resulting in a percentage of missing data in the data base of approximately 11.1%, and, despite the number of indicators considered in the model, this percentage was maintained so the results are comparable. In future works, it would be interesting to increase the percentage of nonresponses by manipulating the number of indicators in each set of variables and considering ordinal missing data.
Considering the two-method suggested by Graham [
1], it would also be interesting and valuable to compare the obtained values, because the study of omissions by design has been underestimated, particularly with misspecified models.
Finally, considering other structural equation models frequently used in psychology studies and understanding the behavior of the fit measures when the data present omissions by design is also challenging.
5. Conclusions
The present work aims to help applied researchers that are using structural equation models with data obtained from a particular planned missing design, a three-form design. The later design allows us to reduce the number of questions in a survey that each participant answered without reducing the overall number of questions in the survey. Consequently, this reduces the time and burden needed to complete each survey, improving the quality and the quantity of the obtained responses. They are particularly important in psychological research, where time and resources are limited. However, is important to understand the consequences of using such a design in the usual fit measures considered in the evaluation of the adjustment of a SEM.
The results obtained in the current study showed that for small samples, TLI and CFI have a similar behavior with omissions from a three-form design with correctly and misspecified models. They have worse values with a strong correlation between factors despite the model size and the factor loading values. The SRMR index is affected by a three-form design with correctly specified models when the factor loadings are low and if the factor loadings are high in misspecified models. The RMSEA index shows a bad fit only if it considers misspecified models with strong correlation between factors.
All the four fit measures did not appear to be affected for the existence of nonresponses from a three-form design, and all of them presented acceptable fit values (good fit) with correctly specified models when considering large sample sizes despite different models or parameter values. However, the existence of a model misspecification leads to worse values for the fit measures, particularly when there is a strong correlation between factors.
The obtained results highlight the necessity of considering nonresponse patterns during model fitting with misspecified models and factors with strong correlations.