Predicting Differences in Model Parameters with Individual Parameter Contribution Regression Using the R Package ipcr

Round 1

Reviewer 1 Report

The paper provides an overview of the individual parameter contribution regression method, along with an application and simulations demonstrating the procedures. I thought it was a very reasonable paper demonstrating software and code that is freely available to everyone. The version that I reviewed had missing citations and figure references, though, so it is difficult to comment about whether the cited literature is appropriate/thorough. This also made it somewhat difficult to know which figure was being described in text.

My primary comment is that the authors could further describe relationships between this method and related procedures (maybe they do, but I could not see the references). SEM has had the "modification indices" for a long time, which seems to be a predecessor to what the authors propose. Additionally, there might be relationships between the bias correction here and the Firth bias correction procedure. Finally, I suspect that there are cases where the Zeileis score tests (from strucchange) are equivalent to the tests that one would obtain from ipcr. This is because Equation (8) is very similar to what enters into the score tests; the main difference is that the score tests do not have the leading "theta-hat" part.

I had one other specific comment about the top of p. 24, and relatedly p. 8: It would be helpful to provide more description about why missing data is difficult to handle. Is it simply not implemented, or are there theoretical issues that are unsolved?

Author Response

Thank you very much for your thoughtful comments and recommendations.

First of all, we would like to apologize for the missing citations as well as figure and table references. We are very grateful that you managed to review the manuscript despite these difficulties.

In the following, we provide a point-by-point response to your suggestions and summarize the changes to the manuscript. Quotes from you are italicized.

Relationships between IPC regression and other methods

My primary comment is that the authors could further describe relationships between this method and related procedures (maybe they do, but I could not see the references). SEM has had the "modification indices" for a long time, which seems to be a predecessor to what the authors propose.

We completely agree that relationships between IPC regression and related procedures should be further described. Therefore, we added a paragraph to the introduction that describes the similarities and dissimilarities between the modification index, expected parameter changes, structural change tests/score-based tests, and IPC regression. The following paragraph can now be found on pages 2 to 3:

„IPCs are calculated by transforming the partial derivative of an objective function with respect to the parameters. Objective functions such as the well-known log-likelihood function are used to estimate model parameters. Various statistical procedures analyze the derivative of an objective function to assess the fit of a statistical model. In the SEM framework, the modification index (MI; Sörbom, 1989) uses the derivative to approximate how much the fit of a model would change after a new parameter is added to the model. As an extension to the MI, which is merely a test statistic, the expected parameter change (EPC; Saris et al., 1987; Saris et al., 2009) has been put forward for obtaining a direct estimate of the added parameter. Although the MI and EPC aim to quantify specification errors, Oberski (2013) demonstrated that the MI and EPC for MGSEMs correspond to IPC regression under certain conditions. However, this equivalency ends in situations that cannot be handled by MGSEMs, such as covariates that are continuous or multiple covariates, making IPC regression a much more flexible method for the investigation of heterogeneity. Other methods that analyze the partial derivative of objective functions are structural change tests (e.g., Hjort & Koning, 2002; Zeileis & Hornik, 2007). Originally used in the detection of change points in time series analysis (Andrews, 1993), structural change tests have been recently popularized by Merkle and Zeileis (2013) and Merkle et al. (2014) to uncover parameter heterogeneity in psychometric models. The difference between structural change tests and IPC regression is that structural change tests provide formal tests whether the parameters of a model are invariant with respect to a covariate. In contrast, IPC regression seeks to model the relationships between parameters and covariates by means of linear regression. In the following, we would like to add some remarks on the Firth bias correction procedure and cases, where the score-based tests put forward by Merkle and Zeileis could be equivalent to IPC regression.”

Firth bias correction procedure

Additionally, there might be relationships between the bias correction here and the Firth bias correction procedure.

We were not aware of the Firth bias correction procedure and are very grateful that Reviewer 1 pointed us towards it. After reading Firth’s 1993 paper “Bias reduction of maximum likelihood estimates,“ we think that Firth’s procedure and iterated IPC regression are distinct. Firth’s procedure seems to be based on modifying the score function in order to decrease the bias of the maximum likelihood estimates. On the other hand, iterated IPC regression removes the bias in a step-wise fashion using a procedure closely related to the Fisher scoring algorithm without modifying the score function. Furthermore, the sources of the biases that both procedures seek to correct are different. Firth’s correction aims at removing a general bias in maximum likelihood estimates, whereas iterated IPC regression removes a bias in the IPCs to a specific parameter caused by heterogeneity in other parameters. The bias of IPC regression is similar to the bias of the expected parameter change (EPC) when the Fisher information matrix differs between null and alternative models (see Saris, Satorra, & Sörbom (1987, page 120-121). The detection and correction of specification errors in structural equation models. Sociological Methodology, 17). In summary, because we do not see a strong correction between Firth’s bias correction procedure and iterated IPC regression, we would prefer not to mention Firth’s procedure in our manuscript.

Possible equivalency between score-based tests and IPC regression

Finally, I suspect that there are cases where the Zeileis score tests (from strucchange) are equivalent to the tests that one would obtain from ipcr. This is because Equation (8) is very similar to what enters into the score tests; the main difference is that the score tests do not have the leading "theta-hat" part.

As pointed out by Reviewer 1, the IPCs, as defined in Equation 8, are very similar to what enters the different score-based tests put forward Merkle, Zeileis, and colleagues. Both approaches scale the scores by pre-multiplying them with the Fisher information matrix. For IPC regression, the scores are scaled to have the same variance as the parameter estimates. For the score-based tests, the scores are scaled so that the resulting standard deviation is unity. IPC regression also centers the scores at the parameter estimates.
Importantly, both approaches analyze the transformed scores differently. IPC regression models the transformed scores by means of linear regression and provides t-values with corresponding p-values to test specific hypotheses about parameter heterogeneity, whereas the score-based tests aggregate the scores into a single test statistic, using the ordering implied by a covariate. Unfortunately, we do not know any cases where the score-based tests and IPC regression provide equivalent results. We investigated such a possible equivalency with simulated data using a simple linear regression model and dichotomous and continuous covariates. However, we could not produce exactly the same results with both methods (same test statistics or p-values). These results lead us to the conclusion that both methods are generally not equivalent. Of course, we cannot exclude the possibility that there are situations in which they are equivalent. If there is interest, we are happy to share the R script with Reviewer 1. 

Lacking support for missing data

I had one other specific comment about the top of p. 24, and relatedly p. 8: It would be helpful to provide more description about why missing data is difficult to handle. Is it simply not implemented, or are there theoretical issues that are unsolved?

Thank you very much for bringing this ambiguity to our attention. Fortunately, no unsolved issues other than time constraints hinder us from implementing support for SEMs fitted on incomplete data. We added the following sentence to page 24 in lines 745 to 756 to clarify that we plan to implement this feature soon:
“We plan to implement support for SEMs fitted on incomplete data in the future.”

Finally, here is an overview of the changes to the manuscript requested by Reviewer 2: 

• We added some words of warning that the heatmap in Figure 3 and IPC regression might produce slightly divergent results. This is because the heatmap visualizes zero-order correlation, whereas IPC regression estimates the partial effects of the covariates on the IPCs. The changes can be found on page 10 in lines 326 to 329.
• Our interpretation of the results of Simulation I were hard to follow. We rephrased the paragraph about the estimated group difference on page 16 in lines 533 to 545.

Reviewer 2 Report

I really enjoyed reading this paper. This paper can be a one-stop entry point for applied users (in particular SEM users) to learn about both the theory and practice of 'icpr'. The paper is well written, and is suitable for publication in 'psych' after a few minor points have been addressed.

1) More guidance is needed as how to interpret the heatmap in Figure 3.  At first sight, this plot seems to suggest that the covariate 'months' is strongly connected to the IPCs of the model parameters 'x2 ~~ x2' and 'visual =~ x3'.  But in the summary() output (on page 10), the  corresponding regression coefficients are very small and not significant. This may be confusing for applied users.

2) I am somewhat worried by the results in Figure 6 (page 16). The pattern of the boxplots is not symmetric, as it should be. In particular, I found it strange that the boxplots for +0.7 (on the far right) were lower than the boxplots for -0.7 (on the far left). The difference can be seen for all three sample sizes and seems to be too large to be only due to sampling variability. Please double-check.

3) All figure and table references were missing. (The references were missing too, but perhaps this was needed for the anonymisation?)

4) This one is just a suggestion. I think it would be nice (and broaden the appeal of the paper) if in section 5 (or in a separate section), the authors  could include a brief example of a linear mixed model, involving the lme4 and merDeriv packages. The paper is now very SEM centric.

Further, a few textual suggestions:

- section 3.1, 4th line: "an objective function are OLS for linear regression": I would suggest replacing 'OLS' (an estimation method) by 'SSE' (the sum of squared residuals, as in eq 11)

- section 3.1, 2nd line under eq (2): "(that is, finding the values for which f is zero)": this is not correct: f (the function value) does not have to be zero; the elements of the first derivative of f() need all to be zero

-page 7, line 212-213: "bread() extracts the Hessian matrix": strictly speaking, this matrix is not always the Hessian; it could also be the  inverse of the expected information matrix; perhaps you can adopt the text from the man page of bread(): "an estimator for the expectation of the negative derivative of the estimating functions , usually the Hessian"

- page 9, Figure 2: explain the meaning of 'phi' and 'psi_1' etc in the figure caption (or remove them from the figure)

- page 11, line 310: the 'header' with the column names of the parameter table looks strange (shifted to the left, Std..Error, Pr...t..)

- page 12, line 389: iteration*s*

- page 12, line 400: coefficient*s*

- page 20, line 603: "t-test": did you mean "z-test"?

- page 25 (Appendix): Appendix B and Appendix C should be subsections of Appendix A; also, the title of Appendix C should be 'Simulation IV'

Author Response

Thank you very much for your thoughtful comments and recommendations.

First of all, we would like to apologize for the missing citations as well as figure and table references. We are very grateful that you managed to review the manuscript despite these difficulties.

In the following, we provide a point-by-point response to your suggestions and summarize the changes to the manuscript. Quotes from you are italicized.

Interpretation of the heatmap in Figure 3

1) More guidance is needed as how to interpret the heatmap in Figure 3.  At first sight, this plot seems to suggest that the covariate 'months' is strongly connected to the IPCs of the model parameters 'x2 ~~ x2' and 'visual =~ x3'.  But in the summary() output (on page 10), the  corresponding regression coefficients are very small and not significant. This may be confusing for applied users.

Thank you very much for bringing this ambiguity to our attention. The heatmap and the IPC regression analysis can differ because the heatmap depicts the zero-order correlations, whereas the IPC regression coefficients are the partial effects. Therefore, we added the following sentences to communicate this fact more clearly (see page 10, lines 326 to 329):

“Note that the heatmap depicts the zero-order correlations between IPCs and covariates, whereas IPC regression estimates the partial effects of the covariates on the IPCs. Zero-order correlations and partial effects might differ, especially if some of the covariates are correlated.”

Asymmetric boxplots in Figure 6

2) I am somewhat worried by the results in Figure 6 (page 16). The pattern of the boxplots is not symmetric, as it should be. In particular, I found it strange that the boxplots for +0.7 (on the far right) were lower than the boxplots for -0.7 (on the far left). The difference can be seen for all three sample sizes and seems to be too large to be only due to sampling variability. Please double-check.

We agree with Reviewer 2 that the asymmetric pattern looks questionable at first sight. However, we are unsure why exactly the pattern of the bias should be symmetric. Nevertheless, we carefully double-checked Simulation I for coding mistakes and repeated the simulation with a larger range of group differences. We found that the asymmetric pattern was even more pronounced for larger group differences. We are happy to share these results with Reviewer 1 if requested.

We chose the values for the group differences in the error variance in a way so that the effect sizes for negative group differences (on the left in Figure 6) and positive group differences (on the right) were equal in terms of their log-variance ratio, that is the logarithm of the ratio of the error variance in group 2 to the error variance in group 1. However, the absolute values of the group differences were quite different: on the left side, the absolute group difference was 0.5, while it was 1 on the right side. Therefore, the group difference was twice as large for the most extreme positive difference than for the negative one. We think that these differences in the absolute size of the group differences were responsible for the asymmetric boxplots.

However, after revisiting the paragraph that described the results shown in Figure 6, we found it a little difficult to understand. Thus, we made some changes and tried to explain the implications of the bias a little bit more clearly. Please find the new version of the paragraph on page 16 in lines 533 to 545.

Example of a linear mixed model using lme4 and merDeriv

4) This one is just a suggestion. I think it would be nice (and broaden the appeal of the paper) if in section 5 (or in a separate section), the authors  could include a brief example of a linear mixed model, involving the lme4 and merDeriv packages. The paper is now very SEM centric.

We completely agree with Reviewer 2 that the manuscript is currently very SEM centric. The reason for the focus on SEMs is that IPC regression is most useful for SEMs because adding covariates to SEMs is often more complicated than adding covariates to other statistical models such as linear regression models or linear regression mixed models. Nevertheless, our manuscript is the first attempt to widen the area of application of IPC regression by discussing linear regression models. We also agree that going one step further and demonstrating the use of IPC regression to uncover parameter heterogeneity in linear mixed models would broaden the appeal of the paper. However, for the time being, we would like to save such a demonstration for a later project. First and foremost, we intended the present manuscript to introduce IPC regression and present the software implementation in greater detail. Adding another demonstration for a linear mixed model would slightly diminish the introductory character of our manuscript since we were required to introduce and discuss some details related to linear mixed models. 

We mostly adopted your textual suggestions and made the following changes:

- section 3.1, 4th line: "an objective function are OLS for linear regression": I would suggest replacing 'OLS' (an estimation method) by 'SSE' (the sum of squared residuals, as in eq 11)

Thank you very much for pointing this out. We followed your advice. 

- section 3.1, 2nd line under eq (2): "(that is, finding the values for which f is zero)": this is not correct: f (the function value) does not have to be zero; the elements of the first derivative of f() need all to be zero

We changed the sentence to: “In practice, f is minimized by finding the roots (the zeros) of the partial derivative of the objective function.” Thank you for finding this mistake.

-page 7, line 212-213: "bread() extracts the Hessian matrix": strictly speaking, this matrix is not always the Hessian; it could also be the  inverse of the expected information matrix; perhaps you can adopt the text from the man page of bread(): "an estimator for the expectation of the negative derivative of the estimating functions , usually the Hessian"

We followed your suggestion.

- page 9, Figure 2: explain the meaning of 'phi' and 'psi_1' etc in the figure caption (or remove them from the figure)

We added the meaning of the symbols to the figure caption of Figure 2.

- page 20, line 603: "t-test": did you mean "z-test"?

We refer to the t-test of the regression coefficients as provide by summary.lm. Are these z-tests? The R documentation also seems to imply that these are t-tests.  

- page 11, line 310: the 'header' with the column names of the parameter table looks strange (shifted to the left, Std..Error, Pr...t..)
- page 12, line 389: iteration*s*
- page 12, line 400: coefficient*s*
- page 25 (Appendix): Appendix B and Appendix C should be subsections of Appendix A; also, the title of Appendix C should be 'Simulation IV'

We corrected all mistakes above. Thank you very much for bringing these issues to our attention.

Finally, as requested by Reviewer 1, we added a paragraph to the introduction that describes the similarities and dissimilarities between the modification index, expected parameter changes, structural change tests/score-based tests, and IPC regression. The new paragraph can now be found on pages 2 to 3 in lines 90 to 110.