A Review of the ‘BMS’ Package for R with Focus on Jointness

Round 1

Reviewer 1 Report

This paper show how to use the R-package BMS for  applying Bayesian Model Averaging. 

The paper is well written and easy to reed. Also, the topic is of potential interest for applied researchers. However, I consider that there are some aspects that need to be improved.

Despite the paper is focused on the usage of the BMS package, I consider that a brief description of the methodology is needed. This description must include at least what posterior distribution are (with a short description of Bayes theorem), the definition of posterior inclusion probabilities and the basis of BMA.   

In section 3.4 I do not understand what do you mean with the sentence: "If the BMS package is used for prediction purposes..." Why do you refer to PIPs when talking about predictions?

The examples and the results need to be described in more detail. Which models are considered? what are fictitious variables? which is the sample size? The number of covariates? which are the economic conclusions that can be extracted from the results. In this regard, more pictures which can illustrate these conclusion are needed.

In the caption of Table 3 you talk about the final model. This a vague reference to what is called the "true model", an essential part of Bayesian model selection. Something needs to be added in this sense. It may be interesting to include a reference to it in the description proposed in my first comment.

Regarding MCMC convergence, can an user have an indicator of the convergence of the chain?  Why Table 4 has less covariates than Table 3 when, in fact, you are talking about a larger model?

Captions of Figure 1 and 2 are wrong. 

Regarding Jointness measurements, There are other packages that can compute them directly? Following Forte et al. 2018, it seems that BayesVarSel can do it. Also, it would be important to add the intuition behind them. Why are they different? do they have the same meaning? 

Following with the jointness measurements, Why the Doppelhofer and Weeks (2009) measurements can not be computed in the example? 

Minor typos: 

When citing a reference using parenthesis, some times the parenthesis appear without any space before them. See, for instance Section 3.1.3 (page 5)

I would remove the code for the function to calculate the measures of jointness including it in an appendix

Pag 19, 3rd line of the last paragraph: It is written hwo instead of how. 

Author Response

Please see attached comments

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper, written by Shahram Amini and Christopher F. Parmeter de- scribes the R package, BMS, that provides a framework for Bayesian Model Averaging (henceforth BMA) for linear regression models. The package excels in allowing for a variety of prior structures, among them the binomial-beta prior on the model space and the so-called hyper-g specifications for Zellner’s g prior. Morevover the provided paper focuses on the joint appearance of the single co- variates within the MCMC sampling procedure. A function is provided that allows to estimate several versions of jointness measures.

I think that the paper has potential to contribute to the literature but at the moment its contribution (or its focus) is to weak to warrant publication in Econometrics. In my opinion the reasons therefore are threefold.

Firstly, in the Journal of Statistical Software (JSS) there is already a paper describing the BMS package, written by Martin Feldkircher and Stefan Zeugner (Bayesian Model Averaging Employing Fixed and Flexible Priors: The BMS Package for R, Journal of Statistical Software, 2015,68 (4)). I strongly propose that the authors disentangle their contributen more from the JSS paper. I.e., they should rewrite their paper to warrant a new contribution to the literature. As mentioned in the title I would propose that the authors focus more deeply on jointness. That leads me on to my second point:

To the best of my knowledge, a vivid discussion on appropriate jointness measure propertiers took place in the literature - with contributions of Ley and Steel, or Doppelhofer (as mentioned by the authors). But Hofmarcher et al 2018 (Bivariate jointness measures in Bayesian Model Averaging: Solving the conundrum) in the Journal of Macroeconomics showed that non of the proposed measures fulfilled the properties. Further they propose a more appropriate jointness measure. I think, the authors should also use this jointness measure and discuss it in their paper. I would also expect that the different jointness measures are used and compared in the empirical example.

Thirdly, I think the authors should remove the code junks from their paper. E.g., the code chunk on page 9 to 11 are standard outputs of the BMS package and should not be discussed in this paper. Moreover 

I would expect that the authors write an add-on package (functions) on the BMS package and provide it as supplemental material. I think this would guarantee sufficient contribution to warrant publication and would increase the likelihood of citation of their work...

Minor comments: 

.)A minor comment: please remove Table 1, which is the ?help output of bms in R.

.)Do not describe each g-prior specification (page 5)

.)remove Figure 1

.) p.14 I would use a table to describe the covariates

 

 

Author Response

Please see attached responses

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I may start saying that the paper has improved in many aspects. For instance, the inclusion of some introduction to the methodology makes it more clear. Also, the description of the examples is now easier to understand.

However, other issues have arisen in this new version with some of them deserving major attention. 

The last paragraph of page 3 is quite confusing and mixes different ideas. You talk about the prior being informative with respect to the likelihood?? That would mean that you inform your prior with the data and that is what you should never do from a bayesian perspective. In fact, the usage of BIC has been largely criticized due to its equivalence to the usage of the UIP prior centered at the MLE of the data. 

In this same paragraph you also talk about a sensitivity analysis (without naming it) and then relate it with the "robustness" of a variable. As expressed here, this idea can be confused with the "importance" of a variable (meaning that the variable may be included in the model), and this is not the case. 

This introductory section needs some more references to the general methodology. Look at the description of the methods in the forthcoming paper of Mark Steel (with a preliminar version available at https://arxiv.org/abs/1709.08221) 

To talk about jointness measurements you need first to introduce what the probability of a variable means (basically the PIPs). 

In the last paragraph of page 6 you talk about substitutes variables. Is this something that has been checked somehow in a paper or is just an intuition? It is very difficult to interpret jointness measurements and I am curious if you find this to be so clear. 

Other minor issues that also need to be taken into account are: 

The usage of \Delta as the symbol for parameters in equations 1 and 2.Notice that just a while after you use \theta_k... why don't you use \theta all the time? It is common practice in the literature about this topic.

The last sentence of section 2 refers to section 4 without acknowledging that there is now the jointness section in the middle.

Also, the text needs to be revised carefully since there are still some typos, for instance in the naming of the package and the usage of the \texttt style. Also, in there is an "infeasible" at the end of  the paragraph before the last one of section 2. 

Author Response

See attached file

Author Response File: Author Response.pdf

Reviewer 2 Report

In the new version, the authors made the jointness aspect much more apparent and fulsome (as I proposed in my review report).

For all the other proposed changes (which are also substantial claims),  the authors replied that the editor suggested to keep it as it is. Given this, I propose that the editor should judge whether the paper is suitable for publication or not.... 

 

Author Response

See attached responses

Author Response File: Author Response.pdf

Round 3

Reviewer 1 Report

Thanks for taking care of all my comments. The introduction and, hence, the paper has now improved a lot.