Bias Discovery in Machine Learning Models for Mental Health

Round 1

Reviewer 1 Report

Authors propose a study on the bias of machine learning techniques when applied to psychiatric data. 
The work is very interesting and it is well written. The problem definition is clear and the experiments have been correctly conducted.

In the following some minor issues that should be addressed to improve the quality and the understandability of the manuscript.

• my main concern regards the fact that different medications have been transformed in the equivalent dose of Diazepam. This transformation introduces a bias in the study itself and it could lead to a lost of information. However, I'm not a medical expert, so my question is: there is a medical reason that allow us to consider as equivalent different medications and thus represent them in terms of equivalent quantity of Diazepam? If so, please clearly explain this point in the manuscript
• Fairness measures have been used and results are shown in tables 5, 6 and 7. These measures must be introduced and described before using them. Please add a brief description of the tool that has been used, and clearly describe the adopted measures, the values they can assume, and how they could be interpreted
• Please refer to recent works on the use of machine learning techniques with psychiatric data (e.g. https://doi.org/10.1016/j.ins.2021.12.049 )
• It is not clear the meaning of the graphs 1, 2, 3, 4, 5, 6. What do you mean with classification threshold? Could you please better explain this concept?

Author Response

Dear Reviewer 1,

Thank you for taking the time to review our paper. We have gone through all your comments and provide
answers below. We hope that you find these satisfactory, but please let us know if there is anything else
that you think we should change. Changes in the manuscript resulting from your comments have been highlighted in blue.

Best regards,

Pablo Mosteiro
Corresponding author

1) Conversion of different medications into Diazepam-equivalent

Thank you for your comment. This is the normal procedure when investigating benzodiazepine use. All benzodiazepines have the same working mechanism. The only differences are half-life time and peak time. So when studying benzodiazepines it is allowed to make an equivalent dose of one specific benzodiazepine. We understand that this might surprise readers, so we have clarified it in a footnote where the conversion is first introduced.

2) Introduction of fairness metrics

Thanks for pointing that out. The metrics were actually described in Section 2.3. We have added a reference thereto at the beginning of Section 4.1.

3) Suggested reference

We have added a citation to that paper at the end of Section 2.1.

4) Explanation of graphs

We have added an explanation at the beginning of Section 3.

Reviewer 2 Report

Thank you for giving me the opportunity to review this research paper written by Pablo Mosteiro and colleagues. The work aims to investigate fairness evaluation and to present bias mitigation strategies by employing a Machine Learning model, trained on real clinical psychiatric data.

I would like to congratulate the authors, as the topic is interesting and relevant and the structure of the paper has a clear and logical flow. The language used is concise. The findings and future work are addressed.

I also have some minor improvement suggestions:

1. In the Introduction, please highlight your original contributions, significance and strengths.
2. A remainder of the paper can be included in the Introduction section as well.
3. A schematic representation of the workflow employed in fairness evaluation and bias mitigation strategies can be added into Materials and Method Chapter in order to enhance the clarity of the approach.

I hope my feedback is useful for the authors in improving their paper and wish them all the best in pursuing this important area of research.

Author Response

Dear Reviewer 2,

Thank you for taking the time to review our paper. We have gone through all your comments and provide
answers below. We hope that you find these satisfactory, but please let us know if there is anything else
that you think we should change. Changes in the manuscript resulting from your comments have been highlighted in brown.

Best regards,

Pablo Mosteiro
Corresponding author

1) In the Introduction, please highlight your original contributions, significance and strengths.

We have edited the last sentence in the next-to-last paragraph in Section 1 to explain that this is our contribution

2) A remainder of the paper can be included in the Introduction section as well.

We have added a paragraph at the end of Section 1

3) A schematic representation of the workflow employed in fairness evaluation and bias mitigation strategies can be added into Materials and Method Chapter in order to enhance the clarity of the approach.

We have added this in Section 2