Explainable Stacked Ensemble Deep Learning (SEDL) Framework to Determine Cause of Death from Verbal Autopsies

Round 1

Reviewer 1 Report

This article focuses on the theme of application of  Artificial  Intelligence (AI) to Verbal autopsies (VA) , commonly used in Low to Medium Income Countries (LMIC) to  determine cause of death (CoD) where death occurs outside clinical settings, with the most commonly  used international gold standard being physician medical certification. Interviewers elicit information  from relatives of the deceased, regarding circumstances and events that might have led to death. This  information is stored in textual format as VA narratives. Method and research approach are clearly explained;  

-Since line 95  the syntax  should be carefully checked (" Other work by [28,37] uses a fusion of CNN and RNN and attain good results using a hierarchical implementation.  [38] investigated multi task learning model for CoD classification. .. Nevertheless, their CNN feed forward network had input of word embeddings and  LDA topics using 10 fold cross validation, and attained a precision of 0.779, recall 0.778 and 100 F1-score of 0.774. However, they used a small dataset and had word clusters that where more frequent and longer than others creating non representative features.).

- table and figures and quite representative;  

- a relevant aspect , in scientific application of AI, is related in this section, that  should  added references appropriately by Authors: 1.2. Explainable AI  Although DL methods have shown promising results in various domains, they are  difficult to interpret. In other words, it is difficult to tell how a model got to a final  prediction, due to their complexity. This is the reason why DL models are know as a black box. 

Minor revision  of English language (syntax) is required. 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Authors,

I wish to express my appreciation for your dedication to advancing scientific research. Your recent article is both captivating and relevant in the current scientific landscape.

In the spirit of constructive feedback, I offer a minor suggestion that, in my humble opinion, could further enhance the scientific merit of your manuscript and pique the interest of the forensic community.

I would recommend considering a more comprehensive introduction to the concept of the verbal autopsy, thereby providing a richer contextual foundation for your readers. For instance, you might consider referencing the enlightening review article published in 2022, which delves into the applicability of the verbal autopsy (freely accessible in full-text): PMID: 36142022.

Once again, I extend my gratitude for your valuable contribution to our field.

Best regards

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

Thank you very much for the opportunity to review this article.

The article describes a complex methodology that is sound in daily professional practice. The manuscript is quite interesting, although some minor aspects could be revised significantly to improve the article's readability.

Some abbreviations need to be explained when they first appear in the text. The authors should review the text and clarify all the abbreviations used the first time they use them in the text of the article.

They should also proofread the manuscript. For example, in lines 164-165 of the PDF sent for review, it is evident that text is missing.

The introduction is engaging and explains the topic quite clearly. However, its length is long, occupying three and a half pages. It would make the article easier to read if the authors reduced its length somewhat. Although long, the authors could devote a little more space to justify their work. This is over-understood in the context presented, but the final part of the introduction would be improved if the authors developed the justification of their work more explicitly. I recommend that they make the objectives more explicit.

In the methods section, it would be correct (and more accessible for readers to understand) if the authors first described the study design. The objectives should be at the end of the introduction. Commonly, by convention, the past tense is usually used to describe the study, and in the submitted manuscript, the authors alternate between the present and the past tense. Another aspect that should be reviewed in the methods section is that the authors offer results. Although this is not a significant flaw, this section should describe what is done and what was obtained in the results section. For example, in section 2.3, the authors describe a process but also show the results of that process. Even if they are simple numerical data, they are results. Describing them in methods is minor, but the methods section should only describe the methodology.

In the methods section, too much theory is offered. I understand that the process and terminology are complex, but the authors present a scientific article, not a book chapter. It is complicated to summarize complex theoretical concepts such as those presented. However, precisely one of the authors' tasks is to reduce as much as possible the contextual information presented so that the core of the manuscript is their results and conclusions. Therefore, they should reduce the theory and explanations as much as possible, simplifying them and using external references. This aspect would speed up the reading of the article.

Narrative comments appear in the text, for example, in lines 348-249. Sections such as methods and results should describe what was done and what was obtained. Text that narrates what is to be done, as in these lines, only increases the length of the manuscript and thus slows down the reading. Line 393, for example, also makes a qualitative assessment, stating that a process is vital. In methods, one should only describe what has been done to make the work reproducible. The assessments, especially the qualitative ones, should go into the discussion. I recommend authors review the entire narrative or assessment text in purely descriptive sections such as methods or results.

In the discussion, the authors should elaborate more on the limitations, both those inherent to the study design and those inherent to the methodology. For example, there is always an unavoidable selection bias that should be evaluated to contextualize the validity of the conclusions obtained.

There are some minor typographical errors.

The manuscript requires moderate revision in terms of general writing and English usage, including chopped sentences, minor typographical errors, lack of abbreviation explanations, and other flaws that make it somewhat annoying to read. Addressing these minor issues will make the text much easier to read.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

The manuscript "Explainable Stacked Ensemble Deep Learning (SEDL) Framework to Determine Cause of Death from Verbal Autopsies" by Mapundu et al. is dedicated to the proper cause of death investigation when real autopsy is not an option. The paper is well written and can be read by a wide scope of specialists in various fields (medics, lawyers etc.).

To achieve their goals, the authors created standard clean data frames from the field observations. Classifications constructed by the authors included 12 disease types. Data was adequately preprocessed and balanced. Learning curves are plotted to demonstrate the quality of the constructed model.

The manuscript can be published in "MAKE" because it is very interesting and well-written. The problem of standardized verbal autopsy reports, that is solved by the algorithm described in the paper, exists in some remote locations. The results of the study are described thoroughly and can be used by other researchers.

The main weakness of the manuscript in my opinion is the lack of the comparisons with other techniques used in analogous situations. Authors claim that their algorithm allows (among other advantages) to overcome errors of human-performed verbal autopsy. However, no data (as far as I can see in the text) supporting this proposition is provided.

Some technical inconsistencies are present in the text:

Line 27 - "0. Introduction".

However, in my opinion, few weak points of the article can not be viewed as an obstacle for publication of the manuscript in the "MAKE" journal.

Author Response

Yes, agreed there is little research done to date that uses deep learning techniques to determine causes of death from verbal autopsy.

However, we make reference to our initial study which applies conventional machine learning approaches to determine causes of death, as well as other studies that have used machine learning approaches in this context.

We cannot share data because of confidentiality purposes, unless explicitly requested.

Yes, agreed the Latex template used automatically numbers headings and seems it started from 0. I presume this will be resolved during manuscript processing.

Author Response File: Author Response.docx

Reviewer 5 Report

Authors present a robust deep learning techniques for the identification of Causes of Death (CoD) from Visual Assessments (VAs). Among these techniques, the Stacked Ensemble Deep Learning (SEDL) methods demonstrated optimal performance, surpassing Long Short Term Memory (LSTM) and Convolutional Neural Network (CNN) models. Our empirical findings indicate the potential for integrating ensemble deep learning methods into the CoD assessment process to aid experts in reaching diagnoses. Ultimately, this integration has the potential to reduce physicians' narrative review time, leading to cost savings and error reduction while facilitating accurate diagnoses.

Article is fine, well written. The use of CNN appears to be obsolete but here it is inserted in a broad context. english is ok

Author Response

Thank you for your feedback, it is really appreciated.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Verbal autopsies (VA) are commonly used in Low to Medium Income Countries (LMIC) to determine cause of death (CoD) where death occurs outside clinical settings, with the most commonly used international gold standard being physician medical certification. Interviewers elicit information  from relatives of the deceased, regarding circumstances and events that might have led to death. This  information is stored in textual format as VA narratives. The narratives entail detailed information  that can be used to determine CoD. However, this approach, still remains a manual task that is  costly, inconsistent, time consuming and subjective (prone to errors) amongst many drawbacks.

The paper in its present form should be accepted for publication.

I have no concerns about the use of the English language.

Author Response

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Authors,

I would like to extend my heartfelt appreciation for taking the time to incorporate my suggestions into the article. Your efforts have truly enhanced its quality.

Great work, and kind regards

Author Response

Thank you for your feedback, it is really appreciated.

Author Response File: Author Response.docx

Reviewer 3 Report

Thank you for the opportunity to review the modifications made to this article.

The authors have made most of the proposed changes. Those they have not completed are minor, and they also adequately justify not making them.

Therefore, the manuscript has improved its readability. Without losing its original quality or technical background, it is now a more readable text that can potentially reach more readers.

Author Response

Author Response File: Author Response.docx