Trials and Tribulations of Collecting Evidence on Effectiveness in Disability-Inclusive Development: A Narrative Review
Round 1
Reviewer 1 Report
The key messages of the paper have become lost.Whilst I have scored the paper high on the question above 'Is the work a significant contribution to the field? - this is on the premise that the paper is revised to have key messages clearly articulated.
The paper has several tensions that come through
- Implementation science is broad and as the authors would know, has contentious terminological issues. Impact analysis and evaluation are sometimes used synonymously in implementation science. Though analysis is the evaluation of the outcomes/results and its value for policy and practice. The authors need to be clear on what they are referring to - either impact analysis or impact evaluation and define what they mean. The paper focuses on outcome measures, program logic and tools all elements linked to proximal impact analysis rather than the distal impact analysis – nor evaluation of the impact, which is the conclusion or measure of effectiveness, value to policy.
- The authors have attempted to use the traditional EBM lens and paradigm which established a hierarchy of the value of evidence and recognises its limitations in real world research with complex interventions such as disability-inclusive development. The EBM lens relies/accepts two research types of methodologies – experimental and observational. The authors could accept that this methodology is necessary, useful but adopt a research lens that uses a less traditional and restrictive approach. The paper's topic perhaps demands critique and commentary on why us of ONLY these traditional approaches should not be relied upon, and are not 'fit for purpose' with disability-inclusive development - boldly take a broader view of scientific knowledge and evidence to include the traditional methodologies but also highlight contextual, expert knowledge and experiential knowledge. These last three sources of evidence are critical in the context of inclusive development.
- There is also a tension that comes through in the paper between an advocacy message/emphasis on rights, methodological complexity and common sense in real-world research.
Comments for author File: 
Comments.pdf
Author Response
Reponse: The reviewer made a number of in-document comments, all of which have been responded to.
The key messages of the paper have become lost. Whilst I have scored the paper high on the question above 'Is the work a significant contribution to the field? - this is on the premise that the paper is revised to have key messages clearly articulated.
Response: A key message has been included at the end of each section.
The paper has several tensions that come through
- Implementation science is broad and as the authors would know, has contentious terminological issues. Impact analysis and evaluation are sometimes used synonymously in implementation science. Though analysis is the evaluation of the outcomes/results and its value for policy and practice. The authors need to be clear on what they are referring to - either impact analysis or impact evaluation and define what they mean. The paper focuses on outcome measures, program logic and tools all elements linked to proximal impact analysis rather than the distal impact analysis – nor evaluation of the impact, which is the conclusion or measure of effectiveness, value to policy.
Response: Thank you for this comment and request to clarify our terminology. In this paper we refer to ‘impact evaluations’, of which randomised controlled trials are a type. We did not define what we meant my impact evaluation, and have added the following at the top of page 2:
"Impact evaluations, ideally randomised controlled trials, estimate the effect of an intervention on outcomes that are important to participants and policy-makers.”
In other words: an impact evaluation is set of research activities that are used to estimate the causal effects of an intervention. We have not gone into detail on the designs and methods in impact evaluations, although we have said that they should be used to understand "what works, for whom, under which circumstances, and which levers will help to achieve disability-inclusive development”. We note that the ideal designs will include qualitative and quantitative data; we have added "including the acceptability of the intervention” to the paragraph at the top of page 2.
In point 3, on page 5, we discuss in detail the need to think critically about the outcomes that are chosen, and to be inclusive about what outcomes are thought to be ‘important’. This may interact with the spectrum of proximal to distal outcomes; for example, policy makers may see the ultimate, and therefore ‘distal’ outcome of a programme to be increased economic output, with improvements to participation as a proximal mediator on the way to that outcome, while the project participants may see participation as an end in itself and, in the long run, that economic output is only so useful inasmuch as it supports participation.
- The authors have attempted to use the traditional EBM lens and paradigm which established a hierarchy of the value of evidence and recognises its limitations in real world research with complex interventions such as disability-inclusive development. The EBM lens relies/accepts two research types of methodologies – experimental and observational. The authors could accept that this methodology is necessary, useful but adopt a research lens that uses a less traditional and restrictive approach. The paper's topic perhaps demands critique and commentary on why us of ONLY these traditional approaches should not be relied upon, and are not 'fit for purpose' with disability-inclusive development - boldly take a broader view of scientific knowledge and evidence to include the traditional methodologies but also highlight contextual, expert knowledge and experiential knowledge. These last three sources of evidence are critical in the context of inclusive development.
Response: Thank you for this comment. It is correct that we have restricted our discussion of the ‘evidence’ to the traditional forms of experimental and observational empirical research that are at the heart of the evidence-based medicine paradigm. We take the point that this cannot always be relied on, and indeed will always need to be filtered through the lens of experience and ‘wisdom’ when applied in practice. However, it is our experience that the greatest divergence between disability-inclusive programming and other areas of development programming is the lack of rigorous empirical research on which to build more nuanced recommendations. We have not tried to present the traditional — and defunct — evidence hierarchy as the sole source of scientific knowledge, only that these traditional methodologies have some use that has been under-used in research on inclusive programming. We have called for greater participation of people with disabilities in decision-making, bringing that experiential and contextual knowledge to bear on the problem.
We have attempted to address the tensions highlighted throughout the text.
Reviewer 2 Report
The manuscript is a narrative review posing interesting questions for current and future research when collecting evidence on effectiveness in disability inclusive development. The issues raised in this narrative review will help advance future research since it takes seriously the need to include members of all collectives in the research that is being conducted on the given collective. The manuscript is well presented. The analysis of the data provided is correct. The conclusions of the review are interesting for the readership of the Journal. The conclusions will be food for thought to many researchers and policymakers on the importance of considering disaggregation of data by impairment for instance to have more specific information to be able to provide better responses. This work offers evidence of the necessity to search for new ways of providing aggregated data about health but in a significant approach for disability-inclusive development interventions. The authors have addressed important long-standing questions on research with regards to collecting evidence on effectiveness in disability inclusive development. I would recommend exploring the communicative methodology and its contributions to social impact assessment not to add it to the manuscript but just for the authors information to analyze.
Before publication a few revisions should be addressed:
Pag 5 line 192 Enhanced participation is achieved in three main ways. Then there are only two categories. Change the information in line 192 and write two or clearly state the third way.
Pag 6 line 236 SINTEF should be explained, not everyone who will read the manuscript will know what it is.
Pag 6 line 258 WASH, explain what is it: Access to clean water, adequate sanitation and hygiene.
Pag 8 line 378 and 379 Authors contributions should be revised. The following is not needed: For research articles with several authors, a short paragraph specifying their individual contributions must be provided. The following statements should be used”
Author Response
- Pag 5 line 192 Enhanced participation is achieved in three main ways. Then there are only two categories. Change the information in line 192 and write two or clearly state the third way.
Response: This has been corrected to highlight that it is two main ways.
- Pag 6 line 236 SINTEF should be explained, not everyone who will read the manuscript will know what it is.
Response: We have clarified that SINTEF is a Norwegian research organization
- Pag 6 line 258 WASH, explain what is it: Access to clean water, adequate sanitation and hygiene.
Response: This change has been made.
- Pag 8 line 378 and 379 Authors contributions should be revised. The following is not needed: For research articles with several authors, a short paragraph specifying their individual contributions must be provided. The following statements should be used”
Response: This change has been made.
Author Response File: Author Response.pdf
Reviewer 3 Report
Overall, I really appreciate this article. I think it summarizes some information that is needed in the field, and will make a contribution, especially to people who are new to considering disability inclusive development, and especially DID research.
The article is well written and only minor changes are needed (e.g. line 192 three main ways but only 2 are provided; line 251 extra/missing spaces; line 341 should be "may require" - remove plural from require). References need to be double-checked as several do not seem to be cited correctly.
There does seem to be a heavy use of the authors' own works, and while they are appropriate, perhaps examples from other teams and other studies would be useful to broaden the reach. There are many other examples that could be cited.
Author Response
- Overall, I really appreciate this article. I think it summarizes some information that is needed in the field, and will make a contribution, especially to people who are new to considering disability inclusive development, and especially DID research. The article is well written and only minor changes are needed (e.g. line 192 three main ways but only 2 are provided; line 251 extra/missing spaces; line 341 should be "may require" - remove plural from require). References need to be double-checked as several do not seem to be cited correctly.
Response: Thank you for these kind comments. These corrections have been made.
- There does seem to be a heavy use of the authors' own works, and while they are appropriate, perhaps examples from other teams and other studies would be useful to broaden the reach. There are many other examples that could be cited.
Response: A broader range of references have been included.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The authors have been responsive to review comments and enhanced the paper by so doing. The paper will certainly add to the discourse on disability inclusive development and research. Congrats to the authors.
