Training for Tuberculosis Elimination in Indonesia: Achievements, Reflections, and Potential for Impact

Round 1

Reviewer 1 Report

Review : The main issue with this paper is the lack of a clear statement of the aims or objectives of the paper. The paper is structured and reads like a report of the training, with a focus on the process and the learning outcomes. While potentially of interest, the link between the training and program outcomes is likely to be of greater interest to readers, and more linked to the overall supplement focus on TB elimination.

Response: clearer statement of the paper’s objectives provided with more focus on the links between the training and TB programs

Review: As a report of learning outcomes, the paper is hampered by the lack of an objective measure of these outcomes, with only self reported changes in knowledge as the measure. While the training methodology demonstrates aspects of best practice, without some objective measure of the learning outcomes, it is difficult to determine its effectiveness.

Response: The paper clarifies that this is a process evaluation and is explicit on the limitations of the evaluation of training effectiveness

Review In terms of program outcomes, again there is no evaluation of the post program project implementation, or contribution to TB elimination, has been undertaken.

Response: This remains a weakness – with only some report of project implementation, but no evaluation of project contribution.

Review The focus on development of small group projects is one of the key innovative elements of the training. The authors could consider greater focus on this element. Even if it is premature to measure the outcome of the projects, the issues selected, the approaches proposed, and their potential contribution to addressing key gaps / problems in TB programs could be described. While Table 2 lists the project titles, there is no information on the rationale / problem being addressed, or the relevance, feasibility, and potential contribution of the projects to TB elimination.

Response: The expanded Table 2 provides a rationale and summary of achievements (in terms of implementation) of each project. The variety and innovation in the projects are some of the most interesting parts of the paper.

Review: A second recommendation relates to the structure of the paper. One of the key elements that would assist a reader in understanding the program, is a more specific statement of the objectives of the program.  The current statement of objectives refers to ‘skills in areas such as’; and to developing and implementing  ‘a project’. More specific objectives could provide a rationale and explanation for the specific features of the design and methodology, and better link to the stated outcomes such as ‘inform TB policy and practice’, and the development of ‘people to people and institutional links’.

Response: This has largely been addressed with additional more specific statements of objectives and outcomes

Review: What are the specific features of the Indonesian TB program that require this approach? More description of the current Indonesian TB program and the specific problems / issued with HR capacities would provide a clearer basis for the program design.  More specific objectives would also provide a rationale for the selection of participants,  the selection of training topics, training methodology, and, in particular, the focus on design and implementation of projects. The location of the courses, the pre and post workshops, and the development of networks / institutional linkages could then be linked with the context, needs, and objectives.

Response: This has been addressed with additional information on the problems and issues for the Indonesian TB program

Review: The section on ‘lessons learned’ gives the impression that the rationale for the design and methodology features emerged as lessons learned after the course, rather than informing the planning of the course. A clearer statement of the objectives and specific contextual issues, and using these to explain / justify the design would make more sense.

Response : The objectives now refer to ‘reflections’ but it is not clear where the reflections are in the paper. Section 3 on strengths and limitations could be considered ‘reflections’ and may be better included under discussion.

Review: The discussion session is short. Table 3, currently included as results, would be more appropriately included in discussion, as most of the content relates to reflection on elements of the course in the light of current recommended training methodology and experience elsewhere. The discussion and conclusion need to link more clearly to the objectives for the paper.

Response: Table 3 has been moved to the discussion section

Suggested revision

As noted above, to explicitly state the section dealing with reflections – suggest that this be section 3 on strengths and limitations.

Author Response

Note that this was the first round of reviews from the 15th June, 2019. From correspondence with Anna Liu, this reviewer has already approved the paper. Please see below.

Dear Steve,

 

We received 1 report for your original reviewer. When I received the revised version tropicalmed-538641, I sent it to the original reviewer,and he give "accept" report. But if we process it further, we need at least 2 reports. So we added more reviewers. Than we collected a "reject" report. Whit "reject+accept" reports, we need 3rd report. So we  added more reviewers. And than we received a "minor" and "major" reports. That's why we added 3 new reviewers. Actually, we only need 2  new reviewers, but 3 reviewers agreed to review and submitted their reports on time.

  Best regards,

   Anna

Author Response File: Author Response.docx

Reviewer 2 Report

Training procedure for TB elimination is innovative, unique and comprehensive. The manuscript explains in detail with too long. Some facts are repeatedly explained. So it can be revised as short and precise.

Author Response

Please see attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

A. General Comments

The paper is a report on a series of TB courses held for health care staff in Indonesia and is mostly a description of the course content although does also entail an evaluation in the form of a questionnaire survey of 75 of in total 76 participants in the course.

There is no research question formulated, but aims of the study are provided, and are multiple: To provide a description of the purpose, design and delivery of the training, reflect on lessons learned from the training, discuss the effectiveness of training methodologies, and highlight key challenges in evaluating trainings. These aims are all well dealt with, but are not very standard objectives for a scientific study, but rather an evaluation report to funders or policy makers. There seems to be very little original data collected and if so, it is not presented in the paper. Hence, the Results only very limitedly answers the questions raised.

The design of the study is not described, but seems to be a cross sectional survey, although the focus is only to a very limited extent on the data collected in the survey, rather than reporting on the content of the course provided. A later qualitative study based on interviews of the participants seems to be in planning, but is not reported here.

The very lengthy paper contains little information which carries importance to the general reader or clinicians in the field, but may be of relevance to policy makers. The article is not really within the TMIH scope nor sufficiently novel and interesting to warrant publication

The main conclusions are that a process evaluation of the training was provided and inherent strengths and limitations in delivering effective, relevant training with genuine and lasting impacts were identified; a conclusion which is only to a limited extent supported by the presented data. It is further concluded that achieving the END TB targets in Indonesia and the wider Asia-Pacific region will require scale up and implementation of new evidence-based strategies, a conclusion which is not really substantiated by any presented data. Finally, the authors conclude that critical component of this implementation is the scale up of human resource capacity through effective training and that the model the authors describe here offers such an example; again a conclusion which may not call for major disagreements, but neither of these statements are supported by any data presented in the paper.

 

B. Major Comments

1. If indeed intended to be reported as a cross sectional survey, then STROBE guidelines should be adhered to, eg describe the design in title and methods, sample size and a flow chart of the selection of informants.

2. Participants are not well described, a table with characteristics would be expected. 76 were selected from 817 applicants, but incl/exclusion criteria are not descrited. Were they representative? 

3. The main outcome measure is not well described, but seems to be self-perceived knowledge increase and acceptability as well as how satisfied participants were with the course. These aspects are briefly mentioned in the Results, but mainly described in to supplementary tables, which is somewhat odd, as they contain the main data of the manuscript, and should therefore not be supplementary but main tables. The course content tables should be supplementary, or better: dropped. The S1+S2 tables are not very helpful as they only contain median scores and does not display how many answered what to the questions.

4. Few conclusions can be made from the limited data presented, and the current conclusions are not warranted.

Author Response

Please see attachment 

Author Response File: Author Response.pdf

Reviewer 4 Report

This is a well-written paper on an interesting topic. It is certainly time to start thinking about TB elimination planning, though harder to imagine in the global context for countries like Indonesia.

The paper is lacking in results and there are not many specific on how to incorporate the interventions presented programmatically. The paper might benefit by being shortened since it is lacking data and/or interventions.

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

I had not understood that this was a invited commentary. I think the revisions are fine, and I have no further comments.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

General comments

The main issue with this paper is the lack of a clear statement of the aims or objectives of the paper. The paper is structured and reads like a report of the training, with a focus on the process and the learning outcomes. While potentially of interest, the link between the training and program outcomes is likely to be of greater interest to readers, and more linked to the overall supplement focus on TB elimination.

As a report of learning outcomes, the paper is hampered by the lack of an objective measure of these outcomes, with only self reported changes in knowledge as the measure. While the training methodology demonstrates aspects of best practice, without some objective measure of the learning outcomes, it is difficult to determine its effectiveness.

In terms of program outcomes, again there is no evaluation of the post program project implementation, or contribution to TB elimination, has been undertaken.

Thus in its current form, it is difficult to recommend publication. However, there is clearly a need for more knowledge on the effectiveness of the many training courses that have been undertaken, and to determine whether this model of training offers advantages in terms of addressing workforce capacity constraints, and contributing to program effectiveness.

I suggest that the authors need to re-consider the purpose and objectives of the paper. Two options to consider are: Is it just a process evaluation of the training ? or does it / can it address the question of how training of the TB workforce can be designed to more explicitly contribute to TB program outcomes?

The focus on development of small group projects is one of the key innovative elements of the training. The authors could consider greater focus on this element. Even if it is premature to measure the outcome of the projects, the issues selected, the approaches proposed, and their potential contribution to addressing key gaps / problems in TB programs could be described. While Table 2 lists the project titles, there is no information on the rationale / problem being addressed, or the relevance, feasibility, and potential contribution of the projects to TB elimination.

A second recommendation relates to the structure of the paper. One of the key elements that would assist a reader in understanding the program, is a more specific statement of the objectives of the program.  The current statement of objectives refers to ‘skills in areas such as’; and to developing and implementing  ‘a project’. More specific objectives could provide a rationale and explanation for the specific features of the design and methodology, and better link to the stated outcomes such as ‘inform TB policy and practice’, and the development of ‘people to people and institutional links’.

What are the specific features of the Indonesian TB program that require this approach? More description of the current Indonesian TB program and the specific problems / issued with HR capacities would provide a clearer basis for the program design.  More specific objectives would also provide a rationale for the selection of participants,  the selection of training topics, training methodology, and, in particular, the focus on design and implementation of projects. The location of the courses, the pre and post workshops, and the development of networks / institutional linkages could then be linked with the context, needs, and objectives.

The section on ‘lessons learned’ gives the impression that the rationale for the design and methodology features emerged as lessons learned after the course, rather than informing the planning of the course. A clearer statement of the objectives and specific contextual issues, and using these to explain / justify the design would make more sense.

The discussion session is short. Table 3, currently included as results, would be more appropriately included in discussion, as most of the content relates to reflection on elements of the course in the light of current recommended training methodology and experience elsewhere. The discussion and conclusion need to link more clearly to the objectives for the paper.

More specific comments on each section follow:

Abstract

Aim / objective of paper not stated

Not clear how the training contributed to the TB program

Introduction

Line 39 Refers to the need for a paradigm shift – how did this program address/ contribute to enabling this shift /What is required for the paradigm shift.

Line 42-45 Role / effectiveness of training in address HR quality issues is doubtful – issues of incentives, supervision, resources also important.

Line 46-52 Rationale for this training in context of Indonesian TB program not explained – source of request / training needs / expected outcomes

Line 53 This statement of the paper’s structure still lacks clarity on the purpose of the paper, and reads more like a report. Can you provide a purpose that links more explicitly the training design and process with the issues / needs of the TB program ?

Line 66 How was course content determined ? how does the content relate to needs ?

Table 1

Goal includes a variety of roles within TB programs and also refers to partnership with Australian professionals and institutions. What is the rationale for the variety of participant roles and the Australian partnerships ?

Methodology refers to projects ‘to inform policy and practice’ – again what was the rationale for the use of projects ; how were they linked to policy and practice needs ?

Course team: ‘multidisciplinary’ – all though all are referred to as medical doctors. In what sense was the course multidisciplinary ?

Design: The reference to identified specific learning objectives is important – more information is needed on how the learning objectives were identified, and some examples / outline of the scope of the objectives.

While the teaching and learning methods reflect best practice, further explanation of why these approaches were selected and what more general learning / behavioural outcomes were sought through these methods in addition to the specific TB program related outcomes.

Content: More explanation / justification for the content, and in particular the second element of skills in project development, is needed. What was the learning objective of the projects ? What were the programmatic objectives of the projects ?

Participant selection: Selection of participants does not seem to be related to the levels of skills or knowledge; or the HR capacity deficits in the Indonesian TB program. Further explanation and rationale for the selection criteria, that linked to the objectives, would be helpful.

The second paragraph suggests that proposed project titles were included in the selection process. More information and explanation on the rationale for this, and what the applicants were asked to do in terms of the projects would be helpful. The selection criteria are useful, but again, would make more sense if the rationale and linkage with the expected outcomes was more explicit.

Outputs: This section provides more detail on the project development element of the program – but would be strengthened by a clearer and more explicit explanation of what the program hoped to achieve through the projects. Were they mainly a learning device ? Were they expected to address gaps / issues in the TB program ? If so what, if any, resources were available ? How were the projects linked with national / local priorities, strategies and resources ?

Evaluation: This is currently focused on self assessed knowledge and satisfaction with the course. While useful, there does not appear to be any objective assessment of changes in participant knowledge or skills and this undermines any claims for the course’s outcomes.

Table 2 Project titles. More information on the specific problem the project aims to address, the proposed approach, and the expected outcomes / results would strengthen the potential links between the projects and the TB program needs.

Page 7 Lessons learned – This section more appropriate belongs as discussion, rather than results

Line 94 ‘the training was successful’ – success would normally be assessed in terms of the objectives and expected outcomes. This is difficult to judge due to the very general nature of the objective statements. Further justification is needed for this statement – particularly to link it more explicitly to the ‘technical and programmatic knowledge’ gained, and to some criteria in regard to the projects. The addition of reference to networks created and motivation is useful – but if these were elements of the program’s success, then they should be reflected in the program’s objectives/ expected outcomes.

The lack of any objective measure of knowledge / skills developed weakens the statement of the training success.

Line 100. Involving local experts. This was not mentioned in the description of the course team and could be added there.

Line 110. Participant selection. The reference to ‘making a positive difference to TB programs and outcomes’ introduces another potential objective / outcome for the program which has not been included in the earlier description of outcomes. Introducing these concepts earlier as part of the rationale and objectives would clarify the rationale for the design and approach.

Page 8 line 118. The mix of skills and expertise among the participants is an important issue and could potentially undermine achieving some of the specific technical capacity improvements.

Page 8 line 120 ff Training methodology. Some important issues raised here in regards to the rationale for the program design. For the reader unfamiliar with the context /course, presenting this information under ‘lessons learned’ suggests that it was not considered prior to the course – whereas one would have expected the course design to have been informed by these considerations.  This section might better focus on what were the additional learnings from the course experience using this design – including both strengths (mixing individual and group learning ) and weaknesses ( dealing with a group with different backgrounds, levels of knowledge and skills).

Line 140. This explanation of the projects would be more useful earlier in the paper.

Line 148-49. An important comment on the evaluation. Further explanation of the rationale for not including more objective measures would be helpful.

Table 3. The source / relationship of Table 3 to the TB course is not clear. They seem to refer to more general theories / concepts of training , but also to lessons learnt from this specific training.

Discussion

Line 158. What is the evidence of the ‘lasting ..professional connections’.

Line 175. Limitation – limited evaluation a key issue. More information on the scope, range, and aims of projects could be a valuable early indicator; as well as progress on implementation of these projects.

However, as TB program performance indicators are influenced by many other factors, they might not be as useful as training indicators.

Line 190 Useful comment on HIV training evaluation. Why have more HIV trainings been evaluated ? This might be a role particularly suited to donor funding.

Limited discussion section. Much of Table 3 might be better considered as discussion rather than results.

Conclusion

Line 210 Comment on projects is not supported by any evidence in the results section. As noted above, more information on the projects would also support an evaluation of the program impact.

In regard to the supplementary materials, it would be helpful to state the title of the tables, so that readers could determine if they wished to download them.