Assessments Used for Summative Purposes during Internal Medicine Specialist Training: A Rapid Review
Round 1
Reviewer 1 Report
I really enjoyed reading this manuscript. The methods are well done and the findings are clearly explained. My only comment is about Figure 1 - I wonder if it can be redone so it is clear. Right now it is very fuzzy.
Author Response
Figure 1 - I wonder if it can be redone so it is clear. Right now it is very fuzzy. 
Response: We have uploaded a high-quality figure to the journal submission portal.
Reviewer 2 Report
Lines 9-21: Abstract: this paragraph needs to be reviewed by all the authors. There are repetitive sentences, redundant information.
Assessments used for summative purposes of patient-facing clinical competency in specialist internal 9 medicine training is high-stakes, both to doctors in training, as it is a prerequisite for qualification, as well as 10 their community of prospective patients rewrite for clarity
A rapid review of the literature evaluated methods of assessments 11 used for summative purposes of patient facing clinical competency during specialist internal medicine training 12 in Australia Repetitive/redundant
Lines 16-19: Ten eligible studies were included. Four studied the mini- 16 clinical evaluation exercise (mini-CEX), two the Royal Australasian College of Physicians short case exam, 17 three a variety of Entrustable Professional Activities (EPAs) or summative entrustment and progression review 18 processes, and one a novel clinical observation tool.
please indicate that it is the authors who are reviewing not a stand-alone number.
Maybe consider adding the Cochrane rapid review guidelines as a non-published supplement. It might be useful for the reading audience.
Cite as: Garritty C, Gartlehner G, Kamel C, King VJ, Nussbaumer-Streit B, Stevens A, Hamel C, Affengruber L. Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group. March 2020. https://methods.cochrane.org/rapidreviews/cochrane-rr-methods
except for the abstract, i found no blaring mistakes or omissions. Very well done.
Comments for author File: 
Comments.pdf
This manuscript was easy to read and understand. If anything, the abstract needs rewriting. Use of the English language was appropriate for this manuscript.
Author Response
Response: Authors have revised this paragraph as follows to improve clarity and reduce redundancies:
Success in clinical assessment is associated with good clinical outcomes. Summative assessment of patient-facing clinical competency in specialist internal medicine training is therefore high-stakes, both to doctors in training, as it is a prerequisite for qualification, as well as their community of prospective patients. The best methods of summative assessment of clinical competency is imperative, but evaluations of such methods are scarce. Here we review published literature to evaluate the methods of summative assessment of patient facing clinical competency that are used during specialist internal medicine training.
Lines 16-19: Please indicate that it is the authors who are reviewing not a stand-alone number
Response: We feel that we have maintained consistency with the Cochrane rapid review guidelines in our manuscript ( Garritty C, Gartlehner G, Kamel C, King VJ, Nussbaumer-Streit B, Stevens A, Hamel C, Affengruber L. Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group. March 2020. https://methods.cochrane.org/rapidreviews/cochrane-rr-methods). We have nonetheless revised the selected text to improve clarity as follows:
Ten eligible studies were included in our rapid review. Four of these focused on the mini-clinical evaluation exercise (mini-CEX), two focused on the Royal Australasian College of Physicians short case exam, three focused on a variety of Entrustable Professional Activities (EPAs) or summative entrustment and progression review processes, and one study focused on a novel clinical observation tool. None of the eligible included studies examined the Objective Structured Clinical examination (OSCE). The mini-CEX assessment demonstrated the most evidence in support of the Ottawa criteria for good assessment.
Reviewer 3 Report
I enjoyed reading this well-written paper, with much to commend. The introduction is clear and appropriately considers summative assessment and the wider context of programmatic assessment. The research question is clear.
You made good use of an academic librarian to help with your search criteria. However, I cannot help wondering if there was some problem with the search strategy, given that you did not detect any studies from the UK. The MRCP PACES examination has been the subject of a number of empirical studies. On quickly reviewing several of them now, they appear to meet your eligibility criteria, so I am very puzzled that they were not included. I think this needs further justification and consideration.
One of your inclusion criteria is that "full text access was available". This obviously varies by institution. You have not clarified if any potential studies were excluded on this basis. It would be helpful to make this clear. You explain later in the discussion that assessment studies related to programmatic assessment were not included. Given your introduction, this seems surprising and it would help the reader to justify this more explicitly in the methods section.
I found it confusing when first reading the methods section when you talked about 329 results and then referred to the remaining 488 studies. When studying your PRISMA diagram, it became much clearer, although the MEDLINE search in figure 1 refers to 351 studies, not 329. It would be helpful to revise the methods section to aid understanding.
It would be helpful to include a sentence or reference to explain Covidence to any reader who is not aware of this.
When discussing the data extraction, you refer to a second reviewer checking the accuracy and completion of extracted data. It is not clear what proportion of papers they reviewed. It would be helpful to clarify this.
The discussion is clearly laid out and there is a helpful acknowledgement of the study's limitations. I expected to see more consideration regarding future research in this area.
Overall, however, I found this a useful addition to the research literature in this area. I hope my feedback will prove helpful in improving your paper.
Author Response
Response: We appreciate this observation. The MRCP PACES exams were not captured by our search strategy because they are not considered summative in the traditional sense. Although they are a hurdle requirement to pass to more advanced training, they can be taken a number of times during training in both formative and summative capacities, rather than in a summative capacity at the end of training. We have informally repeated the search strategy with PACES as another search term and only one reference was captured. This reference was a descriptive review/descriptive piece and as such did not meet eligibility criteria.
One of your inclusion criteria is that "full text access was available". This obviously varies by institution. You have not clarified if any potential studies were excluded on this basis. It would be helpful to make this clear. 
Response: No studies were excluded based on the unavailability of full text; one study included required paywall access which we obtained. The Literature Search section of the methods has been updated to improve clarity as follows:
No studies that met all eligibility criteria were excluded based on the unavailability of full text.
Reviewer 4 Report
The authors describe an interesting rapid review regarding summative assessments for internal medicine residency training.
The topic of this rapid review is timely and important. Assessment of patient-facing clinical skills is of immense importance in internal medicine residency training, not only for the residents, but also for their future patients. It is necessary to address the strengths and weaknesses of currently used methods to ensure the highest standard of training and patient care.
The approach of a literature review using four online databases and a focus on publications after the year 2000 ensures that current and relevant data is captured. The inclusion criteria, based on the 2010 Ottawa Consensus Framework, provide a robust standard for the evaluation of the assessments. The comparison of different methods of assessment and the identification of gaps in the literature appear to be particularly helpful.
The manuscript is well-structured, and the information is presented in a clear and concise manner. There is a logical flow of findings and conclusions drawn from the evidence.
The paucity of published evidence on the best form of summative assessment of patient-centered clinical competence is an important finding and points to the need for broader reviews with more variable inclusion criteria.
In conclusion, this article is a valuable contribution to the field and addresses a gap in the current literature. I appreciate the diligent efforts in this research and recommend the manuscript for acceptance in its current form.
Author Response
No response required.
