Assessments Used for Summative Purposes during Internal Medicine Specialist Training: A Rapid Review

Abstract

Assessments used for summative purposes of patient-facing clinical competency in specialist internal medicine training are high-stakes, both to doctors in training, as it is a prerequisite for qualification, as well as their community of prospective patients. A rapid review of the literature evaluated methods of assessments used for summative purposes of patient-facing clinical competency during specialist internal medicine training in Australia. Four online databases identified literature published since the year 2000 that reported on summative assessment in specialist medical training. Two reviewers screened and selected eligible studies and extracted data, with a focus on evidence of support for the criteria for good assessment as set out in the 2010 Ottawa Consensus framework for good assessment. Ten eligible studies were included. Four studied the mini-clinical evaluation exercise (mini-CEX), two the Royal Australasian College of Physicians short case exam, three a variety of Entrustable Professional Activities (EPAs) or summative entrustment and progression review processes, and one a novel clinical observation tool. The mini-CEX assessment demonstrated the most evidence in support of the Ottawa criteria. There was a paucity of published evidence regarding the best form of summative assessment of patient-facing clinical competency in specialist internal medicine training.

1. Introduction

Internal medicine training in Australia is structured and accredited by the Royal Australasian College of Physicians (RACP), under federal government regulation administered by the Australian Medical Council (AMC). There is evidence to support a correlation between success during clinical training and indices of preventative care and chronic disease management [1,2]. Assessment of medical knowledge and theory during training may be reliably and feasibly evaluated through written examination [3]. However, assessment of patient-facing clinical competencies, such as taking a history or performing a physical examination, requires observation by a skilled assessor [4]. This type of assessment may also involve a real patient in a genuine healthcare environment or a simulated participant (actor) and require assessors who are generally specialists, further increasing logistical difficulty and cost.

Summative assessment evaluates student learning against a defined benchmark or series of learning goals [5]. Assessment used for summative purposes in medical education is high stakes for learners, as the information gained is used to make decisions regarding progression through training or certification [6]. This is especially true in specialist medical training, where summative assessment is additionally high stakes to the wider community since it sets a standard for knowledge and clinical care [7]. A variety of assessment instruments for summative purposes are used in specialist Internal Medicine training programmes (for example, in Australia, New Zealand, the United Kingdom, the United States of America, and Canada). These include a variety of assessments of patient-facing clinical competency, notably long cases, short cases, objective structured clinical examinations (OSCEs), mini-clinical evaluation exercises (mini-CEX), and other work-based assessments [4,8] (see Supplementary Material Table S1). In Australia, the assessment of patient-facing clinical competency is made by a combination of traditional long and short cases in an examination termed the Divisional Clinical Examination [9]. However, this examination has been criticised for inadequate reliability [10,11].

Assessment literature is moving more toward programmatic assessment rather than the study of the properties of individual assessment tools [12]. Programmatic assessments support a portfolio of assessments as being more reliable and authentic than a single assessment [13,14,15]. The change in focus to programmatic assessment emphasises the breadth of assessment types, each with different properties and capabilities to sample the whole curriculum. Single assessments are educationally valuable; multiple assessments combined into a whole can justify summative decisions, with the highest-stakes decisions requiring the most supportive data [13,14,16]. In the programmatic assessment model, individual assessment tools do not need to be as thoroughly analysed as the entire program of assessment. Due to the significant short- and long-term consequences of decisions justified by high-stakes assessments, there is intense scrutiny from all stakeholders: learners, educators, accredited training organisations, and regulators. These are arguably the most important decisions made during medical training [7].

Given the complexity of assessing patient-facing clinical competency and the high stakes involved during specialist internal medicine training, from both the learner and community perspective, it is important that the best methods of assessment are chosen. The Ottawa Consensus identified seven criteria for good assessment: validity, reproducibility, equivalence, feasibility, educational effect, catalytic effect, and acceptability [17]. The most important of the Ottawa criteria for summative assessment is validity (supports the use of the results of an assessment for a defined purpose), reproducibility (assessment results would not change under similar circumstances), and equivalence (assessment scores will be the same if administered in different institutions or cycles of testing), in addition to feasibility (the assessment is practical, realistic and appropriate for the circumstances) and acceptability (stakeholders find the results and assessment process to be credible) [18].

This rapid review aimed to explore the published literature on methods of assessment used for summative purposes of patient-facing clinical competency during specialist internal medicine training, using the critical lens of the criteria for good assessment outlined in the 2010 Ottawa consensus statement [17].

2. Materials and Methods

A rapid review was performed to obtain a contemporary assessment of the literature in a 3-month time period. Rapid reviews are an efficient tool to gather evidence quickly [19]. This can be achieved because a rapid review uses a simplified methodology compared to a systematic review [20]. This review followed the Cochrane Rapid Review Methods Group guidance for rapid reviews [21]. Our rapid review sought to answer the question: ‘What is known about assessments used for summative purposes in specialist internal medicine patient-facing clinical competency?’

Eligibility criteria: The review was limited to studies that were written in the English language, peer-reviewed, published since the year 2000, and with full-text access available. Additional inclusion criteria are that the study was empirical research (quantitative, qualitative, or mixed methods), the context was postgraduate or specialist medical training (that is, post-initial medical qualification), the assessment was for summative purposes (final/progression determining), was a competency assessment (‘shows how’ or ‘does’) of patient-facing skills. Excluded studies were reviews and quality improvement studies, not predominantly adult or pediatric internal medicine (for example, surgery, anaesthetics, intensive care, and family medicine). We did not include programmatic assessments as this is a broad assessment type that involves longitudinal collection of multiple assessment points and types not limited to summative assessment and thus sits outside of the scope of this rapid review. Eligibility criteria were modified iteratively, as the search strategy was trialled to ensure that the literature comprehensively covered the research question and the volume of literature was manageable within a three-month timeframe. Some assessment tools in medical education have formative uses (or, in some cases, are used for both formative and summative purposes). However, for the purpose of this review, only assessments that were used with a summative purpose or intent were included. Therefore, how the tool was used determined whether the study was included in the review. For example, the Royal Australasian College of Physicians does not use the mini-CEx as a summative assessment, but the mini-CEx is used with summative intent in the USA training system. The review, therefore, includes some studies from the USA that reference the use of the mini-CEx but none from Australia.

Literature search: An academic librarian was consulted to construct suitable search terms, with iterative modification as required in setting the eligibility criteria. A search strategy was trialled using a single database, MEDLINE. Only 17 results were generated using the search terms: “summative” AND “internal medicine” AND “performance OR competen* OR assessment OR exam” AND “postgraduate* OR postgraduate* OR graduate*”. As there was insufficient literature to form a satisfactory review, the search parameters were expanded beyond internal medicine. The search strategy was not expanded beyond specialist training (to consider training prior to completion of an initial medical qualification), nor to include assessments without a summative purpose, such as those typically included in programmatic assessment regimes. The rationale for this restriction was that summative assessment in a specialist setting is high-stakes and the final hurdle to independent practice relevant to future public health care quality. A search strategy replacing the search term “internal medicine” with “medicine OR medical OR clinical” generated 329 results. This search strategy was then generalised to four databases which comprised the final search: MEDLINE, PubMed, Web of Science, and Scopus. This search strategy was then generalised to four databases which comprised the final search and yielded a total of 1269 studies imported for screening as follows: MEDLINE (n = 351), PubMed (n = 388), Web of Science (n = 236), and Scopus (n = 294). A further 35 records were identified through handsearching via Google Scholar and web searching. Moreover, 816 duplicate studies were removed, leaving 488 studies. No studies that met all eligibility criteria were excluded based on the unavailability of full text.

Study selection: All papers identified from the final search strategy that included summative assessment in all specialist training programmes were uploaded to Covidence—a software package that streamlines evidence synthesis at each step of the systematic review process. For the remaining 488 studies, two reviewers independently performed title and abstract screening and resolved all conflicts. Moreover, 455 studies were excluded. The most common reason for exclusion was related to the population under assessment being focused on a trainee program other than specialist medicine. References lists from the studies that reached full-text review were also reviewed for any potentially relevant studies, and these were added to the full-text review set. Two reviewers performed full-text reviews and resolved conflicts.

Data extraction: A single reviewer performed data extraction using a piloted data extraction table: date, jurisdiction, specific population and inclusion/exclusion criteria, number of participants, assessment under study, outcome measure/comparator, and attributes relevant to the Ottawa criteria for good assessment. A second reviewer checked the accuracy and completeness of data extraction. All ten studies progressed to this stage.

Risk of bias assessment: Risk of bias assessment was omitted because no judgements were being made about the included study quality.

Data synthesis: For each Ottawa criterion, whether the study provided evidence of support or evidence of concern was assessed. The studies were also assessed for whether support or concern was implied or discussed without supportive evidence.

3. Results

Overall, 1269 records were identified from database searching. An additional 35 records were identified through handsearching. After duplicate, title, and abstract screening, 34 studies proceeded to full-text review, and 10 studies were included. A PRISMA flow diagram is detailed in Figure 1.

4. Study Characteristics

Half (50%) of the included articles were from the USA [22,23,24,25,26], with the remainder from Australia [11,27], The Netherlands [14,28], and Canada [29]. As per the selection criteria, all studies were performed predominantly in an internal medicine training context. Half (50%) of the studies related to adult internal medicine [14,22,23,26,29], three were related to paediatric internal medicine [24,25,28], and two to a combination [11,27]. Trainee cohorts under study ranged from 22 to 1190 individuals (median 105). Six studies had a cross-sectional design [14,22,25,27,28,29], whereas three were single cohort studies [23,24,26]. See

Table 1. Table of study characteristics.

Study ID	Title	Country	Study Design	Population Description	Number of Participants
Durning et al., 2002 [22]	Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training	United States	Cross-sectional study	Postgraduate year doctors (“residents”) at a medical centre	23
Hatala et al., 2006 [29]	Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination	Canada	Cross-sectional study	Postgraduate year 4 resident doctors preparing for RCPSC IM examination	22
Moonen-vanLoon et al., 2013 [14]	Composite reliability of a workplace-based assessment toolbox for postgraduate medical education	Netherlands	Cross-sectional study	Dutch residents at 59 hospitals	953 total (466 first-year subset)
Norcini et al., 2003 [23]	The mini-CEX: a method for assessing clinical skills	United States	Cohort study	Non-systematic volunteer residents at 21 programs	421
Schumacher et al., 2018 [24]	Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions	United States	Cohort study	Paediatric residency programs in Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN)	463 residents, 155 CCC members, 14 PDs
Schumacher et al., 2019 [25]	Justifications for Discrepancies Between Competency Committee and Program Director Recommended Resident Supervisory Roles	United States	Cross-sectional study	Paediatric residency clinical competency committee members and program directors at 14 USA residency programs	98
Smith et al., 2017 [26]	Successful Implementation of a Direct Observation Program in an Ambulatory Block Rotation	United States	Cohort study	Internal medicine residents rotating through an ambulatory care block	57 residents and 14 faculty
Smit et al., 2019 [28]	A National Process to Enhance the Validity of Entrustment Decisions for Dutch Pediatric Residents	Netherlands	Cross-sectional study	Program directors, attending staff, and residents from all Dutch paediatric programs	112 residents and 37 faculty
Wilkinson et al., 2008 [11]	Reliability of the long case	Australia	Other: Statistical analysis of results	RACP Clinical Exam sitters 2005 (Aus only) and 2006 (Aus/NZ)	1190 examinations (915 Adult, 273 Paediatric, 2 supplementary)
Wilkinson et al., 2010 [27]	The reliability of long and short cases undertaken as practice for a summative examination	Australia	Cross-sectional study	DCE candidates from 5 hospitals	59

.

5. Assessment and Comparator/Outcome

The most frequent assessment method reported in the reviewed literature was the Mini Clinical Examination Exercise (mini-CEX) in four studies [14,22,23,29]. Three studies investigated Entrustable Professional Activities (EPAs) and/or Clinical Competence Committee (CCC) decision-making [24,25,28]. Two studies investigated the Royal Australasian College of Physicians (RACP) Divisional Clinical Examination (DCE) short case [11,27]. One study described a novel clinical observation tool. [26]. No studies investigated the Objective Structure Clinical Examination (OSCE). Seven studies used the assessment(s) under study to determine reliability without other outcome measures or comparators. Three studies each used another assessment as a comparator (specifically: American College of Physicians In-Training Examination; Canadian Royal College of Physicians and Surgeons of Canada Internal Medicine examination; and RACP DCE). See

Table 2. Assessment tool and outcome/comparator of included studies Mini-CEX: Mini Clinical Examination Exercise; EPA: Entrustable Professional Activities; CCC: Clinical Competence Committee decision making. Short case: Royal Australasian College of Physicians (RACP) Divisional Clinical Examination (DCE) short case; RCPSC: Royal College of Physicians and Surgeons of Canada; ITE: American College of Physicians–American Society of Internal Medicine In-Training Examination; MEF: American Board of Internal Medicine’s (ABIM’s) monthly evaluation form (MEF); PD: Program Directors Supervisory role.

Study ID	Assessment	Outcome/Comparator
Durning et al., 2002 [22]	Mini-CEX	Itself MEF ITE
Hatala et al., 2006 [29]	Mini-CEX	RCPSC IM examination
Moonen-vanLoon et al., 2013 [14]	Mini-CEX Direct Observation Procedural Skills Multi-Source Feedback	Internal
Norcini et al., 2003 [23]	Mini-CEX	Itself
Schumacher et al., 2018 [24]	Supervisory role entrustment (summative milestone profile)	Between individual CCC members and CCC decision
Schumacher et al., 2019 [25]	CCC entrustment decision	PD supervisory role
Smith et al., 2017 [26]	Direct Observation procedural Skills	Self-reported clinical skills and faculty preference
Smit et al., 2019 [28]	EPA	Acceptability
Wilkinson et al., 2008 [11]	Short case (& long case)	Itself
Wilkinson et al., 2010 [27]	Practice short case (and practise long case)	examination short case (and exam long case)

.

6. Ottawa Criteria for Good Assessment

No study gave evidence for all criteria within the Ottawa framework. The most frequently referenced criterion was validity, where six studies explicitly described supportive evidence, and two others implied support for validity. Five studies explicitly reported evidence for reproducibility. Evidence for equivalence was explicitly stated in one study, implied or discussed in two, and concerns regarding equivalence were raised in another two. The Ottawa criteria for which evidence was least frequently given were educational effect, catalytic effect, and acceptability. The assessment method with the most evidence of support for the Ottawa criteria was the mini-CEX. These results are summarised in

Table 3. Reporting of Ottawa framework criteria in included studies. Legend: evidence of support (with data): +; implied/discussed evidence of support: [+]; implied/discussed evidence of concern: [−]; no evidence or discussion: blank.

	Ottawa Framework Criteria
Study and Assessment Tool	Validity	Reproducibility	Equivalence	Feasibility	Educational Effect	Catalytic Effect	Acceptability
Mini-CEX
Durning 2002 [22]	+	+		[+]
Hatala 2006 [29]	+	+			[+]
Moonen-vanLoon 2013 [14]	[+]	+	[+]	+
Norcini 2003 [23]	+	+	[+]	[+]		[+]	+
Clinical Competency Committee
Schumacher 2018 [24]			[−]
Schumacher 2019 [25]	+
Novel
Smith 2017 [26]	[+]			+	+	+	+
Entrustable Professional Activities
Smit 2019			[−]	+		+	+
RACP short case
Wilkinson 2008 [11]	+		+	+
Wilkinson 2010 [27]	+	+		[+]	[−]	[+]

.

7. Discussion

This rapid review of assessments used for summative purposes of patient-facing clinical skills during specialist internal medicine training in the US, Australia, Canada, and the Netherlands found limited literature to support the efficacy of any single assessment method. Only ten eligible English language publications were found that had been published since the year 2000. From the perspective of the Ottawa criteria of good assessment, the reviewed studies focused on the validity or reliability of the assessment method, with less emphasis on the criteria of educational effect, catalytic effect, and acceptability.

Although all elements within the Ottawa framework are relevant, certain elements have varying importance depending on the purpose of the assessment under consideration. Formative assessment is particularly valued for the educational and catalytic effect on driving current and future learning [18]. This is less relevant for assessments used for summative purposes, and therefore, it is not surprising that these two elements had the least supportive evidence in the studies reviewed. Validity is essential to all assessments, and reproducibility and equivalence are the two other most important elements in effective summative assessment methods. It is notable that only one of the ten reviewed studies explicitly provided evidence for equivalence, with two others providing implied support or discussion. Two further studies implied or discussed evidence of concerns regarding Equivalence. Future work could move past unpacking the Ottawa criteria towards a more integrative review that considers the extent to which the evidence on a suite of clinical assessments provides guidance and narrative on how best to assess patient-facing competencies.

The most reported tool to summatively assess patient-facing clinical skills was the mini-CEX, followed by the RACP DCE short case. Interestingly, literature concerning the OSCE did not meet the specified criteria despite it being in use for many years and employed in the Membership of the Royal Colleges of Physicians (MRCP) Practical Applications of Clinical Examination Skills (PACES) examination in the United Kingdom [30,31]. The review also did not reveal any literature on other work-based assessments that may be used for summative purposes, such as direct observation of procedural skills (DOPS) or Case-based discussion. There is also no mention of ‘eportfolio’ as a summative tool in itself. Some of the most recently published studies explored the combined use of multiple workplace-based assessments, often individually with a primary formative purpose, combined to make summative assessment decisions, often by a Clinical Competency Committee [24,32].

No assessment of the quality of the published studies was made, and therefore, it is impossible to determine the assessment tool with the strongest evidence of support. However, the mini-CEX had the most (numerically) evidence of support, which is in keeping with the considerable literature on the mini-CEX in other disciplines. A systematic review of tools for direct observation and assessment of medical trainees found the mini-CEX to have the strongest validity evidence [33]. A recent review of the mini-CEX found generally good evidence for reliability, validity, feasibility, and acceptability, with less evidence for educational impact [34]. Notably, none of the 58 studies in this published review reported use primarily for summative decision-making [34].

The scope of this rapid review was deliberately narrow, which may have impacted the availability of published studies on summative assessment of patient-facing clinical competency assessment during Internal Medicine specialist training. However, in Australia, Internal Medicine specialists are the largest cohort of specialist doctors after Family Medicine specialists, and therefore, the size of the trainee cohort is not well correlated with the relative lack of investigative evidence [35]. In comparison to the assessment of knowledge or theoretical applications of knowledge, the assessment of performative competencies is difficult, presents considerable logistic difficulties, and is inherently time-consuming and costly [8]. Research on the assessment of performative competencies likely inherits many of these difficulties, and the high-stakes nature of summative assessment further exacerbates these concerns.

Narrowing the search to only assessments used for summative purposes may have excluded a large body of literature on the use of workplace assessments that are used more formatively but also used as part of programmatic assessment to make final decisions on specialists’ progression. The limitations imposed on the search criteria may also mean that opportunities were missed to explore how to better undertake clinical-facing assessments within programmatic assessment. The high-stakes nature of the summative assessments at the late stage in specialist medical training programs should be a powerful driver for a body of evidence to justify training and assessment programs. However, there may be concerns that such research could undermine existing long-standing protocols and expose a training program to challenge, and overall, such evidence is lacking [36]. In jurisdictions where such concerns hinder research, there may be a need for a regulatory imperative to require an evidence-based assessment system.

8. Strengths and Limitations of This Review

A rapid review format was chosen given the desire for a contemporary assessment of the literature. However, the narrow search criteria adopted may have excluded studies that explored educational impact and acceptability. The narrow search terms may also have contributed to the exclusion of studies from ‘non-western’ settings. To provide more meaningful information on how a suite of assessments provides guidance and a narrative on how best to assess patient-facing competencies, a narrative or integrative review may have been a better choice for the literature review. Internal medicine training in resource-rich, developed countries is highly regulated, and there are a limited number of accredited training providers, often only one per country. As such, there may be commercial-in-confidence concerns that have prevented the publication of data that might support (or otherwise) systems of assessment in active use. This context also makes it unlikely that systems of assessment could ever be directly compared within one jurisdiction, as this would result in trainees within a single training program being subjected to different assessments to measure acceptable clinical performance.

9. Conclusions

Summative assessment in specialist medical training is high stakes to learners, educators, and the wider community of prospective future patients. There is a paucity of literature regarding assessments for summative purposes of patient-facing clinical competencies during internal medicine training. The existing, limited literature supports the use of mini-CEX as a valid, reliable, and feasible tool. The lack of supportive literature suggests a systematic impediment to research in this area. Systems of summative assessment require an evidence base to justify their use. The absence of a satisfactory evidence base may be an opportunity for a regulatory imperative.