From GPT-3.5 to GPT-4.o: A Leap in AI’s Medical Exam Performance

Abstract

ChatGPT is a large language model trained on increasingly large datasets to perform diverse language-based tasks. It is capable of answering multiple-choice questions, such as those posed by diverse medical examinations. ChatGPT has been generating considerable attention in both academic and non-academic domains in recent months. In this study, we aimed to assess GPT’s performance on anatomical multiple-choice questions retrieved from medical licensing examinations in Germany. Two different versions were compared. GPT-3.5 demonstrated moderate accuracy, correctly answering 60–64% of questions from the autumn 2022 and spring 2021 exams. In contrast, GPT-4.o showed significant improvement, achieving 93% accuracy on the autumn 2022 exam and 100% on the spring 2021 exam. When tested on 30 unique questions not available online, GPT-4.o maintained a 96% accuracy rate. Furthermore, GPT-4.o consistently outperformed medical students across six state exams, with a statistically significant mean score of 95.54% compared with the students’ 72.15%. The study demonstrates that GPT-4.o outperforms both its predecessor, GPT-3.5, and a cohort of medical students, indicating its potential as a powerful tool in medical education and assessment. This improvement highlights the rapid evolution of LLMs and suggests that AI could play an increasingly important role in supporting and enhancing medical training, potentially offering supplementary resources for students and professionals. However, further research is needed to assess the limitations and practical applications of such AI systems in real-world medical practice.

1. Introduction

The environment and tools used for academic education and research have significantly changed in the past decades. Before the invention of textbooks as an educational tool, lectures were virtually the only option to gather knowledge from an expert. For some decades, students can principally decide by themselves whether or not they will visit a given lecture or learn the study material on their own. Nowadays, the use of textbooks is still very common; however, traditional textbooks are often used in conjunction with other resources, such as podcasts or videos. In parallel, the way students prepare assays significantly changed. While previously, different textbooks in libraries were screened for relevant information, nowadays, online resources such as Google, Bing, or Yandex are available. In addition, different AI tools are continuously developed, which principally can facilitate initial exploration of the literature in a domain in which someone is marginally familiar, for example, iris.ai, scite.ai, or jenni.ai.

In the past year, a new artificial intelligence (AI) model called ChatGPT garnered significant attention due to its ability to perform a diverse array of natural language-based tasks. GPT, which stands for Generative Pre-training Transformer, is a large language model (LLM) developed by OpenAI that uses the Transformer architecture and Reinforcement Learning from Human Feedback approach. The previous class of AI models has primarily been Deep Learning models, which are designed to learn and recognize data patterns. In contrast, the LLM AI algorithm can, if properly trained on sufficiently large amounts of text data, predict the likelihood of a given sequence of words based on the context of the words that come before it, a mechanism called “autoregressive”. These technologies offer a range of opportunities, such as the development of lifelike simulations, digital patients, tailored feedback systems, advanced evaluation techniques, and the ability to overcome language barriers [1]. Thus, the utilization of AI and generative language models holds immense potential in augmenting medical education. Others, however, argue that AI in general and, in particular, LLMs might be a challenge for academic teaching and integrity in the future [2,3,4].

OpenAI, the company behind ChatGPT, was founded in 2015. The first LLM, GPT-1, was introduced in June 2018 and consisted of 117 million pre-trained parameters. Its successor, GPT-2, already comprised 1.5 billion parameters and was released to the public by the end of 2019. In June 2020, GPT-3 followed, having been trained with more than 175 billion parameters, thus being over 100 times larger than GPT-1 and over 10 times larger than GPT-2. GPT-3 was the first model capable of handling comprehensive language tasks such as writing emails, generating code, or translating texts. On 30 November 2022, the company released an improved version, GPT-3.5, or ChatGPT. GPT-4, launched on 14 March 2023, is the latest and most advanced model in the GPT series, representing a significant leap forward. For example, unlike GPT 4.o, GPT-3.5 had no internet connection and, therefore, could not respond to queries on current topics. While the specifics of the GPT4 training data and architecture are not officially announced, they certainly build upon the strengths of GPT-3 and overcome some of its limitations.

In Germany, a regularly structured degree course in medicine takes six years and three months. It is completed by a final state exam (Staatsexamen), with which one obtains a license to practice as a physician (called approbation). The application procedure for a degree course in medicine is highly competitive and complicated due to the complex selection process via Hochschulstart. Medical studies and training in Germany are regulated under federal regulations on medical registration. There are currently three state exams the students have to pass during their studies: the first state exam, also called the M1 exam or formerly “Physikum”, which takes place after the second year; the second state exam, also called the M2 exam, which takes place after the fifth year; and the third state exam, also called the M3 exam, which takes place after the sixth year. The M1 exam consists of an oral and a written examination. The written part is conducted nationwide on two consecutive days and includes 600 multiple-choice questions from all preclinical subjects. The oral exams take place in anatomy, biochemistry, and physiology.

Previous studies have primarily assessed LLM performance in medical contexts using questions from simulated medical exams and open-source datasets [5,6,7,8]. More recent research has evaluated the performance of LLMs in actual examination settings [9,10,11]. In March 2023, OpenAI published a technical report on GPT-4 [12]. This report presents data on GPT-4’s performance in a variety of exams designed for humans. Of note, there was no specific pre-training for these exams. In these tests, GPT-4 performed remarkably well, frequently surpassing the scores of most human participants, and showed a significant improvement compared with the performance of GPT-3.5.

The primary objective of the current article was to assess the performance of GPT-3.5 on official medical board examinations. To this end, test questions from two previous state-wide exams were provided to GPT-3.5, and the percentage of correctly or wrongly answered questions was evaluated. The secondary objective was to evaluate the improvement of GPT technology by comparing the performance of GPT-3.5 and GPT-4o. Finally, we systematically compared the performance of GPT-4.o with that of a large cohort of medical students.

2. Materials and Methods

2.1. General Information and Applied Large Language Models

The study utilized an observational, cross-sectional design to evaluate the performance of GPT 3.5 and GPT-4.0 on 436 questions from M1 medical state exams (subject anatomy) from spring 2021 to autumn 2023. Each question was manually inputted into either GPT-3.5 or GPT-4.o (see below for more detailed information). GPT is a sophisticated language model created by OpenAI, headquartered in San Francisco, CA. It utilizes self-attention mechanisms and an extensive corpus of training data to generate natural language responses to textual input within a conversational context. Unlike other chatbots or conversational systems that have the ability to access external sources of information (such as conducting internet searches or accessing databases) to provide tailored responses to user inquiries, GPT-3.5 was a self-contained model that could not browse the internet. This means that all responses were generated in real-time, solely based on the abstract relationships between words, also known as “tokens”, in the neural network. Initially, this study used GPT-3.5, but additional experiments were conducted approximately 12 months later with GPT-4, which had since gained access to online information. Questions were directed to freely accessible and publicly available versions of GPT-3.5 and to the premium ChatGPT-4.o plus version, accessed through the Firefox web browser. No incognito mode was used.

2.2. Input Source

First, medical state exam (M1 exam, formerly called “Physikum”) multiple-choice questions were obtained from the Thieme examen online website. In contrast to other studies [13], multiple-choice questions (five options: A, B, C, D, or E) were provided to the LLM in German language but were translated to prepare this manuscript using GPT-4o. None of the questions had multiple valid answers.

In the first set of experiments, we assessed the performance of GPT-3.5 on official medical board examinations. To this end, anatomical gross macroscopy test questions from the autumn exam 2022 and the spring exam 2021 were obtained from the Thieme examen online website. Responses were categorized as either correct or incorrect. To investigate the reproducibility of the results, we repeated the experiment the next day using the same exam questions. For the spring 2021 exam questions, GPT-3.5 was re-challenged for the wrong answers by asking the question, “Are you sure?”, and the responses were categorized as follows: correct = after the re-challenge a correct answer was given; no change = after the re-challenge again a wrong answer was given; wrong = while the LLM apologizes for the wrong answer, again a wrong answer was given after the re-challenge. The described evaluations were performed between February and May 2023. Questions related to or including anatomical images or drawings were excluded.

In a second set of experiments, we aimed to compare the performance of GPT-3.5 versus GPT-4.o. To this end, the same questions as in the first part of this study were provided to the more advanced GPT-4.o version in June 2024. Responses were categorized as either correct or incorrect, and their frequencies were statistically compared. Beyond, we aimed to analyze in this part of the study if the potentially improved performance of GPT-4.o might be due to the fact that medical state exam questions are online and can, thus, be used as a template to answer the questions. To rule this out, 30 randomly chosen questions from previous anatomical exams at the university center Rostock were used as an input source for GPT-4o. Responses were categorized as either correct or incorrect.

In a third set of experiments, we aimed to evaluate the performance of GPT-4.o in comparison to medical students. To this end, anatomical gross macroscopy, microscopy, and embryology test questions were obtained from six different exams from the Thieme examen online website (id est; spring 2021–2023 and autumn 2021–2023), and the performance of GPT-4.0 was compared with the results of medical students (retrieved from the data repository of https://www.impp.de/pruefungen/medizin/l%C3%B6sungen-und-ergebnisse.html (accessed on 1 August 2024)). The total number of valid questions in this last part of the study was 344. Again, questions related to or including anatomical images or drawings were excluded, and responses were categorized as either correct or incorrect.

2.3. Analyses

Objective 1: To assess the performance of GPT-3.5 on official medical board examinations, gross anatomical questions from two exams (autumn 2022 and spring 2021) were included. Numbers of correct and incorrect answers, as well as numbers of unanswered questions, were recorded. Based on this information, we calculated the percentage of each categorical score. Reproducibility was defined as the agreement between the answers of the first and second rounds, provided by GPT-3.5, regardless of whether the provided/suggested answer was initially correct or incorrect.

Objective 2: To evaluate the improvement of GPT technology by comparing the performance of GPT-3.5 and GPT-4o. Gross anatomical questions from two exams (autumn 2022 and spring 2021) were included. Numbers of correct and incorrect answers from the first rounds were recorded for both GPT versions and statistically compared using Fisher’s exact test (two-sided).

Objective 3: To assess whether GPT-4.o performs superior to medical students. Gross anatomical, histological, and embryology questions from six exams (spring 2021–2023 and autumn 2021–2023) were included. Numbers of correct and incorrect answers were recorded for both GPT versions and statistically compared using a Mann–Whitney test.

All statistical analyses have been performed using Prism 9.2.0 (GraphPad Software Inc., San Diego, CA, USA). No outliers were excluded from the analyses. p-values of ≤0.05 were considered statistically significant.

3. Results

In the first set of experiments, 56 test questions from the autumn exam in 2022 were obtained from the Thieme examen online website (see Figure 1 and

Table 1. This table demonstrates the results of the M1 autumn exam in 2022 (formerly called physikum). Stars (*) in the row “image” indicate questions that were not further processed because they contained relevant images or schematic drawings. The exam was provided to GPT-3.5 in duplicate, which is indicated by “1st round” and “2nd round”, respectively. The crosses indicate whether GPT-3.5 provided a correct, false, or no answer (see respective columns). Red crosses indicate inconcistent answers.

Autumn Exam in 2022
			1st Round				2nd Round
Index	Image	Topic	Correct	False	No Answer		Correct	False	No Answer
1		penis	x					x
2		A vertebralis	x				x
3		cerebellum		x			x
4	*
5		uterus	x				x
6		neuroanatomy of the optic tract		x				x
7		intestines		x			x
8		respiratory system	x				x
9		segmental innervation	x				x
10	*
11		caecum	x				x
12		lymphatics		x			x
13		stomach		x				x
14	*
15		inguinal region			x			x
16		coronary blood vessel	x				x
17	*
18		face	x					x
19		intestines			x			x
20	*
21		nose	x				x
22		thorax		x			x
23		Bursa omentalis	x				x
24		eye		x				x
25		neuroanatomy of the optic tract	x					x
26		neuroanatomy of the trigeminal nerve			x				x
27	*
28		neuroanatomy of machanosensation	x				x
29		acromioclavicular joint	x				x
30		eye bulb movement	x				x
31		larynx		x			x
32		neuroanatomy of the trigeminal nerve			x				x
33		hand and wrist anatomy	x				x
34		lymphatics	x					x
35		brain stem nuclei		x				x
36		neuroanatomy of the cortex	x				x
37	*
38		bile duct	x				x
39		tympanic	x				x
40		neck and throat	x				x
41		continence organs	x				x
42	*
43		superficial anatomy		x			x
44	*
45		lymphatics		x				x
46		neuroanatomy of the spinal cord	x				x
47	*
48	*
49		hand and wrist anatomy	x				x
50		eyeball	x				x
51		ankle joint and foot		x				x
52		knee joint		x			x
53		abdominal wall	x						x
54		neuroanatomy of the n facialis	x				x
55		hip joint	x				x
56		inguinal region		x				x

). Of those 56 questions, 11 were related to anatomical images and were, thus, excluded from any further analyses (highlighted by a star in Table 1).

In the first round, GPT-3.5 correctly answered 27 of the remaining 45 exam questions, which equals 60%. The suggested answer from GPT-3.5 was wrong in 14 cases. In another 4 cases, no answer was provided at all. To investigate the reproducibility of this result, we repeated the experiment the next day using the same exam questions (see second-round columns in Table 1). In the second round, GPT-3.5 correctly answered 29 of the 45 exam questions, which equals ~64%. The suggested answer from GPT-3.5 was wrong in 13 cases. In another 3 cases, no answer was provided at all. In most cases, those questions that were answered correctly in the first round were also correctly answered in the second round. On the other hand, those questions that were answered wrong in the first round were also wrongly answered in the second round. However, in some cases, false answers in the first round were correct in the second round and vice versa (highlighted in red in Table 1).

The spectrum of how GPT-3.5 responded to the given exam questions varied somehow. In some cases, a brief answer without any explanations was provided. For example, the provided answer for question #19 was “The greater curvature of the stomach is supplied primarily by the left gastric artery, so answer (B) is correct”. In other instances, GPT-3.5 provided more detailed explanations of why the given answer should be the correct one. For example, the provided answer for question #43 was “According to the rule of nines, which is used in first aid for a quick estimate of the affected skin surface in the case of burns, the body is divided into nine regions, each of which is assigned a certain percentage of the total skin surface. The ventral surface of the left leg corresponds to about 9% of the total skin surface. If this area were completely scalded, about 9% of the total skin surface would be affected. So the correct answer is (B) 9%”. Of note, this particular question was answered falsely in the first round but correctly in the second (see Table 1). Interestingly, in some cases, GPT-3.5 did not provide the correct multiple-choice item but correctly described which answer should be the correct one. In some cases, no conclusive answer was provided at all.

In order to reproduce the obtained results, 49 test questions from the M1 spring exam in 2021 were obtained from the Thieme examen online website. Of those 49 questions, 2 questions were related to anatomical images and were, thus, excluded from the analyses (highlighted by a star in

Table 2. This table demonstrates the results of the M1 spring exam in 2021 (formerly called physikum). Stars (*) in the row “image” indicate questions that were not further processed because they contained relevant images or schematic drawings. The exam was provided to GPT-3.5 in duplicate, which is indicated by “1st round” and “2nd round”, respectively. The crosses indicate whether GPT-3.5 provided correct, wrong, or no answer (see respective columns). Red crosses indicate inconcistent answers. In the column “re-challenge”, it is indicated whether GPT-3.5 provided an alternative answer in case simply asking the question, “Are you sure?” for all questions falsely answered in the 2nd round.

Spring Exam in 2021
			1st Round				2nd Round
Index	Image	Topic	Correct	False	No Answer		Correct	False	No Answer	Re-Challenge
1		nose		x				x		wrong
2		vegetativum	x				x
3		reflexes	x				x
4		hand and wrist	x				x
5	*
6		hand and wrist	x					x		correct
7		segental innervation		x			x
8		hand and wrist	x				x
9		N medianus			x				x
10		ankle joint and foot		x			x
11		ankle joint and foot	x				x
12		joints		x				x		no change
13		hip joint	x					x		correct
14		hip joint			x		x
15		N obturatorius		x				x		no change
16		hip joint	x				x
17		gluteal region	x				x
18		pelvis		x				x		no change
19		inguinal region	x				x
20		scrotum	x				x
21		lymphatics		x				x		no change
22		pelvis	x				x
23		abdomen	x				x
24		aorta	x				x
25		heart	x				x
26		heart	x				x
27		lymphatics	x				x
28		abdomen	x				x
29		abdomen		x				x		no change
30		abdomen		x				x		no change
31		abdomen		x			x
32		continence organs		x				x		wrong
33		adrenals		x				x		wrong
34		neck and throat		x				x		wrong
35		neck and throat		x				x		no change
36		neck and throat		x				x		no change
37		neck and throat		x				x		no change
38		vegetativum	x				x
39		head	x				x
40		CNS	x				x
41		CNS	x				x
42		CNS	x					x		correct
43		endocrinum		x			x
44		optic tract			x				x
45		eye		x				x		correct
46		eye	x				x
47		eye		x				x		no change
48	*
49		vestibular	x				x

). In the first round, GPT-3.5 correctly answered 25 of the remaining 47 exam questions, which equals ~53%. The suggested answer from GPT-3.5 was wrong in 19 cases. In another 3 cases, no answer was provided at all. To investigate the reproducibility of this result, we again repeated the experiment using the very same exam questions. In the second round, GPT-3.5 correctly answered 27 of the 47 exam questions, which equals ~57%. The suggested answer from GPT-3.5 was wrong in 18 cases. In another 2 cases, no answer was provided at all. Again, in most cases, those questions that were answered correctly in the first round were also correctly answered in the second round. On the other hand, those questions that were answered wrong in the first round were also wrongly answered in the second round. However, in some cases, false answers in the first round were correct in the second round and vice versa (highlighted in red in Table 2).

In the second round of the spring 2021 exam, no answers were provided for questions #9 (related to the course of the N. medianus) and #44 (related to the symptom “acromegaly”). Since the response for question #9 was “None of the statements is usually true”, next, we re-challenged GPT-3.5 by asking the question, “sure that none of the statements are true”. In contrast to the first response, the next response was correct. In particular, GPT-3.5 stated, “I’m sorry, I accidentally misunderstood your previous question. In fact, one of the statements about the course of the median nerve is correct. The median nerve runs between the heads of the pronator teres”. The response for question #44 was “Given the symptoms, the mass is most likely to be in the area of the pituitary stalk, which connects the hypothalamus and adenohypophysis. Tongue enlargement may be caused by compression of the mass on the hypoglossal nerves, while visual field limitations may be due to optic nerve compression. However, accurate localization of the mass requires imaging”. We, again, re-challenged GPT-3.5 by asking the following question: “what is the correct answer”. After this re-challenge, GPT-3.5 provided the correct answer.

In the next step, we decided to re-challenge GPT-3.5 for the 18 questions that were not correctly answered in the second round of the M1 spring exam 2021 (see Table 2) by simply asking the question “Are you sure?”. As demonstrated in Table 2, in eleven cases, GPT-3.5 persisted on the initially provided answer (highlighted by “no change” in Table 2); in four cases, the LLM apologized for the wrong answer but, again, provided a wrong answer (highlighted by “wrong” in Table 2), and in another four cases the LLM apologized for the wrong answer and provided a correct answer (highlighted by “correct” in Table 2).

So far, we have been able to demonstrate that GPT-3.5 is able to “understand” anatomical multiple-choice questions from medical exams and can provide, in a sufficient number of cases, correct answers. LLMs continuously develop. When starting this study in 2023, GPT-3.5 was available and, thus, used. In May 2024, GPT-4o was released by OpenAi. To analyze the performance of this advanced LLM, we challenged GPT-4o with the 45 test questions from the M1 autumn 2022 exam and the 47 test questions from the M1 spring 2021 exam. Surprisingly, GPT-4o greatly improved its performance with just three wrong answers in the autumn 2022 exam and zero wrong answers in the spring 2021 exam, respectively (see Figure 2). Statistical comparison using Fisher’s exact test rejected the null hypothesis (p = 0.0011 for the autumn 2022 exam and p < 0.0001 for the spring 2021 exam), demonstrating that GPT-4.o outperforms GPT-3.5 in this selection of anatomical medical state exam questions.

Allowing LLMs to access the internet can significantly expand their functionality by enabling them to retrieve up-to-date information, verify facts, and respond to queries that require real-time data. The better performance of GPT-4o versus GPT-3.5 in this study might be due to the fact that medical state exam questions are online and can, thus, potentially be used as a template to answer the questions. We, thus, decided in the next step to investigate the performance of GPT-4o in answering 30 randomly chosen multiple-choice questions from previous anatomical exams at the university center Rostock. Since these questions were prepared by our lecturers, they cannot be found by LLMs in any online repository. Of note, GPT-4o correctly answered 29 of the 30 exam questions, which equals a ~96% test performance.

Finally, we performed a systematic comparison of the performance of GPT-4o versus that of a large cohort of medical students by using entire anatomical questions from six distinct M1 state exams (spring 2011 to autumn 2023). As demonstrated in Figure 3, the GPT-4o outperformed medical students across all six medical examinations. Statistical comparison between GPT-4o and medical students’ test performance revealed that this difference is highly significant (GPT-4o mean 95.54 ± SEM 1.004 versus medical students mean 72.15 ± SEM1.673; p < 0.01).

4. Discussion

ChatGPT, an AI-powered chatbot developed by OpenAI, is currently creating a buzz across various academic and non-academic sectors. Its most promising feature is to respond to questions in a human-like manner. Due to its growing popularity and emerging use, it is crucial to conduct a critical assessment of the potential effects of GPT on research and teaching. In this study, we analyzed the areas of robustness and limitations of GPT’s responses to anatomical multiple-choice questions of the first medical state exam in Germany.

In university education, various examination methods are used, including term papers, oral exams, or multiple-choice tests. The use of LLMs in scientific writing, such as for preparing term papers, is already critically discussed in the literature [14,15,16,17] and represents an explicit limitation of such an assessment format in the future. Comparably, online exams, currently offered by many universities and in various formats, are at risk when students use LLMs [3,18]. Empirical research shows a significant increase in academic dishonesty in online environments [19,20]. For in-person multiple-choice exams, the risk of students using LLMs to cheat may initially appear minimal. However, with the rapid advancement of technologies, it is conceivable that optical systems, like smart glasses, could be developed to recognize text and, when connected to LLMs, assist in answering multiple-choice questions. To mitigate this risk, exam questions should be designed in ways that LLMs cannot easily handle. For instance, LLMs still struggle with logical reasoning [9,21], so questions requiring critical thinking rather than simple fact recall may eventually be safely used. Future research has to show the rules on how to prepare such kinds of questions.

Despite these reservations, some authors argue that the growing use of LLMs can also drive the evolution of university curricula [22]. For example, instead of traditional term papers, students could be tasked in the future to evaluate AI-generated content, learning to assess and refine its responses critically. Additionally, LLMs offer several potential benefits in academia, such as assisting non-native speakers by translating content and providing explanations in different languages, facilitating literature research, and supporting the preparation of multiple-choice exam questions, for example, by verifying the accuracy of possible answers.

Recent studies have shown that several LLMs are highly capable of answering medical exam questions from various specialties, different countries, and varying levels of difficulty. A summary of studies published in 2024 is shown in

Table 3. Literature review of studies investigating the performance of LLMs, particularly GPT, on medical exam questions. Studies published in 2024 until the final publication of this paper have been included. Literature research was conducted in https://pubmed.ncbi.nlm.nih.gov/ using the search terms “ChatGPT AND medical exam”. Review articles were excluded. The term “ChatGPT” is just given in case no information regarding the specific LLM version was provided in the respective manuscript.

Citation	Study Resources	Applied LLM	Main Outcome(s)
[23]	Progress Test of the Brazilian National Medical Exam	GPT-3.5	GPT-3.5 outperformed humans
[24]	European Board of Interventional Radiology exam	GPT-4.o	GPT-3.5 outperformed students and consultants; lower accuracy than certificate holders
[25]	American Registry of Radiologic Technologists Radiography Certification Exam	GPT-4	Higher accuracy on text-based compared to image-based questions; performance varied by domain
[26]	Taiwan plastic surgery board exam	GPT3.5 and GPT-4	ChatGPT-4 outperformed ChatGPT-3.5; ChatGPT-4 passed five out of eight yearly exams
[27]	U.K. medical board exam	Various	All LLMs scored higher on multiple-choice vs. true/false or “choose N” questions; best performance of GPT-4.o
[28]	Chinese Medical Licensing exam	GPT3.5 and GPT-4	ChatGPT-4 outperformed ChatGPT-3.5; GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties
[29]	U.S. otolaryngology board exam (preparation tool BoardVitals)	ChatGPT, GPT-4, and Google Bard	GPT-4 outperformed ChatGPT and Bard
[30]	Polish nephrology speciality exam	GPT3.5 and GPT-4	ChatGPT-4 outperformed ChatGPT-3.5; outperformed humans
[31]	U.S. national radiation oncology in-service examination	GPT3.5 and GPT-4.o	ChatGPT-4 outperformed ChatGPT-3.5; difference in performance by question category
[32]	Member of Royal College of Physicians Part 1 exam	GPT3.5 and GPT-4	ChatGPT-4 outperformed ChatGPT-3.5; both versions above the historical pass mark for MRCP Part 1
[33]	European Board of Urology exam	GPT3.5 and GPT-4	ChatGPT-4 outperformed ChatGPT-3.5; performance varied by domain; GPT-4 passed all exams
[34]	American Board of Physical Medicine and Rehabilitation exam	GPT-3.5, GPT-4, and Google Bard	GPT-4 outperformed ChatGPT and Bard
[35]	Local pulmonology exam with third-year medical students	GPT-3.5	Performance varied by domain and question type; performance closely mirrors that of an average medical student
[36]	Orthopedic Board-Style Written Exam	GPT-3.5	Performs below a threshold likely to pass the American Board of Orthopedic Surgery (ABOS) Part I written exam
[37]	Chinese ophthalmology-related exam	own developed LLM (MOPH)	Good performance
[38]	Chinese residency final exam	GPT-3.5	Potential for personalized Chinese medical education
[39]	French local physiology university exam	GPT-3.5	Outperformed humans
[40]	American urology board exam	GPT-3.5 and GPT-4	GPT-4 outperformed ChatGPT-3.5; performance relatively poor
[41]	Otolaryngology (Rhinology) Standardized Board Exam	GPT-3.5 and GPT-4	GPT-4 outperformed ChatGPT-3.5 and residents
[42]	Vignettes covering emergency conditions in plastic surgery	GPT-4 and Gemini	GPT-4 outperformed Gemini; AI might support clinical decision-making
[43]	American Society for Surgery of the Hand exam	GPT-3.5	Poor performance; performance varied by question type
[44]	Urology Canadian board exam	GPT-4	GPT-4 underperformed compared to residents
[45]	Polish Medical Final exam	GPT-3.5	GPT-3.5 underperformed compared to humans; passed 8 out of 11 exams
[46]	Advanced Burn Life Support exam	GPT-3.5, GPT-4, Bard	GPT-4 outperformed Bard; high accuracy of GPT-3.5 and GPT-4
[47]	Canadian Association of Medical Radiation Technologists exam	GPT-4	Passed exams; bad performance on critical thinking questions
[48]	Japanese Medical Licensing exam	GPT-4	Passed exams; performance comparable to humans; performance varied by question type
[49]	Spanish orthopedic Surgery and Traumatology exam	GPT-3.5, Bard, Perplexity	GPT-3.5 outperformed Bard and Perplexity;
[50]	American Board of Neurological Surgery	ChatGPT	ChatGPT underperformed compared to humans
[51]	European Society of Neuroradiology exam	GPT-3.5, GPT-4, Bard	GPT-4 outperformed GPT-3.5 and Bard
[52]	Interventional cardiology exam	GPT-4	GPT-4 underperformed compared to humans; passed the exam
[53]	Examen Único Nacional de Conocimientos de Medicina min Chile	GPT-3.5, GPT-4, GPT-4V	All versions passed the exam; GPT-4 and GPT-4V outperformed GPT-3.5
[54]	AAMC PREview® exam	GPT-3.5, GPT-4	GPT-4 outperformed GPT-3.5; both versions outperformed humans
[55]	United States Medical Licensing Exam (USMLE) Step 2	GPT-3.5, GPT-4	GPT-4 outperformed GPT-3.5; correctly listed differential diagnoses from case reports
[56]	Thai local 4th year pharmacy exam	GPT-3.5	GPT-3.5 underperformed compared to humans
[57]	Dutch Clinical pharmacy exam (retrieved by parate kennis database)	GPT-4	GPT-4 outperformed pharmacists
[58]	Postgraduate orthopaedic qualifying exam	GPT-3.5, GPT-4, Bard	Bard outperformed GPT-3.5 and GPT-4
[59]	National Board of Medical Exam	GPT-4, GPT-3.5, Claude, Bard	GPT-4 outperformed GPT-3.5, Claude, and Bard
[60]	Anatomopathological Diagnostic and Therapeutic Procedures	ChatGPT	GPT-4 outperformed humans
[61]	American Shoulder and Elbow Surgeons Maintenance of Certification exam	GPT-3.5, GPT-4	GPT-4 underperformed compared to humans; performance varied by question type
[62]	nephrology fellows exam	GLT-4V	GPT-4V underperformed compared to humans; performance varied by question type
[63]	American Society for Surgery of the Hand Self-Assessment exam	GPT-4	GPT-4 underperformed compared to humans
[64]	Japanese Otolaryngology Board Certification exam	GPT-4V	Accuracy rate increased after translation to English; performance varied by question type
[65]	Pediatric Board Preparatory exam	GPT-3.5, GPT-4	GPT-4 outperformed GPT-3.5, limitations with complex questions
[66]	European Board exam in Neurological Surgery	GPT-3.5, Bing, Bard	Bard outperformed GPT-3.5, Bing, and humans; performance varied by question type
[67]	United States Medical Licensing Exam STEP 1-style questions	GPT-4	Passed exam
[68]	Italian gastroenterology-related national residency admission exam	GPT-3.5, Perplexity	GPT-3.5 outperformed Perplexity
[69]	Japanese National Medical Licensing exam	GPT-4V	68% accuracy; input of images does not further improve performance
[11]	Taiwan advanced medical licensing exam	GPT-4	Passed exams; chain-of-thought prompt increased performance
[70]	Israeli Hebrew Internal Medicine National Residency exam	GPT-3.5	Suboptimal performance (~37%)
[71]	Taiwan Nursing Licensing exam	GPT-4	Passed exam; performance varied by domain
[72]	International Society for Clinical Densitometry exam	GPT-3, GPT-4	GPT-4 outperformed GPT-3; GPT-4 passed exam
[73]	Indian National Eligibility cum Entrance Test	GPT-3.5, GPT-4, Bard	GPT-4 outperformed GPT-3.5 and Bard, GPT-4, and GPT-3.5 passed the exam
[74]	Taiwanese Stage 1 of Senior Professional and Technical Exam for Medical Doctors	GPT-4	Passed exam; performance varied by domain
[75]	Chinese national medical licensing exam	GPT-3.5	Does not pass the exams; performance varied by domain and question type
[76]	Local Belgium medical licensing exam of University of Antwerp	GPT-3.5, GPT-4, Bard, Bing, Claude instant, Claude+	All LLMs passed the exam; Bing and GPT-4 outperformed the other LLMs and humans
[77]	Medical licensing exam	GPT.3-5, GPT-4	GPT-4 outperformed GPT-3.5 and humans
[78]	Japanese National Medical Licensing Examination	GPT.3-5, GPT-4	GPT-4 outperformed GPT-3.5; optimization of prompts increases performance
[79]	Psychosomatic medicine	GPT-4	Passed exam
[80]	Otolaryngology-head and neck surgery certification exam	GPT-4	Passed exam
[81]	European Board of Urology (EBU) In-Service Assessment exam	GPT-3.5, GPT-4, Bing	GPT-4 and Bing outperformed GPT-3.5
[82]	Taiwanese Nuclear Medicine Specialty Exam	GPT-4	Passed exam; increased performance when adding chain-of-thoughts method
[83]	Medical cases in ophthalmology	GPT-3.5	GPT-3.5 underperformed compared to residents
[84]	Turkish Neurosurgical Society Proficiency Board exam	GPT-4	GPT-4 outperformed humans
[85]	Multi-Specialty Recruitment Assessment from PassMedicine	GPT-3.5, Llama2, Bard, Bing	Bing outperformed all other LLMs and humans

.

The general outcome of this literature research is that (i) different GPT versions are capable of answering medical exam questions of various disciplines, (ii) the effectiveness of LLMs in answering multiple-choice question assessments varies depending on the type of data input, (iii) the newer GPT-4 versions outperform GPT-3 versions, and (iv) the GPT-4 versions outperform in most cases other LLMs such as Bard or Bing.

Previous research showed that the effectiveness of generative pre-trained transformer (GPT) models in answering multiple-choice question assessments varies depending on the type of data input [21,86,87]. In one study, it was shown that the inclusion of code snippets makes multiple-choice questions more challenging for the GPT models, likely due to a combination of the models being somewhat more limited in handling computer programs compared with natural language and the inherent difficulty of the questions with code [86]. Another study nicely demonstrated that the feasibility of LLMs in providing accurate answers to multiple-choice questions related to feasibility issues is low [21].

Indeed, when ChatGPT was released more than a year ago, a number of studies were initiated addressing the performance of GPT in the context of academic examinations. For example, a study from the University of Minnesota Law School let GPT-3.5 take four final exams and compare the results to human students. Researchers found that GPT-3.5 could pass each exam but was not a great student [88]. In the context of medical exams or health issues, soon after its release, GPT3 has been tested in various disciplines, including hand surgery [89], questions related to pancreatic cancer and its surgical care [90], or basic life support and advanced cardiac life support [91]. Cai and colleagues investigated the ability of generative AI models to answer ophthalmology board-style questions [92]. They evaluated three LLMs with chat interfaces, Bing Chat (Microsoft) and GPT 3.5 and 4.o (OpenAI), and found that GPT-4.o and Bing Chat performed comparably to humans. Beyond, it was shown that ChatGPT-4.o outperformed ChatGPT 3.5, a finding that is in line with our results and results of other groups [8,9,12,87,93,94,95,96,97]. For example, a study using American Pharmacist Licensure Examination MC-questions as an input source for LLMs demonstrated a higher test performance of GPT-4 versus GPT-3 [87]. In another study, GPT-3.5 did not pass Polish medical examinations, whereas the GPT-4 models demonstrated the capability to pass the majority of the exams evaluated [96].

During our studies, we observed that re-feeding GPT-3.5 with a given query does not necessarily lead to the same answer. In a study by van Dis and colleagues [98], the authors asked ChatGPT the question, “how many patients with depression experience relapse after treatment”. As stated in that manuscript, ChatGPT generated an overly general text arguing that treatment effects are typically long-lasting. The authors criticized that ChatGPT did not provide accurate and detailed answers. More precisely, the authors stated that “numerous high-quality studies show … the risk of relapse ranges from 29% to 51% in the first year after treatment completion”. When we repeated the same query, ChatGPT generated a more sophisticated answer, stating that “According to research, approximately 50% of individuals who have experienced a single episode of depression are likely to experience a recurrence of the condition within five years”. Obviously, the underlying AI technology was able to learn from other textual input and, thus, can adopt its textual output. In line with our findings, the authors demonstrated that repeating the same query can generate a more detailed and accurate answer. Furthermore, in line with other reports, GPT-3.5 occasionally made unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain. While chain-of-thought may imitate human reasoning processes, the fundamental nature of LLMs remains that of black-box probabilistic models, lacking a mechanism to guarantee the faithfulness of reasoning.

5. Conclusions

With the release of ChatGPT in 2022, it has become increasingly clear that this form of AI technology will significantly impact the way academic teaching will be organized and tested. Both serious concerns and great advantages and support can potentially be expected from such kinds of technologies [98]. Of note, one limitation of previous GPT versions was that it was trained on data up to September 2021. However, there are now versions available with online access. Beyond, future studies have to show the accuracy of different LLMs such as, for example, Google Bard, BioMedLM (formerly known as PubMed GPT), or Med-PaLM, the latter being the first AI system to surpass the pass mark on the U.S. Medical License Exam (USMLE)-style questions. While AI opens up new possibilities in various fields, including medicine, it also brings challenges that necessitate a thorough expert assessment. For any exams, testers should try to come up with questions that cannot be correctly answered by any of them in our days available LLMs. Limitations of the LLMs in logical reasoning might be a versatile option to construct appropriate multiple-choice exam questions in the future [4,99].