Google Gemini’s Performance in Endodontics: A Study on Answer Precision and Reliability

Abstract

(1) Background: Large language models (LLMs) are revolutionising various scientific fields by providing advanced support tools. However, the effectiveness of these applications depends on extensive, up-to-date databases to ensure certainty and predictive power. Transparency about information sources in Medicine remains a significant issue. (2) Methods: To evaluate Google Gemini’s accuracy and reproducibility in endodontic diagnosis and treatment, 60 questions were designed based on the European Society of Endodontology Position Statements. Thirty questions were randomly selected and answered using Gemini during April 2023. Two endodontic experts independently scored the answers using a 3-point Likert scale. Discrepancies were resolved by a third expert. The relative frequency and absolute percentage of responses were detailed. Accuracy was assessed using the Wald binomial method, and repeatability was assessed using percentage agreement, Brennan and Prediger’s coefficient, Conger’s generalised kappa, Fleiss’ kappa, Gwet’s AC, and Krippendorff’s alpha, all with 95% confidence intervals. Statistical analysis was performed using STATA software. (3) Results: A total of 900 answers were generated. The percentage of correct answers varied from 0% to 100% per question. Overall accuracy was 37.11% with a 95% confidence interval of 34.02–40.32%; (4) Conclusions: Gemini is not currently designed for medical use and therefore needs to be used with caution when considered for this purpose.

1. Introduction

Access to large language models (LLMs) is transforming several scientific disciplines by providing a wide range of support tools. These models, an advanced form of artificial intelligence (AI), allow interaction with information available on the Internet or any database to which they are applied, with the ability to analyse complex algorithms and self-learning. We are entering a new era in medicine. AI can be applied to clinical practice through risk assessment models, improving diagnostic accuracy and workflow efficiency [1].

These models can be used in medical diagnostics as well as in symptom checking applications to provide potential diagnoses and assist users in self-classification [2]. They can also support clinical decision making and patient monitoring, even suggesting inter-professional consultations based on signs and symptoms [3].

However, for these applications to be truly effective, it is essential to have a large number of up-to-date databases. This increases the level of certainty, credibility, and predictive power of the models [4]. One of the main problems with these models is the lack of transparency about the sources of information they use. This can raise concerns about the veracity and reliability of the information provided, as users have no way of verifying the accuracy of the data or the context in which they originated [5].

There are numerous tools on the market. One of the most popular is Chat Generative Pre-trained Transformer (ChatGPT, OpenAI, San Francisco, CA, USA) [6], the latest version of which was released in February 2023. In the same year were also introduced Gemini from Google DeepMind (London, UK) and Google Bard chatbot (Alphabet, Mountain View, CA, USA), although their development dates back to 2020 [7].

Gemini is trained on a multimodal and multilingual dataset that includes web documents, books, and code, as well as image, audio, and video data [8].

The use of chatbots in healthcare could be of significant benefit, as they can assist both professionals and patients [9]. Despite a positive reception from users and early promising signs, further studies are needed to assess the full extent of the utility of these chatbots in healthcare settings [10]. In dentistry, although the scientific literature has already demonstrated the effectiveness of chatbot applications [11,12], the use of Gemini remains untested in terms of determining its reliability and reproducibility in a clinical setting.

Therefore, the aim of this study was to evaluate the accuracy and reproducibility of Gemini-generated answers to diagnostic and treatment questions in endodontics, to test its effectiveness as a diagnostic tool for dentists.

2. Materials and Methods

2.1. Ethical Approval

This research did not require ethical approval since no human participants were involved.

2.2. Question Design

To evaluate the accuracy and repeatability of the Gemini Pro software (Google, Mountain View, CA, USA) in generating answers to questions about diagnosis and treatment in endodontics, two endodontic specialists (M.T. and V.D.F.) designed 60 questions based on the European Society of Endodontology (ESE) Position Statements [13]. These documents were chosen because they contain clinical practice guidelines developed by a group of recognised experts in endodontics.

Each question was preceded by the prompt “To answer this question, imagine that you are an endodontist and I am a general dentist”, thus directing the answers to a specific situation.

Thirty questions were then randomly selected using a random number generator within a range using JavaScript ECMAScript 2021 (Mozilla Corporation, San Francisco, CA, USA). The selected questions are listed in

Table 1. Selected questions used in the answer generation.

Guide Used (DOI)	Question
10.1111/iej.13916	Why does resorption appear in a tooth?
10.1111/iej.13916	Which are the types of root resorption in dentistry?
10.1111/iej.13916	What is the treatment for internal resorption in a tooth?
10.1111/iej.13916	What is the treatment for external resorption in a tooth?
10.1111/iej.12606	What is the objective of regenerative endodontics?
10.1111/iej.12606	When is it contraindicated to perform regenerative endodontics?
10.1111/iej.12606	What are the success criteria for regenerative endodontics?
10.1111/iej.12606	What is the percentage, amount and time of sodium hypochlorite that we should use to irrigate during regenerative endodontics at the first appointment?
10.1111/iej.12606	What is the irrigation protocol during the first appointment in regenerative endodontics?
10.1111/iej.12606	After how much time can we perform the second session in regenerative endodontics?
10.1111/iej.12606	What is the irrigation protocol in the second session of regenerative endodontics?
10.1111/iej.12606	When should we do follow up on regenerative endodontics?
10.1111/iej.12606	What type of material should be placed on top of the blood clot when we do regenerative root canal?
10.1111/iej.13456	What is a surgical extrusion of a tooth?
10.1111/iej.13456	What is an intentional reimplantation of a tooth?
10.1111/iej.13456	What is dental autotransplantation?
10.1111/iej.13456	What is the total extra-alveolar time during an intentional reimplantation?
10.1111/iej.13456	What should splinting be like after intentional reimplantation in dentistry?
10.1111/iej.13456	How long should splinting be maintained after intentional reimplantation?
10.1111/iej.13456	How long is the maximum time to do a root canal after surgical procedures?
10.1111/iej.13456	Should the socket be curetted in a surgical extrusion?
10.1111/iej.13456	Should endodontics be performed on the donor tooth during autotransplantation?
10.1111/iej.13456	Do donor teeth with open apex require root canal treatment in autotransplantation?
10.1111/iej.12741	What are the indications for the use of antibiotics in endodontics?
10.1111/iej.12741	When should we administer antibiotic prophylaxis in endodontics?
10.1111/iej.12741	What are the recommended antibiotics in endodontics?
10.1111/iej.13916	When is performing a CBCT in endodontics indicated?
10.1111/iej.13916	How much FOV should we use in endodontics when performing a CBCT?
10.1111/iej.13916	Should we evaluate the complete CBCT volume?
10.1111/iej.13916	Can CBCTs be performed on children in endodontics?

.

2.3. Generating Answers in Gemini

Thirty answers were generated for each question using Gemini with the ‘new chat’ function on each occasion. The answers were obtained at different times of the day, namely morning, afternoon, and evening, during the month of April 2023. All answers were stored in a spreadsheet (Excel version 16; (Microsoft, Redmond, WA, USA)).

2.4. Evaluation of Gemini Answers by Human Experts

Two experts in endodontics (B.T. and A.S.) independently scored the 900 generated answers using the guidelines used to design the questions, with a 3-point Likert scale (0 = incorrect; 1 = partial or incomplete; 2 = correct). Discrepancies in the ratings were resolved by a third expert (YF).

2.5. Statistical Analysis

The 900 ratings of each answer were analysed using the statistical software STATA, version BE 14 (StataCorp, Texas, TX, USA).

For each of the 30 questions, the relative frequency (n) and absolute percentage (%) of answers that received different ratings from the experts were described. To analyse the accuracy of the answers generated by Gemini, the proportion of answers scoring 2 (correct) was calculated for both the total set of questions and for each individual question, together with its 95% confidence interval using the Wald binomial method.

To assess repeatability, the consistency of ratings across replicates was analysed using weighted concordance analysis for ordinal categories and multiple replicates. Measures included percent agreement, Brennan and Prediger’s coefficient, Conger’s generalised kappa, Fleiss’ kappa, Gwet’s AC, and Krippendorff’s alpha with their corresponding 95% confidence intervals. The 900 evaluations of each response were analysed using STATA statistical software, version BE 14 (StataCorp, Texas, TX, USA).

3. Results

Gemini generated 900 answers, providing 30 answers for each of the 30 questions asked.

Table 2. Frequency of expert grading for Gemini answers.

	Incorrect (0)		Partially Correct or Incomplete (1)		Correct (2)
Question	n	%	n	%	n	%
1	0	0.00	30	100.00	0	0.00
2	0	0.00	30	100.00	0	0.00
3	0	0.00	11	36.67	19	63.33
4	0	0.00	0	0.00	30	100.00
5	0	0.00	30	100.00	0	0.00
6	0	0.00	16	53.33	14	46.67
7	0	0.00	30	100.00	0	0.00
8	0	0.00	12	40.00	18	60.00
9	0	000	0	0.00	30	100.00
10	30	100.00	0	0.00	0	0.00
11	29	96.67	1	3.33	0	0.00
12	0	0.00	30	100.00	0	0.00
13	0	0.00	22	73.33	8	26.67
14	27	90.00	0	0.00	3	10.00
15	0	0.00	0	0.00	30	100.00
16	0	0.00	29	96.67	1	3.33
17	0	0.00	0	0.00	30	100.00
18	4	13.33	26	86.67	0	0.00
19	1	3.33	29	96.67	0	0.00
20	30	100.00	0	0.00	0	0.00
21	0	0.00	22	73.33	8	26.67
22	22	73.33	5	16.67	3	10.00
23	0	0.00	0	0.00	30	100.00
24	0	0.00	17	56.67	13	43.33
25	0	0.00	27	90.00	3	10.00
26	25	83.33	1	3.33	4	13.33
27	0	0.00	0	0.00	30	100.00
28	0	0.00	0	0.00	30	100.00
29	27	90.00	3	10.00	0	0.00
30	0	0.00	0	0.00	30	100.00

shows the frequency of expert grading for Gemini answers. The percentage of correct repetitions for the questions ranged from 0% to 100%, depending on the question. The experts considered that eight questions showed accuracy and repeatability; five questions showed incomplete or partial answers and repeatability, and two questions (12 and 22) showed both inaccuracy and repeatability. A total of 16 questions showed varying degrees of repeatability. The overall accuracy of the answers generated by Gemini was 37.11%, with a 95% confidence interval ranging from 34.02% to 40.32%, as shown in

Table 3. Overall accuracy of the answers.

	Accuracy	Standard Error	CI 95% (Wilson)
Overall 900 responses	37.11%	0.02	34.02%	40.32%

. The repeatability of expert grading of Gemini generated answers ranged from substantial to almost perfect.

It was observed that Gemini could not solve one of the questions (“What are the recommended antibiotics in endodontics?”) in any of the repetitions given, obtaining as an answer “I can’t assist you with that, as I’m only a language model and don’t have the capacity to understand and respond” (Figure 1).

4. Discussion

The introduction of new technologies in the healthcare sector has had a transformative effect on the diagnosis and treatment of diseases as well as facilitating the work of healthcare professionals.

In particular, the use of LLMs by professionals is proving to be a promising tool, not only from a clinical perspective [14], but also in the training of future healthcare professionals [15]. These models can analyse large amounts of medical data, provide preliminary diagnoses and offer treatment recommendations, which can optimise clinical decision making.

However, despite the considerable potential of these applications, the most well-known programmes are currently not specifically designed for medical use or use in medical settings. This means that they need to be used with sufficient caution and under professional supervision [16].

In addition, there are concerns about the transparency and provenance of the data used to train these models, which may affect the reliability of the results.

Although ChatGPT versions 3.5 and 4 (OpenAI, San Francisco, CA, USA) have become the standard reference for studies on the use of chatbots in the medical sciences [17,18], particularly in dentistry [19,20], there is limited evidence on the performance of other LLMs, such as Gemini.

The Gemini software is presented as a new tool that, unlike ChatGPT, is characterised by its multimodal capability [7]. This means that Gemini can understand and process photos, audio, text, and other types of input natively, without the need to implement additional plug-ins.

Its use has been documented in educational activities for dental students [21,22], demonstrating its potential as a didactic tool. However, its utility as a diagnostic tool has not received the same level of scientific development and validation as its competitor, ChatGPT.

The results of this study indicate that the accuracy of the answers provided by the Gemini model is between 34.02% and 40.32% with 95% confidence. These results suggest that although Gemini has the potential to be a useful tool for diagnostic support, its current performance may not be sufficiently reliable for critical clinical applications, as approximately one third of the answers provided by Gemini are correct, which may not be sufficient for applications requiring high precision, such as diagnosis or treatment recommendations.

Although the high repeatability of Gemini’s answers is a positive aspect, providing results considered to be between substantial and near perfect, it is important to note that the consistency of incorrect answers remains an issue, highlighting the need for improvements in the model’s accuracy.

In this context, Ozden et al. [23] analysed the ability of Gemini to answer 25 dichotomous (yes/no) questions about dental trauma based on the International Association of Dental Traumatology guidelines, [24], and obtained 64% correct answers with Gemini, compared to 37.11% in the present study. However, Suárez et al. obtained slightly lower accuracy results than the present study when evaluating the performance of ChatGPT in generating answers in the field of endodontics. In another study [25], ChatGPT-3.5 provided more credible information compared to Bing and Google Bard. Therefore, different factors could influence the different results observed in the literature. The type of question (dichotomous versus open-ended) could influence the results of the tool. However, similar to the present study, several studies [11,23] also used guidelines published by international scientific societies, providing scientific certainty both in the formulation of the questions and in the analysis of the answers obtained.

In the present study, the variability in the percentages of correct answers for diagnosis suggests that the accuracy of the answers depends significantly on how the questions are phrased. This may be due to factors such as the clarity of the question, the relevance of the content, or the inherent difficulty of the question, which is a limitation of the study. Analysis of this variability may help to improve the wording of questions in future surveys or exams, in order to obtain a more accurate measure of Gemini’s knowledge or skills.

On the other hand, during the answer acquisition phase, it was observed that the language used by Gemini differed from that used by ChatGPT [26], even using expressions such as “This is my recommendation” (Figure 2). This use of language suggests a significant level of personalisation and familiarity in the answers. However, this apparent personalisation in Gemini’s answers could be problematic and ethically questionable, especially in sensitive contexts such as health. Such expressions may mislead the user into believing that they are receiving advice from a human professional. Therefore, the structure of Gemini’s answers should ensure that the user understands that they are interacting with an AI model and not a human professional.

The use of chatbots is a promising future tool that can not only assist healthcare professionals, but also make a significant contribution to the sustainability of the healthcare sector. A platform that can access scientific literature databases without the limitations of a manual search, or with the immediacy that current technologies allow, would enable clinicians to make more reliable diagnoses, benefiting both the patient and society as a whole, and reducing the environmental impact associated with traditional medical care processes.

At present, there are places in the world where access to quality scientific information is very complex, both because of the cost involved and the quality of the computer systems However, with just one of these tools it would be possible to have access to millions of pieces of scientific evidence more efficiently and sustainably.

In this sense, the systems that power these applications need to be trained by clinicians to prevent the errors that chatbots can make, as this study shows, which currently limit their use in dentistry. In addition, optimising these tools to increase their sustainability could lead to more efficient resource management and a reduction in waste, thereby contributing to global sustainable development goals.

5. Conclusions

It is important to recognise that Google’s Gemini application is not designed for clinical use, as is clearly stated in its documentation. This limitation underlines the need for extreme caution when using it in medical contexts. Although the use of chatbots promises to be one of the most transformative developments in healthcare, the use of Gemini in diagnostic processes still lacks the necessary scientific robustness. It is therefore essential that its use in medical practice is always monitored by professionals to ensure its effectiveness and compliance with the ethical standards required for such applications. Further studies are needed to determine the performance of chatbots in medical applications, as well as comparisons of the effectiveness of the different LLMs currently available on the market.