Can Generative AI Contribute to Health Literacy? A Study in the Field of Ophthalmology

Abstract

ChatGPT, a generative artificial intelligence model, can provide useful and reliable responses in the field of ophthalmology, comparable to those of medical professionals. Twelve frequently asked questions from ophthalmology patients were selected, and responses were generated both in the role of an expert user and a non-expert user. The responses were evaluated by ophthalmologists using three scales: Global Quality Score (GQS), Reliability Score (RS), and Usefulness Score (US), and analyzed statistically through descriptive study, association, and comparison. The results indicate that there are no significant differences between the responses of expert and non-expert users, although the responses from the expert user tend to be slightly better rated. ChatGPT’s responses proved to be reliable and useful, suggesting its potential as a complementary tool to enhance health literacy and alleviate the informational burden on healthcare professionals.

1. Introduction

Artificial intelligence (AI) has demonstrated remarkable potential in the healthcare field since its inception in the 1970s with pioneering projects such as Mycin [1], which used statistical inference rules to emulate the decision-making process of medical experts through predictive reasoning. However, it is in recent years that there has been a surge in innovative solutions in the healthcare sector, with generative artificial intelligence (AI) focused on creating original content being among the most prominent.

It is essential to understand that artificial intelligence is an umbrella term encompassing various primary techniques, the most used being machine learning (ML), deep learning (DL), and natural language processing (NLP).

Initially launched in 2022 with the ChatGPT-3.5 version and updated to ChatGPT-4 in 2023, this generative AI model falls under the category of chatbots, designed to simulate human conversations using natural language processing (NLP) and machine learning (ML) [2], proving to be useful in various fields, including healthcare, thanks to its iterative learning capability and human feedback. Currently, there are several versions available, ranging from a free basic ChatGPT-4 to subscription versions like ChatGPT-4 Pro. To avoid confusion about the different versions, we will refer to ChatGPT-4 whenever discussing this generative artificial intelligence tool.

ChatGPT is characterized by its iterative capability, meaning it can expand its knowledge through reinforced learning from human feedback. The uses of this chatbot are diverse, including generating conversations, personal assistants, search engines, human resource planning, and even writing scientific articles. It has been asserted that there are no significant differences between a cover letter written by a human and one written by ChatGPT [3].

Numerous articles have been written about the ability to simulate human writing and reasoning, driven by concerns that AI might replace humans. These studies conclude that it is difficult to distinguish the origin of a text [4], whether robotic or human, to the extent that its use for writing scientific articles is acknowledged due to its excellent performance [5,6]. As far as is known, at least four scientific articles have listed ChatGPT as a co-author since its launch in 2022. However, there are also critical voices. In the context of scientific publishing, many publishers oppose the use of AI, some for the entire process and others only allowing it to improve the writing process, but not the interpretation of data [7], as explicitly stated in their publication conditions.

ChatGPT’s ability to integrate multiple types of input (such as patient questions) and generate outputs (detailed and comprehensible responses) offers a unique opportunity to enhance the user experience in the healthcare field. Although this tool does not replace direct interaction with healthcare professionals, it can better prepare patients for their consultations by reducing the barrier of technical knowledge and facilitating a more informed dialogue between the patient and the professional. The importance of online information search tools cannot be overlooked either [8]. The success of traditional search engines is now joined by chatbots, and the habit of online health inquiries is no exception.

There are still few definitive studies on the reliability of the information provided by GPT in the healthcare setting. However, the existing studies offer revealing data on its potential. Madrid-García et al. (2023) established that GPT is an excellent medium for answering medical questions with high precision and that its clinical reasoning is consistent [9].

A concept gaining importance in these times, driven by patient empowerment and active participation, is health literacy. This is defined as the health information an individual needs to know to make good decisions about their health [10]. It involves possessing the necessary knowledge, motivation, and skills to access, understand, and apply information in the healthcare domain.

This literacy is nourished by various sources, one of the most important being mass media, which includes social networks, search engines, and chatbots. All these sources contribute to informing or misinforming about health, including health promotion or public health programs. They even have the capacity to modify public actions and perceptions [11], especially when they come from groups the population identifies as reliable sources of information, such as healthcare professionals [12].

The Ottawa Charter for Health Promotion [13] provides an essential framework for the development of health literacy. According to this framework, as individuals increase their ability to control their health, their overall well-being tends to improve. For this reason, health literacy is recognized as a determinant of health.

The use of digital tools, such as chatbots, provides the population with a set of skills that allow them to access, understand, and use health-related information. If fed with reliable data, these tools can become great allies in the fight against misinformation and will be highly useful in enhancing patient independence and decision-making capacity. This can help avoid productivity losses among professionals by automating repetitive information delivery, where generative AI can be of great assistance.

Few studies have verified the capabilities of information created by generative AI, making it necessary to validate the reliability and usefulness of the information provided by ChatGPT. There is scepticism about whether the information obtained may be incorrect or dangerous for users relying on chatbots for medical information.

This study focuses on evaluating how multimodal technologies, specifically generative AI, can support and enhance health literacy, a fundamental aspect for informed decision-making by patients. It assesses ChatGPT’s ability to match the quality of responses provided by medical professionals in ophthalmology and determines whether its responses contribute to improving health literacy.

Specifically, we aim to (i) determine whether generative AI can provide responses of comparable quality to those of expert professionals, (ii) assess whether the information generated by ChatGPT-3.5 can be effectively understood and used by non-expert users in ophthalmology, and (iii) discuss how the information provided by ChatGPT-3.5 can be integrated into a broader framework of ophthalmological health literacy, which includes the critical interpretation and application of this information under the supervision of healthcare professionals.

2. Materials and Methods

Regarding the use of ChatGPT-3.5 in our study, we would like to clarify that this version was active at the time the research was conducted, and as it was the free version, it was the most accessible and widely used by the public. This decision allowed us to evaluate the potential of the tool in a more realistic and representative context of widespread use among users.

The selection process for the panel of ophthalmologists was designed to ensure the participation of experts with extensive clinical experience and deep knowledge in ophthalmology, thus ensuring the quality and relevance of the questions selected for the study. The only inclusion criterion was that the ophthalmologists be active in clinical practice and have experience in managing a wide variety of ophthalmological conditions.

Ophthalmologists from various leading medical institutions in Spain were invited to participate, forming a diverse panel in terms of both subspecialties within ophthalmology (retina, cornea, glaucoma, among others) and geographical distribution. These experts were asked to provide a list of the most common questions they receive from their patients during consultations, covering a wide range of topics, from diagnostic aspects to treatment and prognosis considerations.

A total of 36 different questions were received, which were reviewed by a member of the research team, who analyzed their relevance, clarity, and representativeness. From these, the 12 most recurrent and representative questions were selected, ensuring they covered a diversity of common topics in ophthalmological practice and were understandable to a general audience.

The selected questions were adapted to improve their clarity and were then used to generate responses via ChatGPT-3.5 in two different scenarios: in the first, without a specific prompt (simulating the interaction of a non-expert user), and in the second, with a prompt that simulated the response of a medical specialist (simulating the interaction of an expert user). It is important to note that “expert” users, such as healthcare professionals, and “non-expert” users, such as patients, have different levels of understanding and expectations regarding the information provided. These differences are reflected in how they interact with the chatbot to obtain information, so the aim is to determine whether the responses generated by ChatGPT-3.5 are equally useful and reliable for users with different levels of health literacy, regardless of their level of knowledge.

The ophthalmologists evaluated ChatGPT-3.5’s responses to the 12 selected questions using three validated scales. The Global Quality Scale (GQS) [14,15], on a five-point scale (where 1 indicates the worst information and 5 the best), was used to assess the quality and ease of understanding of the information. The Reliability Score (RS) scale evaluated the reliability of the medical sources, and the Usefulness Score (US) scale assessed the usefulness of the response for the patient, both on a scale of 0 to 7, where 7 indicates the highest reliability and usefulness, respectively.

A total of 42 questionnaires were received evaluating the questions, 21 for each type of prompt (customized or not). However, one of the questionnaires from the non-expert user group was discarded due to being incomplete. After an initial exploratory analysis, the sample was expanded to 400 questionnaires through synthetic data generation using the bootstrapping technique, which allows for the creation of random samples from an existing dataset through resampling. Python libraries pandas, numpy, and os were used.

The list of questions provided by the panel of experts was as follows:

• Q1. Will my cataracts be operated on with a laser?
• Q2. Is eye pressure related to blood pressure?
• Q3. How many drops of eye drops should I use each time?
• Q4. If I have surgery, will I stop being myopic?
• Q5. Why don’t ophthalmologists get surgery to get rid of their glasses?
• Q6. What is considered normal eye pressure?
• Q7. Until when will my myopia increase?
• Q8. Is vitreous detachment the same as retinal detachment?
• Q9. When will I regain my vision after retinal surgery?
• Q10. Do I have macula, doctor?
• Q11. Will it hurt a lot if I get an injection in my eye?
• Q12. Is astigmatism for distance or near vision?

Descriptive, association, and comparison statistical analyses were performed using the open-source software R version 4.4.1 and the graphical interface RStudio, now called Posit. The Shapiro–Wilk and Levene tests were used to verify the assumptions of normality and homoscedasticity, respectively. The level of statistical significance was set at p < 0.05.

3. Results

The results obtained using ChatGPT-3.5 demonstrated that the generated responses were rated as useful and reliable by the ophthalmologists in both user categories (expert and non-expert). Representative examples of the responses are included to illustrate its performance in the context of ophthalmological questions, highlighting the usefulness and reliability of the answers.

Example 1—Question about surgical procedures For the question “Will my cataract surgery be done with a laser?”, ChatGPT 3.5 generated the following response, as a non-expert user: “Cataract treatment is commonly performed using a technique called phacoemulsification, which uses ultrasound to dissolve the cloudy lens. In some cases, a laser may be used for certain parts of the procedure, but it is not the most common approach”. The ophthalmologists rated this response highly for usefulness (5.5/7) and reliability (5/7) for non-expert users, noting that the answer is accurate and understandable, although more details could be added for expert users.

Example 2—Question about ophthalmological conditions: For the question “What is considered a normal eye pressure?”, ChatGPT 3.5 generated the following response, as an expert user: “Normal intraocular pressure generally ranges between 10 and 21 mmHg. However, higher values may be indicative of glaucoma, especially if other risk factors are present. It is important to have regular check-ups to monitor eye pressure”. This response was rated 6/7 for usefulness and 6.5/7 for reliability by the ophthalmologists, who highlighted that the response is appropriate for an expert context, providing accurate information with a level of detail that supports its validity.

The results show that there are no statistically significant differences between the responses of expert and non-expert users, although the expert user responses received slightly higher ratings. On the GQS scale, both types of responses exceeded the midpoint of the scale, with an average score of 7.66 for the expert user and 7.38 for the non-expert user. Regarding reliability (RS) and usefulness (US), the expert user responses were also rated higher, although some specific questions received better scores when answered without prompt customization.

Regarding the reliability of the responses, on the Global Quality Score (GQS) scale, no statistically significant differences were found between the responses of the expert user and the non-expert user for most questions. Both types of responses exceeded the midpoint of the scale, with a mean of 7.66 for the expert user and 7.38 for the non-expert user. Individually, in questions Q3, Q5, Q8, Q9, and Q10, the non-expert user responses were rated higher than those of the expert user, although these differences were not statistically significant (

Table 1. Results according to the Global Quality Score (GQS).

	Non-Expert User				Expert User				X2	UMW
			95% CI				95% CI
	Mean	Median	Lower	Upper	Mean	Median	Lower	Upper	p	p
Q1	2	2	0.93	3.07	3.63	4	2.63	4.62	0.202	0.024
Q2	2.71	2	1.44	3.99	3.63	3.5	3	4.25	0.149	0.12
Q3	3.71	4	2.69	4.74	3.5	3.5	2.5	4.5	0.966	0.72
Q4	3.14	3	2.15	4.13	3.38	3	2.49	4.26	0.55	0.658
Q5	4.43	4	3.93	4.92	4.25	4.5	3.51	4.99	0.252	0.8
Q6	3.86	4	3.03	4.69	4.38	4	3.94	4.81	0.109	0.211
Q7	4	4	3.08	4.92	4	4.5	3	5	0.733	0.95
Q8	4.29	5	3.26	5.31	4.13	4	3.3	4.95	0.689	0.569
Q9	4.43	5	3.7	5.16	3.75	3.5	2.78	4.72	0.49	0.24
Q10	3.57	4	2.07	5.07	3.38	3.5	2.04	4.71	0.992	0.81
Q11	4	4	3.08	4.92	4.25	4.5	3.51	4.99	0.736	0.617
Q12	4.14	5	3.02	5.27	3.75	3.5	2.78	4.72	0.782	0.497

).

On the Reliability Score (RS) scale, the responses were generally considered reliable, with a mean of 4.99 for the expert user and 4.75 for the non-expert user. In questions Q3, Q8, Q9, and Q10, the non-expert user responses slightly exceeded those of the expert user, without statistically significant differences (

Table 2. Results according to the Reliability Scale (RS).

	Non-Expert User				Expert User				X2	UMW
			95% CI				95% CI
	Mean	Median	Lower	Upper	Mean	Median	Lower	Upper	p	p
Q1	3	2	0.93	5.07	4.5	4.5	3.16	5.84	0.324	0.196
Q2	3.29	3	2.01	4.56	4.38	4	3.04	5.71	0.509	0.214
Q3	5.43	6	4.38	6.48	4.63	4.5	3.22	6.03	0.138	0.289
Q4	4.14	4	3.02	5.27	4.5	4	3.09	5.91	0.46	0.763
Q5	5.29	5	4.26	6.31	5.38	5.5	4.2	6.55	0.808	0.812
Q6	5.43	5	4.25	6.61	6.13	6	6.3	6.95	0.232	0.277
Q7	5.14	5	4.02	6.27	5.13	5	3.83	6.42	0.157	0.953
Q8	5.43	6	3.75	7.1	5.38	6	3.97	6.78	0.919	0.856
Q9	5.57	6	4.39	6.75	5	4.5	3.45	6.55	0.501	0.588
Q10	4.71	6	2.28	7.14	4.25	4.5	1.94	6.56	0.92	0.718
Q11	4.71	4	3.23	6.2	5.63	6	4.37	6.88	0.625	0.259
Q12	4.86	5	3.22	6.5	5	5	3.33	6.67	0.382	0.769

).

Analyzing the usefulness for the patient, the Usefulness Score (US) scale showed a mean of 5.12 for the expert user and 4.73 for the non-expert user. Only in question Q1 were statistically significant differences found (p = 0.008) in favor of the expert user (M = 4.75) compared to the non-expert user (M = 2). In questions Q3, Q9, Q10, and Q12, the non-expert user responses were rated higher than those of the expert user, although these differences did not reach statistical significance (

Table 3. Results according to the Usefulness Scale (US).

	Non-Expert User				Expert User				X2	UMW
			95% CI				95% CI
	Mean	Median	Lower	Upper	Mean	Median	Lower	Upper	p	p
Q1	2	1	0.31	3.96	4.75	5	3.35	6.15	0.035	0.008
Q2	3	3	1.49	4.51	4.75	5	3.51	5.99	0.482	0.064
Q3	5	6	3.4	6.6	4.75	5	3.15	6.35	0.199	0.814
Q4	4.29	4	2.62	4.95	4.75	4	3.51	5.99	0.491	0.513
Q5	6.14	7	5.02	7.27	5.75	6.5	4.43	7.07	0.919	0.659
Q6	5.29	5	4.01	6.56	6.13	6	5.3	6.95	0.267	0.249
Q7	5.14	5	3.9	6.39	5.25	5.5	3.93	6.57	0.808	0.812
Q8	5.57	6	3.98	7.16	5.5	6	4.16	6.84	0.966	0.808
Q9	5.71	5	4.55	6.87	5	5	3.27	6.73	0.688	0.501
Q10	4.57	5	2.51	6.63	4.38	4.5	2.19	6.56	0.829	0.905
Q11	5	5	9.93	6.07	5.5	6	4.32	6.68	0.46	0.406
Q12	5.14	5	3.5	6.78	5	5	3.33	6.67	0.626	0.906

).

The results indicate that, in general, there are no statistically significant differences between the responses generated by ChatGPT under the expert user role and those generated under the non-expert user role. However, the expert user responses tend to be slightly better rated in terms of reliability and usefulness.

4. Discussion

This study has demonstrated the ability of generative AI to generate responses with above-average reliability and usefulness, correctly answering all the questions posed, according to the consulted ophthalmologists. This opens an interesting field of application in patient education and even for future professionals [16].

The responses generated by ChatGPT-3.5 were consistently rated as useful and reliable by the ophthalmologists, both in contexts involving expert and non-expert users. However, it is important to clarify that, while the responses generated by ChatGPT-3.5 exceeded the midpoint of the reliability and usefulness scales, not all observed differences reached a level of statistical significance. This underscores the potential of AI as a complementary tool in education and health literacy, while also highlighting the importance of personalizing and adapting the model to different user profiles.

Differences were observed in the evaluation of responses between expert and non-expert users, with responses in expert user contexts being rated slightly higher. It is noteworthy that 4 questions received better ratings from non-expert users than from expert users, accounting for 33.3% of the questions asked. This may be influenced by the chatbot’s training or the availability of more consistent information. In digital health, this means that AI-generated interfaces and responses might need adjustments depending on the user’s prior knowledge, which could maximize the tool’s usefulness and effectiveness. For the future development of AI in health, this differentiated approach could improve user satisfaction and precision, enhancing the adoption of these technologies in clinical settings.

In the context of health literacy, the model demonstrated its ability to provide accurate and accessible information, which is essential for empowering patients in managing their ophthalmological health. Moreover, the evaluations conducted by the ophthalmologists indicate that, although the responses are consistent and of high quality, there is room to further personalize the prompts to better meet the specific needs of expert users. This suggests the importance of correctly training the model with personalized prompts, guided by medical specialists, to avoid unrepresentative answers.

The responses provided by generative AI proved to be particularly useful for non-expert users, offering clear and accessible explanations on complex ophthalmological topics. This highlights AI’s potential to democratize access to healthcare information, especially in specialized fields like ophthalmology. By providing answers that are easily understood by patients without medical training, ChatGPT can play a crucial role in improving health literacy, helping patients make informed decisions about their eye health. This accessibility is an important step towards health equity, ensuring that even those with fewer resources or knowledge can access accurate and useful information.

The expansion of the study through bootstrapping allowed us to observe specific patterns in the questions where ChatGPT-3.5’s responses were more useful for non-expert users. Identifying these patterns provides valuable insights for designing digital interventions aimed at improving health literacy. Developers of digital health technologies can use these findings to optimize how AI generates educational content, focusing on areas where non-expert users need more support. Additionally, this approach can guide the creation of personalized AI-based educational modules that dynamically adapt to the user’s needs in real-time.

The reliability, accuracy, and usefulness of the information provided can ensure its use as an autonomous information source, with specialists defining its scope of action. The ability of ChatGPT-3.5 to generate responses that meet the expectations of users with different levels of knowledge reinforces its applicability in health education and improving health literacy. This can reduce the informational burden on healthcare professionals, allowing them to focus on more complex cases. Thus, a potential use as a virtual assistant for certain frequently asked questions by patients could relieve specialists from informational bureaucracy or provide a foundational knowledge base to make the consultation with the specialist more understandable. Furthermore, its rapid advancement and implementation benefit all stakeholders in the healthcare sector, including patients [17], so we are likely to witness a true explosion of interventions aimed at them.

There are several limitations associated with the use of ChatGPT, the most notable being the uncertainty regarding data updates and, consequently, the training updates of the models, since it does not perform web searches and rejects information from the past two years.

The study was based on 12 ophthalmological questions selected by a panel of experts. While these questions were chosen to represent a diverse range of common inquiries in clinical practice, the limited number of questions may not capture the full spectrum of possible interactions and concerns that patients may have in a real-world setting. This limitation implies that the study’s results may not be fully representative of all potential ophthalmological consultations. Selecting a greater number of questions could have provided a broader and more detailed view of how ChatGPT-3.5 handles different types of ophthalmological inquiries. Additionally, with only 12 questions, it is possible that some key areas of ophthalmology were not covered, limiting the applicability of the results to more general contexts or specific ophthalmological subspecialties.

Related to the above, the limited number of questions could increase the variance in the responses generated by ChatGPT-3.5, as some questions might be more easily handled by the AI than others, which could bias the results. This aspect may restrict the study’s ability to generalize its findings to other areas of medicine or even to other specialties within ophthalmology. Although the use of techniques like bootstrapping helped increase the sample size and reduce variance, a larger initial sample of questions could have provided a more robust database for analysis.

Another limitation related to ethics is the tendency of these models to amplify biases [18,19], such as automation bias or attributing greater importance to information generated by a computer program. However, improvements and continuous training, due to the iterative process carried out daily by millions of users, are enhancing this aspect.

Finally, we cannot overlook ChatGPT’s so-called hallucinations, which occur due to the need to always provide an answer, even when not all the information is available, creating strange and disconnected compositions from reality.

Artificial intelligence is integrating into healthcare information systems, coexisting and potentially becoming a fundamental pillar of the medicine of the near future [20]. From this statement, new research should begin, exploring different concepts of the relationship between technology and health.

5. Conclusions

This study demonstrates that generative artificial intelligence, such as ChatGPT, has significant potential to complement patient education in the healthcare field by providing responses with above-average reliability and usefulness, as assessed by the consulted ophthalmologists. In this context, we propose that generative AI, specifically ChatGPT, should be considered a tool within the broader process of ophthalmological health literacy, rather than a comprehensive solution. By providing accurate and well-founded information, it can facilitate the first stage of health literacy: access to information. However, we acknowledge that this is only the first step in a more complex process that includes understanding, critical evaluation, and the application of information by the patient—tasks that require the ongoing guidance of healthcare professionals.

The use of ChatGPT in healthcare could alleviate the informational burden on healthcare professionals, allowing them to focus on more critical aspects of patient care. By delivering precise and useful information, ChatGPT can better prepare patients for their consultations, improve their understanding of medical procedures, and increase their independence in managing their health.

To maximize the potential of ChatGPT and ensure its effective integration into healthcare, it is essential to personalize the prompts used to generate responses, ensuring they reflect the accuracy and depth required in the medical field. This will lead to close collaboration with medical specialists to design and oversee these prompts. Additionally, it is necessary to keep the data constantly updated and implement strategies to identify and mitigate biases in the generated responses, ensuring fairness and representativeness in the information provided. Lastly, mechanisms of transparency must be established in the use of generative AI, along with ongoing supervision by experts to validate the quality of the responses and their suitability for patient needs.

Finally, this research not only provides new insights into the applicability of generative AI in the medical field but also offers an interdisciplinary perspective that combines data analysis, artificial intelligence, and communication sciences.