Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive Study
Abstract
:1. Introduction
2. Background
3. Materials and Methods
3.1. Study Design
3.2. Participants
3.3. Data Sources/Measurement
3.3.1. Input Source
3.3.2. Encoding
3.4. Statistical Methods
4. Results
4.1. The Performance of ChatGPT on RNLE
4.2. Evaluation of the Advantages and Disadvantages of Using ChatGPT
5. Discussion
5.1. The Performance of ChatGPT on RNLE
5.2. To Evaluate the Advantages and Disadvantages of Using ChatGPT
6. Limitations
7. Conclusions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med. Educ. 2023, 9, e45312. [Google Scholar] [CrossRef] [PubMed]
- Talan, T.; Kalinkara, Y. The role of artificial intelligence in higher education: ChatGPT assessment for anatomy course. J. Manag. Inf. Syst. Comput. Sci. 2023, 7, 33–40. [Google Scholar] [CrossRef]
- Miao, H.; Ahn, H. Impact of ChatGPT on Interdisciplinary Nursing Education and Research. Asian Pac. Isl. Nurs. J. 2023, 7, e48136. [Google Scholar] [CrossRef] [PubMed]
- O’Connor, S.; ChatGPT. Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse? Nurse Educ. Pract. 2023, 66, 103537. [Google Scholar] [CrossRef] [PubMed]
- Qadir, J. Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. TechRxiv 2022. [Google Scholar] [CrossRef]
- Grunhut, J.; Wyatt, A.T.; Marques, O. Educating Future Physicians in Artificial Intelligence (AI): An Integrative Review and Proposed Changes. J. Med. Educ. Curric. Dev. 2021, 8, 23821205211036836. [Google Scholar] [CrossRef]
- Sallam, M.; Salim, N.; Barakat, M.; Al-Tammemi, A. ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J. 2023, 3, e103. [Google Scholar] [CrossRef]
- Sallam, M.; Salim, N.A.; Barakat, M.; Fayyad, D.; Hallit, S.; Harapan, H.; Hallit, R.; Mahafzah, A.; Salim, N.A.; Al-Tammemi, A.B. ChatGPT Output Regarding Compulsory Vaccination and COVID-19 Vaccine Conspiracy: A Descriptive Study at the Outset of a Paradigm Shift in Online Search for Information. Cureus 2023, 15, e35029. [Google Scholar] [CrossRef]
- Nalli, G.; Amendola, D.; Smith, S. Artificial intelligence to improve learning outcomes through online collaborative activities. In Proceedings of the European Conference on e-Learning, Brighton, UK, 27–28 October 2022; Volume 21, pp. 475–479. [Google Scholar]
- Ali, K.; Barhom, N.; Tamimi, F.; Duggal, M. ChatGPT—A double-edged sword for healthcare education? Implications for assessments of dental students. Eur. J. Dent. Educ. 2023. [Google Scholar] [CrossRef]
- Khan, R.A.; Jawaid, M.; Khan, A.R.; Sajjad, M. ChatGPT-Reshaping medical education and clinical management. Pak. J. Med. Sci. 2023, 39, 605–607. [Google Scholar] [CrossRef]
- Sallam, M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare 2023, 11, 887. [Google Scholar] [CrossRef] [PubMed]
- O’Connor, S.; Yan, Y.; Thilo, F.J.S.; Felzmann, H.; Dowding, D.; Lee, J.J. Artificial intelligence in nursing and midwifery: A systematic review. J. Clin. Nurs. 2023, 32, 2951–2968. [Google Scholar] [CrossRef] [PubMed]
- Seney, V.P.; Desroches, M.L.P.; Schuler, M.S.P. Using ChatGPT to Teach Enhanced Clinical Judgment in Nursing Education. Nurse Educ. 2023, 48, 124. [Google Scholar] [CrossRef] [PubMed]
- Kikuchi, R. Application of Artificial Intelligence Technology in Nursing Studies: A Systematic Review. 2021. Available online: https://www.himss.org/resources/application-artificial-intelligence-technology-nursing-studies-systematic-review (accessed on 10 September 2023).
- Cascella, M.; Montomoli, J.; Bellini, V.; Bignami, E. Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios. J. Med. Syst. 2023, 47, 33. [Google Scholar] [CrossRef]
- OpenAI. GPT-4 Technical Report. 2023. Available online: https://cdn.openai.com/papers/gpt-4.pdf (accessed on 20 September 2023).
- Ahn, C. Exploring ChatGPT for information of cardiopulmonary resuscitation. Resuscitation 2023, 185, 109729. [Google Scholar] [CrossRef] [PubMed]
- Bhutoria, A. Personalized education and artificial intelligence in the United States, China, and India: A systematic review using a Human-In-The-Loop model. Comput. Educ. Artif. Intell. 2022, 3, 100068. [Google Scholar] [CrossRef]
- Hutson, M. Could AI help you to write your next paper? Nature 2022, 611, 192–193. [Google Scholar] [CrossRef]
- O’Connor, S. Artificial intelligence and predictive analytics in nursing education. Nurse Educ. Pract. 2021, 56, 103224. [Google Scholar] [CrossRef]
- Else, H. Abstracts written by ChatGPT fool scientists. Nature 2023, 613, 423. [Google Scholar] [CrossRef]
- Benoit, J. ChatGPT for clinical vignette generation, revision, and evaluation. medRxiv-Med. Educ. 2023. [Google Scholar] [CrossRef]
- Huh, S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study. J. Educ. Eval. Health Prof. 2023, 20, 1. [Google Scholar] [CrossRef] [PubMed]
- Eysenbach, G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation with ChatGPT and a Call for Papers. JMIR Med. Educ. 2023, 9, e46885. [Google Scholar] [CrossRef]
- Wikipedia. Hallucination (Artificial Intelligence). 2023. Available online: https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) (accessed on 12 September 2023).
- Li’evin, V.; Hother, C.E.; Winther, O. Can large language models reason about medical questions? arXiv. 2022. Available online: https://arxiv.org/pdf/2207.08143.pdf (accessed on 5 September 2023).
- Antaki, F.; Touma, S.; Milad, D.; El-Khoury, J.; Duval, R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmol. Sci. 2023, 3, 100324. [Google Scholar] [CrossRef]
- Kung, T.H.; Cheatham, M.; Medenilla, A.; Sillos, C.; De Leon, L.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo, J.; et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit. Health 2023, 2, e0000198. [Google Scholar] [CrossRef]
- Gunawan, J. Exploring the future of nursing: Insights from the ChatGPT model. Belitung Nurs. J. 2023, 9, 1–5. [Google Scholar] [CrossRef] [PubMed]
- Mahmud, M.M.; Yaacob, Y.; Ramachandiran, C.R.; Ching, W.S.; Ismail, O. Theories into Practices: Bloom’s Taxonomy, Comprehensive Learning Theories (CLT) and E-Assessments. In Proceeding Book of 1st International Conference on Educational Assessment and Policy; Center for Educational Assessment: Los Angeles, CA, USA, 2019; Volume 2, pp. 22–27. [Google Scholar] [CrossRef]
- Finlay, G.D.; Rothman, M.J.; Smith, R.A. Measuring the modified early warning score and the Rothman index: Advantages of utilizing the electronic medical record in an early warning system. J. Hosp. Med. 2014, 9, 116–119. [Google Scholar] [CrossRef] [PubMed]
- Robert, N. How artificial intelligence is changing nursing. Nurs. Manag. 2019, 50, 30–39. [Google Scholar] [CrossRef]
- The Rothman Index. Model Development and Scientific Validation. 2023. Available online: www.perahealth.com/the-rothman-index/model-development-and-scientific-validation (accessed on 5 September 2023).
- Ministry of Examination. Regulations on Professional and Technical Examinations. Available online: https://wwwc.moex.gov.tw/main/ExamLaws/wfrmExamLaws.aspx?kind=3&menu_id=320&laws_id=110 (accessed on 11 September 2023).
- Ministry of Examination. Statistics of the Number of Applicants and Acceptance/Pass Rates for the Second Professional and Technical Examinations for Nursing License in the Year. Available online: https://wwwc.moex.gov.tw/main/ExamReport/wHandStatisticsFile.ashx?file_id=2122 (accessed on 11 September 2023).
- Cotton, D.; Cotton, P.; Shipway, R. Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innov. Educ. Teach. Int. 2023. [Google Scholar] [CrossRef]
- Ministry of Examination. Exam Question Query Platform. 2023. Available online: https://wwwq.moex.gov.tw/exam/wFrmExamQandASearch.aspx (accessed on 11 September 2023).
- OpenAI. ChatGPT (Mar 14 Version) [Large Language Model]. 2023. Available online: https://chat.openai.com/chat (accessed on 2 September 2023).
Exams | 2022 1st | 2022 2nd | 2023 1st | 2023 2nd | |
---|---|---|---|---|---|
Number | |||||
Candidates in Taiwan | 6873 | 14,699 | 7476 | 13,990 | |
Number of passes | 1031 | 6018 | 1267 | 5758 | |
Pass percentage | 18.17% | 45.31% | 20.42% | 45.73% |
RNLE Items Subjects * | 2022 1st | 2022 2nd | 2023 1st | 2023 2nd | ||||
---|---|---|---|---|---|---|---|---|
Correct/Incorrect | Scores | Correct/Incorrect | Scores | Correct/Incorrect | Scores | Correct/Incorrect | Scores | |
BMS ** | 53/27 | 66.25 | 24/26 | 48.00 | 39/11 | 78.00 | 35/15 | 70.00 |
FNNA | 49/31 | 61.25 | 30/50 | 37.50 | 42/38 | 52.50 | 51/29 | 63.75 |
MSN | 52/48 | 65.00 | 40/40 | 50.00 | 37/43 | 45.25 | 41/39 | 51.25 |
MPN | 45/35 | 56.25 | 47/33 | 58.75 | 43/37 | 53.75 | 47/33 | 58.75 |
PCHN | 56/24 | 70.00 | 51/29 | 63.75 | 39/41 | 42.50 | 48/32 | 60.00 |
Correct percentage | 63.75% | 58.37% | 54.05% | 60% | ||||
Mean score (SD) | 63.75 ± 5.22 | 51.6 ± 10.16 | 54.4 ± 4.02 | 60.75 ± 6.88 |
Question 1 | Which of the following supplies nourishment to the lungs through the bronchial arteries? Answer: (C) (A) Pulmonary artery (B) Pulmonary vein (C) Thoracic aorta (D) Internal thoracic artery [38]. |
ChatGPT | The answer is (C). The thoracic aorta is responsible for transporting oxygenated blood from the heart to the lungs and supplying the bronchial arteries to nourish lung tissues [39]. |
Question 2 | Mrs. Wang, G1P0, who has had normal prenatal check-ups in the past, presents at 32 weeks of gestation with a complaint of reduced fetal movements and is feeling very concerned. What would be the most appropriate action at this time? Answer: (C) (A) Instruct the patient to return home and record daily fetal movements. (B) Explain that this is a common occurrence at this stage and reassure her not to worry. (C) Schedule a non-stress test. (D) Immediately arrange for an induction of labor [38]. |
ChatGPT | The answer is (C). This test can help assess the well-being of the fetus by monitoring its heart rate and response to its own movements, providing valuable information about the baby’s condition [39]. |
Question 3 | Regarding the description of the Study of Osteoporotic Fractures frailty index (SOF), which of the following statements is incorrect? Answer: (B) (A) There are three assessment items. (B) Weight loss: A weight loss of more than 5 kg compared to one year ago. (C) Lower extremity function: Inability to rise from a chair without using hands five times. (D) Meeting two or more criteria qualifies as frailty [38]. |
ChatGPT | No answer. (A), (B), (C), and (D) are all correct. Based on the description, individuals meeting two or more of these three assessment criteria are classified as frail. In summary, according to the provided description, there are no errors. In summary, based on the descriptions provided in the data, there is no definitive answer to this question [39]. |
Question 4 | Ms. Chen, who has been in labor for 12 h, has given birth to a healthy newborn. Regarding the nursing interventions performed by the nurse during the fourth stage of labor, which of the following is not appropriate? Answer: (D) (A) Encouraging the mother to have eye-to-eye contact with the baby. (B) Wrapping the baby in a blanket to keep warm and encouraging touching the baby. (C) Delaying the administration of vitamin K1 and initiating skin-to-skin contact first. (D) Placing the newborn in an incubator to allow the mother to rest [38]. |
ChatGPT | The answer is (C). During the fourth stage of labor, it is customary for the healthcare team to administer a vitamin K1 injection to the newborn as a preventive measure against hemorrhagic disorders. Placing the newborn in an incubator to allow the mother to rest (D) is a reasonable arrangement to ensure that the mother has an opportunity to rest and recover postpartum [39]. |
Question 5 | Which of the following situations is less likely to result in black stools in a patient? Answer: (D) (A) Duodenal ulcer (B) Rectal bleeding (C) Iron supplementation (D) Overconsumption of pork liver [38]. |
ChatGPT | The answer is (C). Black stools are typically caused by upper gastrointestinal bleeding, and rectal bleeding, iron supplementation, and overconsumption of pork liver generally do not result in black stools. Food undergoes normal digestion, which does not result in black stools [39]. |
Question 6 | Which of the following statements regarding postoperative care for patients undergoing percutaneous transluminal coronary angioplasty (PTCA) for angina is incorrect? (A) Monitor pulses in both lower limbs. (B) Monitor electrocardiogram. (C) Encourage the patient to drink plenty of water. (D) Encourage the patient to perform active lower limb joint exercises. Answer: (D) [38]. |
ChatGPT | The answer is (C). Encouraging the patient to drink plenty of water is not typically a part of the immediate post-procedure care plan. The other options (A, B, and D) are more relevant to the post-PTCA care: The answer (D), it can help prevent complications such as deep vein thrombosis (DVT) and promote circulation. Option (C) is not a standard postoperative care practice for PTCA patients and is therefore the incorrect statement in this context [39]. |
Question 7 | Which of the following statements about home-based care services is less appropriate? Answer: (D). (A) Nursing personnel can independently establish home nursing facilities. (B) Home care can provide care for nasogastric tubes, tracheostomies, and urinary catheters. (C) Home rehabilitation services may be provided when deemed necessary based on an assessment by care management specialists. (D) Home pharmaceutical services are not within the scope of services [38]. |
ChatGPT-1 | The answer is (A). Nursing personnel who wish to independently establish home care facilities must adhere to specific laws and regulations, typically requiring certification and licensing by government authorities. Nursing personnel cannot arbitrarily establish home care facilities but must instead comply with relevant laws and regulations, subject to regulatory oversight [39]. |
ChatGPT-2 | The answer is (D). Home-based care services can include medication management and assistance, so stating that home pharmaceutical services are not within the scope of services is less appropriate [39]. |
Question 8 | According to the Occupational Health and Safety Regulations, how often should an employer arrange a general health examination for employees under the age of 40? (A) Once a year. (B) Once every two years. (C) Once every three years. (D) Once every five years. Answer: (C) [38]. |
ChatGPT | The answer is (B) in Chinese. However, the answer switch (C) in English. As per the Occupational Health and Safety Regulations, employers are required to arrange a general health examination once every two years. This regulation aims to ensure the well-being and health of the workforce by periodically assessing their general health status and identifying any potential health issues [39]. |
Question 9 | When nurses implement physical restraints on patients, which of the following considerations should be taken into account? ① Patients with excessive salivation should be maintained in a lateral position. ② Patients should be turned after being restrained for more than 2 h. ③ If Haloperidol is administered before restraint, blood pressure changes should be closely monitored. ④ Patients typically resist restraint, and it is not necessary to explain the reasons for the restraint. ⑤ After restraint, patients should be allowed to rest, and further discussions with the patient should be avoided. (A) ①②③. (B) ①③⑤. (C) ②③④. (D) ②④⑤. Answer: (A) [38]. |
ChatGPT | The answer is (B). Options ④ and ⑤ are not included in the correct answer because: It is generally recommended to explain the reasons for restraints to the patient to ensure their understanding and cooperation. Allowing the patient to rest after restraint is important, but it doesn’t preclude the possibility of further discussions or consultations with the patient [39]. |
Advantage | Disadvantage |
---|---|
|
|
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Huang, H. Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive Study. Healthcare 2023, 11, 2855. https://doi.org/10.3390/healthcare11212855
Huang H. Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive Study. Healthcare. 2023; 11(21):2855. https://doi.org/10.3390/healthcare11212855
Chicago/Turabian StyleHuang, Huiman. 2023. "Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive Study" Healthcare 11, no. 21: 2855. https://doi.org/10.3390/healthcare11212855