Advancing Medical Assistance: Developing an Effective Hungarian-Language Medical Chatbot with Artificial Intelligence
Abstract
:1. Introduction
1.1. Medical Chatbot Assistants
1.2. Natural Language Processing
1.3. Challenges in Developing Hungarian-Language AI Tools
2. Related Works
2.1. Medical Chatbots
2.2. Ethical Implications
3. Data Collection
4. Dataset
5. Methods
5.1. Long Short-Term Memory
5.1.1. Composition
5.1.2. Layers
5.1.3. Optimization and Loss Function
5.2. Bidirectional Encoder Representations from Transformers
5.2.1. Composition
5.2.2. Layers
5.2.3. Optimization and Loss Function
6. Results
7. Conclusions
8. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
NLP | Natural language processing |
MTS | Manchester Triage System |
GDPR | General Data Protection Regulation |
SVM | Support Vector Machine |
LSTM | Long Short-Term Memory |
RNN | Recurrent Neural Network |
BERT | Bidirectional Encoder Representations from Transformers |
ADAM | Adaptive Moment Estimation |
GPT | Generative Pre-trained Transformers |
AI | Artificial intelligence |
ReLU | Rectified Linear Unit |
NLLLoss | Negative log-likelihood loss |
TP | True positive |
AUC | Area Under the ROC Curve |
KNN | K-Nearest Neighbors |
Appendix A. Confusion Matrix of the LSTM Model
16 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 22 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 2 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 28 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 12 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
1 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 11 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 29 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 33 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 |
References
- Dharwadkar, R.; Deshpande, N. A Medical ChatBot. Int. J. Comput. Trends Technol. 2018, 60, 41–45. [Google Scholar] [CrossRef]
- Anjum, K.; Sameer, M.; Kumar, S. AI Enabled NLP based Text to Text Medical Chatbot. In Proceedings of the 2023 3rd International Conference on Innovative Practices in Technology and Management (ICIPTM), Uttar Pradesh, India, 22–24 February 2023; pp. 1–5. [Google Scholar]
- Kaponis, A.; Kaponis, A.A.; Maragoudakis, M. Case study analysis of medical and pharmaceutical chatbots in digital marketing and proposal to create a reliable chatbot with summary extraction based on users’ keywords. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA ’23, Corfu, Greece, 5–7 July 2023; pp. 357–363. [Google Scholar] [CrossRef]
- Athota, L.; Shukla, V.K.; Pandey, N.; Rana, A. Chatbot for Healthcare System Using Artificial Intelligence. In Proceedings of the 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, 4–5 June 2020; pp. 619–622. [Google Scholar] [CrossRef]
- Ghorashi, N.; Ismail, A.; Ghosh, P.; Sidawy, A.; Javan, R. AI-Powered Chatbots in Medical Education: Potential Applications and Implications. Cureus 2023, 15, e43271. [Google Scholar] [CrossRef]
- Matheny, M.; Israni, S.T.; Ahmed, M. Artificial Intelligence in Healthcare: The Hope, the Hype, the Promise, the Peril; National Academy of Medicine: Washington, DC, USA, 2020; pp. 1–15. [Google Scholar]
- Vincze, J. Virtual Reference Librarians (Chatbots). Libr. Hi Tech News 2017, 34, 5–8. [Google Scholar] [CrossRef]
- Shawar, B.; Atwell, E. Chatbots: Are they Really Useful? LDV Forum 2007, 22, 29–49. [Google Scholar] [CrossRef]
- Wang, J.; Hwang, G.H.; Chang, C.Y. Directions of the 100 most cited chatbot-related human behavior research: A review of academic publications. Comput. Educ. Artif. Intell. 2021, 2, 100023. [Google Scholar] [CrossRef]
- Caldarini, G.; Jaf, S.F.; McGarry, K.J. A Literature Survey of Recent Advances in Chatbots. Information 2021, 13, 41. [Google Scholar] [CrossRef]
- Allen, J.F. Natural language processing. In Encyclopedia of Computer Science; John Wiley and Sons Ltd.: Hoboken, NJ, USA, 2003; pp. 1218–1222. [Google Scholar]
- Jones, K.S. Natural Language Processing: A Historical Review. In Current Issues in Computational Linguistics: In Honour of Don Walker; Springer: Dordrecht, The Netherlands, 1994; pp. 3–16. [Google Scholar] [CrossRef]
- Khurana, D.; Koli, A.; Khatter, K.; Singh, S. Natural language processing: State of the art, current trends and challenges. Multimed. Tools Appl. 2023, 82, 3713–3744. [Google Scholar] [CrossRef]
- Laki, L.; Yang, Z. Sentiment Analysis with Neural Models for Hungarian. Acta Polytech. Hung. 2023, 20, 109–128. [Google Scholar] [CrossRef]
- Ostrogonac, S.J.; Rastović, B.S.; Popović, B. Automatic Job Ads Classification, Based on Unstructured Text Analysis. Acta Polytech. Hung. 2021, 18, 191–204. [Google Scholar] [CrossRef]
- Jurafsky, D.; Martin, J.H. Speech and Language Processing. In International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems; World Scientific Publishing: Singapore, 2019. [Google Scholar]
- Webster, J.J.; Kit, C. Tokenization as the initial phase in NLP. In Proceedings of the 14th Conference on Computational Linguistics, COLING ’92, Nantes, France, 23–28 August 1992; Volume 4, pp. 1106–1110. [Google Scholar] [CrossRef]
- Rai, A.; Borah, S. Study of Various Methods for Tokenization. In Applications of Internet of Things; Mandal, J.K., Mukhopadhyay, S., Roy, A., Eds.; Springer: Singapore, 2021; pp. 193–200. [Google Scholar]
- Mielke, S.J.; Alyafeai, Z.; Salesky, E.; Raffel, C.; Dey, M.; Gallé, M.; Raja, A.; Si, C.; Lee, W.Y.; Sagot, B.; et al. Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP. arXiv 2021, arXiv:2112.10508. [Google Scholar]
- Sun, K.; Qi, P.; Zhang, Y.; Liu, L.; Wang, W.Y.; Huang, Z. Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks. arXiv 2023, arXiv:2212.09912. [Google Scholar]
- Chowdhary, K.R. Fundamentals of Artificial Intelligence; Springer Nature: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
- Alwaisi, S.; Al-Radhi, M.; Németh, G. Multi-speaker child speech synthesis in low-resource Hungarian language. In Proceedings of the 2nd Workshop on Intelligent Infocommunication Networks, Systems and Services, Dubrovnik, Croatia, 24–27 September 2024; pp. 19–24. [Google Scholar] [CrossRef]
- Omar, M.; Choi, S.; Nyang, D.; Mohaisen, D. Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions. arXiv 2022, arXiv:2201.00768. [Google Scholar] [CrossRef]
- Novák, A.; Siklósi, B.; Oravecz, C. A New Integrated Open-source Morphological Analyzer for Hungarian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, 23–28 May 2016; pp. 1315–1322. [Google Scholar]
- Gräf, M.; Knitza, J.; Leipe, J.; Krusche, M.; Welcker, M.; Kuhn, S.; Mucke, J.; Hueber, A.; Hornig, J.; Klemm, P.; et al. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol. Int. 2022, 42, 2167–2176. [Google Scholar] [CrossRef]
- Cotte, F.; Mueller, T.; Gilbert, S.; Blümke, B.; Multmeier, J.; Hirsch, M.C.; Wicks, P.; Wolanski, J.; Tutschkow, D.; Schade Brittinger, C.; et al. Safety of Triage Self-assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study. JMIR Mhealth Uhealth 2022, 10, e32340. [Google Scholar] [CrossRef]
- Lee, H.; Kang, J.; Yeo, J. Medical Specialty Recommendations by an Artificial Intelligence Chatbot on a Smartphone: Development and Deployment. J. Med. Internet Res. 2021, 23, e27460. [Google Scholar] [CrossRef]
- Mohanty, S.; Chatterjee, S. ans Sarma, M.; Puravankara, R.; Bali, M. Diabot: A Predictive Medical Chatbot using Ensemble Learning. Int. J. Recent Technol. Eng. 2019, 8, 6334–6340. [Google Scholar]
- Kooli, C. Chatbots in Education and Research: A Critical Examination of Ethical Implications and Solutions. Sustainability 2023, 15, 5614. [Google Scholar] [CrossRef]
- Mohammad Amini, M.; Jesus, M.; Fanaei Sheikholeslami, D.; Alves, P.; Hassanzadeh Benam, A.; Hariri, F. Artificial Intelligence Ethics and Challenges in Healthcare Applications: A Comprehensive Review in the Context of the European GDPR Mandate. Mach. Learn. Knowl. Extr. 2023, 5, 1023–1035. [Google Scholar] [CrossRef]
- Chatbot Dataset. 2023. Available online: https://www.kaggle.com/datasets/niraliivaghani/chatbot-dataset (accessed on 19 January 2024).
- Himanshu. Sample for ChatBot. 2021. Available online: https://www.kaggle.com/code/himanshu01dadhich/sample-for-chatbot (accessed on 23 January 2021).
- Malik, K. Chatbot. 2020. Available online: https://github.com/Karan-Malik/Chatbot/blob/master/chatbot_codes/intents.json, (accessed on 16 January 2020).
- Pezoa, F.; Reutter, J.L.; Suarez, F.; Ugarte, M.; Vrgoč, D. Foundations of JSON schema. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Montreal, QC, Canada, 11–15 May 2016; pp. 263–273. [Google Scholar]
- SmartOne. 25+ Best Machine Learning Datasets for Chatbot Training in 2023. Available online: https://smartone.ai/blog/best-machine-learning-datasets-for-chatbot-training/ (accessed on 20 January 2024).
- 24 Best Machine Learning Datasets for Chatbot Training. 2023. Available online: https://kili-technology.com/data-labeling/machine-learning/24-best-machine-learning-datasets-for-chatbot-training (accessed on 21 January 2024).
- Laki, L.J.; Yang, Z.G. Neural machine translation for Hungarian. Acta Linguist. Acad. 2022, 69, 501–520. [Google Scholar] [CrossRef]
- Furkó, P. Perspectives on the Translation of Discourse Markers: A Case Study of the Translation of Reformulation Markers from English into Hungarian. Acta Univ. Sapientiae Philol. 2015, 6, 181–196. [Google Scholar] [CrossRef]
- Nikonorov, M.; Nikonorov, M. Create a Chatbot Trained on Your Own Data via the OpenAI API. 2024. Available online: https://www.sitepoint.com/create-data-trained-chatbot-openai-api/ (accessed on 22 January 2024).
- Brownlee, J. How to Develop a Neural Machine Translation System from Scratch. 2020. Available online: https://machinelearningmastery.com/develop-neural-machine-translation-system-keras/ (accessed on 21 February 2024).
- Roccetti, M.; Delnevo, G.; Casini, L.; Mirri, S. An alternative approach to dimension reduction for pareto distributed data: A case study. J. Big Data 2021, 8, 39. [Google Scholar] [CrossRef] [PubMed]
- Graves, A. Long Short-Term Memory. In Supervised Sequence Labelling with Recurrent Neural Networks; Springer: Berlin/Heidelberg, Germany, 2012; pp. 37–45. [Google Scholar] [CrossRef]
- Fjellström, C. Long Short-Term Memory Neural Network for Financial Time Series. arXiv 2022, arXiv:2201.08218. [Google Scholar]
- Lindemann, B.; Müller, T.; Vietz, H.; Jazdi, N.; Weyrich, M. A survey on long short-term memory networks for time series prediction. Procedia CIRP 2021, 99, 650–655. [Google Scholar] [CrossRef]
- Schmidt, R.M. Recurrent Neural Networks (RNNs): A gentle Introduction and Overview. arXiv 2019, arXiv:1912.05911. [Google Scholar]
- Sherstinsky, A. Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Phys. D Nonlinear Phenom. 2020, 404, 132306. [Google Scholar] [CrossRef]
- Marhon, S.A.; Cameron, C.J.F.; Kremer, S.C. Recurrent Neural Networks. In Handbook on Neural Information Processing; Springer: Berlin/Heidelberg, Germany, 2013; pp. 29–65. [Google Scholar] [CrossRef]
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed]
- Hochreiter, S. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst. 1998, 6, 107–116. [Google Scholar] [CrossRef]
- Noh, S.H. Analysis of Gradient Vanishing of RNNs and Performance Comparison. Information 2021, 12, 442. [Google Scholar] [CrossRef]
- Pascanu, R.; Mikolov, T.; Bengio, Y. On the difficulty of training Recurrent Neural Networks. arXiv 2023, arXiv:1211.5063. [Google Scholar]
- Rehmer, A.; Kroll, A. On the vanishing and exploding gradient problem in Gated Recurrent Units. IFAC-PapersOnLine 2020, 53, 1243–1248. [Google Scholar] [CrossRef]
- Ceni, A. Random orthogonal additive filters: A solution to the vanishing/exploding gradient of deep neural networks. arXiv 2022, arXiv:2210.01245. [Google Scholar]
- Lhasiw, N.; Sanglerdsinlapachai, N.; Tanantong, T. A Bidirectional LSTM Model for Classifying Chatbot Messages. In Proceedings of the 2021 16th International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP), Ayutthaya, Thailand, 21–23 December 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Graves, A.; Fernández, S.; Schmidhuber, J. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. In Proceedings of the Artificial Neural Networks: Formal Models and Their Applications—ICANN 2005, Warsaw, Poland, 11–15 September 2005; pp. 799–804. [Google Scholar]
- Cui, Z.; Ke, R.; Pu, Z.; Wang, Y. Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction. arXiv 2019, arXiv:1801.02143. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv 2019, arXiv:1810.04805. [Google Scholar]
- Bao, H.; Dong, L.; Piao, S.; Wei, F. BEiT: BERT Pre-Training of Image Transformers. arXiv 2022, arXiv:2106.08254. [Google Scholar]
- Yuan, M.; Wan, J.; Wang, D. CRM-SBKG: Effective Citation Recommendation by Siamese BERT and Knowledge Graph. In Proceedings of the 2023 IEEE 2nd International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), Changchun, China, 24–26 February 2023; pp. 909–914. [Google Scholar] [CrossRef]
- Hrinchuk, O.; Khrulkov, V.; Mirvakhabova, L.; Orlova, E.; Oseledets, I. Tensorized Embedding Layers for Efficient Model Compression. arXiv 2020, arXiv:1901.10787. [Google Scholar]
- Surkova, A.; Skorynin, S.; Chernobaev, I. Word embedding and cognitive linguistic models in text classification tasks. In Proceedings of the XI International Scientific Conference Communicative Strategies of the Information Society, CSIS’2019, St. Petersburg, Russia, 25–26 October 2019. [Google Scholar] [CrossRef]
- Dinh, T.N.; Pham, P.; Nguyen, G.L.; Vo, B. Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding. Expert Syst. Appl. 2024, 243, 122911. [Google Scholar] [CrossRef]
- Webb, G.I. Encyclopedia of Machine Learning. In Encyclopedia of Machine Learning; Springer: Boston, MA, USA, 2010; p. 744. [Google Scholar] [CrossRef]
- Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv 2015, arXiv:1502.03167. [Google Scholar]
- Boucherouite, S.; Malinovsky, G.; Richtárik, P.; Bergou, E.H. Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization. arXiv 2022, arXiv:2209.07883. [Google Scholar] [CrossRef]
- Fatras, K.; Zine, Y.; Majewski, S.; Flamary, R.; Gribonval, R.; Courty, N. Minibatch optimal transport distances; analysis and applications. arXiv 2021, arXiv:2101.01792. [Google Scholar]
- Gomez, C.; Selman, B.; Weinberger, K.Q.; Bjorck, J. Understanding Batch Normalization. arXiv 2018, arXiv:1806.02375. [Google Scholar]
- Zhang, B.; Zhao, Q.; Feng, W.; Lyu, S. AlphaMEX: A smarter global pooling method for convolutional neural networks. Neurocomputing 2018, 321, 36–48. [Google Scholar] [CrossRef]
- Shustanov, A.; Yakimov, P. Modification of single-purpose CNN for creating multi-purpose CNN. J. Phys. Conf. Ser. 2019, 1368, 052036. [Google Scholar] [CrossRef]
- Fabris, F.; Freitas, A.A. Analysing the Overfit of the Auto-sklearn Automated Machine Learning Tool. In Proceedings of the Machine Learning, Optimization, and Data Science, Siena, Italy, 10–13 September 2019; pp. 508–520. [Google Scholar]
- Sperl, P.; Kao, C.Y.; Chen, P.; Lei, X.; Böttinger, K. DLA: Dense-Layer-Analysis for Adversarial Example Detection. In Proceedings of the 2020 IEEE European Symposium on Security and Privacy (EuroS&P), Genoa, Italy, 7–11 September 2020; pp. 198–215. [Google Scholar] [CrossRef]
- Agarap, A.F. Deep learning using rectified linear units (relu). arXiv 2018, arXiv:1803.08375. [Google Scholar]
- Javid, A.M.; Das, S.; Skoglund, M.; Chatterjee, S. A ReLU Dense Layer to Improve the Performance of Neural Networks. arXiv 2020, arXiv:2010.13572. [Google Scholar]
- Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958. [Google Scholar]
- S, R.; Bharadwaj, A.S.; S K, D.; Khadabadi, M.S.; Jayaprakash, A. Digital Implementation of the Softmax Activation Function and the Inverse Softmax Function. In Proceedings of the 2022 4th International Conference on Circuits, Control, Communication and Computing (I4C), Bangalore, India, 21–23 December 2022; pp. 64–67. [Google Scholar] [CrossRef]
- Kouretas, I.; Paliouras, V. Simplified Hardware Implementation of the Softmax Activation Function. In Proceedings of the 2019 8th International Conference on Modern Circuits and Systems Technologies (MOCAST), Thessaloniki, Greece, 13–15 May 2019; pp. 1–4. [Google Scholar] [CrossRef]
- Zhang, Z. Improved Adam Optimizer for Deep Neural Networks. In Proceedings of the 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), Banff, AB, Canada, 4–6 June 2018; pp. 1–2. [Google Scholar] [CrossRef]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Li, P.; He, X.; Song, D.; Ding, Z.; Qiao, M.; Cheng, X.; Li, R. Improved Categorical Cross-Entropy Loss for Training Deep Neural Networks with Noisy Labels. In Proceedings of the Pattern Recognition and Computer Vision: 4th Chinese Conference, PRCV 2021, Beijing, China, 29 October–1 November 2021; pp. 78–89. [Google Scholar] [CrossRef]
- Banerjee, K.; C, V.P.; Gupta, R.R.; Vyas, K.; H, A.; Mishra, B. Exploring Alternatives to Softmax Function. arXiv 2020, arXiv:2011.11538. [Google Scholar]
- Gordon-Rodríguez, E.; Loaiza-Ganem, G.; Pleiss, G.; Cunningham, J.P. Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning. arXiv 2020, arXiv:2011.05231. [Google Scholar]
- Nitish, S.; Darsini, R.; Shashank, G.S.; Tejas, V.; Arya, A. Bidirectional Encoder Representation from Transformers (BERT) Variants for Procedural Long-Form Answer Extraction. In Proceedings of the 2022 12th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India, 27–28 January 2022; pp. 71–76. [Google Scholar] [CrossRef]
- Cesar, L.B.; Manso-Callejo, M.A.; Cira, C.I. BERT (Bidirectional Encoder Representations from Transformers) for Missing Data Imputation in Solar Irradiance Time Series. Eng. Proc. 2023, 39, 9026. [Google Scholar] [CrossRef]
- Charoenkwan, P.; Nantasenamat, C.; Hasan, M.M.; Manavalan, B.; Shoombuatong, W. BERT4Bitter: A bidirectional encoder representations from transformers (BERT)-based model for improving the prediction of bitter peptides. Bioinformatics 2021, 37, 2556–2562. [Google Scholar] [CrossRef] [PubMed]
- Brown, T.B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language Models are Few-Shot Learners. arXiv 2020, arXiv:2005.14165. [Google Scholar]
- GPT-4 Technical Report. arXiv 2023, arXiv:2303.08774.
- Savelka, J.; Agarwal, A.; An, M.; Bogart, C.; Sakr, M. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research—Volume 1, ICER ’23, Chicago, IL, USA, 7–11 August 2023; pp. 78–92. [Google Scholar] [CrossRef]
- MacNeil, S.; Tran, A.; Mogil, D.; Bernstein, S.; Ross, E.; Huang, Z. Generating Diverse Code Explanations using the GPT-3 Large Language Model. In Proceedings of the 2022 ACM Conference on International Computing Education Research—Volume 2, ICER ’22, Lugano and Virtual Event, Switzerland, 7–11 August 2022; pp. 37–39. [Google Scholar] [CrossRef]
- Luo, X.; Ding, H.; Tang, M.; Gandhi, P.; Zhang, Z.; He, Z. Attention Mechanism with BERT for Content Annotation and Categorization of Pregnancy-Related Questions on a Community Q&A Site. Proc. IEEE Int. Conf. Bioinform. Biomed. 2021, 2020, 1077–1081. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv 2023, arXiv:1706.03762. [Google Scholar]
- Córdova Sáenz, C.A.; Becker, K. Assessing the use of attention weights to interpret BERT-based stance classification. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT ’21, Melbourne, VIC, Australia, 14–17 December 2022; pp. 194–201. [Google Scholar] [CrossRef]
- Cui, B.; Li, Y.; Chen, M.; Zhang, Z. Fine-tune BERT with Sparse Self-Attention Mechanism. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; pp. 3548–3553. [Google Scholar] [CrossRef]
- Pratiwi, H.; Windarto, A.P.; Susliansyah, S.; Aria, R.R.; Susilowati, S.; Rahayu, L.K.; Fitriani, Y.; Merdekawati, A.; Rahadjeng, I.R. Sigmoid Activation Function in Selecting the Best Model of Artificial Neural Networks. J. Phys. Conf. Ser. 2020, 1471, 012010. [Google Scholar] [CrossRef]
- Kalman, B.; Kwasny, S. Why tanh: Choosing a sigmoidal function. In Proceedings of the [Proceedings 1992] IJCNN International Joint Conference on Neural Networks, Baltimore, MD, USA, 7–11 June 1992; Volume 4, pp. 578–581. [Google Scholar] [CrossRef]
- Waoo, A.A.; Soni, B.K. Performance Analysis of Sigmoid and Relu Activation Functions in Deep Neural Network. In Intelligent Systems; Sheth, A., Sinhal, A., Shrivastava, A., Pandey, A.K., Eds.; Springer: Singapore, 2021; pp. 39–52. [Google Scholar]
- Haykin, S. Neural Networks: A Comprehensive Foundation; Prentice Hall PTR: Hoboken, NJ, USA, 1994. [Google Scholar]
- Kelley, H.J. Gradient theory of optimal flight paths. Ars J. 1960, 30, 947–954. [Google Scholar] [CrossRef]
- Wei, R.; Yin, H.; Jia, J.; Benson, A.R.; Li, P. Understanding Non-linearity in Graph Neural Networks from the Bayesian-Inference Perspective. arXiv 2022, arXiv:2207.11311. [Google Scholar]
- de Brébisson, A.; Vincent, P. An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family. arXiv 2015, arXiv:1511.05042. [Google Scholar]
- Zhu, D.; Yao, H.; Jiang, B.; Yu, P. Negative Log Likelihood Ratio Loss for Deep Neural Network Classification. arXiv 2018, arXiv:1804.10690. [Google Scholar]
- de Carvalho, M.C.M.; Dougherty, M.S.; Fowkes, A.S.; Wardman, M.R. Forecasting Travel Demand: A Comparison of Logit and Artificial Neural Network Methods. J. Oper. Res. Soc. 1998, 49, 717–722. [Google Scholar] [CrossRef]
- Goutte, C.; Gaussier, E. A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In Proceedings of the ECIR 2005: Advances in Information Retrieval, Santiago de Compostela, Spain, 21–23 March 2005; Volume 3408, pp. 345–359. [Google Scholar] [CrossRef]
- Kiarash, M.; He, Z.; Zhai, M.; Tung, F. Ranking Regularization for Critical Rare Classes: Minimizing False Positives at a High True Positive Rate. arXiv 2023, arXiv:2304.00049. [Google Scholar]
- Yacouby, R.; Axman, D. Probabilistic Extension of Precision, Recall, and F1 Score for More Thorough Evaluation of Classification Models. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, Online, 20 November 2020; pp. 79–91. [Google Scholar] [CrossRef]
- McHugh, M.L. Interrater reliability: The kappa statistic. Biochem. Med. 2012, 22, 276–282. [Google Scholar] [CrossRef]
- Warrens, M.J. Five ways to look at Cohen’s kappa. J. Psychol. Psychother. 2015, 5, 1–4. [Google Scholar] [CrossRef]
- Ting, K.M. Confusion Matrix. In Encyclopedia of Machine Learning and Data Mining; Springer: Boston, MA, USA, 2017; p. 260. [Google Scholar] [CrossRef]
- Pan, Y.; Li, X.; Yang, Y.; Dong, R. Multi-Source Neural Model for Machine Translation of Agglutinative Language. Future Internet 2020, 12, 96. [Google Scholar] [CrossRef]
Reference | Functionality | Accuracy | Interface | Integration Support |
---|---|---|---|---|
[25] | Diagnostic support | 0.7 | User-friendly | Healthcare data |
[28] | Predictive diagnostics | 0.86 | User-friendly | NLU |
[27] | Specialty matching | 0.96 | Smartphones | Healthcare data system |
[1] | Query processing | 0.95 | User-friendly | API integration |
Symptom | Disease |
---|---|
My eyes are inflamed. | Conjunctivitis |
I feel tired and irritable during the day. | Insomnia |
Warm, red skin over the affected joint. | Arthritis |
Throbbing in the neck or ears. | High blood pressure |
Class | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Precision | 1.0000 | 1.0000 | 1.0000 | 0.6667 | 0.7500 | 1.0000 | 1.0000 | 0.7778 | 1.0000 | 0.0000 | 0.8000 | 1.0000 |
Recall | 0.9412 | 1.0000 | 1.0000 | 0.5000 | 1.0000 | 0.9655 | 1.0000 | 1.0000 | 0.6364 | - | 1.0000 | 1.0000 |
F1-score | 0.9697 | 1.0000 | 1.0000 | 0.5714 | 0.8571 | 0.9824 | 1.0000 | 0.8750 | 0.7778 | - | 0.8889 | 1.0000 |
Class | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 |
Precision | 1.0000 | 1.0000 | - | 0.0000 | 0.6250 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 |
Recall | 0.9333 | 0.9091 | 0.0000 | - | 0.8333 | 1.0000 | 1.0000 | 1.0000 | - | 0.7778 | 0.9286 | 0.0000 |
F1-score | 0.9655 | 0.9524 | - | - | 0.7143 | 1.0000 | 1.0000 | 1.0000 | - | 0.8750 | 0.9630 | 0.0000 |
Class | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 |
Precision | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 | 0.3333 | 0.0000 | 1.0000 | 0.0000 | 0.4000 |
Recall | 1.0000 | 1.0000 | 0.2500 | - | 1.0000 | 1.0000 | - | 1.0000 | - | 1.0000 | - | 1.0000 |
F1-score | 1.0000 | 1.0000 | 0.4000 | - | 1.0000 | 1.0000 | - | 0.5000 | - | 1.0000 | - | 0.5714 |
Colds | 0 | 0 | 0 | 0 | 5 | 2 |
Pneumonia | 0 | 1 | 0 | 0 | 0 | 0 |
Sore throat | 0 | 0 | 0 | 0 | 0 | 0 |
Nasal flushing | 0 | 0 | 0 | 0 | 1 | 0 |
Nasal congestion | 0 | 0 | 0 | 0 | 2 | 3 |
Fever | 0 | 0 | 0 | 0 | 0 | 7 |
Colds | Pneumonia | Sore throat | Nasal flushing | Nasal congestion | Fever |
Chickenpox | 2 | 0 | 0 | 0 |
Fungal skin | 2 | 6 | 0 | 0 |
Eczema | 0 | 0 | 7 | 0 |
Sunburn | 0 | 0 | 0 | 0 |
Chickenpox | Fungal skin | Eczema | Sunburn |
Sprain | 0 | 1 | 0 | 0 | 0 |
Bite | 0 | 0 | 0 | 0 | 0 |
Bruise | 0 | 1 | 0 | 0 | 0 |
Cut | 0 | 1 | 0 | 0 | 0 |
Bleeding | 0 | 2 | 0 | 0 | 0 |
Sprain | Bite | Bruise | Cut | Bleeding |
Hypotension | 16 | 0 | 0 | 0 |
Hypertension | 0 | 13 | 0 | 0 |
Heart attack | 0 | 0 | 13 | 0 |
Anemia | 0 | 0 | 0 | 8 |
Hypotension | Hypertension | Heart attack | Anemia |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Simon, B.; Hartveg, Á.; Dénes-Fazakas, L.; Eigner, G.; Szilágyi, L. Advancing Medical Assistance: Developing an Effective Hungarian-Language Medical Chatbot with Artificial Intelligence. Information 2024, 15, 297. https://doi.org/10.3390/info15060297
Simon B, Hartveg Á, Dénes-Fazakas L, Eigner G, Szilágyi L. Advancing Medical Assistance: Developing an Effective Hungarian-Language Medical Chatbot with Artificial Intelligence. Information. 2024; 15(6):297. https://doi.org/10.3390/info15060297
Chicago/Turabian StyleSimon, Barbara, Ádám Hartveg, Lehel Dénes-Fazakas, György Eigner, and László Szilágyi. 2024. "Advancing Medical Assistance: Developing an Effective Hungarian-Language Medical Chatbot with Artificial Intelligence" Information 15, no. 6: 297. https://doi.org/10.3390/info15060297
APA StyleSimon, B., Hartveg, Á., Dénes-Fazakas, L., Eigner, G., & Szilágyi, L. (2024). Advancing Medical Assistance: Developing an Effective Hungarian-Language Medical Chatbot with Artificial Intelligence. Information, 15(6), 297. https://doi.org/10.3390/info15060297