Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
Abstract
:1. Introduction
1.1. Research Purpose
1.2. Related Work
2. Language Capacity of GPT-4o
2.1. Performance on Exams
2.1.1. Performance on USMLE
2.1.2. Performance on CFA
2.1.3. Performance on SAT
2.1.4. Performance on MBE
2.2. Reasoning
2.3. Language Translation
2.3.1. Data
2.3.2. Evaluation Method
- 1 indicates that the vectors are identical.
- 0 indicates that the vectors are orthogonal (i.e., no similarity).
- −1 indicates that the vectors are opposed.
2.3.3. Results
2.3.4. Limitations
3. Vision Capacity of GPT-4o
3.1. Image Classification: Fruits Classification
3.2. Image Classification: Driver Drowsiness Detection
3.3. Image Classification: Crop Disease Classification
3.4. Image Classification: Glaucoma Detection
3.5. Image Classification: Cancer, Tumor, and Aneurysm Detection
3.6. Image Captioning
4. Speech Capacity of GPT-4o
4.1. Emotion Detection
4.2. Accent Detection
5. Multimodal Capacity of GPT-4o
5.1. Visual Question Answering
5.2. Vision-Language Capabilities
6. Implications, Limitations, and Future Work
6.1. Implications
6.2. Limitations
6.3. Future Work
7. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Gemini Team; Anil, R.; Borgeaud, S.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A.M.; Hauth, A.; Millican, K.; et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv 2024, arXiv:2312.11805. [Google Scholar]
- Korinek, A. Language Models and Cognitive Automation for Economic Research; National Bureau of Economic Research: Cambridge, MA, USA, 2023. [Google Scholar]
- Floridi, L.; Chiriatti, M. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds Mach. 2020, 30, 681–694. [Google Scholar] [CrossRef]
- Dillion, D.; Mondal, D.; Tandon, N.; Gray, K. Large Language Models as Moral Experts? GPT-4o Outperforms Expert Ethicist in Providing Moral Guidance; OSF: Peoria, IL, USA, 2024. [Google Scholar] [CrossRef]
- Ray, S. Google CEO Says Gemini AI’s ‘Unacceptable’ Responses Offended Users and Showed Bias. 2024. Available online: https://www.forbes.com/sites/siladityaray/2024/02/28/google-ceo-says-gemini-ais-unacceptable-responses-offended-users-and-showed-bias/?sh=250e1a1b1103 (accessed on 15 June 2024).
- Ongsulee, P. Artificial intelligence, machine learning and deep learning. In Proceedings of the 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE), Bangkok, Thailand, 22–24 November 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1–6. [Google Scholar]
- Thirunavukarasu, A.J.; Ting, D.S.J.; Elangovan, K.; Gutierrez, L.; Tan, T.F.; Ting, D.S.W. Large language models in medicine. Nat. Med. 2023, 29, 1930–1940. [Google Scholar] [CrossRef]
- Hayawi, K.; Shahriar, S. AI Agents from Copilots to Coworkers: Historical Context, Challenges, Limitations, Implications, and Practical Guidelines. Preprints 2024. [Google Scholar] [CrossRef]
- Aher, G.V.; Arriaga, R.I.; Kalai, A.T. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In Proceedings of the 40th International Conference on Machine Learning, PMLR, Honolulu, HI, USA, 23–29 July 2023; pp. 337–371. Available online: https://proceedings.mlr.press/v202/aher23a.html (accessed on 15 June 2024).
- Mannuru, N.R.; Shahriar, S.; Teel, Z.A.; Wang, T.; Lund, B.D.; Tijani, S.; Pohboon, C.O.; Agbaji, D.; Alhassan, J.; Galley, J.; et al. Artificial intelligence in developing countries: The impact of generative artificial intelligence (AI) technologies for development. Inf. Dev. 2023, 02666669231200628. [Google Scholar] [CrossRef]
- Lund, B.D.; Wang, T.; Mannuru, N.R.; Nie, B.; Shimray, S.; Wang, Z. ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. J. Assoc. Inf. Sci. Technol. 2023, 74, 570–581. [Google Scholar] [CrossRef]
- Hu, B.; Sheng, Q.; Cao, J.; Shi, Y.; Li, Y.; Wang, D.; Qi, P. Bad actor, good advisor: Exploring the role of large language models in fake news detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 26–27 February 2024; Volume 38, pp. 22105–22113. [Google Scholar]
- Koubaa, A. GPT-4 vs. GPT-3.5: A Concise Showdown. Preprints 2023. [Google Scholar] [CrossRef]
- Coyne, S.; Sakaguchi, K.; Galvan-Sosa, D.; Zock, M.; Inui, K. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. arXiv 2023, arXiv:2303.14342. [Google Scholar]
- Salman, S.; Liu, X. Overfitting Mechanism and Avoidance in Deep Neural Networks. arXiv 2019, arXiv:1901.06566. [Google Scholar]
- Shen, X.; Wu, Y.; Backes, M.; Zhang, Y. Voice Jailbreak Attacks Against GPT-4o. arXiv 2024, arXiv:2405.19103. [Google Scholar]
- Ying, Z.; Liu, A.; Liu, X.; Tao, D. Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks. arXiv 2024, arXiv:2406.06302. [Google Scholar]
- Kalyanpur, A.; Saravanakumar, K.; Barres, V.; Chu-Carroll, J.; Melville, D.; Ferrucci, D. LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic. arXiv 2024, arXiv:2406.17663. [Google Scholar]
- Zhang, N.; Sun, Z.; Xie, Y.; Wu, H.; Li, C. The latest version ChatGPT powered by GPT-4o: What will it bring to the medical field? Int. J. Surg. 2024. [Google Scholar] [CrossRef]
- Wang, H.; Xu, J.; Xie, S.; Wang, R.; Li, J.; Xie, Z.; Zhang, B.; Xiong, C.; Chen, X. M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models. arXiv 2024, arXiv:2405.15638. [Google Scholar]
- Sonoda, Y.; Kurokawa, R.; Nakamura, Y.; Kanzawa, J.; Kurokawa, M.; Ohizumi, Y.; Gonoi, W.; Abe, O. Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” Cases. medRxiv 2024, 2024.05.26.24307915. [Google Scholar] [CrossRef]
- Singgalen, Y.A. Analyzing an Interest in GPT 4o through Sentiment Analysis using CRISP-DM. J. Inf. Syst. Inform. 2024, 6, 882–898. [Google Scholar] [CrossRef]
- Pang, S.; Nol, E.; Heng, K. ChatGPT-4o for English language teaching and learning: Features, applications, and future prospects. SSRN Sch. Pap. 2024, 4837988. [Google Scholar] [CrossRef]
- Xu, S.; Wang, Y.; Liu, D.; Xu, C. Collage Prompting: Budget-Friendly Visual Recognition with GPT-4V. arXiv 2024, arXiv:2403.11468. [Google Scholar]
- Zhou, Y.; Ong, H.; Kennedy, P.; Wu, C.C.; Kazam, J.; Hentel, K.; Flanders, A.; Shih, G.; Peng, Y.; Moy, L.; et al. Evaluating GPT-4V (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs. Radiology 2024, 311, e233270. [Google Scholar] [CrossRef] [PubMed]
- Allyn, B. Scarlett Johansson Says She Is “Shocked, Angered” over New ChatGPT Voice. Available online: https://www.npr.org/2024/05/20/1252495087/openai-pulls-ai-voice-that-was-compared-to-scarlett-johansson-in-the-movie-her (accessed on 15 June 2024).
- Li, H.; Ding, W.; Kang, Y.; Liu, T.; Wu, Z.; Liu, Z. CTAL: Pre-training cross-modal transformer for audio-and-language representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7–11 November 2021; pp. 3966–3977. [Google Scholar]
- Federation of State Medical Boards and National Board of Medical Examiners. USMLE Step 1 Content Description and General Information. 2024. Available online: https://www.usmle.org (accessed on 15 June 2024).
- Kung, T.H.; Cheatham, M.; Medenilla, A.; Sillos, C.; Leon, L.D.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo, J.; et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit. Health 2023, 2, e0000198. [Google Scholar] [CrossRef] [PubMed]
- Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med. Educ. 2023, 9, e45312. [Google Scholar] [CrossRef] [PubMed]
- Brin, D.; Sorin, V.; Vaid, A.; Soroush, A.; Glicksberg, B.S.; Charney, A.W.; Nadkarni, G.; Klang, E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci. Rep. 2023, 13, 16492. [Google Scholar] [CrossRef]
- Haleem, A.; Javaid, M.; Qadri, M.A.; Suman, R. Understanding the role of digital technologies in education: A review. Sustain. Oper. Comput. 2022, 3, 275–285. [Google Scholar] [CrossRef]
- Callanan, E.; Mbakwe, A.; Papadimitriou, A.; Pei, Y.; Sibue, M.; Zhu, X.; Ma, Z.; Liu, X.; Shah, S. Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams. arXiv 2023, arXiv:2310.08678. [Google Scholar]
- College Board. The SAT Suite of Assessments. Available online: https://www.collegeboard.org (accessed on 15 June 2024).
- OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F.L.; Almeida, D.; Altenschmidt, J.; Altman, S.; et al. GPT-4 Technical Report. arXiv 2024, arXiv:2303.08774. [Google Scholar]
- National Conference of Bar Examiners. MBE Sample Test Questions. Available online: https://www.ncbex.org (accessed on 15 June 2024).
- Griggs, M. Building a Better Bar Exam. Tex. A&M Law Rev. 2019, 7, 1. [Google Scholar]
- Katz, D.M.; Bommarito, M.J.; Gao, S.; Arredondo, P. Gpt-4 passes the bar exam. Philos. Trans. R. Soc. A 2024, 382, 20230254. [Google Scholar] [CrossRef]
- Huang, J.; Chang, K.C.-C. Towards Reasoning in Large Language Models: A Survey. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, Toronto, ON, Canada, 9–14 July 2023; Rogers, A., Boyd-Graber, J., Okazaki, N., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2023; pp. 1049–1065. [Google Scholar]
- Johnson-Laird, P. Deductive reasoning. WIREs Cogn. Sci. 2010, 1, 8–17. [Google Scholar] [CrossRef]
- Hayes, B.K.; Heit, E.; Swendsen, H. Inductive reasoning. WIREs Cogn. Sci. 2010, 1, 278–292. [Google Scholar] [CrossRef]
- Walton, D. Abductive Reasoning; University of Alabama Press: Tuscaloosa, AL, USA, 2014. [Google Scholar]
- Dalvi, B.; Jansen, P.; Tafjord, O.; Xie, Z.; Smith, H.; Pipatanangkura, L.; Clark, P. Explaining Answers with Entailment Trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, 7–11 November 2021; Moens, M.-F., Huang, X., Specia, L., Yih, S.W., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 7358–7370. [Google Scholar] [CrossRef]
- Weston, J.; Bordes, A.; Chopra, S.; Rush, A.M.; van Merriënboer, B.; Joulin, A.; Mikolov, T. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv 2015, arXiv:1502.05698. [Google Scholar]
- Sinha, K.; Sodhani, S.; Dong, J.; Pineau, J.; Hamilton, W.L. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; Inui, K., Jiang, J., Ng, V., Wan, X., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4506–4515. [Google Scholar] [CrossRef]
- Bhagavatula, C.; Bras, R.L.; Malaviya, C.; Sakaguchi, K.; Holtzman, A.; Rashkin, H.; Downey, D.; Yih, W.; Choi, Y. Abductive Commonsense Reasoning. International Conference on Learning Representations. 2019. Available online: https://openreview.net/forum?id=Byg1v1HKDB (accessed on 15 June 2024).
- López Espejel, J.; Ettifouri, E.H.; Yahaya Alassan, M.S.; Chouham, E.M.; Dahhane, W. GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Nat. Lang. Process. J. 2023, 5, 100032. [Google Scholar] [CrossRef]
- Khoshafah, F. ChatGPT for Arabic-English Translation: Evaluating the Accuracy. Res. Sq. 2023. [Google Scholar] [CrossRef]
- Tiedemann, J. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, 23–25 May 2012; pp. 2214–2218. [Google Scholar]
- Kunchukuttan, A.; Mehta, P.; Bhattacharyya, P. The IIT bombay english-hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 7–12 May 2018. [Google Scholar]
- Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019; Volume 1 (long and short papers), pp. 4171–4186. [Google Scholar]
- Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; Inui, K., Jiang, J., Ng, V., Wan, X., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 3982–3992. [Google Scholar] [CrossRef]
- Cubero, S.; Aleixos, N.; Moltó, E.; Gómez-Sanchis, J.; Blasco, J. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technol. 2011, 4, 487–504. [Google Scholar] [CrossRef]
- Jebraeily, Y.; Sharafi, Y.; Teshnehlab, M. Driver drowsiness detection based on convolutional neural network architecture optimization using genetic algorithm. IEEE Access 2024, 12, 45709–45726. [Google Scholar] [CrossRef]
- Hung, L.P.; Alias, S. Beyond Sentiment Analysis: A Review of Recent Trends in Text Based Sentiment Analysis and Emotion Detection. J. Adv. Comput. Intell. Intell. Inform. 2023, 27, 84–95. [Google Scholar] [CrossRef]
- Shahriar, S. GAN computers generate arts? A survey on visual arts, music, and literary text generation using generative adversarial network. Displays 2022, 73, 102237. [Google Scholar] [CrossRef]
- Graham, C.; Roll, N. Evaluating OpenAI’s Whisper ASR: Performance analysis across diverse accents and speaker traits. JASA Express Lett. 2024, 4, 025206. [Google Scholar] [CrossRef] [PubMed]
- Ahamad, A.; Anand, A.; Bhargava, P. AccentDB: A database of non-native english accents to assist neural speech recognition. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, 11–16 May 2020; European Language Resources Association: Marseille, France, 2020; pp. 5351–5358. Available online: https://www.aclweb.org/anthology/2020.lrec-1.659 (accessed on 15 June 2024).
- Yu, W.; Yang, Z.; Li, L.; Wang, J.; Lin, K.; Liu, Z.; Wang, X.; Wang, L. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv 2023, arXiv:2308.02490. [Google Scholar]
- Zhu, N.; Zhang, N.; Shao, Q.; Cheng, K.; Wu, H. OpenAI’s GPT-4o in surgical oncology: Revolutionary advances in generative artificial intelligence. Eur. J. Cancer 2024, 206, 114132. [Google Scholar] [CrossRef]
- Zhu, H.; Wang, Z.; Shi, Y.; Hua, Y.; Xu, G.; Deng, L. Multimodal Fusion Method Based on Self-Attention Mechanism. Wirel. Commun. Mob. Comput. 2020, 2020, 8843186. [Google Scholar] [CrossRef]
- Zhou, K.; Hassan, F.H.; Hoon, G.K. The State of the Art for Cross-Modal Retrieval: A Survey. IEEE Access 2023, 11, 138568–138589. [Google Scholar] [CrossRef]
- Hayawi, K.; Shahriar, S.; Mathew, S.S. The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD. J. Inf. Sci. 2024, 01655515241227531. [Google Scholar] [CrossRef]
- Shahriar, S.; Al Roken, N.; Zualkernan, I. Classification of Arabic poetry emotions using deep learning. Computers 2023, 12, 89. [Google Scholar] [CrossRef]
- Sivarajkumar, S.; Kelley, M.; Samolyk-Mazzanti, A.; Visweswaran, S.; Wang, Y. An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study. JMIR Med. Inform. 2024, 12, e55318. [Google Scholar] [CrossRef]
- Shahriar, S.; Al Roken, N. How can generative adversarial networks impact computer generated art? Insights from poetry to melody conversion. Int. J. Inf. Manag. Data Insights 2022, 2, 100066. [Google Scholar] [CrossRef]
Model | Total Questions | Correct Answers | Accuracy |
---|---|---|---|
GPT-3.5 | 389 | 201 | 51.67% |
GPT-4 | 80 | 72 | 90.00% |
GPT-4o | 118 | 98 | 83.05% |
Model | Accuracy |
---|---|
GPT-3.5 | 58.80% |
GPT-4 | 73.20% |
GPT-4o | 85.39% |
Test | Total Questions | Correct Answers | Accuracy |
---|---|---|---|
Reading & Writing M1 | 33 | 31 | 93.94% |
Reading & Writing M2 | 33 | 29 | 87.88% |
Math M1 | 27 | 25 | 92.59% |
Math M2 | 27 | 22 | 81.48% |
Model | Reading & Writing | Math |
---|---|---|
GPT-3.5 | 83.75% | 73.75% |
GPT-4 | 88.75% | 87.50% |
GPT-4 (no vision) | 88.75% | 86.25% |
GPT-4o | 90.91% | 87.04% |
Model | Accuracy |
---|---|
GPT-3.5 | 45.10% |
GPT-4 | 75.70% |
GPT-4o | 75.00% |
Category/ Model | Deductive Reasoning | Inductive Reasoning | Abductive Reasoning | ||
---|---|---|---|---|---|
Entailment Bank | bAbI (Task 15) | CLUTRR | bAbI (Task 15) | αNLI | |
GPT-3.5 | 25/30 | 26/30 | 2/30 | 14/30 | 19/30 |
GPT-4 | 27/30 | 30/30 | 11/30 | 28/30 | 25/30 |
GPT-4o | 29/30 | 30/30 | 17/30 | 30/30 | 27/30 |
Language | Translation Accuracy (%) |
---|---|
Spanish | 88 |
Arabic | 78 |
Hindi | 82 |
French | 75 |
Portuguese | 86 |
Russian | 80 |
Class | Precision | Recall | F1-Score |
---|---|---|---|
Banana | 0.91 | 1.00 | 0.95 |
Papaya | 1.00 | 1.00 | 1.00 |
Apple | 1.00 | 1.00 | 1.00 |
Litchi | 1.00 | 1.00 | 1.00 |
Jackfruit | 1.00 | 0.90 | 0.95 |
Hog Plum | 1.00 | 1.00 | 1.00 |
Grapes | 1.00 | 1.00 | 1.00 |
Guava | 1.00 | 1.00 | 1.00 |
Mango | 0.91 | 1.00 | 0.95 |
Orange | 1.00 | 0.90 | 0.95 |
Class | Precision | Recall | F1-Score |
---|---|---|---|
Drowsy | 0.8 | 0.8 | 0.8 |
Natural | 0.8 | 0.8 | 0.8 |
Class | Precision | Recall | F1-Score |
---|---|---|---|
Anthracnose | 0.60 | 0.60 | 0.60 |
Apple Scab | 1.00 | 0.80 | 0.89 |
Black Spot | 0.67 | 1.00 | 0.80 |
Blight | 0.38 | 0.75 | 0.50 |
Blossom End Rot | 1.00 | 1.08 | 1.00 |
Botrytis | 1.00 | 0.20 | 0.33 |
Brown Rot | 1.00 | 0.20 | 0.33 |
Canker | 0.25 | 0.25 | 0.25 |
Cedar Apple Rust | 0.83 | 1.00 | 0.91 |
Clubroot | 1.00 | 1.00 | 1.00 |
Crown Gall | 1.00 | 1.00 | 1.00 |
Downy Mildew | 1.00 | 0.20 | 0.33 |
Fire Blight | 0.80 | 0.80 | 0.80 |
Fusarium | 1.00 | 0.60 | 0.75 |
Gray Mold | 0.43 | 0.75 | 0.55 |
Leaf Spots | 0.40 | 0.80 | 0.53 |
Mosaic Virus | 0.67 | 0.80 | 0.72 |
Nematodes | 1.00 | 1.00 | 1.00 |
Powdery Mildew | 0.71 | 1.00 | 0.83 |
Verticillium | 0.67 | 0.40 | 0.50 |
Class | Precision | Recall | F1-Score |
---|---|---|---|
Glaucoma | 0.58 | 0.86 | 0.69 |
Normal | 0.73 | 0.38 | 0.50 |
Class | Precision | Recall | F1-Score |
---|---|---|---|
Aneurysm | 0.3725 | 0.5758 | 0.4524 |
Cancer | 0.0000 | 0.0000 | 0.0000 |
Tumor | 0.2708 | 0.3939 | 0.3210 |
Metric | Score |
---|---|
BLEU-1 | 0.193 |
BLEU-2 | 0.095 |
BLEU-3 | 0.058 |
BLEU-4 | 0.031 |
Model | Recognition | OCR | Knowledge | Language Generation | Spatial Awareness | Math | Total |
---|---|---|---|---|---|---|---|
GPT-3.5 | 22.3 | 31.4 | 15.6 | 16.6 | 32.9 | 24.0 | 27.6 |
GPT-4 | 34.3 | 66.3 | 25.6 | 36.6 | 60.6 | 72.0 | 48.1 |
GPT-4V | 67.5 | 68.3 | 56.2 | 60.7 | 69.4 | 58.6 | 67.7 |
GPT-4o | 80.6 | 87.5 | 75.7 | 77.1 | 84.7 | 88.1 | 83.9 |
Input Image | Questions|Assessed Capabilities | GPT-4o Responses |
---|---|---|
(a) Q: How many gallons of supreme gasoline can I get with $50? | GPT-4o: (a) With $50, you can get approximately 13.70 gallons of supreme gasoline. | |
(b) Q: What is the price difference between diesel and regular gasoline? Required capabilities: OCR, Math | GPT-4o: The price difference between diesel and regular gasoline is $0.11 per gallon. | |
(a) Q: Based on the image, what is the total international awareness of Tina Fey? | GPT-4o: Based on the image, the total international awareness of Tina Fey is 35%. This percentage is located at the bottom row under the column labeled “Total” for Tina Fey. | |
(b) Q: Based on the image, in which country does Tina Fey have the highest level of total awareness? Required capabilities: OCR, Spatial awareness | GPT-4o: Based on the image, Tina Fey has the highest level of total awareness in Australia, with an awareness level of 62%. | |
(a) Q: What is the price for tomatoes? | GPT-4o: The price for tomatoes is 8.00, as indicated by the sign placed above the pile of tomatoes in the image. | |
(b) Q: What fruit is the right of plums? Required capabilities: Recognition, Spatial awareness | GPT-4o: The fruit to the right of the plums in the image is pears. | |
(a) Q: Which car is on the parking spot 33? | GPT-4o: The parking spot labeled 33 is empty. | |
(b) Q: Which spot is the red car parking on? Required capabilities: OCR, Recognition, Spatial awareness | GPT-4o: The red car is parked in spot number 30. | |
(a) Q: Which category did the worst job in plastic recycling, considering the amount of plastic made and thrown out? | GPT-4o: The category that did the worst job in plastic recycling, considering the amount of plastic made and thrown out, is “Packaging”. In 2015, 146 million metric tonnes of plastic were made for packaging, and 141 million metric tonnes were thrown out. This shows that almost all the plastic made for packaging was discarded as waste. | |
(b) Q: What is the difference in metric tonnes between the amount of plastic produced and the amount discarded? Required capabilities: OCR, Spatial awareness | The difference between the amount of plastic produced (407 million metric tonnes) and the amount discarded (302 million metric tonnes) is: 407 million metric tonnes − 302 million metric tonnes = 105 million metric tonnes So, the difference is 105 million metric tonnes. | |
Q: What does this factory make? Required capabilities: OCR | GPT-4o: The factory in the image belongs to Leica, which is a company known for manufacturing cameras, lenses, and other optical equipment. | |
Q: In which country was this photo taken? Required capabilities: Recognition, Knowledge | GPT-4o: The photo shows Novak Djokovic kissing a trophy that appears to be the Australian Open trophy. Therefore, this photo was likely taken in Australia. | |
Q: Which are producers in this food web? Required capabilities: OCR, Knowledge, Spatial Awareness | GPT-4o: In this food web, the producers are: Phytoplankton Seaweed These are the organisms that produce energy through photosynthesis and serve as the base of the food web. | |
Q: Is it summertime in the scene? Answer it and give the rationale. Required capabilities: Recognition, Knowledge, Language generation | GPT-4o: No, it is not summertime in the scene. The presence of large pumpkins suggests that it is likely fall, as pumpkins are typically associated with the autumn harvest season. Additionally, the characters are wearing long-sleeved clothing, which is more common in cooler weather. | |
Q: Can you explain this meme? Required capabilities: Recognition, OCR, Knowledge, Language generation | GPT-4o: This meme humorously illustrates the common tendency to procrastinate. The text shows an intention to start a task at a specific time (8:00), but when the time is slightly missed (8:05), the person decides to delay the task further, waiting until the next hour (9:00) to start. The image of Kermit the Frog relaxing on a bed adds to the humor by depicting the relaxed attitude towards procrastination. The meme pokes fun at the irrational yet relatable habit of postponing tasks due to missing an arbitrary starting time by a few minutes. | |
Q: What is funny about this image? Required capabilities: Recognition, Knowledge, Language generation | GPT-4o: This image is a popular meme that combines two unrelated photos to create a humorous juxtaposition. On the left side, there is an image of a woman (Taylor Armstrong from “The Real Housewives of Beverly Hills”) angrily pointing and shouting, while another woman (Kyle Richards) tries to calm her down. On the right side, there is a photo of a white cat (known as “Smudge the Cat”) sitting at a dinner table with a confused and displeased expression in front of a plate of vegetables. The humor comes from the contrast between the woman’s intense emotion and the cat’s indifferent and bemused reaction, as if the cat is unbothered by the woman’s outburst. This combination creates a relatable and funny depiction of exaggerated reactions and misunderstandings. | |
Q: How to cook this dish? Required capabilities: Recognition, Knowledge, Language generation | GPT-4o: This image shows a pie with a lattice crust. Here is a basic recipe to make a lattice-top pie, such as a strawberry or mixed berry pie: Ingredients: For the Filling: 4 cups fresh strawberries or mixed berries, 3/4 cup granulated sugar, 1/4 cup cornstarch, 1 tablespoon lemon juice, 1 teaspoon vanilla extract, A pinch of salt For the Crust: 2 1/2 cups all-purpose flour, 1 teaspoon salt, 1 teaspoon sugar, 1 cup unsalted butter, chilled and cut into small pieces, 1/4 to 1/2 cup ice water Instructions: In a large bowl, combine the flour, salt, and sugar. Add the chilled butter pieces and cut them into the flour mixture using a pastry blender or your fingers until the mixture resembles coarse crumbs. Gradually add ice water, a tablespoon at a time, mixing until the dough begins to come together. Do not overwork the dough. Divide the dough into two equal parts, shape them into disks, wrap in plastic wrap, and refrigerate for at least 1 h. In a large bowl, combine the berries, sugar, cornstarch, lemon juice, vanilla extract, and salt. Stir gently to combine. On a lightly floured surface, roll out one disk of dough to fit a 9-inch pie pan. Place the rolled-out dough into the pie pan and trim the edges. Pour the berry mixture into the crust-lined pie pan. Roll out the second disk of dough and cut it into 1-inch-wide strips. Lay 4 to 5 strips over the filling, parallel to each other. Lay another 4 to 5 strips perpendicular to the first set, weaving them to create a lattice pattern. Trim and crimp the edges to seal. Preheat the oven to 375°F (190 °C). Bake: Place the pie on a baking sheet to catch any drips and bake for 45–55 min, or until the crust is golden brown and the filling is bubbly. Cool: Let the pie cool completely before serving to allow the filling to set. Enjoy your homemade lattice-top pie! | |
Q: Can you give a short introduction to this painting? Required capabilities: Recognition, Knowledge, Language generation | GPT-4o: This painting is a historical Chinese artwork titled “Emperor Taizong Receiving the Tibetan Envoy”, created by Yan Liben during the Tang Dynasty. The piece illustrates a significant diplomatic event where Emperor Taizong of the Tang Dynasty meets a Tibetan envoy. This event highlights the political and cultural exchanges between the Tang Dynasty and Tibet. The painting is notable for its detailed depiction of the figures, the hierarchical arrangement, and the ceremonial aspects of the meeting. Yan Liben, a renowned court artist of the Tang Dynasty, is celebrated for his ability to capture the grandeur and solemnity of imperial events. This artwork not only serves as a historical record but also exemplifies the artistic sophistication of the Tang Dynasty. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Shahriar, S.; Lund, B.D.; Mannuru, N.R.; Arshad, M.A.; Hayawi, K.; Bevara, R.V.K.; Mannuru, A.; Batool, L. Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Appl. Sci. 2024, 14, 7782. https://doi.org/10.3390/app14177782
Shahriar S, Lund BD, Mannuru NR, Arshad MA, Hayawi K, Bevara RVK, Mannuru A, Batool L. Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Applied Sciences. 2024; 14(17):7782. https://doi.org/10.3390/app14177782
Chicago/Turabian StyleShahriar, Sakib, Brady D. Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. 2024. "Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency" Applied Sciences 14, no. 17: 7782. https://doi.org/10.3390/app14177782