An Investigation into the Utility of Large Language Models in Geotechnical Education and Problem Solving

Abstract

The study explores the capabilities of large language models (LLMs), particularly GPT-4, in understanding and solving geotechnical problems, a specialised area that has not been extensively examined in previous research. Employing a question bank obtained from a commonly used textbook in geotechnical engineering, the research assesses GPT-4’s performance across various topics and cognitive complexity levels, utilising different prompting strategies like zero-shot learning, chain-of-thought (CoT) prompting, and custom instructional prompting. The study reveals that while GPT-4 demonstrates significant potential in addressing fundamental geotechnical concepts and problems, its effectiveness varies with specific topics, the complexity of the task, and the prompting strategies employed. The paper categorises errors encountered by GPT-4 into conceptual, grounding, calculation, and model inherent deficiencies related to the interpretation of visual information. Custom instructional prompts, specifically tailored to address GPT-4’s shortcomings, significantly enhance its performance. The study reveals that GPT-4 achieved an overall problem-solving accuracy of 67% with custom instructional prompting, significantly higher than the 28.9% with zero-shot learning and 34% with CoT. However, the study underscores the importance of human oversight in interpreting and verifying GPT-4’s outputs, especially in complex, higher-order cognitive tasks. The findings contribute to understanding the potential and limitations of current LLMs in specialised educational fields, providing insights for educators and researchers in integrating AI tools like GPT-4 into their teaching and problem-solving approaches. The study advocates for a balanced integration of AI in education to enrich educational delivery and experience while emphasising the indispensable role of human expertise alongside technological advancements.

1. Introduction

The educational landscape is experiencing a profound transformation, propelled by the rapid advancement and integration of artificial intelligence (AI) technologies [1,2,3,4]. This shift is prominently marked by the emergence of Generative AI as a pivotal force in reshaping educational paradigms, offering innovative approaches to learning and problem-solving [5,6,7]. Central to this AI-driven revolution in education are generative AI models such as large language models (LLMs) like ChatGPT [8,9,10,11,12,13], which have demonstrated considerable potential in enhancing learning outcomes [14,15,16] and aiding in complex problem-solving scenarios. These models, powered by sophisticated algorithms and extensive data, can alter information delivery and fundamentally change how students interact with and comprehend complex topics [17].

The growing incorporation of AI into education, especially in STEM (science, technology, engineering, and math) fields, is underpinned by a burgeoning body of research [18]. Bai and Stede [19] conducted a survey on modern machine learning (ML) methods for automated assessment of students’ natural language free-text responses. This study accentuates the growing adoption of AI in educational contexts and emphasises its transformative role in revolutionising traditional learning approaches. A notable area where AI has made significant inroads is in exam preparation and assessment. ChatGPT has been a focal point in various studies of AI applications in education, showcasing its effectiveness across multiple educational domains [20,21,22,23,24]. For instance, Schulze Balhorn et al. [25] systematically evaluated GPT-3.5’s ability to answer questions in the natural sciences and engineering using 594 questions from 198 Delft University faculty members. Using a structured assessment, participants generally rated the answers as “mostly correct.” However, ratings declined with higher educational levels of questions and when assessing skills beyond scientific knowledge, like critical thinking.

Zhang et al. [26] evaluated the programming capabilities of GPT-3.5 and GPT-4 using Swift-based exam questions from a third-year university course. Their study finds that both models generally outperform average student scores but do not consistently surpass top students. This comparison highlights areas where the GPT models excel and fall short, providing a nuanced view of their current programming proficiency. The study also reveals surprising instances where GPT-3.5 outperforms GPT-4, suggesting complex variations in AI model capabilities.

Katz et al. [27] assessed GPT-4’s zero-shot performance on the entire Uniform Bar Examination (UBE), including the Multistate Bar Examination (MBE), the Multistate Essay Exam (MEE), and the Multistate Performance Test (MPT). GPT-4 significantly outperformed both humans and previous models on the MBE, achieving a 26% improvement over GPT-3.5 and higher scores in five out of seven areas. On the MEE and MPT, GPT-4 scored an average of 4.2/6.0, notably surpassing GPT-3.5. GPT-4 scored 297 points across the UBE, exceeding the passing threshold for all jurisdictions and demonstrating its potential to enhance legal service delivery.

Maitland et al. [28] evaluated GPT-4’s performance on the MRCP (Membership of the Royal College of Physicians) Parts 1 and 2 practice questions. The study found 86.3% and 70.3% accuracy rates for Parts 1 and 2, respectively. The analysis identified eight error types, with factual, context, and omission errors being the most common. Overall, GPT-4 significantly outperformed the passing thresholds, offering insights into the effectiveness of GPT-4 in answering multiple-choice medical exams. Currie et al. [29] analysed GPT-3.5’s effects on academic integrity and its use in medical imaging courses, testing it in exams and written assignments across six subjects in the medical radiation science undergraduate program. Evaluations using standardised rubrics and Turnitin showed that GPT-3.5, generally underperformed compared to students, particularly in advanced subjects, but excelled in basic exams. The study concluded that while GPT-3.5 can enhance learning for more straightforward tasks, it risks compromising academic integrity and is limited in handling complex, discipline-specific content. In addition, the study recommends cautious integration of AI tools to maintain academic standards. Ali et al. [30] evaluated GPT-3.5 and GPT-4 on a 500-question mock neurosurgical medical exam. The study finds both models surpass the passing threshold, with GPT-4 significantly outperforming GPT-3.5, scoring 83.4% compared to GPT-3.5’s 73.4%. Both models showed lower performance on image-based questions, relying only on text clues. Overall, the study demonstrates the potential of LLMs in achieving passing scores on specialised medical exams, especially GPT-4’s enhanced capabilities. Kung et al. [31] assessed GPT-3.5’s capabilities in the exams of the United States Medical Licensing Examination (USMLE). This study also demonstrated the ability of GPT-3.5 to reach the passing threshold without needing specialised training, indicating its potential in medical education. Further corroborating these findings, Gilson et al. [32] and Antaki et al. [33] reported positive outcomes in applying ChatGPT within medical education, suggesting its significant role in this specialised field.

In business education, Terwiesch [34] investigated GPT-3.5’s effectiveness in the final exam of a typical MBA (i.e., Master of Business Administration) core course, operations management. The study revealed that GPT-3.5 showed competence in essential operations management and process analysis questions, including those based on case studies. However, it made significant errors in simple calculations at a 6th-grade math level and struggled with advanced process analysis questions involving multiple products and stochastic effects. Despite these limitations, GPT-3.5 exhibited a notable ability to adapt its responses based on human hints, underscoring its potential as a valuable educational resource in business school settings. Eulerich et al. [35] expanded the scope of ChatGPT’s evaluation to accounting certification exams. This study evaluated the capability of ChatGPT models to pass major accounting certification exams. Their findings suggest that while the initial GPT-3.5 scored an average of 53.1% and failed the exams, significant improvements were observed with the GPT-4. Enhancements, including additional training and using reasoning tools like calculators, raised the average score to 85.1%, allowing ChatGPT to pass all examined certifications.

However, despite these promising results, there is a noticeable gap in the literature regarding ChatGPT’s performance in geotechnical engineering, an area explored by only a few researchers [36,37]. Fatahia et al. [38] discussed generative AI’s role in geotechnical engineering, noting its potential and the need to explore tools like ChatGPT further in geotechnical engineering. This gap in the literature indicates the necessity for comprehensive research into ChatGPT’s capabilities in geotechnical education and problem solving. Geotechnical engineering is inherently dependent on complex mathematical concepts and sophisticated problem-solving abilities, posing unique challenges yet to be thoroughly tested against the capabilities of ChatGPT and similar AI tools. A comprehensive investigation into ChatGPT’s potential in geotechnical engineering is essential. Such an exploration would reveal both the current capabilities and limitations of ChatGPT in addressing complex geotechnical problems, setting the stage for future applications, such as:

• Interactive tutoring where LLMs offer personalised instruction to enhance student understanding of complex topics, such as soil mechanics and foundation design.
• Virtual lab developing where LLMs are combined with virtual reality. Students can perform virtual soil tests and experiments, with real-time guidance and results interpretation from the LLM.

This paper aims to fill the existing research gap by evaluating the depth and scope of GPT-4’s capabilities in geotechnical engineering. Our study delves into the utility of GPT-4 for a diverse range of cohorts, including students, educators, researchers, AI developers, and practitioners, focusing on its proficiency in explaining concepts, answering questions, and crafting educational content. We rigorously scrutinise GPT-4’s ability to handle fundamental undergraduate geotechnical topics and assess its performance in tackling problems of escalating complexity.

A critical part of our research is identifying major error types that hinder GPT-4’s effectiveness in geotechnical contexts. We have categorised these errors into four groups: “Conceptual” errors arising from the inability to retrieve necessary concepts or facts; “Grounding” errors where retrieved concepts are incorrectly applied in equations or constraints; “Calculation” errors involving errors in algebraic and arithmetic manipulation; and “Deficiency”, which include challenges in interpreting images, graphs, charts, and in drawing engineering images. Additionally, our study explores various prompting techniques such as zero-shot, chain-of-thought (CoT) processes, and tailored custom instructional prompts to optimise GPT-4’s output.

We aim to offer a comprehensive overview of LLMs’ role in geotechnical education and problem-solving by investigating these aspects. This research is structured around key questions that assess GPT-4’s capabilities and limitations, providing a holistic view of its utility across different user groups in geotechnical engineering. We utilised the April 2023 version of GPT-4 for this purpose. Ultimately, we seek to contribute to the responsible and effective integration of AI models like GPT-4 into educational settings. Our findings are intended to provide valuable insights that could inform future teaching methodologies and guide generative AI development in geotechnical engineering. This approach involves leveraging these advanced AI tools while continuously refining and developing them to realise their full potential as invaluable educational resources.

2. Dataset

2.1. Data Selection

To ensure statistical validity in assessing the utility of GPT-4 in fundamental or undergraduate early-level geotechnical education and problem solving, we first calculated the requisite sample size according to the formula provided by Daniel and Cross [39]:

$$n=\frac{Z^{2}P\left(1-P\right)}{d^2}$$

(1)

where $n$ denotes the sample size; $Z$ is the Z-score correlating to the chosen confidence level; $P$ represents the anticipated prevalence or proportion; and $d$ signifies the acceptable margin of error. For the present study, a 95% confidence level was chosen, corresponding to a Z-score of approximately 1.96, and a precision (margin of error) of 5%. Following the work of Lwanga and Lemeshow [40], setting $P$ at 0.5 is recommended to obtain a sample size that is sufficiently large to account for variability. This approach dictates a sample question size of 385, which is conducive to a robust statistical analysis. It should be noted that by adopting the above formula, the following assumptions have been made:

• A large population from which the sample is drawn, and the sample itself represents a minor fraction (less than 5%) of the total population;
• The expected outcome has been simplified to a binary variable—the success or failure of the LLM in problem-solving tasks—which aligns with the typical dichotomous nature of the assessment outcomes;
• The proportion’s sampling distribution is presumed to be normal or approximately normal. This assumption is reasonable for the calculated sample size, invoking the central limit theorem to justify the normal approximation.

In alignment with the calculated sample size requirements, we compiled a diverse question bank comprising 391 questions from Das’s authoritative textbook “Principles of Geotechnical Engineering, 8th Edition” in geotechnical engineering. The answers to these questions are obtained from the solution manual.

2.2. Data Categorisation

To effectively assess the capabilities of the LLM in geotechnical engineering education, we categorised our question bank along with topic relevance and cognitive complexity. These classifications allow for a comprehensive analysis of the model’s performance across the breadth of geotechnical topics and the depth of problem-solving skills required.

2.2.1. Categorisation by Topic

Our question bank is structured around core topics essential to undergraduate geotechnical engineering, aligning with Chapters 2–17 from the above textbook. These chapters encompass a comprehensive range of topics, including soil origin, grain size analysis, weight–volume relationships, soil plasticity, classification, compaction, seepage, permeability, in situ stresses, compressibility, shear strength, and lateral earth pressures, as listed

Table 1. Overview of Fundamental Geotechnical Engineering Topics Covered in the Question Bank.

Chapter Code	Topic Description
C2	Origin of Soil and Grain Size
C3	Weight–Volume Relationships
C4	Plasticity and Structure of Soil
C5	Classification of Soil
C6	Soil Compaction
C7	Permeability
C8	Seepage
C9	In Situ Stresses
C10	Stresses in a Soil Mass
C11	Compressibility of Soil
C12	Shear Strength of Soil
C13	Lateral Earth Pressure: At-Rest Rankine and Coulomb
C14	Lateral Earth Pressure: Curved Failure Surface
C15	Slope Stability
C16	Soil Bearing Capacity for Shallow Foundations
C17	Subsoil Exploration

. Figure 1 delineates a comparative analysis of the prevalence of image-based and text-based questions across the sixteen chapters, labelled C2 through C17. This categorisation enables a targeted analysis of the LLM’s proficiency in diverse geotechnical topics.

2.2.2. Categorisation by Cognitive Complexity

Bloom’s taxonomy, a framework consisting of categorised educational goals, helps educators define and distinguish between different levels of human cognition in learning environments [41]. Employing the updated Bloom’s taxonomy proposed by Krathwohl [41], we classify questions into six levels of cognitive complexity: “Remember”, “Understand”, “Apply”, “Analyse”, “Evaluate”, and “Create”. Figure 2 illustrates this hierarchy and provides specific examples for each level. For instance, Level 1: “Remember” is exemplified by tasks such as ‘Recalling and describing common laboratory tests in soil mechanics’. This systematisation aids in gauging the LLM’s proficiency in addressing geotechnical challenges of varying difficulty. Figure 3 represents the typical distribution of cognitive complexity levels in an undergraduate setting. It shows that most tasks (52%) are at the “Apply” level, aligning with undergraduate studies’ practical, application-focused nature. Lower proportions in the “Create” (1%) and “Evaluate” (4%) categories reflect the gradual development of higher-order thinking skills, often more emphasised in higher-level undergraduate and postgraduate education.

3. Experimental Setup

This investigation employs GPT-4 [8], a state-of-the-art LLM, to interrogate our question bank. We have implemented three strategic prompting strategies to evaluate the model’s problem-solving prowess: zero-shot learning, CoT prompting, and custom instructional prompting:

• Zero-shot learning: This approach assesses GPT-4’s baseline problem-solving skills without prior exposure to specific examples. It evaluates the model’s ability to leverage pre-trained knowledge and innate reasoning capabilities.
• CoT: In this approach, we prompted GPT-4 to detail its reasoning steps, akin to a human’s approach to problem-solving. This approach aids in understanding the model’s thought process and checks for logical coherence.
• Custom instructional prompting: This technique proactively mitigates common errors and refines GPT-4’s responses. Initially, we conducted a thorough analysis to identify the most frequent and impactful errors that GPT-4 encounters when tackling geotechnical problems. Armed with this insight, we crafted precise prompts containing targeted instructions addressing these shortcomings. These custom prompts were then used to guide GPT-4’s problem-solving process, optimising its accuracy and relevance in response to geotechnical questions.

Each prompting strategy is designed to progressively improve GPT-4’s problem-solving accuracy in geotechnical engineering. The Zero Shot strategy presents questions in their original form, testing the model’s baseline capabilities without prior specific training. This strategy evaluates the model’s ability to utilise pre-trained knowledge in new problems.

In the CoT approach, questions were again presented in their original form but with an additional directive to “solve the question and do it step by step”. This approach prompted GPT-4 to explicitly outline its reasoning process, offering insight into its logical progression and facilitating the identification of any reasoning gaps or errors.

For custom instructional prompting, we first analysed GPT-4’s performance in the Zero Shot approach to identify areas of improvement. Understanding these errors allowed us to create custom instructions, such as specific formulas or conceptual guidance, to address these identified issues directly. The examples of these custom instructions are detailed in the Appendix A. The custom instruction strategy and CoT were applied selectively, only for questions where the Zero Shot approach was insufficient. This approach represents a more targeted and refined strategy to overcome specific errors and enhance the model’s response quality in complex geotechnical scenarios.

4. Results

4.1. Accuracy and Effectiveness of Prompting Strategies

Figure 4 compares accuracy rates among three prompting strategies: Zero Shot, CoT, and our proposed strategy, Custom Instruction. The accuracy rate of each strategy is marked on the figure, with Zero Shot achieving a 28.9% accuracy and CoT 34%, while Custom Instruction notably leads with an impressive 67% accuracy. This marked difference is represented by the linear progression of points, underscoring the superior effectiveness of Custom Instruction in answering our question bank.

The subsequent subsections will delve deeper into these strategies, offering a comprehensive comparison across various topics and difficulty levels as per Bloom’s taxonomy and their performance in text-based versus image-based questions.

4.1.1. Different Topics

Figure 5 illustrates the accuracy rates of three prompting strategies—Zero Shot (blue), CoT (green), and Custom Instruction (red)—across chapters C2 to C17. Each chapter is represented by a trio of bars showing the performance of each strategy. The figure shows the variability of the strategies’ success across chapters: while some chapters (e.g., C4) show high accuracy for all strategies, others (e.g., C8, C10) demonstrate low effectiveness across the board. This visualisation underlines the importance of strategy selection concerning topic-specific demand. In addition, it can be seen that Custom Instruction consistently outperforms other strategies, indicative of its superior effectiveness. Specifically, chapters C3, C4, C12, and C16 exhibit the pronounced advantage of Custom Instruction where it reaches a 100% correct rate, starkly contrasting with the other strategies, which show significantly lower rates in these chapters. In Chapter C2, Custom Instruction demonstrates a superior outcome with approximately 85% correctness, while Zero Shot and CoT hover around 35% and 37.5%, respectively. These results indicate that Custom Instruction substantially enhances correct response rates in specific chapters, potentially indicating that the content of these chapters may be more amenable to custom-tailored instructional strategies.

4.1.2. Difficulty Level According to Bloom’s Taxonomy

Figure 6 illustrates the overall accuracy of responses categorised by Bloom’s taxonomy levels using Zero Shot, CoT, and Custom Instruction. The accuracy for both the “Remember” and “Understand” levels is consistently high across all prompting strategies, standing at 100%. Notably, there is a marked increase in accuracy for the “Apply”, “Analyse”, and “Evaluate” levels when using Custom Instruction, with accuracy rates of 68.5%, 48.4%, and 50%, respectively, compared to the other strategies. While CoT performs marginally better than Zero Shot in the “Apply” and “Analyse” levels, Custom Instruction markedly outpaces both. At the higher cognitive levels of “Evaluate” and “Create”, CoT does not enhance GPT-4’s responses, yet Custom Instruction significantly elevates its responses. Remarkably, for the “Create” level, Custom Instruction achieves a 100% accuracy rate. In comparison, Zero Shot and CoT yield a 0% accuracy rate, underscoring the substantial impact of Custom Instruction at the highest level of cognitive demand in Bloom’s taxonomy. This graphical analysis highlights the superior performance of Custom Instruction in facilitating higher-order thinking skills in GPT-4.

4.1.3. Question Type

Figure 7 portrays the accuracy of GPT-4 in answering questions with and without images using the three different prompting strategies, i.e., Zero Shot, CoT, and Custom Instruction. For text-based questions, the accuracy rates are 46.4% with Zero Shot, 52.6% with CoT, and markedly higher at 92.8% with Custom Instruction. In contrast, for image-based questions, the accuracies are significantly lower across all strategies, with Zero Shot at 8.8%, CoT at 12.6%, and Custom Instruction at 37.4%. Overall, the data succinctly underscore the better performance of GPT-4 in handling text-based questions and emphasise the pronounced escalation in precision afforded by the Custom Instruction strategy.

4.2. Error Type Distribution

Subsequent to our analysis of accuracy rates, we investigated the error patterns associated with the utilisation of the three prompting strategies, as illustrated in Figure 8. This figure synthesises the occurrence rates of four principal error types—”Conceptual”, “Grounding”, “Calculation”, and “Deficiency”—that were identified in the responses generated by GPT-4. In the Zero Shot strategy, “Deficiency” errors were the most prevalent at 44.2%, followed by “Grounding” errors at 39.2%, “Conceptual” errors at 12.6%, and “Calculation” errors at 4%. The CoT strategy shows a similar distribution, with “Deficiency” errors being the most common at 45.8%, “Grounding” errors at 40.1%, and equal rates for “Conceptual” errors at 12.6% and “Calculation” errors at 1.5%. However, the Custom Instruction strategy reveals a pronounced concentration of “Deficiency” errors, constituting 85.9% of its error profile, while “Calculation” errors emerge as the second most common at 9.4%. This observation potentially implies that the Custom Instruction strategy is more efficient in addressing “Conceptual” and “Grounding” errors than “Deficiency” and “Calculation” errors.

4.2.1. Different Topics

Figure 9 presents a comparative analysis of error frequencies across different chapters using the three prompting strategies—Zero Shot, CoT, and Custom Instruction. Organised into a tripartite heatmap structure, each column delineates one of the strategies, while the rows index the chapters (C2 through C17). Within each cell of the heatmaps, the number of errors is categorised into four types: “Conceptual”, “Grounding”, “Calculation”, and “Deficiency”. The colour intensity within the cells correlates with the count of errors, following a gradient from yellow (no errors) to dark blue (maximum error frequency).

For the Zero Shot strategy, the heatmap reveals a prominent concentration of “Conceptual” errors in the early chapters, with the highest count being 20 errors in C2—Origin of Soil and Grain Size, indicating challenges in grasping foundational concepts in these topics. “Grounding” errors are predominant in C12—Shear Strength of Soil, with 24 occurrences, suggesting difficulties in employing correct equations or constraints within this topic. Notably, “Calculation” errors are minimal, with no more than three instances observed, suggesting a relative strength of the model in handling algebraic and arithmetic operations. “Deficiency” errors related to visual data interpretation are more frequent in later chapters, such as C9 to C11. This trend could be attributed to the prevalence of image-based content in these chapters, posing interpretative challenges that the Zero Shot strategy does not adequately address.

Transitioning to the CoT strategy, a subtle decrement in “Conceptual” errors is discernible, yet “Grounding” and “Deficiency” errors persist with notable severity, especially in C12 and C10, respectively. The strategy shows its strength in curtailing “Calculation” errors, suggesting an aptitude for computational reasoning. However, the resilience of “Deficiency” errors across chapters indicates that the CoT approach may require further refinement to aid visual data comprehension.

The heatmap for Custom Instruction demonstrates a notable decrease in the “Conceptual” and “Grounding” errors, as evidenced by lower error counts in these categories across several chapters. This result indicates the effectiveness of the Custom Instruction strategy in enhancing understanding and context accuracy. However, “Deficiency” errors remain prevalent (e.g., 18–20 errors in later chapters like C9 and C10), highlighting ongoing challenges in dealing with visual information even with customised instructions.

Overall, the Zero Shot strategy shows a higher tendency for conceptual misunderstandings than the CoT and Custom Instruction strategies. The CoT strategy improves conceptual accuracy but does not fully resolve “Grounding” and “Deficiency” errors. The Custom Instruction strategy significantly reduces “Conceptual” and “Grounding” errors but struggles with “Deficiency” errors.

4.2.2. Difficulty Level According to Bloom’s Taxonomy

Figure 10 represents the distribution of error types across different Bloom’s Taxonomy levels for the Zero Shot, CoT, and Custom Instruction strategies. This figure reveals a concentration of errors, especially “Grounding” and “Deficiency”, in more complex cognitive tasks like “Apply”. This pattern suggests that the Zero Shot strategy might struggle with tasks that demand higher-order thinking and application of concepts.

The CoT strategy, as depicted in the heatmap, similarly grapples with “Grounding” and “Deficiency” errors within the “Apply” level, although to a slightly lesser extent than Zero Shot, underscoring a common difficulty in tasks requiring the practical application of knowledge.

The Custom Instruction strategy demonstrates almost complete mitigation of “Conceptual” and “Grounding” errors across most levels of Bloom’s taxonomy, signifying its effectiveness in reinforcing fundamental understanding and contextual application. Nonetheless, it confronts a pronounced incidence of “Deficiency” errors, particularly within the “Apply” category. This observation points to a persistent challenge this strategy faces with tasks demanding the synthesis of visual information and its application.

5. Discussion

5.1. Insights and Implications for Geotechnical Education

In our exploration of GPT-4’s application in geotechnical education, several key insights and implications have emerged:

• Errors and Problem Complexity: We observed a direct correlation between the complexity of geotechnical problems and the increase in errors from GPT-4. This trend underscores the model’s challenges in processing and accurately responding to more complex tasks, indicating a potential area for future AI development and customised training.
• Text-Based versus Image-Based Questions: GPT-4 exhibits proficiency in handling text-based queries, reflected in a lower occurrence of “Conceptual” errors. However, the performance of the version of GPT 4 used in image-based questions is limited due to an inherent inability to process or interpret images for geotechnical problems. This observation calls for integrated multimodal AI systems that can handle a variety of data formats.
• Deficiencies and Limitations: Despite its strengths in processing textual information, GPT-4 exhibits notable deficiencies in tasks requiring interpreting images or generating complex visual representations. This limitation is particularly pronounced when the flow net diagrams are involved. The model struggles with creating or interpreting such visual content and occasionally fails to adhere strictly to the custom instructions provided. These shortcomings highlight two key areas in need of further development. First, there is a pressing need to enhance GPT-4’s spatial and visual processing capability, enabling it to handle tasks beyond text-based information. Second, the model’s occasional deviation from custom instructions underscores the ongoing requirement for refinement in its ability to understand and process user-specific directives accurately. Addressing these limitations is critical for advancing AI technologies like GPT-4 to improve their effectiveness as educational tools in geotechnical engineering and expand their applicability in solving complex, visually oriented problems.
• Custom Instructions for Enhanced Accuracy: The application of tailored instructions has been shown to significantly improve the accuracy of GPT-4. The improvement is particularly notable in reducing “Conceptual” and “Grounding” errors and mitigating “Calculation” errors in tasks requiring numerical and logical reasoning. This aspect demonstrates the importance of context-specific guidance in enhancing AI performance in educational settings.

5.2. Demonstration of GPT-4’s Question Generation Capabilities in Geotechnical Education

Utilising custom instructions, GPT-4 was tasked with generating educational questions and answers for specific chapters in geotechnical engineering, illustrating its application across various cognitive levels of Bloom’s taxonomy. For Chapters 2, 3, and 12, GPT-4 was directed to create questions and provide answers for each level of Bloom’s taxonomy: “Remember”, “Understand”, “Apply”, “Analyse”, “Evaluate”, and “Create”. The following examples in

Table 2. Questions and Answers Generated by GPT-4 across Bloom’s Taxonomy Levels.

Bloom’s Level	Question and Answer Summary
Remember	Question: What is the formula to calculate the degree of saturation (S) in soil mechanics?
	Answer: The degree of saturation (S) in soil mechanics is calculated using the formula $S=\frac{\left(G_s w\right)}{e}$, where $G_s$ is the specific gravity, $w$ is the water content in fraction, and $e$ is the void ratio.
Understand	Question: Explain why the maximum dry density or unit weight of soil is important in soil compaction and how it is related to the moisture content.
	Answer: The maximum dry density or unit weight of soil is crucial in soil compaction as it represents the highest density a soil can be compacted to at a given moisture content. It’s important for ensuring stability and support in engineering projects. The relationship with moisture content is significant as it determines the optimal water content at which the soil achieves its maximum dry density, leading to effective compaction and strength.
Apply	Question: Given principal stresses ${\sigma }_1$ = 150 kPa and ${\sigma }_3$ = 50 kPa in a soil sample, and a friction angle (ϕ) of 30 degrees, calculate the normal stress (σn) and shear stress (τ) on a plane at an angle θ of 45 degrees using the relevant formulas.
	Answer: To calculate the normal stress (${\sigma }_n$) and shear stress (τ) on a plane at an angle θ of 45 degrees, we use the formulas: ${\sigma }_n=\frac{\left({\sigma }_1+{\sigma }_3\right)}{2}+\frac{\left({\sigma }_1-{\sigma }_3\right)}{2} \ast \mathit{cos}\left(2\times 45°\right)$ $\tau =\frac{\left({\sigma }_1-{\sigma }_3\right)}{2}\ast \mathit{sin}\left(2\times 45°\right)$ Substituting the given values: ${\sigma }_1$ = 150 kPa, ${\sigma }_3$ = 50 kPa The calculated normal stress (${\sigma }_n$) and shear stress ($\tau$) on the plane at an angle of 45 degrees are: -Normal stress (${\sigma }_n$) = 100 kPa -Shear stress ($\tau$) = 50 kPa
Analyse	Question: Given data for different soil samples including specific gravity, dry unit weight, and water content, determine the void ratio, degree of saturation, and wet unit weight for each sample.
	Answer: To determine the void ratio, degree of saturation, and wet unit weight for each soil sample, the following steps are taken: -Calculate the void ratio ($e$) using the formula $e=\left(G_s {\gamma }_w)/{\gamma }_d\right)-1.$ -Calculate the degree of saturation ($S$) using $S=\left(G_{s}w\right)/e.$ -Calculate the wet unit weight (${\gamma }_{wet}$) using ${\gamma }_{wet}={\gamma }_d (1+w).$ Specific calculations would depend on the provided data for each soil sample.
Evaluate	Question: Critically assess the applicability of Mohr-Coulomb Failure Theory in predicting shear failure in a clayey soil, considering its cohesion and angle of internal friction.
	Answer: The Mohr-Coulomb Failure Theory is a widely used criterion for predicting shear failure in soils. For clayey soils, which typically have significant cohesion, this theory can be effective in predicting failure. However, the theory assumes linear increase of shear strength with normal stress, which may not always accurately represent the complex behaviour of clay, especially under varied moisture and loading conditions. Thus, while useful, the theory should be applied with consideration of its limitations and possibly supplemented with empirical data specific to the type of clay.
Create	Question: Design a method to experimentally determine the maximum and minimum void ratios of a soil sample, and subsequently calculate its relative density.
	Answer: To experimentally determine the maximum and minimum void ratios, the following method could be designed: -For the maximum void ratio ($e_{max}$) Place a known volume of loosely poured soil in a container and measure its mass. Calculate the void ratio using the soil’s specific gravity and the volume of the container. -For the minimum void ratio ($e_{min}$) Place the same volume of soil in a container and compact it to its maximum density (e.g., using a mechanical compactor), then measure its mass. Calculate the void ratio as before. -With $e_{max}$ and $e_{min}$, calculate the relative density ($D_r$) using $D_r=(e_{max}-e)/(e_{max}-e_{min}).$ This experiment would provide a comprehensive understanding of the soil’s packing characteristics.

showcase the model’s ability after implementing custom instructions to formulate relevant questions and to provide correct answers. It should be noted that the GPT-4’s response is in italics. In addition, we explored GPT-4’s proficiency as an AI tutor by evaluating its ability to correct errors in answering questions associated with geotechnical engineering. This evaluation is detailed in

Table 3. Evaluation of GPT’s Accuracy in Correcting Calculation Errors in Soil Mechanics Problem-Solving.

Prompted Question and Answer:

Question: Given principal stresses ${\sigma }_1$ = 150 kPa and ${\sigma }_3$ = 50 kPa in a soil sample and a friction angle ($\varphi$) of 30 degrees, calculate the normal stress (${\sigma }_n$) and shear stress ($\tau$) on a plane at an angle θ of 45 degrees using the relevant formulas. Answer: To calculate the normal stress (${\sigma }_n$) and shear stress ($\tau$) on a plane at an angle θ of 45 degrees, we use the formulas: ${\sigma }_n=\frac{\left({\sigma }_1+{\sigma }_3\right)}{2}+\frac{\left({\sigma }_1-{\sigma }_3\right)}{2}\cos \left(2\times 45°\right)\tau =\frac{\left({\sigma }_1-{\sigma }_3\right)}{2} \sin \left(2\times 45°\right)$ Substituting the given values: ${\sigma }_1$ = 150 kPa,${\sigma }_3$ = 50 kPa. The calculated normal stress (${\sigma }_n$) and shear stress ($\tau$) on the plane at an angle of 45 degrees are: -Normal stress (${\sigma }_n$) = 120 kPa − Shear stress ($\tau$) = 60 kPa. Can you check if the solution is correct?

GPT’s Evaluation and Response:

The correct normal stress (${\sigma }_n$) and shear stress ($\tau$) on the plane at an angle of 45 degrees, calculated using the given values of${\sigma }_1=150 kPa$ and ${\sigma }_3=50$ kPa, are: Normal stress (${\sigma }_n$) = 100 kPa Shear stress ($\tau$) = 50 kPa The previously provided answer is incorrect.

. This table shows that GPT-4 corrected the initial response, demonstrating its influential role as an AI tutor in identifying and rectifying errors in complex fundamental soil mechanics problems.

6. Conclusions

The paper investigates the application of large language models (LLMs), specifically GPT-4, in geotechnical engineering education. It assesses GPT-4’s performance across various topics and cognitive complexity levels using different prompting strategies. The study reveals that while GPT-4 shows adeptness in handling basic geotechnical concepts and problems, its effectiveness varies with task complexity and prompting methods. Errors in GPT-4 responses are categorised into “Conceptual”, “Grounding”, “Calculation”, and “Deficiency” related to visual interpretation. While the “Regenerate” function of ChatGPT has potential utility for improving the diversity of responses and correcting immediate errors, it should be used carefully in educational settings. Educators and students should employ this function as part of a broader strategy that includes critical evaluation and verification of AI-generated content, particularly in specialised fields such as geotechnical engineering. Our proposed prompting strategy—Custom Instruction—significantly enhanced GPT-4’s performance, particularly in rectifying most errors, albeit with persistent challenges in calculations and visual interpretations, and hence such use is recommended.

A salient aspect of our investigation is the indispensable role of human intervention in refining GPT-4’s outputs for complex tasks, underscoring the need for human–AI collaboration. The study demonstrates GPT-4’s potential as a valuable tool in geotechnical education, such as generating questions, answers and feedback, and its emerging role as an interactive AI tutor for enriching students’ learning experience.

However, our research also brings to light AI’s limitations and reliability concerns in geotechnical engineering. The study suggests that despite ChatGPT’s potential to assist in educational scenarios, its responses often need verification and refinement by knowledgeable individuals. These limitations stem from the model’s general training approach and inherent constraints, often leading to inaccuracies or oversimplifications in addressing complex problems. The lack of domain-specific training and the model’s inherent limitations sometimes lead to inaccuracies or oversimplifications in complex problem-solving scenarios.

Therefore, a significant future direction for AI in education is the development of more sophisticated, domain-tailored generative AI models. These models should be trained with discipline-specific datasets to enhance their accuracy and reliability. Until such advancements are achieved, human oversight remains essential in utilising AI tools like ChatGPT. Educators and students must critically evaluate AI-generated content, ensuring it aligns with factual and theoretical standards.

In summary, while generative AI in geotechnical education opens up exciting avenues for innovative learning and teaching, its current state demands a balanced integration, with human expertise playing a crucial role in guiding and overseeing its application. The future evolution of generative AI in education hinges on technological advancements and its responsible and informed use by educators and learners.