Framework for Evaluating LLM Performance in Undergraduate Calculus

Abstract

Large language models (LLMs) are increasingly being used in education, yet their correctness alone does not capture the quality, reliability, or pedagogical validity of their problem-solving behavior, especially in mathematics, where multi-step logic, symbolic reasoning, and conceptual clarity are critical. Conventional evaluation methods largely focus on final answer accuracy and overlook the reasoning process. To address this gap, we introduce a novel interpretability framework for analyzing LLM-generated solutions using undergraduate calculus problems as a representative domain. Our approach combines reasoning flow extraction and decomposing solutions into semantically labeled operations and concepts with prompt ablation analysis to assess input salience and output stability. Using structured metrics such as reasoning complexity, phrase sensitivity, and robustness, we evaluated the model behavior on real Calculus I–III university exams and compared it with the performances of students enrolled in the courses. Our findings revealed that LLMs often produce syntactically fluent yet conceptually flawed solutions with reasoning patterns sensitive to prompt phrasing and input variation. This framework enables a fine-grained diagnosis of reasoning failures, supports curriculum alignment, and informs the design of interpretable AI-assisted feedback tools. The framework was evaluated on Gemma 3, an open-access large language model, across zero-shot, retrieval-augmented generation, and contextual retrieval configurations, using nine real undergraduate calculus examinations from three course levels. To our knowledge, this is the first paper to apply a combined reasoning flow decomposition and prompt ablation framework to real undergraduate calculus examinations, benchmarked against actual student cohort performance, laying the foundation for the transparent and responsible deployment of AI in STEM learning environments.

1. Introduction

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, question answering, and even basic symbolic reasoning. As these systems become increasingly integrated into educational environments ranging from intelligent tutoring systems to AI-assisted assessment [1,2], the need to evaluate and interpret problem-solving behavior has grown urgent, especially in structured and logic-intensive domains like mathematics. Evaluating whether these systems truly “understand” mathematical reasoning remains an open question, especially when it comes to multi-step logic, symbolic manipulation, and conceptual abstraction. Traditional evaluation methods that focus solely on answer correctness often obscure how an LLM arrives at its solution or whether the reasoning process is pedagogically sound [3,4].

While prior studies have explored LLMs’ performance on mathematical benchmarks [5] and demonstrated that intermediate reasoning steps improve accuracy [6], significantly less attention has been paid to the interpretability and pedagogical validity of those reasoning processes in structured educational settings. Mathematics, particularly calculus, demands precision, symbolic manipulation, and logical structural characteristics that go beyond surface-level fluency. Even when LLMs produce correct final answers, the intermediate steps may contain conceptual errors, skipped justifications, or fragile dependencies on prompt phrasing that are invisible to accuracy-only evaluation [7]. Merely assessing whether a model generates the correct answer is insufficient to understand the mechanisms or rationale behind its conclusions nor does it evaluate the internal consistency of its problem-solving process. For large language models to be responsibly integrated into educational environments, particularly within STEM fields, it is imperative to extend beyond mere accuracy and develop methodologies to systematically interpret reasoning processes [8,9]. To address this gap [10,11], we introduce a structured interpretability framework designed to analyze and explain how LLMs reason through mathematical problems. Our framework consists of two primary components: (1) a reasoning flow analysis that breaks down model outputs into labeled operations, concepts, and complexity scores, and (2) a prompt sensitivity ablation method that quantifies the influence of specific input elements on output behavior. To our knowledge, our interpretability framework contributes a post hoc decomposition pipeline for LLM-generated mathematical solutions that combines structural reasoning flow and ablation analysis, as illustrated in Section 3. By reconstructing how models reason or misreason and identifying which input elements most affect that reasoning, we provide tools for the following:

• Curriculum Alignment: evaluate whether model reasoning follows pedagogically valid patterns.
• Failure Prediction: flag brittle responses to complex or under specified prompts.
• Retrieval Tuning: quantify how retrieval strategies affect the robustness of generated solutions.
• Training Prioritization: identify operations and concepts where models consistently struggle.

This enables deeper insight into model reasoning beyond accuracy metrics. We apply this framework in a real-world case study evaluating the behavior of LLMs on Calculus I–III university exams and benchmarking against real students’ scores. These exams serve as rigorous, structured testbeds, allowing us to examine how well the model handles varying levels of mathematical complexity and abstraction. While we report standard performance scores against student average scores graded by mathematical faculty experts, the primary goal of this paper is to demonstrate the usefulness of our framework in identifying reasoning patterns, inconsistencies, and prompt sensitivities that would otherwise remain hidden while benchmarking the performance and not comprehensively evaluating all LLM model performance on calculus. We further tested the performance of LLMs by incorporating external knowledge using retrieval-augmented generation (RAG) and contextual retrieval methods, evaluating how these affect not only accuracy but also reasoning fidelity. Contrary to expectations, naive retrieval strategies often degrade performance, underscoring the importance of context alignment and interpretability when augmenting LLMs in high-stakes domains. Additionally, we discuss expert feedback on the limitations of LLM performance in calculus problem solving.

1.1. Motivation: Need for Reasoning Centered Interpretability

While large language models are increasingly capable of solving mathematical problems, their deployment in real-world education remains constrained by a lack of transparent reasoning. In high-stakes STEM domains, correctness alone is insufficient; both educators and learners must understand the how and why behind a model’s solution. LLM-generated responses can be superficially fluent yet pedagogically misleading, making it difficult to assess whether reasoning is valid, shallow, or inconsistent.

This gap motivates the need for interpretability frameworks tailored to education. Standard metrics such as rubric scores or end-to-end accuracy fail to capture multi-step logic, symbolic transformations, or sensitivity to prompt phrasing. To support instructional alignment and responsible model use, educators require tools to trace reasoning failures, uncover conceptual misunderstandings, and identify brittle dependencies in model behavior. Our work addresses this by proposing a post hoc interpretability framework that decomposes LLM solutions into reasoning chains, evaluates robustness through controlled perturbations, and quantifies step complexity, sensitivity, and stability. These methods offer actionable insights for debugging models, aligning AI outputs with curricula, and enhancing learner trust.

1.2. Calculus as a Case Study

Calculus provides an ideal domain for evaluating reasoning in LLMs due to its structured progression of complexity and central role in undergraduate STEM education. Most university curricula divide calculus into three sequential courses, Calculus I, II, and III—which are typically completed during the first two years of study. These courses not only build mathematical fluency but also serve as a foundation for advanced topics in engineering, physics, computer science, and economics. Calculus I introduces core concepts such as limits, continuity, derivatives, and introductory integration, emphasizing procedural fluency and practical problem solving. Calculus II deepens abstraction through integration techniques, series, and parametric representations, requiring logical reasoning and conceptual synthesis. Calculus III advances to multivariable functions, vector calculus, and 3D geometric modeling, demanding spatial reasoning and symbolic precision. This progression from concrete computation to abstract generalization offers a natural framework for probing LLM behavior across increasing cognitive demands. By applying our interpretability framework to real Calculus I–III exams, we can assess how LLMs handle procedural steps, symbolic manipulations, and conceptual understanding across a range of mathematical tasks.

1.3. Contributions

This paper makes the following contributions to the fields of educational AI and explainable machine learning:

1.

Interpretability framework for LLMs in mathematics: We introduce a domain-grounded interpretability framework that decomposes LLM-generated solutions into semantically annotated reasoning steps and evaluates sensitivity to prompt phrasing through controlled ablations.

2.

Structured metrics for reasoning analysis: We define and operationalize quantitative metrics such as reasoning complexity, robustness, phrase sensitivity, and step count that enable a fine-grained evaluation of model behavior beyond correctness.

3.

Empirical case study using real calculus exams: We apply the framework to a series of Calculus I–III university exams, benchmarking LLM performance against human student scores and examining the effects of knowledge augmented configurations.

4.

Insights into pedagogical alignment and model limitations: Our analysis reveals common reasoning failures, sensitivity to linguistic variation, and gaps in conceptual understanding, offering guidance for responsible classroom use and instructional alignment.

5.

Qualitative interpretation of reasoning behavior: We present a discussion of model behavior through reasoning flow decomposition and qualitative case studies, identifying both successful multi-step reasoning patterns and concrete failure modes, grounding the quantitative metrics in interpretable examples.

To our knowledge, this is the first study to apply reasoning flow decomposition combined with prompt ablation analysis to real undergraduate calculus examinations benchmarked against actual student cohort performance. By making reasoning processes transparent, our framework supports the design of interpretable and accountable AI tools for use in formative feedback, grading, and tutoring in STEM education. This paper contributes to the informatics literature in two primary respects. First, it introduces a computational framework comprising a rule-based reasoning flow parser, a TF-IDF and Levenshtein ablation pipeline, and a metric aggregation module that can serve as infrastructure for evaluating generative AI in any structured reasoning domain. Second, it addresses a core informatics challenge: designing evaluation systems for generative AI that are both technically rigorous and domain-interpretable, connecting NLP evaluation methodology to applied educational contexts.

The rest of the paper is organized as follows. Section 2 discusses related research. Section 3 explains the proposed interpretability framework. Section 4 elaborates the experimental methodology, and Section 5 presents quantitative results. Section 6 provides a qualitative discussion and interpretation of model behavior. Section 7 presents expert observations, pedagogical implications, and future directions. Section 8 presents the concluding remarks.

2. Related Works

2.1. Explainable AI in Education

Recent advancements in educational AI emphasize the growing importance of interpretability, fairness, and human-centered design, particularly in the deployment of large language models in classroom settings. Prior work by Maity et al. [8] underscores the value of explanations that are not only technically accurate but also pedagogically meaningful, advocating for transparency and cultural sensitivity in AI-generated feedback. In alignment with this vision, our work focuses on making LLM outputs interpretable in a manner that supports student learning and instructor insight into calculus. Chinta et al. [12] explored fairness-aware design in AI for education, noting the potential for algorithmic bias and the necessity of stakeholder-informed development. Their findings motivate our focus on fine-grained reasoning analysis, which reveals disparities in how LLMs handle procedural versus conceptual tasks, which is an important consideration for equitable learning support. Schneider et al. [9] provided a comprehensive review of explainability in generative AI, including interactivity, verifiability, and audience-specific customization. While these principles have seen limited adoption in educational contexts, our framework operationalizes them by decomposing mathematical reasoning into annotated steps, revealing both the strengths and breakdowns of model logic. Concerns about generative AI alignment in education are further highlighted by Gupta et al. [10], who noted risks such as hallucinations, curriculum misalignment, and lack of cultural grounding. These issues are particularly acute in mathematics, where precision and conceptual fidelity are critical. Our interpretability framework responds by exposing subtle flaws and missteps in LLM outputs even when the final answers appear correct.

From a policy standpoint, the U.S. Department of Education (https://www.ed.gov/sites/ed/files/documents/ai-report/ai-report.pdf) (accessed on 4 April 2026) [13] outlines principles for responsible AI in education, calling for contextual adaptation, iterative evaluation, and support for learner agency. This paper echoes this by offering a systematic, data-driven method for evaluating LLM behavior in real curricular scenarios, enabling continuous refinement and oversight. Ethical considerations have also been emphasized by Memarian et al. [14], who advocate for participatory design and stakeholder involvement in AI systems used in higher education. Our approach supports this by providing interpretable artifacts (e.g., reasoning graphs and sensitivity scores) that can be meaningfully used by students and instructors alike. This paper addresses these concerns by presenting structured quantitative metrics tailored to mathematics tasks, grounded in domain expertise, and validated through real exam questions.

Foundational taxonomies of interpretability methods, such as those presented by [15], help situate our method along the spectrum of post hoc explanation tools. By blending symbolic reasoning patterns with semantic annotations, our framework offers a hybrid approach that balances fidelity and usability for both technical and non-technical users. Wang et al. [16] propose LLM-based symbolic programs as a step toward more transparent AI systems, combining neural outputs with rule-based logic. While our research does not rely on symbolic synthesis, it shares a similar motivation for making the internal reasoning of LLMs observable and aligned with human expectations in educational contexts. The theoretical contributions of this paper also shape our approach. Rosé et al. [11] stressed the need for AI explanations that support sensemaking and metacognition rather than merely improving prediction accuracy. Clancey [17] extends this argument by framing explanation as a socially situated process that is dynamic, dialogic, and deeply dependent on context. These ideas emphasize step-level analysis and prompt sensitivity as a means of reconstructing how the model understands and solves mathematical problems. Finally, broader surveys in explainable ML [18,19] identify key metrics of explanatory quality, such as cognitive load, user trust, and task relevance. Our interpretability framework is informed by these principles, prioritizing explanations that are not only technically valid but also cognitively useful in high-stakes, domain-specific learning environments such as undergraduate calculus.

Collectively, these studies provide a robust foundation for evaluating and designing explainable AI systems in education. However, few studies have rigorously examined LLM reasoning on domain-specific assessments at a fine-grained level. This paper addresses this gap by offering a comprehensive interpretability framework that captures reasoning structure, input sensitivity, and step complexity, providing insights beyond accuracy that are essential for responsible educational deployment.

2.2. Reasoning Chains in NLP

Reasoning chains in natural language processing (NLP) are pivotal to enhancing machine understanding and response generation capabilities. They represent the sequences of logical steps or decisions that an AI model takes to arrive at a conclusion. Within the context of NLP, reasoning chains can significantly impact tasks such as question answering, natural language inference, and multi-hop reasoning, where models are required to connect disparate pieces of information to generate coherent and relevant outputs. The development of reasoning chains involves advanced techniques such as Chain-of-Thought (CoT) prompting, which encourages models to generate intermediate reasoning steps before producing a final answer. This method aims to enhance the comprehensiveness and accuracy of the responses generated by language models. For instance, the LLaMA-LoRA (Large Language Model Meta AI with Low-Rank Adaptation) model combines neural prompt engineering with low-rank adaptation to improve logical reasoning capabilities in Chinese NLP, surpassing benchmarks set by models such as GPT-3.5 [20]. Despite advancements, challenges remain, as seen in models such as GPT-5, which excels in certain reasoning tasks but struggles with others, such as commonsense reasoning and summarization. This inconsistency underscores the need for continued evolution in AI models to effectively handle diverse NLP tasks [21]. An extensive survey of natural language reasoning emphasizes the importance of both classical logical reasoning and emerging avenues, such as defeasible reasoning in NLP. The focus is on constructing precise reasoning frameworks that incorporate external knowledge, thereby advancing the state of AI-driven language models [22]. Moreover, natural language inference (NLI) tasks have benefited from utilizing external knowledge bases, highlighting the potential of structured knowledge in enhancing model performance in specific domains, such as science questions [23]. In summary, reasoning chains in NLP are essential for achieving higher-order logic and improving model comprehension. Continued research and refinement in this area promises to address current limitations, guiding the next generation of AI language models toward more robust and nuanced understanding and reasoning capabilities.

2.3. LLMs in Mathematics

The reasoning capabilities of large language models in mathematics have garnered substantial interest owing to the inherent complexities involved in mathematical problem solving. LLMs, such as GPT-3 and GPT-4, demonstrate considerable potential in natural language processing tasks and have extended their capabilities to solving certain reasoning problems in mathematics. However, despite their impressive performance, several aspects underline their limitations in terms of mathematical reasoning. One primary limitation is that LLMs often display shallow reasoning capabilities even with advanced prompting techniques. Formal logic, which is essential for complex reasoning, remains a challenge for LLMs, as they struggle to accurately convert natural language into formal logic. One study highlighted the potential of LLMs in transforming natural language descriptions of logic puzzles into answer set programs, demonstrating that with carefully designed prompts and few learning examples, LLMs could assist in creating complex logic programs. However, errors are common and simple enough for human intervention [24]. LLMs have been subjected to tests such as GSM8K to evaluate their mathematical reasoning. Although their performance has improved, studies indicate that these improvements do not necessarily translate into genuine advancements in reasoning capabilities. The models often replicate reasoning steps from their training data rather than performing genuine logical reasoning [3]. Foundational work on evaluating mathematical reasoning capabilities introduced the MATH dataset, comprising 12,500 challenging competition-level problems, demonstrating that even large transformer models achieve relatively low accuracy and that scaling model size alone is insufficient for robust mathematical reasoning [5]. Similarly, chain-of-thought prompting, which generates intermediate reasoning steps before producing a final answer, has been shown to substantially improve LLM performance on arithmetic and symbolic reasoning tasks, underscoring that how a model reasons matters as much as whether it arrives at the correct answer [6]. Furthermore, a dynamically adaptive framework has been proposed to classify errors in LLMs’ approach to Math Word Problems (MWPs). Traditional error classification is limited by static categories, thus failing to accurately reflect the different error patterns in mathematical reasoning. By analyzing 15 LLMs across several datasets, researchers introduced MWPES-300K, which is a dataset designed for comprehensive error analysis. This study showed that as models become more sophisticated, their error patterns shift from basic to complex, emphasizing the need for error-aware prompting strategies to improve mathematical reasoning performance [4]. Moreover, a distinct prompting strategy termed the Pedagogical Chain-of-Thought was devised to enhance LLMs’ ability to detect reasoning mistakes, particularly in mathematics. By employing educational theory principles, this approach reportedly outperforms traditional prompting strategies in identifying errors, thus laying the groundwork for more reliable math-answer grading [25]. Fine tuning LLMs for mathematical reasoning has shown promise. MetaMath, a specialized model trained on a dataset named MetaMathQA, has demonstrated superior performance on mathematical reasoning benchmarks compared to many open-source models. It outperformed others in the GSM8K and MATH benchmarks by a significant margin, showcasing the potential of targeted fine tuning in enhancing mathematical reasoning capabilities [26]. At the frontier of domain-specific mathematical reasoning, Lewkowycz et al. [27] introduced Minerva, which is a large language model further trained on scientific and mathematical content from arXiv and technical web pages. Minerva achieves strong performance on undergraduate-level quantitative problems across physics, engineering, and mathematics, demonstrating that domain-adapted training substantially improves multi-step symbolic reasoning of the kind required in calculus examinations.

Recent literature also highlights the growing role of large language models in supporting mathematics education, particularly in domains such as calculus. Liu and Yang [28] examine how LLMs assist students in programming related tasks by providing guidance in mathematical logic and reasoning, thereby enhancing conceptual understanding in calculus. In addition to computational support, these models help students engage more deeply with problem-solving processes, especially in engineering contexts where tools such as MATLAB are commonly used. Sharma et al. [1] explore the capacity of LLMs to deliver personalized learning experiences. By adapting responses to individual queries, these systems help tailor instruction to student-specific needs, improving engagement and academic outcomes across a variety of STEM disciplines, including calculus. This personalized support positions LLMs as adaptive educational tools capable of complementing traditional instruction. Bernabei et al. [29] document the increasing use of LLMs, such as ChatGPT, by students for content generation, academic assessment preparation, and mathematical problem solving. These tools act as virtual tutors or writing assistants, enabling students to refine their understanding and articulate solutions more effectively. Despite their promise, the integration of LLMs into educational workflows is not without challenges. Kang et al. [30] and Wang et al. [2] raise concerns regarding the accuracy and ethical implications of AI-generated content. They stress the importance of validating AI outputs and ensuring that LLM-assisted learning preserves academic integrity. These studies collectively call for hybrid educational approaches that combine the efficiency of AI with pedagogically sound, human-guided learning frameworks.

Although existing research has made significant strides in understanding the potential of LLMs and explainable AI in educational settings, several key limitations persist. Most prior studies have focused on high-level applications or generic instructional benefits without delving into domain-specific evaluations of reasoning quality or interpretability at a granular level. Furthermore, few studies offer systematic frameworks for identifying latent errors, evaluating sensitivity to prompt changes, or decomposing mathematical reasoning into meaningful pedagogical components. Our study addresses these gaps by introducing a domain-grounded interpretability framework tailored to undergraduate calculus assessments. Unlike prior work in explainable AI for education, which typically focuses on language tasks or limited numerical feedback, our framework provides symbolic traceability, sensitivity quantification, and cognitive complexity estimation metrics rarely explored in the context of undergraduate mathematics education. By combining symbolic annotation, reasoning decomposition, and robustness analysis, we move beyond accuracy to illuminate how, why, and when LLMs succeed or fail in an essential step toward safe, transparent, and educationally effective AI deployment in mathematics education. This perspective aligns with recent findings that process supervision, which evaluates the correctness of each intermediate reasoning step rather than only the final answer, significantly outperforms outcome-only evaluation for training reliable models on complex mathematical tasks [7]. Our interpretability framework operationalizes a similar principle for the educational context: by decomposing LLM-generated solutions into labeled reasoning steps and quantifying their sensitivity to input perturbations, we provide educators and researchers with step-level diagnostic tools that go beyond correctness to assess the quality and stability of mathematical reasoning.

3. Interpretability Framework and Quantitative Evaluation

To move beyond correctness and enable deeper insight into LLM behavior on mathematical tasks, in this paper, we introduce a structured interpretability framework that analyzes both the reasoning trajectories and input dependencies of LLM-generated solutions, as presented in Figure 1. Our framework comprises two primary complementary modules: (1) the reasoning flow module, which decomposes step-by-step outputs into semantically annotated chains of thought, and (2) the sensitivity module, which quantifies the influence of input components through controlled perturbation. We further grounded our framework components in quantitative metrics, enabling fine-grained comparisons across questions, systems, and retrieval strategies.

3.1. Structured Reasoning Flow Analysis

Our reasoning flow module presents a systematic method for extracting and analyzing the reasoning flows embedded in LLM-generated calculus solutions. Starting from detailed step-by-step outputs produced by the model, we segment these solutions into discrete reasoning steps using rule-based parsing. Each step is then semantically annotated along the three critical dimensions as follows:

• Mathematical Operation: fundamental actions such as differentiation, substitution, simplification, or integration.
• Conceptual Tag: domain-specific concepts including the chain rule, directional derivatives, or the Fundamental Theorem of Calculus.
• Cognitive Complexity: an estimate of abstraction or difficulty level (low, medium, high) derived from symbolic depth and syntactic complexity.

These annotated steps form a directed reasoning graph, where nodes represent individual reasoning steps, and edges capture logical dependencies. From this structure, we compute summary statistics, such as the total operation count, unique concepts used, average complexity per step, and a composite reasoning complexity score. This approach provides a transparent and interpretable reconstruction of the model’s latent reasoning process, enabling the detection of skipped justifications, conceptual errors, or over-generalizations that are not evident from final correctness alone.

3.2. Sensitivity Ablation and Input Salience

Our sensitivity module systematically quantifies the sensitivity of LLM outputs to specific input question components. We implemented a structured ablation procedure, followed by divergence scoring, to quantify the impact score. Each calculus question prompt passed to the LLM is decomposed into constituent elements, including the following:

• Mathematical Operation (e.g., $f\left(x\right)$, $\nabla f$, coordinate vectors);
• Instructional keywords (e.g., “find”, “compute”, “evaluate”);
• Numerical and coordinate references (e.g., $(x,y)=(-1,4)$);
• Linguistic features (e.g., punctuation, casing, or word order).

For each ablation trial, one element was masked or syntactically perturbed while preserving the rest of the prompt. For example, removing the reference $(-1,4)$ from the prompt “Find the gradient $\nabla f$ of $f(x,y)=x^{2}y$ at point $(-1,4)$” results in the modified input: “Find the gradient $\nabla f$ of $f(x,y)=x^{2}y$”. The LLM is queried with both the original and ablated prompts. To evaluate the semantic and structural divergence between the original output and each ablated output, we employ the following:

• TF-IDF Weighted Cosine Similarity: outputs are vectorized using TF-IDF representations, and cosine similarity is computed to assess semantic shifts in topic and terminology.
• Normalized Edit Distance (Levenshtein Distance): measures token-level changes between original and ablated responses, reflecting structural or syntactic perturbations.

These metrics were aggregated across all ablations for a given question to compute two quantized interpretability scores:

• Phrase Sensitivity: the maximum divergence observed across all ablations, reflecting the most influential input token or phrase for each question.
• Robustness Score: the average semantic similarity across ablated responses, capturing the overall output stability for each question.

This analysis reveals how sensitive the model’s reasoning is to specific elements in the prompt, identifying brittle linguistic dependencies and conceptually salient input features. Our approach allows educators and researchers to understand which parts of a problem influence model behavior the most and where reasoning stability may break down.

Table 1. Worked ablation example: original prompt and four ablated variants for the gradient problem. Each ablation removes or substitutes one element type while preserving the rest.

Variant	Prompt Text	Element Removed
Original	Find the gradient $\nabla f$ of $f(x,y)=x^{2}y$ at point $(-1,4)$	—
Ablation 1	Find the gradient $\nabla f$ of $f(x,y)=x^{2}y$	Coordinate reference $(-1,4)$
Ablation 2	Find the gradient $\nabla f$ at point $(-1,4)$	Function expression $f(x,y)=x^{2}y$
Ablation 3	Compute the gradient $\nabla f$ of $f(x,y)=x^{2}y$ at point $(-1,4)$	Instructional keyword “find”
Ablation 4	Find the gradient of $f(x,y)=x^{2}y$ at point $(-1,4)$	Symbolic notation $\nabla f$

provides a worked example of the ablation procedure applied to the gradient and directional derivative question used in our case study (Section 3.1).

3.3. Quantitative Metrics and Aggregation

This section elaborates our rigorous approach for evaluating the proposed interpretability framework by computing the quantitative metrics at the question level that capture key aspects of model behavior, such as semantic stability, reasoning complexity, reasoning pattern details, and sensitivity to phrasing variations. These detailed question-level metrics offer insights into how the model processes individual problems. Subsequently, these metrics are then aggregated to produce exam-level summaries and presented in Section 5 for the dual purposes of (1) characterizing the overall LLM performance and reasoning patterns across the full assessment and (2) the clarity of the presentation of results in this paper. This section first defines and interprets each question-level metric, followed by aggregated exam-level metrics, concluding with a discussion of the importance of these metrics for interpreting large language model outputs in mathematical problem-solving tasks.

Table 2. Question -level metrics: interpretation.

Metric	Interpretation
Robustness	Measures model stability to variations in question phrasing.
Impact Factor	Identifies tokens or critical elements most influential to the model’s output.
Step Count	Reflects the depth and complexity of the model’s multi-step reasoning.
Complexity	Quantifies the difficulty and reasoning demands of the question.
Reasoning Pattern Trace	Captures the model’s reasoning approach and patterns through the question.
Phrase Sensitivity	Measures how sensitive the model’s response is to changes in phrasing.

summarizes question-level metrics and

Table 3. Exam -level interpretation and evaluation metrics.

Metric	Notes
Robustness	Measures model stability to phrasing variations across all questions in an exam.
Complexity	Based on individual question features (e.g., steps, nested logic).
Step Count	Inferred from reasoning path per question, then averaged.
Phrase Sensitivity	Measures change in output with paraphrasing per question.

summarizes exam-level metrics.

• Robustness measures the model’s consistency in producing correct answers across variations in input phrasing or question structure. A robust LLM maintains its performance even when questions are paraphrased or perturbed. High robustness indicates stability and reliability in real-world or noisy user settings. The standard deviation helps assess performance volatility across questions, which are then aggregated at exam-levels. Formally, robustness is defined as shown below:

  $$R={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}cos({\mathbf{v}}_{\mathrm{orig}},{\mathbf{v}}_{{\mathrm{abl}}_i})$$

  (1)

  where ${\mathbf{v}}_{\mathrm{orig}}$ is the TF-IDF vector of the original response, ${\mathbf{v}}_{{\mathrm{abl}}_i}$ is the TF-IDF vector of the i-th ablated response, and n is the total number of ablations per question.
• Impact Factor identifies which parts of the input question most strongly influence the LLM’s response, which are derived from ablation or attention analysis. By pinpointing key tokens or phrases that the model relies upon, researchers can understand the model’s focus areas and whether it attends to relevant information pertaining to mathematical reasoning. This insight helps interpret how the model reasons and can highlight potential biases or errors in understanding, improving transparency at the question level.
• Complexity evaluates the difficulty of the reasoning involved, considering factors such as the number of mathematical operations, nested logic, and conceptual constructs required. This metric contextualizes the model’s performance by linking reasoning demands to output quality at the question level, which we also aggregate at the exam level to understand the complexity of the overall exam. Understanding complexity allows researchers to identify which question types challenge the model and where interpretability methods should focus to better explain intricate reasoning paths.
• Reasoning Pattern Trace captures the sequence and types of logical or mathematical operations the LLM applies during problem solving. It provides a structured map of the model’s thought process, revealing whether it uses appropriate methods such as substitution, differentiation, or evaluation at each step for each question. This trace is crucial for interpretability because it allows researchers to verify the correctness and coherence of the model’s reasoning flow, allowing debugging and refinement.
• Complexity captures the semantic and syntactic difficulty of the questions based on token length, number of operations, or concept depth. This metric contextualizes performance, and higher complexity correlates with increased cognitive load. Comparing performance across complexity levels helps diagnose whether models struggle disproportionately with harder questions. We aggregated the complexity results per question and assigned the overall exam a complexity score. Formally, the complexity score is computed as shown below:

  $$C=0.5\times \left|\mathcal{O}\right|+0.3\times \left|\mathcal{K}\right|+0.2\times \overline{d}$$

  (2)

  where $\left|\mathcal{O}\right|$ is the number of unique mathematical operations, $\left|\mathcal{K}\right|$ is the number of unique conceptual tags, and $\overline{d}$ is the mean syntactic depth per reasoning step. Weights were calibrated to reflect cognitive load consistent with Bloom’s Taxonomy Apply and Analyze levels [31,32].
• Step Count represents the average number of reasoning or computational steps taken (or inferred) by the model in its answer for each question and aggregated at the exam level. Higher step counts suggest multi-hop reasoning or deeper logical chaining. This metric evaluates not only correctness but also how the model arrives at an answer that is important for interpretability and alignment with human-like problem solving. Formally, the step count per exam is shown below:

  $$S={\displaystyle \frac{1}{\left|Q\right|}}\sum _{q\in Q}{s}_q$$

  (3)

  where $s_q$ is the number of discrete reasoning steps segmented by the rule-based parser for question q, and $\left|Q\right|$ is the number of questions in the exam.
• Phrase Sensitivity measures the extent to which minor changes in wording or phrasing impact the model’s output. Low phrase sensitivity implies semantic understanding beyond surface patterns. High phrase sensitivity indicates an over-reliance on specific phrasing or keywords, which undermines generalization. This metric is essential for gauging robustness to natural language variability for each question and then aggregated at the exam level. Formally, phrase sensitivity is defined as shown below:

  $$PS=\underset{i}{max}\left(1-cos({\mathbf{v}}_{\mathrm{orig}},{\mathbf{v}}_{{\mathrm{abl}}_i})\right)$$

  (4)
  This captures the single most influential input element: the ablation that produced the largest semantic divergence from the original output.

We would like to acknowledge that no statistical testing between the student and LLM performances were conducted as we did not access individual student data but only the class averages, which restricts statistical analysis. We believe while statistical analysis is important for understanding the relation between student performance and LLM performance, this is beyond the scope of this paper with the main contribution of interpretability framework and its evaluation. Additionally, a link to our framework implementation in python is provided for reproducible use at the end of this paper.

4. Experimental Setup

In this section, we discuss our experimental setup to evaluate our interpretable framework by selecting a high-performing LLM model and establishing the baseline by comparing the model performance to real student scores across multiple levels of calculus for a semester worth of exams. We also illustrate our knowledge augmentation experiments to evaluate the effect on LLM performance, namely retrieval-augmented generation and contextual retrieval methods. Furthermore, all three experimental configurations are compared in terms of the aggregated metrics introduced in Section 3.3.

4.1. Real World Data

For the benchmarking of LLM scores against real world student performance, aggregated class average for three courses—namely, Calculus I, II, and III—were used from a US institution. Each course has three exams termed here as Exam I, II, and III following a structured curriculum. Each course has approximately 50 students, and the scores presented here are average of the student population. Specifically, Calculus I had approximately 48 students, Calculus II had approximately 51 students, and Calculus III had approximately 47 students, representing complete class populations for the semester. Only pre-computed aggregate class-level averages were used in this paper; no individual student records, scores, or identifiers were accessed or retained. As only anonymized aggregate data were used, IRB review was not required. Further details of the courses (such as course numbers, university names) are not disclosed to ensure the privacy and protection of undergraduate students and faculty members. Additionally, further details of the course are not relevant to this paper in evaluating our proposed framework for the interpretation of mathematical reasoning of LLMs in Calculus.

4.2. Grading Rubric

The following standardized rubric (

Table 4. Calculus exam grading rubric (per question).

Criteria	Points Awarded	Description
Correct Method/Setup	30%	Select appropriate approach (e.g., product rule, substitution, Taylor series, etc.). Correct identification of limits, bounds, or parameterization.
Execution/Computation	40%	Accurate algebra, arithmetic, and calculus steps. Logical progression of work. Minor algebraic or sign errors receive partial credit.
Correct Final Answer	10%	Correct numeric or symbolic answer with appropriate simplification. No credit if work is largely incorrect even if the final answer is correct.
Mathematical Notation & Units	10%	Proper use of notation (integral signs, ${\displaystyle \frac{dx}{dt}}$, etc.). Include units where appropriate.
Clarity/Explanation	10%	Steps are shown clearly. Reasoning is explained when needed (e.g., why a series converges).

) is used to evaluate student and LLM responses to individual exam questions in Calculus I, II, and III. Each question is graded not only on the correctness of the final answer but also on the process, clarity, and mathematical communication. This rubric was developed by the mathematics faculty member responsible for the evaluated courses and follows standard undergraduate calculus assessment practice; the criterion weighting reflects the pedagogical emphasis on process over final answer and is transferable to comparable courses at other institutions.

Partial credit is awarded based on the level of understanding demonstrated in the student’s or LLM’s work. If student or LLM applies an incorrect method but shows some relevant and conceptually related steps, it may receive up to 25–30% of the total points for that question. If the correct method is chosen but significant errors occur in computation, algebra, calculus, or writing proper mathematical notation, typically 40–60% of the points may be awarded, depending on the severity of the mistakes. For minor arithmetic, notation, or sign errors with an otherwise correct method and steps, partial credit of 80–90% is usually given. Full credit is reserved for responses that are mathematically correct, logically organized, and clearly presented. Responses that are completely incorrect, incoherent, or left blank typically receive no credit.

Consider a problem worth 10 points where the task is to evaluate an integral using substitution. The grading breakdown may be as follows:

• Correct substitution and integral setup: 3 points;
• Correct integration steps: 4 points;
• Correct final answer: 1 point;
• Proper notation (e.g., changing limits or including $dx$/$du$): 1 point;
• Clarity of substitution steps and back-substitution: 1 point.

If a student or LLM uses the wrong substitution but proceeds logically, they might still earn partial credit for the structure and attempt. However, omitting key steps or making conceptual errors (e.g., forgetting to change limits in definite integrals) will significantly reduce the score.

4.3. Baseline Model Selection and Prompt Engineering

To establish a performance benchmark and identify the most suitable large language model for our experiments, we conducted baseline assessments using Calculus I Exam 1. This exam serves as a representative introductory assessment, comprising eight questions that cover fundamental topics, such as limits, continuity, derivatives, and function analysis. Each candidate LLM was evaluated using a standardized set of instructions as a system role designed to simulate a realistic testing environment. The final prompt presented here has been modified iteratively to accommodate formatting inconsistencies in LLM-generated text to allow our mathematics experts to grade using their rubrics and promote chain-of-thought output, as expected from a student and not the final answer directly. The structured prompt used across all experiments is shown in

Table 5. System role prompt used for LLM evaluation.

Component	Instruction
Role	You are an undergrad student taking an exam. Answer each question thoroughly, completely and show all steps.
Notation	Do NOT use LaTeX or math markup of any kind. Use plain, human readable math notation only (e.g., $f\left(x\right)=x^2/4$, not \frac{x^2}{4}).
Format	Do not include any $$, \⁢(⁢\⁢), or backslashes. Write math as it would appear on paper using keyboard characters.

.

Consequently, the best model was selected based on the results as illustrated in Section 5 and used as the primary LLM for subsequent experiments, which include evaluations on retrieval-augmented generation, contextual retrieval techniques on more advanced examinations, as well as the evaluation of our proposed interpretability framework.

4.4. Knowledge Augmented Retrieval Strategies

Although zero-shot prompting offers a baseline for evaluating LLM reasoning, real-world LLM deployments often benefit from external knowledge support [33,34,35]. In this section, we explore two external augmentation strategies—namely, retrieval-augmented generation [36] and contextual retrieval (https://www.anthropic.com/engineering/contextual-retrieval) (accessed on 4 April 2026)—to assess their effectiveness in enhancing LLM performance on complex calculus problems requiring domain-specific grounding.

4.4.1. Retrieval-Augmented Generation

Retrieval-augmented generation [36] is a technique that enhances the input of large language models by incorporating retrieved documents or passages from an external knowledge base. This approach enables a model to base its responses on more accurate or domain-specific information [37]. Unlike fine tuning, which necessitates retraining the model, RAG dynamically integrates the relevant context during inference, allowing general-purpose LLMs to specialize in specific domains with minimal engineering effort. In the realm of mathematics education, previous research indicates that RAG can enhance factual accuracy [38,39,40,41], reduce hallucinations [42,43], and facilitate multi-step reasoning [44,45] by providing canonical definitions, formulas, and examples [37,46,47,48]. To assess the effectiveness of retrieval-augmented generation in enhancing reasoning capabilities in large language model-based examinations in calculus, we developed a custom RAG pipeline specifically designed for calculus problem solving. This system processes a reference calculus textbook and course notes by extracting, filtering, and embedding semantically relevant content. Key mathematical topics, such as derivatives, integrals, and optimization, were employed as keywords to filter pertinent textbook segments. The filtered content was divided into overlapping chunks and stored in a Chroma vector database [49] using nomic-embed-text [50] embeddings.

4.4.2. Contextual Retrieval

In contrast to the conventional retrieval-augmented generation approach, we also investigate Anthropic’s Contextual Retrieval framework (https://www.anthropic.com/engineering/contextual-retrieval) (accessed on 4 April 2026), which incorporates external knowledge directly into a large language model’s prompt window by utilizing meticulously curated document contexts without necessitating modifications to the model architecture or the use of a separate retriever pipeline.

Unlike the standard RAG methodology, which generally functions in two stages as a retriever that selects the top k passages from a corpus and a generator that bases its output on those selections, Anthropic’s method integrates multiple pertinent documents directly into the context prompt, thereby enabling the LLM to discern and employ the most beneficial information in a single operation. The advantages over Standard RAG can be summarized as follows:

• Unified architecture: Contextual retrieval does not require a separate dense retriever (e.g., FAISS), simplifying the system and reducing latency.
• Instructional alignment: Since the same LLM handles both retrieval selection and answer generation, the method is more aligned with user intent, particularly in educational domains where nuanced understanding is key.
• Improved fluency and coherence: The model can better maintain consistency across answer reasoning, because it directly “sees” and “reasons” over the inserted passages in a single forward pass.
• Minimal hallucination risk: By grounding responses in the explicitly provided context, factual accuracy can improve without reliance on parametric memory.
• Lower infrastructure overhead: No need to deploy, train, or maintain separate retrieval systems or indexes.

For each question, textbook content and notes were embedded similarly for both RAG and contextual retrieval for consistent comparison. This approach allows us to evaluate whether inline and external knowledge-grounded reasoning enhances accuracy or consistency compared to both the zero-shot baseline (without retrieval) and traditional retrieval-augmented generation, thereby allowing us to assess the relative efficacy of retrieval strategies in enhancing LLM reasoning on complex, structured problems, such as undergraduate calculus examinations.

To evaluate our interpretability framework, we compared and evaluated three model configurations:

• Baseline: zero-shot prompting without retrieval.
• RAG: retrieval-augmented generation with a vector store of textbook and lecture notes.
• Contextual Retrieval: dynamically selected semantically aligned passages as external context from textbook and lecture notes.

4.5. Implementation and Technical Environment

All experiments were conducted locally without access to commercial APIs, which is consistent with the reproducibility constraints described in Section 5. The primary model, Gemma 3, was served via Ollama running on a local machine. The RAG pipeline was implemented in Python 3.13 using LangChain for orchestration, ChromaDB as the vector store, and nomic-embed-text (via Ollama) for embedding generation. Ablation divergence scoring used TF-IDF vectorization and cosine similarity from scikit-learn as well as token-level edit distance from the python-Levenshtein library. Reasoning step segmentation was performed using rule-based parsing with spaCy. The model temperature was set to 0 across all runs to ensure deterministic outputs, which is consistent with the consistency evaluation reported in Section 5.3. The complete framework implementation is publicly available at [51].

5. Results and Analysis

In this section, we conduct a comprehensive evaluation of the mathematical capabilities of the model using selected calculus examinations. Faculty members assessed the model’s responses employing standardized faculty-developed rubrics to determine the accuracy of the Gemma 3 model and compare its performance with the historical average of actual student scores. This baseline analysis offers an essential context for identifying the model’s strengths and weaknesses, thereby establishing a foundation for more in-depth evaluations of advanced retrieval strategies, consistency of model output across iterations, and understanding its limitations.

5.1. Baseline Model Selection

Before applying our interpretability framework, we first establish a quantitative baseline of model performance across calculus exams. This section reports the overall rubric-based scores and error distributions, enabling a high-level understanding of how well the model performs relative to human learners, and setting the stage for deeper structural analysis in subsequent sections.

This section focuses on selecting the baseline model based on mathematical answers, consistency, and the formatting of outputs. We evaluated three models—namely, Gemma 3 [52], LLaMA 2 [20], and LLaVA-7B [53]—keeping in mind our computational capacity restrictions, which did not allow us to run larger models and literature surveys, as discussed in Section 2. Exam questions were presented to each model in a sequential manner with the system role, as illustrated in Section 4.3, and their responses were independently assessed using the official exam rubric applied to human students.

Owing to computational resource constraints and the need for reproducibility on local systems, our evaluation is limited to models that are lightweight and can be run without access to commercial APIs. We selected Gemma 3, LLaMA 2, and LLaVA-7B as representative small-scale models that reflect open-access trends in educational settings and are grounded in our literature survey. Although this paper evaluates a subset of small-scale open-access LLMs, the interpretability framework is designed to be model-agnostic in its architecture; however, the empirical results presented here are derived from a single model (Gemma 3), and metric values should not be generalized to other LLMs without independent evaluation. The goal of this paper is not to rank or optimize model performance but rather to demonstrate the utility of a model-agnostic interpretability framework. The reasoning flow and sensitivity metrics introduced in this paper are designed to be applicable across LLM architectures, including frontier models such as GPT-4 and Claude. Although specific metric values (e.g., robustness or phrase sensitivity) naturally vary depending on the underlying model’s capabilities, the core interpretability methodology of semantic reasoning decomposition and input ablation remains consistent. Future work may explore how these metrics differ across model families, but the framework itself is fundamentally adaptable to any LLM that generates multi-step mathematical solutions.

Among the evaluated models, Gemma 3 exhibited the highest levels of consistency, interpretability, and mathematical prowess. It strictly adhered to formatting instructions, presenting answers in a plain-text notation suitable for human assessment. Furthermore, it provides logically coherent and computationally precise responses across a diverse array of problem types, including domain and range determination, function inversion, limit evaluation, and derivative computation using first principles. In contrast, LLaMA 2 frequently violated prompt constraints by reverting to LaTeX or symbolic markup and committed several conceptual and arithmetic errors, such as incorrect asymptotic analysis and flawed limit logic. LLaVA-7B demonstrated the weakest performance overall with incomplete or incorrect responses and a refusal to engage with certain questions. In particular, neither LLaMA 2 nor LLaVA-7B fully complied with the formatting requirements, limiting their applicability in real-world educational contexts. In summary, Gemma 3 demonstrated superior performance in terms of both accuracy and usability, achieved the highest examination score, and exhibited the most promising potential for application in AI-driven tutoring or automated assessment. Although Gemma 3 did not utilize visual information from graph-based questions, its symbolic reasoning remained logically valid. This indicates that the model is capable of performing effectively under text-only conditions, which is pertinent for most current large-language-model deployment settings.

5.2. Baseline Comparison with Student Performance

We note that only aggregate class-level averages were available for comparison; no individual student records were accessed and no inferential statistical testing was conducted. Comparative observations throughout this section should therefore be interpreted as descriptive rather than statistically validated differences.

To evaluate the effectiveness of large language models such as Gemma within the realm of formal mathematics education, we conducted an assessment utilizing actual undergraduate calculus examinations. Our objective was to compare the performance of LLMs with that of human learners in a progressively challenging academic curriculum as discussed in Section 4.1.

We compared our LLM performance with comprehensive exam scores from undergraduate students who completed standardized versions of the Calculus I, II, and III exams as part of their curricula. These scores served as a baseline for evaluating the performance of the Gemma. Each course adhered to a conventional structure, comprising two midterm exams and a comprehensive final exam, with increasing levels of conceptual complexity and symbolic abstraction from Calculus I to III, as illustrated in Section 1.2. The exams incorporated a combination of procedural, conceptual, and applied problem-solving tasks, which is consistent with widely adopted undergraduate curricula. To evaluate the language model, the Gemma was prompted once per exam using structured inputs that mirrored the wording, format, and structure of the original student assessments. The model’s responses were graded on a per-question basis using the same rubrics applied to student submissions, enabling the computation of both total exam scores and detailed breakdowns by question type and topic (reported in Section 7 by mathematical faculty experts).

Table 6. Course exam performance comparison.

Course Exam	Gemma Score	Gemma Average (%)	Class Average (%)
Calculus I
Exam 1	44/70	62.9	88.8
Exam 2	48/70	94.2	82.1
Final Exam	74/100	74.0	73.0
Calculus II
Exam 1	49/60	80.0	89.6
Exam 2	52/60	86.6	85.2
Final Exam	72/90	80.0	72.9
Calculus III
Exam 1	84/100	84.0	78.0
Exam 2	74/100	74.0	80.3
Final Exam	112/150	74.6	71.3

presents a summary of Gemma’s performance across the three calculus courses in comparison with the average class scores. Each row details the model’s score on a specific examination with the percentage accuracy calculated and compared against the class average. This enabled an analysis of not only Gemma’s ability to successfully complete entire examinations but also the extent to which its reasoning accuracy and consistency were comparable to human performance across varying levels of mathematical complexity. The primary contribution of this paper lies in evaluating the interpretability of large language models in mathematical problem solving. To ensure consistency, all exam outputs generated by the LLM were assessed by the same expert rather than obtaining feedback from multiple faculty members. While the use of a single assessor limits the ability to report inter-rater reliability, the criterion-referenced rubric with explicit point allocations minimizes subjective judgment; future evaluations of this framework should incorporate multiple independent assessors and report inter-rater agreement statistics such as Cohen’s kappa.

In Calculus I, Gemma performed well overall, particularly in Exam 2 (94.2%), exceeding the class average by more than 12 percentage points. The final exam score closely matched the class average (74.0% vs. 73.0%), demonstrating retention across units. In Calculus II, which introduces techniques of integration, sequences and series, and parametric forms, Gemma remained competitive, exceeding the class average in the final exam (80.0% vs. 72.9%) but scoring below the class average in Exam 1. In Calculus III, which covers multivariate calculus and vector analysis, the performance decreased slightly. While Gemma started strongly in Exam 1 (84.0%), it struggled in Exam 2 (74.0%) compared to the class average (80.3%). However, it recovered in the final examination (74.6% vs. 71.3%). The findings indicate that Gemma exhibits significant proficiency in addressing structured calculus problems, particularly in early to intermediate topics. The model’s stronger performance in later exams may reflect familiarity with recurring mathematical structures or better alignment with prompt patterns rather than a cumulative understanding. However, variability in performance, especially in Calculus III, underscores the persistent challenges that LLMs face with spatial reasoning, multi-step symbolic derivation, and tasks necessitating strong conceptual abstraction. This performance benchmark adds to the expanding body of evidence that LLMs can function as additional educational tools in STEM fields, although they require careful evaluation regarding scope, consistency, and domain transfer as task complexity escalates.

5.3. Baseline Consistency Across Multiple Runs

Additionally, for a comprehensive evaluation of Gemma outputs, we evaluated the model in terms of consistency through multiple runs of the same Calculus I Exam I, which were graded by the same mathematics faculty expert in order to keep the grading input consistent. Across five runs, the outputs demonstrated consistency in formatting and solution, received the same marks (59/70), and maintained formatting as well as mathematical notation usage consistency.

5.4. RAG and Contextual Retrieval Results

In this section, we benchmark the performance of the baseline LLM and the investigated augmentation strategies with real student performance across three exams from three different levels of undergraduate calclus examinations. The choice of exams in this experiment is motivated by the need for understanding performance across increasing levels of calculus difficulty as well as the low scores observed for the baseline model on Exam I across all three courses in Table 6 and discussed in Section 5.2.

The performance trends, as illustrated in

Table 7. Performance comparison across calculus exams.

Exam	Model Configuration	Score %
Calculus I Exam I	Human Student Average	88.8
	Gemma (Baseline)	62.9
	Gemma + RAG	70.0
	Gemma + Contextual Retrieval	65.7
Calculus II Exam I	Human Student Average	89.6
	Gemma (Baseline)	80.0
	Gemma + RAG	81.7
	Gemma + Contextual Retrieval	75
Calculus III Exam II	Human Student Average	80.3
	Gemma (Baseline)	74.0
	Gemma + RAG	52.0
	Gemma + Contextual Retrieval	64.0

, reveal the nuanced effects of retrieval augmentation. In Calculus I Exam I, the baseline model achieved a score of 62.9%, which improved with both augmentation strategies by 70.0% using RAG and 65.7% with contextual retrieval. Although still trailing the human student average of 88.8%, this gain suggests that retrieval mechanisms can support factual grounding and improve accuracy when question–context alignment is strong. With contextual retrieval, our expert analysis identified the usage of incorrect functions, fractions, and square roots over RAG. A similar pattern is observed in Calculus II Exam I, where the baseline scored 80.0% and RAG and contextual retrieval achieved 81.7% and 75.0%, respectively, compared to the student average of 89.6%. The marginal improvement by RAG indicates potential benefits in less abstract problem types, while the drop in contextual retrieval is primarily due to challenges in working with fractions and missing answers to questions. In contrast, the performance in Calculus III Exam II demonstrated the limits of augmentation. Despite being the most conceptually challenging examination, the baseline achieved a reasonable score of 74.0%. However, RAG significantly underperformed, yielding just 52.0%, whereas contextual retrieval fared slightly better at 64.0%. Both fell short of the student average of 80.3%. These findings suggest that in higher-difficulty domains, improperly aligned or noisy retrieval content may introduce distractions or inconsistencies that degrade performance. This degradation is attributable to retrieval noise at higher abstraction levels: in vector calculus and multivariable analysis, semantic similarity between exam question tokens and textbook chunk representations is lower than in procedural Calculus I topics, causing the retriever to surface tangentially related content that disrupts rather than supports the model’s reasoning chain.

Interestingly, faculty observations reported that while contextual retrieval consistently improved mathematical notation fidelity and formatting clarity, it struggled with fractions and incorrect function usage. This points to the value of contextual retrieval in enhancing the response structure even when it does not directly boost answer correctness. Overall, these results emphasize that while retrieval-based augmentation can support performance gains in certain cases, its efficacy is highly contingent on the precision of context alignment. In high-complexity mathematical tasks, improper context may obscure logical flow or derail reasoning chains, making retrieval strategies both an opportunity and a liability.

6. Discussion: Interpreting Problem Solving Behavior of LLMs

Having established the reasoning flow framework and described our experimental setup in Section 3 and Section 4, respectively, we now present the principal findings derived from applying this framework to model-generated solutions across Calculus I–III examinations. This section constitutes the discussion of our results. Section 6.2 presents quantitative findings, Section 6.3 provides a qualitative interpretation of selected cases, and the overall conclusions of the paper are presented in Section 8. This analysis extends beyond the mere correctness of LLMs to examine the conceptual structure, procedural validity, and logical dependencies inherent in each solution response. By deconstructing the model’s reasoning into labeled operations and associated concepts, we can identify both strengths, such as alignment with expected steps, and weaknesses, including hallucinated transitions and conceptual mismatches. A comprehensive analysis report generated for each question is presented below.

6.1. Reasoning Flow Analysis Results

This section presents a representative sample of the reasoning flow analysis applied to one examination question. The framework generates a detailed report for each LLM-generated response, decomposing it into operations, concepts, and ablation impact scores. The example below is drawn from a Calculus III question on gradient and directional derivative computation, which is representative of the multi-step, multi-concept problems evaluated in this paper.

Having established the overall performance trends, we turn to the structural decomposition of the model responses using our reasoning flow analysis. This section explores how the model constructs multi-step solutions, the complexity of operations performed, and the concepts it invokes, providing a window into its internal problem-solving process.

LLM Attention Analysis Report: Gradient and Directional Derivative Problem
Timestamp: 20250724_182112	Model:gemma3:latest
Question Analyzed: For $f(x,y)=x^{2}y$, (a) Find $\nabla f$ (b) Find the directional derivative in the direction of $\overrightarrow{w}=-\widehat{𝚤}+\widehat{𝚥}$ at $(-1,4)$ (c) Find the direction of the most rapid increase at $(-1,4)$ (d) Find the maximum rate of change at $(-1,4)$
Ablation Analysis Results: Important phrase: (-1, 4)    Impact: 0.630 Important phrase: $\nabla f$    Impact: 0.562 Important phrase: directional    Impact: 0.477 Important phrase: $f(x,y)$    Impact: 0.379 Important phrase: maximum    Impact: 0.373
Reasoning Flow Analysis: Total steps: 7 Complexity score: 23 Reasoning patterns: substitution, differentiation, differentiation, solving, differentiation, evaluation, solving

Detailed Reasoning Steps: 1. Start solving the calculus problem step by step. Operations: substitution 2. (a) Find $\nabla f$: the gradient of f, using partial derivatives. Operations: derivative, evaluation, solving Concepts: gradient, partial derivatives 3. (b) Compute the directional derivative in the direction of $\overrightarrow{w}=-\widehat{𝚤}+\widehat{𝚥}$ at $(-1,4)$. Operations: derivative, solving Concepts: gradient, directional derivative 4. Compute $\nabla f(-1,4)=\langle 2(-1)\left(4\right),{(-1)}^2\rangle =\langle -8,1\rangle$. Operations: solving Concepts: gradient 5. Find the unit vector in the direction of $\overrightarrow{w}=-\widehat{𝚤}+\widehat{𝚥}$, where $\parallel \overrightarrow{w}\parallel =\sqrt{2}$, so the unit vector is $\langle -{\displaystyle \frac{1}{\sqrt{2}}},{\displaystyle \frac{1}{\sqrt{2}}}\rangle$. Operations: derivative, evaluation, solving Concepts: gradient, directional derivative 6. (c) Direction of most rapid increase at $(-1,4)$ is along $\nabla f=\langle -8,1\rangle$. Operations: evaluation, solving Concepts: gradient 7. (d) Maximum rate of change at $(-1,4)$ is $\parallel \nabla f\parallel =\sqrt{{(-8)}^2+1^2}=\sqrt{65}$. Operations: solving Concepts: gradient, maximum

To systematically interpret the internal problem-solving mechanisms of LLMs, we employ a reasoning flow analysis that dissects each model-generated response into a structured sequence of mathematical operations and conceptual references, as illustrated in Section 3.1. This detailed tracing enabled us to assess whether the solution was aligned with logical expectations, appropriately applied domain principles, and maintained semantic coherence throughout the steps. This analysis allows us to assess whether the model’s stepwise outputs align with expected solution patterns and where they diverge from canonical solution paths used in instruction. In the interest of conciseness, we present a single-sample reasoning flow report for each question across all the examinations reviewed.

6.2. Quantitative Evaluation of the Interpretability Framework

The effectiveness of the interpretability framework was assessed through structured question-level metrics aggregated across each exam.

Table 8. Comparison of LLM performance metrics across courses, exams, and model configurations.

Course	Exam	Model	Robustness	Complexity	Step Count	Phrase Sensitivity
Calc I	Exam I	B	0.726	18.7	6.1	0.488
		R	0.671	17.9	5.9	0.503
		C	0.752	19.3	6.4	0.445
	Exam II	B	0.682	19.2	6.3	0.422
		R	0.608	17.6	6.0	0.468
		C	0.710	18.9	6.2	0.399
	Exam III	B	0.719	18.4	6.1	0.451
		R	0.702	18.9	6.2	0.465
		C	0.731	19.1	6.3	0.442
Calc II	Exam I	B	0.681	18.3	6.1	0.451
		R	0.706	18.7	6.4	0.432
		C	0.699	18.9	6.3	0.428
	Exam II	B	0.702	18.8	6.2	0.417
		R	0.729	19.2	6.5	0.430
		C	0.709	19.1	6.4	0.430
	Exam III	B	0.693	18.6	6.2	0.447
		R	0.723	19.0	6.3	0.443
		C	0.686	18.8	6.1	0.442
Calc III	Exam I	B	0.682	17.18	5.8	0.459
		R	0.721	17.51	5.9	0.471
		C	0.736	17.36	5.9	0.469
	Exam II	B	0.693	17.82	6.0	0.465
		R	0.732	17.91	6.1	0.486
		C	0.757	17.79	6.1	0.494
	Exam III	B	0.684	16.95	5.7	0.460
		R	0.716	17.22	5.9	0.476
		C	0.692	17.20	5.9	0.476

presents the disaggregated results by course, exam, and model variants—namely, baseline (B), RAG (R), and contextual retrieval (C). These metrics quantify four dimensions of interpretability: robustness, complexity, reasoning step count, and phrase sensitivity.

In Calculus I, all three exams yielded analyzable outputs across the baseline (B), retrieval-augmented generation (R), and contextual retrieval (C) models. Exam I exhibited strong overall performance with contextual model (C) achieving the highest metrics: robustness of $0.752$, complexity of $19.3$, and an average of $6.4$ reasoning steps. Phrase sensitivity was moderate ($0.445$), indicating a dependence on specific phrasing. The baseline and RAG models also performed well, although this was consistently below the contextual configuration across most metrics. Exam II followed a similar trend but with slightly reduced values across all dimensions, suggesting a modest decline in reasoning depth and robustness. The contextual model again led in performance with a robustness of $0.710$, complexity of $18.9$, and step count of $6.2$, although phrase sensitivity remained non-negligible ($0.399$). Exam III showed comparable trends to Exam I with the contextual model maintaining its lead: a robustness of $0.731$, complexity of $19.1$, and $6.3$ average steps. Phrase sensitivity ($0.442$) was in line with the results of previous studies. Interestingly, the baseline model for Exam III also performed competitively (robustness $0.719$, complexity $18.4$, step count $6.1$), suggesting an overall stability in LLM reasoning quality across various calculus problem sets. In Calculus II, all three examinations showed consistent metric values across all model variants. The RAG model (R) generally outperformed the baseline and contextual retrieval models in terms of robustness and reasoning step count, peaking at $0.729$ and $6.5$ in Exam II. The contextual retrieval model (C) showed strong and stable performance with lower phrase sensitivity, potentially indicating greater resilience to surface-level linguistic variation. Calculus III examinations revealed significant distinctions among the model variants. Contextual retrieval model (C) demonstrated the highest robustness in Exam II ($0.757$), which was accompanied by elevated step counts ($6.1$) and phrase sensitivity ($0.494$), indicating a more profound yet somewhat phrase-sensitive reasoning process. The RAG model (R) consistently performed well across all three examinations with robustness and complexity metrics generally exceeding those of the baseline model.

In summary, the results demonstrate that retrieval augmented (R) and (C) models improve over the baseline in several key interpretability dimensions, particularly in terms of robustness and reasoning depth. However, neither variant consistently reduced phrase sensitivity, suggesting that improved reasoning comes at the cost of continued reliance on input phrasing. These findings highlight the need for a further refinement of retrieval strategies to balance structured reasoning with linguistic flexibility in large language models applied to formal mathematical domains.

6.3. Qualitative Case Studies: Successful and Failed Reasoning Patterns

To complement the quantitative evaluation in Section 6.2, we present two qualitative case studies drawn from the experimental data that illustrate contrasting reasoning outcomes. These examples ground the abstract metric values in concrete model behavior and illuminate the diagnostic value of the interpretability framework.

Case 1—Success: Gradient and Directional Derivative (Calculus III). The gradient and directional derivative problem reported in the reasoning flow report (Section 3.1) represents a well-handled case. The question asked the model to compute (a) $\nabla f$ for $f(x,y)=x^{2}y$, (b) the directional derivative at $(-1,4)$ in the direction $\overrightarrow{w}=-\widehat{𝚤}+\widehat{𝚥}$, (c) the direction of most rapid increase, and (d) the maximum rate of change at $(-1,4)$. The model produced a logically coherent seven-step solution with operations sequenced as follows: differentiation → differentiation → evaluation → normalization → dot product → identification → computation. It correctly computed $\nabla f(-1,4)=\langle -8,1\rangle$, normalized the direction vector to $\langle -\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}\rangle$, and returned the maximum rate of change as $\parallel \nabla f\parallel =\sqrt{65}$. The framework assigned a complexity score of 23 and a phrase sensitivity of 0.630, with the coordinate reference $(-1,4)$ identified as the highest-impact phrase (impact 0.630), which was followed by $\nabla f$ (0.562). The high phrase sensitivity for the coordinate reference is expected: removing the evaluation point forces the model to produce a general symbolic answer rather than a numerically grounded one, causing substantial output divergence. Importantly, the high complexity score (23) reflects the multi-concept depth of the problem including gradient computation, unit vector normalization, and directional derivative invoked in sequence, yet the model navigated these steps without conceptual gaps. This case illustrates the framework’s ability to confirm valid multi-step reasoning and to identify which prompt elements anchor the solution.

Case 2—Failure: Symbolic Simplification Errors (Calculus I Exam 1). In contrast, an expert assessment of Calculus I Exam 1 recorded the lowest baseline score across all exams (62.9% versus a student average of 88.8%, Table 6), documenting two concrete and recurring failure modes. First, the model failed to evaluate $lne=1$ correctly, instead treating the expression as unreduced or assigning an incorrect numerical value. Second, the model consistently misrepresented composite expressions: for example, $\sqrt{x+2}$ was parsed and output as $\sqrt{x}+2$, decomposing a composite argument into a sum of separate terms. The expert assessor noted that students enrolled in the same course did not make these errors. From the framework’s perspective, both failures manifest at the evaluation step of the reasoning trace: the model reaches the correct procedural structure (identifying that simplification or substitution is required) but produces a symbolically incorrect output at the terminal step. This is precisely the failure mode the framework is designed to surface — syntactically fluent yet conceptually flawed solutions, where final answer correctness diverges from the validity of intermediate steps. The aggregate metrics for Calculus I Exam 1 baseline (robustness: 0.726, complexity: 18.7, phrase sensitivity: 0.488, Table 8) indicate moderate phrase sensitivity that is consistent with the model treating mathematical expressions as surface-level linguistic tokens rather than structured symbolic objects. These cases together demonstrate that the interpretability framework provides diagnostic traction beyond score-level evaluation, enabling the identification of both the step at which reasoning fails and the input features that most influence model behavior.

7. Pedagogy and Future Work

7.1. Educational Theory Linkage: Cognitive and Metacognitive Alignment

Our interpretability framework aligns with foundational principles in educational theory, particularly cognitive load theory [32], Bloom’s taxonomy [31], and metacognitive scaffolding [54]. By decomposing LLM-generated calculus solutions into semantically labeled steps (e.g., operation type, concept tag, and cognitive complexity), our framework surfaces not only what the model produces but also how it arrives at its conclusions. This mirrors procedural and conceptual knowledge acquisition in human learners and aligns with a revised version of Bloom’s taxonomy [55], particularly at the Apply and Analyze levels. Furthermore, the structured reasoning trace and ablation-based sensitivity analysis support metacognitive reflection, which is a core component of constructivist learning. Students and educators can interrogate the reasoning process, identify brittle or incorrect logic, and calibrate trust in AI outputs, thus moving beyond passive consumption to active, reflective learning. Our framework enables an analysis of LLM outputs in terms of instructional learning objectives, offering a foundation for formative feedback, self-assessment, and adaptive instruction grounded in cognitive science.

7.2. Expert Feedback on LLM Solutions

The rapid advancement of large language models has sparked interest in their potential applications across various fields, including mathematics education. While these models have demonstrated impressive capabilities in natural language processing and generation, their performance in solving mathematical problems remains an area of active research and scrutiny.

Our expert observation reveals that while many of the problems were handled correctly, several LLM-generated solutions were difficult to interpret owing to conceptual errors, computational mistakes, and occasional issues with mathematical notation. Graphical limit problems were particularly challenging for LLMs, and solutions that required sketches or graphs were frequently incomplete or missing. Vector cross-products were not represented correctly in tabular format, and most partial derivative problems were solved incorrectly with errors in applying the chain rule. Simple arithmetic mistakes, such as the incorrect addition or subtraction of integers, were also observed and often led to incorrect final answers. Interestingly, some of these mistakes resemble common student errors, such as arithmetic miscalculations or incorrect application of the chain rule. However, students typically handle power, root, fractions, logarithmic, and exponential functions more reliably than LLMs. For instance, the LLM failed to compute $lne=1$ correctly and sometimes misrepresented expressions such as $\sqrt{x+2}$ as $\sqrt{x}+2$, which students are generally more careful about. These issues highlight the need for improved mathematical reasoning, symbolic accuracy, and verification steps in LLM solutions.

7.3. Pedagogical Implications

The proposed interpretability framework offers several actionable insights into educational deployment. Beyond research contribution, our proposed framework has direct pedagogical utility for both learners and educators. Firstly, from the learner’s perspective, visualizing step-by-step reasoning enables a cognitive unpacking of complex calculus solutions. Students can compare their own solution paths with those generated by the model, identify divergences, and reflect on errors, thereby supporting metacognitive skill development. Secondly, as AI models such as large language models are increasingly integrated into intelligent tutoring systems and educational platforms, our framework enables educators to comprehend LLM outputs and reasoning regardless of their prior experience with AI. Familiarity with this framework will aid in the design of assignments where AI assistance is either permitted or prohibited while also ensuring alignment with curricular learning objectives. Third, the metrics of reasoning complexity and robustness can inform curriculum designers and tutors on where LLMs struggle the most, potentially guiding the integration of AI-assisted instruction in calculus coursework.

Additionally, this framework can be employed to instruct students on the proper utilization of AI by verifying procedural steps. Our phrase sensitivity and robustness metrics provide insight into how problem formulation affects model behavior. Educators can use this information to refine question prompts while students gain experience interpreting and rephrasing problems, which is an essential component of AI literacy and responsible usage in academic settings.

To illustrate these implications concretely, we present three representative use-case scenarios:

• Instructor scenario: Before distributing a set of LLM-generated worked examples on the chain rule, an instructor runs the framework over the candidate solution set. Questions with a complexity score above 20 are flagged as potentially too dense for students to follow; questions with a phrase sensitivity score above 0.5 are flagged as likely to produce inconsistent guidance if students rephrase them, which is a common behavior. The instructor can then review or replace flagged items before sharing them with the class.
• Tutoring system scenario: An intelligent tutoring system displays the reasoning flow graph alongside the LLM-generated answer. Students can trace the model’s step-by-step operations and identify the exact step where their own solution path diverges, supporting targeted self-correction and metacognitive reflection without requiring direct instructor intervention.
• AI literacy scenario: Students learning to critically evaluate AI outputs use the phrase sensitivity scores to observe concretely how a minor rephrasing of the same calculus problem changes the model’s response. This hands-on exercise builds awareness of LLM brittleness and reinforces responsible AI use as part of STEM education.

7.4. Limitations and Future Deployment

Although the proposed interpretability framework provides structured insights into the approach of large language models to mathematical problem solving, several limitations warrant discussion. In certain instances, LLMs may produce high-scoring outputs that nonetheless contain subtle conceptual errors, potentially misleading learners who lack the expertise to identify them. Furthermore, our analysis indicates that LLMs tend to excel in procedural or computational tasks but often encounter difficulties with deeper conceptual reasoning, affirming existing research on LLM reasoning. This observation suggests the necessity for hybrid models or ensemble strategies that incorporate symbolic reasoning or retrieval-augmented grounding, particularly for questions requiring higher-order reasoning. The variability in LLM responses attributable to factors such as prompt phrasing, stochastic decoding, or model drift also presents a challenge in their deployment in education. Future iterations of the framework should incorporate confidence-aware modeling or multi-sample consistency checks to enhance robustness and reliability.

Several additional limitations warrant explicit acknowledgment. First, the empirical evaluation is restricted to a single small-scale open-access model (Gemma 3); widely deployed frontier models such as GPT-4, ChatGPT, and Claude exhibit substantially different reasoning profiles and were not evaluated due to API cost constraints. Second, the student performance data were derived from a single US institution, and cross-institutional evaluation would be required to support broader generalizability claims. Third, the ablation procedure masks one element at a time; the sensitivity of the framework metrics to alternative ablation strategies (e.g., phrase-level vs. sentence-level masking, or simultaneous multi-element perturbation) has not been evaluated. Fourth, the grading was conducted by a single mathematics faculty assessor; while the criterion-referenced rubric limits subjectivity, future work should report inter-rater agreement to strengthen evaluation validity.

As future work, we envision deploying this interpretability framework within a student facing an intelligent tutoring system. The interpretability components such as reasoning complexity or input salience would be integrated into adaptive learning environments, automatically identifying questions of appropriate difficulty or flagging LLM-generated solutions that warrant human review. In such a system, both LLM generated step-by-step explanations and their accompanying reasoning flow diagrams can aid learners in diagnosing misunderstandings, reflecting on problem-solving strategies, and transferring knowledge across topics. These interpretable artifacts can support formative assessments and scaffold students’ metacognitive awareness during learning. However, caution should be exercised in educational settings. An overreliance on LLM-generated solutions may give students a false sense of understanding and weaken foundational skill development. To mitigate these risks, future systems should be designed based on the following principles.

• Assistive, not Replacive: LLMs should augment, rather than substitute, core learning processes.
• Failure Aware: Educators must be cognizant of common LLM failure modes, including hallucinated steps, misinterpretations, and omitted justifications.
• Transparent and Interactive: Interfaces should offer traceable reasoning paths and interactive feedback mechanisms that promote active engagement and critical reflection.

Ultimately, the value of LLMs in education will depend not only on their raw capabilities but also on how thoughtfully they are embedded into pedagogical workflows that center on human learning, agency, and oversight.

8. Conclusions

As large language models become increasingly integrated into educational settings, there is a critical need for tools that go beyond accuracy to explain how these models reason, fail, and interact with student input. This paper presents a novel interpretability framework specifically designed for the analysis of large language models in the context of undergraduate mathematics education. Departing from traditional correctness-based evaluations, our approach dissects model-generated solutions into semantically labeled reasoning chains and evaluates input sensitivity through structured ablations. When applied to real-world calculus examinations, this dual-pronged methodology provides detailed insights into model reasoning behavior, identifies breakdowns in logical structure, and reveals the influence of input phrasing on output. Our empirical analysis demonstrated a significant variation in reasoning robustness across different question types, retrieval strategies, and linguistic perturbations. In particular, contextual retrieval consistently enhances both interpretability and robustness metrics, highlighting the importance of semantically aligned contexts in improving model performance in mathematically rigorous domains. By providing a transparent, step-level view of model behavior, our framework offers a novel perspective for diagnosing these issues and assessing the pedagogical readiness of LLMs. As future work, we plan to extend this framework to other STEM disciplines and integrate it into interactive, instructor-facing tools for real-time feedback, auditing, and curriculum alignment. This paper establishes a foundation for developing trustworthy, interpretable, and educationally aligned AI systems capable of supporting learners and educators in high-stakes academic contexts.