Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

Abstract

In the realm of mathematics education, self-explanation stands as a crucial learning mechanism, allowing learners to articulate their comprehension of intricate mathematical concepts and strategies. As digital learning platforms grow in prominence, there are mounting opportunities to collect and utilize mathematical self-explanations. However, these opportunities are met with challenges in automated evaluation. Automatic scoring of mathematical self-explanations is crucial for preprocessing tasks, including the categorization of learner responses, identification of common misconceptions, and the creation of tailored feedback and model solutions. Nevertheless, this task is hindered by the dearth of ample sample sets. Our research introduces a semi-supervised technique using the large language model (LLM), specifically its Japanese variant, to enrich datasets for the automated scoring of mathematical self-explanations. We rigorously evaluated the quality of self-explanations across five datasets, ranging from human-evaluated originals to ones devoid of original content. Our results show that combining LLM-based explanations with mathematical material significantly improves the model’s accuracy. Interestingly, there is an optimal limit to how many synthetic self-explanation data can benefit the system. Exceeding this limit does not further improve outcomes. This study thus highlights the need for careful consideration when integrating synthetic data into solutions, especially within the mathematics discipline.

1. Introduction

The emergence of digital learning platforms has opened a plethora of opportunities for researchers to investigate and comprehend learning behaviors through abundant system interaction data [1]. A notable area of interest among various learning facets is self-explanation, identified as a robust active learning technique. This strategy has been particularly effective in bolstering comprehension in subjects like mathematics [1,2,3]. Self-explanation can be described as a mechanism where learners articulate explanations, elucidate concepts, expand on methods, and immerse in problem-solving to enhance their grasp and absorb fresh insights [4,5].

With the proliferation of computer-driven learning platforms, self-explanation has gained renewed attention and application. Contemporary learning innovations place a premium on self-explanation by crafting intuitive interfaces, creating assessment models rooted in self-explanation behaviors, and formulating tactics to extract profound self-explanations [6,7]. Tools like the one formulated by Crippen and Earl [8] highlight the centrality of self-explanation in methodical problem-solving. Ongoing studies persistently explore the versatile applications of self-explanation in education [9,10], like the adoption of template-driven self-explanations. Such templates equip learners with pre-set frameworks, serving as built-in guides to bolster their explanation processes [1,11,12].

Furthermore, the domain of self-explanation practices reaches beyond traditional boundaries. These methods involve understanding a concept and facilitating multiple educational tools such as feedback systems, crafting practice quiz responses, and generating valuable datasets for automated evaluations [10]. Within this framework, automated assessments play a pivotal role. By analyzing and interpreting self-explanations, both educators and automated systems can delve deeper into the intricacies of a learner’s thought patterns. This knowledge equips them with the capability to tailor educational strategies to better cater to individual needs. Such insights are crucial for tasks like classifying learner responses, which provide a clear view of their comprehension levels. Additionally, they allow for the easy detection of recurring mistakes or topics that consistently stump students [9,10].

However, devising a system capable of automatically grading diverse styles of self-explanation is challenging. A major concern is that self-explanation, due to its time-intensive nature [13], poses a feasibility issue for mass data collection. Additionally, crafting a quality self-explanation requires proficiency in both the specific subject matter and general writing [14,15]. Given these challenges, amassing a vast and diverse collection of self-explanation samples is demanding. This predicament further complicates the development of systems designed to aid learning through extensive sets of self-explanation examples.

To address these challenges and enhance automated scoring of self-explanations, we propose a semi-supervised approach that leverages the LLM. While popular LLMs such as OpenAI’s GPT-3 [16] are commonly employed in English language settings, due to the nature of the target problem we specifically focus on the Japanese variant of the model developed by CyberAgent [17], based on GPT-NeoX [18]. This approach aims to explore the model’s potential for generating self-explanation sentences, which will serve as the foundation for our regression models designed to predict self-explanation scores. By incorporating the semi-supervised methodology and leveraging advanced language models, we aim to improve the accuracy and effectiveness of auto-scoring in the self-explanation learning domain. Our research is anchored by two pivotal questions:

• RQ1: To what extent can the integration of self-explanations generated by the LLM Japanese model and mathematical material be used to enhance the accuracy of the predictive regression model for self-explanation scores?
• RQ2: What is the optimal quantity of artificially generated pseudo-self-explanation data required to effectively improve the predictive performance of the model?

These research questions provide insights into maximizing the utility of the LLM Japanese model and refining data augmentation techniques. The core findings from our research are twofold. First, we propose a strategy for advancing automated scoring in math education by synergizing LLM-generated content and mathematical material. Second, we highlight the ideal quantity of artificial self-explanation data for peak predictive accuracy.

2. Related Work

2.1. Automated Scoring of Self-Explanations: The Imperative for Rich Data

Self-explanation, widely recognized for amplifying learning outcomes in various fields, notably mathematics, has found its stride in the digital learning environment [1,4,5]. Emblematic tools like the iSTART tutoring system have been devised to foster and elevate learners’ grasp and performance [19]. Such platforms urge students to think critically, mirroring the analytical strategies of experts. Notably, the iSTART system utilizes natural language processing (NLP) in its pioneering approach to gauge and rate self-explanations, bolstering understanding across a gamut of texts.

The endeavor to automate the scoring of self-explanation quality has seen the integration of NLP tools and cutting-edge neural network architectures [20]. Techniques like latent semantic analysis (LSA) and recurrent neural network (RNN) interfaced with machine learning underscore the capabilities of automated systems, often outshining traditional manual evaluation in both effectiveness and efficiency [14,20,21,22,23,24]. Furthermore, semi-supervised learning techniques, which capitalize on abundant unlabeled data, have exhibited the potential to refine scoring accuracy [25]. Yet, the quest for more representative samples of self-explanations, especially in languages other than English, remains a prevailing challenge.

2.2. Augmenting Mathematical Self -Explanations Using Large Language Models

Consistent data shortages and imbalances have long impeded automated classification. Techniques like the synthetic minority oversampling technique (SMOTE), adaptive synthetic (ADASYN), and its derivatives aim to counteract these with synthetic data generation [26,27,28,29], but their effectiveness wanes for scant datasets. Although a variety of synthetic data tools and research exist [30,31,32,33,34,35], their applicability is often curtailed when facing complex domain-specific data, particularly in predicting student outcomes [35,36,37]. Navigating this landscape, large language models (LLMs) have made a compelling case as tools for text data augmentation. For instance, Dai et al.’s AugLLM [38] capitalizes on ChatLLM [16] to generate supplemental text, emphasizing the rising importance of LLMs in the realm of mathematical content [39].

However, in the context of mathematical education, the accurate auto-scoring of self-explanations presents stark challenges. Crafting these self-explanations is not just labor-intensive but demands expertise in both the subject and linguistic expression. Coupled with the lack of diverse and representative self-explanation samples, particularly in languages other than English, the task becomes even more daunting.

This paper aims to navigate these challenges by adopting a semi-supervised learning technique anchored by the Japanese variant of LLM. Our objective is twofold: to generate enriched self-explanation content and to amplify the accuracy of automated grading tailored for Japanese mathematical education. The emphasis here is a meticulous alignment with the intricacies of the Japanese language, paired with the nuances of mathematical challenges. In essence, our work hopes to bridge the existing data gap, presenting a more robust and linguistically attuned auto-scoring system.

3. Problem Setting: The Learning Task

In this chapter, we primarily introduce the original human-labeled data, which serves as a foundation for the subsequent pseudo-labeling of the unlabeled samples, thereby bolstering the training process. Prior to delving into methodological details, it is pivotal to define the distinct learning task under examination, which underpins our methodological foundation.

3.1. Collecting Self-Explanations

Self-explanations from learners are gathered via online platforms, as represented in Figure 1. The scope of this approach includes diverse mathematical challenges or quizzes that require written elaboration. We utilized the LEAF platform [40], composed of BookRoll (a digital reading application), and LAViEW (a tool for learning analytics), enabling students and teachers to monitor and reflect on their educational progress. This platform, having been successfully implemented in a Japanese secondary school for several years, captures handwritten responses in vector form, portraying the precise coordinates and velocity of each pen stroke.

The learners interacted with the quiz and recorded their answers using a tablet computer, employing a stylus for handwriting. As shown in Figure 1, the handwritten answer playback and self-explanation input process require students to input an explanation sentence after completing a step of their answer during playback.

3.2. Assessment of Self-Explanation Quality

Self-explanations in our study were assessed based on three main criteria: coherence, clarity, and relevance. Specifically, ‘coherence’ gauges the logical flow of the explanation, ‘clarity’ measures its understandability, and ‘relevance’ ensures the inclusion of all pertinent knowledge concepts and procedural elements. For consistent evaluation, we adapted the rubric and scoring definitions from Nakamoto et al. [10], as depicted in

Table 1. Rubrics and a sample answer of self-explanation in a quiz.

Numbers	Rubric	Sample Answer of Self-Explanations
Step 1	Be able to find the equation of a linear function from two points.	Substituting the y-coordinate of p into the equation of the line AC.
Step 2	Be able to find the equation of the line that bisects the area of a triangle.	Find the area of triangle ABC, then find the area of triangle OPC.
Step 3	Be able to represent a point on a straight-line using letters (P-coordinates).	With the line OC as the base, find the y-coordinate of p, which is the height. P’s coordinate is (t, −1/2t + 4).
Step 4	Be able to represent a point on a straight-line using letters (Q-coordinate).	Since the coordinates of P are (3,5/2), the line OP is y = ⅚x, and the coordinates of Q are (t,5/6).

and

Table 2. Score grading definitions.

Graded Score	Description
1 (Unacceptable)	The number of steps for which self-explanation is filled in for the steps required for the solution is minimal, and there were problematic expressions in the students’ self-explanation (e.g., mistaken patterns, boredom).
2 (Poor)	Self-explanation is mainly provided for the steps required for the solution. Still, they are more like bullet points than explanations.
3 (Fair)	Self-explanation is mainly provided for the steps required for the answer—the average self-explanation level among all respondents.
4 (Very Good)	Self-explanation is provided for most of the steps required for the answer, but there is room for improvement as an explanation (logic, expressions).
5 (Excellent)	Self-explanation is mainly provided for the steps required for the answer, and the explanation is logical and well-written.

, which are well-suited for tasks with varied solutions or strategies [41]. Instead of a detailed sentence-by-sentence breakdown, our approach evaluates explanations on a holistic, quiz-by-quiz basis, offering a comprehensive insight into the learner’s understanding of the topic.

For the evaluation process, two independent evaluators employed these rubrics to rate the collected 2205 self-explanations, scoring them on a scale ranging from 1 to 5. A quadratic weighted Cohen’s kappa coefficient [42] of 0.749 between the evaluators indicated a significant level of agreement. The subsequent analysis used the mean score derived from both evaluators, which categorized the self-explanation scores for a roughly uniform distribution. Descriptive statistics of the collected self-explanations are presented in

Table 3. Descriptive statistics of graded self-explanations.

Data Type	Num of Quiz	Variations in Math Units	Total Answers	Sentence Length (Character Count)		Quality Score
				Mean	SD	Mean	SD
Train	40	8	1420	67.8	56.8	2.94	1.34
Valid	37	8	355	67.3	59.3	2.92	1.31
Test	8	3	431	63.7	53.2	2.81	1.25

.

In anticipation of the machine learning methodologies outlined in the subsequent chapters, our dataset was segmented into three distinct categories. The train dataset, which incorporates 1420 self-explanations, forms the fundamental basis for both training our models and for LLM data augmentation. Meanwhile, the valid dataset, comprising 355 self-explanations, is earmarked for the crucial tasks of fine-tuning our models’ parameters. It also plays a significant role in the evaluation of model accuracy and in ensuring model robustness. Lastly, the test dataset, which consists of 431 self-explanations, is designated to provide a measure of the performance of our finalized models.

3.3. The Text Regression Model Description

Inspired by the work of Wang et al. [43], we employ BERT [44] and a pre-trained BERT Japanese model [45] as the backbone for our regression models, which are intended to predict the quality scores of self-explanations. Wang et al.’s methodology of injecting rubrics into the system influenced the architecture of our model, making it specifically attuned to the grading of short responses. BERT’s deep learning model, grounded on a transformer architecture, has been recognized for surpassing most preceding models in diverse natural language processing tasks [46]. Given its robust performance and compatibility with the Japanese language, BERT is an ideal choice for our study. Our model takes as the input the preprocessed self-explanation text and the corresponding quiz title (Figure 2) and yields as the output the predicted quality score for each self-explanation.

4. The Proposed Method

4.1. Overview or Pseudo-Labeling

In this section, we delve into our proposed method, building on the foundation laid out in Section 3. Our approach, illustrated in Figure 3, seamlessly blends human-labeled mathematical text data described in Section 3 and LLM-generated data to enhance our machine-learning model. Drawing from Cascante–Bonilla’s semi-supervised framework [25], we utilize pseudo-labeling as our primary technique. For human-labeled data, we lean on mathematical self-explanations, while the LLM and mathematical content texts help in producing pseudo-labeled samples to complement them. We gathered over 1420 self-explanation samples for the training model from undergraduate math students, which were further utilized in the Japanese LLM (Step 1). Figure 3 provides a comprehensive visual overview of this pseudo-training mechanism and its integrated phases.

The pseudo-labeling technique commences with the training of an initial model using the labeled dataset (Step 2). This model then assigns labels to the unlabeled data, producing what we term ‘pseudo’ labels (Step 3). These newly formed pseudo-labels are then amalgamated with the original labeled dataset, initiating a cycle of continuous model enhancement (Steps 4–5). As the model’s predictive prowess escalates, the caliber of the pseudo-labels also elevates.

4.2. Pseudo-Labeling Training Algorithm: Dataset Categorization, Function Definitions, and Model Learning

In Figure 3, we present our methodology which combines both human and pseudo-labeled samples to form a mathematically relevant dataset. Using the pseudo-labeling training algorithm (Algorithm 1), we integrate human-labeled datasets, LLM, and math texts and functions.

The human-labeled datasets act as the primary foundation for our machine learning training, and the pseudo-labeled datasets further refine and expand it. A comprehensive breakdown and flow are described in detail in the following sections as definitions and pseudo-code. Figure 4 offers an illustrative representation of the training process, harmonizing with the methodology laid out in Figure 3.

(1)

Dataset categorization:

Hereafter ${\mathbb{D}}_{type}$ represents an unlabeled dataset pertaining to a particular $type$, and ${\mathbb{D}}_{type}^{*}$ does a labeled dataset of the same category as ${\mathbb{D}}_{type}$. ${\mathbb{D}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{v}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{d}}$ is the composite dataset given to the model for training and evaluation, which includes the labeled training set ${\mathbb{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}^{*}$, test dataset ${\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, and the generated unlabeled sample dataset ${\mathbb{D}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}$.

$${\mathbb{D}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{v}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{d}}={\mathbb{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}^{*}+{\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}+{\mathbb{D}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}$$

(1)

(2)

Function definitions:

$$\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}^{*}\right)=M$$

(2a)

$$\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M,{\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}\right)={\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{*}$$

(2b)

$$\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathbb{D},k\right)={\mathbb{D}}_k\left(\mathrm{where}\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathbb{D},n\left(\mathbb{D}\right)\right)=\mathbb{D}\right)$$

(2c)

• $\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}^{*}\right)$: A function that takes a set of parameters, denoted by $\theta$, and a labeled dataset ${\mathbb{D}}^{*}$ to yield a learned model $M$.
• $\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M,{\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}\right)$: A function that accepts a model $M$ and a non-labeled test dataset ${\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, subsequently outputting a labeled test dataset ${\mathbb{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{*}$.
• $\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathbb{D},k\right)$: A function that takes in a dataset $\mathbb{D}$ and a numerical value $k$ where $0\le k\le n\left(\mathbb{D}\right)$ ($n\left(\mathbb{D}\right)$ refers to the total number of data points in dataset $\mathbb{D}$), outputting a selected subset ${\mathbb{D}}_k\subseteq \mathbb{D}$.

(3)

Model learning and final test:

$$M_1=\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}_{\mathrm{train}}^{*}\right)$$

(3a)

$$M_{t+1}=\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}_{\mathrm{train}}^{*}+\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M_t,{\mathbb{D}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}\right),k_t\right)\right)$$

(3b)

$${\mathbb{D}}_{\mathrm{test}}^{*}=\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M_T,{\mathbb{D}}_{\mathrm{test}}\right)\left(\mathrm{where}T\mathrm{is}\mathrm{sufficient}\mathrm{number}\mathrm{of}t\right)$$

(3c)

The model learning procedure follows an iterative process. Initially, the model $M_1$ is trained using $\theta$ and the labeled training dataset ${\mathbb{D}}_{\mathrm{train}}^{*}$. In each subsequent timestep, a new model $M_{t+1}$ is developed with an updated training set, comprising the original labeled dataset ${\mathbb{D}}_{\mathrm{train}}^{*}$ and a selected subset of the pseudo-labeled ${\mathbb{D}}_{\mathrm{sample}}$. After the model learning process has been iterated $T$ times, the final model $M_T$ is evaluated on the original test dataset ${\mathbb{D}}_{\mathrm{test}}$ to output the pseudo-labeled test dataset ${\mathbb{D}}_{\mathrm{test}}^{*}$. This dataset, enriched with pseudo-labels, serves as a vital resource for subsequent analyses and performance evaluations.

(4)

Parameter setting in our study:

Figure 4 provides a comprehensive outline of our experimental approach. Our dataset consists of both human-annotated and unlabeled samples. For the training process, we amassed 2205 self-explanation samples from student contributors. In our setting, $\theta$ stands for a model built using logistic regression with text representation acquired from BERT, a state-of-the-art transformer-based model renowned for its superior performance on numerous NLP tasks. The iterative process continues for 3 timesteps; in other words, $T$ is set to 3. The selection size at each timestep $t$, denoted by $k_t$, varies as follows: $k_1$ is equal to the total number of data points in ${\mathbb{D}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}$, such as $n({\mathbb{D}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}$), whereas for the second time step, $k_2$ could be any one of the following: 128, 256, 512, 1024, 2048, or 4096. To distinguish them, we defined the $\left(t+1\right)$th model, which is the same as $M_{t+1}$ in Formula (3b), which was learned with selected $k_t$ training data as $M_t^{k_t}$. The following formula represents the concrete model learning method in the study:

$$M_1=\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}_{\mathrm{train}}^{*}\right)$$

(4a)

$$M_2=\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}_{\mathrm{train}}^{*}+\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M_1,{\mathbb{D}}_{\mathrm{sample}}\right),n\left({\mathbb{D}}_{\mathrm{sample}}\right)\right)\right)$$

(4b)

$$M_3^{2^{i+7}}=\mathcal{M}\mathcal{o}\mathcal{d}\mathcal{e}\mathcal{l}\left(\theta ,{\mathbb{D}}_{\mathrm{train}}^{*}+\mathcal{S}\mathcal{e}\mathcal{l}\mathcal{e}\mathcal{c}\mathcal{t}\left(\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M_2,{\mathbb{D}}_{\mathrm{sample}}\right),2^{i+7}\right)\right)\left(\mathrm{where}0\le i\le 5\right)$$

(4c)

$${\mathbb{D}}_{\mathrm{test}}^{*}=\mathcal{T}\mathcal{e}\mathcal{s}\mathcal{t}\left(M_3^{2^{i+7}},{\mathbb{D}}_{\mathrm{test}}\right)\left(\mathrm{where}0\le i\le 5\right)$$

(4d)

Algorithm 1: Semi-supervised learning with pseudo-labeling

Input: Labeled dataset: ${\mathbb{D}}_{\mathrm{train}}^{*}$ Unlabeled dataset: ${\mathbb{D}}_{\mathrm{sample}}$ Test dataset: ${\mathbb{D}}_{\mathrm{test}}$ Maximum number of iterations: $T$ Confidence threshold for pseudo-labeling: $confidence$ Convergence threshold for change in consecutive scores: $\epsilon$ Output: Trained model: $NLPmodel$ Final score of the model on ${\mathbb{D}}_{\mathrm{test}}$: $s$ Procedure:  Initialize model:  $NLPmodel\leftarrow trainedNLPmodelwith{\mathbb{D}}_{\mathrm{train}}^{*}$  Initialize values:  $previousscore\leftarrow 0$  $iteration\leftarrow 0$  $convergenceachieved\leftarrow \mathrm{False}$  While $iteration$ $<$ $T$ and not $convergenceachieved$ do   a. Predict pseudo-labels:   $predictions\leftarrow$ predicted labels of ${\mathbb{D}}_{\mathrm{sample}}$ using $NLPmodel$   $confidence\leftarrow$ threshold when labeling of ${\mathbb{D}}_{\mathrm{sample}}$ using $NLPmodel$   b. Filter high confidence predictions:   ${\mathbb{D}}_{\mathrm{sample}}^{*}\leftarrow \left\{sample\right|sample$ in ${\mathbb{D}}_{\mathrm{sample}}$ and $confidenceofthesample>confidence\}$   c. Merge labeled and pseudo-labeled data:   ${\mathbb{D}}_{\mathrm{train}}^{*}\leftarrow {\mathbb{D}}_{\mathrm{train}}^{*}+{\mathbb{D}}_{\mathrm{sample}}^{*}$   d. Retrain the model:   $NLPmodel\leftarrow$ trained $NLPmodel$ with ${\mathbb{D}}_{\mathrm{train}}^{*}$   e. Evaluate current model performance:   $currentscore$ $\leftarrow$ Evaluation metrics when labeling ${\mathbb{D}}_{\mathrm{test}}$ using $NLPmodel$   f. Check for convergence:   If $|currentscore-previousscore|<\epsilon :$   $convergenceachieved\leftarrow \mathrm{True}$   $previousscore\leftarrow currentscore$   g. Update iteration count:   $iteration\leftarrow iteration+1$ End Procedure

4.3. Pseudo-Data Preparation: LLM Usage and Mathematical Material

We employed a pseudo-labeling technique to enrich our dataset, sourcing additional self-explanation data via the Japanese LLM and mathematical material.

Given alternatives such as OpenAI’s GPT-3 [16], our preference leaned towards Cyberagent’s LLM [17] due to its open-source availability and its adeptness in the Japanese language, perfectly complementing our dataset. To gather data, the LLM tackled mathematical contexts and formulated pertinent explanations. Our methodology was as follows:

• Random data selection: We began our process by randomly selecting 30% from our human-labeled training dataset to capitalize on the rich diversity of student-generated self-explanations.
• Keyword extraction: Ten keywords were extracted from each self-explanation, encapsulating its essence, guiding LLM to produce contextually relevant data.
• LLM generation: Armed with the extracted keywords, we then proceeded to prompt the LLM [47]. Specifically, each set of 10 keywords was used as seed input, directing the LLM to generate contextually coherent pseudo-self-explanation data. The model was given a directive to ‘elaborate based on the provided keywords’, ensuring the generated content-maintained relevance to the original self-explanation context.

Approximately 19,000 entries were generated, with a random subset of 4096 used for experiments. This combination of pseudo- and human-labeled data broadened our training set, enhancing the automated scoring system’s performance without compromising quality.

We also leveraged the math quiz texts dataset, populated with standard mathematical solutions. Its rich mathematical material and contextual problem-solving methods made it invaluable for generating mathematical self-explanations.

4.4. Comparative Analysis of Original and LLM-Generated Dataset

In

Table 4. Comparative metrics for original and generated datasets.

Data Type	Counts	Mean Score	Std
Original	2205	2.91	1.33
LLM	4096	3.39	1.44
Math	4096	3.87	1.77

and Figure 5, a detailed comparative analysis between the original and the synthetically generated datasets is elucidated. Upon examination, it becomes evident that the augmented datasets exhibit a modest augmentation in their average quality metrics relative to the foundational dataset.

Table 5. LLM-generated samples.

LLM-Generated Samples (Original)	LLM-Generated Samples (English Translation)	Predicted Self-Explanation Score
その後、α + β = −a−2とαβ = 2aの関係から解と係数の関係が分かる。次に、問題で言及されたαとβを用いて式を展開し整理し、右辺を0にする。さらに式を工夫して代入が可能な形にする。そして、関係式α + β = −a−2とαβ = 2aを式に代入して簡略化し、a^2 + a−6 = 0となる。これを因数分解してaの解を求めると、a = −3とa = 2が得られる。その後、a = 2を代入すると解が重解になり、解が一つしかなくなるため問題に合わない。したがって、正しい答えはa = −3である。その後、重積算や累乗などいろいろな計算が解に出てきて混乱するだろうから丁寧に説明していく。	After that, α + β = −a−2 and αβ = 2a can be understood from the relationship between the solutions and coefficients. Next, expand and organize the expressions using the α and β mentioned in the problem, and set the right side to 0. Further manipulate the equation to transform it into a form that allows for substitution. Then, substitute the relationship α + β = −a−2 and αβ = 2a into the equation and simplify it, resulting in a2 + a−6 = 0. Factorize this to find the solutions for a, which are a = −3 and a = 2. Afterward, when a = 2 is substituted, the answer becomes a repeated root, which means there is only one solution, making it inconsistent with the problem. Therefore, the correct answer is a = −3. Afterward, various calculations such as product of roots and exponentiation may appear in the solution, and it may be confusing, so I will explain it carefully.	5.00
代入してみたら、平方根の中が計算できた。αとβを求めた。αに−2を代入した。解を吟味して、答えは−3だった。その後、a = −4.7x^5 + 7として計算したが誤差があった。そこで右側の5を左側の4に変形すると正しい値が求められた（ただし小数点以下を切り捨て）。この問題は高校数学の範囲外であるが、中学・高校で習った知識で解くことができたので紹介したい。	Substituted it into the formula for solutions. Calculated the value inside the square root. Found α and β. Substituted α = −2. Examined the solution, and the answer was −3. Afterward, calculated it as a = −4.7x^5 + 7, but there was an error. So I rearranged the 5 on the right side to 4, and the correct value was obtained (with the decimal part truncated). This problem is beyond the scope of high school mathematics, but it could be solved using knowledge learned in middle and high school, so I want to introduce it.	4.08
グラフを書いた傾きがー4のy = x^2の接線を求めた。その後、接点を原点とし両辺に点a,bを代入すると「sin-cos」と出たので、「COS(1)」で求めた。これで式の意味が分かったと思う。−5は角度が変化しないからゼロでないといけないのだ！	Used the slope obtained from graphing to find the tangent of y = x^2 with slope −4. Took the point of tangency as the origin and substituted points a and b on both sides. ‘Sin-cos’ was obtained, so I calculated it using ‘COS(1)’. I think I understood the meaning of the equation now. −5 does not change the angle, so it must be nonzero!	3.02
x = 3を代入する。方程式を解き進める。解の吟味を書く。間違いに気づき、修正する。最終的な答えを書く。その後、再帰的に解く。	Substituted x = 3. Proceeded to solve the equation. Wrote the examination of the solutions. Noticed the mistake and corrected it. Wrote the final answer. Afterward, solve it recursively.	2.18
前のは間違えたため、全部消した。その後、通分してみた。	Since the previous one was incorrect, I deleted everything and then performed the common denominator. Afterwards, something like this.	1.23

showcases a range of these samples with their respective translations in English, alongside the scores predicted by our automated system. It is evident from the samples that as the complexity of the self-explanation increases, so does the predicted score. For instance, the first sample, which delves into the relationship between solutions and coefficients, is accorded a high score of 5.00 due to its comprehensive self-explanation. Conversely, the final example, which is a brief reflection on an error and an attempt to rectify it, receives a lower score of 1.23, reflecting its succinct nature.

Table 6. Math texts samples.

Math Texts Samples	Predicted Self-Explanation Score
Angle bisector and ratio, using Ceva’s theorem: Revised version succeed math A problem 349 Δ (Triangle), let Δ have the angle bisector of ∠ and the point where it intersects the side, and the point that divides the side in the ratio. When the line intersects at point, find the length of the side.	5.00
Using Menelaus’s theorem: Segment ratio and area ratio, revised version succeed math A problem 350 Δ, let be the point where it divides the side in the ratio, and the point where the segment is divided in the ratio, and the point where the extension of the segment intersects the side. Find the following segment ratios and area ratios Δ : Δ	4.93
Using the relationship between sides and angles: Range of values for side length in a triangle, revised version succeed math A problem 355, determine the range of values for so that a triangle with the following side lengths exists.	3.84
Using the relationship between the sizes of three sides: Proving inequalities related to segment lengths, revised version succeed math A important example 66, take point inside Δ, and join and prove that . Abbreviated.	3.13
Examining the sizes of the three angles of a triangle, revised version succeed math A important example 64, examine the sizes of the three interior angles of Δ.	2.66

offers various math texts that were considered during our study. These texts, extracted from educational resources, shed light on various mathematical principles and theorems. The predicted self-explanation score gives an indication of the depth and comprehensiveness of the self-explanation provided in each math text. For instance, the math text delving into the ‘angle bisector and ratio, using Ceva’s theorem’ provides a nuanced exploration of the topic, earning it a top score of 5.00. In contrast, the examination of the sizes of the three angles of a triangle is more straightforward, reflected in its score of 2.66. The varied scores underline the diverse complexity levels present in math texts.

The inherent variability in the complexity of mathematical texts and self-explanations, as illustrated by Table 5 and Table 6, underscores the importance of our automated scoring system. It demonstrates the system’s capability to adaptively discern and quantify the nuances in self-explanation quality.

5. Experiments and Evaluations

5.1. Exploring the Influence of Self-Explanation Augmentation on Model Efficiency

We embarked on an exploration to discern the influence of self-explanation augmentation on the efficiency of an automated self-explanation scoring model across diverse datasets. We used mean absolute error (MAE) metrics [48,49] to evaluate model performance, giving insights into the extent of error deviation and the efficacy for individual items.

Table 7. Datasets overview.

Dataset	Base_Line	LLM	Math	Mixed	Only_LLM_Math
Original (N = 1420)	●	●	●	●
LLM- generated (N = 4096)		●		●	●
Math texts (N = 4096)			●	●	●
Total Number of Data	1420	5516	5516	9612	8192

and

Table 8. Model performance for various datasets (MAE).

Data Type	Base_Line	LLM	Math	Mixed	Only_LLM_Math
Test	0.749	0.699	0.646	0.692	1.135
Val	0.602	0.341	0.358	0.336	1.033

lay out the results of our experiments, contrasting performances across different dataset permutations. When we introduced augmented datasets into the mix, distinct variations in performance emerged.

Remarkably, our model, when nurtured with a blend of the ‘math’ and ‘original dataset’, consistently delivered the most desirable MAE results. This underlines its superior predictive precision in assessing self-explanation quality. Such results lend credence to the efficacy of the model when trained with this specific data amalgamation. On another note, the ‘mixed’ model—which weaves together human-graded samples, LLM-crafted pseudo-sentences, and mathematical content—also demonstrated notable improvements. This outcome underscores the model’s robustness and flexibility when fed with diverse data sources. Yet, the model named ‘only_LLM_math’, which exclusively depended on LLM-created sentences, trailed behind the foundational model in terms of performance. This observation underscores the criticality of harmonizing human-judged and machine-produced data to achieve optimal results.

5.2. Evaluating Optimal Quantity of Pseudo-Self-Explanation Data

The data from

Table 9. Test MAE with varying amounts of added pseudo-self-explanation sata.

Dataset	Number of Datasets Added
	128	256	512	1024	2048	4096
base_line	0.75
LLM	0.67	0.63	0.72	0.72	0.71	0.7
math	0.64	0.66	0.67	0.64	0.65	0.65
mixed	0.68	0.66	0.71	0.68	0.73	0.69
only_LLM_math	1.19	0.96	1.02	0.89	1.15	1.14

and

Table 10. Validation MAE with varying amounts of added pseudo-self-explanatory data.

Dataset	Number of Datasets Added
	128	256	512	1024	2048	4096
base_line	0.60
LLM	0.57	0.35	0.51	0.49	0.40	0.34
math	0.40	0.50	0.43	0.35	0.40	0.36
mixed	0.59	0.32	0.52	0.44	0.40	0.34
only_LLM_math	1.19	0.90	0.96	0.81	1.02	1.03

, as well as the visual cues from Figure 6 and Figure 7, underline the importance of calibrating the amount of added pseudo-self-explanation data. While adding more datasets often improves performance, there is a saturation point beyond which the returns diminish. This trend underscores the need for a balanced approach, optimizing the amount of data incorporated to ensure robust and generalizable results. It also emphasizes the intricate interplay between different datasets, with ‘LLM’ and ‘mixed’ data offering particularly promising outcomes.

The ‘baseline’ row signifies the MAE when the model is trained only on the original dataset, devoid of any pseudo-self-explanation data. Each subsequent row shows the MAE when the model is trained with an increasing volume of pseudo-self-explanation data, ranging from 128 to 4096 datasets.

Upon examining the ‘LLM’ model, we note an enhancement in performance when the added datasets increase from 128 to 256. Beyond this, the further addition of generated data does not significantly reduce the MAE, suggesting an optimal balance between data augmentation and model efficacy with an addition of 256 datasets. The ‘math’ model displays a similar trend, with the lowest MAE achieved when 1024 datasets are added. Beyond this point, no substantial performance enhancement is observed with extra data.

For the ‘mixed’ model, we see a consistent improvement in performance with increased data, but this plateaus beyond 2048 datasets, where the MAE slightly increases. Conversely, the ‘only_LLM_math’ model shows erratic trends. Its performance varies noticeably with the quantity of added data and consistently exceeds the baseline model’s MAE, regardless of the added data volume. This reveals potential difficulties when exclusively relying on generated pseudo-self-explanation data.

6. Discussion

6.1. Detailed Analysis of Results (RQ1)

Regarding Research Question 1, an in-depth analysis of the results displayed in Table 8 reveals several noteworthy observations regarding the influence of self-explanation augmentation on the model’s performance. In the test category, we observe an improvement in the model’s performance when transitioning from the baseline to the LLM and math models. Notably, the math model achieves the lowest MAE at 0.646, which aligns with Dai et al.’s [36] proposition that data augmentation at the semantic level improves robustness and consistency. However, the performance slightly deteriorates in the mixed model and substantially plummets in the ‘only_LLM_math’ model. This suggests that an excessive concentration of LLM-generated self-explanations could impair the model’s predictive proficiency.

A similar pattern emerges when examining individual topics within the ‘test’ category. For instance, the model delivers optimal performance for ‘quadratic equations’ with the LLM-generated model, but the performance markedly deteriorates when solely relying on LLM-generated self-explanations. The validation category follows a similar trajectory, with the LLM, math, and mixed models outshining the baseline model. Once again, the mixed model achieves the smallest error. However, the ‘only_LLM_math’ model experiences a decline in performance, further highlighting the advantages of using a diverse dataset that encompasses both human-evaluated and machine-generated explanations.

6.2. Findings and Observations (RQ2)

Regarding Research Question 2, the results presented in Table 9 and Table 10 provide valuable insights into determining the optimal quantity of generated pseudo-self-explanation data that can enhance the model’s performance. For the ‘LLM’ model, an initial improvement in model performance is observed as the number of added datasets increases from 128 to 256. Beyond this point, further augmentation of the generated data does not lead to a significant reduction in MAE, suggesting that adding 256 datasets strikes an optimal balance between data augmentation and model performance. The ‘math’ model exhibits a similar pattern, with the lowest MAE observed when 1024 datasets are added, and no significant performance improvements resulting from further data augmentation. The ‘mixed’ model, on the other hand, shows a general trend of performance enhancement with increased data augmentation, up to a threshold of 2048 datasets, beyond which the MAE slightly increases. In contrast, the ‘only_LLM_math’ model does not present a consistent trend. Its performance fluctuates significantly as the volume of added data increases, and its MAE consistently surpasses that of the baseline model, regardless of the amount of added data. This underscores the challenges of solely leveraging generated pseudo-self-explanation data for augmentation, particularly when the model might lack domain-specific expertise, echoing concerns raised by Dai et al. [36].

Our study underscores that while weaving in generated pseudo-self-explanation data offers potential advantages, there is a point of diminishing returns. Beyond a certain threshold, piling on more data does not invariably translate to superior outcomes. This stunted impact might have its roots in the constraints imposed by our evaluation dataset’s size and imbalance. The compact nature of our training and test datasets might curtail the model’s expansive learning capabilities. Intriguingly, despite the mathematical underpinnings in the LLM and math texts, the ‘only_LLM_math’ model does not match up to either the baseline or its proposed counterparts. The divergences might be attributable to the caliber of the generated self-explanations coupled with the characteristics of the datasets utilized in the training and testing phases.

In conclusion, while integrating generated pseudo-self-explanation data shows potential in enhancing model performance, our results suggest there is a limit. Oversampling methods, such as SMOTE and ADASYN, can expand small datasets and provide a broader learning spectrum for models [29]. Although our focus has been on LLM pseudo-labeling, merging this with oversampling could be a promising direction for future research, aiming to further refine our models. Our study underscores the importance of careful and context-aware adjustments when adopting data augmentation strategies in developing self-explanation auto-scoring models.

6.3. Limitations and Future Research

In our research, several limitations of our study should be highlighted.

• Subject scope: Our dataset is restricted to mathematics, potentially constraining the generalizability of our findings to other subjects.
• Dependency on LLM: Our methodology hinges on the LLM’s ability to generate pseudo-self-explanation data. This dependence may introduce noise and errors into our system.
• Data quality and representativeness: The performance of our approach is contingent on the quality and representativeness of labeled data. Poor or biased data could compromise model efficacy.
• Model performance variability: We identified noticeable disparities in our model’s performance across various mathematical categories. For instance, it predicted the ‘property of a circle’ (0.242) more accurately than ‘quadratic functions’ (0.419) within the validation datasets. These results indicate that self-explanation augmentation’s effectiveness may be influenced by the inherent complexity of a topic and the linguistic nuances present within the self-explanations.
• Evaluation dataset categories and size: The evaluation dataset for some categories is comparatively small, which poses challenges in drawing definitive conclusions. It is essential to consider the ease of inference as it pertains to various mathematical concepts, including linear functions, shapes, equations, and square roots. Certain subjects may be inherently more challenging for machine training due to their linguistic or conceptual intricacies.