Knowledge Graph and Personalized Answer Sequences for Programming Knowledge Tracing

Abstract

Knowledge tracing is a significant research area in educational data mining, aiming to predict future performance based on students’ historical learning data. In the field of programming, several challenges are faced in knowledge tracing, including inaccurate exercise representation and limited student information. These issues can lead to biased models and inaccurate predictions of students’ knowledge states. To effectively address these issues, we propose a novel programming knowledge tracing model named GPPKT (Knowledge Graph and Personalized Answer Sequences for Programming Knowledge Tracing), which enhances performance by using knowledge graphs and personalized answer sequences. Specifically, we establish the associations between well-defined knowledge concepts and exercises, incorporating student learning abilities and latent representations generated from personalized answer sequences using Variational Autoencoders (VAE) in the model. This deep knowledge tracing model employs Long Short-Term Memory (LSTM) networks and attention mechanisms to integrate the embedding vectors, such as exercises and student information. Extensive experiments are conducted on two real-world programming datasets. The results indicate that GPPKT outperforms state-of-the-art methods, achieving an AUC of 0.8840 and an accuracy of 0.8472 on the Luogu dataset, and an AUC of 0.7770 and an accuracy of 0.8799 on the Codeforces dataset. This demonstrates the superiority of the proposed model, with an average improvement of 9.03% in AUC and 2.02% in accuracy across both datasets.

1. Introduction

With the continuous development of Massive Open Online Courses (MOOC), a large amount of online learning data can be used to accurately and timely trace students’ learning states. Knowledge tracing (KT) is a data-driven method in education technology, which uses a series of student interactions with exercises to predict their mastery of Knowledge Concepts (KCs) corresponding to those exercises. However, many existing KT models, such as DKT [1], DKT+ [2], DKVMN [3], SAKT [4], IEKT [5], AKT [6], and GKT [7], suffer from limitations, including insufficient exercise representation and a lack of personalized knowledge state modeling. Details of these works will be reviewed in the subsequent section.

Nowadays, programming has become a fundamental skill for solving real-world problems. However, despite the success of KT in MOOC systems, a noticeable gap in research on KT within programming education can be observed, which presents distinct challenges. Programming KT is mainly based on subjective exercises, and questions are usually answered by students through code submission. The performance of students’ code is judged by the evaluation system using test data under different situations. In this context, we provide an example of students’ answers to programming exercises, as shown in Figure 1, where exercise $p_1$ corresponds to the KC “Struct”, $p_2$ includes the KCs “Struct” and “Pointer”, $p_3$ corresponds to the KC “LinkList”, and $p_4$ includes the KCs “LinkList” and “Bi-LinkList”. In the process of answering exercises in the field of programming, two aspects are worth considering:

(1)

Exercises are not independent—clear predecessor and successor relationships between KCs exist. For example, after the $p_2$ exercise was answered incorrectly, $p_4$ continued to be attempted by student $S_1$. Due to insufficient mastery of the predecessor of $p_4$, the exercise resulted in a timeout. However, after exercises related to the previous KCs, $p_2$ and $p_3$, were completed, $p_4$ was answered correctly by student $S_1$.

(2)

Multiple attempts reflect individualized knowledge mastery. For example, two failed attempts were made by student $S_2$ even after exercise $p_2$ was answered correctly, which may have been submitted for testing reasons and should not be interpreted as a lack of mastery over the KCs involved. Another example is when exercise $p_1$ was faced; students $S_1$ and $S_2$ answered correctly on the first attempt, while three attempts were required by student $S_3$ to get it right. Therefore, the mastery of the knowledge concept “Struct” is clearly different for $S_1$ and $S_2$ compared to $S_3$.

Various methods have been explored in current studies on programming KT. For example, Bayesian Knowledge Tracing (BKT) has been employed for code quality assessment, and code representation learning has been conducted using abstract syntax trees (AST) and token-based methods. While promise has been shown by these methods in tracing students’ knowledge states on individual exercises, the broader relationships between exercises and KCs are often overlooked, and personalization in modeling students’ overall knowledge states is lacking.

To address these challenges, we propose a programming KT model based on a knowledge graph and personalized answer sequences for the diversification of programming exercises, named GPPKT. A knowledge graph is provided as a structured way to represent relationships between entities, making it especially suitable for organizing complex information. In the context of programming KT, the hierarchical and relational structure of knowledge concepts is captured by the knowledge graph, and the connections between these concepts are modeled more accurately. The Variational Autoencoder (VAE) [8] is utilized as a powerful generative model to learn latent representations of data. These latent variables capture the underlying structure and variability in the data. In programming KT, personalized knowledge states are modeled by the VAE, capturing the nuances of students’ answer sequences. Long Short-Term Memory (LSTM) networks [9] are employed for modeling sequential data, as long-term dependencies are captured by them. This makes them an effective choice for KT, as a student’s learning process is influenced by a series of past interactions and answers. By integrating these components, a powerful framework for programming knowledge tracing is provided by GPPKT. Both the representation of programming exercises and the modeling of students’ individual learning processes are improved. In summary, the main contributions of this work are as follows:

• We construct a knowledge graph in the programming field to constrain the embedding of knowledge concepts by perceiving their types. The resulting embedding vectors can effectively connect the KCs in the exercises.
• In response to the phenomenon of students answering consecutive programming exercises, we explore students’ learning behavior and learning ability, and a gating mechanism is introduced to balance their historical and current knowledge states. This approach better reflects the personalized knowledge mastery of students.
• We conduct extensive experiments on two real-world programming datasets, which shows that GPPKT outperforms state-of-the-art models, with an average AUC improvement of 9.0%. Ablation experiments are also performed to ensure the reliability and effectiveness of each component in the study.

The rest of this paper is organized as follows: In Section 2, related work is introduced, and the improvements made by our method on these approaches are highlighted. In Section 3, the problem addressed by programming knowledge tracing is explained, and the components and implementation of the GPPKT model are detailed. Section 4 presents experimental comparisons of GPPKT with baseline methods on two programming datasets, along with ablation studies. Finally, in Section 5, the paper is concluded, and future work and potential applications are discussed.

2. Related Work

In recent years, significant advancements have been made in knowledge tracing through the application of deep learning techniques. Deep Knowledge Tracing (DKT) [1], a model based on Recurrent Neural Networks (RNNs), marked a pivotal shift by eliminating the need for manual feature extraction and achieving higher accuracy in predicting student performance. Following DKT, various deep learning models have been proposed to further refine KT. Notable among these are RNN-based models, such as EERNN [10], DKT-DSC [11], and KQN [12], as well as memory network-based models like Deep-IRT [13] and DKVMN [14]. The integration of attention mechanisms has also been explored, leading to models like AKT [6], ATKT [15], and RKT [16], which offer improved performance by focusing on relevant parts of the input data. In addition, convolutional neural networks have been employed in KT models, such as CKT [17] and CRKT [18], to capture spatial dependencies within the data. Further enhancements in KT have been achieved by incorporating exercise text models (EKT) [19] and memory curves [20,21] to better reflect student learning behavior. Moreover, graph-based approaches have gained traction, with Nakagawa [7] introducing the Graph-based Knowledge Tracing (GKT) model, which represents relationships between KCs as a graph and frames the KT task as a time-series node classification problem within a Graph Neural Network (GNN). Yang [22] expanded on this concept with the Graph-based Interactive Knowledge Tracing (GIKT) model, which leverages the relationships between exercises and KCs for embedding learning. Additionally, the SFBKT model [23] was introduced, incorporating a synthetically forgetting behavior method that utilizes both individual and group forgetting factors to enhance predictions of student performance.

In programming KT, a BKT-based method for evaluating code quality and tracing knowledge states was introduced by Kasurinen [24], relying on manual rules and statistical analysis of code syntax. A programming KT model integrating code features with LSTM networks was developed by Wang [25], using student-submitted code as input. Piech’s [26] recurrent neural network model was also employed, where ASTs are decomposed into subtrees, and a non-parametric model is used for matrix representation of each subtree’s root node. Building on this, a token-based method was employed by Swamy [27], in which code tokens form a vocabulary, with vector elements representing the TF-IDF values of tokens. The code2vec model was used by Shi [28] with DKT to trace student progress, representing code as abstract syntax trees divided into code paths, guided by learned weights. KT models using logistic regression and RNN based on the AST edit distance were proposed by Jiang [29], quantifying the distance between a student’s intermediate solution and the optimal one. Lastly, student code was transformed into vector representations by Liang [30], and error classification was used as a concept indicator, adding a cognitive layer to DKT for inferring students’ conceptual skill levels.

Our proposed method is built upon by these works, specifically addressing the limitations of existing programming KT models. Unlike prior models that often focus on single code features or specific exercises, a knowledge graph is integrated into our method to capture the relationships between diverse exercises and KCs. Additionally, the VAE is employed to model personalized learning trajectories, allowing for a better reflection of students’ individualized learning progress and abilities. This approach combines a knowledge graph with personalized learning trajectory modeling. It not only improves the predictive accuracy of existing methods but also offers a more holistic understanding of students’ knowledge states across different exercises and KCs.

3. Method

In this section, a detailed introduction to our GPPKT model is provided. First, the formal definition of KT is presented. Then, the method for constructing the knowledge graph is described. Next, the use of the VAE to handle student answer sequences and the IRT for modeling students’ personalized learning abilities is introduced. Finally, these features are incorporated into the KT framework to enhance its generality in the programming field, thereby improving KT’s performance.

3.1. Presentation of the Problem

In online judge (OJ) systems, suppose there are $\left|E\right|$ programming exercises. The answer sequence for a particular student is defined as $s=\left\{(e_t,r_t)\right|t=1,\dots ,n\}$, $e_t\in E$ represents the t-th programming exercise attempted by the student, $r_t\in \{0,1\}$ indicates the correctness of the answer, and n represents the number of different exercises which students answered. Typically, $r_t=1$ indicates that the student passed all test cases, otherwise $r_t=0$. As programming exercises involve students repeatedly attempting the same exercise, the student’s answer to exercise $e_t$ is detailed as $(e_t,r_t)=\left\{(e_t,r_t^i)\right|i=1,\dots ,m\}$, where m represents the number of consecutive attempts for exercise $e_t$, and $r_t^i\in \{0,1\}$ indicates the answer for the i-th consecutive attempt on exercise $e_t$.

3.2. Overall Architecture

We propose a deep KT model based on a knowledge graph and students’ personalized answer sequences, namely GPPKT, to solve programming KT. The model architecture is shown in Figure 2, and mainly consists of four parts. The knowledge graph module is responsible for linking the various KCs to obtain the representation of the exercise $e_t$. IRT is used to obtain the student’s personalized learning ability value ${\theta }_t^i$ in the part of learning ability. In the part of answer sequence, the student’s answer sequence $r_t^{seq}$ on the same exercise is modeled by VAE, where the personalized answer state $z_t$ obtained can effectively reflect the student’s learning behavior. The LSTM module introduces an attention mechanism for adaptive aggregation of the current state $k{s}_t$, and a gating mechanism is proposed to balance the student’s historical and current knowledge state, where $k{s}_{tout}$ denotes the student’s final knowledge state.

3.3. Knowledge Graph to Represent Exercise Vectors

3.3.1. Constructing Knowledge Graph

Many kinds of programming exercises exist where the predecessor and successor relationships between KCs are obvious, and these relationships can be reflected by the knowledge graph, as shown in the left part of Figure 2. By using graph embedding methods, embedding vector representations that contain rich semantic and topological information are obtained from the graph. A knowledge graph suitable for the field of programming exercises is constructed in three steps: knowledge acquisition, graph design, and storage.

• Knowledge concept acquisition: We gather information about KCs from the OI-Wiki website. Specifically, hierarchical and relational information about programming concepts, such as “Dynamic Programming” and its associated subtopics, like “Depth First Search” and “Recurrence”, is extracted. To ensure the completeness and correctness of these concepts, the obtained KCs are categorized and organized with the help of information from CSDN and Wikipedia. Additionally, three authoritative textbooks, Data Structure Programming Practice, Introduction to Competitive Programming (2nd Edition), and Introduction to Algorithms, are used for validation. This process results in a knowledge graph with 11 topics and 204 KCs.
• Knowledge graph design: Introductory KCs in the field of programming are first identified based on various knowledge sources. For instance, under the “Fundamentals” category, concepts such as “Divide and Conquer” and “Function” are categorized as basic KCs. Next, predecessor–successor relationships are established, linking more advanced KCs like “Dynamic Programming” to foundational concepts. These relationships are labeled with references to content from Data Structure Programming Practice and OI-Wiki. Further validation of the initial knowledge graph is carried out using Introduction to Competitive Programming (2nd Edition) and Introduction to Algorithms to minimize potential subjective biases. The graph is then optimized by retrieving additional hierarchical relationships for KCs from Wikipedia, ensuring a more comprehensive coverage.
• Knowledge storage: After KC acquisition and knowledge graph design, we obtain the nodes and relationships required for the knowledge graph and finally use Neo4j for knowledge storage.

We extract a portion of the constructed knowledge graph, as shown in Figure 3; the knowledge graph represents different KCs and their interrelationships. For instance, under the category “Dynamic Programming”, concepts such as “Depth First Search” and “Recurrence” are connected, reflecting their related nature. Nodes with the same color are assigned to the same knowledge category, indicating their thematic connections. The lighter, dashed edges are used to illustrate connections between different categories, emphasizing the interconnectedness of programming concepts across various topics.

3.3.2. Knowledge Graph Embedding Model: RotatE-TA

To improve the representation of exercises in our model, we build on the RotatE model, which is a knowledge graph embedding method that uses relational rotation in complex vector space to model entities and relations [31]. RotatE effectively captures the relationships between entities by treating relations as rotations in the complex plane. Building on this foundation, we propose a type-aware extension called RotatE-TA. The knowledge graph structure is incorporated into our model, while type information of KCs is also accounted for to create more accurate embeddings.

• Entity and Relation Definition: We define predecessor KC as head entities $c_i$ and successor KC as tail entities $c_j$, with the relation denoted as r. These two KCs, along with the relation, are represented as a triple $(c_i,r,c_j)$.
• Scoring Function: We define a scoring function for the triple as $d_r(c_i,c_j)$, which evaluates the importance of a candidate triple.
• Type Comparison: The knowledge topics for all KCs in the knowledge graph are defined as $T=\{t_1,t_2,\dots ,t_m\}$, where each KC $c_i$ is associated with a knowledge topic $c_i^t\in T$, and each KC belongs to only one knowledge topic. The comparison of the two knowledge topic types is calculated as follows:

  $$type(c_i,c_j)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{c}_i^t=c_j^t\hfill \\ 0,\hfill & \mathrm{else}\hfill \end{array}\right.$$

  (1)
• Distance Scoring with Type-Awareness: Based on the type comparison function, we introduce type-aware weights into the distance scoring function as follows:

  $$d_r(c_i,c_j)=\left|\right|c_i\circ r-c_j\left|\right|+\lambda \ast \left|\right|m_{c_i}-m_{c_j}\left|\right|\ast type(c_i,c_j).$$

  (2)
  In Equation (2), ∘ represents the Hadamard product, $m_{c_i}$ and $m_{c_j}$ represent the modular lengths of the head entity and tail entity, respectively. $\lambda$ is a hyperparameter representing the magnitude of type-aware weights.

After embedding the knowledge graph, we obtain the vector representation of an exercise by simply adding the embedding vectors of the KCs, as shown in Equation (3):

$$e_t=\sum _{k=1}^m{c}_k,$$

(3)

where m represents the number of KCs involved in the exercise, $c_k$ is the embedding vector for each KC, $e_t$ is the final exercise vector, and this vector can capture the semantic relationships of the exercise in the knowledge graph.

3.4. Modeling Answer Sequences with VAE

In addressing the issue of lacking personalized knowledge states during the answer sequence in the programming field, we propose a personalized modeling method based on the VAE [8], as shown in the middle right part of Figure 2, inspired by the Exercise-Enhanced Recurrent Neural Network (EERNN) [10]. VAE is a generative model that combines the concepts of an autoencoder [32] with variational inference to learn latent representations of data.

3.4.1. Input Representation

In this study, we represent a student’s multiple answers to a specific exercise as a sequence denoted by $r_t^{seq}=\{r_t^1,r_t^2,\cdots ,r_t^n\}$, where $r_t^i$ represents the i-th answer. Since the number of answers a student provides for a particular exercise is uncertain, $r_t^{seq}$ is adjusted to a fixed length N by padding it with $r_t$.

3.4.2. Model Architecture

The model’s architecture is depicted in Figure 4. The student’s answer sequence to the same exercise is used as the VAE input, with hidden layer information $z_t$ being learned and extracted as the personalized response state, thereby enabling the modeling and tracing of the student’s knowledge state. The encoder–decoder structure of the VAE is defined by the following equations:

$$\begin{array}{cc}\hfill z_t& =\mathrm{Encoder}\left(r_t^{seq}\right),\hfill \\ \hfill y& =\mathrm{Decoder}\left(z_t\right).\hfill \end{array}$$

(4)

3.4.3. Objective Function

The objective function $L_{total}$ of the VAE consists of two components: the reconstruction loss $L_{recon}$ and the KL divergence loss $L_{KL}$. These components are calculated as follows:

• Reconstruction Loss: This loss measures the difference between the original input $r_t^{seq}$ and the output of the decoder y. It is defined as:

  $$\begin{array}{cc}\hfill L_{recon}& =-\sum _{i=1}^N\left({r_t^{seq}}_{i}log\left(y_i\right)+(1-{r_t^{seq}}_i)log(1-y_i)\right),\hfill \end{array}$$

  (5)

  where N represents the dimensions of the student’s answer sequence, ${r_t^{seq}}_i$ and $y_i$ represent the i-th element of the input sample $r_t^{seq}$ and the decoder output y, respectively.
• KL Divergence Loss: This loss encourages the learned latent variable distribution to be close to a standard normal distribution. It is defined as:

  $$\begin{array}{cc}\hfill L_{KL}& =-{\displaystyle \frac{1}{2}}\sum _{j=1}^D\left(1+log\left({\left({\sigma }_j\right)}^2\right)-{\left({\mu }_j\right)}^2-{\left({\sigma }_j\right)}^2\right),\hfill \end{array}$$

  (6)

  where D represents the dimensions of the latent variable $z_t$, and ${\mu }_j$ and ${\sigma }_j$ represent the mean and standard deviation of the j-th dimension of the latent variable $z_t$, respectively.

The final objective function is a combination of these two losses:

$$\begin{array}{cc}\hfill L_{total}& =L_{recon}+L_{KL}.\hfill \end{array}$$

(7)

3.5. Students’ Personalized Learning Abilities

3.5.1. First-Order IRT for Ability Assessment

Different students have varying levels of mastery over different exercises, and a single student may have different levels of mastery over different exercises. As shown in the middle left part of Figure 2, this study aims to assess and represent students’ personalized learning abilities using a first-order IRT, thereby reflecting students’ different abilities for each exercise. The equation for first-order IRT is as follows:

$$p_i\left(\theta \right)={\displaystyle \frac{1}{1+e^{-d(\theta -b_i)}}},$$

(8)

where $p_i\left(\theta \right)$ represents the probability of a student answering a particular exercise correctly, $\theta$ represents the student’s personalized learning ability, $b_i$ represents the difficulty parameter of the exercise, and d is a constant set to 1.702 to scale the logistic function to approximate a normal ogive curve.

3.5.2. Inverting Ability Parameters

We utilize students’ accuracy in answering exercises as an indicator of their personalized learning abilities, defined as the ratio of correct answers to the total number of attempts. Additionally, our datasets in the programming field include information about the difficulty of each exercise. Based on this information, the original IRT equation is used to invert the individual ability parameter $\theta$, achieving an accurate measurement of student learning ability:

$${\theta }_t^i=b_i+{\displaystyle \frac{1}{d}}ln{\displaystyle \frac{p_i\left(\theta \right)}{1-p_i\left(\theta \right)}}.$$

(9)

In this study, we obtain the difficulty parameter $b_i$ of the exercise and the student’s answer condition $p_i\left(\theta \right)$. The final obtained ${\theta }_t^i$ represents the personalized ability parameter of the student.

3.6. LSTM Framework

3.6.1. Modeling Knowledge States with LSTM

Due to the unique mechanism in the field of programming, we should consider not only the exercise information and students’ results, but also incorporate the previously mentioned personalized response state $z_t$ and the student’s learning ability ${\theta }_t^i$. In this study, a student’s personalized learning state is composed of these features mentioned above. Subsequently, the interaction projection technique [15] is applied to differentiate whether a student answers an exercise correctly. Additionally, $r_t$ is extended into a K-dimensional all-0 or all-1 vector ${\widehat{r}}_t$. The calculation method for a student’s personalized learning state is as follows:

$$\begin{array}{c}\hfill x_t=\left\{\begin{array}{cc}[e_t,z_t,{\widehat{r}}_t,{\theta }_t^i]\hfill & if{r}_t=1,\hfill \\ {\widehat{r}}_t,e_t,z_t,{\theta }_t^i]\hfill & if{r}_t=0.\hfill \end{array}\right.\end{array}$$

(10)

The above equations represent the concatenation of four features, where $x_t$ signifies the student’s personalized learning state. From this, we obtain the sequence of a student’s personalized knowledge states, $X=\{x_1,x_2,x_3,\dots ,x_t\}$.

Next, LSTM is used to model the student’s knowledge states, as shown in the upper part of Figure 2. The student’s current knowledge state $k{s}_t$ is only dependent on the previous time step’s knowledge state $k{s}_{t-1}$ and the current learning state $x_t$. Therefore, the LSTM computation process can be simplified as follows:

$$k{s}_t=LSTM(k{s}_{t-1},x_t).$$

(11)

3.6.2. Attention Mechanism

Although it is possible to directly use $k{s}_t$ to predict a student’s performance on the next exercise, the programming exercises have many knowledge topics, various KCs, and a wide range of difficulty levels. Therefore, recent exercises may not have an equal impact on a student’s performance. To address this issue, we introduce an attention mechanism to adaptively determine the importance coefficients $a_i$ for each exercise in the early knowledge state sequence, $K{S}_{seq}=\{k{s}_1,k{s}_2,\dots ,k{s}_{t-1}\}$. The computation process is as follows:

$$\begin{array}{cc}\hfill u_i& =tanh(W_w·k{s}_i+b_w),\hfill \\ \hfill a_i& ={\displaystyle \frac{exp\left(u_i^T{u}_w\right)}{{\sum }_{j}exp\left(u_j^T{u}_w\right)}}.\hfill \end{array}$$

(12)

In this process, $k{s}_i$ is first input into a single-layer neural network, and then a softmax operation is performed to obtain the importance coefficient $a_i$. Where $W_w$, $b_w$, and $u_w$ represent the weight matrix, bias, and the weight vector, respectively.

Due to the differential impact of historical and current knowledge states on prediction, this study divides students’ knowledge states into two stages: historical and current. The historical knowledge state $k{s}_{his}$ is defined as the weighted sum of early knowledge states $\{k{s}_1,k{s}_2,\dots ,k{s}_{t-1}\}$. The specific calculation equation is as follows:

$$k{s}_{his}=\sum _{j=1}^{t-1}{a}_{j}k{s}_j.$$

(13)

3.6.3. Gating Mechanism

In this study, $k{s}_t$ is used to represent the current level of knowledge mastery for a student, reflecting their present knowledge state. Meanwhile, the student’s historical knowledge state $k{s}_{his}$ is also considered a significant indicator of their knowledge mastery. Therefore, we introduce a learnable gating mechanism, denoted as g to balance the importance between $k{s}_t$ and $k{s}_{his}$. The final knowledge state acquisition process is illustrated in Figure 5, and the detailed computation process is as follows:

$$\begin{array}{c}\hfill g=\sigma (W_t·k{s}_t+W_{his}·k{s}_{his}+b_g),\\ \hfill k{s}_{tout}=g\odot k{s}_t+(1_D-g)\odot k{s}_{his},\end{array}$$

(14)

where g represents a learnable gating mechanism, $W_t$ and $W_{his}$ denote the weight matrices, and $b_g$ represents the bias term. ⊙ signifies element-wise multiplication, $1_D$ represents a D-dimensional all-ones vector, and $k{s}_{tout}$ represents the student’s final knowledge state.

3.7. Prediction

Once the student’s final knowledge state is obtained, a single layer neural network can be used to predict students’ performance in the next exercise. Assuming the one-hot encoding representation of the next exercise is denoted as ${\widehat{e}}_{t+1}\in R^{1\times K}$, the equation for calculating the probability of the next exercise’s prediction is as follows:

$${\widehat{y}}_{t+1}=\sigma (k{s}_{tout}·W_{out}+b_{out})·{\widehat{e}}_{t+1},$$

(15)

where $W_{out}$ represents the weight matrix, $b_{out}$ is the bias term, ${\widehat{y}}_{t+1}$ signifies the predicted probability of the student answering exercise $t+1$ correctly, and $y_{t+1}$ is the actual student answer to exercise $t+1$. We use the cross-entropy to calculate the loss between $y_{t+1}$ and ${\widehat{y}}_{t+1}$, and the specific equation for this calculation is as follows:

$$L=-\sum _{t+1}(y_{t+1}log{\widehat{y}}_{t+1}+(1-y_{t+1})log(1-{\widehat{y}}_{t+1})).$$

(16)

4. Experiments

4.1. Dataset

4.1.1. Data Source

Traditional KT models often use open datasets, such as Statics [33] or ASSISTments [34]. However, there are significant differences between the programming domain and traditional fields, particularly because most programming exercises are subjective in nature. To ensure the effectiveness of the model proposed in this study, we utilized datasets from two widely used programming practice platforms, Luogu and Codeforces. These platforms offer a diverse range of exercise types and a comprehensive record of student responses, which align well with the unique requirements of programming education. This approach also allows the performance of our model to be evaluated in a more realistic and practical context.

Luogu (https://www.luogu.com.cn/, accessed on 20 December 2022) established by Chinese programming enthusiasts, is a robust platform dedicated to providing a refreshing and efficient programming experience for OIers/ACMers. With a current repository of over ten thousand exercises and several hundred thousand users, Luogu gains popularity in the programming community. Codeforces (https://codeforces.com/, accessed on 27 December 2022) is a Russian website, which serves as an online judging system, and hosts daily programming contests for global programming enthusiasts. Its globally renowned weekly contests attract nearly 30,000 participants, and the platform boasts a collection of over 8000 exercises, earning recognition from top programming students both domestically and internationally.

4.1.2. Data Preprocess

Given that the datasets were sourced from programming practice platforms, the raw data included various types of information, such as knowledge concepts, exercises, and student answer records, all represented as discrete character-type data. To prepare the data for model training, we applied a series of preprocessing steps:

• Filtering Invalid Submissions: We filtered out invalid submission records, such as those labeled “Time limit exceeded”, “Runtime error”, or “Unanswered”, to retain only valid and meaningful submission data. These invalid records were considered noise and could skew the analysis if included.
• Removing Irrelevant Users: We excluded users with an insufficient number of answer sequences, as these would not provide enough data to model effectively. Additionally, non-student users, including administrators and virtual users, were removed to ensure that the dataset reflected genuine student learning behavior.
• Normalizing Sequence Lengths: Since the number of exercises answered by each student varied significantly, directly modeling answer sequences of different lengths would be challenging. To address this, we standardized the sequence length to a fixed value, ensuring that all input sequences were of uniform length for the model. This involved truncating longer sequences and padding shorter ones to the specified length.

The resulting preprocessed subset of the datasets is summarized in

Table 1. Partial information extracted from two datasets.

Data Sources	Students	Exercises	Interactions	Knowledge Concepts
Luogu	2081	3329	299,168	190
Codeforces	1685	8237	630,724	204

.

4.2. Environment

The success of a model analysis experiment heavily relies on having the right hardware and software environment available.

Table 2. Description of the experimental environment.

Configuration Environment	Configuration Parameters
Operating System	Windows 10 64-bit
GPU	RTX 3070ti
CPU	i7-13700H
Memory	16 GB
Programming language	Python 3.7
Deep learning framework	tensorflow 2.1.0
Python library	Scikit-learn, Numpy, Pandas

presents an overview of the key hardware and software environment configurations utilized in this study.

4.3. Evaluation Metrics and Baseline Methods

4.3.1. Evaluation Metrics

To validate the GPPKT model proposed in this research, we choose the area under the curve (AUC) and accuracy (ACC) as evaluation metrics. AUC offers a more precise evaluation of the model’s ability to differentiate positive and negative samples and is independent of threshold values. ACC is a fundamental performance metric in classification models. It quantifies the proportion of correctly classified samples relative to the total number of samples. In the context of KT, ACC is used to measure the accuracy of a model in predicting student learning progress.

4.3.2. Baseline Methods

To thoroughly verify the effectiveness of the GPPKT model, several classical and state-of-the-art KT models were selected as baseline methods. These baselines were chosen as they represent the evolution of knowledge tracing models over time, from the initial models that incorporated deep learning to their improved versions and the latest advances in attention mechanisms and interpretability. The baseline methods include:

• DKT [1]: A pioneering knowledge tracing method using recurrent neural networks.
• DKT+ [2]: An improved DKT model addressing input reconstruction and state fluctuations.
• DKVMN [3]: A memory network-based model with interpretable knowledge and student state representation.
• Deep-IRT [13]: Combines IRT with DKVMN for an interpretable deep knowledge tracing model.
• AKT [6]: Introduces the Rasch model and self-attention for better exercise and interaction modeling.
• ATKT [15]: Adds adversarial perturbations to LSTM-based sequences to reduce overfitting.
• IEKT [5]: Combines individual cognition and learning processes for accurate knowledge state tracing.
• SAKT [4]: Uses self-attention mechanisms in Transformer to handle sparse data in knowledge tracing.
• SAINT [35]: Extends SAKT with additional attention modules for deeper exercise–answer relationship modeling.

4.4. Model Training and Parameter Selection

We randomly divide the dataset; 80% is used as the training set, and 20% is used as the test set. We use the Adam optimizer to optimize all model parameters with the following settings: ${\beta }_1=0.9$, ${\beta }_2=0.999$, $\epsilon =1\times {10}^{-8}$, and a model learning rate of 0.001. In accordance with typical practices in the field of KT, the sequence length is set to $l=200$, considering the typical student answer sequence lengths and the average number of student answers. Finally, each model is trained for a total of 30 epochs.

4.5. Experiment Results

4.5.1. Comparative Experiment Analysis

After multiple experiments, the final performance of all models on the two datasets is shown in

Table 3. Comparative experiments.

Methods	Luogu		Codeforces
	AUC	ACC	AUC	ACC
DKT	0.8114	0.8397	0.6804	0.8574
DKT+	0.8104	0.8388	0.7037	0.8589
Deep-IRT	0.8020	0.8271	0.6702	0.8526
DKVMN	0.8058	0.8300	0.6828	0.8573
IEKT	0.8230	0.8309	0.7308	0.8567
SAKT	0.8301	0.8307	0.6708	0.8550
SAINT	0.8263	0.8254	0.6763	0.8577
AKT	0.8313	0.8394	0.7491	0.8595
ATKT	0.8380	0.8418	0.7543	0.8614
GPPKT (our)	0.8840	0.8472	0.7770	0.8799

, with the best results highlighted in bold. Baseline KT models provided by the pykt [36] library are used. Table 3 compares various KT models using two key metrics: AUC and ACC. The AUC measures the ability of a model to distinguish between correct and incorrect answers, with higher values indicating better performance. The ACC reflects the overall percentage of correct predictions made by the model. These metrics are critical for evaluating the effectiveness of KT models in predicting students’ future performance. Overall, the GPPKT model proposed in this study is shown to perform well on both datasets after the knowledge graph and the student answer sequence are fully considered.

On the Luogu dataset, an average improvement of 7.75% in AUC is exhibited by GPPKT compared to other methods, with a relative improvement of approximately 5.49% observed compared to the second-best method, ATKT. The ACC is recorded at 0.8472, showing an improvement of 1.51% over the average performance of other methods. On the Codeforces dataset, an average AUC improvement of 10.30% is demonstrated by GPPKT compared to other methods, with a relative improvement of about 3.01% seen compared to the second-best method, ATKT. The ACC is recorded at 0.8799, indicating an improvement of 2.53% over the average performance of other methods.

Additionally, it is observed that similar performance is exhibited by DKT and DKT+, which are among the first to use deep knowledge tracing. This is due to the fact that programming exercises primarily consist of subjective questions, and students’ understanding of KCs is relatively stable, leading to fewer fluctuations in the predicted state. DKT+ was developed specifically to address the problem of fluctuations in the prediction state. Although some improvement in preventing overfitting is provided by DKT+, it does not account for the inherent relationships between KCs.

Furthermore, similar performance is exhibited by Deep-IRT and DKVMN. On the Luogu dataset, AUC values of 0.8020 and 0.8058 are recorded, respectively, and on the Codeforces dataset, AUC values of 0.6702 and 0.6828 are recorded, respectively. DKVMN is designed to dynamically update the mastery levels of relevant KCs and performs well on objective exercises. However, in programming KT, where subjective exercises prevail and students often engage in continuous answering behavior, the impact of KC forgetting is not significant. Deep-IRT is an improvement upon the DKVMN model; although students’ learning abilities are also considered, it indirectly reveals the limitations of traditional KT methods when addressing personalized answers for subjective exercises. Both SAKT and SAINT are based on transformer models, and similar AUC values are observed for both. However, relationships between exercises and students’ learning abilities are not considered by either model.

A certain gap between IEKT and GPPKT is also observed. Although the IEKT model integrates individual cognition and learning processes, the relationship between KCs is not considered. It is evident that, besides GPPKT, the AKT and ATKT models demonstrate the best performance. These two models share the commonality of incorporating attention mechanisms. This indirectly highlights the significance of the attention mechanism introduced in this study for accurate prediction when modeling student answer sequences.

The superior performance of GPPKT compared to other models can be attributed to several key factors. First, the integration of knowledge graph structures into the modeling process by GPPKT is effectively achieved, capturing the intricate relationships between exercises and KCs. This enhances the model’s ability to accurately trace students’ knowledge states, particularly in domains with complex, interconnected knowledge. Moreover, the attention mechanism incorporated in GPPKT enables the model to focus on relevant parts of the answer sequences, leading to more accurate predictions. These factors collectively contribute to the significant improvements observed in both AUC and ACC.

4.5.2. Ablation Experiment Analysis

In order to verify the contribution of each part of the model proposed in this paper to the prediction of students’ answers under the same other conditions, we conduct ablation experiment analysis to verify the rationality of GPPKT.

Table 4. Ablation experiments.

Methods	AUC	ACC
(1) LSTM	0.8097	0.7954
(2) LSTM + VAE	0.8576	0.8343
(3) LSTM + KG	0.8654	0.8361
(4) LSTM + ABI	0.8679	0.8380
(5) LSTM + VAE + ABI	0.8762	0.8425
(6) LSTM + VAE + KG	0.8749	0.8392
(7) LSTM + KG + ABI	0.8703	0.8411
(8) LSTM + VAE + KG + ABI	0.8840	0.8472

presents the results of the ablation experiments, which examine the impact of each component of the GPPKT model on its performance. The methods listed represent different combinations of model components: VAE (Variational Autoencoder), KG (Knowledge Graph), and ABI (Ability Index). The AUC and ACC columns show how each combination affects the model’s predictive accuracy and overall performance. The analysis of these values is essential to understand the significance of each component in enhancing the model’s capability. We choose the Luogu dataset for the ablation experiments. The results of the ablation experiments are shown in Table 4 which demonstrates the permutations of each method proposed in this study, where VAE indicates that the student answer sequences processed by VAE are used, KG indicates that the knowledge graph is used, and ABI indicates that students’ learning abilities are used.

Through the ablation experiments in Table 4, we draw the following conclusions: Comparing methods (2) and (5), it is observed that the consideration of students’ personalized learning abilities has a positive impact on the model’s prediction because different students have varying levels of mastery over different KCs. When methods (3) and (6) are compared, it is found that the incorporation of a VAE to process students’ answer sequences has a positive impact on the model’s predictive capabilities. This is because each student’s understanding of the exercises gradually improves during the learning process. Finally, when methods (4) and (7) are compared, it is seen that the introduction of a knowledge graph also benefits the model’s predictions because KCs are interconnected, and the use of a knowledge graph allows exercise information to be better considered.

In summary, the inclusion of a knowledge graph, the use of a VAE to represent answer sequences, and the consideration of students’ personalized learning abilities all have a positive impact on the model’s performance.

4.6. Visualization

4.6.1. Knowledge Graph Visualization

We extract the changes in the knowledge state of a student during the process of answering an exercise, as shown in Figure 6, which reflects the changes in the student’s mastery of other related KCs when answering the KC of “tree”.

In the heatmap, the color intensity represents the depth of the student’s mastery: lighter colors indicate lower mastery levels, while darker colors signify a deeper understanding. As the student progresses through time steps 0 to 7, the heatmap block corresponding to the KC “Tree” gradually deepens in color. This deepening suggests that the student’s understanding of the “Tree” concept is improving steadily. Simultaneously, the heatmap blocks for related KCs such as “Tree Structures”, “Huffman Tree”, “Minimum Spanning Tree”, and “Applications of Trees” also show deepening colors. This indicates that the student’s mastery of these interconnected concepts is strengthening alongside their understanding of “Tree”.

Notably, for the more complex concept “Huffman Tree”, the heatmap shows a slower change in color intensity compared to “Tree”. This slower deepening reflects the greater difficulty the student experiences in mastering “Huffman Tree”, likely due to the complexity of its construction algorithm.

As the student continues answering exercises during time steps 8 to 15, 16 to 23, and 24 to 31, the heatmap reveals further changes in their understanding of various KCs. During time steps 16 to 23, the heatmap block for “Huffman Tree” initially deepens but then lightens, indicating that the student’s grasp of this KC is unstable and errors might occur in subsequent answers. This pattern is mirrored in the heatmap blocks for other related KCs, which also show a cycle of deepening and lightening colors. These fluctuations in color illustrate the dynamic nature of the student’s knowledge state, which shifts in response to their answering performance.

Overall, the evolution of colors in the heatmap provides a visual representation of how the student’s mastery levels for specific KCs, as well as related concepts, change over time. The comprehensive nature of these changes underscores the effectiveness of incorporating a knowledge graph in tracing and visualizing the student’s learning process.

4.6.2. Student Learning Abilities Visualization

We visualize the learning abilities of three students across six exercises in Figure 7a. In this heatmap, darker colors represent higher learning ability on the respective exercise, while lighter colors indicate lower learning ability. Among the three students selected, we observe different overall levels of learning ability: excellent, moderate, and lower.

Students with excellent learning abilities demonstrate consistently high learning ability values on almost all exercises, mostly around 0.9. Students with moderate learning abilities exhibit moderate ability values on some exercises, with some around 0.7 and one exercise at 0.27. While students with lower learning abilities generally display relatively lower ability values on most exercises, with half around 0.27 and a smaller portion around 0.7.

To further analyze the learning process, we examine how the knowledge states of these students evolve over time on two different exercises, as shown in Figure 7b. Here, the heatmap traces changes in their mastery of KCs. The darkening of heatmap blocks over time indicates an increase in the student’s mastery level.

For students with higher learning abilities, the color intensifies quickly, indicating significant and rapid progress in mastering the relevant KCs. Students with moderate learning abilities also experience a darkening of color, though this process is more gradual. Despite the slower progression, these students still achieve noticeable improvements in their knowledge states within a given period. In contrast, students with lower learning abilities show only slight darkening of the heatmap blocks, reflecting limited progress. Even though their knowledge states improve somewhat during the learning process, the final values, around 0.31 and 0.29, suggest they are still far from fully mastering the relevant KCs.

From another perspective, students with higher learning ability need a shorter time to reach a relatively higher knowledge state, while students with lower learning ability need more time to reach a similar level. These findings further highlight the close relationship between learning ability and knowledge state.

4.6.3. Personalized Answer Sequence Visualization

We extract a partial answer sequence from a student, as shown in Figure 8a, where each row represents the student’s responses to different programming exercises. In this figure, lighter colors represent incorrect answers (coded as 0), while darker colors represent correct answers (coded as 1). It is evident that the second and eighth exercises were correctly answered by the student on their first attempt. However, for the third, fifth, and seventh exercises, the student consistently answered incorrectly. There are also cases where the student initially answered incorrectly but eventually arrived at the correct answer after multiple attempts. For example, in the sixth exercise, there was one incorrect attempt before the correct answer was given. The fourth exercise required five attempts to arrive at the correct answer, while the first exercise took 15 attempts before a correct answer was achieved.

The left half of Figure 8b represents the latent variables obtained after processing the student’s answer sequence through the VAE’s encoder, which is a part of our model’s input and clearly reflects the personalized and refined representation of the student’s continuous responses to each programming exercise. The higher value indicates the faster mastery of the problem-solving process for that particular exercise. We also visualize the output of the hidden layer after passing through the VAE’s decoder, as shown in the right half of Figure 8b. It can be observed that the VAE effectively reconstructs the latent variables. For exercises where the student initially answered incorrectly and then correctly after multiple attempts, the gradient of the values reflects the student’s gradually improving mastery of the exercise throughout the answer sequence. Additionally, the different number of attempts is differentially reflected in the answer sequence.

This detailed visualization and analysis underscore the effectiveness of the VAE in capturing and representing the complexity of students’ learning processes, providing valuable insights into their knowledge states.

5. Conclusions

In this paper, GPPKT is proposed to address two significant challenges in programming knowledge tracing: the inaccurate representation of exercises and the neglect of student answer sequence information. A programming knowledge graph was constructed based on authoritative learning resources, and the RotatE-TA knowledge graph embedding method was introduced to vectorize exercise representations. The VAE was employed to model students’ answer sequences on the same exercise, effectively capturing students’ learning behavior. Additionally, students’ learning ability differences were obtained through the IRT to enhance the information provided by students’ answer sequences. LSTM and an attention mechanism were used to adaptively aggregate hidden states, while a gating mechanism was introduced to balance the student’s historical and current knowledge state for performance prediction. Extensive experiments were conducted on two real-world programming datasets to evaluate GPPKT. The results demonstrate that GPPKT outperforms state-of-the-art methods in the field of programming knowledge tracing, with an average improvement of 9.03% in AUC and 2.02% in ACC across both datasets.

The GPPKT model is expected to stimulate the development of KT research within the programming field. Key contributions of this work include the integration of knowledge graphs to enhance exercise representation, the application of VAE for modeling student behavior, and the utilization of IRT to differentiate student learning abilities. These innovations have been shown to significantly improve the accuracy and effectiveness of programming knowledge tracing.

In future work, we will integrate recommendation algorithms to more accurately suggest exercises and learning materials to students, aiming to achieve intelligent and adaptive programming learning. This has important implications for personalized learning and could enhance the effectiveness of programming education. Additionally, we plan to incorporate features such as code submissions and other relevant data into our model to further refine the understanding of student learning processes and outcomes.These advancements in programming knowledge tracing have the potential to transform educational practices by providing educators with deeper insights into student learning trajectories and enabling more targeted interventions. Our work not only contributes to the academic field but also offers practical applications for improving the quality of programming education.