Online Learning Strategy Induction through Partially Observable Markov Decision Process-Based Cognitive Experience Model

Abstract

Inducing learning strategies is a crucial component of intelligent tutoring systems. Previous research has predominantly focused on the induction of offline learning strategies. Although the existing offline learning strategy induction methods can also be used for real-time updates of learning strategies, their update efficiency is not high, making it difficult to capture the characteristics exhibited by learners during the learning process in a timely manner. With the superior performance of the Partially Observable Markov Decision Process (POMDP), this paper proposes a POMDP-based cognitive experience model, which can be quickly updated during interactions and enables the real-time induction of learning strategies by weighting the learning experiences of different learners. Experimental results demonstrate that the learning strategies induced by PCEM are more personalized and exhibit superior performance.

1. Introduction

Intelligent tutoring systems (ITSs) provide personalized support in learners’ cognitive process when the learners are engaged with knowledge concept learning based on the integration of computer science and educational principles [1]. The systems utilize artificial intelligence technology and educational models to offer customized learning paths, personalized feedback, and support, in order to optimize learning outcomes [2,3]. Depending on the type of learning resources, the tutoring methods employed by ITS vary. Figure 1 illustrates one such method, where the system and the learner engage in a repetitive cycle: (a) the system chooses a question, (b) the learner answers the question, and (c) the system checks the answer and reads the explanation. The repetitive cycle is known as the exercise-based cognitive process, which is the primary focus of the discussion of this article. In this process, the key to improving the learner’s knowledge state lies in selecting appropriate exercises from a vast pool of options, which is referred as the learning strategy. Efficiently inducing the optimal learning strategy is the focus of this study.

Studies on optimizing learners’ learning strategies are primarily divided into two categories—model-free and model-based learning strategy optimization. The studies about model-free learning strategy optimization mainly utilize techniques based on reinforcement learning (RL) [4,5,6,7] and its improvements, such as hierarchical reinforcement learning (HRL) [8,9] and deep reinforcement learning (DRL) [10,11,12,13,14]. However, these methods do not explicitly represent how learners understand knowledge concepts in the cognitive processes, meaning they do not model learners’ cognitive processes. Consequently, the resulting learning strategies may be unexplainable, which contradicts the principles of evidence-based learning [15].

In contrast, the model-based methods mainly utilize techniques based on a partially observable Markov decision process (POMDP) [16]. Early related studies primarily explored the feasibility of POMDP in cognitive modeling and learning strategy induction [17]. Subsequent studies improved modeling methods and strategy induction techniques, emphasizing the personalization of models and strategies [18,19,20,21]. Additionally, the online application of model-based methods also deserves attention.

Unlike the need for online planning in POMDPs due to the excessively large planning space, the online methods required by ITS focus more on the real-time update of cognitive models and learning strategies. Specifically, learners’ answers can serve not only as the basis for the next decision, but also as new information to update the cognitive model. In other words, in this scenario, the model and the corresponding optimal strategy are both dynamic. However, achieving this requires addressing the following difficulties.

• Generally, a large amount of calculation is required to complete the model update, which is time-consuming and contrary to the required real-time update.
• The information provided by a single answer is very limited, and will be diluted by other historical records.

The first issue arises because traditional methods demand all historical data for model training to ensure accuracy. In real-time interactions, this would mean learners must collectively wait for model updates. Using all historical data compensates for missing information in individual datapoints. Alternatively, although individual learners’ answer records may be limited, they deeply reflect personal cognitive experiences. We refer to the POMDP model trained solely from the answer records of individual learners as the POMDP-based cognitive experience model (PCEM). For specific temporal answer sequences, which represent an advisee, each cognitive experience model can be regarded as a adviser, and the final learning strategy is derived by synthesizing the suggestions of these advisers. Considering the difference in cognitive abilities among different learners, the weight of each adviser’s suggestion in the final learning strategy should not be the same, but should be determined by the similarity in cognitive abilities between the adviser and the advisee, which precisely addresses the second issue mentioned above.

Our main contributions are summarized as follows:

• To achieve a personalized online update of the model and strategy, we propose the POMDP-based cognitive experience model (PCEM). Unlike models that accurately reflect learners’ cognitive abilities, the cognitive experience model emphasizes learners’ subjective experiences in the cognitive process. Additionally, we introduce an adviser–advisee mechanism to derive learning strategies corresponding to the PCEM.
• This paper proposes new methods for parameter learning and strategy solutions to the PCEM. The parameter learning draws on the ideas of HCPM and makes appropriate modifications based on the PCEM characteristics. The strategy solving method incorporates the similarity between cognitive experiences to improve the performance.
• We demonstrate the PCEM performance in real-time update of learning strategies in several real-world knowledge concept learning domains.

The remaining sections of this article are organized as follows. We start to review the related work of learning strategy induction methods in Section 2. We present the HCPM background knowledge in Section 3.1, and provide a detailed exposition of the parameter learning and strategy induction methods for the PCEM model in Section 3.3 and Section 3.4. Section 4 shows the experimental results and demonstrates the performance of PCEM. Section 5 summarizes this work and discusses further research.

2. Related Works

Obtaining learning/teaching strategies through acquiring or constructing cognitive models can be challenging and expensive. Therefore, researchers have embraced model-free methods for inducing such strategies, which derive them directly from learners’ answer data without relying on precise cognitive process modeling. Tang et al. [5] proposed a RL-based method for personalized learning that adaptively selects learning materials and optimizes recommendation strategies under uncertain information. Zhou et al. [9] introduced and applied an offline, off-policy HRL framework based on Gaussian processes, demonstrating higher effectiveness in guiding teaching strategies compared to general RL through experiments. Ju et al. [11] proposed critical-RL to identify critical pedagogical decisions, and through classroom studies, found that, for certain learners, critical strategies at key decision points were significantly more effective than random strategies. Huang et al. [12] introduced the deep reinforcement learning framework for exercise recommendation (DRE), utilizing two Exercise Q-Networks (EQNM and EQNR) and three domain-specific rewards, showing superior recommendation performance on real datasets.

Zhou et al. [6] demonstrated the improvement of learner–system interaction by combining RL with human-written explanations, with experiments proving that personalized RL decisions paired with explanations are more effective than using RL decisions or explanations alone. Ausin et al. [13,14] showed how combining data-driven methods (such as DRL) with educational strategies, along with simple explanations, enhanced learner interaction and learning performance with ITSs, revealing the effectiveness of learners’ problem-level decisions. Their team also developed the InferNet method to address temporal delayed reward issues, showing its effectiveness in inferring true immediate rewards and significantly improving learner learning outcomes in simulated tasks and empirical studies compared to immediate rewards and previous methods. Kubotani et al. [7] proposed a RL-based teaching framework that optimizes teaching strategies through internal modeling of learners, even with limited learner learning history, and validated its effectiveness through mathematical model experiments.

Model-based learning strategy induction methods help optimize and develop learning/teaching strategies by constructing and using cognitive models. These methods offer theoretical support and data-driven personalized learning paths, improving learning outcomes. However, building and maintaining accurate cognitive models is costly. Considering that deciding on learning/teaching decisions involves reasoning and balancing multiple priorities, Rafferty et al. [17] addressed this by developing a POMDP planning framework. This framework provided a method for selecting learning/teaching actions based on learning models, domain structure, and teaching goals. It demonstrated the ability to select actions in real-time in moderately sized domains and showed that, despite mismatches between learner models and actual learners, POMDP strategies could accelerate learning in concept learning tasks, outperforming other alternative methods. Ramachandran et al. [18] designed the Assistive Tutor POMDP (ATPOMDP) to provide personalized support for learners practicing difficult math concepts over multiple tutoring sessions and demonstrated its effectiveness, significantly enhancing learning gains.

Nioche et al. [19] proposed a modular framework combining online inferences for each user and items with online planning considering learning time constraints, expanding model-based tutoring system research. This framework showed effectiveness in both simulated and real learners, particularly in adapting to changes in learning abilities and item difficulty. Gao et al. [20,21] used ITS as a platform to introduce a new strategy induction method and practice-based cognitive modeling, adopting a POMDP model to optimize learning strategies and refining them using information entropy techniques. This approach aimed to explain learners’ learning performance in knowledge concepts of interest and provide personalized learning strategies. Recognizing that personalized modeling was more suitable than traditional methods for tutoring individual learners on learning knowledge concepts, the team also introduced the homomorphic POMDP (H-POMDP) model and a new cognitive modeling approach to induce learning strategies for individual learners, demonstrating its performance in multiple knowledge concept learning domains.

3. POMDP-Based Cognitive Experience Modeling and Online Personalized Learning Planning

In this section, we provide an overview of the background knowledge pertaining to HCPM [21], which lays the foundation for the development of PCEM. Following that, we delineate the definition of PCEM and introduce the associated method for updating the real-time model and learning strategy.

To facilitate subsequent explanations, we first define the relevant symbols. In the exercise-based cognitive process, learners are denoted by $\mathrm{Le}$, exercises by $\mathrm{Ex}$, and knowledge concepts by $\mathrm{Kc}$, with their respective quantities represented by $\mathrm{L}$, $\mathrm{M}$, and $\mathrm{N}$. A learner’s attempt at a question is considered a learning action, denoted as $\mathrm{a}$, and the correct or incorrect of the answer is denoted as $\mathrm{o}$. Thus, the cognitive process of the $i_{th}$ learner can be represented by the sequence of answer data as ${\mathrm{h}}_i=(({\mathrm{a}}_{i,1},{\mathrm{o}}_{i,1}),({\mathrm{a}}_{i,2},{\mathrm{o}}_{i,2}),\cdots ,({\mathrm{a}}_{i,{\mathrm{T}}_i},{\mathrm{o}}_{i,{\mathrm{T}}_i}))$, where ${\mathrm{T}}_i$ is the number of attempts made by the $i_{th}$ learner. The cognitive processes of all learners can be represented as $\mathbb{H}=\{{\mathrm{h}}_1,{\mathrm{h}}_2,...,{\mathrm{h}}_{\mathrm{L}}\}$.

3.1. Background Knowledge

Figure 2 shows a four-step POMDP model representing the cognitive process of a given learner.

Given a knowledge learning domain, a POMDP-based cognitive model is represented as a 6-tuple $(S,A,\Omega ,T,O,R)$, where S is a set of knowledge states, A is a set of learning actions, $\Omega$ is a set of observations related to the learner’s performance in learning actions, T is a state transition function that indicates changes in knowledge states caused by the learning actions, O is an observation function that describes the possible performance of a learner in a specific learning action given a specific knowledge state, and R is a reward function customized according to specific learning goals.

Compared with the POMDP-based cognitive models that assume the same cognitive ability for learners, HCPM can differentiate between the learning abilities of different individuals, thereby helping to achieve personalized learning strategy optimization. HCPM comprises several POMDP-based cognitive models $\{{\mathcal{M}}_1,{\mathcal{M}}_2,\cdots ,{\mathcal{M}}_{\mathrm{K}}\}$ that satisfy the following properties ($\Phi$, $\Psi$):

$$\begin{array}{cc}\hfill \Phi \leftarrow \{& S_1=S_2=\cdots =S_{\mathrm{K}}=S,A_1=A_2=\cdots =A_{\mathrm{K}}=A,{\Omega }_1={\Omega }_2=\cdots ={\Omega }_{\mathrm{K}}=\Omega \}\hfill \\ \hfill \Psi \leftarrow \{& T_1\ne T_2\ne \cdots \ne T_{\mathrm{K}},O_1=O_2=\cdots =O_{\mathrm{K}}=O,R_1=R_2=\cdots =R_{\mathrm{K}}=R\}\hfill \end{array}$$

These properties enable the HCPM to represent various performances among individuals with different cognitive abilities in the same cognitive task.

This article presents the novel PCEM, evolved from HCPM. It shifts away from precisely describing the cognitive process, emphasizing rapid updates and documentation of learners’ experiences for real-time strategy optimization.

3.2. PCEM Specification

The number of POMDPs $\mathrm{K}$ included in HCPM needs to be determined. Noting that learners’ learning abilities vary to some extent, $\mathrm{K}$ can be set to the number of learners in the data, so that each trained POMDP reflects the learning experience of each learner. Thus, we can define PCEM as follows.

Definition 1. 

Given the number of learners $\mathrm{L}$, each learner is modeled by a POMDP, we have $\{{\mathcal{M}}_1,{\mathcal{M}}_2,\cdots ,{\mathcal{M}}_{\mathrm{L}}\}$, and these POMDPs satisfy the following properties (Φ, Ψ). We refer to these $\mathrm{L}$ POMDPs as the PCEM model ${\mathcal{M}}^{\mathrm{E}}=\{{\mathcal{M}}_1,{\mathcal{M}}_2,\cdots ,{\mathcal{M}}_{\mathrm{L}}\}$.

$$\begin{array}{cc}\hfill \Phi \leftarrow \{& S_1=S_2=\cdots =S_{\mathrm{L}}=S,A_1=A_2=\cdots =A_{\mathrm{L}}=A,{\Omega }_1={\Omega }_2=\cdots ={\Omega }_{\mathrm{L}}=\Omega \}\hfill \\ \hfill \Psi \leftarrow \{& T_1\ne T_2\ne \cdots \ne T_{\mathrm{L}},O_1=O_2=\cdots =O_{\mathrm{L}}=O,R_1=R_2=\cdots =R_{\mathrm{L}}=R\}\hfill \end{array}$$

Similarly to HCPM, all $\mathrm{L}$ POMDPs in PCEM share the same state space S, action space A, and observation space $\Omega$, and have identical observation O and reward functions R, but they have different transition functions $T_1,T_2,\cdots ,T_{\mathrm{L}}$. Unlike HCPM, each POMDP in PCEM can only reflect the cognitive experience of each individual learner. Even if the response sequence used to train a particular POMDP is sufficiently long, it is difficult to approximate the cognitive model. Specifically, a learner’s cognitive process often has an irreversible nature—once a learner has mastered certain knowledge concepts, it is challenging for the learner to forget these concepts during continuous learning. Therefore, even relatively long response sequences often contain only a small amount of information that reflects changes in the learner’s knowledge state, while including a large amount of information about how the learner maintains the knowledge state in the later stages.

3.3. PCEM Parameter Learning

Given a set of learning activity data $\mathbb{H}=\left\{{\mathrm{h}}_1,{\mathrm{h}}_2,\cdots ,{\mathrm{h}}_{\mathrm{L}}\right\}$, where each data point ${\mathrm{h}}_i=\left\{({\mathrm{a}}_{i,1},{\mathrm{o}}_{i,1}),({\mathrm{a}}_{i,2},{\mathrm{o}}_{i,2}),\cdots ,({\mathrm{a}}_{i,{\mathrm{T}}_i-1},{\mathrm{o}}_{i,{\mathrm{T}}_i-1})\right\}$ represents the learner ${\mathrm{Le}}_i$, the objective is to learn the cognitive experience model ${\mathcal{M}}_l=(D_l,T_l,O)$ ($D_l$ is the distribution of the initial state) for each learner. We divide $\mathbb{H}$ into observation sequences $\mathbb{O}=\left\{{\mathrm{O}}_1,{\mathrm{O}}_2,\cdots ,{\mathrm{O}}_{\mathrm{L}}\right\}$ (${\mathrm{O}}_i$ is the observation sequence of the learner ${\mathrm{Le}}_i$) and action sequences $\mathbb{A}=\left\{{\mathrm{A}}_1,{\mathrm{A}}_2,\cdots ,{\mathrm{A}}_{\mathrm{L}}\right\}$ (${\mathrm{A}}_i$ is the action sequence of the learner ${\mathrm{Le}}_i$), with the corresponding state sequences being $\mathbb{S}=\left\{{\mathrm{S}}_1,{\mathrm{S}}_2,\cdots ,{\mathrm{S}}_{\mathrm{L}}\right\}$ (${\mathrm{S}}_i$ is the state sequence of the learner ${\mathrm{Le}}_i$). Then, the probability for all observation sequences $\mathbb{O}$ is given by

$$\begin{array}{c}\hfill P(\mathbb{O}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A})=\sum _{\mathbb{S}}P(\mathbb{O}\mid \mathbb{S},{\mathcal{M}}^{\mathrm{E}},\mathbb{A})P(\mathbb{S}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A})\end{array}$$

where $P(·)$ and $P(·|·)$ represent the probability function and conditional probability function, respectively. The parameter learning can be achieved through the EM algorithm.

• Determine the log-likelihood function of the complete data.

The complete data are

$$\begin{array}{cc}\hfill (\mathbb{A},\mathbb{O},\mathbb{S})=& ({\mathrm{a}}_{1,1},{\mathrm{a}}_{1,2},\dots ,{\mathrm{a}}_{1,{\mathrm{T}}_1},\dots ,{\mathrm{a}}_{\mathrm{L},1},{\mathrm{a}}_{\mathrm{L},2},\dots ,{\mathrm{a}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},{\mathrm{o}}_{1,1},{\mathrm{o}}_{1,2},\dots ,{\mathrm{o}}_{1,{\mathrm{T}}_1},\dots ,{\mathrm{o}}_{\mathrm{L},1},{\mathrm{o}}_{\mathrm{L},2},\dots ,{\mathrm{o}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},\hfill \\ & {\mathrm{s}}_{1,1},{\mathrm{s}}_{1,2},\dots ,{\mathrm{s}}_{1,{\mathrm{T}}_1},\dots ,{\mathrm{s}}_{\mathrm{L},1},{\mathrm{s}}_{\mathrm{L},2},\dots ,{\mathrm{s}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}})\hfill \end{array}$$

and the log-likelihood function of the complete data are $logP(\mathbb{O},\mathbb{S}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A})$.

• E-step of the EM algorithm: Calculate the Q function $Q({\mathcal{M}}^{\mathrm{E}},\overline{{\mathcal{M}}^{\mathrm{E}}})$,

$$\begin{array}{cc}\hfill Q({\mathcal{M}}^{\mathrm{E}},\overline{{\mathcal{M}}^{\mathrm{E}}})=& \sum _{\mathbb{S}}logP(\mathbb{O},\mathbb{S}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A})P(\mathbb{S}\mid \mathbb{O},\overline{{\mathcal{M}}^{\mathrm{E}}},\mathbb{A})\hfill \\ \hfill =& \sum _{\mathbb{S}}logP(\mathbb{O},\mathbb{S}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A}){\displaystyle \frac{P(\mathbb{O},\mathbb{S}\mid \overline{{\mathcal{M}}^{\mathrm{E}}},\mathbb{A})}{P(\mathbb{O}\mid \overline{{\mathcal{M}}^{\mathrm{E}}},\mathbb{A})}}\hfill \end{array}$$

where $\overline{{\mathcal{M}}^{\mathrm{E}}}$ are the current estimates of the parameters, ${\mathcal{M}}^{\mathrm{E}}$ are the parameters to be maximized, and $P(\mathbb{O}\mid \overline{{\mathcal{M}}^{\mathrm{E}}},\mathbb{A})$ is a constant that can be ignored.

$$\begin{array}{cc}\hfill P(\mathbb{O},\mathbb{S}\mid {\mathcal{M}}^{\mathrm{E}},\mathbb{A})=& D_1\left({\mathrm{s}}_{1,1}\right)O({\mathrm{s}}_{1,1},{\mathrm{a}}_{1,1},{\mathrm{o}}_{1,1})T_1({\mathrm{s}}_{1,1},{\mathrm{a}}_{1,1},{\mathrm{s}}_{1,2})\hfill \\ & \dots O(s_{1,{\mathrm{T}}_1},a_{1,{\mathrm{T}}_1},{\mathrm{o}}_{1,{\mathrm{T}}_1})T_1({\mathrm{s}}_{1,{\mathrm{T}}_1},{\mathrm{a}}_{1,{\mathrm{T}}_1},{\mathrm{s}}_{1,{\mathrm{T}}_1+1})\hfill \\ & D_2\left({\mathrm{s}}_{2,1}\right)O({\mathrm{s}}_{2,1},{\mathrm{a}}_{2,1},{\mathrm{o}}_{2,1})T_2({\mathrm{s}}_{2,1},{\mathrm{a}}_{2,1},{\mathrm{s}}_{2,2})\hfill \\ & \dots O(s_{2,{\mathrm{T}}_2},a_{2,{\mathrm{T}}_2},{\mathrm{o}}_{2,{\mathrm{T}}_2})T_2({\mathrm{s}}_{2,{\mathrm{T}}_2},{\mathrm{a}}_{2,{\mathrm{T}}_2},{\mathrm{s}}_{2,{\mathrm{T}}_2+1})\hfill \\ & \dots \hfill \\ & D_{\mathrm{L}}\left({\mathrm{s}}_{\mathrm{L},1}\right)O({\mathrm{s}}_{\mathrm{L},1},{\mathrm{a}}_{\mathrm{L},1},{\mathrm{o}}_{\mathrm{L},1})T_1({\mathrm{s}}_{\mathrm{L},1},{\mathrm{a}}_{\mathrm{L},1},{\mathrm{s}}_{\mathrm{L},2})\hfill \\ & \dots O(s_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},a_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},{\mathrm{o}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}})T_{\mathrm{L}}({\mathrm{s}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},{\mathrm{a}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}},{\mathrm{s}}_{\mathrm{L},{\mathrm{T}}_{\mathrm{L}}+1})\hfill \end{array}$$

Thus, the function $Q({\mathcal{M}}^{\mathrm{E}},\overline{{\mathcal{M}}^{\mathrm{E}}})$ can be expressed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q({\mathcal{M}}^{\mathrm{E}},\overline{{\mathcal{M}}^{\mathrm{E}}})=& \sum _{{\mathrm{S}}_1}log{D}_1({\mathrm{s}}_{1,1})P({\mathrm{O}}_1,{\mathrm{S}}_1\mid \overline{{\mathcal{M}}_1},{\mathrm{A}}_1)+...+\sum _{{\mathrm{S}}_{\mathrm{L}}}log{D}_{\mathrm{L}}({\mathrm{s}}_{\mathrm{L},1})P({\mathrm{O}}_{\mathrm{L}},{\mathrm{S}}_{\mathrm{L}}\mid \overline{{\mathcal{M}}_{\mathrm{L}}},{\mathrm{A}}_{\mathrm{L}})\hfill \\ \hfill +& \sum _{{\mathrm{S}}_1}(\sum _{t=1}^{{\mathrm{T}}_1}log{T}_1({\mathrm{s}}_{1,t},{\mathrm{a}}_{1,t},{\mathrm{s}}_{1,t+1}))P({\mathrm{O}}_1,{\mathrm{S}}_1\mid \overline{{\mathcal{M}}_1},{\mathrm{A}}_1)\hfill \\ \hfill +& ...\hfill \\ \hfill +& \sum _{{\mathrm{S}}_{\mathrm{L}}}(\sum _{t=1}^{{\mathrm{T}}_{\mathrm{L}}}log{T}_{\mathrm{L}}({\mathrm{s}}_{\mathrm{L},t},{\mathrm{a}}_{\mathrm{L},t},{\mathrm{s}}_{\mathrm{L},t+1}))P({\mathrm{O}}_{\mathrm{L}},{\mathrm{S}}_{\mathrm{L}}\mid \overline{{\mathcal{M}}_{\mathrm{L}}},{\mathrm{A}}_{\mathrm{L}})\hfill \\ \hfill +& \sum _{\mathbb{S}}(\sum _{i=1}^{\mathrm{L}}\sum _{t=1}^{{\mathrm{T}}_i}logO({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{o}}_{i,t}))P({\mathrm{O}}_i,{\mathrm{S}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\hfill \end{array}\end{array}$$

(1)

• M-step of the EM algorithm: maximize the Q-function $Q({\mathcal{M}}^{\mathrm{E}},\overline{{\mathcal{M}}^{\mathrm{E}}})$ to estimate the model parameters.

Since the parameters to be maximized appear separately in $(2\mathrm{L}+1)$ terms in Equation (1), it is only necessary to maximize each term individually.

(a)

The first $\mathrm{L}$ terms of Equation (1) can be expressed as

$$\begin{array}{c}\hfill \sum _{{\mathrm{S}}_i}log{D}_i\left({\mathrm{s}}_{i,1}\right)P({\mathrm{O}}_i,{\mathrm{S}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)=\sum _{s\in S}log{D}_i\left(s\right)P({\mathrm{O}}_i,s_{i,1}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\end{array}$$

Note that $D_i\left(s\right)$ satisfies the constraint ${\sum }_{s\in S}{D}_i\left(s\right)=1$, we use the Lagrange multipliers and express the Lagrange function as

$$\begin{array}{c}\hfill \sum _{s\in S}log{D}_i\left(s\right)P({\mathrm{O}}_i,s_{i,1}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)+\gamma (\sum _{s\in S}{D}_i\left(s\right)-1)\end{array}$$

Take the partial derivatives of it and set the results to 0,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial }{\partial D_i\left(s\right)}}[\sum _{s\in S}log{D}_i\left(s\right)P({\mathrm{O}}_i,s_{i,1}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)+\gamma (\sum _{s\in S}{D}_i\left(s\right)-1)]=0\end{array}$$

$$\begin{array}{c}\hfill P({\mathrm{O}}_i,s_{i,1}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)+\gamma D_i\left(s\right)=0\end{array}$$

(2)

Sum over s to obtain $\gamma$

$$\begin{array}{c}\hfill \gamma =-P({\mathrm{O}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\end{array}$$

By substituting it into Equation (2), we obtain the $D_i\left(s\right)$

$$\begin{array}{c}\hfill D_i\left(s\right)={\displaystyle \frac{P({\mathrm{O}}_i,s_{i,1}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)}{P({\mathrm{O}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)}}\end{array}$$

(3)

(b)

The $(\mathrm{L}+1)$th to $\left(2\mathrm{L}\right)$th terms of Equation (1) can be expressed as

$$\begin{array}{cc}& \sum _{{\mathrm{S}}_i}\left(\sum _{t=1}^{{\mathrm{T}}_i}log{T}_i({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{s}}_{i,t+1})\right)P({\mathrm{O}}_i,{\mathrm{S}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\hfill \\ \hfill =& \sum _{s,s^{\prime }\in S}\left(\sum _{t=1}^{{\mathrm{T}}_i}log{T}_i({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{s}}_{i,t+1})\right)P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s,{\mathrm{s}}_{i,t+1}=s^{\prime }\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\hfill \end{array}$$

Similarly to the first $\mathrm{L}$ terms, applying the Lagrange multipliers with the constraint ${\sum }_{s^{\prime }\in S}{T}_i(s,a,s^{\prime })=1$ (only when ${\mathrm{a}}_{i,t}=a$, the partial derivative of $T_i({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{s}}_{i,t+1})$ with respect to $T_i(s,a,s^{\prime })$ is not 0, denoted as $\mathbb{I}({\mathrm{a}}_{i,t}=a)$),

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill T_i(s,a,s^{\prime })=& {\displaystyle \frac{{\sum }_{t=1}^{{\mathrm{T}}_i}P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s,{\mathrm{s}}_{i,t+1}=s^{\prime }\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\mathbb{I}({\mathrm{a}}_{i,t}=a)}{{\sum }_{t=1}^{{\mathrm{T}}_i}P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\mathbb{I}({\mathrm{a}}_{i,t}=a)}}\hfill \\ \hfill \mathbb{I}(\ast )& =\left\{\begin{array}{c}1,\ast istrue\hfill \\ 0,\ast isfalse\hfill \end{array}\right.\hfill \end{array}\end{array}$$

(4)

where $\mathbb{I}(\ast )$ is a indicator function.

(c)

The $(2\mathrm{L}+1)$th term can be expressed as

$$\begin{array}{cc}& \sum _{\mathbb{S}}(\sum _{i=1}^{\mathrm{L}}\sum _{t=1}^{{\mathrm{T}}_i}logO({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{o}}_{i,t}))P({\mathrm{O}}_i,{\mathrm{S}}_i\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\hfill \\ \hfill =& \sum _{s\in S}(\sum _{i=1}^{\mathrm{L}}\sum _{t=1}^{{\mathrm{T}}_i}logO({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{o}}_{i,t}))P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\hfill \end{array}$$

Similarly, we use the Lagrange multipliers with the constraint ${\sum }_{o\in \Omega }O(s,a,o)=1$. Note that the partial derivative of $O({\mathrm{s}}_{i,t},{\mathrm{a}}_{i,t},{\mathrm{o}}_{i,t})$ with respect to $O(s,a,o)$ is not 0 only when ${\mathrm{a}}_{i,t}=a$ and ${\mathrm{o}}_{i,t}=o$, denoted as $\mathbb{I}({\mathrm{a}}_{i,t}=a)$ and $\mathbb{I}({\mathrm{o}}_{i,t}=o)$

$$\begin{array}{c}\hfill \begin{array}{c}\hfill O\left(\mathrm{s},\mathrm{a},\mathrm{o}\right)={\displaystyle \frac{{\sum }_{i=1}^{\mathrm{L}}{\sum }_{t=1}^{{\mathrm{T}}_i}P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\mathbb{I}({\mathrm{a}}_{i,t}=a)\mathbb{I}({\mathrm{o}}_{i,t}=o)}{{\sum }_{i=1}^{\mathrm{L}}{\sum }_{t=1}^{{\mathrm{T}}_i}P({\mathrm{O}}_i,{\mathrm{s}}_{i,t}=s\mid \overline{{\mathcal{M}}_i},{\mathrm{A}}_i)\mathbb{I}({\mathrm{a}}_{i,t}=a)}}\end{array}\end{array}$$

(5)

As shown in Figure 3, for each sequence, there is a corresponding POMDP (Figure 3a). We first initialize the parameters of these $\mathrm{L}$ POMDPs (Figure 3b). We extract the observation sequence and action sequence for each sequence to facilitate subsequent parameter updates based on the update formulas (Figure 3c). We repeatedly iterate and update the parameters according to Equations (3)–(5) until the parameter values converge (e.g., the difference between the parameter values of the last two updates does not exceed a certain threshold) (Figure 3d,e).

We summarize the above process in Algorithm 1. The algorithm generates the state space S, action space A, and observation space $\Omega$ based on the specific problem to be solved (line 1), and randomly initializes the parameters ${\mathcal{M}}^{\mathrm{E}}$ (line 2). Then, given available learning records, the algorithm updates the parameters according to Equations (3)–(5) (lines 3–16), until the termination condition is met (e.g., the error between the parameters updated in the last two iterations is within a certain threshold).

Algorithm 1 Parameter Learning of PCEM

Input: The observation sequences $\mathbb{O}$, the action sequences $\mathbb{A}$, the termination condition Output: The initial state distribution $D_i$, the transition function $T_i$, and the observation function O ($i=1,2,...,\mathrm{L}$).   1: Generate S, A, and $\Omega$.   2: Randomly initialize ${\mathcal{M}}^{\mathrm{E}}$.   3: while the termination condition is not met do   4:       for $i\in \left\{1,2,...,\mathrm{L}\right\}$ do   5:             for $\mathrm{s}\in S$ do   6:                  Calculate $D_i\left(s\right)$ according to Equation (3)   7:             for $\mathrm{s},{\mathrm{s}}^{\prime }\in S,\mathrm{a}\in A$ do   8:                  Calculate $T_i(s,a,s^{\prime })$ according to Equation (4)   9:       for $\mathrm{s}\in S,\mathrm{o}\in \Omega ,\mathrm{a}\in A$ do 10:             Calculate $O(s,a,o)$ according to Equation (5) 11: return $D_i,T_i,O(i=1,2,...,\mathrm{L})$

• Comparison of algorithm efficiency.

Comparing the parameter learning methods of HCPM and PCEM, the parameter learning method of HCPM consumes more time than that of PCEM primarily in two steps: (1) HCPM requires the computation of the membership degree of each time series to each POMDP, whereas in PCEM, these membership degrees are constants (since the time series and POMDPs correspond one-to-one) and do not require calculation; and (2) when calculating the parameters of the state transition function, HCPM uses each time series to (weighted) calculate the parameters of every POMDP, while PCEM only uses each time series to calculate the parameters of its corresponding single POMDP. Therefore, the difference in time complexity between the two can be expressed as follows:

$$\begin{array}{c}\hfill O_{HCPM}-O_{PCEM}=O(\mathrm{K}\mathrm{L}\mid S\mid +\mid A\mid \mathrm{K}\mathrm{L}{\mid S\mid }^2\mathrm{T})\end{array}$$

where O denotes the time complexity (note that this is distinct from the observation function O mentioned earlier), $\mathrm{K}$ is the number of POMDPs in HCPM, $\mathrm{L}$ is the number of time series, T represents the length of the time series (as the lengths of different time series may vary, a representative value of the same order of magnitude is used here), $\left|S\right|$ is the size of the state space, and $\left|A\right|$ is the size of the action space. The first term on the right side of the equation represents the time complexity of calculating the membership degrees, while the second term represents the difference in time complexity between the two models when calculating the state transition functions. Additionally, the time complexity of the parameter learning methods for PCEM and PCPM is the same. However, in practical applications, since each learner is modeled individually (except for the observation function), the parameter updates for each model can be completed locally and independently, which significantly improves computational efficiency.

3.4. PCEM Online Planning

For PCEM, although it is possible to use planning methods to solve each POMDP within it to obtain a strategy corresponding to each learner, it should be noted that these POMDPs only reflect the past cognitive experiences of the corresponding learners. Therefore, these strategies often have difficulty providing correct tutoring for the cognitive process that requires continuous acquisition of new knowledge.

Given the cognitive process is complexity, we propose the adviser–advisee mechanism for real-time planning. The advisee is the learner tackling the upcoming exercise, while other learners engaging with the system are termed advisers. For a particular knowledge concept that the advisee is about to learn, there are two steps in online planning: (a) the advisers put forward suggestions based on their respective cognitive experiences; and (b) the system comprehensively considers these suggestions and selects the next exercise for the advisee. The two steps are detailed as follows.

(a) Based on the advisee’s current answer sequence, each adviser suggests their view of the optimal next exercise. This step can be implemented by dynamic programming [22]. The initial belief of each adviser is equivalent to the distribution of its initial state,

$$\begin{array}{c}\hfill b_1\left({\mathrm{s}}_1\right)=D_j\left({\mathrm{s}}_1\right)\end{array}$$

The update of the belief is as follows:

$$\begin{array}{c}\hfill b_t\left({\mathrm{s}}_t\right)={\displaystyle \frac{{\sum }_{{\mathrm{s}}_{t-1}\in S}{b}_{t-1}\left({\mathrm{s}}_{t-1}\right)O({\mathrm{s}}_{t-1},{\mathrm{a}}_{t-1},{\mathrm{o}}_{t-1})T_j({\mathrm{s}}_{t-1},{\mathrm{a}}_{t-1},{\mathrm{s}}_t)}{{\sum }_{{\mathrm{s}}_{t-1},{\mathrm{s}}_t^{\prime }\in S}{b}_{t-1}\left({\mathrm{s}}_{t-1}\right)O({\mathrm{s}}_{t-1},{\mathrm{a}}_{t-1},{\mathrm{o}}_{t-1})T_j({\mathrm{s}}_{t-1},{\mathrm{a}}_{t-1},{\mathrm{s}}_t^{\prime })}}\end{array}$$

(6)

For any belief, the optimal strategy ${\pi }^{\ast }\left(b_t\right)$ is to choose the action with the highest value.

$$\begin{array}{c}\hfill {\pi }^{\ast }\left(b_t\right)=argma{x}_{{\mathrm{a}}_t\in A}Q(b_t,{\mathrm{a}}_t)\end{array}$$

where the action value is calculated as

$$\begin{array}{c}\hfill Q(b_t,{\mathrm{a}}_t)=R(b_t,{\mathrm{a}}_t)+\gamma {\sum }_{{\mathrm{o}}_t\in \Omega }O(b_t,{\mathrm{a}}_t,{\mathrm{o}}_t)V_{t+1}\left(b_{t+1}\right)\end{array}$$

where the next belief $b_{t+1}$ is updated according to Equation (6).

The belief value $V_t\left(b_t\right)$ is the value of the action that maximizes the new belief value therefore leading to the recursive computation in the DP,

$$\begin{array}{c}\hfill V_t\left(b_t\right)=ma{x}_{{\mathrm{a}}_t\in A}Q(b_t,{\mathrm{a}}_t)\end{array}$$

(b) After each adviser gives his/her estimation of the value of each action, the system takes into account all of the value estimations of the advisers comprehensively. This requires considering the similarity between each adviser and the advisee, and calculating the comprehensive value estimation of each action by weighting with the similarity. The similarity between learner i (advisee) and learner j (adviser) is calculated as follows:

$$\begin{array}{c}\hfill w_{i,j}={\displaystyle \frac{P({\mathrm{O}}_i\mid {\mathcal{M}}_j,{\mathrm{A}}_i)}{{\sum }_{j^{\prime }=1}^{\mathrm{L}}P({\mathrm{O}}_i\mid {\mathcal{M}}_{j^{\prime }},{\mathrm{A}}_i)}}\end{array}$$

The comprehensive estimated value of the action is

$$\begin{array}{c}\hfill Q^c(b_{i,t},{\mathrm{a}}_t)=\sum _{j=1}^{\mathrm{L}}{w}_{i,j}{Q}_j(b_{j,t},{\mathrm{a}}_t)\end{array}$$

The comprehensive optimal strategy is

$$\begin{array}{c}\hfill {{\pi }^c}^{\ast }\left(b_{i,t}\right)=argma{x}_{{\mathrm{a}}_t\in A}Q(b_{i,t},{\mathrm{a}}_t)\end{array}$$

4. Experimental Results

This section presents the results of our rigorous experimentation, validating the efficacy of our approach. We detail the datasets employed, the performance metrics and testing methodologies used, and showcase the experimental outcomes. This is followed by a thorough summary and discussion. Our implementation seamlessly integrates online planning for PCPM, HCPM, and PCEM. Below, we outline the methods used for comparative analysis. It is worth noting that these are the only POMDP-based planning strategies specifically designed for modeling learners’ cognitive processes in knowledge concept learning, as documented in existing literature.

• PCPM [20]: This model incorporates POMDP to represent learners’ cognitive processes, facilitating real-time updates to a learner’s knowledge state based on feedback. This, in turn, enables the adaptation of learning strategies accordingly.
• HCPM [21]: By utilizing H-POMDP, this model models learners’ cognitive processes, allowing dynamic updates to learners’ knowledge states and cognitive abilities based on their feedback. This facilitates the induction of suitable learning strategies.
• PCEM: our method, proposed in Algorithm 1, aims to enhance the performance and efficiency of the system.

We devised three metric measurements (detailed in Equations (7)–(9)) to quantify the compared algorithms’ effectiveness. Furthermore, we conducted statistical analysis (Equation (10)) and evaluated time complexity to comprehensively assess the performance of the algorithms under comparison. For our simulation studies, we carefully chose eight knowledge concept learning domains from three distinct real-world datasets: two publicly accessible datasets, namely Non Skill-Builder data 2009-10 (ASSIST) [23] and Smart Learning Partner (SLP) [24], as well as a proprietary dataset, Quanlang (https://www.quanlangedu.com (accessed on 1 August 2024)). Through extensive experiments, we established the superiority of PCEM in terms of accurately real-time modeling learners’ cognitive processes and adapting learning strategies dynamically based on feedback. The diverse datasets’ results further underlined the efficacy and adaptability of our method. We provided comprehensive results, encompassing comparative studies, statistical analysis, and an evaluation of time complexity. These simulations were executed in a state-of-the-art computing environment, utilizing Python 3.8 on an Ubuntu server equipped with a powerful Core i9-1090K 3.7 GHz processor, GeForce RTX 3090 graphics card, and an ample 128 GB of RAM.

4.1. Datasets

We briefly describe the datasets used in simulation experiments: ASSIST, SLP, and Quanlang.

Table 1. The statistical features of the datasets and sub-datasets in the experiments.

Dataset/Sub-Dataset	Subject	Learners	Knowledge Concepts	Questions	Answer Logs
ASSIST		8096	543	6907	603,128
ASSIST1	Math	2865	3	202	25,963
ASSIST2	Math	3509	7	154	20,513
ASSIST3	Math	3283	6	208	25,642
Quanlang		11,765	399	4871	343,719
Quanlang1	Math	416	4	81	4243
Quanlang2	Math	296	3	62	3089
Quanlang3	Math	515	5	103	5304
SLP		3408	184	6851	344,576
SLP1	Physics	143	5	184	4278
SLP2	Chemistry	81	2	33	1400

summarizes these datasets and sub-datasets. Figure 4 and

Table 2. Knowledge concept descriptions.

Data Set	ID	Knowledge Name	State Description
ASSIST1	${\kappa }_1$	Subtraction whole numbers	(${\kappa }_1$,${\kappa }_2$,${\kappa }_3$)
	${\kappa }_2$	Subtraction whole numbers, Pattern finding
	${\kappa }_3$	Pattern finding
ASSIST2	${\kappa }_1$	Congruence	(${\kappa }_1$,${\kappa }_5$,${\kappa }_2$,${\kappa }_6$,${\kappa }_3$,${\kappa }_7$,${\kappa }_4$)
	${\kappa }_2$	Perimeter of a Polygon
	${\kappa }_3$	Substitution
	${\kappa }_4$	Equation solving more than two steps
	${\kappa }_5$	Congruence, Perimeter of a Polygon
	${\kappa }_6$	Congruence, Perimeter of a Polygon, Substitution
	${\kappa }_7$	Congruence, Perimeter of a Polygon, Substitution, Equation solving more than two steps
ASSIST3	${\kappa }_1$	Multiplication of whole numbers	(${\kappa }_1$,${\kappa }_4$,${\kappa }_2$,${\kappa }_5$,${\kappa }_3$,${\kappa }_6$)
	${\kappa }_2$	Pattern finding
	${\kappa }_3$	Unlabeled
	${\kappa }_4$	Multiplication of whole numbers, Pattern finding
	${\kappa }_5$	Pattern finding, Unlabeled
	${\kappa }_6$	Multiplication of whole numbers, Pattern finding, Unlabeled
Quanlang1	${\kappa }_1$	Positive and negative numbers	(${\kappa }_1$,${\kappa }_2$,${\kappa }_3$,${\kappa }_4$)
	${\kappa }_2$	Absolute value
	${\kappa }_3$	Opposite number
	${\kappa }_4$	Addition of rational numbers
Quanlang2	${\kappa }_1$	Definition of linear equations in one variable	(${\kappa }_1$,${\kappa }_2$,${\kappa }_3$)
	${\kappa }_2$	Formulating linear equations in one variable
	${\kappa }_3$	Solutions to linear equations in one variable
Quanlang3	${\kappa }_1$	Definition of triangles	(${\kappa }_1$,${\kappa }_2$,${\kappa }_3$,${\kappa }_4$,${\kappa }_5$)
	${\kappa }_2$	Classification of triangles
	${\kappa }_3$	Triangular side relations
	${\kappa }_4$	Interior angle properties of triangles
	${\kappa }_5$	Exterior angle properties of triangles
SLP1	${\kappa }_1$	Rectilinear propagation of light	(${\kappa }_1$,${\kappa }_2$,${\kappa }_3$,${\kappa }_4$,${\kappa }_5$)
	${\kappa }_2$	Reflection of light
	${\kappa }_3$	Refraction of light
	${\kappa }_4$	Image formation by converging lens
	${\kappa }_5$	Lens and its applications
SLP2	${\kappa }_1$	Physical and chemical changes	(${\kappa }_1$,${\kappa }_2$)
	${\kappa }_2$	Research into applications of substance property

show the knowledge concept structures and names in eight learning domains. For each domain, the experiments simulate learners’ cognitive processes, considering different abilities and knowledge states, interacting with the system in real-time.

4.2. Metrics for Evaluating Model Performance

To evaluate model effectiveness, we used specific performance metrics. We simulated real-time learning strategy updates for various models and learner–system interactions based on these strategies. Assuming a 5 min response time per question, we simulated $10,000$ learners interacting simultaneously with the system (10 interactions per knowledge concept). We logged the final knowledge states of these simulated learners, and computed their distribution as: $P\left(s\right)={\sum }_{l=1}^{\mathrm{L}}\mathbb{I}(s_{T_l}=s)/\mathrm{L},\forall s\in S$. Three metrics were employed to assess the performance of different online learning strategies, and we analyzed the statistically significant differences of these metrics through a t-test.

• $\delta \left(\kappa \right)\in [0,1]$ represents the average mastery level of the knowledge concept $\kappa$ among the learner group.

  $$\begin{array}{cc}\hfill \delta \left(\kappa \right)& ={\sum }_{s\in S}\mathbb{I}(s,\kappa )P\left(s\right)\hfill \end{array}$$

  (7)

  where $\mathbb{I}(s,\kappa )$ is a indicator function, represents the mastery level (0 or 1) of the learner with knowledge state s for the knowledge concept $\kappa$.
• $\Delta \in [0,\mathrm{N}]$ is the average number of knowledge concepts mastered by the learner group in the learning domain.

  $$\begin{array}{c}\hfill \Delta ={\sum }_{\kappa \in \mathcal{K}}\delta \left(\kappa \right)={\sum }_{s\in S}\mathcal{C}\left(s\right)P\left(s\right)\end{array}$$

  (8)

  where $\mathcal{C}\left(s\right)={\sum }_{\kappa \in \mathcal{K}}\mathbb{I}(s,\kappa )$ is the number of knowledge concepts mastered by a learner with the knowledge state s.
• $\Lambda \in [0,+\infty )$ represents the stability of the learning strategy.

  $$\begin{array}{c}\hfill \Lambda ={\sum }_{s\in S}{[\Delta -\mathcal{C}\left(s\right)]}^{2}P\left(s\right)\end{array}$$

  (9)
• The independent sample t-test statistic is

  $$\begin{array}{c}\hfill t={\displaystyle \frac{{\Delta }_1-{\Delta }_2}{\sqrt{\frac{({\mathrm{L}}_1-1){\Lambda }_1^2+({\mathrm{L}}_2-1){\Lambda }_2^2}{{\mathrm{L}}_1+{\mathrm{L}}_2-2}}(\frac{1}{{\mathrm{L}}_1}+\frac{1}{{\mathrm{L}}_2})}}\end{array}$$

  (10)

  where ${\mathrm{L}}_1={\mathrm{L}}_2=\mathrm{L}=10,000$ in our experiments. And note that the t here is different from the t representing time step in the previous.

4.3. Experiment: Evaluations on Online Learning Strategy Induction

We modeled each dataset using PCEM and HCPM, respectively, recording the number of iterations completed by each method within 5 min to compare their computational efficiency. As shown in

Table 3. Comparison of the number of iterations.

	ASSIST1	ASSIST2	ASSIST3	Quanlang1	Quanlang2	Quanlang3	SLP1	SLP2
PCEM	44	7	8	364	682	39	92	1916
HCPM	14	3	3	91	178	12	38	792

, the computational efficiency of PCEM is significantly higher than that of HCPM. Additionally, by comparing the scales of several datasets, it is evident that computational efficiency is influenced by the number of learners, knowledge concepts, exercises, and records, which is consistent with the analysis in Section 3.3.

Table 4. Experimental results on the learning strategy performance.

Sub-Dataset	Metric	PCPM	HCPM	PCEM
ASSIST1	$\delta \left({\kappa }_1\right)$	0.9977	0.9988	0.9993
	$\delta \left({\kappa }_3\right)$	0.9926	0.9947	0.9959
	$\delta \left({\kappa }_2\right)$	0.9988	0.9986	0.9983
	$\Delta$	2.9890	2.9921	2.9935
	$\Lambda$	0.0180	0.0133	0.0112
ASSIST2	$\delta \left({\kappa }_1\right)$	0.9944	0.9983	0.9995
	$\delta \left({\kappa }_5\right)$	0.9804	0.9913	0.9956
	$\delta \left({\kappa }_2\right)$	0.9950	0.9985	0.9994
	$\delta \left({\kappa }_6\right)$	0.9586	0.9793	0.9889
	$\delta \left({\kappa }_3\right)$	0.9901	0.9953	0.9977
	$\delta \left({\kappa }_7\right)$	0.9337	0.9613	0.9761
	$\delta \left({\kappa }_4\right)$	0.9819	0.9870	0.9903
	$\Delta$	6.8340	6.9110	6.9475
	$\Lambda$	0.4848	0.2427	0.1369
ASSIST3	$\delta \left({\kappa }_1\right)$	0.9966	0.9978	0.9985
	$\delta \left({\kappa }_4\right)$	0.9863	0.9923	0.9955
	$\delta \left({\kappa }_2\right)$	0.9984	0.9994	0.9997
	$\delta \left({\kappa }_5\right)$	0.9869	0.9918	0.9945
	$\delta \left({\kappa }_3\right)$	0.9969	0.9977	0.9982
	$\delta \left({\kappa }_6\right)$	0.9690	0.9795	0.9859
	$\Delta$	5.9340	5.9584	5.9722
	$\Lambda$	0.1647	0.1008	0.0675
Quanlang1	$\delta \left({\kappa }_1\right)$	0.9999	1.0000	1.0000
	$\delta \left({\kappa }_2\right)$	0.9968	0.9987	0.9995
	$\delta \left({\kappa }_3\right)$	0.9891	0.9932	0.9956
	$\delta \left({\kappa }_4\right)$	0.9817	0.9874	0.9911
	$\Delta$	3.9674	3.9793	3.9861
	$\Lambda$	0.0671	0.0391	0.0246
Quanlang2	$\delta \left({\kappa }_1\right)$	0.9998	0.9999	0.9999
	$\delta \left({\kappa }_2\right)$	0.9962	0.9977	0.9985
	$\delta \left({\kappa }_3\right)$	0.9882	0.9917	0.9941
	$\Delta$	2.9843	2.9893	2.9926
	$\Lambda$	0.0237	0.0157	0.0105
Quanlang3	$\delta \left({\kappa }_1\right)$	1.0000	1.0000	1.0000
	$\delta \left({\kappa }_2\right)$	0.9338	0.9337	0.9330
	$\delta \left({\kappa }_3\right)$	0.9317	0.9296	0.9267
	$\delta \left({\kappa }_4\right)$	0.9858	0.9875	0.9889
	$\delta \left({\kappa }_5\right)$	0.9199	0.9211	0.9215
	$\Delta$	4.7711	4.7718	4.7700
	$\Lambda$	0.2494	0.2464	0.2478
SLP1	$\delta \left({\kappa }_1\right)$	1.0000	1.0000	1.0000
	$\delta \left({\kappa }_2\right)$	0.9643	0.9636	0.9627
	$\delta \left({\kappa }_3\right)$	0.9965	0.9976	0.9984
	$\delta \left({\kappa }_4\right)$	0.9881	0.9899	0.9913
	$\delta \left({\kappa }_5\right)$	0.9478	0.9492	0.9501
	$\Delta$	4.8967	4.9004	4.9025
	$\Lambda$	0.1401	0.1298	0.1236
SLP2	$\delta \left({\kappa }_1\right)$	0.9997	0.9997	0.9998
	$\delta \left({\kappa }_2\right)$	0.9963	0.9978	0.9987
	$\Delta$	1.9960	1.9975	1.9984
	$\Lambda$	0.0046	0.0030	0.0020

Bold numbers indicate the statistically significant difference.

presents the results of the experiment where learners’ progress is guided by real-time updated learning strategies. These findings highlight the effectiveness of different learning strategies across various knowledge concept learning domains, shedding light on learners’ performance in these diverse domains. The results provide critical insights into how different strategies impact the final knowledge states that learners achieve, and offer a quantitative measure of these outcomes. For each dataset and learning strategy, we can analyze the distribution of learners’ final knowledge states after the simulation ends, allowing us to calculate and compare the key metrics presented in Table 4. The careful analysis presented in Table 4 reveals profound insights into the effectiveness of various learning strategies. Consider the knowledge learning domain SLP2, where the state space, defined as $({\kappa }_1,{\kappa }_2)\in \{(0,0),(1,0),(1,1)\}$, represents achievable knowledge mastery states. Notably, the probabilities of learners mastering ${\kappa }_1$ and ${\kappa }_2$ are impressively high, at $0.9998$ and $0.9987$, respectively. These figures underscore the remarkable efficacy of the strategies evaluated, offering compelling evidence for their continued application and potential expansion to other educational settings.

Table 5. The statistical significance of the learning strategy performance.

PCEM vs.	ASSIST1	ASSIST2	ASSIST3	Quanlang1	Quanlang2	Quanlang3	SLP1	SLP2
PCPM	2.6378	14.4004	7.9324	6.1880	4.4891	0.1518	1.1302	3.0267
HCPM	0.9550	5.9215	3.3538	2.7149	2.0473	0.2519	0.4169	1.2903

Bold numbers indicate the statistically significant difference.

provides a detailed statistical significance analysis of the performance differences between the online learning strategies induced by the PCEM model and those induced by the PCPM and HCPM models across various knowledge learning domains. The analysis utilizes a two-tailed t-test with a significance level of $0.05$ and degrees of freedom equal to $19,998$, where the critical value is $1.96$.

4.4. Experimental Summary and Discussion

Through rigorous and extensive experiments, we have unequivocally established the superiority of PCEM in accurately modeling learners’ cognitive processes in real-time and dynamically adapting learning strategies based on feedback. The results obtained from diverse datasets further underscore the effectiveness and versatility of our method. We provide comprehensive experimental results, including comparative studies, statistical analysis, and an evaluation of time complexity.

In computational efficiency tests, we modeled datasets using both PCEM and HCPM, recording iterations completed within five minutes. Table 3 clearly shows PCEM’s superior efficiency over HCPM. Analysis across various dataset scales reveals that factors like the number of learners, knowledge concepts, exercises, and records impact computational efficiency, echoing our Section 3.3 analysis.

Across all tested domains, learners who followed PCEM-induced strategies consistently demonstrated a higher level of mastery over knowledge concepts, as illustrated in Table 4. A noticeable pattern emerged: knowledge concepts closer to the root of the knowledge structure were more likely to be mastered, while those nearer to the leaf nodes proved more challenging. This trend reflects the hierarchical nature of knowledge acquisition, wherein mastering core concepts is essential for understanding more complex ideas. This finding aligns with natural cognitive development, indicating that PCEM strategies resonate with learners’ intuitive learning paths. The hierarchical approach inherent in PCEM not only intuitively organizes the learning process, but also enhances efficiency. By giving priority to fundamental concepts, PCEM lays a strong foundation for building upon more complex ideas, facilitating gradual and manageable cognitive progression. This incremental approach mitigates cognitive overload, boosts knowledge retention, and fosters deeper comprehension.

The statistical analysis presented in Table 5 reveals that, although the PCEM model slightly lags behind the PCPM and HCPM models in the Quanlang3 domain, these disparities are statistically insignificant. Crucially, in most other domains, PCEM significantly outperforms its counterparts, showcasing higher proficiency and stability. This underscores the efficacy of PCEM-induced strategies in guiding learners to achieve comprehensive knowledge mastery.

In conclusion, our findings strongly recommend the implementation of PCEM-induced learning strategies. Their hierarchical approach resonates with natural cognitive development, elevating learning efficiency and depth of understanding. The statistical evidence supports their preeminence in most domains, highlighting their adaptability to individual differences and potential for personalized learning.

5. Conclusions and Future Work

In comparison to offline learning strategy induction methods, online personalized learning strategy induction offers enhanced practicality but also poses greater challenges. This paper utilizes ITS as a research platform and introduces the POMDP-based cognitive experience model (PCEM), which serves as the foundation for developing a novel online learning strategy update method. PCEM meticulously captures individual learner characteristics by recording their learning experiences, enabling the induction of personalized online learning strategies. By measuring the similarity between different learners, our method calculates the reference value of each learner’s experience in another’s learning process, facilitating personalized strategy updates. This approach ensures rapid updates to online learning strategies while significantly enhancing their overall performance.

However, our current research is limited to modeling simple state and action spaces, focusing on basic model settings and analysis. Future research will aim to gather more comprehensive information during the data collection phase and explore effective methods to incorporate this information into the modeling process. Our ultimate goal is to develop a tutoring system that considers multiple factors, akin to human instructors’ teaching methods, which will require more complex model design and broader data collection. This presents a substantial challenge in the application of artificial intelligence within ITS, and we look forward to addressing these challenges in our future work.