Reinforcement Q-Learning for PDF Tracking Control of Stochastic Systems with Unknown Dynamics

Abstract

Tracking control of the output probability density function presents significant challenges, particularly when dealing with unknown system models and multiplicative noise disturbances. To address these challenges, this paper introduces a novel tracking control algorithm based on reinforce-ment Q-learning. Initially, a B-spline model is employed to represent the original system, thereby transforming the control problem into a state weight tracking issue within the B-spline stochastic system model. Moreover, to tackle the challenge of unknown stochastic system dynamics and the presence of multiplicative noise, a model-free reinforcement Q-learning algorithm is employed to solve the control problem. Finally, the proposed algorithm’s effectiveness is validated through comprehensive simulation examples.

1. Introduction

The surge in interest in stochastic control stems from its applicability to diverse real-world systems, including aerospace, chemical, textile, and maritime machinery [1,2]. Gaussian processes in control can be managed through mean and variance manipulation [3], but non-Gaussian processes, such as particle scale distribution in paper processing [4], flame shape dynamics [5], and chemical polymerization molecular weight distribution [6], necessitate a more comprehensive approach. Wang’s 1996 innovation introduced probability density function (PDF) control [7,8,9], utilizing B-spline functions to bypass the Gaussian assumption limitations [10]. This has led to a range of stochastic control frameworks in both theoretical and practical realms, particularly in target tracking, where systems often aim to track distributions rather than values [11]. In detail, this method applies a B-spline model to represent the system’s dynamic probability density function (PDF) by mapping the weights of the B-spline function to a dynamic state–space model. This approach shifts the focus from controlling the shape of the PDF to aligning the weights with predefined values. Under this framework, numerous significant papers have been published. For example, Luan’s output PDF control tackles static tracking [12]. However, the controller design in most of the papers within this framework heavily relies on precise knowledge of the PDF model. Specifically, it requires the state–space model, derived from the PDF via the B-spline network, to be accurate for effective controller design. Meeting this requirement is challenging in many real-world industrial systems, limiting the practical application and development of this framework.

Multiplicative noise, which poses significant challenges due to its signal-dependent nature and non-linear characteristics, appears in various applications. However, a great number of studies, including those utilizing the algorithms proposed under Wang’s framework, do not account for the influence of multiplicative noise in their system modeling. Although Yang’s fully probabilistic design under Wang’s framework addresses this issue by incorporating multiplicative noise into the model [13], it necessitates precise knowledge of the system’s model parameters. Consequently, there is still a gap between this method and practical real-world applications.

Another important aspect often neglected in this framework is the consideration of time-varying targets. In many industrial contexts, time-varying targets are essential and present additional challenges for controller design. However, most studies within this framework tend to overlook this factor, which further complicates the practical application and effectiveness of the framework in real-world scenarios  [9].

To address these barriers, this paper employs Wang’s B-spline framework to develop a model-free control approach that enables dynamic weights to adapt to a time-varying target pattern. In our design, we utilize Q-learning-based Linear Quadratic Tracking Control (LQT) as a controller to achieve the tracking goal. Traditional LQT often struggle with the complexities of real-world systems, highlighting the necessity for model-free control strategies. Thus, reinforcement learning, particularly strategy iteration, is employed here to handle control and scheduling tasks without requiring complete model information [14]. As a model-free technique, reinforcement learning optimizes control by learning under given constraints [11,15]. The concept of L-Extra-Sampled (Les)-dynamics was introduced in [16], providing a new perspective for addressing reinforcement learning problems in partially observable linear Gaussian systems. However, the research in this paper primarily focuses on linear Gaussian systems, which may limit the method’s applicability to non-Gaussian systems. In [17], an algorithm based on Q-learning was proposed to handle time-varying linear discrete-time systems with complete dynamic uncertainties. Although the algorithm is model-independent, it may require more computational resources to compute the Q-function and update the policy when dealing with high-dimensional systems. Ref. [18] introduced extreme value theory into reinforcement learning, proposing a new online and offline maximum entropy reinforcement learning update rule [19]. This rule avoids the difficulty of estimating the maximum Q-value in continuous action space. In [20], an efficient offline Q-learning method was proposed to solve the data-driven discrete-time linear quadratic regulator problem. This approach does not require knowledge of the system dynamic model and demonstrates advantages over existing methods in simulations. In [21], a Q-learning based iterative learning control (ILC) framework for fault estimation (FE) and fault-tolerant control (FTC) was proposed to address the actuator fault problem in multiple-input multiple-output (MIMO) systems. However, traditional reinforcement learning methods are inefficient, consuming significant time and requiring continuous adjustment. The efficiency of reinforcement learning can be enhanced by integrating it with optimal control techniques. For instance, in [22], the successful application of Q-learning to handle discrete systems with uncertain parameters was demonstrated, improving tracking control performance. Similarly, Xue’s extension to two-time-scale systems showed favorable outcomes [23]. However, these findings are based on deterministic models and neglect the system randomness and uncertainties. By solving the Riccati equation for quadratic optimal control of linear stochastic systems with unknown parameters, we improve learning efficiency and achieve model-free tracking control. This approach allows for effective control even in the presence of random disturbances, addressing the limitations of traditional methods and enhancing the practical applicability of our framework.

In summary, this paper utilizes Wang’s B-spline framework to develop a model-free control approach, enabling dynamic weights to adapt to a target time-varying pattern. In the case of an unknown stochastic system model, we can change the output PDF shape by controlling the weights. This approach expands the scope of application for stochastic systems in output PDF control. Output PDF control is employed to monitor and manage the distribution of key parameters during the production process, which helps enhance product quality and decrease scrap rates. For instance, in the semiconductor manufacturing industry, controlling the distribution of parameters such as etching depth and doping concentration during wafer fabrication ensures consistent chip performance. Similarly, in the automobile manufacturing industry, controlling the distribution of coating thickness during the painting process improves surface quality and anti-corrosion performance. By leveraging the B-spline basis functions and the model-free reinforcement Q-learning algorithm, we can effectively handle unknown system dynamics and multiplicative noise, achieving precise control over the output PDF shape. This method facilitates accurate PDF shape tracking and establishes a theoretical foundation for PDF monitoring in non-Gaussian stochastic systems, making it highly applicable to real-world industrial contexts. The contributions of this paper can be summarized as follows:

1

By integrating the reinforcement learning method with the LQR control algorithm, the control framework proposed in this paper effectively addresses the challenge of PDF tracking control in stochastic systems with unknown parameters, eliminating the need for precise knowledge of the system’s model parameters.

2

By utilizing B-spline functions to approximate the PDF, our method converts the PDF tracking problem into a state tracking problem with dynamic weights.

3

The multiplicative noise is being considered while modeling the PDF under the B-spline framework, reflecting a more accurate representation of complex and realistic uncertainties.

4

The consideration of time-varying PDF target, which are crucial in many real-world applications but often overlooked in previous studies, enhances the practical applicability of the framework.

5

The framework combines optimal control principles with reinforcement learning, specifically Q-learning, to significantly enhance the performance and accelerate the learning speed of RL algorithms.

The remainder of this paper is organized as follows: Section 2 provides a detailed description of the problem and the B-spline model of the PDF. In Section 3, the optimal control law is derived based on the performance metrics, and the implementation algorithm is presented. Section 4 demonstrates the application of the controller through two numerical examples. Finally, Section 5 summarizes the conclusions and outlines potential future work.

2. Problem Description

2.1. PDF Description Based on B-Spline

Modeling the output PDF of a controlled system by solving partial differential equations can be challenging when using first principles, complicating the development of an effective control strategy [24]. To overcome this, the B-spline approach can be employed to approximate the PDF curve by mapping weights to basis functions. B-spline basis functions are a flexible and widely used class of functions that can adapt to various interpolation and fitting requirements. They can be classified by order: first-order, second-order, third-order and fourth-order B-spline basis functions. Among these, third-order B-spline basis functions are the most commonly used for cubic polynomial functions because they strike a good balance between smoothness and computational complexity. Specifically, given a known interval $[a,b]$, where the output PDF $\gamma \left(y\right)$ is continuous and bounded, the PDF can be expressed using n B-spline basis functions as follows:

$$\gamma \left(y\right)=\sum _{i=1}^n{w}_i{B}_i\left(y\right),$$

(1)

where $w_i$ (with $i=1,2,\dots ,n$) represents the weight and $B_i\left(y\right)$ represents pre-selected n basis functions, which can include Gaussian, radial basis, or wavelet functions. Given that $\gamma \left(y\right)$ is a PDF defined on the interval $[a,b]$, it is subject to the following mathematical constraints:

$${\int }_a^b\gamma \left(y\right)=1.$$

(2)

To satisfy Equation (2), only $n-1$ weights are independent, which allows us to express the distribution in the following form:

$$\begin{array}{c}\gamma \left(y\right)={\mathit{C}}_{\mathbf{0}}\left(y\right)\mathit{x}+L\left(y\right),\end{array}$$

(3)

$$\begin{array}{c}\mathit{x}={[w_1,w_2,\dots ,w_{n-1}]}^T\in {\mathbb{R}}^{n-1\times 1},\end{array}$$

(4)

$$\begin{array}{c}L\left(y\right)={({\int }_a^b{B}_n\left(y\right)dy)}^{-1}{B}_n\left(y\right)\in {\mathbb{R}}^{1\times 1},\end{array}$$

(5)

$$\begin{array}{c}{\mathit{C}}_{\mathbf{0}}\left(y\right)={\left[\begin{array}{c}{B}_1\left(y\right)-{\displaystyle \frac{B_n\left(y\right)}{{\int }_a^b{B}_n\left(y\right)dy}}{\int }_a^b{B}_1\left(y\right)dy\\ B_2\left(y\right)-{\displaystyle \frac{B_n\left(y\right)}{{\int }_a^b{B}_n\left(y\right)dy}}{\int }_a^b{B}_2\left(y\right)dy\\ ⋮\\ B_{n-1}\left(y\right)-{\displaystyle \frac{B_n\left(y\right)}{{\int }_a^b{B}_n\left(y\right)dy}}{\int }_a^b{B}_{n-1}\left(y\right)dy\end{array}\right]}^T\in {\mathbb{R}}^{1\times n-1},\end{array}$$

(6)

where $\mathit{x}$ denotes the weight set, ${\mathit{C}}_{\mathbf{0}}\left(y\right)$ represents the vector of basis functions, and $L\left(y\right)$ is a scalar associated with the basis functions. Based on Equations (5) and (6), we can see that the choice of basis functions determines ${\mathit{C}}_{\mathbf{0}}\left(y\right)$ and $L\left(y\right)$. From Equation (1) to Equation (6), it is evident that the B-spline model enables the control of the output PDF shape by manipulating $n-1$ independent weight vectors [25].

2.2. PDF Tracking Control Problem

The tracking problem is a prevalent issue in the field of control, including in stochastic control systems. In conventional control fields, the objective often involves directing the system to follow a predefined value. Conversely, in stochastic control systems, the goal shifts to having the system track a predetermined probability density function (PDF). Figure 1 illustrates the tracking diagram where system B tracks system A. System A, despite its unknown structure, can monitor its output in real time, allowing access to the output PDF distribution $g_k\left(y\right)$ at any given time k. On the other hand, system B is a controlled system with established dynamics and employs a control input u to produce its output ${\gamma }_k(y,u_k)$. The objective is to align the output distribution of system B with that of system A, with $\mathcal{D}$ quantifying the disparity between the two distributions. The details of the system and tracking control issues outlined above are further elaborated in the subsequent section.

Consider the stochastic system with output PDF ${\gamma }_k\left(y\right)$, whose dynamics is formed as follows:

$${\gamma }_{k+1}\left(y\right)=f({\gamma }_k\left(y\right),u_k),$$

(7)

where the distribution of the system output y is denoted by ${\gamma }_k\left(y\right)$ and $u_k$ represents the system input.

Based on the B-spline model (3), denoting ${\mathit{x}}_k$ as the weights corresponding to the basis functions, the output PDF ${\gamma }_k\left(y\right)$ of the tracking system B can be represented as:

$${\gamma }_k\left(y\right)={\mathit{C}}_{\mathbf{0}}\left(y\right){\mathit{x}}_k+L\left(y\right).$$

(8)

The tracking target $g_k\left(y\right)$ is a dynamic PDF with the following manner:

$$g_k\left(y\right)={\mathit{C}}_{\mathbf{0}}\left(y\right){\mathit{V}}_k+L\left(y\right),$$

(9)

where ${\mathit{V}}_k$ in Equation (9) represent the pre-set target weights corresponding to each basis function.

Subsequently, the shaping of the output PDF $\gamma (y,u_k)$ over the interval $[a,b]$ can be achieved by controlling the weight state ${\mathit{x}}_k$. Within the framework of the B-spline model outlined in [8], the dynamics of the weight states ${\mathit{x}}_k$ for the B-spline model-based PDF are described as follows:

$${\mathit{x}}_{k+1}=\mathit{G}{\mathit{x}}_k+\mathit{H}{u}_k+\mathit{D}{\mathit{x}}_k{E}_k,$$

(10)

where $\mathit{G}\in {\mathbb{R}}^{n-1\times n-1}$ and $\mathit{H}\in {\mathbb{R}}^{n-1\times 1}$ are the corresponding weight system parameters and $E_k$ represents the state-based model randomness, whose distribution is given by:

$$E_k\backsim (0,M),$$

(11)

where M is the variance of $E_k$.

From Equations (7) to (10), we observe that the system’s PDF is dynamic, leading to the derivation of a state–space model for the weights to represent the PDF’s dynamics. However, obtaining precise model parameters, $\mathit{G}$, $\mathit{H}$, and $\mathit{D}$, is challenging. Existing control strategies often rely on precise model parameters, necessitating that $\mathit{G}$, $\mathit{H}$, and $\mathit{D}$ in Equation (10) are known accurately. This assumption is difficult to meet in many real-world industrial processes.

To address this limitation, this study explores the application of data-driven methods, specifically reinforcement learning within machine learning, to mitigate the dependence on model parameters. Reinforcement learning, characterized by its objective to maximize reward through iterative optimization, offers a promising solution to the optimal control problem [26]. By employing Q-learning, one of a reinforcement learning algorithm, the need for explicit model knowledge is alleviated, as it enables the determination of optimal control strategies based solely on system operational data. This approach enables the controlled variable to effectively track the desired trajectory without requiring knowledge of the model parameters.

The control flowchart is depicted in Figure 2. After selecting the B-spline basis functions, the time-varying target weight $r_k$ in Figure 2 is determined based on the target distribution using the B-spline principle. The system input $u_k$ is derived by assessing both the target and system weights using a reinforcement Q-learning control, which will be detailed in the subsequent section. The weight ${\mathit{x}}_{k+1}$ is then updated through B-spline principle modeling and the input $u_k$. The output distribution is obtained by correlating the weight with the basis function. It is important to note that the model error component $E_k$ is characterized by multiplicative noise, and $\mathit{D}$ represents the appropriately dimensioned weight matrix. The weights are iteratively updated according to the model to control the output distribution.

There are many methods to address multiplicative noise, and it is necessary to choose the appropriate method for different application scenarios to effectively reduce the noise’s impact. Techniques such as logarithmic transformation, adaptive filtering, wavelet transformation, statistical methods, and specially designed filters can significantly improve the quality and reliability of signal processing. However, within a reinforcement learning framework, multiplicative noise can be directly learned, thus eliminating the need for these additional processing methods.

Building on this framework, the control objective for such stochastic system is to design a state feedback controller that enables the weight ${\mathit{x}}_k$ to track the target weight $r_k$. Consequently, the PDF shape tracking control problem is transformed into a weight tracking control problem. The details of the controller design will be presented in the next section.

3. Control Algorithm

In this section, we introduce the reinforcement Q-learning control algorithm to achieve the tracking objective of the weight state. The primary motivation for choosing reinforcement learning control is to attain optimal tracking control in the presence of unknown model parameters. Additionally, the reinforcement Q-learning method is employed to establish an optimal control iterative solution structure, which proves to be more efficient than traditional reinforcement learning algorithms. The specific details are elaborated below.

3.1. General Control Solution of LQT for Systems with Multiplicative Noise

In this section, we consider the infinite-horizon linear quadratic tracking problem with multiplicative noise. This developed algorithm will then be utilized as a foundation to create our model-free Q-learning control method.

Denoting the target weight for the LQT problem which are generated by the B-spline model based on the expected PDF output is $r_k$, based on the system dynamics Equation (10), we construct the augmented system:

$$X_{k+1}=\left[\begin{array}{c}{x}_{k+1}\\ r_{k+1}\end{array}\right]=\left[\begin{array}{cc}G& \mathbf{0}\\ \mathbf{0}& I\end{array}\right]\left[\begin{array}{c}{x}_k\\ r_k\end{array}\right]+\left[\begin{array}{c}H\\ \mathbf{0}\end{array}\right]u_k+\left[\begin{array}{cc}D& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right]\left[\begin{array}{c}{x}_k\\ r_k\end{array}\right]E_k\equiv \overline{A}{X}_k+\overline{B}{u}_k+\overline{D}{X}_k{E}_k,$$

(12)

where the augmented state $X_k$ is given by:

$$X_k=\left[\begin{array}{c}{x}_k\\ r_k\end{array}\right].$$

(13)

For the regular infinite-horizon LQT problem, the objective is to design an optimal controller for the system in Equation (10), ensuring that the weight $x_k$ tracks a reference trajectory $r_k$. This can be achieved by minimizing the following infinite-horizon performance index:

$$\begin{array}{c}\hfill J=\mathbb{E}\left\{\sum _{i=0}^{\infty }{\displaystyle \frac{1}{2}}\left[{(x_i-r_i)}^{T}Q(x_i-r_i)+u_i^{T}R{u}_i\right]\right\},\end{array}$$

(14)

where $\mathbb{E}$ denotes the mathematical expectation over the noise {E(0),E(1),…}, and $Q>0$ and $R>0$ are symmetric matrices. The performance index in Equation (14) can be rewritten using Equation (13) as:

$$\begin{array}{c}\hfill J=\mathbb{E}\left\{\sum _{i=0}^{\infty }{\displaystyle \frac{1}{2}}\left[X_i^T{Q}_1{X}_i+u_i^{T}R{u}_i\right]\right\},\end{array}$$

(15)

where $Q_1=C_1^{T}Q{C}_1$ with $C_1=[I,-I]$. The stochastic system (12) is said to be mean-square stabilizable if there exists a state feedback control [27]:

$$u_k^{*}=-K{X}_k.$$

(16)

When the model parameters are known, there are multiple methods to determine the optimal feedback gains and the associated optimal cost. The optimal cost can be derived from the solution of the generalized algebraic Riccati Equation (GARE) [28]:

$$\begin{array}{c}\hfill P=Q_1+{\overline{A}}^{T}P\overline{A}+M{\overline{D}}^{T}P\overline{D}-{\overline{A}}^{T}P\overline{B}{(R+{\overline{B}}^{T}P\overline{B})}^{-1}{\overline{B}}^{T}P\overline{A}.\end{array}$$

(17)

The optimal gain matrix is as follows:

$$\begin{array}{c}\hfill K^{*}=-{\left(R+{\overline{B}}^{T}P\overline{B}\right)}^{-1}{\overline{B}}^{T}P\overline{A}.\end{array}$$

(18)

As verified in [29], the standard conditions for the uniqueness and existence of a solution to the standard ARE require that $(\overline{A},\overline{B})$ is stabilizable and $(\overline{A},\sqrt{Q_1})$ is detectable [30].

3.2. Data-Driven Reinforcement Learning for LQT Problems

In this section, we establish the action-value Q-function for reinforcement learning to iteratively solve the Riccati equation for quadratic optimal control of stochastic systems with unknown model parameters using data. The Q-function is equivalent to the cost function in optimal control, and we need to optimize this Q-function to achieve the optimal value through the control strategy [31]. We construct our Q-function according to the standard control method. Firstly, we need to defined the Q-function. If the control policy, such as in Equation (16), is a mean-square stabilization control policy, then the Q-function should be defined in the desired form. Thus, the Q-function is defined as follows:

$$\begin{array}{cc}\hfill Q(X_k,u_k)& =\mathbb{E}\left\{\sum _{i=0}^{\infty }{\displaystyle \frac{1}{2}}\left[X_i^T{Q}_1{X}_i+u_i^{T}R{u}_i\right]\right\}.\hfill \end{array}$$

(19)

where $Q_1$ is defined in Equation (15). The Bellman equation transforms an infinite sum of terms into a simpler, future-term form. Thus, based on the discrete-time LQT Bellman equation and Equation (19):

$$\begin{array}{cc}\hfill Q(X_k,u_k)& =\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{X}_k^T{Q}_1{X}_k+{\displaystyle \frac{1}{2}}{u}_k^{T}R{u}_k+\mathrm{\Gamma }Q(X_{k+1},u_{k+1})\right\}\hfill \\ \hfill & =\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{X}_k^{T}P{X}_K\right\},\hfill \end{array}$$

(20)

where P is defined in Equation (17). The Q-function can be further extended as follows:

$$Q\left(X_k,u_k\right)=\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{X}_k^T{Q}_1{X}_k+{\displaystyle \frac{1}{2}}{u}_k^{T}R{u}_k+{\displaystyle \frac{1}{2}}\mathrm{\Gamma }{X}_{k+1}^{T}P{X}_{k+1}\right\},$$

(21)

where $0<\mathrm{\Gamma }\le 1$ is the discount factor. This discount factor is crucial because it prevents the reward from increasing to infinity as the time step approaches infinity, thereby making the infinitely long control process evaluable. The discount factor represents the expectation of future rewards. A smaller discount factor places more emphasis on the reward in the current state, while a larger discount factor emphasizes future rewards. However, in the context of an infinite-time control process, the condition $\mathrm{\Gamma }=1$ holds if and only if the reference trajectory $r_k$ is Schur stable. The selection of the discount factor depends on the desired effect, involving a balance between exploration and exploitation of the algorithm. By incorporating augmented system dynamics Equation (12), the Q-function becomes:

$$Q(X_k,u_k)=\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{X}_k^T{Q}_1{X}_k+{\displaystyle \frac{1}{2}}{u}_k^{T}R{u}_k+{\displaystyle \frac{1}{2}}\mathrm{\Gamma }{\left(\overline{A}{X}_k+\overline{B}{u}_k+\overline{D}{X}_k{E}_k\right)}^{T}P\left(\overline{A}{X}_k+\overline{B}{u}_k+\overline{D}{X}_k{E}_k\right)\right\}.$$

(22)

To simplify the computation of the Q-function, we introduce a kernel matrix $\mathcal{H}={\mathcal{H}}^T$, emphasizing its quadratic nature in the variables:

$$\begin{array}{cc}\hfill Q(X_k,u_k)& =\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]}^T\mathcal{H}\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]\right\}\hfill \\ \hfill & =\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]}^T\left[\begin{array}{cc}{\mathcal{H}}_{XX}& {\mathcal{H}}_{Xu}\\ {\mathcal{H}}_{uX}& {\mathcal{H}}_{uu}\end{array}\right]\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]\right\},\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathcal{H}}_{XX}& =Q_1+\mathrm{\Gamma }({\overline{A}}^{T}P\overline{A}+M{\overline{D}}^{T}P\overline{D}),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathcal{H}}_{Xu}& =\mathrm{\Gamma }{\overline{A}}^{T}P{B}_1,\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathcal{H}}_{uX}& =\mathrm{\Gamma }{\overline{B}}^{T}P\overline{A},\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill {\mathcal{H}}_{uu}& =R+\mathrm{\Gamma }{B}_1^{T}P{B}_1,\hfill \end{array}$$

(27)

The optimal control action can then be derived by setting the gradient of Q-Function with respect to to zero, leading to a policy formula based on the Riccati-type solution. Applying the condition ${\displaystyle \frac{\partial Q(X_k,u_k)}{\partial u_k}}=0$ to Equation (23), we can obtain:

$$u_k=-{\mathcal{H}}_{uu}^{-1}{\mathcal{H}}_{uX}{X}_k,$$

(28)

Substituting Equations (26) and (27) into Equation (28), we have:

$$u_k=-{(R+\mathrm{\Gamma }{\overline{B}}^{T}P\overline{B})}^{-1}\mathrm{\Gamma }\left({\overline{B}}^{T}P\overline{A}\right)X_k.$$

(29)

Define the extended state $Z_k$ as:

$$Z_k=\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right].$$

(30)

The Q-function can then be written as the following form by substituting Equation (30) into Equation (20):

$$Q(X_k,u_k)=\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]}^T\mathcal{H}\left[\begin{array}{c}{X}_k\\ u_k\end{array}\right]\right\}=\mathbb{E}\left\{{\displaystyle \frac{1}{2}}{Z}_k^T\mathcal{H}{Z}_k\right\}.$$

(31)

where

$$Z_k^T\mathcal{H}{Z}_k=X_k^T{Q}_1{X}_k+u_k^{T}R{u}_k+\mathrm{\Gamma }{Z}_{k+1}^T\mathcal{H}{Z}_{k+1}.$$

(32)

The multiplicative noise is then incorporated into the kernel matrix $\mathcal{H}$ to participate in the policy iteration without requiring additional consideration.

The algorithm is structured around two main components:

1

Policy Evaluation:

$$\mathbb{E}\left\{Z_k^T{\mathcal{H}}^{j+1}{Z}_k\right\}=\mathbb{E}\left\{X_k^T{Q}_1{X}_k+{\left(u_k^j\right)}^{T}R\left(u_k^j\right)+\mathrm{\Gamma }{Z}_{k+1}^T{\mathcal{H}}^{j+1}{Z}_{k+1}\right\}.$$

(33)

Estimating the Q-function based on current policy and updating the kernel matrix $\mathcal{H}$ using collected data by Equation (33).

2

Policy Improvement:

$$u_k^{j+1}=-{\left({\mathcal{H}}_{uu}^{-1}\right)}^{j+1}{\mathcal{H}}_{uX}^{j+1}{X}_k.$$

(34)

Adjusting the control policy to minimize future Q-function values thereby optimizes system performance over time. Policy iteration can be implemented using least squares (LS) with the data tuple [22].

Through these steps, the reinforcement Q-learning algorithm iteratively refines its estimates and policy, adapting to changes in system dynamics and performance criteria without requiring prior knowledge of the underlying system model. The multiplicative noise is embedded within the kernel matrix using the data-driven method. This approach not only enhances the flexibility of the control system, but also improves its robustness in handling real-world operational variabilities.

Algorithm 1 provides a structured approach to implement the proposed control framework, ensuring that each step is clearly defined and actionable.

Algorithm 1: Tracking control framework with output probability density function

4. Simulation Result

In this section, we demonstrate the effectiveness of the model-free PDF tracking control algorithm through two numerical simulations. During the simulation process, the algorithm does not have any knowledge of the system model and relies solely on numerical calculations.

We start by choosing the B-spline basis functions, which are crucial for transforming the original system into a form suitable for our control algorithm. B-spline basis functions are flexible and can be tailored to various interpolation and fitting requirements. Specifically, we use third-order B-spline basis functions because they provide a good balance between smoothness and computational complexity. The B-spline basis functions selected are as follows:

$$\begin{array}{cc}\hfill & B_1\left(y\right)=0.5(y^2+6y+9)I_1+(-y^2-3y-1.5)I_2+0.5y^2{I}_3,\hfill \\ \hfill & B_2\left(y\right)=0.5(y^2+4y+4)I_2+(-y^2-y+0.5)I_2+0.5(y^2-2y+1)I_4,\hfill \\ \hfill & B_3\left(y\right)=0.5(y^2+2y+1)I_3+(-y^2+y+0.5)I_4+0.5(y^2-4y+4)I_5,\hfill \\ \hfill & B_4\left(y\right)=0.5y^2{I}_4+(-y^2+3y-1.5)I_5+0.5(y^2-6y+9)I_6,\hfill \end{array}$$

(35)

where $I_i=\left\{\begin{array}{cc}1& y\in [i-4,i-3]\\ 0& others\end{array}\right.,i=1,\dots ,6.$

As per constraint (2), only the corresponding weights for three of the four B-spline basis functions are necessary.

• Example 1

The fourth weight is linearly dependent on the first three weights, thereby reducing the model order to three. For example, 1, we use the same model as in reference [32] for comparison to demonstrate the difference between model-based and model-free approaches. By comparing the results, we can highlight the advantages and limitations of the proposed model-free reinforcement Q-learning algorithm against traditional model-based methods. Thus, the coefficient matrix of the system model is given by:

$$\begin{array}{c}\hfill {\mathit{x}}_{k+1}=\mathit{G}{\mathit{x}}_k+\mathit{H}{u}_k+\mathit{D}{\mathit{x}}_k{E}_k,\end{array}$$

(36)

with $\mathit{G}=\left[\begin{array}{ccc}0.555& -0.098& -0.041\\ -0.1& -0.734& 0.181\\ -0.292& 0.02& 0.291\end{array}\right]$, $\mathit{H}=\left[\begin{array}{c}0.275\\ 0.302\\ 0.302\end{array}\right]$, $\mathit{D}=\left[\begin{array}{ccc}0.41& 1.66& 0.51\\ -0.11& 0.215& 0.16\\ 0.31& 0.02& -1.005\end{array}\right]$.

In the described model, $\mathit{G}$ represents the state weight matrix, $\mathit{H}$ is the control matrix, $\mathit{D}$ is the random weight matrix of the noise term, and $E_k$ is the Gaussian noise. Note that the parameters $\mathit{G}$, $\mathit{H}$, and $\mathit{D}$ are assumed unknown. The target weights change every 100 steps, alternating between $r_k={\left[\begin{array}{ccc}0.4229& 0.1217& 0.1487\end{array}\right]}^T$ and $r_k={\left[\begin{array}{ccc}0.5908& 0.1701& 0.2077\end{array}\right]}^T$. Additionally, the initial state of the system is set at ${\mathit{x}}_0={\left[\begin{array}{ccc}0.2673& 0.0969& 0.1897\end{array}\right]}^T$. The noise covariance follows the distribution given by $E_k\sim \mathit{N}(0,0.004)$. The performance index, as indicated in Equation (14), employs $Q=\mathrm{diag}\left[\begin{array}{ccc}20& 20& 20\end{array}\right]$ and $R=0.1$, with a discount factor $\mathrm{\Gamma }=0.8$. The parameter $\sigma$ is chosen to be a suitably small value $=0.01$. The initial kernal matrix ${\mathcal{H}}_0$ is given by:

$${\mathcal{H}}_0=\left[\begin{array}{ccccccc}65.9612& 16.9868& 12.0472& 80.0261& -12.02978& 3.0209& -27.0793\\ 49.9990& 86.9108& -13.9571& 54.1188& -23.0361& -25.0965& 74.9031\\ 3.9340& -17.9644& 38.9088& 70.9414& 13.0338& -32.9645& 79.0096\\ 71.9722& 8.9283& 52.9810& 33.9975& 16.9669& 69.9581& 15.0152\\ 96.0250& -14.9325& 93.0874& 34.9930& 69.0667& 35.0675& 63.9573\\ -30.0573& 86.0075& 70.0079& -0.0794& -10.0277& -39.0610& -23.9975\\ -18.9535& 6.0832& -31.9131& 37.9693& 66.9357& 99.0435& -24.9186\end{array}\right]$$

The simulation results are shown in Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7. Figure 3 illustrates the weight tracking effect during the online learning process. It is evident that the algorithm begins with exploration and converges around the 40th iteration, demonstrating its computational efficiency. Notably, in Figure 4, the red line represents the weight tracking curve under the enhanced Q-learning algorithm, while the blue dashed line represents the target reference curve. The weights $r_1$, $r_2$, and $r_3$ are controlled states, whereas $r_4$, which is linearly related to the other weights and excluded from control due to constraints, is also shown. Figure 4 further shows the control of the four weights and their corresponding target reference curves after the learning is completed. This demonstrates that the reinforcement Q-learning algorithm successfully tracks the specified target weights after learning. Figure 5 presents the system control input, with the red line indicating the control input curve under the reinforcement Q-learning algorithm. This curve aligns synchronically with the target configuration change and is smooth enough for practical use. Figure 6 illustrates the output curve of the reinforcement Q-learning algorithm, which consistently follows the desired PDF shape at each time instant. It also presents the results of the target weights integrated with the PDF output derived from the B-spline model. For example, in controlling the flame shape, this curve represents the flame shape at each moment. Despite the unknown system parameters, the output curve achieves the desired PDF shape, highlighting the effective utilization of the data. Figure 7 shows the tracking error curve of the four states. It can be seen from the figure that, despite the presence of multiplicative noise interference, the tracking effect remains very good. The tracking error only experiences a jump when the tracking weight changes, and in the remaining cases, it is essentially near 0. Finally, the mean square error of the tracking is calculated to be very small, specifically 0.0285, indicating that the tracking effect is good. These results indicate that the proposed control framework effectively enables the system’s PDF to track a predefined PDF shape without requiring specific knowledge of the system parameters.

• Example 2

To strengthen the argument for the algorithm’s applicability in model-free scenarios, we provide another numerical simulation example, with $\mathit{G}=\left[\begin{array}{cc}0.6969& 0.6545\\ -0.0241& 0.8603\end{array}\right]$, $\mathit{H}=\left[\begin{array}{c}0.1298\\ 0.0787\end{array}\right]$, $\mathit{D}=\left[\begin{array}{cc}0.0081& 0.6649\\ 0.1665& 0.1786\end{array}\right]$. The target weights change every 100 steps, alternating between $r_k={\left[\begin{array}{cc}0.4229& 0.1217\end{array}\right]}^T$ and $r_k={\left[\begin{array}{cc}0.8458& 0.2434\end{array}\right]}^T$. Additionally, the initial state of the system is set at ${\mathit{x}}_0={\left[\begin{array}{cc}0.2673& 0.0969\end{array}\right]}^T$. The noise covariance follows the distribution given by $E_k\sim \mathit{N}(0,0.04)$. The initial kernal matrix is ${\mathcal{H}}_0=\left[\begin{array}{ccccc}66.00452& 19.9959& 95.0028& -29.996& 42.9970\\ 23.0017& 32.9949& 60.9976& 54.99558& -33.9953\\ 37.9961& 55.9946& -36.0101& 77.9975& 74.0099\\ 70.9958& -19.0035& 91.0042& 31.0001& 88.0092\\ 33.9968& 16.0032& 57.0020& 2.0017& 31.9961\end{array}\right].$

The simulation results are illustrated in Figure 8. In this figure, the red solid line represents the system state under our control, while the blue dashed line indicates the target weight. As shown, the system states successfully tracks the pre-set time-varying references. Additionally, the mean square error is calculated to be 0.1561, demonstrating the effective control performance. This result clearly shows that even without a model, the second-order stochastic system can accurately track the target weight.

Through two numerical simulation examples, we observe that the overall tracking error is minimal, the response speed is rapid, and the learning efficiency is high in both systems. This indicates that our control method is effective across different systems. Additionally, the mapping relationship between the systems is established using B-spline function fitting, which allows for controlling the PDF shape by tracking the target weight. Furthermore, the presented approach does not require prior knowledge of the model, enabling flexible back-and-forth switching control of the PDF shape.

5. Conclusions

This paper addresses the challenge of output distribution shape tracking in stochastic distribution systems by developing a learning-based, model-free control algorithm within the B-spline model framework to approximate the PDF. This method simplifies the complex issue of PDF shaping into a more manageable problem of dynamic weight modification, treating the system’s inherent randomness and inaccuracies as state-dependent noises, which closely mirror real-world complexities. To handle time-varying targets and reduce dependency on precise model knowledge, an extended Reinforcement Q-learning algorithm is applied in this framework. Simulation results confirm the method’s effectiveness, demonstrating its ability to accurately track varying distribution shapes. Using the data-driven method to control, it greatly removes the limitation that most control methods need system model parameters. Additionally, this approach eliminates the need to address complexities arising from multiplicative noise issues.