The Adaptive Optimal Output Feedback Tracking Control of Unknown Discrete-Time Linear Systems Using a Multistep Q-Learning Approach

Abstract

This paper investigates the output feedback (OPFB) tracking control problem for discrete-time linear (DTL) systems with unknown dynamics. To solve this problem, we use an augmented system approach, which first transforms the tracking control problem into a regulation problem with a discounted performance function. The solution to this problem is derived using a Bellman equation, based on the Q-function. In order to overcome the challenges of unmeasurable system state variables, we employ a multistep Q-learning algorithm that surpasses the advantages of the policy iteration (PI) and value iteration (VI) techniques and state reconstruction methods for output feedback control. As such, the requirement for an initial stabilizing control policy for the PI method is removed and the convergence speed of the learning algorithm is improved. Finally, we demonstrate the effectiveness of the proposed scheme using a simulation example.

1. Introduction

The optimization of performance costs has always been a crucial concern in controller design problems as it can lead to energy savings and, subsequently, have a positive impact on the environment. The development of practical requirements has significantly contributed to the advancement of optimal control [1,2,3,4]. The key challenge in optimal control lies in solving the Riccati equation for linear systems. In the case of linear systems, computationally efficient iterative algorithms [5,6] can be employed to obtain the solution to the Riccati equation. However, this method is only applicable when a comprehensive understanding of the system dynamics is available. In control engineering, online learning controllers have commonly been designed without complete knowledge of the system dynamics [7,8,9,10,11]. Notably, a data-based approach was proposed in [12] for analyzing the controllability and observability of discrete-time linear (DTL) systems without the precise knowledge of system parameters.

Reinforcement learning (RL) is a powerful method for optimizing rewards via interactions with the environment [13]. Utilizing RL techniques, controller performance can be enhanced based on reward signals [14] and controller parameters can be updated to achieve optimal design criteria for adaptive control. Consequently, RL has provided valuable insights into the field of control systems [14,15], augmented by the introduction of the adaptive dynamic programming (ADP) approach, which aims to achieve optimal performance indices for (partially) model-free scenarios [15,16,17,18,19]. Extensive research has been conducted on developing optimal control schemes based on the ADP concept, particularly for applications in linear quadratic regulator (LQR) and linear quadratic tracking (LQT) problems, which was outlined comprehensively in [2,20,21,22,23] and other related references. It is worth noting that learning schemes in reinforcement learning generally involve two iterative steps: policy evaluation and policy update (with the latter focusing on policy improvement). However, it is essential to acknowledge that reinforcement learning based on value function approximation (VFA) introduces deliberate exploration noise to fully investigate systems, thereby undermining the algorithm’s convergence [23,24,25]. Furthermore, the policy iteration (PI) scheme within the adaptive dynamic programming (ADP) framework necessitates an initially admissible policy, which demands a priori knowledge of unknown systems to design robust controllers [22,26]. To overcome this requirement, recent studies have adopted value iteration (VI) methods [23,27,28] within value function approximation (VFA) schemes. Recently, event-triggered control approaches have also been applied to solve the adaptive optimal output regulation problem using PI and VI methods with one-step learning [29].

Most current studies in the field of control engineering have relied on the ability to measure the complete state information of systems [23,30], which is often challenging to achieve in practical engineering applications [31]. As such, the development of output feedback learning controllers has become essential. In the literature, dynamic output feedback controllers have been investigated [32], which rely on the Q-learning algorithm to solve the LQR control problem for discrete-time linear systems. Additionally, a state parameterization method for reconstructing system states based on filtered input and output signals has been proposed. In contrast, static output feedback designs are popular due to their simplicity and have been used to solve the LQR problem for continuous-time linear systems [33]. However, obtaining static output feedback controllers requires not only the complete state variable information of systems during the learning phase but also model-free state estimation techniques based on neural networks [34,35]. An alternative approach, first proposed in [24], is to use the measurements of past inputs, outputs, and reference trajectories in a system as substitutes for the unmeasurable system state to learn the output feedback LQR controller. This approach has also been extended to solve the output feedback LQT problem by employing the VFA technique [25]. Furthermore, model-free state reconstruction techniques have recently been applied to solve output feedback Q-learning PI schemes for $H_{\infty }$ control problems [36,37].

In this paper, we propose a tracking control approach that utilizes a static output feedback multistep Q-learning algorithm in conjunction with state reconstruction techniques. A separate adaptation mechanism was introduced in [38] to estimate unknown feedforward tracking terms. However, the static OPFB design we propose owes its popularity to its simplicity in terms of structure.

The key contributions of this work can be summarized as follows:

• Compared to the results reported in [23,39], the proposed approach does not apply an actor–critic structure, which are dependent on actor and critic NNs, to approximate control policy or value function. Moreover, the proposed model-free learning approach removes the requirement for the measurability of system state variables by collecting past input, output, and reference trajectory data. This is particularly advantageous in practical scenarios in which obtaining full state information may be challenging or costly;
• VFA-based learning [23,39] can ruin algorithm convergence due to the exploration noise that is intentionally added to evaluated policies to sufficiently excite systems. However, we apply the Q-learning scheme [40], which creates no biases in the estimated parameters of Q-function Bellman equations;
• Using the proposed multistep Q-learning technique [41], which surpasses the advantages of PI and VI methods, we are able to remove the requirement for an initial stabilizing control strategy. Moreover, this combination improves the convergence speed of the algorithm, leading to more efficient control performance.

The rest of this paper is organized as follows: Section 1 formulates the problem statement, Section 2 presents the proposed methodology, Section 3 displays the simulation results, and finally, Section 4 concludes the paper with some discussion and future research directions.

2. Problem Statement

This section will first review the problem of infinite-horizon LQT for DTL systems. Then, we will present some fundamental results for solving a discrete-time Bellman equation.

Consider a time-invariant DTL system described by the following state and output equations:

$$\begin{array}{cc}\hfill x_{k+1}& =A{x}_k+B{u}_k\hfill \\ \hfill y_k& =C{x}_k\hfill \end{array}$$

(1)

where $x_k\in {\mathbb{R}}^n$, $u_k\in {\mathbb{R}}^m$, and $y_k\in {\mathbb{R}}^p$ represent the state, input, and output, respectively. The matrices A, B, and C are constant matrices, where the pairs $(A,B)$ and $(A,C)$ are controllable and observable, respectively.

The reference trajectory is generated by the exogenous system:

$$r_{k+1}=F{r}_k$$

(2)

where $r_k\in {\mathbb{R}}^p$ and F is a constant matrix.

The tracking error is defined as follows:

$$e_k=y_k-r_k.$$

(3)

The goal is to create an optimal control policy, $u_k$, that allows the output, $y_k$, to track the reference trajectory, $r_k$, in an optimal way. This is achieved by minimizing the following discounted performance index:

$$J(x_k,r_k)=\frac{1}{2}\sum _{i=k}^{\infty }{\gamma }^{i-k}(e_i^{T}Q{e}_i+u_i^{T}R{u}_i)$$

(4)

where Q and R are positive definite weighting matrices, and $0<\gamma \le 1$ represents the discount factor.

Remark 1.

As stated in [40], the discount factor γ in (4) allows for a more general solution to the LQT problem compared to the standard setting. Importantly, the matrix F need not be stable, thus permitting a broader range of permissible reference signals for the tracking control problem with the quadratic performance index. Additionally, this framework allows for simultaneous optimization of both feedback and feedforward components of the control input, leading to a causal solution to the infinite-horizon LQT problem. It is worth noting that the use of the discount factor γ does not sacrifice generality, as one can set $\gamma =1$ when F is Hurwitz, reducing the LQT problem to an LQR problem with the specified output trajectory exponentially decaying to zero.

2.1. Offline Solution for LQT

By denoting $X_k={\left[\begin{array}{cc}{x}_k^T& r_k^T\end{array}\right]}^T$, we obtain the following augmented system:

$$\begin{array}{cc}\hfill X_{k+1}& =T{X}_k+B_1{u}_k\hfill \\ \hfill e_k& =C_1{X}_k\hfill \end{array}$$

(5)

where $T=\left[\begin{array}{cc}A& 0\\ 0& F\end{array}\right],B_1=\left[\begin{array}{c}B\\ 0\end{array}\right]$, and $C_1=\left[\begin{array}{cc}C& -I\end{array}\right]$.

It can be shown by Lemma 1 of [40] that, with the choice of $u_k=-K{X}_k$, where $K=\left[\begin{array}{cc}{K}_x& K_r\end{array}\right]$, the discounted performance index (4) can be expressed in a quadratic form as follows:

$$V(x_k,r_k)=V(X_k)=\frac{1}{2}{X}_k^{T}P{X}_k$$

(6)

where $P=P^T>0$.

Using Formula (4), the cost function can be expressed as follows:

$$J(x_k,r_k)=\frac{1}{2}(e_k^{T}Q{e}_k+u_k^{T}R{u}_k)+\frac{1}{2}\sum _{i=k+1}^{\infty }{\gamma }^{i-(k+1)}(e_i^{T}Q{e}_i+u_i^{T}R{u}_i)$$

(7)

Using Equation (6), the cost function $J(x_k,r_k)$ can be rewritten as $V(x_k,r_k)$, which can be expressed as follows: H = r + Cv

$$V(x_k,r_k)=\frac{1}{2}{e}_k^{T}Q{e}_k+\frac{1}{2}{u}_k^{T}R{u}_k+\gamma V(x_{k+1},r_{k+1})$$

(8)

Substituting Equation (6) into Equation (8) yields the LQT Bellman equation for P:

$$X_k^{T}P{X}_k=X_k^T\mathrm{\Pi }{X}_k+u_k^{T}R{u}_k+\gamma X_{k+1}^{T}P{X}_{k+1}$$

(9)

where $\mathrm{\Pi }=\left[\begin{array}{cc}{C}^{T}QC& -C^{T}Q\\ -QC& Q\end{array}\right]$.

Define the LQT Hamiltonian as

$$\frac{1}{2}H(X_k,u_k)=\frac{1}{2}{X}_k^T\mathrm{\Pi }{X}_k+\frac{1}{2}{u}_k^{T}R{u}_k+\gamma V(X_{k+1})-V(X_k)$$

(10)

By solving the stationary condition [40,42], i.e.,

$$\frac{\partial H(X_k,u_k)}{\partial u_k}=0$$

(11)

we can find the optimal control input

$$u_k=-K{X}_k=-K_x{x}_k-K_r{r}_k$$

(12)

where $K={(R+\gamma B_1^{T}P{B}_1)}^{-1}\gamma B_1^{T}PT$ and P satisfies the augmented algebraic Riccati equation (ARE):

$$\begin{array}{c}\hfill \mathrm{\Pi }-P+\gamma T^{T}PT-{\gamma }^2{T}^{T}P{B}_1{(R+\gamma B_1^{T}P{B}_1)}^{-1}{B}_1^{T}P=0\end{array}$$

(13)

Remark 2.

The augmented ARE (13) has a unique, positive definite solution P if the pair $(A,\sqrt{Q}C)$ is observable and ${\gamma }^{1/2}F$ is stable [25]. Additionally, a lower bound has been established for the discount factor to ensure the stability of the augmented system [43].

A direct solution to (13) is challenging due to the nonlinear relationship in the unknown parameter. Instead, we substitute (12) into (9) to obtain the augmented LQT Lyapunov equation:

$$\mathrm{\Pi }-P+K^{T}RK+\gamma {(T-B_{1}K)}^{T}P(T-B_{1}K)=0.$$

(14)

To address this issue, an offline PI algorithm [5] has been proposed as an iterative approach to compute the solution to (14). However, it requires complete knowledge of the augmented system dynamics. To overcome this limitation, a Q-learning scheme [40] was developed to solve the model-free LQT problem.

2.2. Q-Function Bellman Equation

Let $Z_k={\left[\begin{array}{cc}{X}_k^T& u_k^T\end{array}\right]}^T$; then, the discrete-time Q-function can be defined as follows:

$$Q(Z_k)=\frac{1}{2}{X}_k^T\mathrm{\Pi }{X}_k+\frac{1}{2}{u}_k^{T}R{u}_k+\gamma V(X_{k+1})$$

(15)

By substituting the augmented system dynamics (5) into (15), we obtain:

$$Q(Z_k)=\frac{1}{2}{Z}_k^T\tilde{H}{Z}_k$$

(16)

where

$$\tilde{H}=\left[\begin{array}{cc}\mathrm{\Pi }+\gamma T^{T}PT& \gamma T^{T}P{B}_1\\ \gamma B_1^{T}PT& R+\gamma B_1^{T}P{B}_1\end{array}\right]\equiv \left[\begin{array}{cc}{\tilde{H}}_{XX}& {\tilde{H}}_{Xu}\\ {\tilde{H}}_{uX}& {\tilde{H}}_{uu}\end{array}\right]$$

(17)

and $\tilde{H}$ is a kernel matrix and $\tilde{H}={\tilde{H}}^T$.

By applying $\frac{\partial Q(Z_k)}{\partial u_k}=0$, we can solve for $u_k$ as follows:

$$u_k=-{({\tilde{H}}_{uu})}^{-1}{\tilde{H}}_{uX}{X}_k$$

(18)

Furthermore, noticing that $Q(Z_k)=V(X_k)$ leads to the Q-function Bellman equation:

$$Z_k^T\tilde{H}{Z}_k=X_k^T\mathrm{\Pi }{X}_k+u_k^{T}R{u}_k+\gamma Z_{k+1}^T\tilde{H}{Z}_{k+1}$$

(19)

This equation expresses the connection between the Q-function and the kernel matrix $\tilde{H}$.

2.3. PI-Based Q-Learning for LQT

Based on the Q-function Bellman equation (19), the PI-based Q-learning solution for the LQT problem can be implemented using Algorithm 1, without relying on the system dynamics [40].

Algorithm 1 PI Q-learning Algorithm for LQT.

Initialization: Start with an admissible control policy $u_k^0$ with ${\tilde{H}}^0$. Procedure: 1: Policy Evaluation: For $j=0,1,\dots$, collect samples under $u_k^j$ to solve ${\tilde{H}}^{j+1}$ using the Q-function Bellman equation: $Z_k^T{\tilde{H}}^{j+1}{Z}_k=X_k^T\mathrm{\Pi }{X}_k+{(u_k^j)}^{T}R(u_k^j)+\gamma Z_{k+1}^T{\tilde{H}}^{j+1}{Z}_{k+1}$ 2: Policy Improvement: Compute the improved control policy as follows: $u_k^{j+1}=-{({\tilde{H}}_{uu}^{j+1})}^{-1}{\tilde{H}}_{uX}^{j+1}{X}_k$ 3: Stopping Criterion: Stop the iteration if $\parallel {\tilde{H}}^{j+1}-{\tilde{H}}^j\parallel <\epsilon$ for some specified small positive number $\epsilon$. Otherwise, let $j=j+1$ and go back to iteration. End Procedure

Algorithm 1 performs repeated iterations between policy evaluation and policy improvement until convergence. In contrast to the offline algorithm [40], Algorithm 1 conducts the policy improvement step using the learned kernel matrix ${\tilde{H}}^{j+1}$. This allows finding the optimal policy even under completely unknown dynamic conditions.

3. Methods

3.1. Multistep Q-Learning

Lemma 1.

[24] When the pair (A, C) of the DTL system (1) is observable, the state $X_k$ of the augmented system can be reconstructed from the past input, output, and reference signal trajectories:

$$X_k=\left[\begin{array}{ccc}{M}_u& M_y& M_r\end{array}\right]\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\end{array}\right]$$

(20)

where ${\overline{u}}_{k-1,k-N}={[u_{k-1}^T,u_{k-2}^T,\cdots ,u_{k-N}^T]}^T$ and ${\overline{y}}_{k-1,k-N}=[y_{k-1}^T,y_{k-2}^T,\cdots ,y_{k-N}^T]$, $N\le n$ are the sequences of input and output signals over the time interval $[k-N,k-1]$, respectively, and

$$M_u=\left[\begin{array}{c}{U}_N-A^N{W}_N^{+}{D}_N\\ 0\end{array}\right],M_y=\left[\begin{array}{c}{A}^N{W}_N^{+}\\ 0\end{array}\right],M_r=\left[\begin{array}{c}0\\ F^N\end{array}\right]$$

$$U_N=\left[\begin{array}{ccccc}{B}_1& A{B}_1& A^2{B}_1& \cdots & A^{N-1}{B}_1\end{array}\right]$$

$$W_N={\left[\begin{array}{ccccc}{\left(C{A}^{N-1}\right)}^T& {\left(C{A}^{N-2}\right)}^T& \cdots & CA& C\end{array}\right]}^T$$

$$D_N=\left[\begin{array}{ccccc}0& CB& CAB& \cdots & C{A}^{N-2}B\\ 0& 0& CB& \cdots & C{A}^{N-3}B\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& 0& CB\\ 0& 0& 0& 0& 0\end{array}\right]$$

$$W_N^{+}={\left(W_N^T{W}_N\right)}^{-1}{W}_N^T$$

Lemma 1 states that the Q-function Bellman equation (19) can be transformed by using the past input, output, and reference trajectory sequences. By substituting Equation (20) into Equation (16), we obtain

$$\begin{array}{c}Q\left(Z_k\right)=\frac{1}{2}{Z}_k^T\tilde{H}{Z}_k=\frac{1}{2}{\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\\ u_k\end{array}\right]}^{T}H\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\\ u_k\end{array}\right]\\ \hspace{1em}\hspace{1em}\stackrel{\Delta }{=}\frac{1}{2}{z}_k^{T}H{z}_k\end{array}$$

(21)

where

$$z_k=\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\\ u_k\end{array}\right]$$

$$H=H^T=\left[\begin{array}{cccc}{H}_{\overline{u}\overline{u}}& H_{\overline{u}\overline{y}}& H_{\overline{u}r}& H_{\overline{u}u}\\ H_{\overline{y}\overline{u}}& H_{\overline{y}\overline{y}}& H_{\overline{y}r}& H_{\overline{y}u}\\ H_{r\overline{u}}& H_{r\overline{y}}& H_{rr}& H_{ru}\\ H_{u\overline{u}}& H_{u\overline{y}}& H_{ur}& H_{uu}\end{array}\right]$$

$$\begin{array}{c}{H}_{\overline{u}\overline{u}}=M_u^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_u=M_u^T{\tilde{H}}_{XX}{M}_u\hfill \\ H_{\overline{u}\overline{y}}=M_u^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_y=M_u^T{\tilde{H}}_{XX}{M}_y\hfill \\ H_{\overline{u}r}=M_u^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_r=M_u^T{\tilde{H}}_{XX}{M}_r\hfill \\ H_{\overline{u}u}=\gamma M_u^T{T}^{T}P{B}_1=M_u^T{\tilde{H}}_{Xu}\hfill \\ H_{\overline{y}\overline{y}}=M_y^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_y=M_y^T{\tilde{H}}_{XX}{M}_y\hfill \\ H_{\overline{y}r}=M_y^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_r=M_y^T{\tilde{H}}_{XX}{M}_r\hfill \\ H_{\overline{y}u}=\gamma M_y^T{T}^{T}P{B}_1=M_y^T{\tilde{H}}_{Xu}\hfill \\ H_{rr}=M_r^T\left(\mathrm{\Pi }+\gamma T^{T}PT\right)M_r=M_r^T{\tilde{H}}_{XX}{M}_r\hfill \\ H_{ru}=\gamma M_r^T{T}^{T}P{B}_1=M_r^T{\tilde{H}}_{Xu}\hfill \\ H_{uu}=R+\gamma B_1^{T}P{B}_1={\tilde{H}}_{uu}\hfill \end{array}$$

According to the principle of optimality, the optimal control policy should satisfy $\frac{\partial Q\left(z_k\right)}{\partial u_k}=$ 0. Solving for $u_k$ from Equation (21) yields the optimal control policy $u_k^{*}$ as:

$$\begin{array}{c}{u}_k^{*}=-{\left(H_{uu}\right)}^{-1}\left(H_{u\overline{u}}{\overline{u}}_{k-1,k-N}+H_{u\overline{y}}{\overline{y}}_{k-1,k-N}+H_{ur}{r}_{k-N}\right)\hfill \\ =-{\left(H_{uu}\right)}^{-1}\left[\begin{array}{ccc}{H}_{u\overline{u}}& H_{u\overline{y}}& H_{ur}\end{array}\right]\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\end{array}\right]\hfill \\ =-K^{*}\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}\\ {\overline{y}}_{k-1,k-N}\\ r_{k-N}\end{array}\right]\hfill \end{array}$$

(22)

where $K^{*}={\left(H_{uu}\right)}^{-1}\left[\begin{array}{ccc}{H}_{u\overline{u}}& H_{u\overline{y}}& H_{ur}\end{array}\right]$.

By substituting Equation (21) into Equation (19) with the utility function $r\left({\tau }_k,u_k\right)={\tau }_k^T\mathrm{\Gamma }{\tau }_k+u_k^{T}R{u}_k$, we have the Q-function Bellman equation incorporating input, output, and reference trajectory sequences, which is expressed as follows:

$$z_k^{T}H{z}_k=r\left({\tau }_k,u_k\right)+\gamma z_{k+1}^{T}H{z}_{k+1}$$

(23)

where ${\tau }_k=\left[\begin{array}{c}{y}_k\\ \\ r_k\end{array}\right]$ and $\mathrm{\Gamma }=\left[\begin{array}{cc}Q& -Q\\ -Q& Q\end{array}\right]$.

Define the optimal value function $V^{*}(z_k)\triangleq V_{u^{*}}(z_k)$. According to the optimal control theory [1], $V^{*}(z_k)$ satisfies the following Bellman equation:

$$V^{*}(z_k)=\underset{u(z)}{min}\left\{r\left({\tau }_k,u_k\right)+\gamma V^{*}(z_{k+1})\right\}$$

(24)

and the optimal control is

$$u_k^{*}=arg\underset{u(z)}{min}\left\{r\left({\tau }_k,u_k\right)+\gamma V^{*}(z_{k+1})\right\}$$

(25)

It is known that the policy evaluation step in the VI scheme is expressed as follows [23]:

$$z_k^T{H}^{\mathrm{j}+1}{z}_k=r({\tau }_k,u_k^j)+\gamma z_{k+1}^T{H}^j{z}_{k+1}$$

(26)

Transforming Equation (26) yields

$$\begin{array}{cc}\hfill z_k^T{H}^{j+1}{z}_k& =r\left({\tau }_k,u_k^j\right)+\gamma z_{k+1}^T{H}^j{z}_{k+1}\hfill \\ \hfill & =r\left({\tau }_k,u_k^j\right)+\gamma \left(r\left({\tau }_{k+1},u_{k+1}^j\right)+\gamma z_{k+2}^T{H}^j{z}_{k+2}\right)\hfill \\ \hfill & =r\left({\tau }_k,u_k^j\right)+\gamma r\left({\tau }_{k+1},u_{k+1}^j\right)+{\gamma }^2\left(r\left({\tau }_{k+2},u_{k+2}^j\right)+\gamma z_{k+3}^T{H}^j{z}_{k+3}\right)\hfill \\ \hfill & ⋮\hfill \\ \hfill & =\sum _{i=k}^{k+N_j-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j}{z}_{k+N_j}^T{H}^j{z}_{k+N_j}\hfill \end{array}$$

(27)

Thus, the convergence of the VI method [23] can be accelerated by introducing a multistep utility function in the policy evaluation. The resulting multistep Q-learning VI algorithm based on output feedback is described as follows:

• Step 1. Initialization: Set $j=0$ and iterate from any initial control policy $u_k^0$, which does not need to be stable or controllable, and $H^0$.
• Step 2. Multistep policy evaluation: Use the Q-function Bellman equation to solve $H^{j+1}$, where

  $$z_k^T{H}^{j+1}{z}_k=\sum _{i=k}^{k+N_j-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j}{z}_{k+N_j}^T{H}^j{z}_{k+N_j}$$

  (28)
• Step 3. Policy improvement: Update the control policy $u_k^{j+1}$ as follows:

  $$u_k^{j+1}=-{\left(H_{uu}^{j+1}\right)}^{-1}\left(H_{u\overline{u}}^{j+1}{\overline{u}}_{k-1,k-N}+H_{u\overline{y}}^{j+1}{\overline{y}}_{k-1,k-N}+H_{ur}^{j+1}{r}_{k-N}\right)$$

  (29)
• Step 4. Termination condition: Check if $∥H^{j+1}-H^j∥\le l$, where l is a very small threshold with pre-set algorithmic accuracy. If this condition is satisfied, terminate the iteration and obtain the optimal control policy $u_k^{j+1}$. Otherwise, return to Step 2 and repeat the iteration.

Remark 3.

As indicated in Equation (28), it is observed that the resultant value function in each policy evaluation step is the sum of the one-step utility function and the previous value function. When $N_j=1$, Equation (28) simplifies to the policy evaluation step, which uses the VI framework for one-step learning [23,28]. The difference in policy evaluation leads to contrasting merits in value iteration. Unlike the traditional value iteration method, the proposed multistep Q-learning VI algorithm takes advantage of a finite-sum utility function instead of a one-step calculation as in value iteration. Consequently, the proposed algorithm leads to an improvement on the learning convergent speed, as demonstrated in the simulation example.

3.2. Adjustment Rules for Step Size $N_j$

During each iteration of the multistep policy evaluation (28), the step size is adjusted. The value iteration algorithm [23] employs a one-step policy evaluation to eliminate the requirement for an initial stabilizing control policy. On the other hand, the policy iteration algorithm uses an infinite-step strategy evaluation to speed up convergence [39]. However, the speed of convergence of the multistep Q-learning algorithm depends on the chosen step size. Initially, a small step size is used in the iteration to avoid the need for an initial stabilizing control policy. Then, the step size is gradually increased to speed up convergence. To adaptively adjust the step length, we use the following rule [41]:

$$N_j=1+\left[\beta \sqrt{j}\right]$$

(30)

where $\beta \ge 0,[·]$ means rounding down. When $\beta =0$ and $N_j=1$, it is equivalent to the VI method with one-step policy evaluation [23].

3.3. Implementation

By using the least squares method, the linear parametric expression of $z_k^T{H}^{j+1}{z}_k$ is given as follows:

$$z_k^T{H}^{j+1}{z}_k={\left({\overline{H}}^{j+1}\right)}^T\overline{z}(k)$$

(31)

where

$${\overline{H}}^{j+1}=vec\left(H^{j+1}\right)\in {\mathbb{R}}^{l(l+1))/2}\equiv {\left[H_{1l}^{j+1},2H_{12}^{j+1},\cdots ,2H_{1l}^{j+1},H_{21}^{j+1},\cdots ,2H_{2l}^{j+1},\cdots ,H_{ll}^{j+1}\right]}^T$$

Here, $H_{ik}^{j+1}$ represents the element in the i-th row and k-th column of matrix $H^{j+1}$, where $i,k=1,2,\cdots ,l$, and $l=mN+pN+m$. The Kronecker product ${\overline{z}}_k=z_k\otimes z_k$ is defined as $\left[z_1^2,z_1{z}_2,\cdots ,z_1{z}_l,z_2^2,z_2{z}_3,\cdots ,z_2{z}_l,\cdots ,z_l^2\right]\in {\mathbb{R}}^{l(l+1)/2}$.

Combining Equation (31) with Equation (28), it can be simplified as:

$${\left({\overline{H}}^{j+1}\right)}^T{\overline{z}}_k=\sum _{i=k}^{k+N_j-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j}{\left({\overline{H}}^j\right)}^T{\overline{z}}_{k+N_j}$$

(32)

The symmetric matrix $H^{j+1}$ has dimensions of $l\times l$, resulting in a total of $l(l+1)/2$ independent elements. Consequently, Equation (32) requires collecting at least $L\ge l(l+1)/2$ sets of ${\overline{z}}_k$ for solving.

The least squares expression for Equation (28) is

$${\overline{H}}^{j+1}={\left\{{\left\{{\mathrm{\Phi }}^j\right\}}^T\left\{{\mathrm{\Phi }}^j\right\}\right\}}^{-1}{\left\{{\mathrm{\Phi }}^j\right\}}^T\left\{{{\rm Y}}^j+{\gamma }^{N_j}{\mathrm{\Psi }}^j{\overline{H}}^j\right\}$$

(33)

where

$${\mathrm{\Phi }}^j=\left[\begin{array}{c}{\overline{z}}_k\\ {\overline{z}}_{k+1}\\ ⋮\\ {\overline{z}}_{k+L-1}\end{array}\right]\in {\mathbb{R}}^{L\times l(l+1)/2}$$

$${{\rm Y}}^j=\left[\begin{array}{c}\sum _{i=k}^{k+N_j-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)\\ \sum _{i=k}^{k+N_j}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)\\ ⋮\\ \sum _{i=k}^{k+N_j+L-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)\end{array}\right]\in {\mathbb{R}}^{L\times 1}$$

$${\mathrm{\Psi }}^j=\left[\begin{array}{c}{\overline{z}}_{k+N_j}\\ {\overline{z}}_{k+N_j+1}\\ ⋮\\ {\overline{z}}_{k+N_j+L-1}\end{array}\right]\in {\mathbb{R}}^{L\times l(l+1)/2)}$$

Remark 4.

When $k\le N$, the input–output data ${\overline{u}}_{k-1,k-N}$ and ${\overline{y}}_{k-1,k-N}$ will be unavailable. To address this issue, the internal model principle can be utilized to collect the missing data. Additionally, the internal model principle allows for asymptotic tracking control in the presence of small variations in system parameters, resulting in data that contain more intrinsic information for learning the optimal control solution.

Remark 5.

In Equation (33), the vector ${\overline{H}}^{j+1}$ represents the jth estimated value of $H^{j+1}$ under the current control policy. By using the components of ${\overline{H}}^{j+1}$, we can infer the corresponding components of matrix $H^{j+1}$ and use them, along with the policy update step (29), to solve for the next step’s control policy. This updated policy can then be used to gather a new set of data for each iteration until obtaining an optimal control policy. To ensure uniqueness in solving Equation (33), a persistent excitation condition is proposed in the literature [20,21], where probing noise $w_k$ is added to the control input, ensuring that ${\mathrm{\Phi }}^j$ is full rank and ${({\mathrm{\Phi }}^j)}^T({\mathrm{\Phi }}^j)$ is invertible. However, using the VFA method may result in bias in finding an optimal solution [23,25]. On the other hand, the Q-learning approach does not produce bias during the parameter estimation process and, hence, does not lead to bias in finding an optimal solution.

3.4. Convergence Analysis

In reference [41], a multistep Q-learning algorithm based on state feedback is proposed for solving the optimal output regulation problem of DTL systems, and the convergence of the proposed algorithm is derived. This paper investigates a multistep Q-learning algorithm based on output feedback to solve the optimal output tracking control problem of discrete-time linear systems. Unlike the optimal output regulation problem studied in reference [41], this paper introduces a discount factor $\gamma$ into the performance index function of the optimal output tracking control problem, resulting in changes to the corresponding Bellman equation. The system state is reconstructed using input, output, and reference signals. Therefore, it is necessary to verify the convergence of the output feedback multistep Q-learning algorithm for solving the optimal output tracking control problem of DTL systems.

The convergence of the algorithm illustrated in Section 2.1 can be proven by first noticing that Equation (28) can be rewritten as follows:

$$Q^{j+1}\left(z_k\right)=\frac{1}{2}\sum _{i=k}^{k+N_j-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j}{Q}^j\left(z_{k+N_j}\right)$$

(34)

Here, we are ready to obtain the following theorem, which indicates that the policy matrix $H^{j+1}$ converges to the optimal value $H^{*}$.

Theorem 1.

Let $\left\{Q^j\left(z_k\right)\right\},where Q^j\left(z_k\right)=\frac{1}{2}{z}_k^T{H}^j{z}_k$, be the sequence generated by the multistep $\mathrm{Q}$-learning algorithm. If $N_j\ge 1$ and

$$Q^0\left(z_k\right)\ge \underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^0\left(z_{k+1}\right)\right\}$$

(35)

holds, then

(i)

For any j,

$$Q^{j+1}\left(z_k\right)\le \underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^j\left(z_{k+1}\right)\right\}\le Q^j\left(z_k\right)$$

(36)

holds;

(ii)

${lim}_{j\to \infty }{Q}^j\left(z_k\right)=Q^{*}\left(z_k\right)$, where $Q^{\infty }\left(z_k\right)$ is the optimal solution to the Q-function Bellman equation.

Proof .

(i) We will use mathematical induction to prove the result (36). From Equations (34) and (35), we have

$$\begin{array}{cc}\hfill Q^1\left(z_k\right)& =\frac{1}{2}\sum _{i=k}^{k+N_0-1}{\gamma }^{i-k}r\left({\tau }_i,u_i^0\right)+{\gamma }^{N_0}{Q}^0\left(z_{k+N_0}\right)\hfill \\ \hfill & =\frac{1}{2}\sum _{i=k}^{k+N_0-2}{\gamma }^{i-k}r\left({\tau }_i,u_i^0\right)+\frac{1}{2}{\gamma }^{N_0-1}r\left({\tau }_{k+N_0-1},u_{k+N_0-1}^0\right)+{\gamma }^{N_0}{Q}^0\left(z_{k+N_0}\right)\hfill \\ \hfill & =\frac{1}{2}\sum _{i=k}^{k+N_0-2}{\gamma }^{i-k}r\left({\tau }_i,u_i^0\right)+{\gamma }^{N_0-1}\underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_{k+N_0-1},u_{k+N_0-1}\right)+\gamma Q^0\left(z_{k+N_0}\right)\right\}\hfill \\ \hfill & \le \frac{1}{2}\sum _{i=k}^{k+N_0-2}{\gamma }^{i-k}r\left({\tau }_i,u_i^0\right)+{\gamma }^{N_0-1}{Q}^0\left(z_{k+N_0-1}\right)\hfill \\ \hfill & ⋮\hfill \\ \hfill & \le \frac{1}{2}r\left({\tau }_k,u_k^0\right)+\gamma Q^0\left(z_{k+1}\right)\hfill \\ \hfill & =\underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^0\left(z_{k+1}\right)\right\}\le Q^0\left(z_k\right)\hfill \end{array}$$

(37)

which means that Equation (36) holds for $j=0$.

Next, assume that Equation (36) is satisfied for $j-1$, i.e.,

$$Q^j\left(z_k\right)\le \underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^{j-1}\left(z_{k+1}\right)\right\}\le Q^{j-1}\left(z_k\right)$$

Then,

$$\begin{array}{cc}\hfill Q^j(z_k)& =\frac{1}{2}\sum _{i=k}^{k+N_{j-1}-1}{\gamma }^{i-k}r({\tau }_i,u_i^{j-1})+{\gamma }^{N_{j-1}}{Q}^{j-1}(z_{k+N_{j-1}})\hfill \\ \hfill & \ge \frac{1}{2}\sum _{i=k}^{k+N_{j-1}-1}{\gamma }^{i-k}r({\tau }_i,u_i^{j-1})+{\gamma }^{N_{j-1}}\underset{u(z)}{min}\left\{\frac{1}{2}r({\tau }_{k+N_{j-1}},u_{k+N_{j-1}})+\gamma Q^{j-1}(z_{k+N_{j-1}+1})\right\}\hfill \\ \hfill & =\frac{1}{2}r({\tau }_k,u_k^{j-1})+\frac{1}{2}\sum _{i=k+1}^{k+N_{j-1}-1}{\gamma }^{i-k}r({\tau }_i,u_i^{j-1})+{\gamma }^{N_{j-1}}{Q}^{j-1}(z_{k+N_{j-1}+1})\hfill \\ \hfill & =\frac{1}{2}r({\tau }_k,u_k^{j-1})+\gamma \left\{\sum _{i=k+1}^{k+N_{j-1}}\frac{1}{2}{\gamma }^{i-\left(k+1\right)}r({\tau }_i,u_i^{j-1})+{\gamma }^{N_{j-1}}{Q}^{j-1}(z_{k+N_{j-1}+1})\right\}\hfill \\ \hfill & =\frac{1}{2}r({\tau }_k,u_k^{j-1})+\gamma Q^j(z_{k+1})\ge \underset{u(z)}{min}\left\{\frac{1}{2}r({\tau }_k,u_k)+\gamma Q^j{z}_{k+1}\right\}\hfill \end{array}$$

(38)

Using Equations (34) and (38), we have

$$\begin{array}{cc}\hfill Q^{j+1}\left(z_k\right)& =\sum _{i=k}^{k+N_j-1}\frac{1}{2}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j}{Q}^j\left(z_{k+N_j}\right)\hfill \\ \hfill & =\sum _{i=k}^{k+N_j-2}\frac{1}{2}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j-1}\underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_{k+N_j-1},u_{k_{+N_j-1}}\right)+\gamma Q^j\left(z_{k+N_j}\right)\right\}\hfill \\ \hfill & \le \sum _{i=k}^{k+N_j-2}\frac{1}{2}{\gamma }^{i-k}r\left({\tau }_i,u_i^j\right)+{\gamma }^{N_j-1}{Q}^j\left(z_{k+N_j-1}\right)\hfill \\ \hfill & ⋮\hfill \\ \hfill & \le \frac{1}{2}r\left({\tau }_k,u_k^j\right)+\gamma Q^j\left(z_{k+1}\right)\hfill \\ \hfill & =\underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^j\left(z_{k+1}\right)\right\}\hfill \end{array}$$

(39)

Thus, Equation (36) holds for all j.

(ii)

According to the conclusion (36), $\left\{Q^j\left(z_k\right)\right\}$ is a monotonically non-increasing sequence with a lower bound of $Q^j\left(z_k\right)\ge 0$. For a bounded monotone sequence, we can always have a limit denoted by $Q^{\infty }\left(z_k\right)\triangleq {lim}_{j\to \infty }{Q}^j\left(z_k\right)$. Take the limit of Equation (36):

$$Q^{\infty }\left(z_k\right)\le \underset{u(z)}{min}\left\{\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^{\infty }\left(z_{k+1}\right)\right\}\le Q^{\infty }\left(z_k\right)$$

(40)

Here, we have

$$Q^{\infty }\left(z_k\right)=\frac{1}{2}r\left({\tau }_k,u_k\right)+\gamma Q^{\infty }\left(z_{k+1}\right)$$

(41)

Notice that Equation (41) is the solution to the Q-function Bellman equation (23). As a result, $Q^{\infty }\left(z_k\right)=Q^{*}\left(z_k\right)$.

 □

Remark 6.

It is worth noting the choice of the initial value function $Q^0$ in multistep Q-learning algorithms. According to Theorem 1, $Q^0$ must satisfy Condition (35). However, (35) is only a sufficient condition, not a necessary one. Therefore, in practical systems, $Q^0$ can be a positive definite function over a large range and can be chosen through trial and error.

4. Simulation Experiment

4.1. Controlled Object

To validate the proposed algorithm, a simulation experiment was conducted on a single-phase voltage source uninterruptible power supply (UPS) inverter, an essential component of the smart grid. With the development of new energy technologies, it is important for the control engineer to design a controller that makes UPS provide efficient and stable sinusoidal output voltages with optimal performance even in the presence of unknown loads. The circuit diagram of the single-phase voltage source UPS inverter is illustrated in Figure 1.

The dynamic equations of the inverter can be expressed as follows:

$$\begin{array}{c}{C}_f\frac{d{v}_o}{dt}=i_L-i_o\\ L_f\frac{d{i}_L}{dt}+r{i}_L=u{V}_s-v_o\end{array}$$

(42)

where $L_f$ represents the filter inductance; $C_f$ represents the filter capacitance; r represents the inductance resistance; $i_L$ represents the filter inductance current; $v_0$ represents the output voltage of the inverter, which is the output voltage of the pulse width modulation (PWM) inverter bridge represented as $u{V}_s$; $i_o=\frac{v_o}{R_o}$ represents the output current; and $R_o$ represents the resistance value of the power grid.

Choosing $v_0$ and $i_L$ as the system state variables, $u{V}_s$ as the system input, and $v_0$ as the system output, we can obtain the state space representation of the single-phase voltage source UPS inverter as follows:

$$\begin{array}{c}\dot{x}=\overline{A}x+\overline{B}u=\left[\begin{array}{cc}-\frac{1}{R_o{C}_f}& \frac{1}{C_f}\\ -\frac{1}{L_f}& -\frac{r}{L_f}\end{array}\right]x+\left[\begin{array}{c}0\\ \frac{1}{L_f}\end{array}\right]u\\ y=\overline{C}x=\left[\begin{array}{cc}1& 0\end{array}\right]x\end{array}$$

(43)

In the above equations, the inverter model parameters are as follows: $L_f=3.56$$\mathrm{mH}$, $C_f=9.92$$\mathrm{\mu F}$, $r=0.4$$\mathrm{\Omega }$, and $R_o=50$$\mathrm{\Omega }$. The initial values for the capacitor voltage and inductor current are set as $x_0={[0,0]}^T$.

By discretizing Equation (43), the state space representation of the discrete-time system can be obtained as follows:

$$\begin{array}{c}{x}_{k+1}=A{x}_k+B{u}_k\\ y_k=C{x}_k\end{array}$$

(44)

where $A=e^{\overline{A}T}$, $B={\int }_0^T{e}^{\overline{A}\tau }d\tau \overline{B}$, and $C=\overline{C}$. The sampling interval is $T={10}^{-4} \mathrm{s}$. Substituting the inverter model parameters into the equations, we have:

$$A=\left[\begin{array}{cc}0.6969& 806545\\ -0.0241& 0.8603\end{array}\right],B=\left[\begin{array}{c}0.1290\\ 0.0267\end{array}\right],C=\left[\begin{array}{cc}1& 0\end{array}\right]$$

(45)

A sinusoidal signal is a typical type of reference signal in power electronics control. The state space representation of a continuous-time system with a sinusoidal signal of magnitude $220\sqrt{2} \mathrm{V}$ and frequency $f=50 \mathrm{Hz}$ is given by:

$$\begin{array}{c}{\dot{x}}_d=\left[\begin{array}{cc}0& 2\pi f\\ -2\pi f& 0\end{array}\right]x_d=\left[\begin{array}{cc}0& 100\pi \\ -100\pi & 0\end{array}\right]x_d\\ r_d=\left[\begin{array}{cc}1& 0\end{array}\right]x_d\end{array}$$

(46)

where the initial state $x_d(0)={[0,1]}^T$. The state space expression for the DTL system corresponding to Equation (46) can be described in the form shown in Equation (2), where

$$F=\left[\begin{array}{cc}0.9995& 0.0314\\ \\ -0.0314& 0.9995\end{array}\right]$$

4.2. Experiment

We select $Q=0.1,R=0.001$, and $\gamma =0.009$ as the parameter values. For each step, we set $\beta =4$ for $N_j$. The detection noise $w_k$ is defined as follows:

$$w_k=0.001\left(7sin(k)+5cos(2k)+9sin(8k)+2cos(6k)\right)$$

(47)

The controlled system used in the simulation experiment has 28 independent variables in ${\overline{H}}^{j+1}$, as shown in Equation (33). Therefore, a minimum of 28 sets of data are required for each iteration. In this simulation, we collected 30 sets of data. The following are the simulation results obtained using MATLAB.

The tracking curve of the output feedback multistep Q-learning algorithm is depicted in Figure 2. In this figure, $r_d$ represents the sinusoidal reference signal, and $y_o$ represents the actual output of the controlled system. After a certain number of iterations, the system output $y_o$ successfully tracks the reference signal $r_d$. The corresponding tracking error curve is illustrated in Figure 3. From this figure, it can be observed that the tracking error reaches zero at 0.012 s, achieving the tracking goal. Figure 4 illustrates the norm of the difference between the Q-function matrices after two consecutive iterations in the output feedback multistep Q-learning algorithm. In the figure, it is shown that $\left|H^{j+1}-H^j\right|<{10}^{-6}$ in the third iteration, indicating that the desired control accuracy has been achieved, and the algorithm has successfully converged. The simulation for the one-step learning case when $N_j=1$ has also been conducted for clarity. It can be observed from Figure 5 that $\left|H^{j+1}-H^j\right|\approx 4.12\times {10}^{-5}$ for the third iteration, which is larger than the value of $9.01\times {10}^{-7}$ in Figure 4. Therefore, the proposed multistep Q-learning scheme improves the learning convergence speed.

Figure 6 shows the variations in each component of the control gain K during each iteration process of the output feedback multistep Q-learning algorithm. Through iterations, it converges to

$$K^3=\left[\begin{array}{cccccc}0.03769& 0.02188& 0.1879& -0.1472& -0.1163& -0.01107\end{array}\right]$$

with

$$H^3=\left[\begin{array}{ccccccc}0.001758& 0.001603& 0.0056& -0.01079& -0.01316& -0.0008375& 3.563e-5\\ 0.01603& 0.001473& 0.01897& -0.009919& -0.01215& -0.00077& 2.243e-5\\ 0.02056& 0.01897& 0.2449& -0.1277& -0.1569& -0.009916& 0.0001926\\ -0.01079& -0.00919& -0.1277& 0.06677& 0.08181& 0.005183& -0.0001508\\ -0.01316& -0.01215& -0.1569& 0.8181& 0.1005& 0.006352& -0.001192\\ -0.008375& -0.0077& -0.009916& 0.05183& 0.006352& 0.004024& -1.135e-5\\ 3.863e-5& 2.243e-5& 0.001926& -0.001508& -0.001192& -1.135e-5& 0.001025\end{array}\right]$$

By solving the algebraic Riccati equation (13), the theoretical optimal values for the control gain K and the matrix H can be obtained as follows:

$$K^{*}=\left[\begin{array}{cccccc}0.0377& 0.0219& 0.1879& -0.1472& -0.1163& -0.0111\end{array}\right]$$

and

$$H^{*}=\left[\begin{array}{ccccccc}0.001758& 0.001603& 0.02056& -0.01079& -0.01316& -0.0008375& 3.828e-5\\ 0.001603& 0.001473& 0.01897& -0.009919& -0.01215& -0.00077& 2.221e-5\\ 0.02056& 0.01897& 0.2449& -0.1277& -0.1569& -0.009916& 0.0001909\\ -0.01079& -0.009919& -0.1277& 0.06677& 0.08181& 0.005183& -0.0001495\\ -0.01316& -0.01215& -0.1569& 0.08181& 0.1005& 0.006352& -0.0001181\\ -0.0008375& -0.00077& -0.009916& 0.005183& 0.006352& 0.0004024& -1.125e-5\\ 3.828e-5& 2.221e-5& 0.0001909& -0.0001495& -0.0001181& -1.125e-5& 0.001016\end{array}\right]$$

The Q function matrix $H^3$ obtained using the proposed learning algorithm is observed to be nearly equal to the theoretical optimal value $H^{*}$, indicating the effectiveness of the proposed output feedback multistep Q-learning algorithm for model-free tracking control.

Figure 7 shows the input signal trajectory of the actual tracking control system, which is a sinusoidal signal after the first four iterations. Figure 8 depicts the waveform of the excitation noise signal, which becomes zero at 0.0091 s, indicating the end of the algorithm learning phase without further noise excitation input. Figure 9 illustrates the tracking error of the system as the value of the system resistance parameter varies within the range of 40 $\mathrm{\Omega }\le R_o\le 60$$\mathrm{\Omega }$. With the change in resistance values, the system maintains its ability to achieve asymptotic tracking, demonstrating the adaptive characteristic of the algorithm. Figure 10 shows the variation in the step size $N_j$ in the multistep Q-learning algorithm. It can be observed that $N_j$ gradually increases as the iterations increase.

5. Conclusions

In this paper, we investigate a value iteration (VI)-based multistep Q-learning algorithm for model-free optimal tracking controller design of unknown discrete-time linear (DTL) systems. By utilizing the augmented system approach, we transform this problem into a regulation problem with a discounted performance function, that depends on the Q-function Bellman equation. To solve the Bellman equation, we employe the VI learning mechanism and develop a multistep Q-learning algorithm that eliminates the need for an initial admissible policy and only requires measurements of past input, output, and reference trajectory data. As a result, our proposed approach offers a novel solution that does not require state measurements and has improved convergence learning speed. To validate the effectiveness of the proposed design, we demonstrate its application through a simulation example. Future work will involve extending the proposed multistep Q-learning scheme to unknown discrete-time systems with time delays and/or sampling errors. Additionally, it would be interesting to explore how to balance the computational demands of the algorithm with the available arithmetic power in practical experimental platforms.