Reinforcement Learning-Based Tracking Control under Stochastic Noise and Unmeasurable State for Tip–Tilt Mirror Systems

Abstract

The tip–tilt mirror (TTM) is an important component of adaptive optics (AO) to achieve beam stabilization and pointing tracking. In many practical applications, the information of accurate TTM dynamics, complete system state, and noise characteristics is difficult to achieve due to the lack of sufficient sensors, which then restricts the implementation of high precision tracking control for TTM. To this end, this paper proposes a new method based on noisy-output feedback Q-learning. Without relying on neural networks or additional sensors, it infers the dynamics of the controlled system and reference jitter using only noisy measurements, thereby achieving optimal tracking control for the TTM system. We have established a modified Bellman equation based on estimation theory, directly linking noisy measurements to system performance. On this basis, a fast iterative learning of the control law is implemented through the adaptive transversal predictor and experience replay technique, making the algorithm more efficient. The proposed algorithm has been validated with an application to a TTM tracking control system, which is capable of quickly learning near-optimal control law under the interference of random noise. In terms of tracking performance, the method reduces the tracking error by up to 98.7% compared with the traditional integral control while maintaining a stable control process. Therefore, this approach may provide an intelligent solution for control issues in AO systems.

1. Introduction

The tip–tilt mirror (TTM) tracking control system is one of the most important components of the adaptive optics (AO) system, which is primarily used in various optical systems to compensate for beam jitter caused by the atmosphere, mechanical vibrations, and other factors in the optical path. The aim is to achieve beam stabilization and enhance the pointing accuracy or imaging quality of the system [1,2,3]. Although the traditional integral controllers most used in TTM systems are simple in design and stable, they have limited control bandwidth, and thus it is increasingly difficult to satisfy the system’s needs in increasingly complex application scenarios. With the advancement of artificial intelligence (AI), many studies have integrated machine learning techniques into system control schemes to establish intelligent systems and enhance performance [4,5,6]. The authors in [7] developed a reinforcement learning (RL) method based on recurrent neural networks (RNNs) for jitter suppression in TTM systems, achieving control performance superior to that of the optimal gain integrator. However, most existing machine learning-based methods focus on the development of neural networks as the key to controller design, and complex network models require a large amount of sampling data and time consumption during training, which is not conducive to the application needs of practical systems.

In order to implement a simpler and more efficient reinforcement learning method in the system, Werbos [8] attempted to interpret the reinforcement learning concept from the perspective of optimal control computation and proposed the Adaptive Dynamic Programming (ADP) algorithm. Compared to traditional deep reinforcement learning algorithms, the ADP method provides a convex optimization control index and replaces the slow update of neural networks with efficient iterative numerical solutions, greatly improving the efficiency and stability of the algorithm. This makes ADP-based RL a powerful tool for solving the dynamic optimal control problems of unknown systems (including optimal tracking control) [9,10,11]. And therefore, the research based on ADP-based RL has been widely carried out in simulated systems such as autonomous vehicles [12,13], energy systems [14,15], and robot control [16,17]. However, in many of the TTM systems, there are two more factors that affect the design and performance of a controller except for the unknown dynamics. The first is the unmeasurable system state. The complexity of real systems and the absence of matching sensors results in incomplete system state information, making it tough to control such partially observed Markov decision processes (POMDPs) [18] through standard state feedback methods. The second one is the stochastic noise introduced by the sensors or by the environment, which requires us to analyze the problem from the perspective of stochastic system control.

On the one hand, in addressing scenarios with unmeasurable state information, numerous studies have leveraged neural networks for state estimation instead of traditional full-state observers [19,20,21]. Despite their ability to capture system nonlinearities, these networks require meticulous design to ensure stability and a large amount of training time for convergency. Alternative approaches [22,23,24] focus on output feedback (OPFB), enabling linear state reconstruction from input–output data, thereby facilitating optimal control outcomes without estimation errors. The aforementioned methods, while showing progress in managing unmeasurable states, are tailored for deterministic systems. Those methods that rely on input–output data for system state reconstruction will become ineffective when measurement data are corrupted by stochastic noise. This susceptibility to noise significantly restricts their applicability in practical scenarios, where both measurement and computational noise are inherent.

On the other hand, for the control problem of stochastic systems containing noise, a system transformation method was applied in [25] to convert the stochastic linear-quadratic (SLQ) control problem with signal-dependent noise into a deterministic framework. The authors of [26] introduced an optimistic least-squares policy iteration (O-LSPI) strategy for controlling linear stochastic systems subjected to additive noise. Furthermore, the control challenges for stochastic systems with both signal-dependent and additive noise have been studied in [27,28]. To deal with stochastic systems with unmeasurable state, the authors of [29] designed a data-driven two-stage control strategy to obtain the optimal observer and optimal controller, respectively. It is worth noting that only signal-dependent noise is considered in [25], while actual measurement noise is typically additive in nature. The system state is assumed to be known in [26,28], and the work in [20,27] relies on the assumption that the noise is measurable.

The TTM systems are typically stochastic systems that simultaneously contain unknown noise and unmeasurable state. The above discussion implies that it still remains a challenge to design optimal controllers for TTM systems using only noisy input–output data when both the system state and the noise information are unavailable. Motivated by this, this paper develops an improved RL-based tracking control scheme for TTM systems. To the best of our knowledge, there is currently little relevant research on this issue, which means that our work has greater potential and advantage in solving tracking control problems in practice. We validate the effectiveness of the scheme to achieve tracking control in a TTM system. The main contributions of the proposed scheme are summarized as follows:

• A RL-based noisy-OPFB Q-learning scheme is proposed, which achieves a rapid and effective solution for the near optimal tracking controller according to state estimation theory, transversal prediction, and experience replay techniques.
• In contrast to the works in [25,27,28], the additive noise independent of state and inputs are taken into account, and the measurable assumption of full state or noise information is removed; only measured input–output data are used during the entire learning process. Also, an additional state observation step is not required, so it is more convenient to implement than the methods in [29].
• The effectiveness of the algorithm is demonstrated through an application example in a TTM tracking control system. Meanwhile, the application prospects of the method in the control of intelligent optical systems have been illustrated by a comparison with the traditional integral controller.

The remainder of this paper is organized as follows: The problem and some preliminaries are stated in Section 2. Section 3 describes the main development of this paper, where a noisy-OPFB Q-learning scheme for stochastic linear quadratic tracking systems is established. Section 4 presents a TTM tracking control system that demonstrates the effectiveness of the proposed algorithm through numerical application examples and analysis. Finally, some concluding remarks are summarized in Section 5.

2. Problem Formulation

A schematic diagram of a classic TTM control system is shown in Figure 1. The far-field camera detects the positional error of the focused spot and feeds it back to the controller. The controller calculates the control voltage and drives the TTM to rapidly deflect the mirror in the opposite direction, compensating for the beam jitter. In principle, this process can also be seen as a tracking control of the beam jitter signal. It can be seen that without adding additional sensors, the system can only obtain the tracking error signal contaminated by the camera’s stochastic noise. The model and state information of the controlled TTM system and the reference jitter signal are unknown, and the noise is unmeasurable. Therefore, designing a stochastic optimal tracking controller under these conditions is challenging.

Consider the representation of the above TTM system as a state space model of the form

$$\left\{\begin{array}{l}{x}_{k+1}=A{x}_k+B{u}_{k-2}+w_k\\ y_k=C{x}_k+v_k\end{array}\right.,$$

(1)

where $x_k\in {ℝ}^{n_x}$ is the unknown system state, $u_k\in {ℝ}^{n_u}$ is the control input, and $y_k\in {ℝ}^{n_y}$ is the system output; for simplicity, we assume $n_y\le n_u$. A, B, and C are unknown system matrices with proper dimensions. $w_k\in {ℝ}^{n_x}$ and $v_k\in {ℝ}^{n_y}$ are uncorrelated zero-mean Gaussian white noise with their semi-positive defined covariance matrices denoted as W₁ and V. In addition, this TTM system is considered to have a time delay of two frames.

The time delay of system (1) reflects the aftereffect of the control input; we could rewrite the time delay system in a delay-free form using state augmentation. By combining the delayed inputs with the original state, the augmented state can be obtained as ${\xi }_k={\left[\begin{array}{ccc}{x}_k^T& u_{k-2}^T& u_{k-1}^T\end{array}\right]}^T$, and the delay-free system is

$$\left\{\begin{array}{l}{\zeta }_{k+1}=\left[\begin{array}{ccc}A& B& 0\\ 0& 0& I_{n_u}\\ 0& 0& 0\end{array}\right]\left[\begin{array}{c}{x}_k\\ u_{k-2}\\ u_{k-1}\end{array}\right]+\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]u_k+\left[\begin{array}{c}{w}_k\\ 0\\ 0\end{array}\right]\triangleq A_1{\zeta }_k+B_1{u}_k+{\tilde{w}}_k\\ y_k=\left[\begin{array}{ccc}C& 0& 0\end{array}\right]{\zeta }_k+v_k\triangleq C_1{\zeta }_k+v_k\end{array}\right..$$

(2)

The beam jitter signal to be tracked by system (1) can be expressed in the following reference trajectory model:

$$\left\{\begin{array}{l}{g}_{k+1}=E{g}_k\\ r_k=F{g}_k\end{array}\right.,$$

(3)

where $g_k\in {ℝ}^{n_r}$ is the state of system (1). $E\in {ℝ}^{n_r\times n_r}$ and $F\in {ℝ}^{n_y\times n_r}$ are unknown constant matrices.

Remark 1.

System (3) is known as a command generator model [30], where E is not required to be Hurwitz. Such a model is able to generate many common trajectories, such as sinusoidal signals, square waveforms, the ramp, and so on.

Without loss of generality, we give the following assumptions: $rank(A)=n_x$, the pair $\left(A,B\right)$ is controllable, and $\left(A,C\right)$ and $\left(E,F\right)$ are observable. Considering the system model (1) and the reference trajectory model (3), the following augmented system can be constructed:

$$\left\{\begin{array}{l}{X}_{k+1}=\left[\begin{array}{c}{\zeta }_{k+1}\\ g_{k+1}\end{array}\right]=\left[\begin{array}{cc}{A}_1& 0\\ 0& E\end{array}\right]\left[\begin{array}{c}{\zeta }_k\\ g_k\end{array}\right]+\left[\begin{array}{c}{B}_1\\ 0\end{array}\right]u_k+\left[\begin{array}{c}{\tilde{w}}_k\\ 0\end{array}\right]\triangleq \overline{A}{X}_k+\overline{B}{u}_k+{\overline{w}}_k\\ e_k=\left[\begin{array}{cc}{C}_1& -F\end{array}\right]X_k+v_k\triangleq \overline{C}{X}_k+v_k\end{array}\right.,$$

(4)

where the augmented state is $X_k\in {ℝ}^{n_x+n_r}$, ${\overline{w}}_k$ is the augmented noise term with its covariance matrix $W=\left[\begin{array}{cc}{W}_1& 0\\ 0& 0\end{array}\right]$, and $e_k\in {ℝ}^{n_y}$ is the augmented system output, which represents the tracking error.

For jitter tracking and suppression in TTM systems, the stochastic optimal tracking problem aims to find an optimal feedback control sequence $u_k,k=0,1,\dots$ to minimize the following cost function:

$$J_k=V\left(X_k\right)=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}\left(X_i^T\overline{Q}{X}_i+u_i^{T}R{u}_i\right)}\right],$$

(5)

where $\gamma \in \left(0,1\right)$ is a discount factor, which is necessary here to make sure that $J_k$ is bounded even though disturbed by noises. $Q\ge 0$ and $R>0$ are symmetric weight matrices, respectively; $\overline{Q}={\overline{C}}^{T}Q\overline{C}$ and $\left({\overline{Q}}^{1/2},\overline{A}\right)$ are observable. The cost function (5) is also known as the value function.

Under given assumptions, it is clear that when noise terms $w_k$ and $v_k$ are not considered, the problem mentioned above degenerates into the infinite-horizon linear quadratic tracking problem and has an optimal solution.

$$u_k^{*}=-{\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)}^{-1}\gamma {\overline{B}}^{T}P\overline{A}{X}_k\triangleq -K{X}_k,$$

(6)

where $P$ is the unique positive defined solution to the following algebraic Riccati equation (ARE):

$$P=\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}-{\gamma }^2{\overline{A}}^{T}P\overline{B}{\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)}^{-1}{\overline{B}}^{T}P\overline{A}.$$

(7)

When noises are not neglectable, the certainty equivalence principle shows that the optimal controllers in stochastic and deterministic environments have exactly the same form. Thus, the controller (6) is equally capable of minimizing the cost function (5), subject to the constraints of system (4). Since the ARE (7) is nonlinear and is difficult to have a direct solution, the following value iteration (VI) algorithm [31] is proposed to obtain the optimal solution in an iterative way, as shown in Algorithm 1.

Algorithm 1: VI algorithm

1:  Initialization: Select an arbitrary control gain $K_0$, $j\leftarrow 0$, $\epsilon >0$. 2:  Repeat: 3:     Policy evaluation: Solve $P_{j+1}$ using $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{P}_{j+1}=\overline{Q}+K_j^{T}R{K}_j+\gamma {\left(\overline{A}-\overline{B}{K}_j\right)}^T{P}_j\left(\overline{A}-\overline{B}{K}_j\right).$ 4:     Policy improvement: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{K}_{j+1}={\left(R+\gamma {\overline{B}}^T{P}_{j+1}\overline{B}\right)}^{-1}\gamma {\overline{B}}^T{P}_{j+1}\overline{A}.$ 5:     $j\leftarrow j+1$ 6:  Until ${‖P_j-P_{j+1}‖}_2<\epsilon$

The VI algorithm has been proved to have the convergency properties ${\lim }_{j\to \infty }{P}_j=P^{*}$ and ${\lim }_{j\to \infty }{K}_j=K^{*}$. However, Algorithm 1 requires knowledge of system dynamics (4), and the full-state feedback controller form necessitates complete state information. Such information is typically difficult to obtain in practical systems, making analytical solutions challenging. Therefore, the following sections will introduce how to design an optimal controller using a fully data-driven approach by incorporating model-free RL methods and the concept of a state observer.

3. Principle of Noisy-OPFB Q-Learning Scheme

3.1. State Reconstruction Using Data History

Since there is no access to complete system state information, an information history vector is first defined, which consists of the input and output data history available at time k, as shown below.

$$I_k\triangleq \left[e_0,\dots ,e_k,u_0,\dots ,u_{k-1}\right], I_0=e_0,$$

(8)

and this information history can be used to describe the same evolution nature of system dynamics (4) by $I_{k+1}=\left(I_k,e_{k+1},u_k\right)$.

Based on this, the minimum-variance estimate of the system state is given as

$${\widehat{X}}_k:=E\left[X_k|I_k\right],$$

(9)

together with the estimation error ${\widehat{e}}_k$ and its covariance matrix $P_{{\widehat{e}}_k}$ defined by

$$\left\{\begin{array}{l}{\widehat{e}}_k:=X_k-{\widehat{X}}_k\\ P_{{\widehat{e}}_k}:=E\left[{\widehat{e}}_k{\widehat{e}}_k^T|I_k\right]\end{array}\right..$$

(10)

The separation principle points out that for a linear stochastic system (4), the design of optimal control policy $u_k^{*}=-K{\widehat{X}}_k$ can be divided into two parts: (a) the optimal control gain $K$ designed under the condition of complete state information for a deterministic system, and (b) the optimal state estimate ${\widehat{X}}_k$ for the stochastic system based on information history.

Here, consider the state observer for system (4) as in form of

$${\widehat{X}}_k={\overline{X}}_k+K_{ss}\left(e_k-\overline{C}{\overline{X}}_k\right)\triangleq {\overline{X}}_k+L_{ss}{\nu }_k,$$

(11)

and

$${\overline{X}}_k=\overline{A}{\widehat{X}}_{k-1}+\overline{B}{u}_{k-1},$$

(12)

where ${\overline{X}}_k$ denotes the a priori estimate and ${\nu }_k$ is the innovation information of the observer. $L_{ss}$ is the steady-state observer gain and $M_{ss}=E\left[(X_k-{\overline{X}}_k){(X_k-{\overline{X}}_k)}^T|I_k\right]$ is the a priori estimation error covariance matrix.

By substituting (12) into (11) the state estimate can be rearranged into an iterative form

$$\begin{array}{l}{\widehat{X}}_k={\overline{X}}_k+L_{ss}(e_k-\overline{C}{\overline{X}}_k)\\ \hspace{1em}\hspace{1em}=(\overline{A}-L_{ss}\overline{C}\overline{A}){\widehat{X}}_{k-1}+(\overline{B}-L_{ss}\overline{C}\overline{B})u_{k-1}+L_{ss}{e}_k\\ \hspace{1em}\hspace{1em}\triangleq A_K{\widehat{X}}_{k-1}+B_K{u}_{k-1}+L_{ss}{e}_k,\end{array}$$

(13)

and expanding it on a time horizon $\left[k-N,k\right]$, one has

$$\begin{array}{l}{\widehat{X}}_k=A_K^N{\widehat{X}}_{k-N}\\ \hspace{1em}\hspace{1em}+[B_K A_K{B}_K \cdots A_K^{N-1}{B}_K]\left[\begin{array}{c}{u}_{k-1}\\ u_{k-2}\\ ⋮\\ u_{k-N}\end{array}\right]+[L_{ss} A_K{L}_{ss} \cdots A_K^{N-1}{L}_{ss} 0]\left[\begin{array}{c}{e}_k\\ e_{k-1}\\ ⋮\\ e_{k-N}\end{array}\right].\end{array}$$

(14)

Note the state observer is asymptotically stable and converges to a steady-state value, which means $A_K$ has all its eigenvalues within the unit circle. So the first term on the right side of (14) becomes small enough to be neglected when given a sufficiently large number N, and that makes the state estimate a linear combination of input and output data history.

$$\begin{array}{l}{\widehat{X}}_k={\displaystyle \sum _{i=0}^{N-1}{A}_K^i{L}_{ss}{e}_{k-i}+}{\displaystyle \sum _{j=1}^N{A}_K^{i-1}{B}_K{u}_{k-j}}\\ =\left[\begin{array}{ccccccc}{L}_{ss}& \cdots & A_K^{N-1}{L}_{ss}& 0 & B_K & \cdots & A_K^{N-1}{B}_K\end{array}\right]\left[\begin{array}{c}{\overline{e}}_{k,k-N}\\ {\overline{u}}_{k-1,k-N}\end{array}\right]\\ \triangleq \Gamma S\left(I_k\right),\end{array}$$

(15)

where $\Gamma$ is a constant coefficient matrix and $S\left(I_k\right)$ is the sufficient statistic [32], which is a summary of all the essential information history in $I_k$ that is actually needed for feedback control, resulting in data reduction when the amount of data in information history is too large for practice.

Remark 2.

According to (15), the state estimate can be reconstructed directly from the available information history $I_k$. However, the solution to the steady-state observer gain $L_{ss}$ still requires the complete knowledge of the system model. In the following discussion, an output feedback Q learning scheme is developed to release this requirement.

3.2. Online Output Feedback Q-Learning Scheme

Have a look back at the value function (5); it could be rewritten in terms of nested conditional expectations when complete state information is unavailable.

$$V\left(X_k\right)=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}E\left[X_i^T\overline{Q}{X}_i+u_i^{T}R{u}_i|I_i\right]}\right].$$

(16)

Based on the relationship between the state and its estimate, substituting (10) into (16) yields

$$\begin{array}{l}V\left(X_k\right)=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}E\left[{\left({\widehat{X}}_i+{\widehat{e}}_i\right)}^T\overline{Q}\left({\widehat{X}}_i+{\widehat{e}}_i\right)+u_i^{T}R{u}_i|I_i\right]}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=E{\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}E\left[tr\left(\overline{Q}\left({\widehat{X}}_i+{\widehat{e}}_i\right){\left({\widehat{X}}_i+{\widehat{e}}_i\right)}^T+R{u}_i{u}_i^T\right)|I_i\right]}\right]}_{.}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}\left[tr\left(\overline{Q}E\left[{\widehat{e}}_i{\widehat{e}}_i^T|I_i\right]\right)+{\widehat{X}}_i^T\overline{Q}{\widehat{X}}_i+u_i^{T}R{u}_i\right]}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}\left({\widehat{X}}_i^T\overline{Q}{\widehat{X}}_i+u_i^{T}R{u}_i\right)}\right]+{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}tr\left(P_{\widehat{e}}\overline{Q}\right)},\end{array}$$

(17)

where $tr\left(·\right)$ denotes the trace operation. The second to third equation holds because of the orthogonal property of the estimation error that $E\left[{\widehat{e}}_k{\widehat{X}}_i^T\right]=0, i=1,\dots ,k-1$. Note $P_{\widehat{e}}\overline{Q}$ is independent of the control input $u_i$; therefore, the term ${\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}tr\left(P_{\widehat{e}}\overline{Q}\right)}$, which characterizes the effect of the estimation error, makes no contribution to the design of the optimal control policy and will be discarded to formulate an equivalent value function.

$$\overline{V}\left({\widehat{X}}_k\right)=E\left[{\displaystyle \sum _{i=k}^{\infty }{\gamma }^{i-k}\left({\widehat{X}}_i^T\overline{Q}{\widehat{X}}_i+u_i^{T}R{u}_i\right)}\right].$$

(18)

Note the conditioning is on ${\widehat{X}}_k$, which is a sufficient statistic of the information history $I_k$; the Bellman equation can then be given by writing (18) into the form of a difference equation

$$\begin{array}{l}\overline{V}\left({\widehat{X}}_k\right)={\widehat{X}}_k^T\overline{Q}{\widehat{X}}_k+u_k^{T}R{u}_k+E{\left[{\displaystyle \sum _{i=k+1}^{\infty }{\gamma }^{i-k}\left({\widehat{X}}_i^T\overline{Q}{\widehat{X}}_i+u_i^{T}R{u}_i\right)}\right]}_{.}\\ \hspace{1em}\hspace{1em}\hspace{1em} \triangleq U_k+\gamma E\left[\overline{V}\left({\widehat{X}}_{k+1}\right)|I_k\right].\end{array}$$

(19)

To relate the cost function to the control inputs, define the following Q function:

$$\begin{array}{l}{Q}^{\pi }\left({\widehat{X}}_k,u_k\right)=U_k+\gamma E\left[\overline{V}\left({\widehat{X}}_{k+1}\right)|I_k\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =U_k+\gamma E\left[Q^{\pi }\left({\widehat{X}}_{k+1},\pi \left({\widehat{X}}_{k+1}\right)\right)|I_k\right],\end{array}$$

(20)

which constitutes exactly the form of the Bellman equation.

Lemma 1.

The value function (19) has a quadratic structure of the state estimate ${\widehat{X}}_k$, attached by a penalty term introduced by an estimation error that $\overline{V}\left({\widehat{X}}_k\right)={\widehat{X}}_k^{T}P{\widehat{X}}_k+c$ and $c=\gamma tr\left(L_{ss}^{T}P{L}_{ss}\left(\overline{C}{M}_{ss}{\overline{C}}^T+V\right)\right)/\left(1-\gamma \right)$.

Proof of Lemma 1.

Suppose for $\forall k>0$, one has $\overline{V}\left({\widehat{X}}_{k+1}\right)={\widehat{X}}_{k+1}^{T}P{\widehat{X}}_{k+1}+c$.

Substituting (11) into (19), one has

$$\begin{array}{l}\overline{V}\left({\widehat{X}}_k\right)={\widehat{X}}_k^T\overline{Q}{\widehat{X}}_k+u_k^{T}R{u}_k+\gamma E\left[\overline{V}\left({\widehat{X}}_{k+1}\right)|I_k\right]\\ \hspace{1em}\hspace{1em}\hspace{1em} ={\widehat{X}}_k^T\overline{Q}{\widehat{X}}_k+u_k^{T}R{u}_k+\gamma E\left[{\widehat{X}}_{k+1}^{T}P{\widehat{X}}_{k+1}+c|I_k\right]\\ \hspace{1em}\hspace{1em}\hspace{1em} ={\widehat{X}}_k^T\left(\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}\right){\widehat{X}}_k+u_k^T\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)u_k+\gamma u_k^T{\overline{B}}^{T}P\overline{A}{\widehat{X}}_k\\ \hspace{1em}\hspace{1em}\hspace{1em} +\gamma {\widehat{X}}_k^T{\overline{A}}^{T}P\overline{B}{u}_k+\gamma \left(tr\left(K_{ss}^{T}P{K}_{ss}\left(\overline{C}{M}_{ss}{\overline{C}}^T+V\right)\right)+c\right)\\ \hspace{1em}\hspace{1em}\hspace{1em} ={\widehat{X}}_k^T\left(\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}\right){\widehat{X}}_k+u_k^T\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)u_k+\gamma u_k^T{\overline{B}}^{T}P\overline{A}{\widehat{X}}_k\\ \hspace{1em}\hspace{1em}\hspace{1em} +\gamma {\widehat{X}}_k^T{\overline{A}}^{T}P\overline{B}{u}_k+\gamma tr\left(L_{ss}^{T}P{L}_{ss}\left(\overline{C}{M}_{ss}{\overline{C}}^T+V\right)\right)/\left(1-\gamma \right).\end{array}$$

(21)

Given the discrete-time Hamiltonian function

$$H\left({\widehat{X}}_k,u_k\right)={\widehat{X}}_k^T\overline{Q}{\widehat{X}}_k+u_k^{T}R{u}_k+\gamma E\left[\overline{V}\left({\widehat{X}}_{k+1}\right)|I_k\right]-\overline{V}\left({\widehat{X}}_k\right).$$

(22)

The optimal controller can be obtained by satisfying the stationary condition $\partial H\left({\widehat{X}}_k,u_k\right)/u_k=0$ to have $u_k=-{\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)}^{-1}\gamma {\overline{B}}^{T}P\overline{A}{\widehat{X}}_k$, and substituting the $u_k$ back into (22) yields

$$\overline{V}\left({\widehat{X}}_k\right)={\widehat{X}}_k^{T}P{\widehat{X}}_k+c$$

(23)

Hence (22) holds for arbitrary $k>0$ and this completes the proof. □

According to Lemma 1 and by substituting (11) into (20), we yield a quadratic form of the Q function:

$$\begin{array}{l}{Q}^{\pi }\left({\widehat{X}}_k,u_k\right)=U_k+\gamma E\left[{\widehat{X}}_{k+1}^{T}P{\widehat{X}}_{k+1}+c|I_k\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\widehat{X}}_k^T\left(\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}\right){\widehat{X}}_k+u_k^T\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)u_k\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{1em} +\gamma u_k^T{\overline{B}}^{T}P\overline{A}{\widehat{X}}_k+\gamma {\widehat{X}}_k^T{\overline{A}}^{T}P\overline{B}{u}_k+c\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\left[\begin{array}{c}{\widehat{X}}_k\\ u_k\end{array}\right]}^T\left[\begin{array}{cc}\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}& \gamma {\overline{A}}^{T}P\overline{B}\\ \gamma {\overline{B}}^{T}P\overline{A}& R+\gamma {\overline{B}}^{T}P\overline{B}\end{array}\right]\left[\begin{array}{c}{\widehat{X}}_k\\ u_k\end{array}\right]+c,\end{array}$$

(24)

where the superscript $\pi$ indicates the certain control policy $u_k=\pi \left(X_k\right)$ that the control input follows. The effect of $c$ can be discarded considering that it is not related to the control input $u_k$ that is to be optimized.

Now, the output feedback Q function can be formulated by replacing ${\widehat{X}}_k$ in (24) with the sufficient statistic based on the result of (15) to have

$$\begin{array}{l}{Q}^{\pi }\left(S\left(I_k\right),u_k\right)={\left[\begin{array}{c}\Gamma S\left(I_k\right)\\ u_k\end{array}\right]}^T\left[\begin{array}{cc}\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}& \gamma {\overline{A}}^{T}P\overline{B}\\ \gamma {\overline{B}}^{T}P\overline{A}& R+\gamma {\overline{B}}^{T}P\overline{B}\end{array}\right]\left[\begin{array}{c}\Gamma S\left(I_k\right)\\ u_k\end{array}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\left[\begin{array}{c}S\left(I_k\right)\\ u_k\end{array}\right]}^T\left[\begin{array}{cc}{\Gamma }^T\left(\overline{Q}+\gamma {\overline{A}}^{T}P\overline{A}\right)\Gamma & \gamma {\Gamma }^T{\overline{A}}^{T}P\overline{B}\\ \gamma {\overline{B}}^{T}P\overline{A}\Gamma & R+\gamma {\overline{B}}^{T}P\overline{B}\end{array}\right]\left[\begin{array}{c}S\left(I_k\right)\\ u_k\end{array}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\triangleq Z_k^{T}H{Z}_k,\end{array}$$

(25)

where $H\triangleq \left[\begin{array}{cc}{H}_{xx}& H_{xu}\\ H_{ux}& H_{uu}\end{array}\right]\in {ℝ}^{n_H\times n_H}$ is a symmetric kernel matrix with its dimension equals $n_H=n_{y}N+n_{u}N+n_y+n_u$.

Hence, substituting (25) into (20) gives the Bellman equation in a quadratic form

$$Z_k^{T}H{Z}_k=U_k+\gamma E\left[Z_{k+1}^{T}H{Z}_{k+1}|I_k\right],$$

(26)

and the corresponding control input is then obtained by satisfying the stationary condition $\partial Q^{\pi }\left(S\left(I_k\right),u_k\right)/u_k=0$ to have

$$u_k=-\overline{K}S\left(I_k\right)=-H_{uu}^{-1}{H}_{ux}S\left(I_k\right)=-{\left(R+\gamma {\overline{B}}^{T}P\overline{B}\right)}^{-1}\gamma {\overline{B}}^{T}P\overline{A}\Gamma S\left(I_k\right).$$

(27)

Remark 3.

It should be emphasized that the Q function (23) maintains the quadratic form of the input and output data and is weighted by the kernel matrix H. The control input (27) has almost the same form as in (6) but features the advantage of using only available historical measurements, avoiding the need for direct measurements and calculations of the system matrices, system state, or observer gain.

According to (26) and (27), a noisy-OPFB Q-learning algorithm can be built as expressed in Algorithm 2.

Algorithm 2: Improved noisy-OPFB Q-learning algorithm

1:  Initialization: Set $j\leftarrow 0$, $\epsilon >0$. Select an arbitrary control policy ${\overline{K}}_0$. 2:  Repeat: 3:     Policy evaluation: Solve $H_{j+1}$ using           $Z_k^T{H}_{j+1}{Z}_k=U_k+\gamma E\left[Z_{k+1}^T{H}_j{Z}_{k+1}|I_k\right]$ 4:     Policy improvement: Obtain an improved control gain by           ${\overline{K}}_{j+1}=-{(H_{j+1})}_{uu}^{-1}{(H_{j+1})}_{ux}$ 5:     $j\leftarrow j+1$ 6:  Until ${‖H_j-H_{j+1}‖}_2<\epsilon$

Similar to the VI method, the ${\overline{K}}_j$ and $H_j$ obtained by solving Algorithm 2 also satisfy the convergence properties ${\lim }_{j\to \infty }{H}_j=H^{*}$ and ${\lim }_{j\to \infty }{\overline{K}}_j={\overline{K}}^{*}$.

Remark 4.

Although Algorithm 2 gives the idea of using the noisy-OPFB Q-learning scheme to solve the stochastic tracking control problem, the limitations of unknown system dynamics and incomplete state information make it difficult to calculate the mathematical expectation in the Bellman Equation (26).

3.3. Algorithm Implementation

Considering that it is difficult to compute the mathematical expectation when the information of system matrices, state, and noise is unknown, this subsection introduces the adaptive transversal predictor (ATP) that eliminates the conditional expectation computation by predicting the tracking error at time k + 1.

Notably, the expectation on the right side of Equation (26) is applied to all potential random variables, specifically to the system output $e_{k+1}$ that is not included in the information history $I_k$. To cope with this, a possible approach is to estimate ${\widehat{e}}_{k+1}$ with $I_k$. As shown in [33], based on the AutoRegresive Moving Average with eXogeneous input (ARMAX) model, a one-step linear prediction of the system output ${\widehat{e}}_{k+1}$ can be given as

$${\widehat{e}}_{k+1}={\displaystyle \sum _{i=1}^M\left({\alpha }_i{e}_{k+1-i}+{\beta }_i{u}_{k+1-i}\right)},$$

(28)

where M is the length of data history that is required for prediction. ${\alpha }_i$ and ${\beta }_i$ are the coefficients of the predictor. This predictor is thought to approximate the prediction performance of the optimal Kalman predictor when it converges.

Note that the information history used in (28) is fully contained in the data history $I_k$; define ${\widehat{Z}}_{k+1}$ by

$${\widehat{Z}}_{k+1}={\left[\begin{array}{cc}{S}^T\left(I_k,{\widehat{e}}_{k+1},u_k\right)& {\pi }^T\left(S\left(I_k,{\widehat{e}}_{k+1},u_k\right)\right)\end{array}\right]}^T,$$

(29)

where $u_{k+1}=\pi \left(S\left(I_k,{\widehat{e}}_{k+1},u_k\right)\right)$ is to emphasize that the control input uses only currently available information history, so it is deterministic under the given condition $I_k$. Substituting (29) into (26) gives the Bellman equation in a deterministic form

$$Z_k^{T}H{Z}_k=U_k+\gamma {\widehat{Z}}_{k+1}^{T}H{\widehat{Z}}_{k+1}.$$

(30)

Define the operators $\phi \left(X\right)={\left[x_1^2, \dots , x_1{x}_m, x_2^2, \dots , x_2{x}_m, \dots , x_m^2\right]}^T\in {ℝ}^{m(m+1)/2\times 1}$ for $\forall X\in {ℝ}^{m\times 1}$ and $vec\left(H\right)={\left[h_{11}, 2h_{12}, \dots , 2h_{1n}, h_{22}, 2h_{23}, \dots , h_{nn}\right]}^T\in {ℝ}^{n(n+1)/2\times 1}$ for any symmetric matrix $H\in {ℝ}^{n\times n}$, where $x_i$ and $h_{ij}$ are elements of $X$ and $H$, respectively. The Q function can be rewritten as a product of a set of quadratic basis functions and a set of coefficients that

$$Q^{\pi }\left(S\left(I_k\right),u_k\right)={\phi }^T\left(Z_k\right)vec\left(H\right).$$

(31)

Now the Bellman Equation (31) can be solved by rewriting it as the following least squares problem:

$${\Theta }_{j}vec\left(H_j\right)={\Xi }_j,$$

(32)

where ${\Theta }_j={\left[\phi \left(Z_j^1\right),\phi \left(Z_j^2\right),\dots ,\phi \left(Z_j^i\right)\right]}^T$ represents the regression vector and ${\Xi }_j={\left[U_j^1+ve{c}^T\left(H_{j-1}\right)\phi \left({\widehat{Z}}_{j+1}^1\right),\dots , U_j^i+ve{c}^T\left(H_{j-1}\right)\phi \left({\widehat{Z}}_{j+1}^i\right)\right]}^T$. The subscript $j$ indicates the number of iterations and superscript $i=1,2,\dots ,l$ represents the different samples in a batch. To guarantee the solvability of of Equation (32) and the uniqueness of the solution, the following full-rank condition must be satisfied:

$$rank\left(\Theta \right)=n_H(n_H+1)/2.$$

(33)

The full-rank condition mandates a sufficiently large batch size to ensure adequate representation while also necessitating the incorporation of exploration noise with adequate intensity into the control input. This exploration noise can be selected as multiple sinusoidal signals or as random Gaussian noise, linearly integrated into the system’s control input. The exploration noise is widely used in adaptive control methods to facilitate the fulfillment of persistence of excitation (PE) conditions [34] and benefits from disrupting the correlations among elements of $\Theta$ and enhancing the diversity of the sampled data. Consequently, the full-rank condition is essential for ensuring convergence.

Furthermore, to enhance data sampling efficiency, this paper employs the experience replay mechanism [35], a standard component in the majority of RL algorithms. The input–output data generated during the interaction with real systems will form a tuple according to $\left\{Z_k, {\overline{U}}_k, {\widehat{Z}}_{k+1}\right\}$ and then get sequentially stored in a replay buffer $\mathcal{D}$. For each iteration step, data are randomly sampled from $\mathcal{D}$. This method also diminishes the strong temporal correlations within the sample data, enabling data reuse. It is important to note that the data utilized for policy updates are derived from the initialized admissible controller. This characteristic indicates that the proposed scheme is off-policy [36], distinguishing them from the conventional on-policy algorithms.

Based on the above discussion, the implementation of the noisy-OPFB Q-learning algorithm can be detailed in Algorithm 3.

Algorithm 3: Implementation of the noisy-OPFB Q-learning algorithm

1:  Initialization: Set iteration step $j=0$, initialize replay buffer $\mathcal{D}$. Select proper $M>0$,$N>0$,$\epsilon >0$. Select an arbitrary control policy ${\pi }_0=-K_{0}S\left(I_k\right)+{\mu }_k$, where ${\mu }_k$ is the exploration noise. 2:  Data sampling and pretraining: Generate samples $\left\{Z_k, U_k, {\widehat{Z}}_{k+1}\right\}$ by running policy ${\pi }_j$ and append them to $\mathcal{D}$. ATP is trained here until convergency. 3:  Repeat: 4:     Policy evaluation: Randomly sample a mini batch of data from $\mathcal{D}$ and solve $H_j$ using (31). 5:     Policy improvement: Obtain an improved controller by              ${\overline{K}}_j=-{(H_j)}_{uu}^{-1}{(H_j)}_{ux}.$ 6:     $j\leftarrow j+1$ 7:  Until ${‖H_j-H_{j+1}‖}_2<\epsilon$

Remark 5.

It is important to note that the entire learning process of the ATP is conducted online during the data sampling phase, precluding the need for any offline computation. This approach optimizes the utilization of time during data sampling, thereby significantly enhancing algorithmic efficiency. Once the ATP converges, the parameters become fixed, with no further learning occurring unless there is a change in the environment. Consequently, this does not interfere with the learning process of the controller.

Remark 6.

Compared with existing works, this paper has the following advantages: The noise is assumed to be available in [37], and complete state information is considered to be measurable in [28]; these assumptions are removed in this paper. In contrast to the method in [29], we do not need an extra learning phase for optimal state estimation. In addition, this work does not require the generation of multiple trajectories as in [38] to approximate mathematical expectations.

Remark 7.

To reduce the computational complexity, the data length N of $S\left(I_k\right)$must be limited, inevitably introducing estimation errors during the learning process. Consequently, the result obtained from Algorithm 3 does not precisely equate to the analytical solution of the Bellman Equation (26) and should be considered a near-optimal solution, which is influenced by the accuracy of estimation and prediction. However, we will demonstrate in the forthcoming numerical experiments that the near-optimal solution does not deviate significantly from the analytical solution, and the tracking control performance is satisfactory.

4. Results and Discussion

In this section, we verify the effectiveness of the proposed noisy-OPFB Q-learning algorithm through tracking control simulation experiments of a TTM system. The results confirm that our proposed algorithm can flexibly and effectively solve the stochastic tracking control problem in the presence of unknown noise and unknown system states.

4.1. Application Example of Tracking Control for a Tip–Tilt Mirror System

Figure 1 and its system dynamic model (1) are given here. The TTM, as an important controlled device, can be equated to a second-order resonance model from its structural schematic in Figure 2, which shows that its inherent elastic structure will generate mechanical resonance.

We therefore model the TTM system as follows:

$$\left\{\begin{array}{l}A=\left[\begin{array}{cc}0& 1\\ -{\omega }_n^2& -2\xi {\omega }_n\end{array}\right]\\ B={\left[\begin{array}{cc}0& {\omega }_n^2\end{array}\right]}^T,C=\left[\begin{array}{cc}1& 0\end{array}\right]\end{array}\right.,$$

(34)

where ${\omega }_n$ and $\xi$ represent the natural frequency and damping factor of the TTM, respectively.

For the external reference signal, a sinusoidal signal with an amplitude of 5 urad is generated using model (3), and its system matrix can be expressed as

$$\left\{\begin{array}{l}E=\left[\begin{array}{cc}\cos (w_r{T}_s)& \sin (w_r{T}_s)/w_r\\ -w_r\sin (w_r{T}_s)& \cos (w_r{T}_s)\end{array}\right]\\ F=\left[\begin{array}{cc}5& 0\end{array}\right]\end{array}\right.,$$

(35)

where $w_r=2\pi f$. $f$ is the center frequency of the sinusoidal signal and $T_s$ is the system’s sampling rate. The entire control block diagram of applying the proposed noisy-OPFB Q-learning algorithm to the TTM tracking control system is shown in Figure 3. It should be emphasized that all the data used throughout the operation and optimization of the TTM tracking control system include only the control input $u_k$ and the noisy tracking error $e_k$ and do not depend on any of the system state information. This highlights the advantages of our proposed method over other Q-learning based methods.

For simulation purposes, we consider only one axis of the TTM, and the sampling rate was set to 1 KHz. The parameters of the TTM are the same as those identified in [39]: ${\omega }_n=532.9165 rad/s$ and $\xi =0.0645$. The cost function remains in the form of (18), where we let $\gamma =0.995$, $Q=100I$, and $R=0.01I$. The lengths of data history used for state estimation and output prediction were selected by $N=M=15$ and then $H\in {ℝ}^{32\times 32}$. Thus, there are a total of 528 unknown parameters to be learned in the kernel matrix H and 31 unknown parameters to be learned in the controller. The rank condition is satisfied by adding an exploration noise with zero mean and unit variance $I$; the batch size is chosen as l = 600. In the subsequent simulations, we set the size of the experience replay buffer to 1000, which means that the system requires a 1 s sampling period to obtain a sufficient amount of training data, and these 1000 sets of data constitute the entire dataset needed for training. The initial controller gain was selected as $K_0=0$.

4.2. Convergency Analysis

The convergence performance of the proposed algorithm is first verified. The process noise and measurement noise are assumed to be Gaussian white noise with distribution $w_k\sim \mathcal{N}\left(0, 0.01I\right)$ and $v_k\sim \mathcal{N}\left(0, 0.01I\right)$. Considering a 10 Hz reference jitter signal and running the TTM control system shown in Figure 3 for 3 s, the tracking control trajectory is shown in Figure 4. It is observed that the controller converges after 1.75 s of learning, and the convergence curves of the controller parameters are displayed in Figure 5. The results show that the controller converges after only seven iterations, which is due to the fact that the proposed noisy-OPFB Q-learning algorithm is updated through efficient numerical solutions rather than stochastic gradient methods, ensuring a rapid and stable convergence process for the controller parameters. To further demonstrate that the trained controller is near optimal, we executed the algorithm 20 times under stochastic noise conditions and compared all solutions with the optimal analytical solution, as shown in Figure 6. The root-mean-square error (RMSE) between the near optimal solutions and the optimal analytical solution is only 0.027. This indicates that despite the introduction of approximation errors and random noise interference during the computation process, the proposed algorithm can still converge well near the optimal solution.

4.3. Tracking Control Performance Analysis

Considering the proposed algorithm directly optimizes the controller using noisy measurements, it is essential to analyze the impact of different camera detection noise levels on the jitter tracking performance. Figure 7, Figure 8 and Figure 9 illustrate the tracking effects on a 10 Hz reference jitter signal under measurement noise standard deviation (STD) ${\sigma }_v$ of 0, 0.5, and 1. The results show that almost complete suppression of jitter is achieved in the absence of noise, while the jitter suppression effect deteriorates with increasing measurement noise STD. Figure 10 presents how the jitter tracking RMSEvaries with the noise STD. It can be observed that there is a roughly linear relationship between the noise STD and tracking RMSE, indicating that the noise level is directly related to tracking performance, and the estimation error introduced by noise is the primary factor affecting Algorithm 3. Nonetheless, fluctuations in the intensity of measurement noise over a wide range do not have a major impact on the tracking effectiveness and stability of the controller. This suggests that our proposed algorithm can accurately deduce the system’s dynamic characteristics from noisy measurements, thereby correctly calculating a near-optimal tracking controller. It eliminates the need for assumptions about measurable state information and the design of state observers, and thus is more generalizable and practical.

Furthermore, we verified the tracking control effectiveness of the proposed algorithm for beam jitter signals of different frequencies with measurement noise STD ${\sigma }_v=0.1$. The jitter tracking RMSE of the proposed algorithm was compared with that of a traditional integral controller, with the results presented in

Table 1. Beam jitter tracking RMSE (urad) of different control methods.

Controller	5 Hz	15 Hz	30 Hz	100 Hz	150 Hz
Reference jitter	3.537	3.537	3.537	3.537	3.537
Integrator	0.747	2.344	5.567	2.429	3.798
Noisy-OPFB Q-learning	0.058	0.069	0.067	0.072	0.069

. From the results, it can be seen that with the increase in the jitter frequency, the integrator, limited by its control bandwidth, struggles to maintain tracking performance. Meanwhile, the phase lag effect causes an amplification of the jitter signal at 30 Hz, leading to a decrease in system stability. In contrast, the proposed algorithm achieved precise and stable tracking for both low and high frequencies; the RMSE is reduced up to 98.7%. Figure 11 shows a comparison example of the tracking error of a 100 Hz reference jitter. From the comparison results of power spectral density (PSD), the tracking error of the proposed method at 100 Hz is only −30 dB, which is 40.5 dB lower compared to the error of integral control. Therefore, in contrast to the traditional integrator, the noisy-OPFB Q-learning is independent of control bandwidth and can autonomously tailor the controller to various reference jitter signals, thus achieving broadband jitter tracking. This comparison underscores the effectiveness and adaptability of the proposed algorithm when applied to tilting mirror systems, making it a potent solution for beam jitter control issues.

Finally, the algorithm’s adaptability to time-varying reference jitter signals and system models was analyzed, and the corresponding tracking trajectory is shown in Figure 12. To simulate environmental and reference signal changes, the frequency of the external jitter signal was changed from the initial 6 Hz to 13 Hz at 3 s. At the same time, the natural resonance frequency and the damping coefficient of the TTM model are changed to ${\omega }_n=333.9293$ and $\xi =0.06799$, respectively. The trajectory in Figure 12 indicates that after a period of adaptation and learning, the controller can once again accurately track the jitter signal. The learning curve of the controller parameters in Figure 13 demonstrates that Algorithm 3 can continue to learn on the basis of an existing controller without the need for an additional parameter tuning process. It should be noted that the whole trajectory is closed-loop continuous, and the proposed algorithm does not cause any degradation of system stability during the updating process. This illustrates that the proposed algorithm possesses online learning and adaptive capabilities and could effectively address the tracking control issues of dynamic beam jitter in the TTM system caused by environmental changes.

5. Conclusions

This paper develops a noisy-OPFB Q-learning scheme to address the optimal tracking control problem for TTM systems under conditions of additive unknown noise interference, unmeasurable system states, and unknown system dynamics. This method adjusts the Bellman equation through the estimation theory and employs an adaptive transversal predictor and experience replay technique, which is able to obtain a fast and stable near-optimal solution in spite of interference from stochastic noise. The numerical iterative computation process based on Q-learning releases the reliance on neural network design, thus facilitating easier deployment in practical systems. We have applied the algorithm to an application example of a TTM tracking control simulation system, demonstrating the effectiveness of the proposed method. Compared to traditional integral control, our reinforcement learning-based approach exhibits better control stability, dynamic adaptability, and tracking performance. This offers the potential for realizing intelligent control of optical systems. Our future work will focus on deploying the scheme in actual systems and finding better methods to reduce the computational costs of complex systems.