Data-Driven Active Learning Control for Bridge Cranes

Abstract

For positioning and anti-swing control of bridge cranes, the active learning control method can reduce the dependence of controller design on the model and the influence of unmodeled dynamics on the controller’s performance. By only using the real-time online input and output data of the bridge crane system, the active learning control method consists of the finite-dimensional approximation of the Koopman operator and the design of an active learning controller based on the linear quadratic optimal tracking control. The effectiveness of the control strategy for positioning and anti-swing of bridge cranes is verified through numerical simulations.

1. Introduction

Bridge cranes are widely used transportation tools mainly employed in loading and transporting goods in the current large-scale production industry. In order to solve the positioning and anti-swing control problems of bridge crane systems, domestic and foreign scholars have carried out in-depth research. In [1], the authors proposed a novel time-varying sliding mode control of variable parameters, Tysse et al. [2] developed a Lyapunov-based damping controller with nonlinear MPC control, Roman et al. [3] developed a hybrid data-driven fuzzy active disturbance rejection control, and Rigatos et al. [4] developed a robust control. Although the above methods can realize the positioning and anti-swing control of the bridge crane system, they are all model-based control methods. The actual bridge crane system is a very complex nonlinear system, which may have possible sources of nonlinearity that are hard to model (i.e., friction, backlash, flexible ropes, dead zones, etc.).

Therefore, data-driven controllers which do not depend on the model of the system itself have been studied in recent years. At present, the data-driven control method has been developed and improved continuously, and has been recognized symbolically both at home and abroad [5]. In [6], the authors developed a data-driven optimal PID type iterative learning control (ILC), Chi et al. [7] developed an indirect adaptive iterative learning control, Estakhrouiyeh et al. [8] proposed iterative feedback tuning algorithm, and Yuan and Tang [9] proposed a novel time–space network flow formulation and approximate dynamic programming approach. With recent advances in optimization techniques and computing power, machine learning technology is now widely used to build data-driven models of bridge cranes. However, deep neural networks (DNN) commonly lack interpretability, which has recently been noted as challenging for applications with safety requirements and remains a cutting-edge research topic. Furthermore, due to the nonlinear activation functions, the obtained dynamic model is not easy to use for designing a linear optimal controller such as a model predictive control and linear quadratic regulator. In recent years, the Koopman operator has become regarded as a powerful tool for capturing the intrinsic characteristics of nonlinear system via linear evolution in the lifted observable space. The Koopman operator governs the evolution of scalar observables defined on the state space of a nonlinear system, which requires a dictionary of scalar observables. There are many possible ways to choose this dictionary; it can be comprised of polynomial functions, Fourier modes, radial basis functions, or other sets of functions of the full-state observable. Therefore, the Koopman operator framework has received extensive attention due to its global linearization capability for nonlinear dynamics identification, and plays an important role in data-driven systems. The resulting linear representation allows for control of the nonlinear system using tools from linear optimal control [10,11,12], which are often easier and faster to implement than nonlinear methods. Beyond the computational speed and the reduction in feedback complexity, linear representation-based control can lead to better performance compared to controllers based on the original nonlinear system [13]. In [14], dynamic mode decomposition (DMD), a data-driven approach to obtain a finite-dimensional approximation of the Koopman operator, was proposed. This method uses time-shifted snapshots (measurements) of the system states to approximate the Koopman operator in a least-squares fashion. This method can be limiting, however, and sometimes fails to capture all the nonlinearities of the system. In [15], the authors proposed extended DMD (EDMD), in which snapshots of nonlinear measurement functions (observables) can be augmented with the system states to obtain a “lifted" finite-dimensional approximation of the Koopman operator. Recently, Ref. [16] extended EDMD for controlled dynamical systems. As a result, the Koopman operator is a promising framework for data-driven system identification. In [17], a data-driven control approach based on the Koopman operator was proposed for bridge cranes. However, the controller design requires a large amount of offline input and output data of the controlled system in order to train data-driven models.

Active learning in robotics has recently become a topic of interest [18,19,20]. Much work has been carried out in active learning for parameter identification [21,22] as well as in active learning for state-control mappings in reinforcement learning [19,23,24] and adaptive control [25]. In particular, the mentioned works refer to exciting the system dynamics using information theoretic measures [21,22,26], reward functions in reinforcement learning, and other methods [27,28] in order to obtain the best set of measurements that resolve a parameter or the best case mapping (either of the state control map or of the dynamics). In this paper, we use active learning to enable the system to learn the Koopman operator representations of a system’s own dynamic process.

In this paper, we propose a data-driven active learning control for bridge cranes. In the proposed algorithm, the linear structure of the Koopman operator is used to enhance Linear Quadratic Optimal Tracking (LQT) control. Then, the mode insertion gradient is derived to improve the accuracy of the LQT controller. Furthermore, a data-driven active learning control is designed by minimizing the mode insertion gradient.

The main contribution of this paper is the construction of a data-driven active learning control scheme for bridge cranes. Unlike other data-driven control methods, the proposed method does not require the data-driven model to be trained in advance, and only requires the real-time input and output data of the bridge crane system to accurately build the Koopman model by learning the Koopman operator. Furthermore, the Koopman model is a linear model in the lifted space with a nonlinear mapping from the original state space, thereby enhancing the linear quadratic optimal tracking (LQT) control. Therefore, the influence of the unmodeled dynamics and system model parameter uncertainties can be avoided while ensuring that the proposed method is feasible and robust.

The rest of this paper is organized as follows. Section 2 formulates the dynamics of the bridge crane and introduces the Koopman operator theory along with its finite-dimensional approximation. In Section 3, the data-driven active learning control approach is presented. Section 4 exhibits the simulation results. Finally, Section 5 summarizes the conclusions of this work.

2. Problem Formulation

In this section, we introduce the dynamics of bridge cranes along with the theory behind the Koopman operator and its finite-dimensional approximation for nonlinear systems with control using the EDMD algorithm.

2.1. Bridge Crane Dynamics

In this paper, we adopt a type of bridge crane consisting of a rope, load, and trolley. Its corresponding 2D simplified physical model is shown in Figure 1. The dynamics equation is as follows:

$$\begin{array}{cc}\hfill & (M+m)\ddot{y}+ml\ddot{\theta }cos\theta -ml{\dot{\theta }}^{2}sin\theta +f_{\mu }\left(\dot{y}\right)=u\hfill \\ \hfill & m{l}^2\ddot{\theta }+mlcos\theta \ddot{y}+mglsin\theta =0,\hfill \end{array}$$

(1)

where M and m represent the respective masses of the trolley and load, $\theta$ represents the vertical direction angle of the load, g represents the gravitational acceleration, l represents the length of a hoisting rope, which is fixed during transportation, $f_{\mu }\left(\dot{y}\right)=\mu \dot{y}$ represents the friction between the trolley and the platform (where $\mu$ represents the coefficient of friction), u is the control force (which can be linear control or nonlinear control, such as saturated control or dead zone control), and y represents the horizontal displacement. According to the method proposed in this paper, it is not necessary to know the dynamic characteristics of the bridge crane. The dynamics in Equation (1) are only used for producing closed-loop experimental data.

Because the dynamics of the bridge crane are unknown, we denote $x={[y,\dot{y},\theta ,\dot{\theta }]}^{\top }$ as the system state. Then, the dynamics Equation (1) can be described as follows:

$$\begin{array}{c}\hfill \dot{x}=f(x,u),\end{array}$$

(2)

where $\dot{x}$ is the rate of change of the state x and the mapping f is unknown.

2.2. Koopman Operator Theory

The Koopman operator is commonly used to capture the intrinsic characteristics via a linear dynamical evolution for unforced nonlinear dynamics [14,15]. With a slight change, the Koopman operator can be used to represent systems with control inputs as well. According to [29], the generalization of the Koopman operator for model (2) relies on an extended state variable $\overline{x}={[x^{\top },u^{\top }]}^{\top }$. In line with [29], the Koopman operator on (1) with extended state $\overline{x}$ is described as

$$\begin{array}{c}\hfill \psi \left(\overline{x}\left(t_{k+1}\right)\right)=\mathcal{K}\psi \left(\overline{x}\left(t_k\right)\right),\end{array}$$

(3)

where $\mathcal{K}$ denotes the Koopman operator, which is infinitely dimensional, $\psi \left(\overline{x}\right)\in \mathbb{R}$ is the observable in the lifted space, and $\overline{x}\left(t_k\right)$ and $u\left(t_k\right)$ are created to respectively store $\overline{x}\left(t\right)$ and $u\left(t\right)$ at different time instants $t=t_k$.

For a continuous and sufficiently smooth dynamics, it is possible to find the continuous-time infinitesimal generator of the Koopman operator, denoted as $\mathcal{G}$. Similar to (3), $\mathcal{G}$ induces a linear dynamical system in continuous time:

$$\begin{array}{c}\hfill \dot{\psi }\left(\overline{x}\left(t\right)\right)=\mathcal{G}\psi \left(\overline{x}\left(t\right)\right).\end{array}$$

(4)

Please refer to [29] for more details on the definition of the Koopman operator.

A finite-dimensional approximation of $\mathcal{K}$ is of interest for controller design. As the state is of infinite dimension, we adopt $\Psi \left(\overline{x}\left(t_k\right)\right)={[\Psi {\left(x\left(t_k\right)\right)}^{\top },u{\left(t_k\right)}^{\top }]}^{\top }={[{\psi }_1\left(x\left(t_k\right)\right),{\psi }_2\left(x\left(t_k\right)\right),\dots ,{\psi }_L\left(x\left(t_k\right)\right),u{\left(t_k\right)}^{\top }]}^{\top }$ as a group of observables for the practical calculation, where L is the number of observable functions on x, i.e., ${\psi }_i\left(x\right)$. The main idea behind computing the Koopman approximation for a continuous-time system with EDMD consists of three steps. First, select the observable functions as basis functions, e.g., polynomial functions or radial basis functions (RBF) [30]. Second, compute a finite-dimensional approximation of the Koopman operator for discrete time system by the least-squares method. To this end, compute the finite-dimensional approximation of the continuous-time infinitesimal generator $\mathcal{G}$.

To obtain an approximate Koopman operator $\tilde{\mathcal{K}}$, we adopt the least-squares method shown in [14]. This minimization takes the following form:

$$\begin{array}{c}\hfill {\tilde{\mathcal{K}}}^{*}=\underset{\tilde{\mathcal{K}}}{argmin}\sum _{k=0}^{M-1}\frac{1}{2}{\parallel \Psi \left(\overline{x}\left(t_{k+1}\right)\right)-\tilde{\mathcal{K}}\Psi \left(\overline{x}\left(t_k\right)\right)\parallel }^2,\end{array}$$

(5)

where $\Psi \left(\overline{x}\left(t_k\right)\right)={[\Psi {\left(x\left(t_k\right)\right)}^{\top },u{\left(t_k\right)}^{\top }]}^{\top }$, M is the number of measurements, and each measurement is a set consisting of an initial state $x\left(t_k\right)$, final state $x\left(t_{k+1}\right)$, and the actuation applied at the same instants $u\left(t_k\right)$ and $u\left(t_{k+1}\right)$, respectively.

The above expression has a closed-form solution provided by

$$\begin{array}{c}\hfill {\tilde{\mathcal{K}}}^{*}=V{W}^{†},\end{array}$$

(6)

where † denotes the Moore–Penrose pseudoinverse and

$$\begin{array}{}\mathrm{(7a)}& \hfill V& =\frac{1}{M}\sum _{k=0}^{M-1}\Psi \left(\overline{x}\left(t_{k+1}\right)\right)\Psi {\left(\overline{x}\left(t_k\right)\right)}^{\top },\hfill \mathrm{(7b)}& \hfill W& =\frac{1}{M}\sum _{k=0}^{M-1}\Psi \left(\overline{x}\left(t_k\right)\right)\Psi {\left(\overline{x}\left(t_k\right)\right)}^{\top }.\hfill \end{array}$$

Lastly, it is possible to switch between the continuous-time and discrete-time operators via $\tilde{\mathcal{G}}=ln\left(\tilde{\mathcal{K}}\right)/t_s$ [31], where $t_s$ is the time between measurements $x\left(t_k\right)$ and $x\left(t_{k+1}\right)$. Thus, the differential equation for the observables $\Psi$ is

$$\begin{array}{c}\hfill \dot{\Psi }\left(\overline{x}\right)=\tilde{\mathcal{G}}\Psi \left(\overline{x}\right),\end{array}$$

(8)

where $\dot{\Psi }$ is the rate of the observable function.

Due to $\Psi \left(\overline{x}\left(t\right)\right)={[\Psi {\left(x\left(t\right)\right)}^{\top },u{\left(t\right)}^{\top }]}^{\top }$, this notation allows us to rewrite (8) as

$$\begin{array}{c}\hfill \left[\begin{array}{c}\dot{\Psi }\left(x\left(t\right)\right)\\ \dot{u}\left(t\right)\end{array}\right]=\left[\begin{array}{cc}A& B\\ ·& ·\end{array}\right]\left[\begin{array}{c}\Psi \left(x\right(t\left)\right)\\ u\left(t\right)\end{array}\right]\end{array}$$

(9)

where A and B are sub-matrices of $\tilde{\mathcal{G}}$ that describe the dynamics of the observables $\Psi \left(x\right(t\left)\right)$ that depend only on the states and change only when $\tilde{\mathcal{G}}$ is updated. Note that the term $(·)$ in (9) refers to terms that evolve the observations on control u; these are ignored here, as there is no ambiguity in their evolution because they are determined by the controller. Thus, the Koopman model can be written as

$$\begin{array}{}\mathrm{(10a)}& & \dot{z}\left(t\right)=f_z(z,u)=Az\left(t\right)+Bu\left(t\right),\mathrm{(10b)}& & x\left(t\right)=Cz\left(t\right),\mathrm{(10c)}& & z\left(0\right)=\Psi \left(x\right(0\left)\right).\end{array}$$

Because the state x is often selected as part of the observables $\Psi$, we adopt $C=\left[\begin{array}{c}{I}_{n\times n},{\mathbf{0}}_{n\times L-n}\end{array}\right]$ as the matrix to project z to the original state x.

This model trades the nonlinearity of an n-dimensional ODE (1) for a nonlinear “lifting” (10c) of the initial condition $x\left(t_0\right)$ to higher-dimensional $(L\gg n)$ Koopman-invariant coordinates (10a) such that the original state can be linearly reconstructed via (10b). To achieve active learning, it is necessary to update the Koopman operator using the real-time input and output data of system. Therefore, we write a recursive least-squares update [25] which adaptively updates $\tilde{\mathcal{G}}$ as more data are acquired; these matrices then vary in response to how the incoming state measurements change the solution to (5).

3. Control

This section presents the controller framework, as shown in Figure 2. The proposed controller can achieve positioning and anti-swing control of the bridge crane, and the design of controller does not depend on the model of the system itself. Unlike other data-driven control methods, the proposed controller does not require the data-driven model to be trained in advance, and only requires real-time input and output data of the bridge crane system to accurately build the Koopman model by learning the Koopman operator.

3.1. Objective Function Design

The active learning controller allows the bridge crane to guide itself towards the important regions of the state space while tracking the trajectory, thereby improving data collection and the quality of the learned Koopman model. Therefore, we consider a general objective function of the form

$$\begin{array}{c}\hfill J_1={\int }_0^{\top }{\mathcal{l}}_{\mathrm{learn}}(z\left(t\right),u\left(t\right))+{\mathcal{l}}_{\mathrm{task}}(z\left(t\right),u\left(t\right))dt+{\mathcal{l}}_{\mathrm{final}}\left(z\left(T\right)\right),\end{array}$$

(11)

where $z\left(t\right)$ is the value of the function observables at time t subject to the Koopman model in (10), starting from initial condition $z\left(0\right)=\Psi \left(x\right(0\left)\right)$ and with $u\left(t\right)$ as the input at time t, T the prediction horizons, ${\mathcal{l}}_{\mathrm{task}}$ designed to accurately evaluate the proximity to the desired trajectory, ${\mathcal{l}}_{\mathrm{learn}}$ designed to evaluate the accuracy of the learned Koopman model, and ${\mathcal{l}}_{\mathrm{final}}$ the terminal cost.

In this work, we use the Fisher information [32] to generate an information measure in ${\mathcal{l}}_{\mathrm{learn}}$. The Fisher information is a way of measuring how much information a random variable has with respect to a set of parameters. We can compute the Fisher information matrix F over the parameters that compose the approximate continuous-time operator $\tilde{\mathcal{G}}$, with the elements in row a and column b of the Fisher information matrix being

$$\begin{array}{c}\hfill F_{a,b}={\frac{\partial f_z}{\partial {\tilde{\mathcal{G}}}_{a,b}}}^{\top }{\Sigma }^{-1}\frac{\partial f_z}{\partial {\tilde{\mathcal{G}}}_{a,b}},\end{array}$$

(12)

where ${\tilde{\mathcal{G}}}_{a,b}$ represents the elements in row a and column b of the approximate continuous-time operator $\tilde{\mathcal{G}}$ and $\Sigma$ is the noise covariance matrix. Because the Fisher information F defined here is positive semi-definite, we use the trace of the Fisher information matrix F as the T-optimality measure $\mathfrak{J}\left(\tilde{\mathcal{G}}\right)$ [33]. Thus, the T-optimality measure is defined as

$$\begin{array}{c}\hfill \mathfrak{J}\left(\tilde{\mathcal{G}}\right)=\mathrm{tr}\left(F\right),\end{array}$$

(13)

where $\mathrm{tr}(·)$ represents the trace of the matrix.

Therefore, the learning cost ${\mathcal{l}}_{\mathrm{learn}}$ is defined as

$$\begin{array}{c}\hfill {\mathcal{l}}_{\mathrm{learn}}(z\left(t\right),u\left(t\right))=\frac{\sigma }{\mathfrak{J}\left(\tilde{\mathcal{G}}\right)+ϵ},\end{array}$$

(14)

where $ϵ\ll 1$ is a small number to prevent singular solutions due to the positive semi-definite Fisher information matrix, $\sigma$ is the information weight, and $\mathfrak{J}$ is computed using the evaluation of $\tilde{\mathcal{G}}$ at the current time.

Remark 1.

Typically, the information weight σ is time-varying and should be set to gradually decrease during the learning process [20]. For instance, we can set $\sigma \left(t\right)={\sigma }_0{(1-0.01)}^t$, with ${\sigma }_0$ being an initial information weight.

At the same time, in order to match the desired trajectory, the running cost ${\mathcal{l}}_{\mathrm{task}}$ is defined as

$$\begin{array}{c}\hfill {\mathcal{l}}_{\mathrm{task}}(z\left(t\right),u\left(t\right))={∥z\left(t\right)-z_{\mathrm{ref}}\left(t\right)∥}_{\overline{Q}}+{∥u\left(t\right)∥}_{\overline{R}},\end{array}$$

(15)

where $\overline{Q}$ represents the lifted state cost matrix provided by $[Q0;00]$, with Q penalizing the system states and $\overline{R}$ the control cost matrix, $z_{\mathrm{ref}}=\Psi \left(x_{\mathrm{ref}}\right)$, and the desired trajectory of the Koopman model (10) obtained by lifting the desired trajectory of the original system (1) using observables $\Psi$.

The terminal cost is defined as

$$\begin{array}{c}\hfill {\mathcal{l}}_{\mathrm{final}}\left(z\left(T\right)\right)=z\left(T\right)Q_{f}z\left(T\right),\end{array}$$

(16)

where $Q_f$ represents terminal cost matrix and $z\left(T\right)$ is the value of the function observables at time T subject to the Koopman model in (10) starting from initial condition $z\left(0\right)$.

3.2. Linear Quadratic Optimal Tracking

Next, we design a linear quadratic regulator (LQR) in the lifted states in order to track the reference trajectories using a linear predictor.

Consider the Koopman model (10). To track reference trajectories $x_{\mathrm{ref}}$, we design an LQT controller that minimizes a running cost (15) and terminal cost (16) in a receding-horizon fashion. At the current time, we lift the original state $x\left(0\right)$ to find $z\left(0\right)$ using the observables $\Psi (·)$; the optimal control $u_{\mathrm{lqt}}\left(0\right)$ can be obtained by solving the following optimization problem:

$$\begin{array}{c}\hfill \underset{u_{\mathrm{lqt}}}{min}{\int }_0^{\top }{\mathcal{l}}_{\mathrm{task}}(z\left(t\right),u_{\mathrm{lqt}}\left(t\right))dt+{\mathcal{l}}_{\mathrm{final}}\left(z\left(T\right)\right)\end{array}$$

(17a)

subject to

$$\begin{array}{}\mathrm{(17b)}& & \dot{z}\left(t\right)=Az\left(t\right)+B{u}_{\mathrm{lqt}}\left(t\right),\mathrm{(17c)}& & z\left(0\right)=\Psi \left(x\right(0\left)\right).\hfill \end{array}$$

Because the running cost (15) is quadratic and the constraints are linear, the solution can be found by solving the algebraic Riccati equation.

The LQT controller takes the form

$$\begin{array}{c}\hfill u_{\mathrm{lqt}}\left(t\right)=-K\left(t\right)z\left(t\right)+s\left(t\right),\end{array}$$

(18)

where $K\left(t\right)={\overline{R}}^{-1}{B}^{\top }P\left(t\right)$, the matrix $P\left(t\right)$, and the tracking compensation term $s\left(t\right)$ can be satisfied via backwards integration of the Riccati differential equation

$$\begin{array}{c}\hfill \dot{P}\left(t\right)=-\overline{Q}+P\left(t\right)B{\overline{R}}^{-1}{B}^{\top }P\left(t\right)-P\left(t\right)A-A^{\top }P\left(t\right),\end{array}$$

(19a)

$$\begin{array}{c}\hfill \dot{s}\left(t\right)=-A^{\top }s+P\left(t\right)B{\overline{R}}^{-1}{B}^{\top }s\left(t\right)+\overline{Q}{z}_{ref},\end{array}$$

(19b)

from the final conditions

$$\begin{array}{c}\hfill P\left(T\right)=Q_f,\end{array}$$

(20a)

$$\begin{array}{c}\hfill s\left(T\right)=-Q_f{z}_{\mathrm{ref}}\left(T\right).\end{array}$$

(20b)

3.3. Active Learning Controller

When the amount of data obtained by the controller is small, there is a significant gap between the learned Koopman model and the infinite-dimensional Koopman operator model. Therefore, the LQT controller (18) designed based on the Koopman model cannot ensure that the trolley reaches the target position, and may even deviate far from the target position.

In light of Equation (11), we want to synthesize a controller that can ensure that the trolley reaches the target position while allowing for improvements in the information measure used for active learning. To achieve this, we design an active learning controller by minimizing the objective function $J_1$ (11) based on the LQT controller. For this, we need to know how sensitive the objective function $J_1$ (11) is to switching between the LQT control $u_{\mathrm{lqt}}$ and the active learning control $u_{\mathrm{al}}$ at time $\tau$ for a time duration $\lambda$.

The sensitivity of switching from $u_{\mathrm{lqt}}$ to $u_{\mathrm{al}}$ for all $\tau \in [0,T]$ for an infinitesimally small $\lambda$ (known as the mode insertion gradient [34]) is provided by

$$\begin{array}{c}\hfill \frac{\partial J_1}{\partial \lambda }{|}_{\tau ,\lambda =0}=\rho {\left(\tau \right)}^{\top }(f_2-f_1),\end{array}$$

(21)

where $z\left(t\right)$ is a solution to the Koopman model (10) under control $u\left(t\right)=u_{\mathrm{lqt}}\left(t\right)$ (18), $f_1=f_z(z\left(\tau \right),u_{\mathrm{lqt}}\left(\tau \right))$, $f_2=f_z(z\left(\tau \right),u_{\mathrm{al}}\left(\tau \right))$, and

$$\begin{array}{cc}\hfill \dot{\rho }\left(t\right)=& -\frac{\partial {\mathcal{l}}_{\mathrm{learn}}}{\partial z}-\frac{\partial {\mathcal{l}}_{\mathrm{task}}}{\partial z}-{\frac{\partial u_{\mathrm{lqt}}}{\partial z}}^{\top }(\frac{\partial {\mathcal{l}}_{\mathrm{learn}}}{\partial u}+\frac{\partial {\mathcal{l}}_{\mathrm{task}}}{\partial u})-(\frac{\partial f_z}{\partial z}+\frac{\partial f_z}{\partial u}\frac{\partial u_{\mathrm{lqt}}}{\partial z})\rho \left(t\right)\hfill \end{array}$$

(22)

subject to the terminal condition $\rho \left(T\right)=0$.

By minimizing the mode insertion gradient within the predicted time period $t\in [0,T]$, we are able to obtain the desired active learning controller $u_{\mathrm{al}}\left(t\right)$. Therefore, we can write an unconstrained optimization problem using a secondary objective function

$$\begin{array}{c}\hfill J_2={\int }_0^{\top }\frac{\partial J_1}{\partial \lambda }{|}_{\tau =t,\lambda =0}+\frac{1}{2}{\parallel u_{\mathrm{al}}\left(t\right)-u_{\mathrm{lqt}}\left(t\right)\parallel }_{\widehat{R}}^{2}dt,\end{array}$$

(23)

where $\widehat{R}\in {\mathbb{R}}^{m\times m}$ bounds the gap between $u_{\mathrm{al}}\left(t\right)$ and $u_{\mathrm{lqt}}\left(t\right)$ and where $\frac{\partial J_1}{\partial \lambda }{|}_{\tau =t,\lambda =0}$ is evaluated at $\tau =t$.

Proposition 1.

The active learning controller $u_{\mathrm{al}}\left(t\right)$ that minimizes (23) is

$$\begin{array}{c}\hfill u_{\mathrm{al}}\left(t\right)=-{\widehat{R}}^{-1}{B}^{\top }\rho \left(t\right)+u_{\mathrm{lqt}}\left(t\right).\end{array}$$

(24)

Proof. 

Because (23) is separable in time, we take the derivative of (23) with respect to $u_{\mathrm{al}}\left(t\right)$ at each point in t, which provides the following expression:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial u_{\mathrm{al}}}{J}_2=& {\int }_0^T\frac{\partial }{\partial u_{\mathrm{al}}}{(\rho \left(\tau \right)}^{\top }(f_2-f_1)\hfill \\ \hfill & +\frac{1}{2}(u_{\mathrm{al}}\left(t\right)-u_{\mathrm{lqt}}\left(t\right))\widehat{R}(u_{\mathrm{al}}\left(t\right)-u_{\mathrm{lqt}}\left(t\right)))dt\hfill \\ \hfill =& {\int }_0^T{B}^{\top }\rho \left(t\right)+\widehat{R}(u_{\mathrm{al}}\left(t\right)-u_{\mathrm{lqt}}\left(t\right))dt.\hfill \end{array}$$

The resulting equation can be obtained as follows:

$$\begin{array}{c}\hfill {\int }_0^T{B}^{\top }\rho \left(t\right)+\widehat{R}(u_{\mathrm{al}}\left(t\right)-u_{\mathrm{lqt}}\left(t\right))dt=0.\end{array}$$

(25)

Finally, solving for $u_{\mathrm{al}}\left(t\right)$ in (25) results in the solution

$$\begin{array}{c}\hfill u_{\mathrm{al}}\left(t\right)=-{\widehat{R}}^{-1}{B}^{\top }\rho \left(t\right)+u_{\mathrm{lqt}}\left(t\right).\end{array}$$

(26)

□

The active learning control $u_{\mathrm{al}}\left(t\right)$ at time $t=0$ is taken as the actual control acting on the controlled system (1) at the current moment. The approximate continuous-time operator $\tilde{\mathcal{G}}$ is updated by obtaining the real-time input and output states of the bridge crane. The Koopman model (3) is reconstructed using the updated continuous-time operator in the next control cycle for optimization and finding a solution. The above process is repeated to complete rolling optimization within the control time domain and realize active learning control for the bridge crane.

Remark 2.

The system parameters of the bridge crane system may change; for instance, there may be changes in the friction coefficient caused by the weather or other factors, changes in load quality, changes in rope length, etc. Traditional data-driven algorithms are based on offline data used to train models in advance. When the parameters change, these trained models may no longer be a good match, resulting in unsatisfactory control performance. On the other hand, the active learning approach is based on online data, which continues to be applicable even when the system parameters change.

The steps in the proposed active learning control methodology are as follows.

Step 1.

Define a set of observable functions; given an initial approximate continuous-time operator, construct the Koopman model.

Step 2.

Pre-set the desired trajectory and lift it using the observable function.

Step 3.

Design the linear quadratic optimal tracking controller based on the Koopman model and the desired trajectory.

Step 4.

Consider both the learning and running costs, design an active learning controller based on the linear quadratic optimal tracking controller.

Step 5.

Apply the active learning controller to the bridge crane at the current time and obtain the output after the control is applied.

Step 6.

Update the approximate continuous-time operator using the online input and output data.

Step 7.

Reconstruct the Koopman model using the updated continuous-time operator and repeat Steps 3–7.

4. Simulation

In order to verify the effectiveness of the proposed active learning method, simulations were conducted in the Python environment. According to the motion characteristics of the bridge crane, the constraints for the system during the simulation tests were as follows: the horizontal displacement constraint of trolley was $[0,2]\mathrm{m}$; the velocity constraint of the trolley was $[-5,5]\mathrm{m}/\mathrm{s}$; the swing angle constraint of the load was $[-90,90]\mathrm{deg}$; and the control input constraint was $[-25,25]\mathrm{N}$. The physical parameters of the bridge crane and the parameters of the active learning controller are provided in

Table 1. The parameters of the bridge crane and active learning controller.

Parameters	Value
Mass of trolley $M\left(\mathrm{kg}\right)$	5
Mass of load $m\left(\mathrm{kg}\right)$	10
Gravitational acceleration $g\left(\mathrm{m}/{\mathrm{s}}^2\right)$	$9.8$
Length of hoisting rope $l\left(\mathrm{m}\right)$	1
Friction coefficient $\mu$	$0.2$
Predicted horizons $T\left(\mathrm{s}\right)$	0.1
Sampling interval $t_s\left(\mathrm{s}\right)$	0.01
Dimension of the Koopman model L	78
State penalty weight matrix Q	$\left[\begin{array}{cccc}12000& 0& 0& 0\\ 0& 1200& 0& 0\\ 0& 0& 3000& 0\\ 0& 0& 0& 500\end{array}\right]$
Control weight matrix $\overline{R}$	1
Initial information weight $\sigma$	200
Regularization weight $\widehat{R}$	100
Terminal cost $Q_f$	0

. We denote $\xi :={[y,\theta ,\dot{y},\dot{\theta },sin\left(y\right),sin\left(\theta \right),sin\left(\dot{y}\right),sin\left(\dot{\theta }\right),cos\left(y\right),cos\left(\theta \right),cos\left(\dot{y}\right),cos\left(\dot{\theta }\right)]}^{\top }\in {\mathbb{R}}^{12}$, and the observable in the lifted space is chosen as $\Psi \left(x\right)={[{\xi }_1,\dots ,{\xi }_{12},{\xi }_1{\xi }_2,\dots ,{\xi }_{11}{\xi }_{12}]}^{\top }\in {\mathbb{R}}^{78}$.

4.1. Performance Evaluation for Active Learning Controller without Training in Advance

To assess the effectiveness of the proposed method, the active learning controller was applied to the bridge crane system (1) for the first time without training in advance. Instead of obtaining historical data to train the Koopman model in advance, we used random initialization. Considering a control without any nonlinear characteristic, we ran the simulation for $t=60\mathrm{s}$ with a sampling rate of $t_s$, as shown in Figure 3.

At first, the desired position of the trolley was set as $y_{\mathrm{ref}}=0.50\mathrm{m}$. Due to the lack of prior training, in the first 15 s the controller needs to learn the Koopman operator in order to establish the Koopman model. Therefore, the trolley does not immediately reach the desired position, and the load swing angle is large. After the controller learns the Koopman operator and establishes the model, the trolley immediately moves to the desired position of $0.5$ m while ensuring that there is no load swing. At $t=40$ s, the desired position of the trolley is set as $y_{\mathrm{ref}}=1.50\mathrm{m}$. Due to the controller having learned an accurate Koopman model, the trolley quickly reaches the desired position and the load swing remain small.

4.2. Comparative Study

For the purpose of better verifying the performance of the proposed method, two other control methods, namely, conventional sliding mode control (CSMC) [1] and PID [35], were selected to compare the control effect on the bridge crane system. The expressions of the two controllers used for comparison are provided below.

4.2.1. CSMC Controller

$$\begin{array}{cc}\hfill u\left(t\right)=& (M+msin{\left(\theta \left(t\right)\right)}^2)[-K{s}_p\left(t\right)-\eta \mathrm{sgn}\left(s_p\left(t\right)\right)-c_1\dot{y}\left(t\right)-c_2\dot{\theta }\left(t\right)\hfill \\ \hfill & +\frac{\mu \dot{y}\left(t\right)-mgsin\left(\theta \left(t\right)\right)cos\left(\theta \left(t\right)\right)-ml{\dot{\theta \left(t\right)}}^{2}sin\left(\theta \left(t\right)\right)}{M+m\ast sin{\left(\theta \left(t\right)\right)}^2}+\frac{gsin\left(\theta \right(t\left)\right)}{l}]\hfill \end{array}$$

(27)

where $c_1,c_2,c_3$ denote the sliding constants, $\eta ,K$ represents the gain of sliding mode surface, and $s_p$ is the position of the sliding mode surface, which is defined as follows:

$$\begin{array}{c}\hfill s_p\left(t\right)=\dot{y}\left(t\right)+c_1(y\left(t\right)-y_{\mathrm{ref}}\left(t\right))+c_2\theta \left(t\right)\end{array}$$

(28)

where $y_{\mathrm{ref}}$ is the desired horizontal displacement trajectory of y.

4.2.2. PID Controller

$$\begin{array}{cc}\hfill u\left(t\right)=& k_{yp}(y\left(t\right)-y_{\mathrm{ref}}\left(t\right))+k_{yi}{\int }_0^t(y\left(\tau \right)-y_{\mathrm{ref}}\left(\tau \right))d\tau +k_{1d}\dot{y}\left(t\right)\hfill \\ \hfill & +k_{\theta p}\theta \left(t\right)+k_{\theta i}{\int }_0^t\theta \left(\tau \right)d\tau +k_{\theta d}\dot{\theta }\left(t\right),\hfill \end{array}$$

(29)

where $k_{yp},k_{yi},k_{yd},k_{\theta p},k_{\theta i},k_{\theta d}$ represent the control gains.

The parameters of the CSMC controller and PID controller used in the experiments are provided in

Table 2. Parameters of the CSMC and PID controllers.

Controller	Parameters	Value
CSMC controller	$c_1$	0.3
	$c_2$	10
	$\eta$	0.001
	K	10
PID controller	$k_{yp}$	4
	$k_{yi}$	0.001
	$k_{yd}$	25
	$k_{\theta p}$	−0.1
	$k_{\theta i}$	0.1
	$k_{\theta d}$	−1

. Figure 4 presents the performance curves of all three controllers. It can be seen that the CSMC controller requires less crane operating positioning time and has payload swing compared to the PID controller. With our active learning control, the system can reach the desired position quickly with tolerable oscillations, while with CSMC the system reaches the desired position somewhat slower and with more limited oscillations. Note that CSMC is a model-based controller designed based on the model’s knowledge, which means that the design of the CSMC controller requires a known system model and accurate parameters. On the other hand, our approach uses a data-driven approach. The active learning control can achieve similar control effects when the system model is unknown; in this case, the absence of accurate system parameters means that the system has better robustness when using the proposed approach.

4.3. Robustness Study

To further validate the robustness of the bridge crane system when using the proposed approach, the following three groups of simulation experiments were set up based on different working conditions, such as the payload, rope length, and friction coefficient changes in engineering applications.

4.3.1. Simulation Group 1

Simulation experiments were conducted with the system parameter settings in Table 1 remaining the same except for the payload mass. In these experiments, the mass of the payload m was selected as 10 kg, 15 kg, and 20 kg.

Figure 5 refers to the variation curves of various state quantities of the bridge crane system with the active learning controller under different payload mass conditions.

4.3.2. Simulation Group 2

To assess the influence of changes in the rope length on the bridge crane system, the rope length was changed while keeping the other controller parameters consistent. The rope length l was selected as $1.0$ m, $1.5$ m, and $2.0$ m. The simulation results are shown in Figure 6.

It is shown that when the rope length changes while keeping the controller parameters unchanged, the proposed method achieves better positioning control and a relatively small maximum load swing angle.

4.3.3. Simulation Group 3

This group describes the changes in the control effect in the whole process with the active learning controller and under different track friction conditions. The other system parameter settings remained the same, while the friction coefficient between the crane and the track was adopted as $\mu =0.02$, $0.2$, and 1. The results are shown in Figure 7.

With a changing friction coefficient, the sensitivity of the system is even lower. From Figure 7a,b, it can be seen that the control performance curves of the system under the three friction coefficients are essentially superimposed on one another.

In summary, these three simulation experiments verify that the active learning controller is less sensitive than others to changes in model parameters, specifically, the load mass, rope length, and friction coefficient, and that the proposed system has strong robustness.

4.4. Performance with Dead Zone

Due to the location of the transmission device between the gear and the track, a dead zone characteristic may exist in the bridge crane system. Therefore, in this subsection we consider the controller in a scenario with a dead zone, as follows:

$$\begin{array}{c}\hfill u=\left\{\begin{array}{cc}{u}_r-r\hfill & \hfill u_r>r\\ 0\hfill & \hfill -r\le u_r\le r\\ u_r+r\hfill & \hfill u_r<-r\end{array}\right.\end{array}$$

(30)

where $u_r\left(t\right)=u_{\mathrm{al}}\left(t\right)$ represents the control input calculated by the active learning controller and r is the dead zone parameter, which reflects the level of the dead zone.

Simulation experiments were conducted with the same system parameter settings provided in Table 1. We performed numerical simulations to assess the capability of the proposed active learning control in a scenario with the dead zone characteristic under different dead zone parameters. The dead zone parameter r was selected as $0.1$, $0.5$, 1, and 2. The desired position of trolley was set as $y_{\mathrm{ref}}=0.5$ m. Figure 8 presents the simulation results of the bridge crane system with the active learning controller and input dead zone parameters. Figure 8a,b shows that when $r=0.1$, i.e., the dead zone characteristic is not significant, the bridge crane system can quickly reach the desired position with a steady-state error of only $0.008$ m and with only smooth and minor swinging of the rope. As the value of the dead zone parameter r increases, the steady-state error between the real position and the desired position gradually increases and the swinging of the rope grows more prominent. Due to the existence of the dead zone characteristic, the nonlinear controller u in (30) is applied to the system instead of the desired active learning control input $u_r$ shown in Figure 8c. Therefore, the larger the dead zone parameter, the worse the control effect becomes.

5. Conclusions

In this paper, we have proposed an active learning control for bridge cranes without the need for training in advance. In the proposed approach, the Koopman operator is used to enhance the control effect of the bridge crane system. Using only the real-time online input and output data of the bridge crane system, the active learning control method consists of finite-dimensional approximation of Koopman operator and the design of an active learning controller based on linear quadratic optimal tracking control. The effectiveness of this method has been verified through numerical simulations. The proposed control method has a simple structure, good tracking performance, and strong robustness for bridge cranes.