Adaptive Machine Learning for Robust Diagnostics and Control of Time-Varying Particle Accelerator Components and Beams

Abstract

Machine learning (ML) is growing in popularity for various particle accelerator applications including anomaly detection such as faulty beam position monitor or RF fault identification, for non-invasive diagnostics, and for creating surrogate models. ML methods such as neural networks (NN) are useful because they can learn input-output relationships in large complex systems based on large data sets. Once they are trained, methods such as NNs give instant predictions of complex phenomenon, which makes their use as surrogate models especially appealing for speeding up large parameter space searches which otherwise require computationally expensive simulations. However, quickly time varying systems are challenging for ML-based approaches because the actual system dynamics quickly drifts away from the description provided by any fixed data set, degrading the predictive power of any ML method, and limits their applicability for real time feedback control of quickly time-varying accelerator components and beams. In contrast to ML methods, adaptive model-independent feedback algorithms are by design robust to un-modeled changes and disturbances in dynamic systems, but are usually local in nature and susceptible to local extrema. In this work, we propose that the combination of adaptive feedback and machine learning, adaptive machine learning (AML), is a way to combine the global feature learning power of ML methods such as deep neural networks with the robustness of model-independent control. We present an overview of several ML and adaptive control methods, their strengths and limitations, and an overview of AML approaches.

1. Introduction

A simple code for the adaptive control algorithm used here can be downloaded from: https://github.com/alexscheinker/ES_adaptive_optimization, accessed on 8 April 2021. Machine learning (ML) [1] tools such as neural networks (NN) [2], Gaussian processes (GP) [3], and reinforcement learning (RL) in which NNs are incorporated to represent system models and optimal feedbacks [4], have been growing in popularity for particle accelerator applications. Although these methods have been around for decades, their recent growth in popularity can be attributed to recent growth in computing power with high performance computers and especially graphics processing units (GPUs) becoming very inexpensive. Furthermore, powerful and easy to use software packages such as tensorflow are now freely available for anyone to easily develop their own sophisticated ML tools specifically tailored to their accelerator problems.

Recent ML applications for accelerators include ML-enhanced genetic optimization [5], utilizing surrogate models for simulation-based optimization studies and for estimating beam characteristics [6,7,8,9,10], Bayesian and GP approaches for accelerator tuning [11,12,13,14,15,16], various applications at the Large Hardon Collider including optics corrections and detecting faulty beam position monitors [17,18,19], powerful polynomial chaos expansion-based surrogate models for uncertainty quantification have been developed [20], and RL tools have been developed for online accelerator optimization [21,22,23,24,25]. One challenge faced by many ML approaches is the fact that as accelerators and their beams change with time, the ML models that were trained with previously collected data are no longer accurate because they are being applied to a different system than the one which they have been trained for. In order to provide accurate control and diagnostics in the presence of time varying systems a real time feedback adaptive ML approach is required.

Recently, novel adaptive feedback algorithms have been developed which are able to tune large groups of parameters simultaneously based only on noisy scalar measurements with analytic proofs of convergence and analytically known guarantees on parameter update rates, which makes them especially well suited for particle accelerator problems [26]. Such methods can be easily implemented via custom python scripts that read and write from machine components via network systems such as EPICS [27] and have been implemented in powerful optimization software such as OCELOT for online accelerator tuning [28]. The main benefit of adaptive methods is that they can be applied online in real time to drifting accelerator systems. For example, these methods have now been applied to automatically and quickly maximize the output power of FEL light at both the LCLS and the European XFEL and are able to compensate for un-modeled time variation in real time while optimizing 105 parameters simultaneously [29]. Another example of the benefit of these approaches is for multi-objective optimization which is typically done offline via extremely lengthy simulation studies. Adaptive methods have been demonstrated for real-time online multi-objective optimization of the electron beam line at AWAKE at CERN for simultaneous emittance growth minimization and trajectory control [30]. These methods have also been demonstrated at FACET to provide non-invasive LPS diagnostics that can actually predict and actively track time-varying TCAV measurements as both accelerator components and initial beam distributions drift with time [31].

Adaptive Machine Learning

The first adaptive ML approach combining ML and adaptive feedback for time-varying particle accelerator systems was recently developed for real time automatic control of the longitudinal phase space (LPS) of the LCLS electron beam [32]. The adaptive ML approach was demonstrated to combine the best of each family of tools: the power of ML tools such as NNs to learn complex relationships directly from measured data and the robust ability of adaptive feedback to handle time-varying noisy and unknown system dynamics. This adaptive ML approach has the potential to solve one of the main limitations in terms of matching the predictions of models (physics-based or surrogate) to actual accelerator beams, by adaptively identifying the initial beam distributions entering the accelerators [33], whose knowledge is needed for accurate model-based predictions.

In this work, our primary interest is in developing adaptive controls and diagnostics for time-varying conditions where it is possible that both the accelerator parameters and the initial beam entering the accelerator change as a function of time in unpredictable ways. Such uncertain time variation is one of the main challenges of accelerators and what makes it so difficult to run online models as real time non-invasive diagnostics because even the perfect model will not be predictive if it is not initialized with the correct parameter values or the correct initial beam distribution whose dynamic evolution is then simulated.

In this paper, we review the difficulties associated with time-varying systems and present a general formulation of applying machine learning to time-varying and uncertain systems and how to use output-based feedbacks to adjust the ML models and their inputs and outputs in real time to make them robust to un-modeled time-variation of the system dynamics and changing initial conditions. Furthermore, we demonstrate our approach with a detailed simulation-based study of a complex 22-dimensional particle accelerator system in which a beam with an unknown, not measurable, and time varying initial distribution enters a section of a particle accelerator in which its transverse size is maintained with a 22 quadrupole magnet lattice. We demonstrate that by using only available non-invasive measurements of the transverse root mean square X and Y beam sizes at the end of this accelerator section, we can adaptively feedback and tune the predictions of a trained neural network to accurately predict the beam envelope evolution throughout the accelerator section, thereby serving as an online, non-invasive, adaptive virtual beam diagnostic.

2. Unknown Time-Varying Systems and Adaptive Feedback Control

Control in the presence of uncertainty and time-variation is extremely challenging even for linear systems. Linear systems are very popular because they are simple local approximations of much more complicated nonlinear dynamics and can be analytically solved. Consider a general n-dimensional nonlinear system

$$\begin{array}{ccc}\hfill \dot{\mathbf{x}}& =& \mathbf{f}(\mathbf{x},\mathbf{u}),\hfill \\ \hfill \underset{\dot{\mathbf{x}}}{\underbrace{\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\\ ⋮\\ {\dot{x}}_n\end{array}\right]}}& =& \underset{\mathbf{f}(\mathbf{x},\mathbf{u})}{\underbrace{\left[\begin{array}{c}{f}_1(\mathbf{x},\mathbf{u})\\ f_2(\mathbf{x},\mathbf{u})\\ ⋮\\ f_n(\mathbf{x},\mathbf{u})\end{array}\right]}},\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right],\mathbf{u}=\left[\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_m\end{array}\right].\hfill \end{array}$$

(1)

Within a small enough neighborhood of any point $({\mathbf{x}}_0,{\mathbf{u}}_0)\in {\mathbb{R}}^{n\times m}$, we can approximate the system (1), by a linear system, based on the Jacobian matrices of $\mathbf{f}(\mathbf{x},\mathbf{u})$. Via Taylor series expansion, for $∥(\mathbf{x},\mathbf{u})-({\mathbf{x}}_0,{\mathbf{u}}_0)∥\ll 1$, we can approximate the $f_i(x,t)$ as:

$$\begin{array}{ccc}\hfill f_i(\mathbf{x},\mathbf{u})& \approx & f_i({\mathbf{x}}_0,{\mathbf{u}}_0)\hfill \\ & & +\frac{\partial f_i({\mathbf{x}}_0,{\mathbf{u}}_0)}{\partial x_1}(x_1-x_{0,1})+\cdots +\frac{\partial f_i({\mathbf{x}}_0,{\mathbf{u}}_0)}{\partial x_n}(x_n-x_{0,n})\hfill \\ & & +\frac{\partial f_i({\mathbf{x}}_0,{\mathbf{u}}_0)}{\partial u_1}(u_1-u_{0,1})+\cdots +\frac{\partial f_i({\mathbf{x}}_0,{\mathbf{u}}_0)}{\partial u_m}(u_m-u_{0,m})\hfill \end{array}$$

Thereby we approximate the nonlinear system

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u})$$

with

$$\begin{array}{ccc}\hfill \underset{\dot{\mathbf{x}}}{\underbrace{\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\\ ⋮\\ {\dot{x}}_n\end{array}\right]}}& \approx & \underset{\mathrm{Jacobian}\mathrm{with}\mathrm{respect}\mathrm{to}\mathbf{x}}{\underbrace{\left[\begin{array}{cccc}\frac{\partial f_1}{\partial x_1}& \frac{\partial f_1}{\partial x_2}& \cdots & \frac{\partial f_1}{\partial x_n}\\ \frac{\partial f_2}{\partial x_1}& \frac{\partial f_2}{\partial x_2}& \cdots & \frac{\partial f_2}{\partial x_n}\\ ⋮& ⋮& ⋮& ⋮\\ \frac{\partial f_n}{\partial x_1}& \frac{\partial f_n}{\partial x_2}& \cdots & \frac{\partial f_n}{\partial x_n}\end{array}\right]}}\underset{\mathbf{x}-{\mathbf{x}}_0}{\underbrace{\left[\begin{array}{c}{x}_1-x_{0,1}\\ x_2-x_{0,2}\\ ⋮\\ x_n-x_{0,n}\end{array}\right]}}\hfill \\ & & +\underset{\mathrm{Jacobian}\mathrm{with}\mathrm{respect}\mathrm{to}\mathbf{u}}{\underbrace{\left[\begin{array}{cccc}\frac{\partial f_1}{\partial u_1}& \frac{\partial f_1}{\partial u_2}& \cdots & \frac{\partial f_1}{\partial u_m}\\ \frac{\partial f_2}{\partial u_1}& \frac{\partial f_2}{\partial u_2}& \cdots & \frac{\partial f_2}{\partial u_m}\\ ⋮& ⋮& ⋮& ⋮\\ \frac{\partial f_n}{\partial u_1}& \frac{\partial f_n}{\partial u_2}& \cdots & \frac{\partial f_n}{\partial u_m}\end{array}\right]}}\underset{\mathbf{u}-{\mathbf{u}}_0}{\underbrace{\left[\begin{array}{c}{u}_1-u_{0,1}\\ u_2-u_{0,2}\\ ⋮\\ u_n-u_{0,m}\end{array}\right]}}+\underset{\mathbf{f}({\mathbf{x}}_0,{\mathbf{u}}_0)}{\underbrace{\left[\begin{array}{c}{f}_1({\mathbf{x}}_0,{\mathbf{u}}_0)\\ f_2({\mathbf{x}}_0,{\mathbf{u}}_0)\\ ⋮\\ f_n({\mathbf{x}}_0,{\mathbf{u}}_0)\end{array}\right]}},\hfill \end{array}$$

with all of the above derivatives calculated at the point $({\mathbf{x}}_0,{\mathbf{u}}_0)$. For any point $({\mathbf{x}}_0,{\mathbf{u}}_0)$, we can always change coordinates to

$$(\mathbf{y},\mathbf{v})=(\mathbf{x},\mathbf{u})-({\mathbf{x}}_0,{\mathbf{u}}_0),$$

which satisfy

$$\dot{\mathbf{y}}=\mathbf{f}(\mathbf{y}+{\mathbf{x}}_0,\mathbf{u}+{\mathbf{u}}_0)={\mathbf{f}}_2(\mathbf{y},\mathbf{u}).$$

Therefore without loss of generality we can consider the case $({\mathbf{x}}_0,{\mathbf{u}}_0)=(\mathbf{0},\mathbf{0})$, and from now on we ignore the term $\mathbf{f}(\mathbf{0},\mathbf{0})$, recognizing that it is just a constant disturbance. Next, if we define the constants

$$a_{ij}=\frac{\partial f_i}{\partial x_j}(\mathbf{0},\mathbf{0}),b_{ij}=\frac{\partial f_i}{\partial u_j}(\mathbf{0},\mathbf{0}),$$

then we can write our system as

$$\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\\ ⋮\\ {\dot{x}}_n\end{array}\right]=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& ⋮& ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]+\left[\begin{array}{cccc}{b}_{11}& b_{12}& \cdots & b_{1m}\\ b_{21}& b_{22}& \cdots & b_{2m}\\ ⋮& ⋮& ⋮& ⋮\\ b_{n1}& b_{n2}& \cdots & b_{nm}\end{array}\right]\left[\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_m\end{array}\right]$$

or

$$\dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}.$$

(2)

If a linear feedback control of the form

$$\mathbf{u}=-K\mathbf{x}$$

(3)

is used, the resulting closed loop system is still linear

$$\dot{\mathbf{x}}=\left(A-BK\right)\mathbf{x},$$

(4)

and can be analytically solved with the help of Laplace transforms to get

$$\mathbf{x}\left(t\right)=e^{(A-BK)t}\mathbf{x}\left(0\right)=e^{A_{c}t}\mathbf{x}\left(0\right),A_c=A-BK.$$

(5)

Finally, the transient dynamics and stability of the system (4) are completely defined by the eigenvalues of the closed loop system matrix $A_c$ because for any square matrix, $A_c$, there exists an invertible matrix P such that

$$P^{-1}{A}_{c}P=J=\mathrm{Block}\mathrm{Diagonal}[J_1,J_2,\cdots ,J_r],$$

(6)

where $J_i$ is a Jordan block associated with the eigenvalue ${\lambda }_i$ of $A_c$, where Jordan blocks of order one are

$$J_i={\lambda }_i,$$

and a Jordan blocks or order $m_i$ are:

$$J_i={\left[\begin{array}{cccccc}{\lambda }_i& 1& 0& \cdots & \cdots & 0\\ 0& {\lambda }_i& 1& 0& \cdots & 0\\ ⋮& 0& \ddots & 0& 0& ⋮\\ ⋮& 0& 0& \ddots & 0& 0\\ ⋮& 0& 0& 0& {\lambda }_i& 1\\ 0& 0& 0& 0& 0& {\lambda }_i\end{array}\right]}_{m_i\times m_i}.$$

Using the matrix expansion (6), we can rewrite the matrix exponential as

$$\begin{array}{ccc}\hfill e^{A_{c}t}& =& e^{PJ{P}^{-1}t}=P{e}^{Jt}{P}^{-1},\hfill \end{array}$$

(7)

to get the solution

$$\mathbf{x}\left(t\right)=e^{(A-BK)t}\mathbf{x}\left(0\right)=e^{A_{c}t}\mathbf{x}\left(0\right)=P{e}^{Jt}{P}^{-1}\mathbf{x}\left(0\right),A_c=PJ{P}^{-1},$$

(8)

which, after some algebra, can be rewritten as

$$\mathbf{x}\left(t\right)=\left(\sum _{i=1}^r\sum _{k=1}^{m_i}{t}^{k-1}{e}^{{\lambda }_{i}t}{R}_{ik}\right)\mathbf{x}\left(0\right).$$

The eigenvalues of $A_c=A-BK$ have found their way into the exponential terms $e^{{\lambda }_{i}t}$. Expanding eigenvalues as a sum of real and imaginary parts, ${\lambda }_i={\lambda }_{r,i}+i{\lambda }_{I,i}$, we can expand the exponential as

$$e^{{\lambda }_{i}t}=e^{({\lambda }_{r,i}+i{\lambda }_{I,i})t}=e^{{\lambda }_{r,i}t}{e}^{i{\lambda }_{I,i}t}=e^{{\lambda }_{r,i}t}\left[cos\left({\lambda }_{I,i}t\right)+isin\left({\lambda }_{I,i}t\right)\right].$$

Clearly the imaginary parts of the eigenvalues define the resonant frequencies of the system while the real parts control whether the trajectories of the dynamics converge or diverge exponentially. If we choose our feedback controller $-Kx$ such that the closed loop matrix $A-BK$, is Hurwitz, so its eigenvalues have negative real parts, then all ${\lambda }_{r,i}<0$, and the system’s trajectory exponentially converges to the origin, and the system is globally exponentially stable. This simple analysis of linear systems has lead to the popularity of simple proportional integral derivative (PID) control which is by far the most common form of feedback control.

If however, the original nonlinear system that we are controlling is time-varying

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u},t),$$

(9)

then by the same arguments as above, we can approximate our system (9) at any instant of time, within a small neighborhood of some point, by the linear time-varying system

$$\dot{\mathbf{x}}=A\left(t\right)\mathbf{x}+B\left(t\right)\mathbf{u},$$

(10)

but that is where the similarity ends. For time-varying systems an eigenvalue analysis is completely useless (except in some very special cases, such as periodic or arbitrarily slowly varying systems) and determining whether system (10) is stable or not and therefore designing stabilizing controllers, is incredibly difficult and requires a nonlinear Lyapunov analysis [34]. The uselessness of eigenvalues for determining whether time-varying systems are stable or unstable is demonstrated by a few simple examples taken from [35,36] and the references within. The system

$$\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\end{array}\right]=\left[\begin{array}{cc}-1+1.5{cos}^2\left(t\right)& 1-1.5sin\left(t\right)cos\left(t\right)\\ -1-1.5sin\left(t\right)cos\left(t\right)& -1+1.5{sin}^2\left(t\right)\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right],$$

has constant eigenvalues with negative real parts ${\lambda }_{\pm }=-0.25\pm 0.25\sqrt{7}i,$ but is exponentially unstable with solution

$$\left[\begin{array}{c}{x}_1\left(t\right)\\ x_2\left(t\right)\end{array}\right]=\left[\begin{array}{cc}{e}^{0.5t}cos\left(t\right)& e^{-t}sin\left(t\right)\\ -e^{0.5t}sin\left(t\right)& e^{-t}cos\left(t\right)\end{array}\right]\left[\begin{array}{c}{x}_1\left(0\right)\\ x_2\left(0\right)\end{array}\right].$$

The linear time-varying system

$$\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\end{array}\right]=\left[\begin{array}{cc}-1+5cos\left(t\right)sin\left(t\right)& -5{cos}^2\left(t\right)\\ 5{sin}^2\left(t\right)& -1-5cos\left(t\right)sin\left(t\right)\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right],$$

has constant negative real eigenvalues ${\lambda }_i=-1$ and is also unstable. The system

$$\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{x}}_2\end{array}\right]=\left[\begin{array}{cc}-\frac{11}{2}+\frac{15}{2}sin\left(12t\right)& \frac{15}{2}cos\left(12t\right)\\ \frac{15}{2}cos\left(12t\right)& -\frac{11}{2}-\frac{15}{2}sin\left(12t\right)\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right],$$

has constant eigenvalues, one of which is positive real, ${\lambda }_i=\{2,-13\}$, but is stable.

As seen from the examples above, even when we know everything about a system, if it is time-varying its stability properties are not obvious. On top of this a whole new level of difficulty is added when the system is also unknown. The main difficulties faced in feedback control is when the sign of the control signal relative to system dynamics is unknown. Standard control methods, such as direct model reference adaptive control, exist for systems with uncertainties of known sign. For example, consider an unknown system of the form

$$\dot{x}\left(t\right)=ax\left(t\right)+bu\left(t\right),$$

(11)

where the values of a and b are unknown, but the sign of b is known. The goal here is in controlling this system so that it behaves like a known reference model system

$${\dot{x}}_m\left(t\right)=a_m{x}_m\left(t\right)+b_m{r}_m\left(t\right),$$

(12)

in which $a_m$, $b_m$, and $r_m\left(t\right)$ are chosen by the user. If $b>0$ then the standard adaptive feedback approach is to design a controller for (11) of the form

$$\begin{array}{ccc}\hfill u\left(t\right)& =& c_1\left(t\right)r_m\left(t\right)+c_2\left(t\right)x\left(t\right),\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill {\dot{c}}_1\left(t\right)& =& -k\left(x\left(t\right)-x_m\left(t\right)\right)r_m\left(t\right),\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\dot{c}}_2\left(t\right)& =& -k\left(x\left(t\right)-x_m\left(t\right)\right)x\left(t\right),\hfill \end{array}$$

(15)

with adaptive gain $k>0$. The closed loop system with the dynamics (11) under the influence of adaptive controller (13)–(15) converges to the model reference dynamics (12) with the convergence to zero of the error $|x\left(t\right)-x_m\left(t\right)|$.

The adaptive control described above immediately fails for unknown sign of b. Consider an extremely simple 1D example of a one parameter system with dynamics

$$\dot{x}=x+b\left(t\right)u_1(x,t),$$

(16)

where $u_1(x,t)$ is a feedback controller and $b\left(t\right)$ is an unknown time-varying function such as $b\left(t\right)=bcos\left(\omega t\right)$. The stabilization of such a system with unknown control direction, $b\left(t\right)$, was an open problem in control theory for a long time. In 1985, a solution was proposed for the simple time-invariant case where $b\left(t\right)\equiv b$ was an unknown constant value [37,38], this solution suffered from an unbounded growing overshoot for time-varying $b\left(t\right)$ with changing sign, eventually destabilizing and destroying any physical system. In 2012, problem (16) was finally solved with a novel model-independent approach [39,40], whose feedback forces the dynamics (16) to have an average behavior described by a new system of the form

$$\dot{\overline{x}}=\overline{x}-b^2\left(t\right)u_2(\overline{x},t),\parallel x\left(t\right)-\overline{x}\left(t\right)\parallel <ϵ$$

(17)

with arbitrarily small $ϵ>0$, where the unknown control direction, $b\left(t\right)$, has become squared and can be stabilized automatically. Because of arbitrarily close proximity of the trajectories of (17) to (16), stabilization of (17) is equivalent to stabilizing (16).

The results in [39,40] actually solved the much more general problem of stabilizing a n-dimensional nonlinear time-varying unknown system of the form

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},t)+\mathbf{g}(\mathbf{x},t)\mathbf{u}(\mathbf{x},t),$$

where $\mathbf{f}$ and $\mathbf{g}$ are both nonlinear, time-varying, and analytically unknown. The method has now been generalized further with analytical proofs of stability for non-differentiable systems as well as systems not affine in control [41,42], of the form

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u},t),$$

and has been utilized in various particle accelerator applications [26,29,30,31,32,43].

3. Machine Learning for Time-Varying Systems

One application of machine learning is to try to learn unknown system relationships directly from data. This has been especially popular in the accelerator community for surrogate models and diagnostics based on them. Consider a family of $N_p$ parameters in an accelerator, $\mathbf{p}=(p_1,\cdots ,p_{N_p})$, which might include the currents of magnet power supplies and RF cavity amplitude and phase settings throughout an accelerator. Theoretically each set of parameter settings ${\mathbf{p}}_i$ maps, via some complicated nonlinear function, F, to an observable $O_i$ according to

$$O_i=F\left({\mathbf{p}}_i\right),$$

(18)

which may be, for example, the 2D $(z,E)$ LPS distribution of the accelerator beam at a particular location. An ML tool can learn a close approximation $\widehat{F}$ of the unknown function F directly from a data set D which contains enough pairs of parameter settings and their corresponding beam distributions

$${\widehat{O}}_i=\widehat{F}\left({\mathbf{p}}_i\right),D=\left\{({\mathbf{p}}_1,O_1),\cdots ,({\mathbf{p}}_m,O_m)\right\},e=\sum _{i=1}^m{\left({\widehat{O}}_i-O_i\right)}^2$$

(19)

by minimizing some measure of error, such as e above, between ML predictions and observed data over the entire data set. Such a surrogate model approach has become popular recently for particle accelerator applications. For example, this has been done to map accelerator parameters to the LPS of the LCLS beam with the approximation $\widehat{F}$ taking the form of a deep neural network [8].

Such machine learning tools have also been applied in the field of optimal control. The original optimal control problems were formulated for dynamic systems of the form (here we stick to a scalar example, but the same holds for vector-valued systems)

$$\dot{x}=f(x,u),J=C_T\left(x\left(T\right)\right)+{\int }_0^T{C}_t(x\left(t\right),u\left(t\right))dt,$$

(20)

where $f(x,u)$ is the known nonlinear system dynamics and J is a functional of both the control effort $u\left(t\right)$ and the resulting trajectory $x\left(t\right)$, which are penalized through some cost function $C_t(x\left(t\right),u\left(t\right))$ as well as a terminal cost $C_T\left(x\left(T\right)\right)$. The solution to this optimal control problem was formulated by Richard Bellman in the 1950s with a method known as Dynamic programming with the use of Lagrange multipliers [44], and many generalizations have developed over the years in the control theory community [45,46]. A quick introduction to the Dynamic programming approach is presented here because it is not only the foundation for many optimal control approaches, but is also the underlying foundation for many reinforcement learning (RL) approaches.

The starting point is that the dynamic system and cost function in (20) can both be approximated via the finite difference approximation of the derivative as

$$x(t+{\Delta }_t)\approx x\left(t\right)+{\Delta }_{t}f(x\left(t\right),u\left(t\right)),C\approx C_T\left(x\left(N{\Delta }_t\right)\right)+{\Delta }_t\sum _{n=1}^N{C}_t(x\left(n{\Delta }_t\right),u\left(n{\Delta }_t\right)),$$

(21)

which is then simply rewritten as an iterative discrete system

$$x(n+1)=x\left(n\right)+f_d(x\left(n\right),u\left(n\right)),C\approx C_{T,d}\left(x\left(N\right)\right)+\sum _{n=1}^N{C}_{t,d}(x\left(n\right),u\left(n\right)).$$

(22)

The dynamic programming approach is to start from the optimal position $x^{*}\left(N\right)$ and the associated optimal cost $C^{*}$ and work backwards considering the optimal way to arrive at state $x^{*}\left(N\right)$ from all possible states $x^{*}(n-1)$. The jump from any single step state $x\left(n\right)$ to the state $x(n+1)$ is then rewarded or penalized via the resulting cost C in order to develop an optimal controller for the optimal state’s evolution when starting from any point $x\left(n\right)$. This method is highly computationally expensive, but for a system with analytically known dynamics and cost function the solution is exact and globally optimal. Recently, methods such as reinforcement learning have been developed for this optimal control problem for unknown systems with unknown dynamics f and measurable, but analytically unknown cost function C for which it is impossible to calculate the optimal control policy with the standard Dynamic Programming approach [4]. One example is to use adaptive methods which can be applied for online RL in which optimal feedback control policies are learned directly from data to learn optimal feedback control policies which are parametrized by a chosen set of basis functions whose coefficients must be adaptively tuned online [43]. Other RL approaches learn either the unknown dynamics f, or the cost function C, or both directly from measured data and represent them as trained neural networks, as described above. Although optimal control and RL are very similar and have closely tied foundations, the communities have diverged and new terms for the same things are being used, such as “agents” for parameter states in phase space, “policies” for feedback controllers, and “rewards/penalties” for cost functions.

For time-varying systems in general, and for accelerators in particular, the performance of any model-based approaches, whether surrogate models or RL methods based on trained neural networks, will degrade if the systems for which they have been trained start to change with time. For example, one of the most impressive demonstrations of RL in the world is Google’s AlphaZero program which defeats world champions in complex games such as chess, shogi, and Go [47]. However, if in the middle of a game the rules of chess were to change, AlphaZero would most likely completely fail and would require a new round of millions of virtual games to re-train its neural networks, whereas a grand-master could quickly adapt to the new rules and continue to play. Therefore, we believe it is important to build on the strength of data-based methods which train powerful neural networks, by building in flexibility via adaptive feedback. These kinds of changes are not just theoretical, but show up in time-varying systems, such as accelerators, as described in more detail below.

It is known that both accelerator components and the initial beam distributions that are formed on photocathodes vary unpredictably with time and measurements are typically noisy and have arbitrary offsets, such as phase shifts of RF cables and analog components due to temperature and changing relationships between magnetic fields and power supplies due to hysteresis. Furthermore, relationships between system inputs and outputs can change. For example, depending on the time-varying initial conditions of a beam entering a certain accelerator section, increasing a quadrupole’s magnetic field might result in less beam loss at one instance and more beam loss at another. Our goal is to tackle the problem of time varying systems in which (18) is replaced with

$$O_i\left(t\right)=F({\mathbf{p}}_i\left(t\right),{\mathit{\rho }}_0\left(t\right),t),$$

(23)

where we emphasize that the parameters ${\mathbf{p}}_i\left(t\right)$ drift with time, that the initial beam distribution ${\mathit{\rho }}_0\left(t\right)$ entering any accelerator section drifts with time, and that the overall relationship between beam observables and parameters drifts with time $\left(t\right)$. Furthermore, we recognize that at any observations of the accelerator and its beam are actually of the form $(\widehat{\mathbf{p}}{\left(t\right)}_i,O_i)$ and ${\widehat{\mathit{\rho }}}_0\left(t\right)$, where ${\widehat{\mathbf{p}}}_i\left(t\right)$ is an estimate of parameters ${\mathbf{p}}_i\left(t\right)$ and ${\widehat{\mathit{\rho }}}_0\left(t\right)$ an estimate of the uncertain initial beam distribution ${\mathit{\rho }}_0\left(t\right)$. Therefore, a surrogate model such as a deep neural network will only result in an approximation that is valid for a small time interval, whose performance will drift as the accelerator’s characteristics drift away from any collected data set.

Our approach is to implement adaptive ML which combines online simulations, adaptive feedback, and ML for systems of the form (23), which can be summarized as

$$\begin{array}{ccc}& {\widehat{O}}_i\left(t\right)=\widehat{F}({\widehat{\mathbf{p}}}_i\left(t\right),{\widehat{\mathit{\rho }}}_0\left(t\right),\mathbf{w}\left(\mathbf{t}\right)),& {\widehat{\mathit{\rho }}}_0\left(t\right)={\widehat{F}}_{\mathit{\rho }}({\widehat{\mathbf{p}}}_i\left(t\right),{\mathbf{w}}_{\mathit{\rho }}\left(\mathbf{t}\right)),\hfill \end{array}$$

(24)

$$\begin{array}{ccc}& \widehat{\psi }\left(t\right)=M_{\psi }(\widehat{\mathbf{p}}\left(t\right),{\widehat{\mathit{\rho }}}_0\left(t\right)),& e_{\widehat{\psi }}\left(t\right)=\int {\left(\widehat{\psi }(x,t)-\psi (x,t)\right)}^{2}dx,\hfill \end{array}$$

(25)

where $\widehat{F}$ represents an adaptive model-ML hybrid data-based learned representation of the observable O, where a trained surrogate model such as a convolutional neural network initially predicts the observable $\widehat{O}$, with the prediction then refined by an online model. The adaptive ML model $\widehat{F}$ receives time-varying parameter estimates $\widehat{\mathbf{p}}\left(t\right)$ and an estimate of the beam’s phase space ${\widehat{\mathit{\rho }}}_0\left(t\right)$ from a second model which is adaptively tuned based on the error $e_{\widehat{\psi }}\left(t\right)$ quantifying the match between its prediction $\widehat{\psi }\left(t\right)$ and the detected $\psi \left(t\right)$ value of a rich non-invasively measured beam characteristic such as the beam’s energy spread spectrum. The parameter estimates, $\widehat{\mathbf{p}}\left(t\right)$, ML weights $\mathbf{w}\left(t\right)$, and initial beam distribution $\widehat{\mathit{\rho }}\left(t\right)$ are all adaptively tuned utilizing a model-independent adaptive feedback control method.

4. Controls and Diagnostics for a 22 Dimensional System

To demonstrate some of the ideas described above, we perform a simulation-based study of an incredibly high-dimensional system in which we control 22 parameters which are the quadrupole magnets in the low energy beam transport (LEBT) of the Los Alamos Neutron Science Center (LANSCE) linear accelerator, as shown in Figure 1. The fact that our system has 22 dimensions makes it clear that ML-based studies are very challenging because even a very coarse grid search of 10 steps per parameter results in a staggering ${10}^{22}$ data points. In this example, we demonstrate that by building in adaptive feedback into our ML architecture, we can automatically estimate and track the time-varying unknown input beam distribution properties without access to their measurements, based only on measurable output measurements of the dynamic system which is used to adaptively guide the ML data-based model.

In this approach it would be possible to adaptively control all of the 22 quadruple magnets in addition to the input beam distributions, but in practice unless a major construction project has taken place or major components requiring disassembly of the beam line have taken place, the quadrupole magnets in a beam line do not move very much or very quickly. Their fields do suffer from hysteresis, but this can be mapped and taken care of with careful setting procedures. On the other hand, the input beam distribution is known to change as the source performance drifts over the course of a few weeks and definitely changes after each source recycle, which takes place every few weeks. Therefore those parameters, the initial beam characteristics, are the ones that are most difficult to measure during operations. Their measurements require lengthy invasive scans that interrupt regular operations, and so they were naturally the right choice for the adaptively tuned parameters for this demonstration.

In this work, we simulate beam dynamics according to the Kapchinsky–Vladimirsky (K-V) equations [48] which describe the evolution of the rms beam sizes $\left(X\right(z),Y(z\left)\right)$ as

$$\begin{array}{ccc}\hfill X^{″}\left(z\right)& =& \frac{{ϵ}_{rx}^2}{X^3\left(z\right)}+\frac{2K}{X\left(z\right)+Y\left(z\right)}-\mathrm{\Theta }\left(z\right)X\left(z\right),\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill Y^{″}\left(z\right)& =& \frac{{ϵ}_{ry}^2}{Y^3\left(z\right)}+\frac{2K}{X\left(z\right)+Y\left(z\right)}+\mathrm{\Theta }\left(z\right)Y\left(z\right),\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill \mathrm{\Theta }\left(z\right)& =& \sum _{n=1}^{22}{I}_n\left(z\right){\mathrm{\Theta }}_n,z\in [0,L=11.7\mathrm{m}]\hfill \end{array}$$

(28)

where K is the beam perveance, a measure of how space-charge dominated the beam is, ${ϵ}_{rx}$ and ${ϵ}_{ry}$ are beam emittance in the x and y axis respective, $I_n\left(z\right)$ are indicator functions which are non-zero at the locations of the quadrupole magnets and ${\mathrm{\Theta }}_n$ are the quarupole focusing strengths defined as

$${\mathrm{\Theta }}_n=\sqrt{\frac{|q{G}_n|}{mc\gamma \beta }},G_n=\frac{0.8\pi IN\nu }{r^2},\beta =v/c,\gamma =\frac{1}{\sqrt{1-\frac{v^2}{c^2}}}$$

(29)

where $G_n$ is the magnetic field of quadrupole n in Gauss/cm, r is the magnet pole-tip radius, N is the number of turns per pole, $\nu$ is the efficiency factor and I is the magnet current.

In designing an adaptive diagnostic, one must make a choice about what parameter measurements are accurate and which might drift with time. In this case, we assume that we can accurately measure magnet settings and create an adaptive ML model for this system which serves as a non-invasive diagnostic by mapping magnet settings to beam profiles along the accelerator section, this NN-based model will be referred to as $M_{\mathrm{\Theta }\to X,Y}$ and its output is two vectors of length 200 each, which represent estimates of $\left(X\right(z),Y(z\left)\right)$ over the $z\in [0,L=11.7\mathrm{m}]$ range with 58 cm longitudinal resolution.

A typical surrogate model approach would simply run many iterations of the simulation (26)–(28) to learn a mapping from quadrupole settings to beam size along the LEBT. However, in our case, we know that the beam’s initial size $\left(X\right(z=0),Y(z=0\left)\right)$ coming out of the source will slowly drift unpredictably with time according to some unknown time-varying functions:

$$X(z=0,t)=F_X\left(t\right),Y(z=0,t)=F_Y\left(t\right).$$

(30)

Such a drift is impossible to measure in real time during accelerator operations, it requires lengthy wire scanner-based measurements that interrupt operations. Taking this into account, although we do not expect to be able to measure $\left(X\right(z=0,t),Y(z=0,t\left)\right)$, we still design our adaptive diagnostic with estimates $(\widehat{X}(z=0,t),\widehat{Y}(z=0,t))$ to be used as inputs. These estimate inputs will give our ML model the flexibility to be adaptive, to track changes in real time. In order to update our estimates, we need one final part which is some measurement of the beam that can be compared to the ML-based prediction. In this case the beam size can be estimated in real time at the end of the LEBT, $\left(X\right(z=L,t),Y(z=L,t\left)\right)$ by looking at the amount of current lost on a pair of vertical and horizontal slits as a non-invasive real-time diagnostic. The current intercepted on these slits grows and decays in proportion to the beam size and is always present as typically only ∼80% transmission is achieved in the LEBT (this helps cut off particles which are far from the axis due to high divergence). For very low current beams that are very finely focused, it is possible that the loss measurements would be negligible requiring slit adjustment or the use of a small aperture. Using an aperture would result in a beam size measurement without detailed $(X,Y)$ knowledge. In this case it is expected that the adaptive solution for the initial conditions would no longer be unique.

Based on the beam measurements the approach is to compare the model’s prediction of the final horizontal and vertical beam sizes to their measurements and calculate an error

$$e\left(t\right)=|\widehat{X}(z=L,t)-X(z=L,t)|+|\widehat{Y}(z=L,t)-Y(z=L,t)|,$$

(31)

and use that error to adaptively tune the input beam size estimates by using the methods in [41,42] to track the time-varying input beam distribution:

$$\frac{d\widehat{X}(z=0,t)}{dt}=\sqrt{\alpha \omega }cos(\omega t+ke\left(t\right)),\frac{d\widehat{Y}(z=0,t)}{dt}=\sqrt{\alpha \omega }sin(\omega t+ke\left(t\right)).$$

(32)

The adaptive tuning approach (32) is chosen because of its ability to optimize analytically unknown output functions of unknown time-varying dynamic systems with many coupled parameters, based only on noisy measurements of the unknown functions. As demonstrated below this method is able to track the unknown initial conditions very accurately and quickly in real-time based only on a measurement of the difference between the measured and predicted final beam sizes as quantified by (31). A high level overview of this adpative ML diagnostic approach is shown in Figure 2, where the NN $M_{\mathrm{\Theta }\to X,Y}$ has dense connected layers with relu activation functions.

To train the model we simulated the dynamics (26)–(28) 600 thousand times and used 590 thousand of the generated trajectories and their corresponding quadrupole magnet values for training, reserving 10 thousand for model validation. Each simulation generated random initial conditions $\left(X\right(0),Y(0)$ by uniformly sampling distributions with mean values of 2 mm and 1.75 mm, respectively, over a range of ±0.5 mm, which is realistic based on experimental data. The quadrupole magnet values were also sampled from uniform distributions centered on a known good setup over a range of ±1.

Considering the root-mean-square error (RMSE) as the metric of accuracy for both the X and Y trajectories,

$${\mathrm{RMSE}}_x=\sqrt{\sum _{n=1}^{200}\frac{{\left(\widehat{X}\left(n\right)-X\left(n\right)\right)}^2}{200}},{\mathrm{RMSE}}_y=\sqrt{\sum _{n=1}^{200}\frac{{\left(\widehat{Y}\left(n\right)-Y\left(n\right)\right)}^2}{200}},$$

(33)

a histogram of the sum of these errors is shown in Figure 3 for the 10 k validation set as well as the worst and best predictions from this set. In Figure 3, we also show the sum of these errors for the validation set if the input beam sizes to the NN are simply set as the average values of 2 mm and 1.75 mm. Figure 4 shows the beam envelopes for the best, average, and worst predictions of the NN when using the correct initial conditions as input.

In Figure 5, we show the results of using this adaptive ML-based diagnostic to track time varying initial beam sizes and compare the error (31) with and without the adaptive tracking. We also display the actual and tracked beam envelopes showing that the adaptive ML approach maintains good overlap throughout the tuning process.

By a similar principal, it is possible to design a second model which can be used to guide feedback control for accelerator tuning, by solving the inverse problem of mapping beam sizes to the quadrupole values that are required to achieve them, referred to as $M_{X,Y\to \mathrm{\Theta }}$. A high level view of such an adaptive control approach is shown in Figure 6. Such a model would require a diagnostic for comparing achieved and desired beam profiles and this type of approach was first demonstrated on the LCLS for automatic control of the LPS whose measurement was available in real time via a transverse deflecting cavity (TCAV) [32].

5. Conclusions

We have shown how combining adaptive model-independent methods with machine learning tools for adaptive ML combines the robustness of adaptive methods to time variation and uncertainty, with the global and extremely fast predictions of complex nonlinear dynamics. The main trick is to introduce extra inputs which may be tuned to modify the ML structure’s predictions based on some measurement. These inputs do not have to be physically measurable quantities, but they should have a physically meaningful effect on the predictions.