Singularly Perturbed Forward-Backward Stochastic Differential Equations: Application to the Optimal Control of Bilinear Systems

Abstract

We study linear-quadratic stochastic optimal control problems with bilinear state dependence where the underlying stochastic differential equation (SDE) has multiscale features. We show that, in the same way in which the underlying dynamics can be well approximated by a reduced-order dynamics in the scale separation limit (using classical homogenization results), the associated optimal expected cost converges to an effective optimal cost in the scale separation limit. This entails that we can approximate the stochastic optimal control for the whole system by a reduced-order stochastic optimal control, which is easier to compute because of the lower dimensionality of the problem. The approach uses an equivalent formulation of the Hamilton-Jacobi-Bellman (HJB) equation, in terms of forward-backward SDEs (FBSDEs). We exploit the efficient solvability of FBSDEs via a least squares Monte Carlo algorithm and show its applicability by a suitable numerical example.

1. Introduction

Stochastic optimal control is one of the important fields in mathematics which has attracted the attention of both pure and applied mathematicians [1,2]. Stochastic control problems also appear in a variety of applications, such as statistics [3,4], financial mathematics [5,6], molecular dynamics [7,8] or materials science [9,10], to mention just a few. For some applications in science and engineering, such as molecular dynamics [8,11], the high dimensionality of the state space is an important challenge when solving optimal control problems. Another issue when solving optimal control problems by discretising the corresponding dynamic programming equations in space and time are multiscale effects that come into play when the state space dynamics exhibits slow and fast motions.

Here we consider such systems that have slow and fast scales and that are possibly high-dimensional. Several techniques have been developed to reduce the spatial dimension of control systems (see e.g., [12,13] and the references therein), but these techniques treat the control as a possibly time-dependent parameter (“open loop control”) and do not take into account that the control may be a feedback control that depends on the state variables (“closed loop control”). Needless to say that homogenisation techniques for stochastic control systems have been extensively studied by applied analysts using a variety of different mathematical tools, including viscosity solutions of the Hamilton-Jacobi-Bellman equation [14,15], backward stochastic differential equations [16,17], or occupation measures [18,19]. However the convergence analysis of multiscale stochastic control systems is quite involved and non-constructive, in that the limiting equations of motion are not given in explicit or closed form, which makes these results of limited practical use; see [20,21] for notable exceptions, dealing mainly with the case when the dynamics is linear.

In general, the elimination of variables and solving control problems do not commute, so one of the key questions in control engineering is under which conditions it is possible to eliminate variables before solving an optimal problem. We call this the model reduction problem. In this paper, we identify a class of stochastic feedback control problems with bilinear state dependence that have the property that they admit the elimination of variables (i.e., model reduction) before solving the control problem. These systems turn out to be relevant in the control of high-dimensional transport PDEs, such as Fokker-Planck equations or the evolution equations of open quantum systems [22,23]. The possibility of applying model reduction before solving the corresponding optimal control problem means that it is possible to treat the control in the original equation simply as a parameter. This is in accordance with the general model reduction strategy in control engineering that is motivated by the fact that solving a dimension-reduced control problem, rather than the original one, is numerically much less demanding; see e.g., [12,13] and the references therein. We will show that this strategy, under certain assumptions, yields a good approximation of the high-dimensional optimal control, which implies that the reduced-order optimal control can be used to control the full systems dynamics almost optimally.

Our approach is based on a Donsker-Varadhan type duality principle between a linear Feynman-Kac PDE and the semi-linear dynamic programming PDE associated with a stochastic control problem [24]. Here we exploit the fact that the dynamic programming PDE can be recast as an uncoupled forward-backward stochastic differential equation (see e.g., [25,26]) that can be treated by model reduction techniques, such as averaging or homogenization. The relation between semilinear PDEs of Hamilton-Jacobi-Bellman type and forward-backward stochastic differential equations (FBSDE) is a classical subject that has been first studied by Pardoux and Peng [27] and since then received lot of attention from various sides, e.g., [28,29,30,31,32,33]. The solution theory for FBSDEs has its roots in the work of Antonelli [34] and has been extended in various directions since then; see e.g., [35,36,37,38]. From a theoretical point of view, this paper goes beyond our previous works [24,39] in that we prove strong convergence of the value function and the control without relying on compactness or periodicity assumptions for the fast variables, even though we focus on bilinear systems only, which is the weakest form of nonlinearity. (Many nonlinear systems however can be represented as bilinear systems by a so-called Carleman linearisation.) It also goes beyond the classical works [20,21] that treat systems that are either fully linear or linear in the fast variables. We stress that we are mainly aiming at the model reduction problem, but we discuss alongside with the theoretical results some ideas to discretise the corresponding FBSDE [40,41,42,43,44], since one of the main motivations for doing model reduction is to reduce the numerical complexity of solving optimal control problems.

1.1. Set-Up and Problem Statement

We briefly discuss the technical set-up of the control problem considered in this paper. In this work, we consider the linear-quadratic (LQ) stochastic control problem of the following form: minimise the expected cost

$$J(u;t,x)=\mathbf{E}\left[{\int }_t^{\tau }\left(q_0\left(X_s^u\right)+{\left|u_s\right|}^2\right)ds+q_1\left(X_{\tau }^u\right)|X_t^u=x\right]$$

(1)

over all admissible controls $u\in \mathcal{U}$ and subject to:

$$d{X}_s^u=\left(a\left(X_s^u\right)+b\left(X_s^u\right)u_s\right)ds+\sigma \left(X_s^u\right)d{W}_s,0⩽t⩽s⩽\tau .$$

(2)

Here $\tau <\infty$ is a bounded stopping time (specified below), and the set of admissible controls $\mathcal{U}$ is chosen such that (2) has a unique strong solution. The denomination linear-quadratic for (1)–(2) is due to the specific dependence of the system on the control variable u. The state vector $x\in {\mathbb{R}}^n$ is assumed to be high-dimensional, which is why we seek a low-dimensional approximation of (1)–(2).

Specifically, we consider the case that $q_0$ and $q_1$ are quadratic in x, a is linear and $\sigma$ is constant, and the control term is an affine function of x, i.e.,

$$b\left(x\right)u=\left(Nx+B\right)u.$$

In this case, the system is called bilinear (including linear systems as a special case), and the aim is to replace (2) by a lower dimensional bilinear system

$$d{\overline{X}}_s^v=\overline{A}{\overline{X}}_s^{v}ds+\left(\overline{N}{\overline{X}}_s^v+\overline{B}\right)v_{s}ds+\overline{C}d{W}_s,0⩽t⩽s⩽\tau ,$$

with states $\overline{x}\in {\mathbb{R}}^{n_s}$, $n_s\ll n$ and an associated reduced cost functional

$$\overline{J}(v;\overline{x},t)=\mathbf{E}\left[{\int }_t^{\tau }\left({\overline{q}}_0\left({\overline{X}}_s^v\right)+{\left|v_s\right|}^2\right)ds+{\overline{q}}_1\left({\overline{X}}_{\tau }^v\right)|{\overline{X}}_t^v=\overline{x}\right],$$

that is solved instead of (1)–(2). Letting $v^{*}$ denote the minimizer of $\overline{J}$, we require that $v^{*}$ is a good approximation of the minimiser $u^{*}$ of the original problem where “good approximation” is understood in the sense that

$$J(v^{*};·,·)\approx J(u^{*};·,·).$$

It is a priori not clear how the symbol “≈” in the last equation must be interpreted, e.g., pointwise for all initial data $(x,t)\in {\mathbb{R}}^n\times [0,T)$ for some $T<\infty$, or uniformly on all compact subsets of ${\mathbb{R}}^n\times [0,T)$.

One situation in which the above approximation property holds is when $u^{*}\approx v^{*}$ uniformly in t and the cost is continuous in the control, but it turns out that this requirement will be too strong in general and overly restrictive. We will discuss alternative criteria in the course of this paper.

1.2. Outline

The paper is organised as follows: In Section 2 we introduce the bilinear stochastic control problem studied in this paper and derive the corresponding forward-backward stochastic differential equation (FBSDE). Section 3 contains the main result, a convergence result for the value function of a singularly perturbed control problem with bilinear state dependence, based on a FBSDE formulation. In Section 4 we present a numerical example to illustrate the theoretical findings and discuss the numerical discretisation of the FBSDE. The article concludes in Section 5 with a short summary and a discussion of future work. The proof of the main result and some technical lemmas are recorded in the Appendix A.

2. Singularly Perturbed Bilinear Control Systems

We now specify the system dynamics (2) and the corresponding cost functional (1). Let $(x_1,x_2)\in {\mathbb{R}}^{n_s}\times {\mathbb{R}}^{n_f}$ with $n_s+n_f=n$ denote a decomposition of the state vector $x\in {\mathbb{R}}^n$ into relevant (slow) and irrelevant (fast) components. Further let $W={\left(W_t\right)}_{t\ge 0}$ denote a ${\mathbb{R}}^m$-valued Brownian motion on a probability space $(\mathrm{\Omega },\mathcal{F},P)$ that is endowed with the filtration ${\left({\mathcal{F}}_t\right)}_{t\ge 0}$ generated by W. For any initial condition $x\in {\mathbb{R}}^n$ and any $\mathcal{A}$-valued admissible control $u\in \mathcal{U}$, with $\mathcal{A}\subset \mathbb{R}$, we consider the following system of Itô stochastic differential equations

$$d{X}_s^{ϵ}=A{X}_s^{ϵ}ds+(N{X}_s^{ϵ}+B)u_{s}ds+Cd{W}_s,X_t^{ϵ}=x,$$

(3)

that depends parametrically on a parameter $ϵ>0$ via the coefficients

$$A=A^{ϵ}\in {\mathbb{R}}^{n\times n},N=N^{ϵ}\in {\mathbb{R}}^{n\times n},B=B^{ϵ}\in {\mathbb{R}}^n,\mathrm{and}C=C^{ϵ}\in {\mathbb{R}}^{n\times m},$$

where for brevity we also drop the dependence of the process on the control u, i.e., $X_s^{ϵ}=X_s^{u,ϵ}$. The stiffness matrix A in (3) is assumed to be of the form

$$A=\left(\begin{array}{cc}\begin{array}{c}{A}_{11}\end{array}& \begin{array}{c}{ϵ}^{-1/2}{A}_{12}\end{array}\\ {ϵ}^{-1/2}{A}_{21}& {ϵ}^{-1}{A}_{22}\end{array}\right)\in {\mathbb{R}}^{(n_s+n_f)\times (n_s+n_f)},$$

(4)

with $n=n_s+n_f$. Control and noise coefficients are given by

$$N=\left(\begin{array}{cc}\begin{array}{c}{N}_{11}\end{array}& \begin{array}{c}{N}_{12}\end{array}\\ {ϵ}^{-1/2}{N}_{21}& {ϵ}^{-1/2}{N}_{22}\end{array}\right)\in {\mathbb{R}}^{(n_s+n_f)\times (n_s+n_f)}$$

(5)

and

$$B=\left(\begin{array}{c}{B}_1\\ {ϵ}^{-1/2}{B}_2\end{array}\right)\in {\mathbb{R}}^{(n_s+n_f)\times 1},C=\left(\begin{array}{c}{C}_1\\ {ϵ}^{-1/2}{C}_2\end{array}\right)\in {\mathbb{R}}^{(n_s+n_f)\times m},$$

(6)

where $Nx+B\in \mathrm{range}\left(C\right)$ for all $x\in {\mathbb{R}}^n$; often we will consider either the case $m=1$ with $C_i=\sqrt{\rho }{B}_i$, $\rho >0$, or $m=n$, with C being a multiple of the identity when $ϵ=1$. All block matrices $A_{ij},N_{ij}$, $B_i$ and $C_j$ are assumed to be order 1 and independent of $ϵ$.

The above $ϵ$-scaling of the coefficients is natural for a system with $n_s$ slow and $n_f$ fast degrees of freedom and arises, for example, as a result of a balancing transformation applied to a large-scale system of equations; see e.g., [22,45]. A special case of (3) is the linear system

$$d{X}_s^{ϵ}=\left(A{X}_s^{ϵ}+B{u}_s\right)ds+Cd{W}_s.$$

(7)

Our goal is to control the stochastic dynamics (3)—or (7) as a special variant—so that a given cost criterion is optimised. Specifically, given two symmetric positive semidefinite matrices $Q_0,Q_1\in {\mathbb{R}}^{n_s\times n_s}$, we consider the quadratic cost functional

$$J(u;t,x)=\mathbf{E}\left[\frac{1}{2}{\int }_t^{\tau }({\left(X_{1,s}^{ϵ}\right)}^{\top }{Q}_0^{}{X}_{1,s}^{ϵ}+|u_s{|}^2)ds+\frac{1}{2}{\left(X_{1,\tau }^{ϵ}\right)}^{\top }{Q}_1^{}{X}_{1,\tau }^{ϵ}\right],$$

(8)

that we seek to minimise subject to the dynamics (3). Here the expectation is understood as an expectation over all realisations of ${\left(X_s^{ϵ}\right)}_{s\in [t,\tau ]}$ starting at $X_t^{ϵ}=x$, and as a consequence J is a function of the initial data $(t,x)$. The stopping time is defined as the minimum of some time $T<\infty$ and the first exit time of a domain $D=D_s\times {\mathbb{R}}^{n_f}\subset {\mathbb{R}}^{n_s}\times {\mathbb{R}}^{n_f}$ where $D_s$ is an open and bounded set with smooth boundary. Specifically, we set $\tau =min\{{\tau }_D,T\}$, with

$${\tau }_D=inf\{r\ge t:X_r^{ϵ}\notin D\}.$$

In other words, $\tau$ is the stopping time that is defined by the event that either $r=T$ or $X_r^{ϵ}$ leaves the set $D=D_s\times {\mathbb{R}}^{n_f}$, whichever comes first. Please note that the cost function does not explicitly depend on the fast variables $x_2$. We define the corresponding value function by

$$V^{ϵ}(t,x)=\underset{u\in \mathcal{U}}{inf}J(u;t,x).$$

(9)

Remark 1.

As a consequence of the boundedness of $D_s\subset {\mathbb{R}}^{n_s}$, we may assume that all coefficients in our control problem are bounded or Lipschitz continuous, which makes some of the proofs in the paper more transparent.

We further note that all of the following considerations trivially carry over to the case $N=0$ and a multi-dimensional control variable, i.e., $u\in {\mathbb{R}}^k$ and $B\in {\mathbb{R}}^{n\times k}$.

From Stochastic Control to Forward-Backward Stochastic Differential Equations

We suppose that the matrix pair $(A,C)$ satisfies the Kalman rank condition

$$\mathrm{rank}\left(C\right|AC\left|A^{2}C\right|\dots \left|A^{n-1}C\right)=n.$$

(10)

A necessary—and in this case sufficient—condition for optimality of our control problem is that the value function (9) solves a semilinear parabolic partial differential equation of Hamilton-Jacobi-Bellman type (a.k.a. dynamic programming equation) [46]

$$-\frac{\partial V^{ϵ}}{\partial t}=L^{ϵ}{V}^{ϵ}+f(x,V^{ϵ},C^{\top }\nabla V^{ϵ}),V^{ϵ}{|}_{E^{+}}=q_1,$$

(11)

where

$$q_1\left(x\right)=\frac{1}{2}{x}_1^{\top }{Q}_1{x}_1^{}$$

and $E^{+}$ is the terminal set of the augmented process $(s,X_s^{ϵ})$, precisely $E^{+}=\left([0,T)\times \partial D\right)\cup \left(\left\{T\right\}\times D\right)$. Here $L^{ϵ}$ is the infinitesimal generator of the control-free process,

$$L^{ϵ}=\frac{1}{2}C{C}^{\top }:{\nabla }^2+\left(Ax\right)·\nabla ,$$

(12)

and the nonlinearity f is given by

$$f(x,y,z)=\frac{1}{2}{x}_1^{\top }{Q}_0{x}_1^{}-\frac{1}{2}{\left|\left(x^{\top }{N}^{\top }+B^{\top }\right){\left(C^{\top }\right)}^{♯}z\right|}^2.$$

(13)

Please note that f is independent of y where ${\left(C^{\top }\right)}^{♯}$ denotes the Moore-Penrose pseudoinverse that is is unambiguously defined since $z=C^{\top }\nabla V^{ϵ}$ and $(Nx+B)\in \mathrm{range}\left(C\right)$, which by noting that ${\left(C^{\top }\right)}^{♯}{C}^{\top }$ is the orthogonal projection onto $\mathrm{range}\left(C\right)$ implies that

$$|(x^{\top }{N}^{\top }+B^{\top })\nabla V^{ϵ}{|}^2=|(x^{\top }{N}^{\top }+B^{\top }){\left(C^{\top }\right)}^{♯}z{|}^2.$$

The specific semilinear form of the equation is a consequence of the control problem being linear-quadratic. As a consequence, the dynamic programming Equation (11) admits a representation in form of an uncoupled forward-backward stochastic differential equation (FBSDE). To appreciate this point, consider the control-free process $X_s^{ϵ}=X_s^{ϵ,u=0}$ with infinitesimal generator $L^{ϵ}$ and define an adapted process $Y_s^{ϵ}=Y_s^{ϵ,x,t}$ by

$$Y_s^{ϵ}=V^{ϵ}(s,X_s^{ϵ}).$$

(14)

(We abuse notation and denote both the controlled and the uncontrolled process by $X_s^{ϵ}$.) Then, by definition, $Y_t^{ϵ}=V^{ϵ}(t,x)$. Moreover, by Itô’s formula and the dynamic programming Equation (11), the pair ${(X_s^{ϵ},Y_s^{ϵ})}_{s\in [t,\tau ]}$ can be shown to solve the system of equations

$$\begin{array}{cc}\hfill d{X}_s^{ϵ}& =A{X}_s^{ϵ}ds+Cd{W}_s,X_t^{ϵ}=x\hfill \\ \hfill d{Y}_s^{ϵ}& =-f(X_s^{ϵ},Y_s^{ϵ},Z_s^{ϵ})ds+Z_s^{ϵ}d{W}_s,Y_{\tau }^{ϵ}=q_1\left(X_{\tau }^{ϵ}\right),\hfill \end{array}$$

(15)

with $Z_s^{ϵ}=C^{\top }\nabla V^{ϵ}(s,X_s^{ϵ})$ being the control variable. Here, the second equation is only meaningful if interpreted as a backward equation, since only in this case $Z_s^{ϵ}$ is uniquely defined. To see this, let $f=0$ and $q_1\left(x\right)=x$ and note that the ansatz (14) implies that $Y_s^{ϵ}$ is adapted to the filtration generated by the forward process $X_s^{ϵ}$. If the second equation was just a time-reversed SDE then $(Y_s^{ϵ},Z_s^{ϵ})\equiv (X_{\tau }^{ϵ},0)$ would be the unique solution to the SDE $d{Y}_s^{ϵ}=Z_s^{ϵ}d{W}_s$ with terminal condition $Y_{\tau }^{ϵ}=X_{\tau }^{ϵ}$. However, such a solution would not be adapted, because $Y_s^{ϵ}$ for $s<\tau$ would depend on the future value $X_{\tau }^{ϵ}$ of the forward process.

Remark 2.

Equation (15) is called an uncoupled FBSDE because the forward equation for $X_s^{ϵ}$ is independent of $Y_s^{ϵ}$ or $Z_s^{ϵ}$. The fact that the FBSDE is uncoupled furnishes a well-known duality relation between the value function of an LQ optimal control problem and the cumulate generating function of the cost [47,48]; specifically, in the case that $N=0$, $B=C$ and the pair $(A,B)$ being completely controllable, it holds that

$$V^{ϵ}(t,x)=-log\mathbf{E}\left[exp\left(-{\int }_t^{\tau }{q}_0\left(X_s^{ϵ}\right)ds-q_1\left(X_{\tau }^{ϵ}\right)\right)\right],$$

(16)

with

$$q_0\left(x\right)=\frac{1}{2}{x}_1^{\top }{Q}_0{x}_1^{}.$$

Here the expectation on the right hand side is taken over all realisations of the control-free process $X_s^{ϵ}=X_s^{ϵ,u=0}$, starting at $X_t^{ϵ}=x$. By the Feynman-Kac theorem, the function ${\psi }^{ϵ}=exp(-V^{ϵ})$ solves the linear parabolic boundary value problem

$$\left(\frac{\partial }{\partial t}+L^{ϵ}\right){\psi }^{ϵ}=q_0\left(x\right){\psi }^{ϵ},{\psi }^{ϵ}{|}_{E^{+}}=exp\left(-q_1\right),$$

(17)

which is equivalent to the corresponding dynamic programming Equation (11).

3. Model Reduction

The idea is to exploit the fact that (15) is uncoupled, which allows us to derive an FBSDE for the slow variables ${\overline{X}}_s^{ϵ}=X_{1,s}^{ϵ}$ only, by standard singular perturbation methods. The reduced FBSDE as $ϵ\to 0$ will then be of the form

$$\begin{array}{cc}\hfill d{\overline{X}}_s& =\overline{A}{\overline{X}}_{s}ds+\overline{C}d{W}_s,{\overline{X}}_t=x_1\hfill \\ \hfill d{\overline{Y}}_s& =-\overline{f}({\overline{X}}_s,{\overline{Y}}_s,{\overline{Z}}_s)ds+{\overline{Z}}_{s}d{W}_s,{\overline{Y}}_{\tau }={\overline{q}}_1\left({\overline{X}}_{\tau }\right),\hfill \end{array}$$

(18)

where the limiting form of the backward SDE follows from the corresponding properties of the forward SDE. Specifically, assuming that the solution of the associated SDE

$$d{\xi }_s=A_{22}{\xi }_{s}ds+C_{2}d{W}_s,$$

(19)

that is governing the fast dynamics as $ϵ\to 0$, is ergodic with unique Gaussian invariant measure $\pi =\mathcal{N}(0,\mathrm{\Sigma })$, where $\mathrm{\Sigma }={\mathrm{\Sigma }}^{\top }>0$ is the unique solution to the Lyapunov equation

$$A_{22}\mathrm{\Sigma }+\mathrm{\Sigma }{A}_{22}^{\top }=-C_2{C}_2^{\top },$$

(20)

we obtain that, asymptotically as $ϵ\to 0$,

$$X_{2,s}^{ϵ}\sim {\xi }_{s/ϵ},s>0.$$

(21)

As a consequence, the limiting SDE governing the evolution of the slow process $X_{1,s}^{ϵ}$—in other words: the forward part of (18)—has the coefficients

$$\overline{A}=A_{11}^{}-A_{12}^{}{A}_{22}^{-1}{A}_{21}^{},\overline{C}=C_1-A_{12}^{}{A}_{22}^{-1}{C}_2^{},$$

(22)

which follows from standard homogenisation arguments [49]; a formal derivation is given in the Appendix A. By a similar reasoning we find that the driver of the limiting backward SDE reads

$$\overline{f}(x_1,y,z_1)=\underset{{\mathbb{R}}^{n_f}}{\int }f((x_1,x_2),y,(z_1,0))\pi \left(d{x}_2\right),$$

(23)

specifically,

$$\overline{f}(x_1,y,z_1)=\frac{1}{2}{x}_1^{\top }{\overline{Q}}_0^{}{x}_1^{}-\frac{1}{2}|\left(x_1^{\top }{\overline{N}}_{}^{\top }+{\overline{B}}_{}^{\top }\right)z_1^{}{|}^2+K_0^{},$$

(24)

with

$${\overline{Q}}_0^{}=Q_0,\overline{N}=C_1^{♯}{N}_{11}^{},\overline{B}=C_1^{♯}\left(B_1^{}+N_{12}^{}{\mathrm{\Sigma }}_{}^{1/2}\right).$$

(25)

The limiting backward SDE is equipped with a terminal condition ${\overline{q}}_1$ that equals $q_1$, namely,

$${\overline{q}}_1\left(x_1\right)=\frac{1}{2}{x}_1^{\top }{Q}_1{x}_1^{}.$$

(26)

3.1. Interpretation as an Optimal Control Problem

It is possible to interpret the reduced FBSDE again as the probabilistic version of a dynamic programming equation. To this end, note that (10) implies that the matrix pair $(\overline{A},\overline{C})$ satisfies the Kalman rank condition [50]

$$\mathrm{rank}(\overline{C}|A\overline{C}|A^2\overline{C}\left|\dots \right|A^{n_s-1}\overline{C})=n_s.$$

As a consequence, the semilinear partial differential equation

$$-\frac{\partial V}{\partial t}=\overline{L}V+\overline{f}(x_1,V,{\overline{C}}^{\top }\nabla V){,V|}_{E_s^{+}}={\overline{q}}_1,$$

(27)

with $E_s^{+}=\left([0,T)\times \partial D_s\right)\cup \left(\left\{T\right\}\times D_s\right)$ and

$$\overline{L}=\frac{1}{2}\overline{C}{\overline{C}}^{\top }:{\nabla }^2+\left(\overline{A}{x}_1\right)·\nabla$$

(28)

has a classical solution $V\in C^{1,2}([0,T)\times D)\cap C^{0,1}\left(E_s^{+}\right)$. Letting ${\overline{Y}}_s:=V(s,{\overline{X}}_s)$, $0⩽t⩽s⩽\tau$, with initial data ${\overline{X}}_t=x_1$ and ${\overline{Z}}_s={\overline{C}}^{\top }\nabla V(s,{\overline{X}}_s)$, the limiting FBSDE (18) can be readily seen to be equivalent to (27). The latter is the dynamic programming equation of the following LQ optimal control problem: minimise the cost functional

$$\overline{J}(v;t,x_1)=\mathbf{E}\left[\frac{1}{2}{\int }_t^{\tau }({\overline{X}}_s^{\top }{\overline{Q}}_0^{}{\overline{X}}_s^{}+|v_s{|}^2)ds+\frac{1}{2}{\overline{X}}_{\tau }^{\top }{\overline{Q}}_1^{}{\overline{X}}_{\tau }^{}\right],$$

(29)

subject to

$$d{\overline{X}}_s=\overline{A}{\overline{X}}_{s}ds+\left(\overline{M}{\overline{X}}_s+\overline{D}\right)v_{s}ds+\overline{C}d{w}_s,{\overline{X}}_t=x_1,$$

(30)

where ${\left(w_s\right)}_{s\ge 0}$ denotes standard Brownian motion in ${\mathbb{R}}^{n_s}$ and we have introduced the new control coefficients $\overline{M}=\overline{C}\overline{N}$ and $\overline{D}=\overline{C}\overline{B}$.

3.2. Convergence of the Control Value

Before we state our main result and discuss its implications for the model reduction of linear and bilinear systems, we recall the basic assumptions that we impose on the system dynamics. Specifically, we say that the dynamics (3) and the corresponding cost functional (8) satisfy Condition LQ if the following holds:

• $(A,C)$ is controllable, and the range of $b\left(x\right)=Nx+B$ is a subspace of $\mathrm{range}\left(C\right)$.
• The matrix $A_{22}$ is Hurwitz (i.e., its spectrum lies entirely in the open left complex half-plane) and the matrix pair $(A_{22},C_2)$ is controllable.
• The driver of the FBSDE (15) is continuous and quadratically growing in Z.
• The terminal condition in (15) is bounded; for simplicity we set $Q_1=0$ in (8).

Assumption 2 implies that the fast subsystem (19) has a unique Gaussian invariant measure $\pi =\mathcal{N}(0,\mathrm{\Sigma })$ with full topological support, i.e., we have $\mathrm{\Sigma }={\mathrm{\Sigma }}^{\top }>0$. According to ([51], Prop. 3.1) and [33], existence and uniqueness of (15) is guaranteed by Assumptions 3 and 4 and the controllability of $(A,C)$ and the range condition, which imply that the transition probability densities of the (controlled or uncontrolled) forward process $X_s^{ϵ}$ are smooth and strictly positive. As a consequence of the complete controllability of the original system, the reduced system (30) is completely controllable too, which guarantees existence and uniqueness of a classical solution of the limiting dynamic programming Equation (27); see, e.g., [52].

Uniform convergence of the value function $V^{ϵ}\to V$ is now entailed by the strong convergence of the solution to the corresponding FBSDE as is expressed by the following Theorem.

Theorem 1.

Let the assumptions of Condition LQ hold. Further let $V^{ϵ}$ be the classical solution of the dynamic programming Equation (11) and V be the solution of (27). Then

$$V^{ϵ}\to V,$$

uniformly on all compact subsets of $[0,T]\times D$.

The proof of the Theorem is given in Appendix A.2. For the reader’s convenience, we present a formal derivation of the limit equation in the next subsection.

3.3. Formal Derivation of the Limiting FBSDE

Our derivation of the limit FBSDE follows standard homogenisation arguments (see [49,53,54]), taking advantage of the fact that the FBSDE is uncoupled. To this end we consider the following linear evolution equation

$$\left(\frac{\partial }{\partial t}-L^{ϵ}\right){\varphi }^{ϵ}=0,{\varphi }^{ϵ}(x_1,x_2,0)=g\left(x_1\right)$$

(31)

for a function ${\varphi }^{ϵ}:{\overline{D}}_s\times {\mathbb{R}}^{n_f}\times [0,T]$ where

$$L^{ϵ}=\frac{1}{ϵ}{L}_0+\frac{1}{\sqrt{ϵ}}{L}_1+L_2,$$

(32)

with

$$\begin{array}{cc}\hfill L_0& =\frac{1}{2}{C}_2{C}_2^{\top }:{\nabla }_{x_2}^2+\left(A_{22}{x}_2\right)·{\nabla }_{x_2}\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill L_1& =\frac{1}{2}{C}_1{C}_2^{\top }:{\nabla }_{x_2{x}_1}^2+\frac{1}{2}{C}_2{C}_1^{\top }:{\nabla }_{x_1{x}_2}^2+\left(A_{12}{x}_2\right)·{\nabla }_{x_1}+\left(A_{21}{x}_1\right)·{\nabla }_{x_2}\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill L_2& =\frac{1}{2}{C}_1{C}_1^{\top }:{\nabla }_{x_1}^2+\left(A_{11}{x}_1\right)·{\nabla }_{x_1}\hfill \end{array}$$

(35)

is the generator associated with the control-free forward process $X_s^{ϵ}$ in (15). We follow the standard procedure of [49] and consider the perturbative expansion

$${\varphi }^{ϵ}={\varphi }_0+\sqrt{ϵ}{\varphi }_1+ϵ{\varphi }_2+\dots$$

that we insert into the Kolmogorov Equation (31). Equating different powers of $ϵ$ we find a hierarchy of equations, the first three of which read

$$L_0{\varphi }_0=0,L_0{\varphi }_1=-L_1{\varphi }_0,L_0{\varphi }_2=\frac{\partial {\varphi }_0}{\partial t}-L_1{\varphi }_1-L_2{\varphi }_0.$$

(36)

Assumption 2 on page 7 implies that $L_0$ has a one-dimensional nullspace that is spanned by functions that are constant in $x_2$, and thus the first of the three equations implies that ${\varphi }_0$ is independent of $x_2$. Hence the second equation—the cell problem—reads

$$L_0{\varphi }_1=-\left(A_{12}{x}_2\right)·\nabla {\varphi }_0(x_1,t).$$

(37)

The last equation has a solution by the Fredholm alternative, since the right hand side averages to zero under the invariant measure $\pi$ of the fast dynamics that is generated by the operator $L_0$, in other words, the right hand side of the linear equation is orthogonal to the nullspace of $L_0^{*}$ spanned by the density of $\pi$. Here $L_0^{*}$ is the formal $L^2$ adjoint of the operator $L_0$, defined on a suitable dense subspace of $L^2$. The form of the equation suggests the general ansatz

$${\varphi }_1=\psi \left(x_2\right)·\nabla {\varphi }_0(x_1,t)+R(x_1,t)$$

where the function R plays no role in what follows, so we set it equal to zero. Since $L_0\psi =-{\left(A_{12}{x}_2\right)}^{\top }$, the function $\psi$ must be of the form $\psi =Q{x}_2$ with a matrix $Q\in {\mathbb{R}}^{n_s\times n_f}$. Hence

$$Q=-A_{12}{A}_{22}^{-1}.$$

Now, solvability of the last of the three equations of (36) requires again that the right hand side averages to zero under $\pi$, i.e.,

$${\int }_{{\mathbb{R}}^{n_f}}\left(\frac{\partial \varphi }{\partial t}+L_1\left[\left(A_{12}{A}_{22}^{-1}{x}_2\right)·\nabla \varphi \right]-L_2\varphi \right)\pi \left(d{x}_2\right)=0,$$

(38)

which formally yields the limiting equation for $\varphi ={\varphi }_0(x_1,t)$. Since $\pi$ is a Gaussian measure with mean 0 and covariance $\mathrm{\Sigma }$ given by (20), the integral (38) can be explicitly computed:

$$\left(\frac{\partial }{\partial t}-\overline{L}\right)\varphi =0,\varphi (x_1,0)=g\left(x_1\right),$$

(39)

where $\overline{L}$ is given by (28) and the initial condition $\varphi (·,0)=g$ is a consequence of the fact that the initial condition in (31) is independent of $ϵ$. By the controllability of the pair $(\overline{A},\overline{C})$, the limiting Equation (39) has a unique classical solution and uniform convergence ${\varphi }^{ϵ}\to \varphi$ is guaranteed by standard results, e.g., ([49], Thm. 20.1).

Since the backward part of (15) is uniformly bounded in $ϵ$, the final form of the homogenised FBSDE (18) is found by averaging over $x_2$, with the unique solution of the corresponding backward SDE satisfying $Z_{2,s}=0$ as the averaged backward process is independent of $x_2$.

4. Numerical Studies

In this section, we present numerical results for linear and bilinear control systems and discuss the numerical discretisation of uncoupled FBSDE associated with LQ stochastic control problems. We begin with the latter.

4.1. Numerical FBSDE Discretisation

The fact that (15) or (18) are decoupled entails that they can be discretised by an explicit time-stepping algorithm. Here we utilise a variant of the least-squares Monte Carlo algorithm proposed in [41]; see also [55]. The convergence of numerical schemes for FBSDE with quadratic nonlinearities in the driver has been analysed in [56].

The least-squares Monte Carlo scheme is based on the Euler discretisation of (15):

$$\begin{array}{cc}\hfill {\widehat{X}}_{n+1}& ={\widehat{X}}_n+\Delta tA{\widehat{X}}_n+\sqrt{\Delta t}C{\xi }_{n+1}\hfill \\ \hfill {\widehat{Y}}_{n+1}& ={\widehat{Y}}_n-\Delta tf({\widehat{X}}_n,{\widehat{Y}}_n,{\widehat{Z}}_n)+\sqrt{\Delta t}{\widehat{Z}}_n·{\xi }_{n+1}\hfill \end{array}$$

(40)

where $({\widehat{X}}_n,{\widehat{Y}}_n)$ denotes the numerical discretisation of the joint process $(X_s^{ϵ},Y_s^{ϵ})$, where we set $X_s^{ϵ}=X_{{\tau }_D}^{ϵ}$ for $s\in ({\tau }_D,T]$ when ${\tau }_D<T$, and ${\left({\xi }_k\right)}_{k\ge 1}$ is an i.i.d. sequence of normalised Gaussian random variables. Now let

$${\mathcal{F}}_n=\sigma \left(\left\{{\widehat{W}}_k:0⩽k⩽n\right\}\right)$$

be the $\sigma$-algebra generated by the discrete Brownian motion ${\widehat{W}}_n:=\sqrt{\Delta t}{\sum }_{i⩽n}{\xi }_i$. By definition the joint process $(X_s^{ϵ},Y_s^{ϵ})$ is adapted to the filtration generated by ${\left(W_r\right)}_{0⩽r⩽s}$, therefore

$${\widehat{Y}}_n=\mathbf{E}\left[{\widehat{Y}}_n|{\mathcal{F}}_n\right]=\mathbf{E}\left[{\widehat{Y}}_{n+1}+\Delta tf({\widehat{X}}_n,{\widehat{Y}}_n,{\widehat{Z}}_n)|{\mathcal{F}}_n\right],$$

(41)

where we have used that ${\widehat{Z}}_n$ is independent of ${\xi }_{n+1}$. To compute ${\widehat{Y}}_n$ from ${\widehat{Y}}_{n+1}$ we use the identification of $Z_s^{ϵ}$ with $C^{\top }\nabla V^{ϵ}(s,X_s^{ϵ})$ and replace ${\widehat{Z}}_n$ in (41) by

$${\widehat{Z}}_n=C^{\top }\nabla V^{ϵ}(t_n,{\widehat{X}}_n),$$

(42)

and the parametric ansatz (44) for $V^{ϵ}$ makes the overall scheme explicit in ${\widehat{X}}_n$ and ${\widehat{Y}}_n$.

4.2. Least-Squares Solution of the Backward SDE

To evaluate the conditional expectation ${\widehat{Y}}_n=\mathbf{E}[·|{\mathcal{F}}_n]$ we recall that a conditional expectation can be characterised as the solution to the following quadratic minimisation problem:

$$\mathbf{E}\left[S|{\mathcal{F}}_n\right]=\underset{Y\in L^2,{\mathcal{F}}_n-\mathrm{measurable}}{\mathrm{argmin}}{\mathbf{E}\left[\right|Y-S|}^2].$$

Given M independent realisations ${\widehat{X}}_n^{\left(i\right)}$, $i=1,\dots ,M$ of the forward process ${\widehat{X}}_n$, this suggests the approximation scheme

$${\widehat{Y}}_n\approx \underset{Y=Y\left({\widehat{X}}_n\right)}{\mathrm{argmin}}\frac{1}{M}\sum _{i=1}^M|Y-{\widehat{Y}}_{n+1}^{\left(i\right)}-\Delta tf\left({\widehat{X}}_n^{\left(i\right)},{\widehat{Y}}_{n+1}^{\left(i\right)},C^{\top }\nabla {\widehat{Y}}_{n+1}^{\left(i\right)}\right){|}^2,$$

(43)

where ${\widehat{Y}}^{\left(i\right)}$ is defined by ${\widehat{Y}}^{\left(i\right)}=Y\left({\widehat{X}}^{\left(i\right)}\right)$ with terminal values

$${\widehat{Y}}_N^{\left(i\right)}=q_1^{}\left(X_N^{\left(i\right)}\right),\tau =N\Delta t.$$

(Please note that $N=N_D$ is random.) For simplicity, we assume in what follows that the terminal value is zero, i.e., we set $q_1=0$. (Recall that the existence and uniqueness result from [33] requires $q_1$ to be bounded.) To represent ${\widehat{Y}}_n$ as a function $Y\left({\widehat{X}}_n\right)$ we use the ansatz

$$Y\left({\widehat{X}}_n\right)=\sum _{k=1}^K{\alpha }_k\left(n\right){\phi }_k\left({\widehat{X}}_n\right),$$

(44)

with coefficients ${\alpha }_1(·),\dots ,{\alpha }_K(·)\in \mathbb{R}$ and suitable basis functions ${\phi }_1,\dots ,{\phi }_K:{\mathbb{R}}^n\to \mathbb{R}$ (e.g., Gaussians). Please note that the coefficients ${\alpha }_k$ are the unknowns in the least-squares problem (43) and thus are independent of the realisation. Now the least-squares problem that has to be solved in the n-th step of the backward iteration is of the form

$$\widehat{\alpha }\left(n\right)=\underset{\alpha \in {\mathbb{R}}^K}{\mathrm{argmin}}{∥A_n\alpha -b_n∥}^2,$$

(45)

with coefficients

$$A_n={\left({\phi }_k\left({\widehat{X}}_n^{\left(i\right)}\right)\right)}_{i=1,\dots ,M;k=1,\dots ,K}$$

(46)

and data

$$b_n={\left({\widehat{Y}}_{n+1}^{\left(i\right)}-\Delta tf\left({\widehat{X}}_n^{\left(i\right)},{\widehat{Y}}_{n+1}^{\left(i\right)},C^{\top }\nabla {\widehat{Y}}_{n+1}^{\left(i\right)}\right)\right)}_{i=1,\dots ,M}.$$

(47)

Assuming that the coefficient matrix $A_n\in {\mathbb{R}}^{M\times K}$, $K⩽M$ defined by (46) has maximum rank K, then the solution to the least-squares problem (45) is given by

$$\widehat{\alpha }\left(n\right)={\left(A_n^{\top }{A}_n^{}\right)}^{-1}{A}_n^{\top }{b}_n^{}.$$

(48)

The thus defined scheme is strongly convergent of order 1/2 as $\Delta t\to 0$ and $M,K\to \infty$ as has been analysed by [41]. Controlling the approximation quality for finite values $\Delta t,M,K$, however, requires a careful adjustment of the simulation parameters and appropriate basis functions, especially with regard to the condition number of the matrix $A_n$.

4.3. Numerical Example

Illustrating our theoretical findings of Theorem 1, we consider a linear system of form (7) where the matrices $A,B$ and C are given by

$$A=\left(\begin{array}{cc}\begin{array}{c}0\end{array}& \begin{array}{c}{ϵ}^{-1/2}{I}_{n\times n}\end{array}\\ -{ϵ}^{-1/2}{I}_{n\times n}& -\gamma {ϵ}^{-1}{I}_{n\times n}\end{array}\right)\in {\mathbb{R}}^{2n\times 2n},$$

and

$$B=C=\left(\begin{array}{c}0\\ \sigma {ϵ}^{-1/2}{I}_{n\times n}\end{array}\right)\in {\mathbb{R}}^{2n\times n}.$$

This is an instance of a controlled Langevin equation with friction and noise coefficients $\gamma ,\sigma >0$ which are assumed to fulfil the fluctuation-dissipation relation

$$2\gamma ={\sigma }^2.$$

In the example we let $\gamma =1/2$ and $\sigma =1$. The quadratic cost functional (8) is determined by the running cost via $Q_0=I_{n\times n}\in {\mathbb{R}}^{n\times n}$ and we apply no terminal cost, i.e., $Q_1=0.$

The associated effective equations are given by (29)–(30), where

$$\overline{A}=-{\gamma }^{-1}{I}_{n\times n},\overline{D}=\overline{C}=\sigma {\gamma }^{-1},\overline{M}=0,{\overline{Q}}_0=I_{n\times n},{\overline{Q}}_1=0\in {\mathbb{R}}^{n\times n}.$$

We apply the previously described FBSDE scheme (40), (44)–(48), which was shown to yield good results in [57], to both the full and the reduced system, and we choose $n=3$, i.e., the full system is six dimensional. To this end we choose the basis functions

$${\varphi }_{k,n}^{{\mu }_k,\delta }\left(x\right)=exp\left(-\frac{{({\mu }_k-x)}^2}{2\delta }\right)$$

where $\delta =0.1$ is fixed but ${\mu }_k={\mu }_k\left(n\right)$ changes in each timestep such that the basis follows the forward process. For this, we simulate K additional forward trajectories $X^{\left(k\right)},k=1,\dots ,K$ and set ${\mu }_k\left(n\right)=X_n^{\left(k\right)}$.

We choose the parameters for the numerics as follows. The number of basis functions K is given by $K=9$ for the reduced system and $K^{ϵ}=40$ for the full system. We choose these values because the maximally observed rank of the coefficient matrices $A_n$ defined in (46) is 9 for the reduced system and they should not be rank deficient. For the full system we could have used a greater values for K, but we want to keep the computational effort reasonable. Further, we choose $\Delta t=5\times {10}^{-5}$ , the final time $T=0.5$ and the number of realisations $M=400.$

We let the whole algorithm run five times and compute the distance between the value functions of the full and reduced systems

$$E\left(ϵ\right):=|V^{ϵ}(0,x)-V(0,x)|$$

for which convergence of order $1/2$ was found in the proof of Theorem 1. Indeed, we observe convergence of order $1/2$ in our numerical example as can be seen in Figure 1 where we depict the mean and standard deviation of $E\left(ϵ\right)$.

4.4. Discussion

We shall now discuss the implications of the above simple example when it comes to more complicated dynamical systems. As a general remark the results show that it is possible to to apply model reduction before solving the corresponding optimal control problem where the control variable in the original equation can simply be treated as a parameter. This is in accordance with the general model reduction strategy in control engineering; see e.g., [12,13] and the references therein. Our results not only guarantee convergence of the value function via convergence of $Y^{ϵ}$, but they also imply strong convergence of the optimal control, by the convergence of the control process $Z^{ϵ}$ in $L^2$. (See the Appendix A for details.) This means that in the case of a system with time scale separation, our result is highly valuable since we can resort to the reduced system for finding the optimal control which can then be applied to the full systems dynamics.

We stress that our results carry over to fully nonlinear stochastic control problems which have a similar LQ structure [24]. Clearly, for realistic (i.e., high-dimensional or nonlinear) systems the identification of a small parameter $ϵ$ remains challenging, and one has to resort to e.g., semi-empirical approaches, such as [58,59].

If the dynamics is linear, as is the case here, small parameters may be identified using system theoretic arguments based on balancing transformations (see, e.g., [22,45]). These approaches require that the dynamics is either linear or bilinear in the state variables, but the aforementioned duality for the quasi-linear dynamic programming equation can be used here as well in order to change the drift of the forward SDE from some nonlinear vector field b to a linear vector field, say, $b_0=Ax$. Assuming that the noise coefficient C is square and invertible and ignoring $ϵ$ and the boundary condition for the moment, it is easy to see that the dynamic programming PDE (11) can be recast as

$$\begin{array}{c}\hfill -\frac{\partial V^{ϵ}}{\partial t}=\tilde{L}V+\tilde{f}(x,V^{ϵ},C^{\top }{\nabla }_x{V}^{ϵ})=0.\end{array}$$

Here

$$\tilde{L}=\frac{1}{2}C{C}^{\top }+b\left(x\right)·\nabla$$

is the generator of a forward SDE with nonlinear drift b, and

$$\tilde{f}(x,y,z)=f(x,y,z)+C^{-1}(Ax-b\left(x\right))·z.$$

is the driver of the corresponding backward SDE. Even though the change of drift is somewhat arbitrary, it shows that by changing the driver in the backward SDE it is possible to reduce the control problem to one with linear drift that falls within the category that is considered in this paper, at the expense of having a possibly non-quadratic cost functional.

Remark 3.

Changing the drift may be advantageous in connection with the numerical FBSDE solver. In the martingale basis approach of Bender and Steiner [41], the authors have suggested to use basis functions that are defined as conditional expectations of certain linearly independent candidate functions over the forward process, which makes the basis functions martingales. Computing the martingale basis, however, comes with a large computational overhead, which is why the authors consider only cases in which the conditional expectations can be computed analytically. Changing the drift of the forward SDE may thus be used to simplify the forward dynamics so that its distribution becomes analytically tractable.

5. Conclusions and Outlook

We have given a proof of concept that model reduction methods for singularly perturbed bilinear control systems can be applied to the dynamics before solving the corresponding optimal control problem. The key idea here was to exploit the equivalence between the semi-linear dynamic programming PDE corresponding to our stochastic optimal control problem and a singularly perturbed forward-backward SDE which is decoupled. Using this equivalence, we could derive a reduced-order FBSDE which was then interpreted as the representation of a reduced-order stochastic control problem. We have proved uniform convergence of the corresponding value function and, as an auxiliary result, obtained a strong convergence result for the optimal control. As we have argued the latter implies that the optimal control computed from the reduced system can be used to control the original dynamics.

We have presented a numerical example to illustrate our findings and discussed the numerical discretisation of uncoupled FBSDEs, based on the computation of conditional expectations. For the latter, the choice of the basis functions played an essential role, and how to cleverly choose the ansatz functions, possibly exploiting that the forward SDE has an explicit solution (see e.g., [41]) is an important aspect that future research ought to address, especially with regard to high dimensional problems.

Another class of important problems not considered in this article are slow-fast systems with vanishing noise. The natural question here is how the limit equation depends on the order in which noise and time scale parameters go to zero. This question has important consequences for the associated deterministic control problem and its regularisation by noise. We leave this topic for future work.