On the Numerical Integration of the Fokker–Planck Equation Driven by a Mechanical Force and the Bismut–Elworthy–Li Formula

Abstract

Optimal control theory aims to find an optimal protocol to steer a system between assigned boundary conditions while minimizing a given cost functional in finite time. Equations arising from these types of problems are often non-linear and difficult to solve numerically. In this article, we describe numerical methods of integration for two partial differential equations that commonly arise in optimal control theory: the Fokker–Planck equation driven by a mechanical potential for which we use the Girsanov theorem; and the Hamilton–Jacobi–Bellman, or dynamic programming, equation for which we find the gradient of its solution using the Bismut–Elworthy–Li formula. The computation of the gradient is necessary to specify the optimal protocol. Finally, we give an example application of the numerical techniques to solving an optimal control problem without spacial discretization using machine learning.

1. Introduction: Optimal Control in Stochastic Thermodynamics and the Fokker–Planck Equation

Thermodynamic transitions at nanoscale occur in highly fluctuating environments. For instance, nanoscale bio-molecular machines operate within a power output range between ${10}^{-16} \mathrm{W}$ and ${10}^{-17} \mathrm{W}$ per molecule while experiencing random environmental buffeting of approximately ${10}^{-8} \mathrm{W}$ at room temperature [1]. Nanomachines experience topological randomness as their motion occurs in inherently non-smooth surroundings due to the fact that machine constituent dimensions are close to those of the atom [1]. The dynamics of nanosystems, therefore, need to be described in terms of stochastic [2] or, more generally, random differential equations [3]. Consequently, the laws of macroscopic thermodynamics are replaced by identities involving functions of indicators of the state of the system that are naturally expressed by stochastic processes. Addressing fundamental and technological questions of nanoscale physics has thus propelled interest in the field of stochastic thermodynamics over the last few years [4,5,6,7].

A class of important questions in stochastic thermodynamics revolves around finding efficient protocols that natural or artificial nanomachines adopt to perform useful work at nanoscale. Optimal control theory provides a natural mathematical formulation for this type of question [8]. For instance, the conversion of chemical energy into mechanical work typically implies steering the system probability distribution between two assigned values. Schrödinger bridge problems [9] (English translation in [10]) and their extensions, see, e.g., [11,12,13], depict this idea mathematically. In these types of problems, protocols optimizing a given functional of the stochastic process describing the state of a nanosystem are determined by solving a coupled Hamilton–Jacobi–Bellman equation [14] and Fokker–Planck equation. The solution of the Hamilton–Jacobi–Bellman equation determines the value of the optimal action (force) steering the dynamics at any instant of time in the control horizon. However, the boundary condition at the end of the control horizon is assigned on the system probability distribution. Solving the Fokker–Planck equation thus becomes necessary to fully determine the optimal protocols.

Possibly the most prominent physical application of such a setup is the derivation of a tight lower bound on the entropy production in classical stochastic thermodynamics [15]. Remarkably, when the system dynamics is modelled by Langevin–Smoluchowski (overdamped) dynamics, the problem maps into the Monge–Ampère–Kantorovich equations and becomes essentially integrable [16]. This allows one to extricate relevant quantitative information about molecular engine efficiency [17,18,19,20] and minimum heat release [21]. The situation is, however, more complicated for more realistic models of nanosystem evolution. Specifically, if we adopt an underdamped (Langevin–Kramers) model of the dynamics [22,23], then even the equation connecting the optimal mechanical potential to the value function solving the dynamic programming equation is not analytically integrable. In the Gaussian case, the solution of a Lyapunov equation [24] specifies the optimal mechanical potential [25]. In general, optimal control duality methods [26] and multiscale perturbative expansions [25] yield lower bounds on the entropy production and approximate expressions of optimal protocols. More detailed quantitative information calls for exact numerical methods. This is particularly challenging, as integration strategies must be adapted to take into account the boundary conditions at the end of the control horizon imposed on the system probability density. Hence, the development of accurate and scaleable methods for numerical integration of the Fokker–Planck equation becomes an essential element of optimization algorithms.

Traditional numerical methods from hydrodynamics, such as the pseudo-spectral method, see, e.g., [27], are certainly accurate, but require boundary conditions that are periodic in space, and may not suit problems in stochastic thermodynamics. An even more serious limitation is the exponentially fast increase in computational complexity with the degrees of freedom of the problem. Monte Carlo averages over Lagrangian particles paths, i.e., realizations of the solution of the stochastic differential equation associated with the Fokker–Planck, circumvent the curse of dimensionality; see [28,29] for example applications to classical and quantum physics. The drawback is, however, that these methods are best suited to computing expectation values of smooth indicators of the stochastic process. They lack accuracy when computing the probability density itself, as this involves averaging over Dirac distributions. These considerations motivate recent works [30,31], which use machine learning methods to construct solutions of the Fokker–Planck equation in the system’s state space. These approaches consider the associated probability flow equation [16,30,31], which use the score function (or gradient of the log probability density [32]) to turn the Fokker–Planck into a mass conservation equation. The score function can be parametrized by, for example, a neural network [31], and the probability density can be recovered through a deterministic transport map.

In this article, we propose a Monte Carlo method adapted to the numerical integration of Fokker–Planck equations of diffusion processes driven by a time-dependent mechanical force. Although mathematically non-generic, they are recurrent in applications of stochastic thermodynamics, as they describe the evolution of a system under a mechanical potential, which may vary in time because of a feedback. We encounter this type of equation in generalized Schrödinger bridge problems instantiating refinements of the second law of thermodynamics [15,21,25]. In this context, the Fokker–Planck equation describes the evolution of the optimal distribution of the state throughout the time interval. Integrating this directly offers a challenging problem, particularly when the driving mechanical potential is non-linear or for systems of high dimension. By using the well-known Girsanov change-of-measure formula [33], we couch the solution to the Fokker–Planck in terms of a numerical expectation that can be evaluated from sampled trajectories of the dynamics.

In addition, we also take a look at an application of the Bismut–Elworthy–Li formula [34,35,36] to compute the gradient of the solution to the Hamilton–Jacobi–Bellman equation. This equation determines the value function, which enforces the system dynamics over the control horizon, and is coupled to the Fokker–Planck through its boundary conditions. Direct access to the gradient of the value function is important, since the stationarity condition in control problems often links the optimal control protocol through the gradient of the value function. The Bismut–Elworthy–Li formula is commonly used in finance, for the calculation of the Greek derivatives [37,38,39]. It has also been used in the numerical integration of non-linear parabolic partial differential equations [40]. We apply the Bismut–Elworthy–Li formula in the underdamped, or degenerate, dynamics [36], alongside a numerical example.

Numerical approaches to the Schrödinger bridge are often iterative. For example, ref. [41] turns the problem into a pair of Fokker–Planck equations and iteratively integrates them to recompute the boundary conditions via a proximal operator-based numerical integration method [42]. Machine learning techniques have been used to iteratively solve half-bridge problems [43,44]. We bring together the described Monte Carlo methods for the Fokker–Planck and the Hamilton–Jacobi–Bellman equation in a prototype numerical example to solve a Schrödinger bridge minimizing the Kullback–Leibler divergence from a free diffusion. This is done by an iteration between updating the drift, parametrized by a neural network, with the stationarity condition, and updating the value function with the probability density. Using Monte Carlo integration allows us to compute the update steps quickly and without spacial discretization.

The next Sections are organized as follows. In Section 2, we leverage the Girsanov theorem to express the solution of a Fokker–Planck equation as an expectation which can be evaluated numerically. This is complemented by analytical and numerical examples. In Section 3, we extend this setup into the underdamped dynamics, with an accompanying numerical example from a stochastic optimal control model in the underdamped dynamics. Section 4 discusses the application of the Bismut–Elworthy–Li formula to non-degenerate dynamics, with an analytic example application in Section 4.3, and a numerical example in Section 4.4 of the Hamilton–Jacobi–Bellman equation coupled to the Fokker–Planck from Section 2.2.4. In Section 5, we extend the application of the Bismut–Elworthy–Li formula to the degenerate case. The formula is applied analytically in Section 5.2, and numerically in Section 5.3. Finally, we give an example use case for the derived formulae, by solving an optimal control problem in the overdamped dynamics by machine learning.

2. Fokker–Planck for a Time-Dependent Mechanical Overdamped Diffusion Process

We consider the Langevin–Smoluchowski stochastic differential equation

$$\begin{array}{c}\hfill \mathrm{d}{\mathcal{q}}_t=-\mu (\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right) \mathrm{d}t+\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} \mathrm{d}{\mathcal{w}}_t\end{array}$$

(1)

where ${\mathcal{w}}_t$ denotes a standard Wiener process [2,45]. The diffusion coefficient ${\beta }^{-1}$ is proportional to the temperature of the environment surrounding the system. The positive constant $\mu$ is the motility with canonical dimensions of a time over a mass. The drift in (1) is the gradient of a time-dependent potential

$$\begin{array}{c}\hfill U_t\left(\mathit{q}\right):[t_{\iota } , \infty ) \times {\mathbb{R}}^d\mapsto {\mathbb{R}}_{+}\end{array}$$

(2)

that we assume to be sufficiently regular and confining.

Remark 1.

Following a well-established convention in stochastic thermodynamics (see, e.g., [46]), we denote functional dependence upon time, i.e., the dynamical parameter, with an underscript. Round brackets express dependence upon state coordinates in configuration or phase space of the diffusion.

The probability density distribution of the solution of (1) at any instant of time t satisfies the Fokker–Planck equation

$$\begin{array}{c}\hfill {\partial }_t{\mathtt{p}}_t\left(\mathit{q}\right)-\mu 〈 {\mathsf{\partial }}_{\mathit{q}} , (\mathsf{\partial }{U}_t)\left(\mathit{q}\right){\mathtt{p}}_t\left(\mathit{q}\right) 〉-{\displaystyle \frac{\mu }{\beta }} {\mathsf{\partial }}_{\mathit{q}}^2{\mathtt{p}}_t\left(\mathit{q}\right)=0\end{array}$$

(3)

whose solution is fully specified by the assignment of an initial datum at time $t=t_{\iota }$. The assumption of a confining potential (2) guarantees that the probability density is integrable in ${\mathbb{R}}^d$. The connection between (1) and (3) stems from the representation of the transition probability density as a Monte Carlo average:

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)={\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{y} E_{\mathcal{P}}\left({\delta }^{\left(d\right)}(\mathit{q}-{\mathcal{q}}_t)|{\mathcal{q}}_{t_{\iota }}=\mathit{y}\right) {\mathtt{p}}_{t_{\iota }}\left(\mathit{q}\right)\end{array}$$

(4)

The expectation value $E_{\mathcal{P}}$ is over the probability measure $\mathcal{P}$ weighing the realizations of the solutions of (1). The singular nature of the Dirac delta distribution prevents the accurate evaluation of the transition probability density as a Monte Carlo average. For this reason, we look for the solution in the form

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)=e^{-\beta U_t\left(\mathit{q}\right)} {\mathtt{f}}_t\left(\mathit{q}\right)\end{array}$$

(5)

Upon inserting into (3), we arrive at

$$\begin{array}{c}\hfill {\partial }_t{\mathtt{f}}_t\left(\mathit{q}\right)+\mu 〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,(\mathsf{\partial }{\mathtt{f}}_t)\left(\mathit{q}\right) 〉-{\displaystyle \frac{\mu }{\beta }} {\mathsf{\partial }}_{\mathit{q}}^2{\mathtt{f}}_t\left(\mathit{q}\right)=\beta {\mathtt{f}}_t\left(\mathit{q}\right) {\partial }_t{U}_t\left(\mathit{q}\right)\end{array}$$

(6)

Proposition 1.

The solution of (3) admits the representation

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)=e^{-\beta U_t\left(\mathit{q}\right)} E_{{\mathcal{P}}^{♭}}\left({\mathtt{p}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) e^{\beta U_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right)+\beta {\int }_{t_{\iota }}^t\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)}|{\mathcal{q}}_t=\mathit{q}\right)\end{array}$$

(7)

where ${\mathcal{P}}^{♭}$ is the probability measure over the paths of the backward diffusion process

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\mathrm{d}}^{♭}{\mathcal{q}}_t:={\mathcal{q}}_t-{\mathcal{q}}_{t-\mathrm{d}t}=\mu (\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right) \mathrm{d}t+\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} {\mathrm{d}}^{♭}{\mathcal{w}}_t\hfill \\ \hfill & {\mathrm{d}}^{♭}{\mathcal{w}}_t={\mathcal{w}}_t-{\mathcal{w}}_{t-\mathrm{d}t}\hfill \end{array}\end{array}$$

(8)

naturally complemented by conditions assigned for some $t_f\ge t$.

Idea of the proof: We start by recalling that, for any test function ${\mathtt{F}}_t$, Itô’s lemma for backward differentials yields

$$\begin{array}{c}\hfill {\mathrm{d}}^{♭}{\mathtt{F}}_t\left({\mathcal{q}}_t\right)=\mathrm{d}t\left({\partial }_t{\mathtt{F}}_t\left({\mathcal{q}}_t\right)+\mu 〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,(\mathsf{\partial }{\mathtt{F}}_t)\left(\mathit{q}\right) 〉-{\displaystyle \frac{\mu }{\beta }}{\mathsf{\partial }}_{\mathit{q}}^2{\mathtt{F}}_t\left(\mathit{q}\right)\right)+\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} 〈 {\mathrm{d}}^{♭}{\mathcal{w}}_t ,(\mathsf{\partial }{\mathtt{F}}_t)\left(\mathit{q}\right) 〉\end{array}$$

We emphasize that ${\mathrm{d}}^{♭}{\mathcal{w}}_t$ is just a standard Wiener process but evolving backward in time. In the stochastic analysis jargon, (8) is a diffusion process with respect to a backward filtration as in, e.g., [47]. As is well known, the stochastic integrals and martingale properties are the same as in forward calculus once one exchanges the pre-point rule with the post point, see, e.g., [48]. Let us define the auxiliary function

$$\begin{array}{c}\hfill {\mathtt{g}}_t\left(\mathit{q}\right)=e^{\beta {\int }_t^{t_f}\mathrm{d}s {\partial }_s{U}_s\left(\mathit{q}\right)}{\mathtt{f}}_t\left(\mathit{q}\right)\end{array}$$

Then, Itô’s lemma and (6) immediately imply

$$\begin{array}{cc}\hfill {\mathrm{d}}^{♭}{\mathtt{g}}_t\left({\mathcal{q}}_t\right)& =- e^{\beta {\int }_t^{t_f}\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)}{\mathtt{f}}_t\left({\mathcal{q}}_t\right) \beta {\partial }_t{U}_t\left({\mathcal{q}}_s\right) \mathrm{d}t+e^{\beta {\int }_t^{t_f}\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)} \mathrm{d}{\mathtt{f}}_t\left({\mathcal{q}}_t\right)\hfill \\ \hfill & =e^{\beta {\int }_t^{t_f}\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)}\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} 〈 {\mathrm{d}}^{♭}{\mathcal{w}}_t ,(\mathsf{\partial }{\mathtt{f}}_t)\left(\mathit{q}\right) 〉\hfill \end{array}$$

The equation tells us that the auxiliary function is a local martingale of the backward diffusion (see, e.g., Chapter 7 of [2]). Since we assume that the confining potential also guarantees integrability, we infer that the martingale property

$$\begin{array}{c}\hfill E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_f}\left({\mathcal{q}}_{t_f}\right) | {\mathcal{q}}_{t_f}=\mathit{q}\right)=E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) | {\mathcal{q}}_{t_f}=\mathit{q}\right)\end{array}$$

must also hold true. By construction, we know that

$$\begin{array}{c}\hfill E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_f}\left({\mathcal{q}}_{t_f}\right)|{\mathcal{q}}_{t_f}=\mathit{q}\right)={\mathtt{f}}_{t_f}\left(\mathit{q}\right)\end{array}$$

and we conclude

$$\begin{array}{c}\hfill {\mathtt{f}}_{t_f}\left(\mathit{q}\right)=E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) | {\mathcal{q}}_{t_f}=\mathit{q}\right)=E_{{\mathcal{P}}^{♭}}\left(e^{\beta {\int }_{t_{\iota }}^{t_f}\mathrm{d}s {\partial }_s{U}_s\left(\mathit{q}\right)}{\mathtt{f}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) | {\mathcal{q}}_{t_f}=\mathit{q}\right)\end{array}$$

Replacing $t_f$ with t in the above chain of identities completes the proof. □

The upshot is that we can use the Feynman–Kac formula over a backward diffusion to compute the solution of a forward Fokker–Planck equation. Next, we take advantage of Girsanov’s change-of-measure formula (see, e.g., Chapter 10 of [2] or 3.5 of [45]) to evaluate the conditional expectation in (7) directly over the paths of the Wiener process, or, more generally, over the paths of any diffusion that generates a measure with respect to which ${\mathcal{P}}^{♭}$ is absolutely continuous. Girsanov’s change-of-measure formula is thus the basis of statistical inference for diffusion processes, see, e.g., [49]. We emphasize that we make use of the Girsanov formula while dealing with backward diffusions as, e.g., in [48]. As time is evolving from a larger to smaller value, correspondingly, the role of “past” and “future” events must be exchanged.

Remark 2.

As increments of any Wiener process are independent, from now on we write

$$\begin{array}{c}\hfill {\mathrm{d}}^{♭}{\mathcal{w}}_t=\mathrm{d}{\mathcal{w}}_t\end{array}$$

to alleviate the notation.

2.1. Use of Girsanov’s Formula

We denote by $\mathcal{Q}$ the probability measure over the path of

$$\begin{array}{c}\hfill {\mathrm{d}}^{♭}{\mathcal{q}}_t=\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} \mathrm{d}{\mathcal{w}}_t\end{array}$$

(9)

Our aim is to use Girsanov’s formula to express expectations with respect to the path measure ${\mathcal{P}}^{♭}$ of (8) in terms of expectations with respect to $\mathcal{Q}$:

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)=e^{-\beta U_t\left(\mathit{q}\right)} E_{\mathcal{Q}}\left({\mathtt{p}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) e^{\beta U_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right)+\beta {\int }_{t_{\iota }}^t\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)}{\displaystyle \frac{\mathrm{d}{\mathcal{P}}^{♭}}{\mathrm{d}\mathcal{Q}}} | {\mathcal{q}}_t=\mathit{q}\right)\end{array}$$

where

$$\begin{array}{c}\hfill {\displaystyle \frac{\mathrm{d}{\mathcal{P}}^{♭}}{\mathrm{d}\mathcal{Q}}}=exp{\int }_{t_{\iota }}^t\left(\sqrt{{\displaystyle \frac{\beta \mu }{2}}}〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉-{\displaystyle \frac{\beta \mu }{4}}\mathrm{d}s{∥(\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right)∥}^2\right)\end{array}$$

(10)

is the Radon–Nikodym derivative. The symbol ∘ emphasizes that we define the stochastic integral in (10) using the post-point prescription:

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉:=\underset{\begin{array}{c}\mathrm{d}t\downarrow 0\\ N\mathrm{d}t=t-t_{\iota }\end{array}}{lim}\sum _{i=1}^N〈 {\mathcal{w}}_{t_{i+1}}-{\mathcal{w}}_{t_i} ,(\mathsf{\partial }{U}_{t_{i+1}})\left({\mathcal{q}}_{t_{i+1}}\right) 〉\end{array}$$

i.e., the function $(\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right)$ is evaluated at the end of each time interval. Accordingly, (10) is a martingale with respect to the backward filtration (i.e., the family of $\sigma$-algebras increases as $t_{\iota }$ decreases) to which we associate the probability measure $\mathcal{Q}$. In other, rougher, words, (10) is a martingale conditional to events occurring at times larger than or equal to the upper bound of integration t.

Writing the stochastic integral in the standard pre-point form allows us to simplify the expression of the probability density. We notice that (9) trivially implies

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t\sqrt{{\displaystyle \frac{\beta \mu }{2}}}〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉={\int }_{t_{\iota }}^t{\displaystyle \frac{\beta \mu }{2}}〈 {\mathrm{d}}^{♭}{\mathcal{q}}_s\stackrel{\circ }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉\end{array}$$

Next, we use the relation between stochastic integrals in the post-point, mid-point (or Stratonovich, denoted by the ⋄-symbol), and pre-point prescriptions:

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t〈 \mathrm{d}{\mathcal{q}}_s\stackrel{\diamond }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉={\int }_{t_{\iota }}^t{\displaystyle \frac{〈 \mathrm{d}{\mathcal{q}}_s\stackrel{\circ }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉+〈 \mathrm{d}{\mathcal{q}}_s ,(\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉}{2}}\end{array}$$

Finally, we recall that ordinary differential calculus holds for the stochastic differentials in Stratonovich form:

$$\begin{array}{c}\hfill U_t\left({\mathcal{q}}_t\right)-U_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right)={\int }_{t_{\iota }}^t\left(\mathrm{d}s {\partial }_s{U}_s\left({\mathcal{q}}_s\right)+〈 \mathrm{d}{\mathcal{q}}_s\stackrel{\diamond }{ ,} (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉\right)\end{array}$$

Putting these observations together, we obtain the following representation of the solution for the Fokker–Planck Equation (3):

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)= E\left({\mathtt{p}}_{t_{\iota }}\left({\mathcal{q}}_{t_{\iota }}\right) e^{-{\displaystyle \frac{\beta }{2}}{\int }_{t_{\iota }}^t\left(〈 \mathrm{d}{\mathcal{q}}_s , (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) 〉+{\displaystyle \frac{\mu }{2}} \mathrm{d}s {∥(\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right)∥}^2\right)} | {\mathcal{q}}_t=\mathit{q}\right)\end{array}$$

(11)

In practice, this means that, to compute the probability density at the configuration space point $\mathit{q}$ at time $t \ge t_{\iota }$, we need to average the initial density over solutions of (9) evolved backward in time to $t_{\iota }$ and weighted by a path-dependent change-of-measure factor.

2.2. Examples and Path Integral Representation

Let us summarize the meaning of (11) in words. Formula (7) tells us that a Fokker–Planck equation of a forward diffusion process with gradient drift, i.e., of the form (3), admits a Feynman–Kac representation in terms of a backward diffusion process. This is because we can use the potential specifying the drift to turn the forward Fokker–Planck into a non-homogeneous backward Kolmogorov equation with respect to the backward diffusion process. This latter equation, as is well known, generically specifies a problem with initial data. We now turn to illustrate this fact with two examples.

2.2.1. Analytical Example

Consider a quadratic potential

$$\begin{array}{c}\hfill U_t\left(\mathit{q}\right)={\displaystyle \frac{1}{2}}〈 \mathit{q} ,{\mathsf{U}}_t\mathit{q} 〉\end{array}$$

with ${\mathsf{U}}_s$ denoting a $d \times d$ real symmetric time-dependent matrix. The backward stochastic differential Equation (8) reduces to

$$\begin{array}{c}\hfill {\mathrm{d}}^{♭}{\mathcal{q}}_t=\mu {\mathsf{U}}_t{\mathcal{q}}_t\mathrm{d}t+\sqrt{{\displaystyle \frac{2 \mu }{\beta }}}\mathrm{d}{\mathcal{w}}_t\end{array}$$

Letting $\mathsf{F}$ denote the flow solution of the deterministic ordinary differential equation

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\displaystyle \frac{\mathrm{d}}{\mathrm{d}t}}{\mathsf{F}}_{t,s}=-\mu {\mathsf{U}}_t {\mathsf{F}}_{t,s}\end{array}\end{array}$$

then the solution of the backward stochastic differential equation is

$$\begin{array}{cc}\hfill & {\mathcal{q}}_t={\mathsf{F}}_{t_f,t}^{\top } \mathit{q}-\sqrt{{\displaystyle \frac{2 \mu }{\beta }}}{\int }_t^{t_f}{\mathsf{F}}_{s,t}^{\top } \mathrm{d}{\mathit{w}}_s\hfill \\ \hfill & {\mathcal{q}}_{t_f}=\mathit{q}\hfill \end{array}$$

The symbol ⊤ as usual denotes matrix transposition. The corresponding transition probability density is Gaussian with mean

$$\begin{array}{cccc}\hfill & E\left({\mathcal{q}}_t|{\mathcal{q}}_u=\mathit{q}\right)={\mathsf{F}}_{u,t}^{\top } \mathit{q}\hfill & \hfill & u \ge t\hfill \end{array}$$

(recalling that, for standard backward differential equations, the martingale property arises upon conditioning on future events [47]) and variance matrix

$$\begin{array}{cccc}\hfill & E\left(({\mathcal{q}}_t-E{\mathcal{q}}_t)\otimes ({\mathcal{q}}_t-E{\mathcal{q}}_t)|{\mathcal{q}}_u=\mathit{q}\right)={\displaystyle \frac{2 \mu }{\beta }}{\int }_t^u\mathrm{d}s {\mathsf{F}}_{s,t}^{\top }{\mathsf{F}}_{s,t}\hfill & \hfill & u \ge t\hfill \end{array}$$

We need to compute

$$\begin{array}{c}\hfill f_t\left(\mathit{q}\right)=E\left(f_{t_{\iota }}\left(\mathit{q}\right) e^{\beta {\int }_{t_{\iota }}^t\mathrm{d}s{\displaystyle \frac{〈 {\mathcal{q}}_s ,{\dot{\mathsf{U}}}_s{\mathcal{q}}_s 〉}{2}}}|{\mathit{q}}_t=\mathit{q}\right)\end{array}$$

If we couch this expression into the form of a path integral [50], we obtain

$$\begin{array}{c}\hfill f_t\left(\mathit{q}\right)={\int }_{{\mathcal{q}}_t=\mathit{q}}\mathcal{D}\left[{\mathsf{q}}_{t:t_{\iota }}\right]e^{-{\int }_{t_{\iota }}^t\mathrm{d}s\left(\beta {\displaystyle \frac{\parallel {\dot{\mathsf{q}}}_s-\mu {\mathsf{U}}_s{\mathsf{q}}_s{\parallel }^2}{4 \mu }}-\beta {\displaystyle \frac{〈{\mathsf{q}}_s ,{\dot{\mathsf{U}}}_s{\mathsf{q}}_s 〉 }{2}}\right)}f\left({\mathsf{q}}_{t_{\iota }}\right)\end{array}$$

Here, $\mathcal{D}\left[{\mathsf{q}}_{t:t_{\iota }}\right]$ denotes the limit over finite dimensional approximations over time lattices in $[t_{\iota },t]$ of paths satisfying the terminal condition ${\mathcal{q}}_t=\mathit{q}$. We are free to interpret the path integral in the mid-point sense, because any change of discretization generates a path-independent Jacobian that can be reabsorbed into the normalization constant. As the integral is Gaussian, we can compute it by infinite dimensional stationary phase using ordinary differential calculus. We are left with

$$\begin{array}{c}\hfill f_t\left(\mathit{q}\right)=e^{U_t\left(\mathit{q}\right)}{\int }_{{\mathcal{q}}_t=\mathit{q}}\mathcal{D}\left[{\mathsf{q}}_{t:t_{\iota }}\right]e^{-{\int }_{t_{\iota }}^t\mathrm{d}s\left(\beta {\displaystyle \frac{\parallel {\dot{\mathsf{q}}}_s{\parallel }^2}{4 \mu }}+\beta {\displaystyle \frac{2〈{\dot{\mathsf{q}}}_s ,{\mathsf{U}}_s{\mathsf{q}}_s 〉+\mu 〈{\mathsf{U}}_s{\mathsf{q}}_s ,{\mathsf{U}}_s{\mathsf{q}}_s 〉}{4}}\right)}{p}_{t_{\iota }}\left({\mathsf{q}}_{t_{\iota }}\right)\end{array}$$

We now readily recognize that

$$\begin{array}{c}\hfill {\mathtt{p}}_{t,s}(\mathit{q}\mid \mathit{y})={\int }_{{\mathcal{q}}_s=\mathit{y}}^{{\mathsf{q}}_t=\mathit{q}}\mathcal{D}\left[{\mathsf{q}}_{t:t_{\iota }}\right]e^{-{\int }_{t_{\iota }}^t\mathrm{d}s\left(\beta {\displaystyle \frac{\parallel {\dot{\mathsf{q}}}_s{\parallel }^2}{4 \mu }}+\beta {\displaystyle \frac{2〈{\dot{\mathsf{q}}}_s ,{\mathsf{U}}_s{\mathsf{q}}_s 〉+\mu 〈{\mathsf{U}}_s{\mathsf{q}}_s ,{\mathsf{U}}_s{\mathsf{q}}_s 〉}{4}}\right)}\end{array}$$

is the path integral expression of the transition probability density of the forward stochastic differential equation

$$\begin{array}{c}\hfill \mathrm{d}{\mathcal{q}}_t=-\mu {\mathsf{U}}_t{\mathcal{q}}_t \mathrm{d}t+\sqrt{{\displaystyle \frac{2 \mu }{\beta }}} \mathrm{d}{\mathcal{w}}_t\end{array}$$

We therefore recover the Chapman–Kolmogorov representation of the solution of (3):

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)={\int }_{\mathbb{R}}\mathrm{d}\mathit{y} {\mathtt{p}}_{t,s}(\mathit{q}\mid \mathit{y}) {\mathtt{p}}_{t_{\iota }}\left(\mathit{q}\right)\end{array}$$

(12)

as expected.

2.2.2. Path-Integral Representation in General

The path integral representation of (7) is

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)={\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{y}{\int }_{{\mathcal{q}}_{t_{\iota }}=\mathit{y}}^{{\mathcal{q}}_t=\mathit{q}}\mathcal{D}\left[{\mathsf{q}}_{t:t_{\iota }}\right]e^{-{\int }_{t_{\iota }}^t\mathrm{d}s\left(\beta {\displaystyle \frac{\parallel {\dot{\mathsf{q}}}_s{\parallel }^2}{4 \mu }}+\beta {\displaystyle \frac{〈{\dot{\mathsf{q}}}_s ,(\mathsf{\partial }{U}_s)\left({\mathsf{q}}_s\right) 〉+\mu {\parallel (\mathsf{\partial }{U}_s)\left({\mathsf{q}}_s\right)\parallel }^2}{4}}\right)}{p}_{t_{\iota }}\left(\mathit{y}\right)\end{array}$$

As the stochastic integral term is evaluated in the pre-point representation, the path integral exactly recovers the path integral expression of the transition probability density. We have thus verified that (12) holds in general. We refer the reader unfamiliar with path integral calculus to, e.g., [50].

2.2.3. Numerical Example: Time-Independent Drift

In this section, we demonstrate how the method described can be applied to find a numerical solution of a Fokker–Planck equation driven by a mechanical potential. We consider the Fokker–Planck of the form (3) with a time-independent drift. By applying the Girsanov theorem as above, we couch the solution of the Fokker–Planck at time $t\in [t_{\iota },t_f]$ into a numerical average of simulated trajectories of the auxiliary dynamics, given by

$$\begin{array}{cc}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)& =E_{\mathcal{Q}}\left(P_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right)exp\left(-{\int }_{t_{\iota }}^t\sqrt{{\displaystyle \frac{\beta \mu }{2}}}〈{\mathrm{d}\mathcal{w}}_r , {\mathsf{\partial }}_{\mathit{q}}{U}_r\left({\mathcal{q}}_r\right)〉+\mathrm{d}r {\displaystyle \frac{\beta \mu }{4}}{∥{\mathsf{\partial }}_{\mathit{q}}{U}_r\left({\mathcal{q}}_r\right)∥}^2\right) | {\mathcal{q}}_t=\mathit{q}\right)\hfill \end{array}$$

(13)

where $\mathcal{Q}$ is the measure generated by the backwards diffusion (9). We approximate the expectation value numerically by repeated sampling of trajectories of the process (9). The trajectories are approximated on a discretization of the time interval $[t_{\iota },t_f]$ given by

$$t_{\iota }=t_0<t_1<\dots <t_N=t_f$$

(14)

Trajectories of (9) are sampled using the Euler–Maruyama scheme:

$${\mathcal{q}}_{t_{n-1}}={\mathcal{q}}_{t_n}-\sqrt{{\displaystyle \frac{2 \mu }{\beta }}|t_{n-1}-t_n|} ϵ$$

(15)

where $ϵ$ is an increment of Brownian noise, sampled independently from a standard normal distribution. The Girsanov factor g is computed as a running cost:

$$g=\sum _{n=0}^{N-1}{\displaystyle \frac{\beta \mu }{4}}|t_{n-1}-t_n{|\parallel (\partial \mathit{U})}_{t_{n-1-i}}\left({\mathcal{q}}_{t_{n-1-i}}\right){|}^2+\sqrt{{\displaystyle \frac{\mu \beta }{2}}|t_{n-1}-t_n|} (\partial {\mathit{U}}_{t_{n-1-i}})\left({\mathcal{q}}_{t_{n-1-i}}\right) ϵ$$

This computation is summarized in Algorithm 1. In Figure 1, we integrate an example Fokker–Planck equation driven by a time-independent mechanical potential in two ways. The results of Algorithm 1 are compared to the proximal gradient descent method of [42]. In this method, the solution is found via gradient descent on the space of probability distributions by solving a proximal fixed point recursion at each time step. Both methods discretize the time interval, but do not require spacial discretization. In our implementation, the Monte Carlo method performs significantly faster.

Algorithm 1 Integrating Fokker–Planck equation using Girsanov theorem

Initialize ${\mathcal{q}}_{t_n}=\mathit{q}\in {\mathbb{R}}^d$ Initialize $g=0$ Initialize $\partial U_{t_n}$ for $n\in \{0,1,\dots ,N\}$ Initialize ${\delta }_t=|t_{n-1}-t_n|$ for $n\in \{0,1,\dots ,N\}$ for i in $0,\dots ,n-1$ do    Sample Brownian noise: $ϵ\sim \mathcal{N}(0,1)$    Evolve one step of (8): ${\mathcal{q}}_{t_{n-i-1}}={\mathcal{q}}_{t_{n-i}}-\sqrt{2 {\delta }_t\mu /\beta } ϵ$    Add to running total: $$g=g+{\delta }_t{\displaystyle \frac{\mu \beta }{4}} {\parallel (\partial {\mathit{U}}_{t_{n-1-i}})\left({\mathcal{q}}_{t_{n-1-i}}\right)\parallel }^2+\sqrt{{\delta }_t{\displaystyle \frac{\mu \beta }{2}}} (\partial {\mathit{U}}_{t_{n-1-i}})\left({\mathcal{q}}_{t_{n-1-i}}\right) ϵ$$ end for Return ${\mathtt{p}}_{t_n}\left(\mathit{q}\right)=P_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right) e^{-g}$

2.2.4. Numerical Example: “Föllmer’s Drift”

In this section, we apply (7) to a non-trivial example of gradient drift. Specifically, we consider the Föllmer drift solution of the dynamic Schrödinger bridge that steers the system between assigned boundary conditions while minimizing the Kullback–Leibler divergence from a free diffusion [10], given by

$$D_{\mathcal{KL}}={\displaystyle \frac{\beta \mu }{4}} E\left({\int }_{t_{\iota }}^{t_f}\mathrm{d}t {\parallel (\partial U_t)\left({\mathcal{q}}_t\right)\parallel }^2\right)$$

(16)

in a finite time interval $t\in [t_{\iota },t_f]$. The boundary conditions are assigned on the initial and final distributions of the position, denoted $P_{\iota }$ for the initial at time $t=t_{\iota }$ and $P_f$ for the final at time $t=t_f$. We consider boundary conditions of the form

$$\begin{array}{cc}\hfill P_{\iota }\left(\mathit{q}\right)& ={\displaystyle \frac{e^{-\beta U_{\iota }\left(\mathit{q}\right)}}{{\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{y} e^{-\beta U_{\iota }\left(\mathit{y}\right)}}}\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill P_f\left(\mathit{q}\right)& ={\displaystyle \frac{e^{-\beta U_f\left(\mathit{q}\right)}}{{\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{y} e^{-\beta U_f\left(\mathit{y}\right)}}}.\hfill \end{array}$$

(17b)

Föllmer drifts are relevant to machine learning applications, see, e.g., [44,51]. We refer to [12] or [25] and references therein for further details on the mathematics and physics backgrounds, respectively.

We summarize how to construct the Föllmer drift by solving a Schrödinger bridge problem using an iterative method of [41]. In doing so, we also obtain the solution of the Fokker–Planck Equation (3) that we use for comparison with the numerical expression provided by (7). The Schrödinger bridge problem is formulated as the minimization of a Bismut–Pontryagin functional [25]. In this framework, we find that the intermediate density $\mathrm{p}$ and a value function V imposing the boundary conditions satisfy the coupled partial differential equations

$$\begin{array}{cc}\hfill {\partial }_t{\mathtt{p}}_t\left(\mathit{q}\right)-\mu 〈 {\mathsf{\partial }}_{\mathit{q}} ,(\mathsf{\partial }{U}_t)\left(\mathit{q}\right){\mathtt{p}}_t\left(\mathit{q}\right) 〉-{\displaystyle \frac{\mu }{\beta }}{\mathsf{\partial }}_{\mathit{q}}^2{\mathtt{p}}_t\left(\mathit{q}\right)& =0;\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill {\partial }_t{V}_t\left(\mathit{q}\right)-\mu 〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,{\mathsf{\partial }}_{\mathit{q}}{V}_t\left(\mathit{q}\right)〉+{\displaystyle \frac{\mu }{\beta }} {\mathsf{\partial }}_{\mathit{q}}^2{V}_t\left(\mathit{q}\right)+{\displaystyle \frac{\beta \mu }{4}}{\left((\mathsf{\partial }{U}_t)\left(\mathit{q}\right)\right)}^2& =0\hfill \end{array}$$

(18b)

along with the stationarity condition

$${\mathsf{\partial }}_{\mathit{q}}{V}_t\left(\mathit{q}\right)={\displaystyle \frac{2}{\beta }}{\mathsf{\partial }}_{\mathit{q}}{U}_t\left(\mathit{q}\right)$$

(19)

We identify (18a) as the Fokker–Planck equation, and (18b) as the Hamilton–Jacobi–Bellman equation which is discussed in later sections. For a known U, we can apply the Girsanov theorem to integrate (18a). We find a reference solution to the system (18a) and (18b) using an adaptation of the method of [41], which is briefly described below. The transformation

$$\begin{array}{cc}\hfill {\mathtt{p}}_t\left(\mathit{q}\right)& ={\varphi }_t\left(\mathit{q}\right){\widehat{\varphi }}_t\left(\mathit{q}\right)\hfill \\ \hfill V_t\left(\mathit{q}\right)& =-log\left({\varphi }_t\left(\mathit{q}\right)\right)\hfill \end{array}$$

(20)

applied to (18a) and (18b) yields the linear coupled equations

$$\begin{array}{cc}\hfill {\partial }_t{\varphi }_t\left(\mathit{q}\right)+{\displaystyle \frac{\mu }{\beta }} {\mathsf{\partial }}_{\mathit{q}}^2{\varphi }_t\left(\mathit{q}\right)& =0\hfill \end{array}$$

(21a)

$$\begin{array}{cc}\hfill {\partial }_t{\widehat{\varphi }}_t\left(\mathit{q}\right)-{\displaystyle \frac{\mu }{\beta }} {\mathsf{\partial }}_{\mathit{q}}^2{\widehat{\varphi }}_t\left(\mathit{q}\right)& =0\hfill \end{array}$$

(21b)

with boundary conditions

$$\begin{array}{cc}\hfill {\varphi }_{t_f}\left(\mathit{q}\right)& =P_f\left(\mathit{q}\right) / {\widehat{\varphi }}_{t_f}\left(\mathit{q}\right)\hfill \\ \hfill {\widehat{\varphi }}_{t_{\iota }}\left(\mathit{q}\right)& =P_{\iota }\left(\mathit{q}\right) / {\varphi }_{t_{\iota }}\left(\mathit{q}\right)\hfill \end{array}$$

We make an initial guess for ${\varphi }_{t_{\iota }}$ which we use to integrate (21a), recompute the boundary conditions and then integrate (21b), recomputing ${\varphi }_{t_{\iota }}$, and repeating this process until convergence: see [41] or Section 8.2 of [25] for a more detailed treatment. We reconstruct the value function and intermediate densities using (20).

With these results, we have a numerical approximation of the drift which maps an assigned initial probability density into an assigned final density while minimizing the Kullback–Leibler divergence on the interval $[t_{\iota },t_f]$. We use this drift to compute the solution of the Fokker–Planck (18a) ${\mathtt{p}}_t$ via Algorithm 1, and compare it to the density resulting from the iteration method of [41] in Figure 2.

3. Fokker–Planck for a Time-Dependent Mechanical Underdamped Diffusion Process

We now turn our attention to the underdamped dynamics [52]:

$$\begin{array}{cc}\hfill & \mathrm{d}{\mathcal{q}}_t={\displaystyle \frac{{\mathcal{p}}_t}{m}} \mathrm{d}t\hfill \\ \hfill & \mathrm{d}{\mathcal{p}}_t=-\left({\displaystyle \frac{{\mathcal{p}}_t}{\tau }}+(\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right)\right) \mathrm{d}t+\sqrt{{\displaystyle \frac{2 m}{\tau \beta }}} \mathrm{d}{\mathcal{w}}_t\hfill \end{array}$$

(22)

This is probably the most popular model of an open classical system in contact with a bath at temperature ${\beta }^{-1}$, driven by a mechanical force and subject to a linear friction force dissipating energy at Stokes rate $\tau$. As in Section 2, we assume that the mechanical potential is confining. The corresponding Fokker–Planck equation

$$\begin{array}{cc}\hfill & \left({\partial }_t+〈 {\displaystyle \frac{\mathit{p}}{m}} ,{\partial }_{\mathit{q}} 〉-〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,{\partial }_{\mathit{p}} 〉-〈 {\mathsf{\partial }}_{\mathit{p}} ,{\displaystyle \frac{\mathit{p}}{\tau }} 〉-{\displaystyle \frac{m}{\beta \tau }}{\mathsf{\partial }}_{\mathit{p}}^2\right){\mathtt{p}}_t\left(\mathit{x}\right)=0\hfill \end{array}$$

(23)

relaxes to a Maxwell–Boltzmann equilibrium for any time-independent confining potential

$$\begin{array}{c}\hfill {\mathtt{p}}_{\mathrm{eq}}\left(\mathit{x}\right)={\displaystyle \frac{e^{-\frac{\beta {∥\mathit{p}∥}^2}{2 m}-\beta U\left(\mathit{q}\right)}}{Z}}\end{array}$$

with Z a normalizing constant. Our aim is to express the solution of (23) as a suitable Monte Carlo average over an initial datum at time $t_{\iota }$. We proceed analogously to Section 2 and posit

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=e^{-{\displaystyle \frac{\beta {∥\mathit{p}∥}^2}{2 m}}}{\mathtt{f}}_t\left(\mathit{x}\right)\end{array}$$

where $\mathit{x}={[\mathit{q},\mathit{p}]}^{\top }$. The symplectic component of the drift acts in the same manner on the probability density and the auxiliary function ${\mathtt{f}}_t$. The dissipative component changes sign:

$$\begin{array}{c}\hfill \left({\partial }_t+〈 {\displaystyle \frac{\mathit{p}}{m}} ,{\partial }_{\mathit{q}} 〉-〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right)-{\displaystyle \frac{\mathit{p}}{\tau }} ,{\mathsf{\partial }}_{\mathit{p}}〉-{\displaystyle \frac{m}{\beta \tau }}{\mathsf{\partial }}_{\mathit{p}}^2\right){\mathtt{f}}_t\left(\mathit{x}\right)=-{\displaystyle \frac{\beta }{m}}〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,\mathit{p} 〉{\mathtt{f}}_t\left(\mathit{x}\right)\end{array}$$

(24)

The upshot is that the resulting equation admits the interpretation of a non-homogeneous backward Kolmogorov equation associated with the backward process

$$\begin{array}{cc}\hfill & {\mathrm{d}}^{♭}{\mathcal{q}}_t={\displaystyle \frac{{\mathcal{p}}_t}{m}} \mathrm{d}t\hfill \\ \hfill & {\mathrm{d}}^{♭}{\mathcal{p}}_t=\left({\displaystyle \frac{{\mathcal{p}}_t}{\tau }}-(\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right)\right) \mathrm{d}t+\sqrt{{\displaystyle \frac{2 m}{\tau \beta }}} \mathrm{d}{\mathcal{w}}_t\hfill \end{array}$$

(25)

Specifically, it is possible to prove the following:

Proposition 2.

The solution of the Fokker–Planck Equation (23) can be couched into a conditional expectation

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=e^{-{\displaystyle \frac{\beta {∥\mathit{p}∥}^2}{2 m}}}{E}_{{\mathcal{P}}^{♭}}\left(e^{{\displaystyle \frac{\beta {∥{\mathcal{p}}_{t_{\iota }}∥}^2}{2 m}}}{\mathtt{p}}_{t_{\iota }}\left({\mathcal{x}}_{t_{\iota }}\right) e^{-{\displaystyle \frac{\beta }{m}}{\int }_{t_{\iota }}^t\mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉} | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

(26)

with respect to the path measure ${\mathcal{P}}^{♭}$ generated by (25).

Idea of the proof: The proof mirrors that of the overdamped case. We define the auxiliary function

$$\begin{array}{c}\hfill {\mathtt{g}}_t\left({\mathcal{x}}_t\right)=e^{-{\displaystyle \frac{\beta }{m}}{\int }_t^{t_f}\mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉}{\mathtt{f}}_t\left({\mathcal{x}}_t\right)\end{array}$$

for any ${\mathrm{f}}_t$ solution of (24). Differentiation backward in time along the paths of (25) yields

$$\begin{array}{c}\hfill {\mathrm{d}}^{♭}{\mathtt{g}}_t\left({\mathcal{x}}_t\right)=e^{-{\displaystyle \frac{\beta }{m}}{\int }_t^{t_f}\mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉}\sqrt{{\displaystyle \frac{2 m}{\tau \beta }}}〈 {\mathrm{d}}^{♭}{\mathcal{w}}_t ,{\mathsf{\partial }}_{{\mathcal{p}}_t}{\mathtt{g}}_t\left({\mathcal{x}}_t\right) 〉\end{array}$$

We conclude that, in any time integral where ${\mathrm{g}}_t$ is integrable, it is also a martingale with respect to the measure ${\mathcal{P}}^{♭}$ generated by (25):

$$\begin{array}{c}\hfill E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_{\iota }}\left({\mathcal{x}}_{t_{\iota }}\right) | {\mathcal{x}}_{t_f}=\mathit{x}\right)=E_{{\mathcal{P}}^{♭}}\left({\mathtt{g}}_{t_f}\left({\mathcal{x}}_{t_f}\right) | {\mathcal{x}}_{t_f}=\mathit{x}\right)={\mathtt{f}}_{t_f}\left(\mathit{x}\right)\end{array}$$

The chain of identities yields the claim. □

In the case of particular physical interest, when the system probability density at time $t_{\iota }$ takes the Maxwell–Boltzmann form

$$\begin{array}{c}\hfill {\mathtt{p}}_{t_{\iota }}\left(\mathit{x}\right)={\displaystyle \frac{e^{-\frac{\beta {∥\mathit{p}∥}^2}{2 m}-\beta U_{\iota }\left(\mathit{q}\right)}}{Z}}\end{array}$$

(26) reduces to

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=e^{-{\displaystyle \frac{\beta {∥\mathit{p}∥}^2}{2 m}}}{E}_{{\mathcal{P}}^{♭}}\left({\displaystyle \frac{e^{-\beta U_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right)}}{Z}} e^{-{\displaystyle \frac{\beta }{m}}{\int }_{t_{\iota }}^t \mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉} | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

3.1. Representation with Respect to a Time Reversal Invariant Measure

We now turn our attention to the path measure $\mathcal{I}$ generated by the forward process

$$\begin{array}{cc}\hfill & \mathrm{d}{\mathcal{q}}_t={\displaystyle \frac{{\mathcal{p}}_t}{m}} \mathrm{d}t\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill & \mathrm{d}{\mathcal{p}}_t=-(\mathsf{\partial }{U}_t)\left({\mathcal{q}}_t\right) \mathrm{d}t+\sqrt{{\displaystyle \frac{2 m}{\tau \beta }}} \mathrm{d}{\mathcal{w}}_t\hfill \end{array}$$

(27b)

The drift of the process is divergenceless. This, together with the statistical invariance under time reversal of the Wiener process, imply that we can also interpret the equation

$$\begin{array}{c}\hfill \left({\partial }_t+〈 {\displaystyle \frac{\mathit{p}}{m}} ,{\partial }_{\mathit{q}} 〉-〈 (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,{\partial }_{\mathit{p}} 〉+{\displaystyle \frac{m}{\beta \tau }}{\mathsf{\partial }}_{\mathit{p}}^2\right){\mathrm{f}}_t(\mathit{q},\mathit{p})=0\end{array}$$

both as a backward Kolmogorov equation or as the Fokker–Planck equation associated with (27a) and (27b) if we replace forward differentials with backward differentials. This means that both $\mathcal{P}$ and ${\mathcal{P}}^{♭}$ are absolutely continuous with respect to $\mathcal{I}$. In particular, we find that

$$\begin{array}{c}\hfill {\displaystyle \frac{\mathrm{d}{\mathcal{P}}^{♭}}{\mathrm{d}\mathcal{I}}}=exp{\int }_{t_{\iota }}^t\left(\sqrt{{\displaystyle \frac{\tau \beta }{2 m}}}〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} {\displaystyle \frac{{\mathcal{p}}_s}{\tau }} 〉-{\displaystyle \frac{\tau \beta }{4 m}} \mathrm{d}s {∥{\displaystyle \frac{{\mathcal{p}}_s}{\tau }}∥}^2\right)\end{array}$$

(28)

As in Section 2, the symbol ∘ denotes that the post-point prescription in the construction of the stochastic integral

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} {\displaystyle \frac{{\mathcal{p}}_s}{\tau }} 〉=\sum _{t_{\iota }<{\mathfrak{t}}_i \le t}〈 {\mathcal{w}}_{{\mathfrak{t}}_i}-{\mathcal{w}}_{{\mathfrak{t}}_{i-1}} ,{\displaystyle \frac{{\mathcal{p}}_{{\mathfrak{t}}_i}}{\tau }} 〉\end{array}$$

The immediate consequence is the representation of the solution of the Fokker–Planck Equation (23):

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=e^{-{\displaystyle \frac{\beta {∥\mathit{p}∥}^2}{2m}}}{E}_{\mathcal{I}}\left({\displaystyle \frac{e^{-\beta U_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right)}}{Z}} e^{-{\displaystyle \frac{\beta }{m}}{\int }_{t_{\iota }}^t\mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉}{\displaystyle \frac{\mathrm{d}{\mathcal{P}}^{♭}}{\mathrm{d}\mathcal{I}}} | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

(29)

Some further simplifications are possible. In view of the identities

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t\mathrm{d}s 〈 (\mathsf{\partial }{U}_s)\left({\mathcal{q}}_s\right) ,{\mathcal{p}}_s 〉={\int }_{t_{\iota }}^t〈 \sqrt{{\displaystyle \frac{2 m}{\tau \beta }}}\mathrm{d}{\mathcal{w}}_s-\mathrm{d}{\mathcal{p}}_s\stackrel{\circ }{ ,} {\mathcal{p}}_s 〉\end{array}$$

and

$$\begin{array}{c}\hfill \mathrm{d}{\mathcal{p}}_t^2=2 〈 {\mathcal{p}}_t\stackrel{\circ }{ ,} \mathrm{d}{\mathcal{p}}_t 〉-{\displaystyle \frac{2 m d}{\tau \beta }} \mathrm{d}t\end{array}$$

we can couch (29) into the form

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=e^{{\displaystyle \frac{(t-t_{\iota })d}{\tau }}}{E}_{\mathcal{I}}\left({\displaystyle \frac{e^{-\beta U_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right)-\frac{\beta {∥{\mathcal{p}}_{t_{\iota }}∥}^2}{2 m}-{\int }_{t_{\iota }}^t\left(\sqrt{\frac{\beta \tau }{2 m}}〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} \frac{{\mathcal{p}}_s}{\tau } 〉+\frac{\tau \beta }{4 m}\mathrm{d}s {∥\frac{{\mathcal{p}}_s}{\tau }∥}^2\right)}}{Z}} | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

Finally, we can re-write the stochastic integral in the pre-point discretization using the identity

$$\begin{array}{c}\hfill {\int }_{t_{\iota }}^t\sqrt{{\displaystyle \frac{\tau \beta }{2 m}}}〈 \mathrm{d}{\mathcal{w}}_s\stackrel{\circ }{ ,} {\displaystyle \frac{{\mathcal{p}}_s}{\tau }} 〉={\int }_{t_{\iota }}^t\sqrt{{\displaystyle \frac{\tau \beta }{2 m}}}〈 \mathrm{d}{\mathcal{w}}_s ,{\displaystyle \frac{{\mathcal{p}}_s}{\tau }} 〉+{\displaystyle \frac{(t-t_{\iota })d}{\tau }}\end{array}$$

Thus, we arrive at

$$\begin{array}{c}\hfill {\mathtt{p}}_t\left(\mathit{x}\right)=E_{\mathcal{I}}\left({\displaystyle \frac{1}{Z}} e^{-\beta U_{\iota }\left({\mathcal{q}}_{t_{\iota }}\right)-{\displaystyle \frac{\beta {∥{\mathcal{p}}_{t_{\iota }}∥}^2}{2 m}}-{\int }_{t_{\iota }}^t\left(\sqrt{{\displaystyle \frac{\beta \tau }{2 m}}}〈 \mathrm{d}{\mathcal{w}}_s , {\displaystyle \frac{{\mathcal{p}}_s}{\tau }} 〉+{\displaystyle \frac{\tau \beta }{4 m}} \mathrm{d}s {∥{\displaystyle \frac{{\mathcal{p}}_s}{\tau }}∥}^2\right)}|{\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

The advantage of taking averages with respect to the measure $\mathcal{I}$ is that it allows us to use paths of (27a) and (27b) to integrate both the Fokker–Planck equation and a coupled Hamilton–Jacobi–Bellman equation, necessary when constructing numerical solutions of Schrödinger bridge problems.

3.2. Numerical Example

In this section, we illustrate the numerical integration of a Fokker–Planck Equation (23) governing the evolution of the joint density of the momentum and position processes following underdamped dynamics (23). We once again consider the optimal control problem of minimizing the Kullback–Leibler divergence from a free diffusion (16) on the interval $[t_{\iota },t_f]$. This optimal control problem is formulated as the minimization of a Bismut–Pontryagin functional, resulting in the coupled partial differential equations

$$\begin{array}{cc}\hfill \left({\partial }_t+〈 {\displaystyle \frac{\mathit{p}}{m}} ,{\partial }_{\mathit{q}} 〉-\langle (\mathsf{\partial }{U}_t)\left(\mathit{q}\right) ,{\mathsf{\partial }}_{\mathit{p}} \rangle -\langle {\mathsf{\partial }}_{\mathit{p}} ,{\displaystyle \frac{\mathit{p}}{\tau }} \rangle -{\displaystyle \frac{m}{\beta \tau }}{\mathsf{\partial }}_{\mathit{p}}^2\right) {\mathtt{p}}_t\left(\mathit{x}\right)& =0\hfill \end{array}$$

(30a)

$$\begin{array}{cc}\hfill \left({\partial }_t+\langle {\displaystyle \frac{\mathit{p}}{m}} , {\mathsf{\partial }}_{\mathit{p}}\rangle -\langle \left({\displaystyle \frac{\mathit{p}}{\tau }}+(\mathsf{\partial }{U}_t)\left(\mathit{q}\right)\right) , {\mathsf{\partial }}_{\mathit{q}}\rangle +{\displaystyle \frac{m}{\beta \tau }} {\mathsf{\partial }}_{\mathit{p}}^2\right)V_t\left(\mathit{x}\right)& =-{\displaystyle \frac{\beta m}{4 {\tau }^2}}{\parallel (\mathsf{\partial }{U}_t)\left(\mathit{q}\right)\parallel }^2\hfill \end{array}$$

(30b)

with the stationarity condition

$$\begin{array}{c}\hfill {\partial }_{\mathit{q}}{U}_t\left(\mathit{q}\right)={\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{p} {\displaystyle \frac{{\mathrm{p}}_t(\mathit{p},\mathit{q})}{{\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{p} {\mathrm{p}}_t(\mathit{p},\mathit{q})}} {\partial }_{\mathit{p}}{V}_t(\mathit{p},\mathit{q})\end{array}$$

(31)

We identify (30a) as the Fokker–Planck and (30b) as the Hamilton–Jacobi–Bellman equation. Using the Girsanov theorem, we find an expression for the intermediate density, the solution of (30a), as an expectation

$${\mathrm{p}}_t\left(\mathit{x}\right)=E\left(P_{\iota }({\mathcal{p}}_{t_{\iota }} , {\mathcal{q}}_{t_{\iota }})e^{-{\displaystyle \frac{\tau \beta }{4 m}}{\int }_{t_{\iota }}^t\mathrm{d}s{\parallel {\mathcal{p}}_s\parallel }^2-\sqrt{{\displaystyle \frac{\tau \beta }{2 m}}}{\int }_{t_{\iota }}^t\langle {\mathrm{d}\mathcal{w}}_s , {\mathcal{p}}_s\rangle } | {\mathcal{x}}_t=\mathit{x}\right)$$

(32)

taken over trajectories of the process (27a) and (27b).

We consider the case where the boundary conditions are assigned on the joint distribution at the initial and final times

$$\begin{array}{cc}\hfill P_{\iota }(\mathit{p},\mathit{q})& ={\displaystyle \frac{1}{Z_{\iota }}}exp\left(-{\displaystyle \frac{{\beta \parallel \mathit{p}\parallel }^2}{2}}-\beta U_{\iota }\left(\mathit{q}\right)\right)\hfill \end{array}$$

(33a)

$$\begin{array}{cc}\hfill P_f(\mathit{p},\mathit{q})& ={\displaystyle \frac{1}{Z_f}}exp\left(-{\displaystyle \frac{{\beta \parallel \mathit{p}\parallel }^2}{2}}-\beta U_f\left(\mathit{q}\right)\right)\hfill \end{array}$$

(33b)

with $Z_{\iota },Z_f$ normalizing constants.

The numerical computation is summarized in Algorithm 2. For the optimal control potential and benchmark solution, we use numerical predictions from [25]. There, predictions are made for the optimal protocol in the underdamped dynamics using a multiscale perturbative expansion around the overdamped problem; for more detail, see Section 8.2 of [25]. The prediction for the optimal control protocol is used as the drift in the integration of the Fokker–Planck (30a), with the results shown in Figure 3. The convergence of the numerical scheme is emphasized in Figure 4. We compute relative distances from the assigned final boundary data (33b) and the variance of the results of Algorithm 2 as the number of sampled trajectories increases and the step size remains constant.

Algorithm 2 Integrating a Fokker–Planck for an underdamped diffusion process using Girsanov theorem

Initialize ${\mathcal{p}}_{t_n}, {\mathcal{q}}_{t_n}=\mathit{p}, \mathit{q}\in \mathbb{R}$ Initialize $g=0$ Initialize $\partial U_{t_n}$ for $n\in \{0,1,\dots ,N\}$ Initialize ${\delta }_t=|t_{n-1}-t_n|$ for i in $0,\dots ,n-1$ do    Sample Brownian noise: $ϵ\sim \mathcal{N}(0,1)$    Evolve one backward step of (29): $\left\{\begin{array}{c}{\mathcal{q}}_{t_{n-1-i}}={\mathcal{q}}_{t_{n-i}}-{\displaystyle \frac{1}{m}}{\mathcal{p}}_{t_i} {\delta }_t\hfill \\ {\mathcal{p}}_{t_{n-1-i}}={\mathcal{p}}_{t_{n-i}}+(\partial U_{t_{n-i}})\left({\mathcal{q}}_{t_{n-i}}\right) {\delta }_t-\sqrt{{\delta }_t{\displaystyle \frac{2m }{\tau \beta }}} ϵ\hfill \end{array}\right.$    Add Girsanov weight: $g=g+{\delta }_t{\displaystyle \frac{\tau \beta }{4 m}} {\mathit{p}}_{t_{n-1-i}}^2+\sqrt{{\delta }_t{\displaystyle \frac{\tau \beta }{2 m}} } ϵ {\mathit{p}}_{t_{n-1-i}}$ end for Return ${\mathtt{p}}_{t_n}(\mathit{p},\mathit{q})=P_{\iota }({\mathcal{p}}_{t_{\iota }},{\mathcal{q}}_{t_{\iota }}) e^{-g}$

4. Bismut–Elworthy–Li Monte Carlo Representation of Gradients

Numerical integration of Schrödinger bridge type problems, in the overdamped [12,13,15,41] and underdamped [22,23,25,53] cases require the solution of a Hamilton–Jacobi–Bellman (also known as a dynamic programming) equation, specifying the optimal control potential. In the simplest overdamped setup, the mechanical force is given by (19). The function

$$\begin{array}{c}\hfill V_t\left(\mathit{q}\right):[t_{\iota },t_f] \times {\mathbb{R}}^d\mapsto \mathbb{R}\end{array}$$

solves a Burgers type equation. More generally, optimization problems often require computing gradients of scalar functions satisfying a non-homogeneous backward Kolmogorov equation in $[t_{\iota } ,t_f]$ of the form

$$\begin{array}{cc}\hfill \left({\partial }_t+〈 {\mathit{b}}_t\left(\mathit{x}\right) ,{\mathsf{\partial }}_{\mathit{x}} 〉+〈 {\mathsf{A}}_t\left(\mathit{x}\right){\mathsf{A}}_t^{\top }\left(\mathit{x}\right) ,{\mathsf{\partial }}_{\mathit{x}}\otimes {\mathsf{\partial }}_{\mathit{x}} 〉\right)V_t\left(\mathit{x}\right)& = - F_t\left(x\right)\hfill \end{array}$$

(34a)

$$\begin{array}{cc}\hfill V_{t_f}\left(\mathit{x}\right)& =\phi \left(\mathit{x}\right)\hfill \end{array}$$

(34b)

The left hand side of (34a) is the mean forward derivative (see Chapter 11 of [54]) of $V_t$ along the paths of the n-dimensional system of Itô stochastic differential equations:

$$\begin{array}{c}\hfill \mathrm{d}{\mathcal{x}}_t={\mathit{b}}_t\left({\mathcal{x}}_t\right) \mathrm{d}t+{\mathsf{A}}_t\left({\mathcal{x}}_t\right) \mathrm{d}{\mathit{w}}_t\end{array}$$

(35)

In (34a) and (35), we consider drift $\mathit{b}$ and volatility fields $\mathsf{A}$ of a more general form than in Section 2 and Section 3. This choice means that the following discussion is applicable to both overdamped and underdamped cases, as well as to more general situations, including non-linear problems [40]. In non-linear problems, the expression of the solution of (34a) and (34b) and its gradient are iteratively computed in sequences of infinitesimal time horizons $[t_{\iota },t_f]$ to construct the solution of partial differential equations in which $\mathit{b}$, $\mathsf{A}$ and F depend upon the unknown field V.

It is well known that Dynkin’s formula (see, e.g., Chapter 6 of [2]) yields a Monte Carlo representation of the solution of (34a) and (34b):

$$\begin{array}{c}\hfill V_t\left(\mathit{x}\right)=E_{\mathcal{P}}\left(\phi \left({\mathcal{x}}_{t_f}\right)+{\int }_t^{t_f}\mathrm{d}s F_s\left({\mathcal{x}}_s\right) | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

(36)

Our goal is to find an analogous expression for the gradient of $V_t$. The Bismut–Elworthy–Li formula [34,35,55] accomplishes this task.

Remark 3.

In what follows, to neaten mathematical formulae we adopt the push-forward notation for the Jacobian matrix of a vector field. We refer to Section $\mathcal{O}.j$ pag. $\mathrm{xlii}$ of [56] for a geometrical justification of the notation.

For any $\mathit{v}:{\mathbb{R}}^d\mapsto {\mathbb{R}}^d$, we write

$$\begin{array}{c}\hfill 〈 {\mathit{e}}_i ,{\mathit{v}}_{\ast }{\mathit{e}}_j 〉:={\partial }_{x^j}{v}_t^i\left(\mathit{x}\right)\end{array}$$

(37)

where ${\mathit{e}}_i$ and ${\mathit{e}}_j$ are, respectively, the i-th and j-th elements of the canonical basis of ${\mathbb{R}}^d$. Under our regularity assumptions, we regard the solution of (35) satisfying the condition

$$\begin{array}{cccc}\hfill & {\mathcal{x}}_s=\mathit{x}\hfill & \hfill & s \le t\hfill \end{array}$$

as the image of the stochastic flow $\mathit{X}:\mathbb{R}\times \mathbb{R}\times {\mathbb{R}}^d\mapsto {\mathbb{R}}^d$ [57] such that

$$\begin{array}{c}\hfill {\mathcal{x}}_t={\mathit{X}}_{t,s}\left({\mathcal{x}}_s\right)\end{array}$$

and omit reference to the initial data on the left hand side, when no ambiguity arises. According to (37), we denote the cocycle obtained by differentiating the flow ${\mathit{X}}_{t,s}$ with respect to its argument as ${\mathcal{x}}_{\ast t,s}$ implying that

$$\begin{array}{c}\hfill 〈 {\mathit{e}}_i ,{\mathcal{x}}_{\ast t,s}{\mathit{e}}_j 〉:={\partial }_{{\mathcal{x}}^j}{X}_{t,s}^i\left(\mathit{x}\right)\end{array}$$

By definition, ${\mathcal{x}}_{\ast t,s}$ enjoys the cocycle property [3], meaning that

$$\begin{array}{cc}{\mathcal{x}}_{\ast t,s}\left({\mathcal{x}}_s\right)={\mathcal{x}}_{\ast t,u}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,s}\left({\mathcal{x}}_s\right)\hfill & \hfill \forall s\le u\le t\\ \hfill \ast \ast & \hfill \end{array}$$

Here, we present a heuristic, physics-style derivation of the formula based on Malliavin’s stochastic variational calculus [58] which draws from the mathematically more rigorous exposition in [36], and is close to the original treatment in [34]. To this end, we observe that if ${\mathit{e}}_i$ is the i-th element of the canonical basis of ${\mathbb{R}}^n$

$$\begin{array}{c}\hfill 〈{\mathit{e}}_i ,(\mathsf{\partial }{V}_t)\left(\mathit{x}\right)〉=E\left(〈{\mathcal{x}}_{\ast t_f,t}{\mathit{e}}_i ,(\mathsf{\partial }\phi )\left({\mathcal{x}}_{t_f}\right)〉+{\int }_t^{t_f}\mathrm{d}s 〈{\mathcal{x}}_{\ast t_f,t}{\mathit{e}}_i ,(\mathsf{\partial }{F}_s)\left({\mathcal{x}}_s\right)〉|{\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

(38)

where ${\mathcal{x}}_{\ast t_f,t}$ denotes the matrix valued process obtained by varying (35) with respect to its initial datum. In other words, if we suppose

$$\begin{array}{cccc}\hfill & {\mathcal{x}}_s=\mathit{x}\hfill & \hfill & s \le t\hfill \end{array}$$

then

$$\begin{array}{cc}\hfill & \mathrm{d}{\mathcal{x}}_{\ast t,s}=({\mathit{b}}_{\ast t}\left({\mathcal{x}}_t\right) \mathrm{d}t+{\mathsf{A}}_{\ast t}\left({\mathcal{x}}_t\right) \mathrm{d}{\mathit{w}}_t) {\mathcal{x}}_{\ast t,s}\hfill \\ \hfill & {\mathcal{x}}_{\ast s,s}=1_n\hfill \end{array}$$

The identity (38) allows us to derive the Bismut–Elworthy–Li formula from Malliavin’s integration by parts formula.

4.1. Integration by Parts Formula

Let us consider the equation

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \mathrm{d}{\mathcal{x}}_t^{\left(\epsilon \right)}=\left({\mathit{b}}_t\left({\mathcal{x}}_t^{\left(\epsilon \right)}\right)+\epsilon {\mathcal{h}}_t\right) \mathrm{d}t+{\mathsf{A}}_t\left({\mathcal{x}}_t^{\left(\epsilon \right)}\right) \mathrm{d}{\mathit{w}}_t\hfill \\ \hfill & {\mathcal{x}}_s^{\left(\epsilon \right)}=\mathit{x}\hfill \end{array}\end{array}$$

(39)

We assume ${\mathcal{h}}_t$ to be a differentiable process, although rigorous constructions of integration by parts formula, see, e.g., [58], weaken this assumption to processes of bounded variation (see Chapter 1 of [2]). Differentiating at $\epsilon =0$ yields the variational equation

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \mathrm{d}{\mathcal{x}}_t^{\prime }=({\mathit{b}}_{\ast t}\left({\mathcal{x}}_t\right) \mathrm{d}t+{\mathsf{A}}_{\ast t}\left({\mathcal{x}}_t\right) \mathrm{d}{\mathit{w}}_t) {\mathcal{x}}_t^{\prime }+{\mathcal{h}}_t \mathrm{d}t\hfill \\ \hfill & {\mathcal{x}}_s^{\prime }=0\hfill \end{array}\end{array}$$

(40)

We can always write the solution of this latter equation in terms of the push-forward of the flow of (35):

$$\begin{array}{c}\hfill {\mathcal{x}}_t^{\prime }={\int }_s^t\mathrm{d}u {\mathcal{x}}_{\ast t,u} {\mathcal{h}}_u\end{array}$$

(41)

Therefore, for sufficiently small $\epsilon$,

$$\begin{array}{c}\hfill {\mathcal{x}}_t^{\left(\epsilon \right)}={\mathcal{x}}_t+\epsilon {\mathcal{x}}_t^{\prime }+\mathrm{h}.\mathrm{o}.\mathrm{t}\end{array}$$

(42)

allows us to regard the solution of (40) as a functional of the solution of (35) (h.o.t. stands for higher order terms). The conclusion is that we can compute the expectation value of any integrable function g of a solution of (40) by expressing it as a function of the solution of (35) via (42) and then averaging with respect to the measure $\mathcal{P}$ generated by (35):

$$\begin{array}{c}\hfill E_{{\mathcal{P}}^{\epsilon }}\left(g\left({\mathcal{x}}_t\right) | {\mathcal{x}}_s=\mathit{x}\right)=E_{\mathcal{P}}\left(g\left({\mathcal{x}}_t^{\left(\epsilon \right)}\right) | {\mathcal{x}}_s=\mathit{x}\right)\end{array}$$

(43)

A second connection comes from Girsanov’s change-of-measure formula. Namely, if ${\mathcal{P}}^{\epsilon }$ is the path measure generated by (39), then, for any test function g, we obtain the identity

$$\begin{array}{c}{E}_{{\mathcal{P}}^{\epsilon }}\left(g\left({\mathcal{x}}_t\right) | {\mathcal{x}}_s=\mathit{x}\right)=E_{\mathcal{P}}\left(g\left({\mathcal{x}}_t\right){\displaystyle \frac{\mathrm{d}{\mathcal{P}}^{\epsilon }}{\mathrm{d}\mathcal{P}}} | {\mathcal{x}}_s=\mathit{x}\right)\hfill \\ =E_{\mathcal{P}}\left(g\left({\mathcal{x}}_t\right) exp{\int }_s^t\left(〈 \mathrm{d}{\mathcal{w}}_u ,\epsilon {\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{h}}_u 〉-\mathrm{d}u {\displaystyle \frac{{∥\epsilon {\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right) {\mathcal{h}}_u∥}^2}{2}}\right) | {\mathcal{x}}_s=\mathit{x}\right)\hfill \end{array}$$

(44)

If g is also sufficiently regular, upon differentiating (43) and (44) at $\epsilon =0$, we arrive at Malliavin’s integration by parts formula:

$$\begin{array}{c}\hfill E_{\mathcal{P}}\left(〈 {\mathcal{x}}_t^{\prime } ,(\mathsf{\partial }g)\left({\mathcal{x}}_t\right) 〉 | {\mathcal{x}}_s=\mathit{x}\right)=E_{\mathcal{P}}\left(g\left({\mathcal{x}}_t\right){\int }_s^t〈 \mathrm{d}{\mathcal{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{h}}_u 〉 | {\mathcal{x}}_s=\mathit{x}\right)\end{array}$$

4.2. Application to Non-Degenerate Diffusion

We set

$$\begin{array}{c}\hfill {\mathcal{h}}_u={\mathcal{x}}_{\ast u,s}{\mathit{e}}_i\end{array}$$

(45)

where ${\mathit{e}}_i$ is the i-th element of the canonical basis of ${\mathbb{R}}^n$. This is legitimate because, under standard regularity assumptions, ${\mathcal{x}}_{\ast u,s}$ is a process of finite variation. Upon inserting into (41), we obtain

$$\begin{array}{c}\hfill {\mathcal{x}}_{t,s}^{\prime }=(t-s) {\mathcal{x}}_{\ast t,s} {\mathit{e}}_i\end{array}$$

The integration by parts formula becomes

$$\begin{array}{c}\hfill E_{\mathcal{P}}\left(〈 {\mathcal{x}}_{\ast t,s}{\mathit{e}}_i ,(\mathsf{\partial }g)\left({\mathcal{x}}_t\right) 〉 | {\mathcal{x}}_s=\mathit{x}\right)=E_{\mathcal{P}}\left({\displaystyle \frac{g\left({\mathcal{x}}_t\right)}{t-s}}{\int }_s^t〈 \mathrm{d}{\mathcal{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,s}{\mathit{e}}_i 〉 | {\mathcal{x}}_s=\mathit{x}\right)\end{array}$$

The identity holds for arbitrary $t \ge s$. Hence, we can apply it to (38) in order to derive the expression of the gradient of the solution of (34a) and (34b):

$$\begin{array}{cc}\hfill 〈{\mathit{e}}_i ,(\mathsf{\partial }{V}_t)\left(\mathit{x}\right)〉& =E\left({\displaystyle \frac{\phi \left({\mathcal{x}}_{t_f}\right)}{t_f-t}}{\int }_t^{t_f}〈 \mathrm{d}{\mathcal{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,t}{\mathit{e}}_i 〉 | {\mathcal{x}}_t=\mathit{x}\right)\hfill \\ \hfill & +E\left({\int }_t^{t_f}\mathrm{d}s {\displaystyle \frac{F_s\left({\mathcal{x}}_s\right)}{s-t}}{\int }_t^s〈 \mathrm{d}{\mathcal{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,t}{\mathit{e}}_i 〉 | {\mathcal{x}}_t=\mathit{x}\right)\hfill \end{array}$$

(46)

provided the volatility field $\mathsf{A}$ is always non-singular.

Application to the Transition Probability Density

It is worth noticing the following consequence of (46) when $F_s$ vanishes. In such a case, (46) reduces to

$$\begin{array}{c}\hfill {\mathsf{\partial }}_{\mathit{x}}{\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{y} \phi \left(\mathit{y}\right) p_{t_f,t}(\mathit{y} | \mathit{x})={\displaystyle \frac{1}{t_f-t}} E_{\mathcal{P}}\left(\phi \left({\mathcal{x}}_{t_f}\right){\int }_t^{t_f}〈 \mathrm{d}{\mathit{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right)){\mathcal{x}}_{\ast u,t}〉 | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

As the identity must hold true for any $\phi$, we can also write

$$\begin{array}{c}\hfill {\mathsf{\partial }}_{\mathit{x}}{p}_{t_f,t}(\mathit{y} | \mathit{x})={\displaystyle \frac{1}{t_f-t}} E_{\mathcal{P}}\left({\delta }^{\left(d\right)}(\mathit{y}-{\mathcal{x}}_{t_f}){\int }_t^{t_f}〈 \mathrm{d}{\mathit{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,t}〉 | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

(47)

A result by Molchanov, Section 5 of [59], allows us to express (47) in terms of an expectation value with respect to a reciprocal process, see, e.g., [60]. Namely, given a Markov process in $[t_{\iota },t_f]$, we can use it to construct a reciprocal process, i.e., a process conditioned at both ends of the time horizon from the relations

$$\begin{array}{cccc}\hfill & {\mathtt{p}}_{t,t_f,t_{\iota }}(\mathit{x}\mid \mathit{z},\mathit{y})={\displaystyle \frac{{\mathtt{p}}_{t_f,t}(\mathit{z}\mid \mathit{x}) {\mathtt{p}}_{t,t_{\iota }}(\mathit{x}\mid \mathit{y})}{{\mathtt{p}}_{t_f,t_{\iota }}(\mathit{z}\mid \mathit{y})}}\hfill & \hfill & t_{\iota } \le t \le t_f\hfill \\ \hfill & {\mathtt{p}}_{t_{\mathfrak{2}},t_{\mathfrak{1}}{t}_f,t_{\iota }}({\mathit{x}}_{\mathfrak{2}},{\mathit{x}}_{\mathfrak{1}}\mid \mathit{z},\mathit{y})={\displaystyle \frac{{\mathtt{p}}_{t_f,t_{\mathfrak{2}}}(\mathit{z}\mid {\mathit{x}}_{\mathfrak{2}}) {\mathtt{p}}_{t_{\mathfrak{2}},t_{\mathfrak{1}}}({\mathit{x}}_{\mathfrak{2}}\mid {\mathit{x}}_{\mathfrak{1}}) {\mathtt{p}}_{t_{\mathfrak{1}},t_{\iota }}({\mathit{x}}_{\mathfrak{1}}\mid \mathit{y})}{{\mathtt{p}}_{t_f,t_{\iota }}(\mathit{z}\mid \mathit{y})}}\hfill & \hfill & t_{\iota } \le t_{\mathfrak{1}} \le t_{\mathfrak{2}} \le t_f\hfill \\ \hfill & \mathrm{etc.}\hfill \end{array}$$

(48)

Upon contrasting (48) with (47), we thus arrive at Bismut’s formula (page 78 of [34]) for the gradient of the transition probability density:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{{\mathsf{\partial }}_{\mathit{x}}{p}_{t_f,t}(\mathit{y}\mid \mathit{x})}{p_{t_f,t}(\mathit{y}\mid \mathit{x})}}=E_{\mathcal{P}}\left({\int }_t^{t_f}〈 \mathrm{d}{\mathit{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{x}}_{\ast u,t}〉 | {\mathcal{x}}_{t_f}=\mathit{y},{\mathcal{x}}_t=\mathit{x}\right)\hfill \end{array}$$

(49)

The subscript $\mathcal{P}$ here means that we construct all finite dimensional approximations to the reciprocal process from the transition probability density of (35) according to (47).

Unfortunately, (49) does not directly provide a Monte Carlo representation of the score function because the derivative acts on the variable expressing the condition. It is, however, possible to use ideas similar to these and from the previous sections to obtain a Monte Carlo representation of the score function.

4.3. Analytical Example

It is worth illustrating the use of the Dynkin’s and Bismut–Elworthy–Li formulas in a case where all calculations can be performed explicitly. To this end let us consider

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \mathrm{d}{\mathcal{q}}_t={\displaystyle \frac{{\mathcal{p}}_t}{m}}+\sqrt{\eta {\displaystyle \frac{2 \tau }{m \beta }}} \mathrm{d}{w}_t^{\left(\mathfrak{1}\right)}\hfill \\ \hfill & \mathrm{d}{\mathcal{p}}_t=-{\displaystyle \frac{{\mathcal{p}}_t}{\tau }}+\sqrt{{\displaystyle \frac{2 m}{\beta \tau }}} \mathrm{d}{w}_t^{\left(\mathfrak{2}\right)}\hfill \end{array}\end{array}$$

(50)

whose solution is simply

$$\begin{array}{cc}\hfill & {\mathcal{q}}_t={\mathcal{q}}_{t_{\iota }}+\sqrt{\eta {\displaystyle \frac{2 \tau }{m \beta }}} w_t^{\left(\mathfrak{1}\right)}+{\int }_{t_{\iota }}^t\mathrm{d}s {\displaystyle \frac{{\mathcal{p}}_s}{m}}\hfill \\ \hfill & {\mathcal{p}}_t={\mathcal{p}}_{t_{\iota }}{e}^{-{\displaystyle \frac{t}{\tau }}}+\sqrt{{\displaystyle \frac{2 m}{\beta \tau }}}{\int }_{t_{\iota }}^t\mathrm{d}{w}_s^{\left(\mathfrak{2}\right)} e^{-{\displaystyle \frac{t-s}{\tau }}}\hfill \end{array}$$

Let us consider the partial differential equation

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\partial }_{t}V(q,p)+{\displaystyle \frac{p}{m}} {\partial }_{q}V(q,p)-{\displaystyle \frac{p}{\tau }} {\partial }_{p}V(q,p)+{\displaystyle \frac{\eta \tau }{m \beta }} {\partial }_q^{2}V(q,p)+{\displaystyle \frac{m}{\beta \tau }} {\partial }_p^{2}V(q,p)=0\hfill \\ \hfill & V_{t_f}(q,p)=p\hfill \end{array}\end{array}$$

(51)

It is straightforward to verify that at any time $t\le t_f$

$$\begin{array}{c}\hfill V_t(p,q)=p e^{-{\displaystyle \frac{t_f-t}{\tau }}}\end{array}$$

Upon applying Dynkin’s formula (36), we verify that

$$\begin{array}{c}\hfill V_t(p,q)=E\left({\mathcal{p}}_{t_f} | {\mathcal{q}}_t=q,{\mathcal{p}}_t=p\right)\end{array}$$

Next, we wish to apply Bismut–Elworthy–Li to recover

$$\begin{array}{c}\hfill {\partial }_p{V}_t(p,q)=e^{-{\displaystyle \frac{t_f-t}{\tau }}}\end{array}$$

(52)

To this end, we determine the cocycle solution of the linearized dynamics. The cocycle equation is

$$\begin{array}{cccc}\hfill & {\dot{\mathcal{x}}}_{\ast t,s}=\left[\begin{array}{cc}0& {\displaystyle \frac{1}{m}}\\ 0& -{\displaystyle \frac{1}{\tau }}\end{array}\right]{\mathcal{x}}_{\ast t,s} \hfill & \hfill & {\mathcal{x}}_{\ast s,s}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\hfill \end{array}$$

from where we obtain the unique solution

$$\begin{array}{c}\hfill {\mathcal{x}}_{\ast t,s}=\left[\begin{array}{cc}1& {\displaystyle \frac{1}{m}}\left(1-e^{-{\displaystyle \frac{t-s}{\tau }}}\right)\\ 0& e^{-{\displaystyle \frac{t-s}{\tau }}}\end{array}\right]\end{array}$$

To evaluate the Bismut–Elworthy–Li formula, we also need the inverse of the volatility matrix, which is

$$\begin{array}{c}\hfill A^{-1}=\left[\begin{array}{cc}\sqrt{{\displaystyle \frac{\beta m}{2 \eta \tau }}}& 0\\ 0& \sqrt{{\displaystyle \frac{\beta \tau }{2 m}}}\end{array}\right]\end{array}$$

We thus obtain

$$\begin{array}{cc}\hfill {\partial }_p{V}_t(p,q)& =E({\displaystyle \frac{{\mathcal{p}}_t e^{-\frac{t_f-t}{\tau }}+\sqrt{\frac{2 m}{\beta \tau }}{\int }_t^{t_f} \mathrm{d}{w}_s^{\left(\mathfrak{2}\right)} e^{-\frac{t_f-s}{\tau }}}{t_f-t}}\times \hfill \\ \hfill & \left({\int }_t^{t_f}\mathrm{d}{w}_u^{\left(\mathfrak{1}\right)}\sqrt{{\displaystyle \frac{\beta m}{2 \tau }}} {\displaystyle \frac{1-e^{-\frac{u-t}{\tau }}}{m}}+\sqrt{{\displaystyle \frac{\beta \tau }{2 m}}} \mathrm{d}{w}_u^{\left(\mathfrak{2}\right)}{e}^{-{\displaystyle \frac{u-t}{\tau }}}\right) | {\mathcal{q}}_t=q, {\mathcal{p}}_t=p)\hfill \end{array}$$

Using standard properties of stochastic integrals [2], we recover the expected result:

$$\begin{array}{c}\hfill {\partial }_p{V}_t(p,q)={\int }_t^{t_f}\mathrm{d}s {\displaystyle \frac{e^{-\frac{1}{\tau }(t_f-s)}{e}^{-\frac{1}{\tau }(s-t)}}{t_f-t}}=e^{-{\displaystyle \frac{t_f-t}{\tau }}}\end{array}$$

4.4. Numerical Example

In this section, we apply the Bismut-Elworthy- Li formula to compute the gradient of the value function in the optimal control problem minimizing the Kullback–Leibler divergence (16) in the overdamped dynamics: the gradient of the solution to (18b). This is calculated as a numerical average over sampled trajectories of (3). We use the same approximation of the optimal control potential U as in the Fokker–Planck example of Section 2.2.4. We find

$${\mathcal{q}}_{\ast s,t}=\left\{\begin{array}{cc}1\hfill & \mathrm{for} s=t\hfill \\ e^{-\mu {\int }_t^s\mathrm{d}r {\partial }^2{U}_r\left({\mathcal{q}}_r\right)} \hfill & s > t\hfill \end{array}\right.$$

(53)

Hence, (46) becomes

$$\begin{array}{cc}\hfill & \left({\partial }_{\mathit{q}}{V}_t\right)\left(\mathit{q}\right)=\sqrt{{\displaystyle \frac{\beta }{2 \mu }}}E({\displaystyle \frac{\phi \left({\mathcal{q}}_{t_f}\right)}{t_f-t}}{\int }_t^{t_f}〈{\mathrm{d}\mathcal{w}}_s , exp\left(-\mu {\int }_t^s\mathrm{d}r {\partial }^2{U}_r\left({\mathcal{q}}_r\right)\right)〉\hfill \\ \hfill & +{\displaystyle \frac{\beta \mu }{4}}{\int }_t^{t_f}\mathrm{d}s {\displaystyle \frac{\parallel \partial U_s\left({\mathcal{q}}_s\right){\parallel }^2}{s-t}} {\int }_t^s〈{\mathrm{d}\mathcal{w}}_r , exp\left(-\mu {\int }_t^r\mathrm{d}v {\partial }^2{U}_v\left({\mathcal{q}}_v\right)\right)〉 | {\mathcal{q}}_t=\mathit{q})\hfill \end{array}$$

(54)

We repeatedly sample trajectories of process (3) using the Euler–Maruyama discretization scheme and compute the integrals as running costs over each trajectory, finally taking a numerical expectation. The calculation is summarized in Algorithm 3 and numerical results shown in Figure 5.

From the physics point of view, note that we can conceptualize the motility constant with the ratio

$$\begin{array}{c}\hfill \mu ={\displaystyle \frac{\tau }{m}}\end{array}$$

(55)

for consistency with the underdamped equations.

Algorithm 3 Monte Carlo integration for gradient of the value function

Initialize ${\mathcal{q}}_{t_n}=\mathit{q}\in \mathbb{R}$ Initialize ${\iota }_1=0$ Initialize ${\iota }_2=0$ Initialize ${\iota }_3=0$ Initialize drift $\partial U_{t_n}$ for $n\in \{0,1,\dots ,N\}$ for i in $n,\dots ,N-1$ do    Sample Brownian noise: $ϵ\sim \mathcal{N}(0,1)$    Compute the BEL weights:    if i == n then      ${\iota }_2={\iota }_2+\sqrt{{\delta }_t} ϵ$    else      ${\iota }_1={\iota }_1+\mu {\delta }_t \left({\partial }^2{U}_{t_n}\right)\left({\mathcal{q}}_{t_n}\right)$      ${\iota }_2={\iota }_2+\sqrt{{\delta }_t} ϵ e^{ -{\iota }_1}$    end if    ${\iota }_3={\iota }_3+{\delta }_t {\parallel \left({\partial }_q{U}_{t_i}\right)\left({\mathcal{q}}_{t_i}\right)\parallel }^2 {\iota }_2$    Evolve one step of (3): ${\mathcal{q}}_{t_{i+1}}={\mathcal{q}}_{t_i}-\mu \left({\partial }_q{U}_{t_i}\right)\left({\mathcal{q}}_{t_i}\right) {\delta }_t+\sqrt{{\delta }_t{\displaystyle \frac{2 \mu }{\beta }}} ϵ$ end for Return $\partial V_{t_n}\left(\mathit{q}\right)=\sqrt{{\displaystyle \frac{\beta \mu }{2}}} \left({\displaystyle \frac{\phi \left({\mathcal{q}}_{t_f}\right)}{t_f-t_n}} {\iota }_2+{\displaystyle \frac{\beta \mu }{4}} {\iota }_3\right)$

5. Application of Bismut–Elworthy–Li to Degenerate Diffusion

For a degenerate diffusion we cannot directly apply (45) as it is because the expression is written in terms of the inverse of a degenerate matrix. Nevertheless the Bismut–Elworthy–Li formula continues to hold. To give an idea of how this comes about, we consider the counterpart to (35), while referring to [36] for the mathematically rigorous reader. Our starting point is

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \mathrm{d}{\mathcal{q}}_t={\mathit{a}}_t\left({\mathcal{x}}_t\right) \mathrm{d}t\hfill \\ \hfill & \mathrm{d}{\mathcal{p}}_t={\mathit{b}}_t\left({\mathcal{x}}_t\right) \mathrm{d}t+{\mathsf{A}}_t\left({\mathcal{x}}_t\right) \mathrm{d}{\mathcal{w}}_t\hfill \end{array}\end{array}$$

(56)

with

$$\begin{array}{c}\hfill {\mathcal{x}}_t=\left[\begin{array}{c}{\mathcal{q}}_t\\ {\mathcal{p}}_t\end{array}\right]:{\mathbb{R}}_{+}\mapsto {\mathbb{R}}^{2 d}\end{array}$$

and $\mathsf{A}$ a non-singular matrix field. The variational equation is

$$\begin{array}{cc}\hfill & \mathrm{d}{\mathcal{q}}_t^{\prime }={\mathit{a}}_{\ast t}\left({\mathcal{x}}_t\right){\mathcal{x}}_t^{\prime } \mathrm{d}t\hfill \\ \hfill & \mathrm{d}{\mathcal{p}}_t^{\prime }={\mathit{b}}_{\ast t}\left({\mathcal{x}}_t\right){\mathcal{x}}_t^{\prime } \mathrm{d}t+{\mathsf{A}}_{\ast t}\left({\mathcal{x}}_t\right){\mathcal{x}}_t^{\prime } \mathrm{d}{\mathcal{w}}_t+{\mathcal{h}}_t \mathrm{d}t\hfill \end{array}$$

where we suppose again

$$\begin{array}{c}\hfill {\mathcal{x}}_s^{\prime }=0\end{array}$$

We notice that we can always write the solution of the variational equation as

$$\begin{array}{c}\hfill {\mathcal{x}}_t^{\prime }={\mathcal{x}}_{★t,s}{\mathit{c}}_t\end{array}$$

for some vector valued process ${\mathit{c}}_t:\mathbb{R}\mapsto {\mathbb{R}}^{2 d}$ such that

$$\begin{array}{c}\hfill {\mathit{c}}_s=0\end{array}$$

Upon differentiation, we readily verify that the self-consistency condition is

$$\begin{array}{c}\hfill {\mathcal{x}}_{★ t,s} {\dot{\mathit{c}}}_t=\left[\begin{array}{c}0\\ {\mathcal{h}}_t\end{array}\right]\end{array}$$

whose solution is

$$\begin{array}{c}\hfill {\mathit{c}}_t={\int }_s^t\mathrm{d}u {\mathcal{x}}_{★u,s}^{-1}\left(\mathit{x}\right)\left[\begin{array}{c}0\\ {\mathcal{h}}_u\end{array}\right]\end{array}$$

(57)

We now avail ourselves of the fact that the above relations hold for a sufficiently regular but otherwise arbitrary vector field ${\mathcal{h}}_u$ and choose it such that

$$\begin{array}{c}\hfill {\mathit{c}}_{t_f}=\left[\begin{array}{c}0\\ \mathit{v}\end{array}\right]\end{array}$$

(58)

Here, $\mathit{v}$ is a unit vector that specifies the direction of the gradient in the Bismut–Elworthy–Li formula. Namely, given

$$\begin{array}{c}\hfill V_t\left(\mathit{x}\right)=E\left(\phi \left({\mathcal{x}}_{t_f}\right) | {\mathcal{x}}_t=\left[\begin{array}{c}\mathit{q}\\ \mathit{p}\end{array}\right]\right)\end{array}$$

the Bismut–Elworthy–Li formula [36] continues to hold according to the chain of identities

$$\begin{array}{cc}\hfill (\mathit{v} · {\mathsf{\partial }}_{\mathit{p}})V_t\left(\mathit{x}\right) & =E\langle {\mathcal{x}}_{\ast t_f,t} {\mathit{c}}_{t_f} ,(\mathsf{\partial }\phi )\left({\mathcal{x}}_{t_f}\right)\rangle =E\langle {\mathcal{x}}_{t_f}^{\prime } ,(\mathsf{\partial }\phi )\left({\mathcal{x}}_{t_f}\right)\rangle \hfill \\ \hfill & =E\left(\phi \left({\mathcal{x}}_{t_f}\right){\int }_t^{t_f}〈 \mathrm{d}{\mathcal{w}}_u ,{\mathsf{A}}_u^{-1}\left({\mathcal{x}}_u\right){\mathcal{h}}_u 〉 | {\mathcal{x}}_t=\left[\begin{array}{c}\mathit{q}\\ \mathit{p}\end{array}\right]\right)\hfill \end{array}$$

(59)

provided the conditions (57), (58) are satisfied.

Similarly, we obtain a representation of the derivative with respect to the $\mathit{q}$ variables by alternative choices of ${\mathcal{h}}_t$ such that

$$\begin{array}{c}\hfill {\mathit{c}}_{t_f}=\left[\begin{array}{c}\mathit{v}\\ 0\end{array}\right]\end{array}$$

5.1. A Strategy for the Explicit Construction of a Variational Field Enforcing the Boundary Conditions

Drawing from [36], we present a straightforward way to construct a variational field on the interval $[t,t_f]$ such that, e.g., (58) holds true. Let

$$\begin{array}{c}\hfill {\mathbb{H}}_u:={\displaystyle \frac{\mathrm{d}{\mathcal{x}}_{\ast u,t}}{\mathrm{d}u}} {\mathcal{x}}_{\ast u,t}^{-1}\end{array}$$

For clarity, we drop the subscripts $t,t_f$ in the following. However, there is still an implicit dependence on these parameters, with u taking values in $[t,t_f]$. Consider the differential system

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\dot{\mathit{g}}}_u\\ {\dot{\mathit{f}}}_u\end{array}\right]={\mathbb{H}}_u\left[\begin{array}{c}{\mathit{g}}_u\\ {\mathsf{\ell }}_u\end{array}\right]\end{array}$$

(60)

with ${\mathsf{\ell }}_u$ arbitrarily assigned (but sufficiently regular), and ${\mathit{g}}_u$ and ${\mathit{f}}_u$ determined by the identity. Then,

$$\begin{array}{c}\hfill {\dot{\mathcal{x}}}_u^{\prime }+\left[\begin{array}{c}{\dot{\mathit{g}}}_u\\ {\dot{\mathit{f}}}_u\end{array}\right]={\mathbb{H}}_u\left({\mathcal{x}}_u^{\prime }+\left[\begin{array}{c}{\mathit{g}}_u\\ {\mathsf{\ell }}_u\end{array}\right]\right)+\left[\begin{array}{c}0\\ {\mathcal{h}}_u\end{array}\right]\end{array}$$

holds by construction for $u\in [t,t_f]$. Hence, if we require

$$\begin{array}{c}\hfill {\dot{\mathsf{\ell }}}_u={\dot{\mathit{f}}}_u-{\mathcal{h}}_u\end{array}$$

we see that

$$\begin{array}{c}\hfill {\mathcal{y}}_u={\mathcal{x}}_u^{\prime }+\left[\begin{array}{c}{\mathit{g}}_u\\ {\mathsf{\ell }}_u\end{array}\right]\end{array}$$

satisfies

$$\begin{array}{cccc}\hfill & \begin{array}{cc}\hfill & {\dot{\mathcal{y}}}_u={\mathbb{H}}_u{\mathcal{y}}_u\hfill \\ \hfill & {\mathcal{y}}_t=\left[\begin{array}{c}{\mathit{g}}_t\\ {\mathsf{\ell }}_t\end{array}\right]\hfill \end{array} \hfill & \hfill & \forall u\in [t,t_f]\hfill \end{array}$$

We solve (60) with the “initial condition”

$$\begin{array}{c}\hfill {\mathit{g}}_t=0\end{array}$$

As a consequence, we arrive at

$$\begin{array}{c}\hfill {\mathcal{x}}_u^{\prime }={\mathcal{x}}_{\ast u,t}\left(\mathit{x}\right)\left[\begin{array}{c}0\\ {\mathsf{\ell }}_t\end{array}\right]-\left[\begin{array}{c}{\mathit{g}}_u\\ {\mathsf{\ell }}_u\end{array}\right]\end{array}$$

The identity we just obtained shows that, in order to obtain a representation of the gradient according to (59), we must restrict the choice of vector fields ${\mathsf{\ell }}_u$ to those satisfying the boundary conditions

$$\begin{array}{cc}\hfill & {\mathsf{\ell }}_t=\mathit{v}\hspace{1em}\hspace{1em}\&\hspace{1em}\hspace{1em}{\mathsf{\ell }}_{t_f}=0\hfill \end{array}$$

(61)

so that at time $u=t_f$

$$\begin{array}{c}\hfill {\mathcal{x}}_{t_f}^{\prime }={\mathcal{x}}_{\ast t_f,t}\left(\mathit{x}\right)\left[\begin{array}{c}0\\ \mathit{v}\end{array}\right]\end{array}$$

(62)

holds true. Once all the above conditions are satisfied, we can determine the right hand side of (59) from

$$\begin{array}{cccc}\hfill & {\mathcal{h}}_u={\dot{\mathit{f}}}_u-{\dot{\mathsf{\ell }}}_u\hfill & \hfill & \forall u \in [t,t_f]\hfill \end{array}$$

(63)

A Case of Particular Interest

There are an infinite number of ways to choose ${\mathsf{\ell }}_u$ such that condition (62) holds true. We detail here a choice of particular interest for physics. Let us consider the generator of the linearized dynamics around a path solution of (22):

$$\begin{array}{c}\hfill {\mathbb{H}}_u\left({\mathcal{x}}_u\right)=\left[\begin{array}{cc}0& {\displaystyle \frac{1}{m}}\\ -(\mathsf{\partial }\otimes \mathsf{\partial }{U}_u)\left({\mathcal{q}}_u\right)& -{\displaystyle \frac{1}{\tau }}\end{array}\right]\end{array}$$

In such a case, the instantiation of (60) is the differential system

$$\begin{array}{cc}\hfill & {\dot{\mathit{g}}}_u={\displaystyle \frac{1}{m}}{\mathsf{\ell }}_u\hfill \\ \hfill & {\dot{\mathit{f}}}_u=-(\mathsf{\partial }\otimes \mathsf{\partial }{U}_s)\left({\mathcal{q}}_u\right) {\mathit{g}}_u-{\displaystyle \frac{1}{\tau }}{\mathsf{\ell }}_u\hfill \\ \hfill & {\dot{\mathit{f}}}_u-{\mathcal{h}}_u={\dot{\mathsf{\ell }}}_u\hfill \end{array}$$

We assign

$$\begin{array}{c}\hfill {\mathsf{\ell }}_u=\mathit{v} {\displaystyle \frac{t_f-u}{t_f-t}}+{\mathit{v}}_{\mathfrak{1}} {\displaystyle \frac{(t_f-u)(u-t)}{2 {(t_f-t)}^2}}\end{array}$$

and obtain

$$\begin{array}{c}\hfill {\mathit{g}}_u=\mathit{v}{\int }_t^u\mathrm{d}s {\displaystyle \frac{t_f-s}{m (t_f-t)}}+{\mathit{v}}_{\mathfrak{1}}{\int }_t^u\mathrm{d}s {\displaystyle \frac{(t_f-s)(s-t)}{2 m {(t_f-t)}^2}}\end{array}$$

We fix ${\mathit{v}}_{\mathfrak{1}}$ by requiring

$$\begin{array}{c}\hfill {\mathit{g}}_{t_f}=0⟹{\mathit{v}}_{\mathfrak{1}}=-6 \mathit{v}\end{array}$$

We thus obtain

$$\begin{array}{cc}\hfill & {\mathsf{\ell }}_u=\mathit{v} {\displaystyle \frac{(t_f-u)(t_f+2 t-3 u)}{{(t_f-t)}^2}}\hfill \end{array}$$

(64)

$$\begin{array}{cc}\hfill & {\mathit{g}}_u=\mathit{v} {\displaystyle \frac{{(t_f-u)}^2(u-t)}{m {(t_f-t)}^2}}\hfill \end{array}$$

(65)

and therefore

$$\begin{array}{c}\hfill {\dot{\mathit{f}}}_u=-{\displaystyle \frac{1}{\tau }}\mathit{v} {\displaystyle \frac{(t_f-u)(t_f+2 t-3u)}{{(t_f-t)}^2}}-(\mathsf{\partial }\otimes \mathsf{\partial }{U}_u)\left({\mathcal{q}}_u\right) \mathit{v} {\displaystyle \frac{{(t_f-u)}^2(u-t)}{m {(t_f-t)}^2}}\end{array}$$

5.2. Analytical Example

We return to the elementary case (50), (51) but set $\eta =0$. The momentum gradient (52) does not depend on $\eta$, yet the application of the Bismut–Elworthy–Li formula requires the inverse of the volatility which in turn appears to depend on $\eta$. As in the present example, the potential is identically vanishing:

$$\begin{array}{c}\hfill U=0\end{array}$$

(66)

we arrive at

$$\begin{array}{cc}\hfill & {\partial }_p{V}_t(p,q)=\hfill \\ \hfill & E\left(\left({\mathcal{p}}_t{e}^{-{\displaystyle \frac{t_f-t}{\tau }}}+\sqrt{{\displaystyle \frac{2 m}{\beta \tau }}}{\int }_t^{t_f}\mathrm{d}{w}_s^{\left(\mathfrak{2}\right)} e^{-{\displaystyle \frac{t_f-s}{\tau }}}\right){\int }_t^{t_f}\sqrt{{\displaystyle \frac{\beta \tau }{2 m}}}\mathrm{d}{w}_u^{\left(\mathfrak{2}\right)}\left(-{\displaystyle \frac{{\mathsf{\ell }}_u}{\tau }}-{\dot{\mathsf{\ell }}}_u\right) | {\mathcal{q}}_t=q,{\mathcal{p}}_t=p\right)\hfill \end{array}$$

Using again the properties of stochastic integrals, the expectation value reduces to

$$\begin{array}{c}\hfill {\partial }_p{V}_t(p,q)=-{\int }_t^{t_f}\mathrm{d}s {\displaystyle \frac{\mathrm{d}}{\mathrm{d}s}}\left(e^{-{\displaystyle \frac{t_f-s}{\tau }}}{\mathsf{\ell }}_s\right)=-{\mathsf{\ell }}_{t_f}+e^{-{\displaystyle \frac{t_f-t}{\tau }}}{\mathsf{\ell }}_t\end{array}$$

whence we recover the correct expression of the gradient once we recall the boundary conditions imposed on the function ℓ in $[t,t_f]$.

This example also indicates that the existence of the Bismut–Elworthy–Li formula for Langevin–Kramers equations of the form (56) can be recovered from the limit $\eta$ tending to zero of a non-degenerate model owing to the vanishing of products of Itô stochastic integrals with respect to independent Wiener processes.

5.3. Numerical Example

We demonstrate here a numerical example of using the Bismut–Elworthy–Li formula to find the gradient of a value function satisfying the Hamilton–Jacobi–Bellman Equation (30b). We look at the case where the initial and final conditions assigned on the density are Gaussian distributions. In the case of Gaussian boundary conditions, we can determine the value function and optimal control protocol in the underdamped dynamics as the numerical solution of a system of differential equations; see Section IV in [25]. The value function $V_t$ is quadratic in the momentum and position variables, and in the two-dimensional phase space case reads

$$\begin{array}{c}\hfill V_t(p,q)=v_t^{\left(0\right)}+v_t^{\left(p\right)}p+v_t^{\left(q\right)}q+{\displaystyle \frac{1}{2}}\left(v_t^{(p,p)}{p}^2+2v_t^{(p,q)}pq+v_t^{(q,q)}{q}^2\right)\end{array}$$

for time-dependent coefficients $v_t^{\left(0\right)}, v_t^{\left(p\right)}, v_t^{\left(q\right)}, v_t^{(p,p)}, v_t^{(p,q)}$ and $v_t^{(q,q)}$ found as in Section IV of [25]. The solution of (34a) and (34b) can be found as

$$\begin{array}{c}\hfill V_t\left(\mathit{x}\right)=E\left(\phi \left({\mathcal{x}}_{t_f}\right)+{\displaystyle \frac{\beta \tau }{4 m}}{\int }_t^{t_f}s {\parallel {\partial }_q{U}_s\left({\mathcal{q}}_s\right)\parallel }^2 | {\mathcal{x}}_t=\mathit{x}\right)\end{array}$$

Applying the Bismut–Elworthy–Li formula with ${\mathcal{h}}_t$ gives the following expression for the gradient of the value function with respect to momentum:

$$\begin{array}{cc}\hfill {\partial }_p{V}_t\left(\mathit{x}\right)& =\sqrt{{\displaystyle \frac{\beta \tau }{2 m}}} E(\phi \left({\mathcal{x}}_{t_f}\right){\int }_t^{t_f}\langle \mathrm{d}{\mathcal{w}}_s , {\mathcal{h}}_s\rangle \hfill \\ \hfill & +{\displaystyle \frac{\beta \tau }{4 m}}{\int }_t^{t_f}\mathrm{d}s {\parallel \partial U_s\left({\mathcal{q}}_s\right)\parallel }^2{\int }_t^s\langle \mathrm{d}{\mathcal{w}}_u , {\mathcal{h}}_u\rangle | {\mathcal{x}}_t=\mathit{x})\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill {\mathcal{h}}_u& =-{\dot{\mathsf{\ell }}}_u-{\displaystyle \frac{{\mathsf{\ell }}_{\mathcal{u}}}{\tau }}-(\mathsf{\partial }\otimes \mathsf{\partial }{U}_u)\left({\mathcal{q}}_u\right) {\mathcal{g}}_u\hfill \\ \hfill & =-\mathit{v} {\displaystyle \frac{1}{\tau {(t_f-t)}^2}} ((t_f-u)(t_f+2t-3u)-\tau (4t_f-6u+2t)\hfill \\ \hfill & +{\displaystyle \frac{\tau }{m}} (\mathsf{\partial }\otimes \mathsf{\partial }{U}_u)\left({\mathcal{q}}_u\right){(t_f-u)}^2(u-t))\hfill \end{array}$$

(67)

The computation is summarized in Algorithm 4 and results shown in Figure 6.

Algorithm 4 BEL for degenerate diffusion

Initialize ${\mathcal{q}}_{t_n}=q\in \mathbb{R}$ Initialize ${\mathcal{p}}_{t_n}=p\in \mathbb{R}$ Initialize ${\iota }_1=0$ Initialize drift function ${\partial }_q{U}_t$ Initialize $h(u,t,T)$ as in (67) for i in $n,\dots ,N-1$ do    Sample Brownian noise: ${ϵ}_i\sim \mathcal{N}(0,{\delta }_t)$    if $i>n$ then      Add to running cost: ${\iota }_1={\iota }_1+{\delta }_t{\parallel {\partial }_q{U}_{t_i}\left({\mathcal{q}}_{t_i}\right)\parallel }^2\left({\sum }_{j=n}^i{ϵ}_j h(t_j,t_n,t_i)\right)$    end if    Evolve one step of (22): $\left\{\begin{array}{c}{\mathcal{q}}_{t_{i+1}}={\mathcal{q}}_{t_i}+{\displaystyle \frac{{\mathcal{p}}_{t_i}}{m}} {\delta }_t\hfill \\ {\mathcal{p}}_{t_{i+1}}={\mathcal{p}}_{t_i}-\left({\displaystyle \frac{{\mathcal{p}}_{t_i}}{\tau }}+{\partial }_q{U}_{t_i}\left({\mathcal{q}}_{t_i}\right)\right) {\delta }_t+\sqrt{{\displaystyle \frac{2 m}{\tau \beta }}} {ϵ}_i\hfill \end{array}\right.$ end for Return ${\partial }_p{V}_{t_n}\left(\mathcal{x}\right)=\sqrt{{\displaystyle \frac{\beta }{2}}}\left(\phi \left({\mathcal{x}}_{t_f}\right)\left({\sum }_{j=n}^{N-1}{ϵ}_j h(t_j,t_n,t_f)\right)+{\displaystyle \frac{\beta }{4}} {\iota }_1\right)$

6. Application to Machine Learning

In this section, we return to the overdamped dynamics and demonstrate an application of numerical methods we discuss above. We present a prototype example for the optimal control problem in the overdamped dynamics of minimizing the Kullback–Leibler divergence (16). Inspired by the seminal works [62,63], we model the optimal control protocol by a neural network, and use gradient descent to iteratively update it based on the stationarity condition (19).

As before, we formulate the problem in terms of a Bismut–Pontryagin cost functional. Additionally, we enforce the assigned boundary conditions (initial and final conditions on the density of the form (17a) and (17b)) through a Lagrangian multiplier $\lambda$. This gives

$$\begin{array}{cc}\hfill \mathcal{A}[\mathtt{p},U,V]& ={\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{q}\left(-V_{t_f}\left(\mathit{q}\right){\mathtt{p}}_{t_f}\left(\mathit{q}\right)+\lambda \left(\mathit{q}\right) ({\mathtt{p}}_{t_f}\left(\mathit{q}\right)-P_f\left(\mathit{q}\right))+V_{t_{\iota }}\left(\mathit{q}\right)P_{\iota }\left(\mathit{q}\right)\right)\hfill \\ \hfill & +{\int }_{t_{\iota }}^{t_f}\mathrm{d}t {\int }_{{\mathbb{R}}^d}{\mathrm{d}}^d\mathit{q} {\mathtt{p}}_t\left(\mathit{q}\right)\left({\displaystyle \frac{\beta \mu }{4}}{\parallel (\mathsf{\partial }{U}_t)\left(\mathit{q}\right)\parallel }^2+\left({\partial }_t-\mu \langle \left({\mathsf{\partial }}_{q}U\right)\left(\mathit{q}\right) , {\mathsf{\partial }}_q\rangle +{\displaystyle \frac{\mu }{\beta }}{\mathsf{\partial }}_q^2\right)V_t\left(\mathit{q}\right)\right)\hfill \end{array}$$

Taking stationary variation with respect to the density $\mathrm{p}$, control protocol U and value function V yields the coupled partial differential Equations (18a) and (18b), and the stationarity condition (19). We identify

$$V_{t_f}\left(\mathit{q}\right)=\lambda \left(\mathit{q}\right)$$

(68)

and the following update rule:

$${\lambda }^{\left(\mathrm{new}\right)}\left(\mathit{q}\right)={\lambda }^{\left(\mathrm{old}\right)}\left(\mathit{q}\right)+{\gamma }_1 \left(log{\displaystyle \frac{{\mathrm{p}}_{t_f}\left(\mathit{q}\right)}{P_f\left(\mathit{q}\right)}}\right)$$

(69)

chosen in this way to preserve the integrability conditions of the value function. The stationarity condition gives an update rule for the drift of the control protocol:

$${(\mathsf{\partial }{U}_t)}^{\left(\mathrm{new}\right)}\left(\mathit{q}\right)={(\mathsf{\partial }{U}_t)}^{\left(\mathrm{old}\right)}\left(\mathit{q}\right)-{\gamma }_2 \left({(\mathsf{\partial }{U}_t)}^{\left(\mathrm{old}\right)}\left(\mathit{q}\right)-{\displaystyle \frac{\beta }{2}}(\mathsf{\partial }{V}_t)\left(\mathit{q}\right)\right)$$

(70)

The parameters ${\gamma }_1,{\gamma }_2>0$ control the step sizes of the gradient descent, known as a learning rate. The update for the Lagrange multiplier is a gradient ascent rather than descent [64].

The right hand sides of both (69) and (70) can be computed using Monte Carlo integration techniques discussed in this article. With appropriate parametrization of the gradient of the control protocol and the Lagrangian multiplier $\lambda$, the method could therefore scale to high dimensions. In this prototype example, we use a polynomial regression for fitting $\lambda$ and a neural network for the gradient of the control protocol. The polynomial regression could be replaced with any suitable parametrization, in particular, with a second neural network.

The gradient of the optimal control protocol $\mathsf{\partial }{U}_t$ is modelled by a neural network, denoted by ${\mathbb{U}}_t$. We use a feed-forward neural network: connected layers, representing affine transformations with non-linear functions (known as activation functions) between them. The neural network has a set of parameters (weights and biases) associated with the layers, which we denote by $\mathsf{\Theta }$. The network takes the time t and space coordinates $\mathit{q}$ as input. Using a neural network allows for evaluating the optimal control protocol on new space coordinates without using interpolation, meaning that it can easily be used as the drift in the computation of the density and value functions using Algorithms 1 and 3.

The training process can be summarized as follows. Firstly, the functions $\lambda$ and $\mathbb{U}$ with a set of parameters ${\mathsf{\Theta }}^{\left(0\right)}$ are initialized. We use these to find the final density ${\mathrm{p}}_{t_f}$ under this drift with Algorithm 1. The Lagrange multiplier is updated using (69). The new $\lambda$ is used as the terminal condition of the value function. We then use Algorithm 3 to compute the gradient of the value function, ${\partial }_q{V}_t$, using the current drift and terminal condition. The neural network parameters $\mathsf{\Theta }$ are updated so that the new drift satisfies (70). Under the updated drift, the final density is recomputed and the process is repeated until convergence. The whole iteration is summarized in Algorithm 5.

The results of Algorithm 5 are illustrated in Figure 7. We show the final density in panel (a). Panels (b)–(g) show the approximation of the gradient of the control protocol by the trained neural network.

Algorithm 5 Learning an Optimal Control Protocol by Gradient Descent

Initialize a neural network $\mathbb{U}$ with parameters ${\mathsf{\Theta }}^{\left(0\right)}$ Initialize ${\lambda }^{\left(0\right)}$ as a polynomial with coefficients zero Initialize ${\gamma }_1,{\gamma }_2$ learning rates while $\ell \le L$ max iters do    Initialize a batch $\mathit{q}={\left\{q_k\right\}}_k$ of K points    Compute ${\mathrm{p}}_{t_f}^{(\ell )}\left(\mathit{q}\right)$ using Algorithm 1 with $\mathbb{U}$ as drift    Update ${\lambda }^{(\ell +1)}\left(\mathit{q}\right)={\lambda }^{(\ell )}\left(\mathit{q}\right)+{\gamma }_1\left(log{\displaystyle \frac{{\mathrm{p}}_{t_f}^{(\ell )}\left(\mathit{q}\right)}{P_f\left(\mathit{q}\right)}}\right)$    Set $V_{t_f}^{(\ell +1)}\left(\mathit{q}\right)={\lambda }^{(\ell +1)}\left(\mathit{q}\right)$    for $t_n$, $n=0,\dots ,N$ do      Compute $(\partial V_{t_n}^{(\ell +1)})\left(\mathit{q}\right)$ using Algorithm 3 with $\mathbb{U}$ as drift      Update ${\mathsf{\Theta }}^{(\ell +1)}$ such that $\parallel \mathbb{U}(\mathit{q},t_n; {\mathsf{\Theta }}^{(\ell +1)})-{\displaystyle \frac{\beta }{2}} (\partial V_{t_n}^{(\ell +1)})\left(\mathit{q}\right){\parallel }^2$ is minimized    end for end whilereturn Approximation of the gradient of the optimal control protocol $\mathbb{U}$

7. Conclusions

In this article, we discuss two integration methods for partial differential equations which frequently appear in optimal control problems. We show how we can use the Girsanov theorem such that a Fokker–Planck equation driven by a mechanical potential can be integrated by taking a numerical expectation of Monte Carlo trajectories of an auxiliary stochastic process. This method can be applied when the auxiliary stochastic process is non-degenerate or degenerate. Secondly, we use the Bismut–Elworthy–Li formula to find expressions for the gradient of the value function satisfying a Hamilton–Jacobi–Bellman equation. We show this for both a non-degenerate and degenerate diffusion.

The discussed numerical methods are supported by computational examples. We examine the dynamic Schrödinger bridge problem, or the minimization of the Kullback–Leibler divergence from a free diffusion while satisfying boundary conditions on the density at the initial and final time. For the overdamped dynamics, our integration shows good agreement with the iterative approach of Caluya and Halder [41] in Figure 2 and Figure 5. In the underdamped case, we integrate the associated Fokker–Planck equation to support the consistency of the multiscale perturbative approach used in [25]. In particular, we compute an estimate of the evolution of the joint density function of the system state for this problem in Figure 3. We also verify the stationarity condition using the Bismut–Elworthy–Li for a degenerate diffusion in Figure 6. Finally, we demonstrate an application of both integrations in a simple machine learning model in Figure 7.

The optimal control problem discussed here has many applications. One possibility is application in machine learning, for instance in the development of diffusion models for image generation [44]. Here, we find an optimal steering protocol between a noise distribution (e.g., a Gaussian) and a target (e.g., an image) by minimizing the Kullback–Leibler divergence. Optimal control problems in the underdamped dynamics are particularly interesting. Underdamped dynamics take into account random thermal fluctuations, noise and the effects of inertia; hence, they are well suited to model non-equilibrium transitions at nanoscale. Models of certain biological systems require considering complex dynamics, for example, because of random external noise from the environment [30]. Such models then result in non-linear partial differential equations, making them difficult to integrate. While the implementation of machine learning to solve an optimal control problem we use here is a prototype, it may be possible to extend it to a more general setting. Specifically, we have in mind transitions obeying underdamped dynamics and occurring at minimum entropy production such as those considered in [53].