Fractional Stochastic Search Algorithms: Modelling Complex Systems via AI

Abstract

The aim of this article is to establish a stochastic search algorithm for neural networks based on the fractional stochastic processes $\{B_t^H,t\ge 0\}$ with the Hurst parameter $H\in (0,1)$. We define and discuss the properties of fractional stochastic processes, $\{B_t^H,t\ge 0\}$, which generalize a standard Brownian motion. Fractional stochastic processes capture useful yet different properties in order to simulate real-world phenomena. This approach provides new insights to stochastic gradient descent (SGD) algorithms in machine learning. We exhibit convergence properties for fractional stochastic processes.

1. Introduction

The gradient descent methodology is not computationally efficient in all applications. Sometimes, optimization algorithms become stuck in the flat regions of manifolds. In those cases, the optimization algorithm requires a long time to escape. This is the challenge of a vanishing gradient where, for instance, $\nabla z\left(\theta \right)$ is almost zero (see Section 4). The method of stochastic gradient descent (SGD) generally overcomes this problem.

Recent advancements in the field of factional stochastic processes exhibit the theoretical benefits of modeling complex systems [1,2,3,4]. Yet, so far no literature exists on fractional stochastic gradient descent (fSGD) or fractional stochastic networks. This paper sketches the potential of such new literature for modeling complex systems. Moreover, we exhibit that fractional stochastic processes are an advancement in machine learning (ML) and artificial intelligence (AI).

The methodology of fractional stochastic gradient descent and the role of stochastic neural networks are based on a generalized assumption of randomness. Mandelbrot and Van Ness defined a fractional Brownian motion (fBM), $B_t^H$, together with a Hurst parameter in 1968 [5]. For $H=\frac{1}{2}$, we obtain a standard Browning motion. Yet, for $H\ne \frac{1}{2}$, we obtain new forms of randomness or stochastic processes that match real-world phenomena.

The new feature of a fractional Brownian motion (fBM) is that increments are interdependent. In the literature, this is called self-similarity. A self-similar stochastic process reveals invariance with respect to the time scale (scaling invariance). A standard Brownian motion or a Lévy process displays different properties. They have independent increments and belong to the famous class of Markov processes.

However, in science, there is ubiquitous evidence that fractional stochastic processes are of relevance. For instance, we frequently observe probability densities with sharp peaks, which is related to the phenomena of long-range interdependence. In many real-world observations and applications, we find the presence of interdependence, too. This pattern can be captured by fractional stochastic processes.

Nonetheless, some phenomena are even more complicated and require further generalization towards sub-fractional stochastic processes. The literature on sub-fBM’s demonstrates that those stochastic processes are useful in scientific applications [6]. A sub-fractional Brownian motion provides a nexus between a Brownian motion and fractional stochastic process. Those processes were introduced by Tudor et al. [7,8] and Bojdecki et al. [9]. Note that, as sub-fractional stochastic processes are not martingale processes, the basic tools of stochastic analysis are insufficient. However, researchers have developed new machinery to handle fractional stochastic processes, such as [10] or [11,12,13,14,15].

In this paper, our purpose is to develop and study the idea of fractional stochastic gradient descent algorithms. Our approach generalizes the existing literature on stochastic gradient descent (SGD) and stochastic neural networks. For instance, Hopfield [16] developed neural networks consisting of several perceptrons with randomness. Similarly, a Boltzmann network is a type of stochastic neural network wherein the output of the activation function is interpreted as a probability.

A study already exists about stochastic gradient descent and its challenges in machine learning [17,18]. Recent developments in the theory and applications of stochastic gradient descent are discussed in the following papers: Schmidt et al. [19], Haochen and Sra [20], Gotmare et al. [21], Curtis and Scheinberg [22], de Roos et al. [23]. The focus of our research is the motivation of fractional stochastic gradient descent (fSGD) algorithms. Thus, our research is beyond the scope of current literature and focuses on the possibility of fractional stochastic gradient descent in theory. We neglect potential computational limitations in machine learning.

The paper is organized as follows. Section 2 provides preliminary definitions. Subsequently, we introduce the foundations of fractional stochastic processes in Section 3. Section 4 introduces the idea of fractional stochastic search algorithms and derives the convergence results in general. Finally, in Section 5, we apply the method to two different cases. Section 6 concludes the paper.

2. Preliminaries

Machine learning is mainly based on neural networks and efficient optimization algorithms. The most primitive neural model is inspired by the work of Rosenblatt [24]. In the following section, we define the major elements from a machine learning perspective.

Definition 1.

A stochastic neuron is defined by n-inputs $\mathit{X}={(x_1,\dots ,x_n)}^T$, n-weighting factors $\mathit{W}={(w_1,\dots ,w_n)}^T$ and an n-dimensional vector of biases $\mathit{B}$ together with a sigmoid activation function ${\sigma }_c\left(\zeta \right)=\frac{1}{1+e^{-c\zeta }}$ with $c\ge 0$ and a stochastic output $\mathit{Y}$

$$\mathit{Y}=\sigma \left({\mathit{W}}^{\mathit{T}}\mathit{X}+\mathit{B}\right),$$

where $\sigma \left({\mathit{W}}^{\mathit{T}}\mathit{X}+\mathit{B}\right)\in (0,1)$. Hence, we define the output for $Y=1$ by $Prob(\mathit{Y}=\mathbf{1};\mathit{X})=\sigma \left({\mathit{W}}^{\mathit{T}}\mathit{X}+\mathit{B}\right)$ and for $Y=0$ by the inverse probability: $Prob(\mathit{Y}=\mathbf{0};\mathit{X})=1-\sigma \left({\mathit{W}}^{\mathit{T}}\mathit{X}+\mathit{B}\right)=\sigma \left(-{\mathit{W}}^{\mathit{T}}\mathit{X}-\mathit{B}\right)$.

Note that, if the activation potential is greater than zero, such as ${\mathit{W}}^{\mathit{T}}\mathit{X}+\mathit{B}>0$, then this neural network is not necessarily activated according to Definition 1. An activation value of one only occurs with the probability of the activation function.

In machine learning, the gradient descent algorithm is omnipresent in all optimization problems. Yet, it does not provide robust solutions in each case. There are computational obstacles, such as when the algorithm becomes stuck in a local minimum or lost in a plateau from which it takes a long time to get out. A plateau is defined as a flat surface region where the gradient $\nabla {\sigma }_c$ is very small (or almost zero).

The optimization algorithm of a neural network always has the goal of finding the optimal weighting parameters $\mathit{\theta }=(\mathbf{W},\mathbf{B})$. The standard algorithm used to optimize the parameters is frequently reformulated in order to minimize the cost function. This is called the gradient descent method. This method is closely related to Newton’s algorithm in numerical computing. The following definition summarizes the algorithm from a machine learning vantage point.

Definition 2.

The gradient descent algorithm is defined by

$${\mathit{\theta }}_{\mathit{t}+\mathbf{1}}={\mathit{\theta }}_{\mathit{t}}-{\lambda }_t{\mathbf{C}}_t{\mathbf{g}}_t,$$

(1)

where ${\mathbf{g}}_t=\nabla \mathcal{L}\left({\mathit{\theta }}_{\mathit{t}}\right)$ is the gradient of a cost function, ${\mathbf{C}}_t$ is an optional conditioning matrix, and ${\lambda }_t$ is the learning rate.

The stochastic gradient descent (SGD) method overcomes the obstacle if the gradient is close to zero. Indeed, SGD reaches a minimum along a non-linear stochastic process. In the following sections, we first discuss the literature and then generalize the approach to fractional stochastic processes.

3. Fractional Stochastic Processes

3.1. General Definitions

Consider a stochastic process with a Hurst parameter H. Subsequently, we define the elementary tools in fractional calculus.

Definition 3.

Let $a,b\in \mathbb{R}$, $a<b$. Let $f\in L^1(a,b)$ and $\alpha >0$. The left- and right-sided fractional integrals of f of order α are defined for $x\in (a,b)$, respectively, as

$${}_a{D}_x^{-\alpha }f\left(x\right)={}_a{I}_x^{\alpha }f\left(x\right)=\frac{1}{\mathsf{\Gamma }\left(\alpha \right)}{\int }_a^x{(x-u)}^{\alpha -1}f\left(u\right)du-\infty \le a\le x,$$

and

$${}_b{D}_x^{-\alpha }f\left(x\right)={}_b{I}_x^{\alpha }f\left(x\right)=\frac{1}{\mathsf{\Gamma }\left(\alpha \right)}{\int }_x^b{(u-x)}^{\alpha -1}f\left(u\right)du-\infty \le x\le b.$$

This is the fractional integral of the Riemann–Liouville type. In the same vein, we define factional derivatives where we distinguish between left- and right-sided derivatives.

Definition 4.

The factional left- and right-sided derivatives, for $f\in I_a^{\alpha }\left(L^p\right)$ and $0<\alpha <1$, are defined by

$${}_a{I}_x^{-\alpha }f\left(x\right)={}_a{D}_x^{\alpha }f\left(x\right)=\frac{1}{\mathsf{\Gamma }(1-\alpha )}\left(\frac{d}{dx}\right){\int }_a^x{(x-u)}^{-(\alpha +1)}f\left(u\right)du$$

(2)

and

$${}_b{I}_x^{-\alpha }f\left(x\right)={}_b{D}_x^{\alpha }f\left(x\right)=\frac{{(-1)}^{\alpha }}{\mathsf{\Gamma }(1-\alpha )}\left(\frac{d}{dx}\right){\int }_x^b{(u-x)}^{-(\alpha +1)}f\left(u\right)du,$$

(3)

for all $x\in (a,b)$ and $I_a^{\alpha }\left(L^p\right)$ is the image of $L^p(a,b)$.

Let us assume $f\in I_a^1\left(L^1\right)$, then we obtain

$${}_a{D}_x^{\alpha }{}_a{D}_x^{1-\alpha }f\left(x\right)=Df\left(x\right),{}_b{D}_x^{\alpha }{}_b{D}_x^{1-\alpha }f\left(x\right)=Df\left(x\right).$$

(4)

Notably, $D^{\alpha }f\left(x\right)$ exists for all $f\in C^{\beta }\left([a,b]\right)$ if $\alpha <\beta$. Given those definitions, we are ready to define a Brownian motion:

Definition 5.

Let H be $0<H<1$, and let $B_0$ be an arbitrary real number. We call $B^H(t,\omega )$ a fractional Brownian motion (fBM) with Hurst parameter H and starting value $B_0$ at time 0, such as

1. 

$B^H(0,\omega )=B_0$, and;

2. 

$B^H(t,\omega )-B^H(0,\omega )=\frac{1}{\mathsf{\Gamma }(H+\frac{1}{2})}[{\int }_{-\infty }^0[{(t-s)}^{H-\frac{1}{2}}-{(-s)}^{H-\frac{1}{2}}]dB(s,\omega )+$

${\int }_0^t{(t-s)}^{H-\frac{1}{2}}dB(s,\omega )]$ [Wyle fractional integral];

3. 

Equivalent to the Riemann–Liouville integral:

$B^H(t,\omega )-B^H(0,\omega )=\frac{1}{\mathsf{\Gamma }(H+\frac{1}{2})}{\int }_0^t{(t-s)}^{H-\frac{1}{2}}dB(s,\omega )$.

Next, let us consider the following corollary:

Corollary  1.

Consider $H=\frac{1}{2}$ and $B_0=0$. Then the Brownian motion is of $B(t,\omega )=B^{\frac{1}{2}}(t,\omega )$.

Proof. 

Let $H=\frac{1}{2}$, we find $B^{\frac{1}{2}}(t,\omega )-B^{\frac{1}{2}}(0,\omega )=\frac{1}{\mathsf{\Gamma }\left(1\right)}{\int }_0^{t}dB(s,\omega )=B(t,\omega ).$ □

In the literature, there exists an alternative, yet useful, definition:

Definition 6.

A fractional Brownian motion is a Gaussian process $B^H\left(t\right)$ for $t\ge 0$ defined by the following covariance function

$$R^{fBM}(t,s)=\mathbb{E}\left[B^H\left(t\right)B^H\left(s\right)\right]=\frac{1}{2}{\left[\right|t|}^{2H}+{\left|s\right|}^{2H}{-|t-s|}^{2H}],$$

(5)

where the Hurst index is denoted by $H\in (0,1)$.

Since the covariance of a Brownian motion is given in the literature, it is easy to extend the definition to an fBM with Hurst index H, such as

$$\begin{array}{cc}\hfill Var\left[B\right(t)-B(s\left)\right]& =\mathbb{E}\left[{(B\left(t\right)-B\left(s\right))}^2\right]=|t-s|\hfill \\ \hfill \iff Var[B^H\left(t\right)-B^H\left(s\right)]& =\mathbb{E}\left[{(B^H\left(t\right)-B^H\left(s\right))}^2\right]={|t-s|}^{2H},\hfill \end{array}$$

where we obtain the definition of a Brownian motion for $H=\frac{1}{2}$. Following Herzog [15], we derive the covariance step-by-step:

$$\begin{array}{cc}\hfill Cov\left[B^H\left(t\right)B^H\left(s\right)\right]& =\mathbb{E}\left[(B^H\left(t\right)-\mathbb{E}\left[B^H\left(t\right)\right])(B^H\left(s\right)-\mathbb{E}\left[B^H\left(s\right)\right])\right]=\mathbb{E}\left[B^H\left(t\right)B^H\left(s\right)\right]\hfill \\ \hfill & =\frac{1}{2}\left[\mathbb{E}\left[B^H{\left(t\right)}^2\right]+\mathbb{E}\left[B^H{\left(s\right)}^2\right]-\mathbb{E}\left[{(B^H\left(t\right)-B^H\left(s\right))}^2\right]\right]\hfill \\ \hfill & =\frac{1}{2}{\left[\right|t|}^{2H}+{\left|s\right|}^{2H}{-|t-s|}^{2H}].\hfill \end{array}$$

Corollary  2.

Consider a fractional Brownian motion. The expectation values of non-overlapping increments are $\mathbb{E}[B^H\left(t\right)-B^H\left(s\right)]\ne 0$ and the variance is of $\mathbb{E}\left[{(B^H\left(t\right)-B^H\left(s\right))}^2\right]={|t-s|}^{2H}$ for all $t,s\in \mathbb{R}$

Proof. 

See [15]. □

3.2. Properties

Next, we consider the properties of the fBM over time for different Hurst parameters. Suppose $0<H<\frac{1}{2}$ or $\frac{1}{2}<H<1$. If we assume that the Hurst parameter is of $0<H<\frac{1}{2}$, we say the fractional stochastic process has a short memory. Conversely, if $\frac{1}{2}<H<1$, we obtain the property of long-range dependence. Figure 1 illustrates sample processes for the three ranges of the Hurst parameters, H.

Figure 1. Different fractional Brownian motions with the following Hurst index: (Left-panel) $H=0.25$, (middle-panel) $H=0.50$ (standard BM), and (right-panel) $H=0.75$.

Proposition 1.

Given a fractional Brownian motion, we obtain the following properties:

1. 

The fBM has stationary increments: $B_t^H-B_s^H\stackrel{\mathit{dis}.}{=}{B}_u^H-B_s^H$;

2. 

The fBM is H-self-similar, such as $B^H\left(at\right)=a^H{B}^H\left(t\right)$;

3. 

The fBM is H-self-similar, such as $B^H\left(at\right)=a^H{B}^H\left(t\right)$;

Proof. 

The proof follows Herzog [15]. In order to prove the stationary of increments, we set $t_1<t_2<t_3<t_4$. The equality of the covariance implies $Y:=B^H\left(t_2\right)-B^H\left(t_1\right)$. Moreover, it has the same distribution, such as $X:=B^H\left(t_4\right)-B^H\left(t_3\right)$. Subsequently, we find

$$\begin{array}{cc}\hfill \mathbb{E}\left[{(B^H\left(t_2\right)-B^H\left(t_1\right))}^2\right]& ={(t_2-t_1)}^{2H}={\left(\Delta t\right)}^{2H}\hfill \\ \hfill \mathbb{E}\left[{(B^H\left(t_4\right)-B^H\left(t_3\right))}^2\right]& ={(t_4-t_3)}^{2H}={(\Delta t)}^{2H},\hfill \end{array}$$

where $t_1<t_2$ and $t_3<t_4$ with $\Delta t=t_2-t_1=t_4-t_3$. This demonstrates that the increments and the time evolution of the increments are the same at any given point. Consequently, we obtain stationary increments.

The second property of Proposition 1 is self-similarity. Consider the following definition,

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\left(B^H\left(at\right)\right)}^2\right]& =\frac{1}{2}[{\left(at\right)}^{2H}+{\left(at\right)}^{2H}-{(at-at)}^{2H}]={\left(at\right)}^{2H}=a^{2H}{t}^{2H}\hfill \\ \hfill & =a^{2H}\mathbb{E}\left[{\left(B^H\left(t\right)\right)}^2\right].\hfill \end{array}$$

Here, we find that ${\left(B^H\left(at\right)\right)}^2=a^{2H}{\left(B^H\left(t\right)\right)}^2$ and $B^H\left(at\right)=a^H{B}^H\left(t\right)$. Part (3) is already given in Corollary 2. □

3.3. Definition of Sub-Fractional Processes

In a recent paper, Herzog [15] described a sub-fractional Brownian motion (sub-fBM) as an intermediate between a Brownian motion and a fractional Brownian motion. Without loss of generality, a sub-fBM is a self-similar Gaussian process. Note that both the fBM and sub-fBM have the properties of self-similarity and long-range dependence, yet a sub-fBM does not have stationary increments [9].

Any Brownian motion is uniquely defined by its covariance. For the sub-fBM we denote covariance by $Cov({\xi }_t^H,{\xi }_s^H)$.

Definition 7.

Consider a sub-fractional Brownian motion with Hurst parameter H and a centered mean zero Gaussian process ${\xi }^H=\{{\xi }_t^H,t\ge 0\}$ with the following covariance function

$$R^{sfBM}(t,s):=\mathbb{E}\left[{\xi }_t^H{\xi }_s^H\right]=s^{2H}+t^{2H}-\frac{1}{2}[{(s+t)}^{2H}{+|s-t|}^{2H}],$$

(6)

where ${\xi }_0^H=0$ and $\mathbb{E}\left[{\xi }_t^H\right]=0$.

Note, a fractional Brownian motion coincides with a Brownian motion if the Hurst parameter is $H=\frac{1}{2}$. Thus, a Brownian motion on the real line has a covariance of $Cov({\xi }_t^H,{\xi }_s^H)=s\wedge t:=min[s,t]$. The process ${\xi }_t^H$ has the following representation for $H>\frac{1}{2}$ (see [25]):

$$\begin{array}{cc}\hfill {\xi }_t^H& ={\int }_0^t{K}^H(t,s)d{W}_s,\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill K^H(t,s)& =c_H\left(H-\frac{1}{2}\right)s^{1/2-H}{\int }_s^t{(u-s)}^{H-3/2}{u}^{H-1/2}du.\hfill \end{array}$$

(8)

The kernel function of a sub-fractional Brownian motion is given by

$${\varphi }^{sfBM}(s,t)=\frac{{\partial }^{2}Cov({\xi }_t^H,{\xi }_s^H)}{\partial s\partial t}=H(2H-1)\left[{|s-t|}^{2H-2}-{(s+t)}^{2H-2}\right].$$

(9)

3.4. Properties of Sub-Fractional Processes

In this subsection, we reiterate useful properties of sub-fractional Brownian motions such as those described in Herzog [15].

Lemma 1.

Consider ${\xi }_t^H$ be a sub-fBM for all t. The properties of the sub-fBM are:

1. 

$\mathbb{E}\left[{\left({\xi }_t^H\right)}^2\right]=(2-2^{2H-1})t^{2H}$.

2. 

$\mathbb{E}\left[{({\xi }_t^H-{\xi }_s^H)}^2\right]=-2^{2H-1}(t^{2H}+s^{2H})+{(t+s)}^{2H}+{(t-s)}^{2H}$.

3. 

If $H\ne \frac{1}{2}$, then ${\xi }_t^H-{\xi }_s^H\stackrel{\mathit{dis}.}{\ne }{\xi }_u^H-{\xi }_s^H$, i.e., the increments are non-stationary.

Proof. 

See [15]. □

Finally, we follow Herzog [15] and prove the following proposition:

Proposition 2.

Let $B_t^H$ be a fractional Brownian motion and ${\xi }_t^H$ be a sub-fractional Brownian motion. For $H\in (\frac{1}{2},1)$, the following holds:

1. 

$\mathbb{E}\left[{\left({\xi }_t^H\right)}^2\right]<\mathbb{E}\left[{\left(B_t^H\right)}^2\right]$;

2. 

$R_{{\xi }_t^H}(s,t)\le R_{B_t^H}(s,t)$.

Proof. 

Obviously, an fBM has the following variance: $Var\left[B_t^H\right]={\left|t\right|}^{2H}$. Similarly, we obtain the variance of $Var\left[{\xi }_t^H\right]=(2-2^{2H-1}){\left|t\right|}^{2H}$ for a sub-fBM. Subsequently, we have $0<(2H-1)ln2$ if $H>\frac{1}{2}$.

The second part follows for $s,t>0$:

$$\begin{array}{cc}\hfill s^{2H}+t^{2H}-\frac{1}{2}{[(s+t)}^{2H}{+|t-s|}^{2H}]& \le \frac{1}{2}{\left[\right|t|}^{2H}+{\left|s\right|}^{2H}{-|t-s|}^{2H}]\hfill \\ \hfill s^{2H}+t^{2H}& \le {(s+t)}^{2H}.\hfill \end{array}$$

In the case of $s=t=0$ or $s=0,t\ne 0$, we have equality. □

4. Fractional Stochastic Search

Let ${\mathit{X}}_{\mathit{t}}$ be a an m-dimensional stochastic process driven by a fractional Brownian motion ${\mathit{B}}_{\mathit{t}}^{\mathit{H}}$, where $H=\frac{1}{2}$. The respective stochastic process ${\mathit{X}}_{\mathit{t}}$ is as follows:

$$d{\mathit{X}}_{\mathit{t}}=a\left({\mathit{X}}_{\mathit{t}}\right)dt+\sigma \left({\mathit{X}}_{\mathit{t}}\right)d{\mathit{B}}_{\mathit{t}}^{\mathit{H}},\mathrm{and}{\mathit{X}}_{\mathbf{0}}=x_0,$$

(10)

where ${\mathit{X}}_{\mathbf{0}}$ is the initial value and ${\mathit{B}}_{\mathit{t}}^{\mathit{H}}=(B_1^H\left(t\right),\dots ,B_m^H\left(t\right))$ is an m-dimensional fractional Brownian motion. Next, consider a cost function $z:{\mathbb{R}}^m\to \mathbb{R}$ which needs to be optimized. Hence, we study the vector field for which the auxiliary function $z\left(\mathit{X}\left(\mathit{t}\right)\right)$ is decreasing. This requires us to find the expectation value:

$$\kappa \left(t\right)=\mathbb{E}\left[z\right(\mathit{X}\left(\mathit{t}\right)\left)\right].$$

(11)

Thus, the function $z\left(\mathit{X}\left(\mathit{t}\right)\right)$ is stochastic and dependent on time t. In general, an optimization algorithm of a neural network minimizes the expectation value of this function. Utilizing the machinery of stochastic analysis, Dynkin’s formula, among others, and following the approach described in [26], we obtain

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \kappa \left(t\right)& =\hfill & z\left({\mathit{X}}_{\mathbf{0}}\right)+{\int }_0^t\mathbb{E}\left[\mathcal{A}\left(z\left(\mathit{X}\left(\mathit{s}\right)\right)\right)\right]ds\\ \hfill \kappa (t+dt)& =\hfill & z\left({\mathit{X}}_{\mathbf{0}}\right)+{\int }_0^{t+dt}\mathbb{E}\left[\mathcal{A}\left(z\left(\mathit{X}\left(\mathit{s}\right)\right)\right)\right]ds,\end{array}\end{array}$$

(12)

where the operator $\mathcal{A}={\sum }_k{a}_k\frac{\partial }{\partial X_k}+\frac{1}{2}{\sum }_{i,j}\left(\sigma {\sigma }^T\right)\frac{{\partial }^2}{\partial X_i\partial X_j}$. The usage of Taylor-series approximation and the differencing of Equations (12) yields

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \Delta \kappa \left(t\right)& =\hfill & \kappa (t+dt)-\kappa \left(t\right)={\int }_t^{t+dt}\mathbb{E}\left[\mathcal{A}\left(z\left(\mathit{X}\left(\mathit{s}\right)\right)\right)\right]ds\\ & =\hfill & \mathbb{E}\left[\mathcal{A}\left(z\left(\mathit{X}\left(\mathit{t}\right)\right)\right)\right]+\mathcal{O}\left(d{t}^2\right).\end{array}\end{array}$$

(13)

The method of steepest descent computes the gradient of $\Delta \kappa \left(t\right)$ such that the process ${\mathit{X}}_{\mathit{t}}$, is as negative as possible. However, if ${\mathit{X}}_{\mathit{t}}$ is a stochastic process, we need to study the expectation of the gradient, particularly where the value $\Delta \kappa \left(t\right)$ is as negative as possible, such as $\mathbb{E}\left[\mathcal{A}\right(z\left(\mathit{X}\left(\mathit{t}\right)\right)\left)\right]<0$.

In order to construct a stochastic process ${\mathit{X}}_{\mathit{t}}$ with this property, we specify $a\left({\mathit{X}}_{\mathit{t}}\right)$ and $\sigma \left({\mathit{X}}_{\mathit{t}}\right)$ in Equation (10), respectively. Next, we specify the diffusion term in Equation (10), $\sigma$, or the product $\sigma {\sigma }^T$, which is a matrix, such that the algorithm in Equation (1) converges efficiently. Indeed, if we set the term of $\sigma {\sigma }^T$ as being inversely proportional to the Hessian matrix, $\mathcal{H}$, then ${\left(\sigma {\sigma }^T\right)}_{ij}=\tau {\left({\mathcal{H}}_z\right)}_{ij}^{-1}$ with $\tau >0$. Through this process we can show the convergence of the algorithm and the existence of the solution.

Given that function z is of class ${\mathcal{C}}^2$ and strictly convex, then, according to [27], the Hessian matrix ${\mathcal{H}}_z$ is symmetric, real, positive definite, and non-degenerate. This guarantees that the Hessian matrix ${\mathcal{H}}_z$ has an inverse, which is also positive definite. Efficient computation can be achieved by utilizing the Cholesky decomposition. One can show that the diffusion term $\sigma$ is a lower triangular matrix satisfying ${\left(\sigma {\sigma }^T\right)}_{ij}=\tau {\left({\mathcal{H}}_z\right)}_{ij}^{-1}$. Under those conditions, we compute $\mathcal{A}z\left(x\right)$

$$\begin{array}{ccc}\hfill \mathcal{A}z\left(x\right)& =& \sum _k{a}_k\frac{\partial z\left(x\right)}{\partial X_k}+\frac{1}{2}\tau \sum _{i,j}{\left({\mathcal{H}}_z^{-1}\right)}_{ij}{\left({\mathcal{H}}_z\right)}_{ij}\hfill \\ & =& ⟨\nabla z\left(x\right),a\left(x\right)⟩+\frac{n}{2}\tau .\hfill \end{array}$$

In order to minimize the gradient, $\mathcal{A}z\left(x\right)$, we have to minimize the first-term, because the second term is a constant. Choosing $a\left(x\right)=-\lambda \nabla z\left(x\right)$ and assuming $\mathcal{A}z\left(x\right)<0$ obtains the following condition:

$$\begin{array}{ccc}\hfill \mathcal{A}z\left(x\right)& =& ⟨\nabla z\left(x\right),-\lambda \nabla z\left(x\right)⟩+\frac{n}{2}\tau \hfill \\ & =& -{\lambda \parallel \nabla z\left(x\right)\parallel }^2+\frac{n}{2}\tau <0\hfill \\ \hfill \frac{\tau }{\lambda }& <& \frac{2}{n}{\parallel \nabla z\left(x\right)\parallel }^2.\hfill \end{array}$$

Using the square vector norm and the assumption of $\xi ={inf}_x{\parallel \nabla z\left(x\right)\parallel }^2\ne 0$, it is sufficient to set $\tau$ and $\lambda$ as the main parameters in the SGD algorithm, such that $\tau =\frac{2\xi }{n}\lambda$. In the sequel, we apply this algorithm to fractional stochastic search problems.

5. Application of Fractional Stochastic Search

In this section, we demonstrate the working of a fractional stochastic search. We exhibit the convergence of a fractional stochastic search within neural networks.

5.1. Stochastic Search: Case I

Suppose that we have a neural network with the following cost function: $z\left(x\right)=\frac{1}{2}{x}^2$ for $x\in (a_1,a_2)$ with $a_i>0$ and $a_1<a_2$. The stochastic gradient descent method searches the minimum of this cost function.

Mathematically, the solution is obvious for this problem. The first derivative is of $z^{\prime }\left(x\right)=x$ for $x\in (a_1,a_2)$. Hence, the minimum is at $z^{\prime }\left(a_1\right)=a_1$, and consequently, the minimum value is $z\left(a_1\right)=\frac{1}{2}{a}_1^2$. Next, we show that we can obtain the same value under a fractional stochastic search algorithm in a neural network.

In step one, we establish an adequate stochastic differential equation according to $d{X}_t=a\left(X_t\right)dt+\sigma \left(X_t\right)d{B}_t^H$ for $X_0=x_0\in (a_1,a_2)$. The gradient of the cost function, which is equal to the first derivative $\nabla z\left(x\right)=x$, as well as the Hessian of the cost function and the second derivative, is defined as $\mathcal{H}\left(x\right)={\nabla }^{2}z\left(x\right)=1$. Both conditions enable us to compute the Lipschitz continuous coefficient functions. For $a\left(X_t\right)=-\lambda \nabla z\left(x\right)=-\lambda x$. For ${\sigma }^2\left(X_t\right)=\tau {\mathcal{H}}^{-1}\left(x\right)=\frac{\tau }{{\nabla }^{2}z\left(x\right)}=\frac{\tau }{1}=\tau$. Hence, we obtain $\sigma \left(X_t\right)=\sqrt{\tau }$. The stochastic differential equation for $H=\frac{1}{2}$ has the form:

$$d{X}_t=-\lambda X_{t}dt+\sqrt{\tau }d{B}_t^{\frac{1}{2}}.$$

(14)

The SDE in Equation (14) is an Ornstein–Uhlenbeck process driven by a Brownian motion $B^H\left(t\right)$ with the Hurst parameter $H=\frac{1}{2}$ [28].

The solution is divided into two parts: In part one, we solve the non-stochastic problem $d{X}_t=-\lambda X_{t}dt$. This is an ordinary differential equation and has the solution $X_t=X_0{e}^{-\lambda t}$. In part two, we define an auxiliary function $Y_t=X_t{e}^{\lambda t}$ and apply the Itô-Doeblin’s lemma:

$$\begin{array}{ccc}\hfill d{Y}_t& =& X_t{e}^{\lambda t}\left(\lambda \right)dt+e^{\lambda t}d{X}_t\hfill \\ & =& X_t{e}^{\lambda t}\left(\lambda \right)dt+e^{\lambda t}[-\lambda X_{t}dt+\sqrt{\tau }d{B}_t^H]\hfill \\ & =& \sqrt{\tau }{e}^{\lambda t}d{B}_t^H.\hfill \end{array}$$

Note that, in this case, the derivation coincides with a standard Brownian motion. Next, integrating the last line yields $Y_t-Y_0=\sqrt{\tau }{\int }_0^t{e}^{\lambda s}d{B}_s^H$. Hence, we obtain $X_t=Y_t{e}^{-\lambda t}$, which is

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill X_t& =\hfill & \left[Y_0+\sqrt{\tau }{\int }_0^t{e}^{\lambda s}d{B}_s^H\right]e^{-\lambda t}\\ & =\hfill & X_0{e}^{-\lambda t}+\sqrt{\tau }{\int }_0^t{e}^{-\lambda (t-s)}d{B}_s^H.\end{array}\end{array}$$

(15)

Based on Equation (15), we find the expectation of $\mathbb{E}\left[X_t\right]=X_0{e}^{-\lambda t}$. Note that the expected integral of a Brownian motion is zero. For $t\to \infty$, the expected value is ${lim}_{t\to \infty }\mathbb{E}\left[X_t\right]=0$. Next, utilizing the general condition of $\tau =\frac{2\xi }{n}\lambda$ for $\xi ={inf\parallel \nabla z\left(x\right)\parallel }^2$ and $n=1$ in Section 4, we obtain

$$\tau =\frac{2\xi }{n}\lambda =\frac{2inf\parallel a_1{\parallel }^2}{1}\lambda =2a_1^2\lambda \iff \frac{\tau }{2\lambda }=a_1^2.$$

Finally, it remains to show that the SDE in Equation (14) converges to the minimum value. Hence, we study the convergence sequence:

$$\begin{array}{ccc}\hfill \kappa \left(t\right)& =& \mathbb{E}\left[z\left(X_t\right)|X_0\right]=\frac{1}{2}\mathbb{E}\left[{\left(X_t\right)}^2|X_0\right]=\hfill \\ & =& \frac{1}{2}\mathbb{E}\left[X_0^2{e}^{-2\lambda t}+2X_0{e}^{-\lambda t}\sqrt{\tau }{\int }_0^t{e}^{-\lambda (t-s)}d{B}_s^H+\tau {\left({\int }_0^t{e}^{-\lambda (t-s)}d{B}_s^H\right)}^2\right],\hfill \end{array}$$

where, for $\mathbb{E}\left[{\left(X_t\right)}^2\right]$, we have substituted Equation (15). Next, we use the property that the expected stochastic integral is zero and the variance of the Brownian motion is $Var\left(d{B}_s^H\right)=ds$. Thus, we obtain

$$\begin{array}{ccc}\hfill \kappa \left(t\right)& =& \frac{1}{2}\left[X_0^2{e}^{-2\lambda t}+\tau {\int }_0^t{e}^{-2\lambda (t-s)}ds\right]=\hfill \\ & =& \frac{1}{2}\left[X_0^2{e}^{-2\lambda t}+\tau \frac{e^{-2\lambda (t-s)}}{2\lambda }{|}_0^t\right]=\hfill \\ & =& \frac{1}{2}\left[X_0^2{e}^{-2\lambda t}+\frac{\tau }{2\lambda }\left(1-e^{-2\lambda t}\right)\right].\hfill \end{array}$$

In order to show the convergence, we compute the limit of the sequence for time to infinity. We obtain the following:

$$\underset{t\to \infty }{lim}\kappa \left(t\right)=\underset{t\to \infty }{lim}\frac{1}{2}\left[0+\frac{\tau }{2\lambda }\right]=\frac{1}{2}\frac{\tau }{2\lambda }=\frac{1}{2}{a}_1^2.$$

Indeed, we find that the (fractional) stochastic algorithm converges to the same minimum value of our function $z\left(x\right)=\frac{x^2}{2}$ for $x\in (a_1,a_2)$.

5.2. Stochastic Search: Case II

Conversely, suppose a fractional stochastic differential equation with a Hurst index $H\ne \frac{1}{2}$ of the form

$$d{X}_t=\mu X_{t}dt+\lambda X_{t}d{B}_t^H,$$

(16)

where we define $\mu :=-\eta$ and $X_0=x>0$. We search the minimum of the function $z\left(X_t\right)=X_t$, where $X_t$ is the solution of the SDE in Equation (16). This equation can be rewritten in the fractional Hida space ${\left(\mathcal{S}\right)}_H^{*}$ as

$$\begin{array}{ccc}\hfill \frac{d{X}_t}{dt}& =& \mu X_t+\lambda X_t\diamond W_t^H\hfill \\ & =& \left(\mu +\lambda W_t^H\right)\diamond X_t,\hfill \end{array}$$

where ⋄ is defined as the Wick product. Using Wick calculus, we find the solution as

$$X_t=x{e}^{\diamond \left(\mu t+\lambda {\int }_0^t{W}_s^{H}ds\right)}=e^{\diamond \left(\mu t+\lambda B_t^H\right)},$$

(17)

where we have used the definition $B_t^H={\int }_0^t{W}_s^{H}ds$. By applying the following definitions $e^{\diamond ⟨w,f⟩}=e^{\left({\int }_{\mathbb{R}}{z}_{t}d{B}_t^H-\frac{1}{2}{\parallel z\parallel }_H^2\right)}$, where ${\parallel z\parallel }_H^2={\int }_{\mathbb{R}}{\int }_{\mathbb{R}}z\left(s\right)z\left(t\right)\varphi (s,t)dsdt$ and $\varphi (s,t)={H(2H-1)|s-t|}^{2H-2}$ for $s,t\in \mathbb{R}$, we obtain the final solution:

$$X_t=x{e}^{\left(\mu t+\lambda B_t^H-\frac{1}{2}{\lambda }^2{t}^{2H}\right)}.$$

(18)

The solution of Equation (18) has an expectation of $\mathbb{E}\left[X_t\right]=X_0{e}^{-\eta t}$. Hence, for $t\to \infty$, the expected value is zero: ${lim}_{t\to \infty }\mathbb{E}\left[X_t\right]=0$. It remains to show the convergence of the fractional SDE in machine learning:

$$\begin{array}{ccc}\hfill \kappa \left(t\right)& =& \mathbb{E}\left[z\left(X_t\right)|X_0\right]=\mathbb{E}\left[x{e}^{\left(\mu t+\lambda B_t^H-\frac{1}{2}{\lambda }^2{t}^{2H}\right)}|X_0\right]\hfill \\ & =& x\mathbb{E}\left[e^{\left(\mu t+\lambda B_t^H-\frac{1}{2}{\lambda }^2{t}^{2H}\right)}\right]\hfill \\ & =& x{e}^{-\eta t}.\hfill \end{array}$$

In order to show convergence, we compute the limit of the sequence for time to infinity. We equally obtain ${lim}_{t\to \infty }\kappa \left(t\right)={lim}_{t\to \infty }x{e}^{-\eta t}=0$.

There are notable limitations of fractional stochastic gradient descent in general. Fractional calculus is built around the Riemann–Liouville integral, which is a non-local operator, lacks in uniqueness, and relies on the initial conditions. Given that a fractional process is not a martingale, the common stochastic tools are not applicable. Whether those properties constrain fractional stochastic gradient descent remains an open research question. Computational aspects might also be a limiting factor. However, for the first time, this research studies the idea of fractional search analogous to stochastic gradient descent in machine learning.

6. Conclusions

This article discovers fractional stochastic gradient descent algorithms for the optimization of neural networks. In the standard case, the fractional stochastic approach follows the well-known stochastic gradient descent method in machine learning. We discuss two special cases. First, we exhibit that fractional stochastic algorithms find the minima. This result might enhance algorithmic optimization in machine learning. Second, we discover the generalized patterns and properties of fractional stochastic processes. These insights may create a universal optimization approach in machine learning and AI in the future. We highlight the need for further research in that direction, particularly for the computational issues.