Methodologies for Improved Optimisation of the Derivative Order and Neural Network Parameters in Neural FDE Models

Abstract

This work presents and compares different methodologies for the joint optimisation of the fractional derivative order and the parameters of the right-hand-side neural network in Neural Fractional Differential Equation models. The proposed strategies aim to tackle the training difficulties typically encountered when learning the fractional order $\alpha$ together with the network weights. One approach is based on regulating the gradient magnitude of the loss function with respect to $\alpha$, encouraging more stable and effective updates. Another strategy introduces an online pre-training scheme, where the network parameters are initially optimised over progressively longer time intervals, while $\alpha$ is updated more conservatively using the full time trajectory. The study focuses only on a foundational setting with one-dimensional problems, and numerical experiments demonstrate that the proposed techniques improve both training stability and accuracy. Nonetheless, the issue of non-uniqueness in the optimal derivative order remains, particularly in less well-posed scenarios, suggesting the need for further research in data-driven modelling of fractional-order systems.

1. Introduction

Neural Fractional Differential Equations (Neural FDEs) form a class of neural architectures aimed at approximating solutions $\mathit{h}\left(t\right)$ for time-dependent data $\{{\mathit{x}}_0,{\mathit{x}}_1,\dots ,{\mathit{x}}_N\}$, where each ${\mathit{x}}_i$ represents data collected at time $t_i$, typically from experiments or simulations over a time interval $[t_0,T]$ [1,2]. Neural FDEs have attracted attention and are currently an emerging architecture in the literature, with extensions to Graph Neural Networks [3], variable-order operators [4], integrating physical knowledge [5], and more efficient adjoint algorithms to accelerate training [6]. While Neural FDEs can also be applied to model problems beyond time-series data, such applications are beyond the scope of this work [7].

These models incorporate a fractional derivative of Caputo type (the Caputo derivative definition is used due to its popularity in real-world formulations as it uses integer-order initial conditions) [8,9] (or any other fractional derivative [10,11,12,13,14,15,16]), denoted ${{}_0{}^{C}D}_t^{\alpha }\mathit{h}\left(t\right)$, into a neural network-driven dynamical system:

$$\left\{\begin{array}{c}{{}_0{}^{C}D}_t^{\alpha }\mathit{h}\left(t\right)={\mathit{f}}_{\mathit{\theta }}(t,\mathit{h}\left(t\right)),\hfill \\ \mathit{h}\left(t_0\right)={\mathit{x}}_0,t\in (t_0,T],\hfill \end{array}\right.$$

(1)

where ${\mathit{f}}_{\mathit{\theta }}$ is a neural network with parameters $\mathit{\theta }$, and $\alpha$ is the order of the fractional derivative. The Caputo fractional derivative for a scalar function $g\left(t\right)$ with $0<\alpha <1$ is defined as follows:

$${{}_0{}^{C}D}_t^{\alpha }g\left(t\right)={\displaystyle \frac{1}{\Gamma (1-\alpha )}}{\mathit{\int }}_0^t{(t-s)}^{-\alpha }{g}^{\prime }\left(s\right)ds,$$

(2)

with $\Gamma (·)$ representing the Gamma function $\Gamma \left(z\right)={\mathit{\int }}_0^{\infty }{t}^{z-1}{e}^{-t}dt$.

A remarkable feature of this framework is the flexibility to learn not only the network parameters $\mathit{\theta }$ but also the fractional order $\alpha$. When $\alpha =1$, the formulation recovers the well-known Neural ODEs [17].

In the approach proposed in [2], the fractional order $\alpha$ is parametrised by a dedicated neural network, denoted by ${\alpha }_{\mathit{\varphi }}$, with trainable parameters $\mathit{\varphi }$. This formulation is especially advantageous when dealing with systems where the fractional order of differentiation is a function of time, accommodating non-stationary dynamics. It is worth noting that, when $\alpha$ is constant, it can be treated as a single trainable parameter of the model, rather than being represented by a neural network, as is done in this work. Thus, learning involves jointly optimising $\mathit{\theta }$ and $\mathit{\varphi }$ (or $\alpha$) by minimising the difference between the predicted trajectory $\widehat{\mathit{h}}\left(t_i\right)$ (obtained from the numerical solution of (1)) and the true data ${\mathit{x}}_i$; that is:

$$\begin{gathered} \begin{array}{cc}\hfill \mathrm{minimize} \\ \mathit{\theta }\in {\mathbb{R}}^{n_{\theta }},\mathit{\varphi }\in {\mathbb{R}}^{n_{\varphi }}\hfill & \mathcal{L}(\mathit{\theta },\mathit{\varphi })={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\parallel \widehat{\mathit{h}}(t_i)-{\mathit{x}}_i{\parallel }_2^2\hfill \\ \hfill \mathrm{subject}\mathrm{to}\hfill & \widehat{\mathit{h}}(t_i)=\mathrm{FDESolve}({\alpha }_{\mathit{\varphi }},f_{\mathit{\theta }},x_0,\{t_0,\dots ,t_N\})\hfill \end{array} \end{gathered}$$

(3)

where $\left|\right|·{\left|\right|}_2^2$ is the ${\mathrm{L}}_2$ norm.

Here, $\mathrm{FDESolve}(·)$ denotes an appropriate numerical solver for the Fractional Differential Equation (FDE) (1) (see [2] for more details), and the loss function is defined as the Mean Squared Error (MSE) between the model prediction and the observed data.

Although the study in [2] reports that the fractional order $\alpha$ can be inferred from data for various datasets, the estimated values often differ significantly from the true (ground-truth) values, and different initialisations of $\alpha$ lead to different optimal $\alpha$ values.

Building upon this, the work in [18] further investigated numerically the dependency between $\alpha$ and ${\mathit{f}}_{\mathit{\theta }}$, showing that the neural network in the right-hand side of the Neural FDE struggles to consistently adjust its parameters to accurately represent the system’s dynamics for arbitrary values of $\alpha$. As a result, Neural FDEs do not necessarily converge to a unique optimal fractional order; instead, they may rely on a wide range of $\alpha$ values to achieve a reasonable fit to the data. This flexibility, however, comes at a cost. Every time the value of $\alpha$ is modified during training, the underlying function ${\mathit{f}}_{\mathit{\theta }}$ must be re-learnt, as the optimal parameter set $\mathit{\theta }$ for one fractional order might differ substantially from that of another. This results in an inefficient and potentially unstable optimisation process, since the function learnt by the network may change substantially with small variations in $\alpha$.

To better understand the complex dependence of the neural network on the order of the derivative, it is helpful to first examine an analytical problem (specifically, a fractional initial value problem) for which there are known results concerning the continuity of the solution with respect to the problem data [9,19]:

Theorem 1.

Let $0<\alpha <1$, $t_0\ge 0$, and $z_0\in \mathbb{R}$. Consider the Caputo fractional initial-value problem

$$\left\{\begin{array}{c}{{}_0{}^{C}D}_t^{\alpha }z\left(t\right)=f\left(t,z\left(t\right)\right),\hfill \\ z\left(t_0\right)=z_0,\hfill \end{array}\right.$$

and two perturbed problems on the same interval $[t_0,T]$:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{{}_0{}^{C}D}_t^{\alpha }u\left(t\right)=\tilde{f}\left(t,u\left(t\right)\right),\hfill \\ u\left(t_0\right)=z_0,\hfill \end{array}\right.\left\{\begin{array}{c}{{}_0{}^{C}D}_t^{\alpha +\epsilon }v\left(t\right)=f\left(t,v\left(t\right)\right),\hfill \\ v\left(t_0\right)=z_0.\hfill \end{array}\right.\end{array}$$

where $\tilde{f}$ satisfies the same Lipschitz (with respect to the second variable) and continuity hypotheses as f, and ε is a small scalar perturbation of the derivative order.

Define

$${\delta }_f:=\underset{(t,z)\in D}{max}\left|f(t,z)-\tilde{f}(t,z)\right|,{\delta }_{\alpha }:=\left|\epsilon \right|$$

with D a suitable domain where the solution exists and is unique. Then, there exists $T>t_0$ and a constant $C>0$ such that the unique solutions $z\left(t\right)$, $u\left(t\right)$ and $v\left(t\right)$ exist on $[t_0,T]$ and satisfy

$$\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-u\left(t\right)\right|\le C{\delta }_f,\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-v\left(t\right)\right|\le C{\delta }_{\alpha },$$

or,

$$\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-u\left(t\right)\right|=O\left({\delta }_f\right),\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-v\left(t\right)\right|=O\left({\delta }_{\alpha }\right).$$

This means that small perturbations in either f or the fractional order $\alpha$ lead to only small changes in the solution, so the problem is well posed in the sense of Hadamard [9,19]. Note that similar results can be obtained for the perturbation of the initial value $z_0$.

In light of this, one can formulate the following lemma:

Lemma 2.

Let f be Lipschitz continuous in its second argument with constant L, then

$$\underset{t_0\le t\le T}{sup}\left|f\left(t,z\left(t\right)\right)-f\left(t,v\left(t\right)\right)\right|=O\left({\delta }_{\alpha }\right)$$

Proof. 

Since f is Lipschitz continuous in its second argument with constant L, the previous theorem gives

$$\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-v\left(t\right)\right|\le C{\delta }_{\alpha }$$

for some $C>0$. Hence

$$\underset{t_0\le t\le T}{sup}\left|f\left(t,z\left(t\right)\right)-f\left(t,v\left(t\right)\right)\right|\le L\underset{t_0\le t\le T}{sup}\left|z\left(t\right)-v\left(t\right)\right|\le LC{\delta }_{\alpha }.$$

In other words, there exists a constant $K=LC$ such that

$$\underset{t_0\le t\le T}{sup}\left|f\left(t,z\left(t\right)\right)-f\left(t,v\left(t\right)\right)\right|\le K{\delta }_{\alpha },$$

showing that a perturbation ${\delta }_{\alpha }$ in the fractional order causes at most an $O\left({\delta }_{\alpha }\right)$ change in the right-hand side along the solution.    □

These estimates should be regarded with care, since their practical relevance depends on the magnitude of the constant K (or C). While they guarantee continuous dependence of the solution and the right-hand side on the data, a large constant may amplify even a small perturbation $\delta$ into a significant deviation. Moreover, because the bounds involve the supremum over $t\in [t_0,T]$, there can be instants at which

$$\left|z\left(t\right)-v\left(t\right)\right|\mathrm{or}\left|f\left(t,z\left(t\right)\right)-f\left(t,v\left(t\right)\right)\right|$$

attains large values (smaller than the supremum), thus producing notable differences in the behaviour of $f(t,z(t\left)\right)$ versus $f(t,v(t\left)\right)$.

Another point to note is that, during the training of a Neural FDE model, the optimiser often drifts toward $\alpha \approx 1$. Three factors may contribute to this bias: the gradient can become unstable for smaller $\alpha$ values due to the singular behaviour of the solution, right-hand-side function or fractional kernel; the loss landscape typically features a broader, more stable basin around $\alpha =1$; and accumulated numerical errors in the approximate solution of the FDE are minimised when $\alpha$ is close to one.

Based on the above, there is a clear need for new techniques to regulate both the learning of the optimal value of $\alpha$ and its interaction with the parameters $\mathit{\theta }$ during the optimisation process. To this end, we propose novel strategies aimed at enhancing the numerical robustness of the training procedure, with a particular emphasis on reducing the sensitivity of the learnt parameters $\mathit{\theta }$ to variations in the fractional order $\alpha$.

This paper is organised as follows. Section 2 introduces two distinct methodologies designed to address the challenges associated with learning the order of the fractional derivative (in the Neural FDE model). In Section 3, we present numerical results obtained using the Neural FDE model (together with the new proposed methodologies) under various ground-truth scenarios. The paper ends with the Conclusion in Section 4. Furthermore, Appendix A contain additional plots of the numerical experiments.

2. Method

We propose two methodologies to address the challenges associated with learning the order of the fractional derivative in Neural FDEs: Neural FDE training with α-gradient clipping (Algorithm 1) and Neural FDE training with online pre-training (Algorithm 2).

Algorithm 1 Neural FDE training with $\alpha$-gradient clipping.

1: Input: start time $t_0$, end time $t_N=T$, initial condition $\mathit{y}\left(t_0\right)={\mathit{y}}_0$, mesh, maximum number of epochs MAXITER, clipping value c;   2: Choose $Optimiser$;   3: ${\mathit{f}}_{\mathit{\theta }}=DynamicsNN\left(\right)$;   4: Initialise $\mathit{\theta }$;   5: Initialise ${\alpha }_{\mathit{\varphi }}$;   6: for$k=1:MAXITER$do   7:       ${\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N}\leftarrow \mathrm{FDESolve}({\alpha }_{\mathit{\varphi }},{\mathit{f}}_{\mathit{\theta }},{\mathit{y}}_0,\{t_0,t_1,\dots ,t_N\})$;   8:       Evaluate loss $\mathcal{L}$;   9:       $\nabla \mathcal{L}\leftarrow \mathrm{Compute}\mathrm{gradients}\mathrm{of}\mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$; 10:       ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}\leftarrow Clip\left({\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}},c\right)$ 11:       $\mathit{\theta },\mathit{\varphi }\leftarrow Optimiser.Step(\nabla \mathcal{L})$; 12: end for 13: Return: $\mathit{\theta },{\alpha }_{\mathit{\varphi }}$;

Algorithm 2 Neural FDE training with online pre-training.

1: Input: start time $t_0$, end time $t_N=T$, initial condition $\mathit{y}\left(t_0\right)={\mathit{y}}_0$, mesh, maximum number of epochs MAXITER, loss change threshold;   2: Choose $Optimiser$;   3: ${\mathit{f}}_{\mathit{\theta }}=DynamicsNN\left(\right)$;   4: Initialise $\mathit{\theta }$;   5: Initialise ${\alpha }_{\mathit{\varphi }}$;   6: Initialise $los{s}_{diff},los{s}_{prev}$;   7: $itrs\leftarrow 0$;   8: while$los{s}_{diff}>$ threshold do$▹$ online pre-training stage   9:       $itrs+=1$ 10:       for $t_i=[t_0,\dots ,T]$ do 11:             $\widehat{\mathit{y}}\left(t_i\right)\leftarrow \mathrm{FDESolve}({\alpha }_{\mathit{\varphi }},{\mathit{f}}_{\mathit{\theta }},{\mathit{y}}_0,\{t_0,t_i\})$; 12:             Evaluate loss $\mathcal{L}(\mathit{\theta },\mathit{\varphi };\mathit{y}\left(t_i\right),\widehat{\mathit{y}}\left(t_i\right))$; 13:             $\nabla \mathcal{L}\leftarrow \mathrm{Compute}\mathrm{gradients}\mathrm{of}\mathcal{L}(\mathit{\theta },\mathit{\varphi };\mathit{y}\left(t_i\right),\widehat{\mathit{y}}\left(t_i\right))$; 14:             $\mathit{\theta }\leftarrow Optimiser.Step(\nabla \mathcal{L}(\mathit{\theta },\mathit{\varphi };\mathit{y}\left(t_i\right),\widehat{\mathit{y}}\left(t_i\right)))$; 15:       end for 16:       ${\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N}\leftarrow \mathrm{FDESolve}({\alpha }_{\mathit{\varphi }},{\mathit{f}}_{\mathit{\theta }},{\mathit{y}}_0,\{t_0,t_1,\dots ,t_N\})$; 17:       Evaluate loss $\mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$; 18:       $\nabla \mathcal{L}\leftarrow \mathrm{Compute}\mathrm{gradients}\mathrm{of}\mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$; 19:       $\mathit{\varphi }\leftarrow Optimiser.Step(\nabla \mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N}))$; 20:       $los{s}_{diff}\leftarrow los{s}_{prev}-\mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$ 21:       $los{s}_{prev}\leftarrow \mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$ 22: end while 23: for$k=itrs:MAXITER$do$▹$ original Neural FDE training 24:       ${\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N}\leftarrow \mathrm{FDESolve}({\alpha }_{\mathit{\varphi }},{\mathit{f}}_{\mathit{\theta }},{\mathit{y}}_0,\{t_0,t_1,\dots ,t_N\})$; 25:       Evaluate loss $\mathcal{L}$; 26:       $\nabla \mathcal{L}\leftarrow \mathrm{Compute}\mathrm{gradients}\mathrm{of}\mathcal{L}(\mathit{\theta },\mathit{\varphi };{\left\{\mathit{h}\left(t_i\right)\right\}}_{i=1\dots ,N},{\left\{\widehat{\mathit{h}}\left(t_i\right)\right\}}_{i=1\dots ,N})$; 27:       $\mathit{\theta },\mathit{\varphi }\leftarrow Optimiser.Step(\nabla \mathcal{L})$; 28: end for 29: Return: $\mathit{\theta },{\alpha }_{\mathit{\varphi }}$;

2.1. $\alpha$-Gradient Clipping

Following the experimental results presented in [2,18], in this work, to prevent a rapid change in the $\alpha$ parameter during training, we propose clipping the gradients of ${\alpha }_{\mathit{\varphi }}$. To maintain the generality of the methodologies throughout this work, the general notation ${\alpha }_{\mathit{\varphi }}$ will be adopted to represent a neural network-based approximation of the fractional order. However, the numerical experiments will focus on the case where $\alpha$ is a single scalar parameter, as this is the most common modelling approach in the context of FDEs.

Since ${\alpha }_{\mathit{\varphi }}$ and $\mathit{\theta }$ are jointly optimised, large or erratic gradient updates in ${\alpha }_{\mathit{\varphi }}$ can hinder the optimisation process. Thus, by bounding the gradients we reduce the risk of such adverse interactions and promote more stable training and better convergence [20,21]. Gradient clipping can be easily integrated into the Neural FDE training process as described in Algorithm 1.

2.2. Online Pre-Training

Another source of instability in the Neural FDE training process is the update of the $\mathit{\varphi }$ at the same frequency of $\mathit{\theta }$ since, as stated in Section 1, the optimal parameter set $\mathit{\theta }$ for one fractional order $\alpha$ may be remarkably distinct for another. Thus, even small changes in $\alpha$ can require substantial re-learning of ${\mathit{f}}_{\mathit{\theta }}$.

To address this we introduce an online pre-training stage to the Neural FDE optimisation process. Instead of solving the FDE over the entire time domain and updating the parameters based on the full trajectory predictions in each epoch, this stage adopts a progressive, short-horizon training approach. The FDE is solved gradually by increasing the integration horizon, and at each step, only the prediction at the current terminal time point $t_i$ is used to compute the loss and update the dynamics parameters $\mathit{\theta }$. This process, named online training [22], effectively performs one update per time step over the interval $t\in [t_0,T]$, resulting in T iterations per epoch, while keeping the fractional-order parameters $\mathit{\varphi }$ fixed.

This gradual training allows the model to first capture local dynamics and stabilise its predictions before being exposed to global behaviour, mitigating gradient instability and poor convergence. Once a full epoch of short-horizon updates is completed, $\mathit{\varphi }$ is updated using a loss computed over the entire trajectory while $\mathit{\theta }$ is fixed. The online pre-training stage ends when the change in loss between epochs falls below a predefined threshold, avoiding unnecessary computation and reducing the risk of overfitting. This stage can be easily introduced in the Neural FDE optimisation process by adding a second training loop before the traditional optimisation, as in Algorithm 2.

3. Numerical Experiments

To analyse and compare the behaviour of the Neural FDE model trained with $\alpha$-gradient clipping (clipped Neural FDE, Algorithm 1) and with online pre-training (pre-trained Neural FDE, Algorithm 2) against the baseline approach from [2] (original Neural FDE), we present numerical results for three carefully selected case studies, each chosen for its distinct characteristics. In each case, the datasets were generated using the analytical solution of the FDE under study, for different values of the fractional order $\alpha$. The objective was to assess the performance of each training method by evaluating the MSE and the accuracy in learning the correct $\alpha$ value. Results are reported as averages over three independent runs, together with the corresponding standard deviations (std). In addition, the evolution of $\alpha$ throughout training, along with its gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}$ and the corresponding loss function, were analysed and compared.

All experiments were implemented in PyTorch 3.8 [23], making use of the FDEint 0.1.1 library [24]. Training was conducted using the Adam optimiser [25] with a learning rate of 0.0025. In the numerical experiments, ${\alpha }_{\mathit{\varphi }}$ was treated as a single trainable parameter. Gradient clipping was applied within the range $[-1,1]$, and the stopping criterion for the online pre-training phase was set to $1\times {10}^{-6}$. Each Neural FDE model was trained for a total of 2000 epochs. It is worth noting that, in the online pre-training methodology, the two training stages together comprise the 2000 epochs, ensuring a fair comparison with the original and gradient clipping approaches.

3.1. Case Study 1

Consider the following initial value problem involving a Caputo fractional derivative of order $\alpha \in (0,1)$:

$${{}_0{}^{C}D}_t^{\alpha }y\left(t\right)=\lambda y\left(t\right),t\in (0,T],y\left(0\right)=1,$$

(4)

where $\lambda \in \mathbb{R}$ is a constant parameter. The analytical solution to this problem is given by [26],

$$y\left(t\right)=E_{\alpha }\left(\lambda t^{\alpha }\right),$$

(5)

where the Mittag-Leffler function $E_{\alpha }\left(z\right)$ is defined as $E_{\alpha }\left(z\right)={\sum }_{k=0}^{\infty }{\displaystyle \frac{z^k}{\Gamma (\alpha k+1)}},\alpha >0$. More generally, the two-parameter Mittag-Leffler function is given by,

$$E_{\alpha ,\gamma }\left(z\right)=\sum _{k=0}^{\infty }{\displaystyle \frac{z^k}{\Gamma (\alpha k+\gamma )}},\alpha ,\gamma >0,$$

(6)

and satisfies $E_{\alpha ,1}\left(z\right)=E_{\alpha }\left(z\right)$.

Note that the right-hand-side function $f(t,y(t\left)\right)=\lambda y\left(t\right)$ is proportional to the solution itself. Therefore, any perturbation in the solution directly translates into a proportional perturbation in the right-hand-side function. As such, the absolute difference between solutions resulting from different values of $\alpha$ coincides (up to a multiplicative factor $\lambda$) with the corresponding difference in the right-hand-side functions.

We consider $\lambda =-1$ and compare the solution for $\alpha =0.80$ with a perturbed order $\alpha =0.85$. Table 1 shows the values of $y\left(t\right)$ for both cases and the absolute difference $\left|\mathsf{\Delta }y\left(t\right)\right|=|y_{0.85}\left(t\right)-y_{0.80}\left(t\right)|$ at selected time points. Varying $\alpha$ induces small changes in both the solution and the right-hand-side function.

Table 1. Comparison between the solutions of problem (4), with $\lambda =-1$, for the fractional orders $\alpha =0.80$ and $\alpha =0.85$.

t	${\mathit{y}}_{0.80}\left(\mathit{t}\right)$	${\mathit{y}}_{0.85}\left(\mathit{t}\right)$	$\left|\mathbf{\Delta }\mathit{y}\right(\mathit{t}\left)\right|$
0.1	0.846	0.863	0.017
0.5	0.562	0.572	0.010
1.0	0.387	0.381	0.006
5.0	0.088	0.066	0.022
7.5	0.057	0.040	0.017
10.0	0.043	0.029	0.014

• Smoothness and Singularities: The Mittag-Leffler function $E_{\alpha }(-t^{\alpha })$ is entire for $t>0$ and continuous at $t=0$. Its small-t expansion

  $$y\left(t\right)=1-{\displaystyle \frac{t^{\alpha }}{\Gamma (1+\alpha )}}+{\displaystyle \frac{t^{2\alpha }}{\Gamma (1+2\alpha )}}-\dots$$

  yields

  $$y^{\prime }\left(t\right)\sim -{\displaystyle \frac{\alpha }{\Gamma (1+\alpha )}}{t}^{\alpha -1},y^{″}\left(t\right)\sim -{\displaystyle \frac{\alpha (\alpha -1)}{\Gamma (1+\alpha )}}{t}^{\alpha -2}(t\to 0^{+}).$$
  Since $\alpha -1<0$ and $\alpha -2<-1$, both $y^{\prime }\left(t\right)$ and $y^{″}\left(t\right)$ diverge as $t\to 0^{+}$. The singularity in the slope is of order $t^{\alpha -1}$ (weakly integrable), while the curvature diverges more strongly like $t^{\alpha -2}$. For larger t, y decreases monotonically from 1 toward 0, beginning with strong convexity (since $y^{″}>0$ near zero) and transitioning eventually to mild concave-down behaviour as the algebraic decay dominates. The right-hand side $f(t,y)=-y\left(t\right)$ is equally smooth on $(0,4]$ and continuous at 0, with $f^{\prime }=-y^{\prime }$ and $f^{″}=-y^{″}$. Thus f has the same singular orders but opposite sign curvature: strongly concave near $t=0$ and gently concave elsewhere.

The Neural FDEs employed in this case study consist of a neural network ${\mathit{f}}_{\mathit{\theta }}$ with the following architecture: an input layer with a single neuron and a rectified linear unit (ReLU) activation function, one hidden layer with 10 neurons also using ReLU activation, and an output layer with a single neuron. The fractional order $\alpha$ was initialised to $0.99$.

Four datasets corresponding to $\alpha =0.4,0.5,0.8,0.99$ were generated using the analytical solution. Each training set consisted of 500 points over the interval $t\in [0,3]$, while the testing sets included 700 points over $t\in [0,4]$.

It is crucial to emphasise that the training data originates from an analytical solution, which is then numerically transformed using an approximation of the Caputo derivative. The neural network ${\mathit{f}}_{\mathit{\theta }}$ learns from this transformed data, meaning the accuracy of the numerical approximation directly impacts the reliability of the training data and, consequently, the optimisation outcomes. This numerical treatment may either suppress or artificially smooth potential singularities in the right-hand-side function $f(t,y(t\left)\right)$.

The results are summarised in Table 2, Table 3 and Table 4, while the evolution of the learnt $\alpha$, its gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}$, and the loss throughout training are illustrated in Figure 1, Figure 2, Figure 3 and Figure 4 for the case $\alpha =0.5$. Additional results for $\alpha =0.4$, $0.8$, and $0.99$ are provided in Appendix A.1, Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6, Figure A7, Figure A8, Figure A9, Figure A10, Figure A11 and Figure A12.

Table 2. Final training loss (average MSE ± standard deviation over 3 runs) of the Neural FDE model using the three different training methods for Case Study 1.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.4	0.000090 ± 0.000050	0.000120 ± 0.000060	0.000080 ± 0.000070
0.5	0.000030 ± 0.000020	0.000130 ± 0.000100	0.000100 ± 0.000100
0.8	0.000030 ± 0.000020	0.000030 ± 0.000010	0.000010 ± 0.000001
0.99	0.000001 ± 0.000001	0.00001 ± 0.000010	0.000001 ± 0.000001

Table 3. Test MSE (average MSE ± standard deviation over 3 runs) of the Neural FDE using the three training methods, applied to Case Study 1.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.4	0.00016 ± 0.00008	0.00014 ± 0.00002	0.00015 ± 0.00016
0.5	0.00004 ± 0.00003	0.00017 ± 0.00014	0.00021 ± 0.00023
0.8	0.00011 ± 0.00004	0.00013 ± 0.00007	0.00003 ± 0.00001
0.99	0.00001 ± 0.00001	0.00018 ± 0.00024	0.00002 ± 0.00002

Table 4. Learnt values of $\alpha$ across three runs of the Neural FDE model using the three different training methods for Case Study 1. Results are reported as the mean MSE ± standard deviation.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.4	0.60053 ± 0.03442	0.79190 ± 0.17455	0.55023 ± 0.29157
0.5	0.68963 ± 0.10462	0.83263 ± 0.16090	0.47243 ± 0.15194
0.8	0.94540 ± 0.04414	0.95633 ± 0.02652	0.88103 ± 0.06997
0.99	0.98280 ± 0.00332	0.97977 ± 0.00466	0.98657 ± 0.00180

Figure 1. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 2. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 3. Training loss across epochs for the best run in Case Study 1 with with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 4. Evolution of the best run’s $\alpha$ gradient during training for Case Study 1 with with $\alpha =0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Table 2 (final training losses) and Table 3 (test MSE over 3 runs) confirm that, as expected, there are no significant differences in performance between the original Neural FDE method [2] and the two strategies proposed in this work.

However, Table 4 shows that while both the original Neural FDE and the pre-trained version produce predicted values of $\alpha$ close to the ground truth, the pre-trained model significantly outperforms the original method by achieving notably more accurate estimates of $\alpha$, demonstrating a clear improvement in learning.

In contrast, the worst performance is observed with the gradient clipping approach, as illustrated also in Figure 1, which displays the data fitting for the case $\alpha =0.5$.

The good fitting performance of the original Neural FDE method (also visible in Figure 1) may be attributed to one of two reasons: either the neural network with parameters $\mathit{\theta }$ adapted effectively to the learnt value $\alpha \approx 0.69$, or the discrepancy between the true value $\alpha =0.5$ and the learnt one was not significant enough to affect the prediction quality in this particular case. The latter explanation is consistent with the sensitivity analysis discussed earlier (Table 1).

Figure 2 (evolution of $\alpha$ during training) and Figure 3 (evolution of the loss function) indicate that gradient clipping negatively affects the update dynamics of $\alpha$. Specifically, the clipped-$\alpha$ strategy appears to slow down convergence, requiring more training epochs to achieve the same level of accuracy as the other two methods.

Figure 4 shows that the gradients of the loss function with respect to $\alpha$ are very small in this case study. This may explain why the gradient clipping technique does not yield improved results. The Adam optimiser combines momentum and adaptive learning rates by computing exponential moving averages of both the gradients and their squared values. As a result, it adjusts the step size dynamically based on the history of gradients. However, when the gradients are consistently small, the adaptive nature of Adam may result in updates that are too conservative, thereby limiting the optimiser’s ability to escape flat regions of the loss landscape or to sufficiently update $\alpha$. The effectiveness of the gradient clipping methodology will be better assessed in the following case studies, where larger gradients are expected to arise.

For the remaining values of $\alpha$, namely 0.4, 0.8 and 0.99, the results are qualitative similar to those observed for $\alpha =0.5$. These results are presented in Appendix A.1 to avoid cluttering of figures.

3.2. Case Study 2

Consider the following initial value problem involving a Caputo fractional derivative of order $\alpha \in (0,1)$:

$${{}_0{}^{C}D}_t^{\alpha }y\left(t\right)=f(t,y\left(t\right)),t\in (0,T],$$

(7)

subject to the initial condition $y\left(0\right)=1$, where the right-hand side is given by,

$$f(t,y\left(t\right))={\displaystyle \frac{2}{\Gamma (3-\alpha )}}{t}^{2-\alpha }-{\displaystyle \frac{1}{\Gamma (2-\alpha )}}{t}^{1-\alpha }$$

The analytical solution to this problem is $y\left(t\right)=t^2-t+1$ [26].

This particular example is noteworthy because the analytical solution does not depend on the value of $\alpha$, allowing the model to converge to an arbitrary fractional order. However, it is important to note that the neural network ${\mathit{f}}_{\mathit{\theta }}$ is trained to approximate the fractional derivative itself, not the solution $y\left(t\right)$. As such, its output inherently depends on the order $\alpha$. Although it is not possible to directly compute the error between the predicted and the true value of $\alpha$ in this case, the model’s behaviour during the search for the optimal solution provides insight into how it adapts to the problem structure.

• Smoothness and Singularities: The polynomial $y\left(t\right)$ is $C^{\infty }$ on $[0,4]$ with

  $$y^{\prime }\left(t\right)=2t-1,y^{″}\left(t\right)=2,$$

  so it exhibits uniform convexity and a single turning point at $t=0.5$. No singularities arise in y.
  The function $f\left(t\right)$ is $C^{\infty }$ on $(0,4]$ and tends to zero as $t\to 0^{+}$, because both exponents $2-\alpha >1$ and $1-\alpha \in (0,1)$ are positive. Its derivatives,

  $$f^{\prime }\left(t\right)={\displaystyle \frac{2(2-\alpha )}{\Gamma (3-\alpha )}}{t}^{1-\alpha }-{\displaystyle \frac{1-\alpha }{\Gamma (2-\alpha )}}{t}^{-\alpha },$$

  $$f^{″}\left(t\right)={\displaystyle \frac{2(2-\alpha )(1-\alpha )}{\Gamma (3-\alpha )}}{t}^{-\alpha }-{\displaystyle \frac{\alpha (1-\alpha )}{\Gamma (2-\alpha )}}{t}^{-\alpha -1},$$

  diverge as $t\to 0^{+}$ with orders $t^{-\alpha }$ and $t^{-\alpha -1}$, respectively. Thus f has a mild slope singularity but a strong curvature singularity at the origin. On $(0,4]$, f changes sign once and its curvature $f^{″}\left(t\right)$ also changes sign, making f non-monotonic and non-convex overall.

The Neural FDEs considered in this case study consist of a neural network ${\mathit{f}}_{\mathit{\theta }}$ with one input layer containing a single neuron using a ReLU activation function; one hidden layer with 10 neurons and ReLU activation; and one output layer with a single neuron.

A dataset was generated from the analytical solution, with each training set comprising 500 points over the interval $t\in [0,3]$, and testing sets containing 700 points over $t\in [0,4]$. To examine the influence of the learnt $\alpha$ and its initialisation on the Neural FDE models, we present results for two initial values of $\alpha$: 0.5 and 0.99.

The outcomes are summarised in Table 5, Table 6 and Table 7, while Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 display the training evolution of the learnt $\alpha$, its gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}$, and the loss for both initialisations.

Table 5. Final training loss (average MSE ± standard deviation over 3 runs) of the Neural FDE model using the three different training methods for Case Study 2.

$\mathit{\alpha }$ Initialisation	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.061450 ± 0.035730	0.046410 ± 0.030470	0.043890 ± 0.031410
0.99	0.041230 ± 0.029030	0.035010 ± 0.007570	0.023510 ± 0.005310

Table 6. Test MSE (average MSE ± standard deviation over 3 runs) of the Neural FDE using the three training methods, applied to Case Study 2.

$\mathit{\alpha }$ Initialisation	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	2.140550 ± 0.476330	2.386290 ± 3.141570	2.527410 ± 3.354300
0.99	2.253340 ± 2.918830	1.505450 ± 0.456910	0.465760 ± 0.514360

Table 7. Learnt values of $\alpha$ across three runs of the Neural FDE model using the three different training methods for Case Study 2. $\alpha$ initialised at $0.5$.

Run	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
1	0.6072	0.6151	0.5827
2	0.5881	0.6515	0.7150
3	0.5510	0.5496	0.6826

Figure 5. Plot of the predicted and ground-truth curves from the best run in Case Study 2 with $\alpha$ initialised at $0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 6. Plot of the predicted and ground-truth curves from the best run in Case Study 2 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 7. Evolution of the best run’s $\alpha$ during training for Case Study 2 with $\alpha$ initialised at $0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 8. Evolution of the best run’s $\alpha$ during training for Case Study 2 with $\alpha$ initialised at $0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 9. Evolution of the best run’s $\alpha$ gradient during training for Case Study 1 with $\alpha$ initialised at $0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 10. Evolution of the best run’s $\alpha$ gradient during training for Case Study 1 with $\alpha$ initialised at $0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

From Table 5 and Table 6, it is evident that the initialisation of $\alpha$ slightly influences the training performance, with an initial value of $0.99$ consistently yielding better results across all three methodologies. In the testing dataset, performances are comparable among the different initialisations; however, initialising at $0.5$ results in less consistency across training runs, as reflected by a higher standard deviation.

All three Neural FDE training approaches learn values of $\alpha$ that remain extremely close to their initialisation value of $0.5$ (and $0.99$) (see Table 7 and Table 8). This indicates that, in this particular case where any $\alpha$ should (in theory) yield a good solution (the analytical solution does not depend on $\alpha$), the methodology is robust enough not to deviate significantly from the initialised $\alpha$.

Table 8. Learnt values of $\alpha$ across three runs of the Neural FDE model using the three different training methods for Case Study 2. $\alpha$ initialised at $0.99$.

Run	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
1	0.9908	0.9925	0.9934
2	0.9921	0.9934	0.9921
3	0.9964	0.9941	0.9906

Note that when $\alpha$ is initialised at $0.5$, the learnt values across the three runs exhibit greater variability (see Table 7).

Figure 5 and Figure 6 display the predicted and ground-truth curves from the best runs in Case Study 2, with $\alpha$ initialised at 0.5 and 0.99, respectively. For the case $\alpha =0.5$, the pre-trained methodology appears to deviate from the ground truth in the testing region; however, overall, this approach delivers the best results.

During training (Figure 7, where $\alpha =0.5$), the parameter $\alpha$ exhibits a sharp increase for both the original and gradient-clipped Neural FDEs. A similar trend is observed when $\alpha =0.99$, although in this case, the clipped model displays a smoother evolution compared to the scenario with $\alpha =0.5$ (Figure 8).

Furthermore, both the original and clipped models show significant oscillations in the gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial \alpha }}$ throughout training. While the pre-trained Neural FDE exhibits similar behaviour, the oscillations are noticeably more stable, suggesting that the pre-training phase contributes to a more robust and consistent learning process (see Figure 9 and Figure 10).

It is also worth noting that the amplitude of the oscillations is substantially larger in the case where $\alpha =0.5$ (the scales used in Figure 9 and Figure 10 are different). For values of $\alpha$ closer to 1, the right-hand-side function becomes smoother and, in principle, easier to learn. As a result, the optimisation process may tend to favour higher values of $\alpha$ within the range $[0,1]$.

These oscillations are reflected in the behaviour of the loss function. Figure 11 and Figure 12 show that the pre-training method exhibits a smoother evolution of the loss compared to the other two approaches.

Figure 11. Training loss across epochs for the best run in Case Study 2 with $\alpha$ initialised at $0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 12. Training loss across epochs for the best run in Case Study 2 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

3.3. Case Study 3

Consider the following initial value problem involving the Caputo fractional derivative of order $\alpha \in (0,1)$:

$${{}_0{}^{C}D}_t^{\alpha }y\left(t\right)=f(t,y\left(t\right)),t\in (0,T],$$

(8)

subject to the initial condition $y\left(0\right)=0$, where the right-hand side is defined as,

$$f(t,y\left(t\right))={\displaystyle \frac{40320}{\Gamma (9-\alpha )}}{t}^{8-\alpha }-3{\displaystyle \frac{\Gamma (5+\alpha /2)}{\Gamma (5-\alpha /2)}}{t}^{4-\alpha /2}+{\displaystyle \frac{9}{4}}\Gamma (\alpha +1)+{\left({\displaystyle \frac{3}{2}}{t}^{\alpha /2}-t^4\right)}^3-y{\left(t\right)}^{3/2}.$$

(9)

We consider the interval $t\in [0,1]$, for which the analytical solution is given by

$$y\left(t\right)=t^8-3t^{4+{\displaystyle \frac{\alpha }{2}}}+{\displaystyle \frac{9}{4}}{t}^{\alpha },$$

(10)

as established in [9]. In this case, the analytical solution includes a term of the form $t^{\alpha }$, which introduces a more pronounced non-smoothness to the solution.

We compare the solution for $\alpha =0.80$ with a perturbed order $\alpha =0.85$. Table 9 shows the values of $y\left(t\right)$ for both cases and the absolute differences $\left|\mathsf{\Delta }y\left(t\right)\right|=|y_{0.85}\left(t\right)-y_{0.80}\left(t\right)|$, $\left|\mathsf{\Delta }f(t,y\left(t\right))\right|=|f_{0.85}\left(t\right)-f_{0.80}\left(t\right)|$ at selected time points.

Table 9. Comparison between the solutions and right-hand-side terms of the fractional ODE for $\alpha =0.80$ and $\alpha =0.85$.

t	${\mathit{y}}_{0.80}\left(\mathit{t}\right)$	${\mathit{y}}_{0.85}\left(\mathit{t}\right)$	$\left|\mathbf{\Delta }\mathit{y}\right(\mathit{t}\left)\right|$	${\mathit{f}}_{0.80}\left(\mathit{t}\right)$	${\mathit{f}}_{0.85}\left(\mathit{t}\right)$	$\left|\mathbf{\Delta }\mathit{f}\right(\mathit{t},\mathit{y}\left(\mathit{t}\right)\left)\right|$
0.10	0.356482	0.317708	0.038773	2.093102	2.124758	0.031655
0.25	0.735508	0.686033	0.049475	2.027857	2.052026	0.024169
0.50	1.154093	1.112515	0.041579	1.307284	1.264756	0.042528
0.75	1.041513	1.022050	0.019464	−0.781984	−0.971532	0.189548
1.00	0.250000	0.250000	0.000000	−2.571448	−2.747086	0.175638

Varying $\alpha$ induces small changes in both the solution and the right-hand-side function. However, these variations are slightly more pronounced than those observed in Case Study 1.

• Smoothness and Singularities:

  $$y^{\prime }\left(t\right)=8t^7-3\left(4+\frac{\alpha }{2}\right)t^{3+{\displaystyle \frac{\alpha }{2}}}+\frac{9}{4}\alpha t^{\alpha -1},$$

  $$y^{″}\left(t\right)=56t^6-3\left(4+\frac{\alpha }{2}\right)\left(3+\frac{\alpha }{2}\right)t^{2+{\displaystyle \frac{\alpha }{2}}}+\frac{9}{4}\alpha (\alpha -1)t^{\alpha -2}.$$
  Near $t=0$, the term $\frac{9}{4}\alpha t^{\alpha -1}$ in $y^{\prime }$ and $\frac{9}{4}\alpha (\alpha -1)t^{\alpha -2}$ in $y^{″}$ dominate, producing a weak slope singularity of order $t^{\alpha -1}$ and a strong curvature singularity of order $t^{\alpha -2}$. As t increases, the high-degree monomials $t^7$ and $t^6$ prevail, introducing inflection points and reversals of concavity.
  The function $f(t,y(t\left)\right)$ is a sum of fractional powers and highly nonlinear terms. Its first and second derivatives consist of sums of powers $t^p$, where near $t=0$ the most singular exponent arises from differentiating ${\left(t^{\alpha /2}\right)}^3$. One finds curvature singularities of order up to $t^{\alpha /2-2}$, which can be exceptionally strong if $\alpha$ is small. Throughout $[0,1]$, f exhibits multiple sign changes, several inflection points, and rapid oscillations of curvature, reflecting high analytical complexity.

For this case study, four datasets were generated corresponding to different values of the fractional order, namely $\alpha =0.5$, $0.8$, and $0.99$. The training set consists of 300 data points uniformly distributed over the interval $t\in [0,0.6]$, while the testing set includes 400 points in the extended range $t\in [0,0.7]$.

The Neural FDE models employ a neural network ${\mathit{f}}_{\mathit{\theta }}$ with the following architecture: one input layer with a single neuron using a tanh activation function; two hidden layers with 128 neurons each—where the first uses the exponential linear unit (ELU) activation and the second uses tanh; and a final output layer with one neuron. The fractional order $\alpha$ was initialised at $0.99$. In this case, a more robust neural network was considered in order to deal the characteristics of the problem at hands.

The corresponding results are presented in Table 10, Table 11 and Table 12. Figure 13, Figure 14, Figure 15 and Figure 16 illustrate the training dynamics for the case $\alpha =0.5$, including the evolution of the learnt order $\alpha$, its gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}$, and the loss function. Additional results for $\alpha =0.8$ and $\alpha =0.99$ are provided in Appendix A.2, in Figure A13, Figure A14, Figure A15, Figure A16, Figure A17, Figure A18, Figure A19 and Figure A20.

Table 10. Final training loss (average MSE ± standard deviation over 3 runs) of the Neural FDE model using the three different training methods for Case Study 3.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.000140 ± 0.000020	0.000170 ± 0.000020	0.000100 ± 0.000001
0.8	0.000020 ± 0.000001	0.000010 ± 0.000001	0.000001 ± 0.000001
0.99	0.000001 ± 0.000001	0.000001 ± 0.000001	0.000010 ± 0.000010

Table 11. Test MSE (average MSE ± standard deviation over 3 runs) of the Neural FDE using the three training methods, applied to Case Study 3.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.001690 ± 0.000110	0.001810 ± 0.000100	0.001190 ± 0.000050
0.8	0.000610 ± 0.000040	0.000610 ± 0.000010	0.000290 ± 0.000060
0.99	0.000330 ± 0.000010	0.000330 ± 0.000020	0.000150 ± 0.000050

Table 12. Learnt values of $\alpha$ across three runs of the Neural FDE model using the three different training methods for Case Study 3. Results are reported as the mean MSE ± standard deviation.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.978230 ± 0.002030	0.980170 ± 0.000250	0.974870 ± 0.000210
0.8	0.988530 ± 0.000050	0.988700 ± 0.000080	0.988400 ± 0.000370
0.99	0.990430 ± 0.000050	0.990430 ± 0.000050	0.989570 ± 0.000260

Figure 13. Plot of the predicted and ground-truth curves from the best run in Case Study 3 with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 14. Evolution of the best run’s $\alpha$ during training for Case Study 3 with $\alpha =0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 15. Training loss across epochs for the best run in Case Study 3 with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 16. Evolution of the best run’s $\alpha$ gradient during training for Case Study 3 with $\alpha =0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

From Table 10 and Table 11, all three methodologies have similar training and testing performance. Furthermore, the learnt $\alpha$ of all three methodologies are similar, close to 0.99, independently of the ground-truth $\alpha$ value of the dataset being modelled. These results highlight the difficulty the model faces in accurately learning the correct value of $\alpha$ using any of the methods. The loss function consistently favours values of $\alpha$ close to 1 (even when considering different initialisations of $\alpha$).

Remark: 

It should be noted, however, that increasing the number of training epochs leads to a gradual decrease in the value of $\alpha$ towards its expected value. Nevertheless, the convergence rate is so slow that the associated computation times become prohibitively high, rendering this approach inefficient in practice.

From Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6, Figure A7, Figure A8, Figure A9, Figure A10, Figure A11, Figure A12, Figure A13, Figure A14, Figure A15, Figure A16, Figure A17, Figure A18, Figure A19 and Figure A20, unlike in the previous case study, both the loss and the gradient with respect to $\alpha$ remain stable throughout training. However, when analysing the prediction curves (Figure 13, Figure A13 and Figure A17), it becomes clear that although the models fit the data well up to $t=0.5$, they struggle to generalise beyond that point.

Figure 17. Plot of the predicted and ground-truth curves from the best run in Case Study 4 with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure 18. Evolution of the best run’s $\alpha$ during training for Case Study 4 with $\alpha =0.5$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure 19. Plot of the best run $\alpha$ gradient value through training for Case Study 4, with $\alpha =0.5$: (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure 20. Training loss across epochs for the best run in Case Study 4 with $\alpha =0.5$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

This weaker performance in the latter part of the time interval is likely due to a shift in the underlying dynamics, which were not present in the training data. For the models to successfully extrapolate, additional training data in the interval $t\in [0.6,0.7]$ would be necessary.

3.4. Case Study 4

Consider again the initial value problem:

$${{}_0{}^{C}D}_t^{\alpha }y\left(t\right)=\lambda y\left(t\right),t\in (0,T],y\left(0\right)=1,$$

(11)

but now with $\lambda =1$. We compare the solution ($y\left(t\right)=E_{\alpha }\left(\lambda t^{\alpha }\right)$) for $\alpha =0.80$ with a perturbed order $\alpha =0.85$. Table 13 shows the values of $y\left(t\right)$ for both cases and the absolute difference $\left|\mathsf{\Delta }y\left(t\right)\right|=|y_{0.85}\left(t\right)-y_{0.80}\left(t\right)|$ at selected time points:

Table 13. Comparison between the solutions of problem (11), with $\lambda =1$, for the fractional orders $\alpha =0.80$ and $\alpha =0.85$.

t	${\mathit{y}}_{0.80}\left(\mathit{t}\right)$	${\mathit{y}}_{0.85}\left(\mathit{t}\right)$	$\left|\mathbf{\Delta }\mathit{y}\right(\mathit{t}\left)\right|$
0.1	1.189	1.163	0.026
0.5	1.928	1.846	0.082
1.0	3.295	3.125	0.169
5.0	185.471	174.573	10.898
7.5	2260.018	2127.085	132.932
10.0	27,533.053	25,913.470	1619.583

For $t=10$, we observe that:

$$\left|f\left(10,z\left(10\right)\right)-f\left(10,v\left(10\right)\right)\right|=\left|E_{0.80}\left({10}^{0.80}\right)-E_{0.85}\left({10}^{0.85}\right)\right|\approx 1619.583.$$

This difference satisfies the inequality

$$\left|f\left(10,z\left(10\right)\right)-f\left(10,v\left(10\right)\right)\right|\le K{\delta }_{\alpha },$$

where ${\delta }_{\alpha }=0.85-0.80=0.05$, which implies that $K\approx$ 32,392.

Such a large value of K highlights the strong sensitivity of the solution with respect to small perturbations in the fractional order $\alpha$, particularly for large t. This indicates that the problem may exhibit a not so well-posed behaviour, where small variations in the problem data result in substantial changes in the solution (or right-hand-side function).

• Smoothness and Singularities: Similar to Case Study 1. Both $y^{\prime }$ and $y^{″}$ diverge with orders $t^{\alpha -1}$ and $t^{\alpha -2}$. For $t>0$, all derivatives exist and remain positive, so y grows monotonically and stays strictly convex (no inflection). The right-hand side $f(t,y)=y\left(t\right)$ shares these properties exactly.

Using the analytical solution, four datasets were generated for $\alpha =0.5$, $0.8$, and $0.99$. Each dataset contains 1500 points in the interval $t\in [0,3]$ for training and 1500 points in the interval $t\in [0,4]$ for testing. The Neural FDE models use a neural network ${\mathit{f}}_{\mathit{\theta }}$ with the following architecture: one input layer with a single neuron and a hyperbolic tangent (tanh) activation function; two hidden layers with 64 neurons each and ReLU activations; and one output layer with a single neuron. The fractional order $\alpha$ was initialised at $0.99$ for all experiments.

The corresponding results are summarised in Table 14, Table 15 and Table 16. The evolution of the learnt order $\alpha$, its gradient ${\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_{\mathit{\varphi }}}}$, and the training loss are depicted in Figure 17, Figure 18, Figure 19 and Figure 20 for $\alpha =0.5$, and in Appendix A.3, Figure A21, Figure A22, Figure A23, Figure A24, Figure A25, Figure A26, Figure A27 and Figure A28, for $\alpha =0.8$ and $\alpha =0.99$.

Table 14. Final training loss (average MSE ± standard deviation over 3 runs) of the Neural FDE model using the three different training methods for Case Study 4.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.000020 ± 0.000030	0.000110 ± 0.000140	0.000180 ± 0.000260
0.8	0.000040 ± 0.000030	0.000020 ± 0.000020	0.005830 ± 0.002250
0.99	0.000040 ± 0.000001	0.000050 ± 0.000040	0.000200 ± 0.000260

Table 15. Test MSE (average MSE ± standard deviation over 3 runs) of the Neural FDE using the three training methods, applied to Case Study 4.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.021130 ± 0.003410	0.024390 ± 0.012660	0.027110 ± 0.019670
0.8	0.014270 ± 0.010150	0.014860 ± 0.006110	0.032400 ± 0.010170
0.99	0.020810 ± 0.006660	0.009430 ± 0.001780	0.035060 ± 0.013860

Table 16. Learnt values of $\alpha$ across three runs of the Neural FDE model using the three different training methods for Case Study 4. Results are reported as the mean MSE ± standard deviation.

$\mathit{\alpha }$	Original Neural FDE	Clipped Neural FDE	Pre-Trained Neural FDE
0.5	0.995400 ± 0.000670	0.996800 ± 0.001120	0.997230 ± 0.001340
0.8	0.996070 ± 0.000740	0.995330 ± 0.001600	0.990330 ± 0.001670
0.99	0.995600 ± 0.001950	0.996330 ± 0.001140	0.997270 ± 0.001430

The results presented in Table 14, Table 15 and Table 16 indicate that all three methodologies exhibit similar performance in terms of fitting the data and predicting the solution’s evolution. However, as shown in Figure 18, the best results for the most successful run were achieved using the pre-trained methodology.

It is worth noting that the clipped methodology produced very poor results, despite achieving a low final training loss. This discrepancy suggests a potential issue or instability with the training process that may depend on the initialisations used.

Once again, the model failed to accurately learn the correct order of the derivative. In all cases, the optimal value of $\alpha$ converged to approximately 1, regardless of the true value used during training (see also Figure 18). Furthermore, the gradient of $\alpha$ exhibited large-amplitude oscillations when using the pre-trained method, suggesting that this approach struggles to handle the irregular behaviour observed in this particular case study (see Table 13).

4. Conclusions

This work presents a fundamental study of two methodologies for the joint optimisation of the fractional derivative order and the parameters of the right-hand-side neural network in Neural FDE models. One approach involves bounding the gradient magnitude of the loss function with respect to $\alpha$, promoting more stable and effective updates. Another strategy introduces an online pre-training scheme, in which the network parameters are first optimised over progressively longer time intervals, while $\alpha$ is updated more conservatively using the full time trajectory.

Several case studies were analysed, encompassing problems where the analytical solution and the right-hand-side function exhibit different levels of regularity and varying degrees of dependence on data.

In every case the solution $y\left(t\right)$ and the forcing $f(t,y(t\left)\right)$ are $C^{\infty }$ on the open interval, but typically admit a weak slope singularity of order $t^{\alpha -1}$ and a stronger curvature singularity of order $t^{\alpha -2}$ (or worse) at $t=0$. The quadratic case (Study 2) is the only example without singularities in y itself; all others exhibit algebraic bends near the origin whose strength and complexity depend on the lowest fractional exponent. The curvature profiles range from simple constant concavity (Study 2) to multiple inflections and high nonlinearity (Study 3).

In scenarios with mildly smooth solutions, the proposed methodologies successfully converged to values of $\alpha$ that approximate the ground truth. In cases where the solution itself does not depend on $\alpha$, but the right-hand-side function does, the optimisation algorithm was able to maintain the initial value of $\alpha$, without drifting unnecessarily.

The final two case studies demonstrated the limitations of the proposed methods. In Case Study 3, convergence to the correct fractional order was not achieved within a practical number of epochs, although it is likely that significantly increasing the training time could improve results. In the last case study, small perturbations in the fractional order caused large variations in the right-hand-side function. As a result, despite a good fit to the data, the optimisation algorithm consistently drifted towards $\alpha =1$ (even for different initialisations of $\alpha$).

This work aims to encourage further developments towards methods that not only achieve a good fit to data (a goal successfully attained in the present study) but also ensure convergence to the ground-truth order of the derivative. While the proposed approaches demonstrate promising results in favourable scenarios, they fall short in more challenging problems, particularly those involving low regularity or high sensitivity to the fractional order. These limitations highlight the need for more robust optimisation strategies capable of addressing such complexities.

While our experiments demonstrate the efficacy of Neural FDEs, fractional derivatives are indeed known to enhance the modelling of periodic and chaotic dynamics in nonlinear systems. For instance, fractional-order variants of the Lorenz and Rössler systems exhibit rich bifurcation behaviours and chaotic attractors, while fractional extensions of the KdV and nonlinear Schrödinger equations support soliton solutions with anomalous dispersion [27,28,29,30]. Although such cases are beyond the scope of this work, our framework is theoretically compatible with these systems, as it learns the fractional dynamics directly from data without presuming a specific structure. Future research will explore Neural FDEs for chaotic systems and soliton dynamics, where the non-local memory effects of fractional derivatives could prove particularly advantageous.

Appendix A.1. Case Study 1

Figure A1. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.4$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A1. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.4$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A2. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.4$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A2. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.4$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A3. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.4$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A3. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.4$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A4. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.4$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A4. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.4$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A5. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A5. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A6. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A6. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A7. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A7. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A8. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A8. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A9. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A9. Plot of the predicted and ground-truth curves from the best run in Case Study 1 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A10. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A10. Evolution of the best run’s $\alpha$ during training for Case Study 1 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A11. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A11. Training loss across epochs for the best run in Case Study 1 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A12. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A12. Plot of the best run $\alpha$ gradient value through training for Case Study 1, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Appendix A.2. Case Study 3

Figure A13. Plot of the predicted and ground-truth curves from the best run in Case Study 3 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A13. Plot of the predicted and ground-truth curves from the best run in Case Study 3 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A14. Evolution of the best run’s $\alpha$ during training for Case Study 3 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A14. Evolution of the best run’s $\alpha$ during training for Case Study 3 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A15. Training loss across epochs for the best run in Case Study 3 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A15. Training loss across epochs for the best run in Case Study 3 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A16. Plot of the best run $\alpha$ gradient value through training for Case Study 3, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A16. Plot of the best run $\alpha$ gradient value through training for Case Study 3, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A17. Plot of the predicted and ground-truth curves from the best run in Case Study 3 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A17. Plot of the predicted and ground-truth curves from the best run in Case Study 3 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A18. Evolution of the best run’s $\alpha$ during training for Case Study 3 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A18. Evolution of the best run’s $\alpha$ during training for Case Study 3 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A19. Training loss across epochs for the best run in Case Study 2 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A19. Training loss across epochs for the best run in Case Study 2 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A20. Plot of the best run $\alpha$ gradient value through training for Case Study 3, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A20. Plot of the best run $\alpha$ gradient value through training for Case Study 3, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Appendix A.3. Case Study 4

Figure A21. Plot of the predicted and ground-truth curves from the best run in Case Study 4 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A21. Plot of the predicted and ground-truth curves from the best run in Case Study 4 with $\alpha$ initialised at $0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A22. Evolution of the best run’s $\alpha$ during training for Case Study 4 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A22. Evolution of the best run’s $\alpha$ during training for Case Study 4 with $\alpha =0.8$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A23. Training loss across epochs for the best run in Case Study 4 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A23. Training loss across epochs for the best run in Case Study 4 with $\alpha =0.8$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A24. Plot of the best run $\alpha$ gradient value through training for Case Study 2, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A24. Plot of the best run $\alpha$ gradient value through training for Case Study 2, with $\alpha =0.8$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A25. Plot of the predicted and ground-truth curves from the best run in Case Study 4 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A25. Plot of the predicted and ground-truth curves from the best run in Case Study 4 with $\alpha$ initialised at $0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A26. Evolution of the best run’s $\alpha$ during training for Case Study 4 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A26. Evolution of the best run’s $\alpha$ during training for Case Study 4 with $\alpha =0.99$: (a) original Neural FDE, (b) clipped Neural FDE, and (c) pre-trained Neural FDE.

Figure A27. Training loss across epochs for the best run in Case Study 4 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A27. Training loss across epochs for the best run in Case Study 4 with $\alpha =0.99$: (a) the original, (b) clipped, and (c) pre-trained Neural FDE models.

Figure A28. Plot of the best run $\alpha$ gradient value through training for Case Study 2, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.

Figure A28. Plot of the best run $\alpha$ gradient value through training for Case Study 2, with $\alpha =0.99$, for (a) original, (b) clipped and (c) pre-trained Neural FDE.