A Novel Neural Network with the Ability to Express the Extreme Points Distribution Features of Higher Derivatives of Physical Processes

Abstract

Higher derivatives are important to interpret the physical process. However, higher derivatives calculated from measured data often deviate from the real ones because of measurement errors. A novel method for data fitting without higher derivatives violating the real physical process is developed in this paper. Firstly, the research on errors’ influence on higher derivatives and the typical functions’ extreme points distribution were conducted, which demonstrates the necessity and feasibility of adopting extreme points distribution features in neural networks. Then, we proposed a new neural network considering the extreme points distribution features, namely, the extreme-points-distribution-based neural network (EDNN), which contains a sample error calculator (SEC) and extreme points distribution error calculator (EDEC). With recursive automatic differentiation, a model calculating the higher derivatives of the EDNN was established. Additionally, a loss function, embedded with the extreme points distribution features, was introduced. Finally, the EDNN was applied to two specific cases to reduce the noise in a second-order damped free oscillation signal and an internal combustion engine cylinder pressure trace signal. It was found that the EDNN could obtain higher derivatives that are more compatible with physical trends without detailed differentiation equations. The standard deviation of derivatives’ error of the EDNN is less than 62.5 percent of that of traditional neural networks. The EDNN provides a novel method for the analysis of physical processes with higher derivatives compatible with real physical trends.

1. Introduction

Higher derivatives are important to theoretical research and engineering application [1,2,3,4,5,6] for deepening the understanding of physical processes and improving analysis efficiency [7,8]. However, it is difficult to obtain higher derivatives directly from a real signal because of measurement errors [9]. Though varied techniques such as the Savitzky–Golay polynomial [10,11], Fourier transform [12,13] and wavelet transform [14,15] have been developed to obtain higher derivatives of measured signals, the higher derivatives obtained with the above techniques often deviate from the real one in trends. It is a challenge and urgent demand [9] to develop a data fitting method without higher derivatives violating real trends.

With the development of physics-informed neural networks (PINN) [16,17,18] and the advancement in automatic differentiation [19,20,21,22], complex nonlinear feature recognition [23,24,25,26] and parameter identification of physical processes with well-defined differential equations [27,28] were extensively studied, which demonstrate the neural networks’ ability to obtain higher derivatives.

Though neural networks can fit any continuous function [29,30,31], the higher-derivatives-constrained PINN could reduce the disturbance of noise [32,33,34] in data fitting. However, when lacking detailed mathematical models, the relations between the derivatives could not be established, and PINN could not be applied in fitting those physical processes.

In order to fit the measured signal without a discrepancy in the trend of higher derivatives, stemming from the local monotonicity of the real physical process [35,36], the necessity and feasibility of adopting the number of extreme points as constraints to data fitting with neural networks are proved. Then, an extreme-points-distribution-based neural network (EDNN) is proposed and applied in noise reduction. The research indicates that the EDNN could obtain higher derivatives that are more compatible to physical trends without detailed differential relations and could be applied in the denoising of measured signal with higher derivatives conforming to real physical trends. The proposed EDNN provides a novel way for signal denoising that retains higher derivatives compatible with real physical process trends.

2. Analysis of Noise’s Influence on Higher Derivatives and Extreme Points Distribution of Typical Process Functions

2.1. Noise’s Influence on the Higher Derivatives of the Measured Signal

Generally, the signal to be measured $f(t)$ is continuous in the time domain. In the neighborhood of $t_0$, it can be approximated using a Taylor series as in expression (1).

$$f(t)=f(t_0)+f^{\prime }(t_0)(t-t_0)+\frac{1}{2!}{f}^{″}(t_0){(t-t_0)}^2+\cdots +\frac{1}{n!}{f}^{(n)}(t_0){(t-t_0)}^n+o({(t-t_0)}^n)$$

(1)

Suppose that there is a continuous noise $\epsilon \left(t\right)$ in the vicinity of $t_0$; the noise $\epsilon \left(t\right)$ can also be approximated using a Taylor series as in expression (2).

$$\epsilon \left(t\right)=\epsilon \left(t_0\right)+{\epsilon }^{\prime }(t_0)(t-t_0)+\frac{1}{2!}{\epsilon }^{″}(t_0){(t-t_0)}^2+\cdots +\frac{1}{n!}{\epsilon }^{(n)}(t_0){(t-t_0)}^n+o({(t-t_0)}^n)$$

(2)

The signal to be measured and noise can be superposed together. Then, the measured signal $C\left(t\right)$ in the vicinity of $t_0$ can be approximated using a Taylor series as in expression (3).

$$\begin{array}{l}C\left(t\right)=f(t)+\epsilon \left(t\right)\\ =f(t_0)+f^{\prime }(t_0)(t-t_0)+\frac{1}{2!}{f}^{″}(t_0){(t-t_0)}^2+\cdots \\ +\frac{1}{n!}{f}^{(n)}(t_0){(t-t_0)}^n+o({(t-t_0)}^n)+\epsilon \left(t_0\right)+{\epsilon }^{\prime }(t_0)(t-t_0)\\ +\frac{1}{2!}{\epsilon }^{″}(t_0){(t-t_0)}^2+\cdots +\frac{1}{n!}{\epsilon }^{(n)}(t_0){(t-t_0)}^n+o({(t-t_0)}^n)\end{array}$$

(3)

Because the noise $\epsilon \left(t\right)$ is random, there exists the probability that the first k terms of the Taylor series at $t_0$ are zero and the term after the k + 1 term can be ignored; thus:

$$\begin{array}{ll}\epsilon \left(t\right)& =\epsilon \left(t_0\right)+{\epsilon }^{\prime }(t_0)(t-t_0)+\frac{1}{2!}{\epsilon }^{″}(t_0){(t-t_0)}^2+\cdots +\frac{\epsilon \left(t_t\right)}{{(t_t-t_0)}^k}{(t-t_0)}^k\\ & +\cdots \frac{1}{n!}{\epsilon }^{(n)}(t_0){(t-t_0)}^n+o({(t-t_0)}^n)=\frac{1}{k!}{\epsilon }^{(k)}(t_0){(t-t_0)}^k\end{array}$$

(4)

Suppose the noise amplitude at the time $t_t$ is ${\epsilon }_t$, i.e., $\epsilon \left(t_t\right)={\epsilon }_t$, then:

$$\epsilon \left(t_t\right)=\frac{1}{k!}{\epsilon }^{(k)}(t_0){(t_t-t_0)}^k$$

(5)

$${\epsilon }^{(k)}(t_0)=\frac{\epsilon \left(t_t\right)}{\frac{1}{k!}{(t_t-t_0)}^k}=\frac{{\epsilon }_t}{\frac{1}{k!}{(t_t-t_0)}^k}$$

(6)

For $\forall$$A_{\epsilon }$, when $\left|{\epsilon }_t\right|>\left|A_{\epsilon }·\frac{1}{k!}{(t_t-t_0)}^k\right|$, $\exists$$\left|{\epsilon }^{(k)}(t_0)\right|>\left|A_{\epsilon }\right|$.

Therefore, when the noise amplitude ${\epsilon }_t$ is a constant, as $t_t$ approaches $t_0$, $\frac{\epsilon \left(t_t\right)}{{(t_t-t_0)}^k}$ will become larger and larger, as will ${\epsilon }^{(k)}(t_0)$. That is, the kth order derivative of the noise at $t_0$ becomes larger and larger. Subsequently, the kth order derivative of the measured signal $C\left(t\right)$ also becomes larger and larger. When ${\epsilon }_t$ is large enough or $\left(t_t-t_0\right)$ is small enough, the kth order derivative of noise $\epsilon \left(t\right)$ at $t_0$ will be large enough, and the kth order derivative of measured signal $C\left(t\right)$ at $t_0$ will deviate seriously from the corresponding derivative of the signal $f(t)$.

The above analysis shows that even if the Taylor series expansion of the noise has only the kth order term, provided the kth order derivative is large enough, the kth order derivative of the measured signal can be seriously interfered with. Additionally, the kth order derivative of the measured signal will deviate substantially from the kth order derivative of the real physical process. Therefore, it is necessary to reduce the noise disturbance to obtain the higher derivatives of the measured data conforming to the real physical trend.

2.2. Extreme Points Distribution of Typical Process Functions

When fitting a measured signal using neural networks, noise often leads to the higher derivatives of the fitted signal to deviate from the true physical trend [37].

Generally, the relation between the physical quantities and time of a physical process can be modeled by a polynomial, Gaussian function, cosine function, or exponential function. The extreme points of those functions’ higher derivatives on a specific interval are distributed in certain patterns. When extreme points distribution is involved in data fitting, the approximation between the higher derivatives of the measured signal and the higher derivatives of the real physical process could be improved. It is necessary to analyze the extreme points’ distribution pattern of those typical process functions and their higher derivatives to involve the extreme points distribution in data fitting with neural networks.

An nth order polynomial modeling a continuous process is shown in expression (7).

$$p\left(x\right)=\sum _{i=0}^n{a}_i·x^i$$

(7)

Additionally, the kth order derivative of an nth order polynomial can be obtained in (8).

$$p^{(k)}\left(x\right)=k!a_k+\frac{(k+1)!}{1!}{a}_{(k+1)}x+\frac{(k+2)!}{2!}{a}_{(k+2)}{x}^2+\cdots +\frac{n!}{(n-k)!}{a}_n{x}^{n-k}$$

(8)

It can be seen that the kth order derivative of nth order polynomial is an (n − k)th-order polynomial. Therefore, the number of the kth-order derivative’s extreme points decreases with the increase of order $k$. When fitting a measured signal modeled by an nth-order polynomial, the extreme point number could be applied to test the consistency between the fitted kth order derivative and that of the real physical process.

An exponential function describing the growth or decline process is shown in (9).

$$f\left(x\right)=a^x$$

(9)

Additionally, the kth-order derivative of the exponential function is shown in (10).

$$f^{(k)}\left(x\right)={{\left[ln\left(a\right)\right]}^{k}a}^x$$

(10)

According to (10), the number of extreme points of the 0–kth-order derivatives of the exponential function is always zero. When fitting a measured signal modeled using the exponential function, the number of extreme points should always remain zero. Any extreme point appearing in the fitting results indicates a deviation from the physical trend.

A Gaussian function describing a random process is shown in expression (11).

$$g\left(x\right)=e^{a{x}^2}$$

(11)

The kth-order derivative of the Gaussian function is shown in expression (12).

$$\frac{d^{n}g(x)}{d{x}^n}=e^{a{x}^2}{\sum }_{i=0}^n{C}_{it}^n(a)x^m$$

(12)

where $C_{it}^n(a)=\frac{{(2a)}^t}{1+i{\epsilon }_n^{+}}{F}_i(n)[(n-i)-{\epsilon }_n^{+}]!!$, $m=i+{\epsilon }_n^{-}$, ${\epsilon }_n^{\pm }=\frac{1}{2}[1\pm {(-1)}^n]$, $t=\frac{(n+i)}{2}+{\epsilon }_n^{-}$ and $F_i(n)=\frac{n!}{i!(n-i)!}$ is the binomial coefficient.

The highest degree of $x^m$ in expression (12) increases as the derivative order increases [38]. Then, the number of the derivative’s extreme points increases as the derivative order increases. When fitting a measured signal modeled using the Gaussian function, the extreme point number could be applied to test the consistency between the fitted kth-order derivative and that of the real physical process.

The cosine function describing a simple periodic process is shown in (13).

$$h(x)=cos\left(x\right)$$

(13)

The kth-order derivative of the cosine function is shown in expression (14).

$$h^{(k)}(x)=cos(\frac{k\pi }{2}+x)$$

(14)

Cosine function shifted left by $k\pi /2$ yields its kth-order derivative. If the number of extreme points of the cosine function is $N_{hp}$ on any arbitrary interval $\left[-a,+a\right]$, the number of extreme points of the cosine function’s kth-order derivative is $N_{hp}-1$, $N_{hp}$ or $N_{hp}+1$. When fitting a measured signal using a cosine function, it is feasible to judge whether the higher derivative deviates from the physical trend according to whether an abnormal number of extreme points appears.

In summary, there are definite relations between the number of extreme points and the derivative order for the typical process function. Moreover, the pattern of a real physical process could be modeled using a typical function or its combinations. Therefore, the number of extreme points of the higher derivatives could be applied as constraints to fit the measured signal for denoising purpose, which will get noise-reduced signals with higher derivatives conforming to the physical trend.

3. The Extreme-Points-Distribution-Based Neural Network (EDNN)

For similar physical processes, the functions modelling the physical quantities varying with time and space have the same form. Additionally, the numbers of the extreme points of those functions and their derivatives are confined in a certain range. Therefore, a novel neural network that introduces the extreme points distribution as constraints was proposed in order to acquire higher derivatives conforming to physical trends. The outline of building the EDNN is shown in Figure 1.

3.1. The Architecture of the Extreme-Points-Distribution-Based Neural Network (EDNN)

The extreme-points-distribution-based neural network (EDNN) is composed of an input layer, hidden layers, an output layer, an automatic differentiation layer, an extreme points distribution feature layer and a loss function containing extreme points distribution errors. As shown in Figure 2, the EDNN contains a sample error calculator (SEC) and an extreme points distribution error calculator (EDEC). The total loss function of the EDNN is composed of the sample error loss function LossSp and the extreme points distribution error loss function LossEV.

The sample error loss function LossSp is calculated from the difference between the output f_s of the sample error calculator (SEC) and the corresponding target f_t. The extreme points distribution feature error loss function LossEV is calculated using the extreme points distribution error calculator (EDEC). For each x_d in the measured signal’s definition domain, the automatic differentiation layer (ADLayers) calculates the 0–(k+1)th-order derivatives. Additionally, the extreme points distribution feature layer (EVLayer) calculates the number of extreme points of the 0–kth-order derivatives. Then, LossEV can be calculated from the difference between the calculated extreme points number and extreme points number of the real physical process.

The total loss function of the EDNN (Loss) is the weighted summation of LossSp and LossEV. The EDNN is trained, and the weight and the bias are updated until the total loss drops below the stopping criterion or the gradient is less than the stopping criterion.

3.2. Recursive Automatic Differentiation

Calculating the derivatives of the output with respect to the input is necessary for the EDNN. A recursive formulation is established for calculating the derivatives. The current layer’s derivatives with respect to the input are modeled as a function of the previous layer’s derivatives with respect to the input.

3.2.1. Derivatives of Hidden Layers

The derivative of the (l + 1)th hidden layer with respect to the input layer can be formulated as a function of the derivatives of the lth hidden layer with respect to the input layer.

The ith node’s output of the (l + 1)th layer is denoted as ${\mathit{a}}_i^{l+1}\left(\mathit{x}\right)$, and its mth-order partial derivative with respect to the input layer’s nth node ${\mathit{x}}_n^I$ is denoted as $D^m\left({\mathit{a}}_i^{l+1}\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$.

According to $\mathrm{F}\mathrm{a}\stackrel{`}{\mathrm{a}}$ di Bruno’s formula [39,40], $D^m\left({\mathit{a}}_i^{l+1}\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ can be expressed as:

$$\begin{array}{l}{D}^m\left({\mathit{a}}_i^{l+1}\left(\mathit{x}\right),{\mathit{x}}_n^I\right)\\ =\sum \frac{m!}{b_1!b_2!\cdots b_m!}{\left(f_i^{l+1}\right)}^{\left(k\right)}\left({\sum }_{j=1}^{S^l}{\mathit{w}}_{i,j}^{l+1,l}·{\mathit{a}}_j^l\left(\mathit{x}\right)+{\mathit{b}}_i^{l+1}\right)\\ ·{\left(\frac{\sum _{j=1}^{S^l}{\mathit{w}}_{i,j}^{l+1,l}·D^1\left({\mathit{a}}_j^l\left(\mathit{x}\right),{\mathit{x}}_n^I\right)}{1!}\right)}^{b_1}\\ ·{\left(\frac{\sum _{j=1}^{S^l}{\mathit{w}}_{i,j}^{l+1,l}·D^2\left({\mathit{a}}_j^l\left(\mathit{x}\right),{\mathit{x}}_n^I\right)}{2!}\right)}^{b_2}\cdots {\left(\frac{\sum _{j=1}^{S^l}{\mathit{w}}_{i,j}^{l+1,l}·D^m\left({\mathit{a}}_j^l\left(\mathit{x}\right),{\mathit{x}}_n^I\right)}{m!}\right)}^{b_m}\end{array}$$

(15)

where the sum is over all different solutions in non-negative integers $b_1,b_2,\cdots {,b}_m$ of $b_1+{2b}_2+\cdots +m{b}_m=m$ and $k=b_1+b_2+\cdots +b_m$; ${\left(f_i^{l+1}\right)}^{\left(k\right)}\left({\sum }_{j=1}^{S^l}{\mathit{w}}_{i,j}^{l+1,l}·{\mathit{a}}_j^l\left(\mathit{x}\right)+{\mathit{b}}_i^{l+1}\right)$ is the kth-order derivative of the activation function; $S^l$ is the number of nodes in layer $l$; ${\mathit{w}}_{i,j}^{l+1,l}$ is the weight of the jth node in the lth layer to the ith node in the (l + 1)th layer; ${\mathit{b}}_i^{l+1}$ is the bias of the ith node of the (l + 1)th layer; $l=1,\cdots ,\left(L_h-1\right)$, $L_h$ is the total number of hidden layers.

When there is no transform in the input layer, the mth-order partial derivative $D^m\left({\mathit{a}}_i^1\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ of the ith node’s output ${\mathit{a}}_i^1\left(\mathit{x}\right)$ in the first hidden layer with respect to the nth node of the input layer is shown in expression (16).

$$D^m\left({\mathit{a}}_i^1\left(\mathit{x}\right),{\mathit{x}}_n^I\right)={\left(f_i^1\right)}^{\left(m\right)}\left({\sum }_{j=1}^{S^I}{\mathit{w}}_{i,j}^{1,I}·{\mathit{x}}_j^I+{\mathit{b}}_i^1\right)·{\left({\mathit{w}}_{i,n}^{1,I}\right)}^m$$

(16)

where ${\left(f_i^1\right)}^{\left(m\right)}\left({\sum }_{j=1}^{S^I}{\mathit{w}}_{i,j}^{1,I}·{\mathit{x}}_j^I+{\mathit{b}}_i^1\right)$ is the mth-order derivative of the activation function; $S^I$ is the number of nodes of the input layer; ${\mathit{w}}_{i,j}^{1,I}$ is the weight of the jth node in the input layer to the ith node in the first hidden layer; ${\mathit{b}}_i^1$ is the bias of the ith node of the first hidden layer.

3.2.2. Derivatives of Output Layers

The partial derivatives of the output layer with respect to the input layer are modeled as the functions of the partial derivatives of the last hidden layer with respect to the input layer. The mth-order partial derivative of the output layer’s ith node with respect to the nth node in the input layer is shown in expression (17).

$$D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)={\sum }_{j=1}^{S^{L_h}}{\mathit{w}}_{i,j}^{O,L_h}·\frac{{\partial }^m\left[{\mathit{a}}_j^{L_h}\left(\mathit{x}\right)\right]}{{\partial \left({\mathit{x}}_n^I\right)}^m}={\sum }_{j=1}^{S^{L_h}}{\mathit{w}}_{i,j}^{O,L_h}·D^m\left({\mathit{a}}_i^{L_h}\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$$

(17)

where $L_h$ is the total number of hidden layers; $S^{L_h}$ is the number of nodes in the $(L_h)\mathrm{t}\mathrm{h}$ hidden layer; ${\mathit{w}}_{i,j}^{O,L_h}$ is the weight of the jth node in the ${(L}_h)\mathrm{t}\mathrm{h}$ hidden layer to the ith node in the output layer.

3.3. Extreme Points Distribution Feature Layer

The role of the extreme points distribution feature layer is calculating the extreme points number. The extreme points of the mth-order partial derivatives $D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ are derived from the (m + 1)th-order derivative $D^{m+1}\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$.

Let:

$$D^{m+1}\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)=0$$

(18)

Equation (18) is an equation for x.

The solution of Equation (18) is a set:

$${Sol}_{m,i,n}=\left\{\mathit{x}\left|D^{m+1}\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)=0\right.\right\}$$

(19)

The number of extreme points of $D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ is the rank of the set ${Sol}_{m,i,n}$:

$$R_{m,i,n}=Rank\left(\mathit{x}\left|D^{m+1}\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)=0\right.\right)$$

(20)

where $Rank\left(\right)$ is the ranking function, $R_{m,i,n}$ is the number of extreme points of $D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ and $R_{m,i,n}$ is defined as the extreme points distribution feature.

3.4. Loss Function Containing Extreme Points Distribution Feature Errors

The number of extreme points of the real physical process is the reference for calculating the extreme points distribution feature error.

If the number of extreme points of derivative $D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ is $R_{m,i,n}$ and the number of extreme points of the real physical process is ${TCE}_{n,i,m}$, then the extreme points distribution feature error of derivative $D^m\left({\mathit{a}}_i^O\left(\mathit{x}\right),{\mathit{x}}_n^I\right)$ is defined as:

$${Er}_{m,i,n}=R_{m,i,n}-{TCE}_{n,i,m}$$

(21)

The extreme points distribution feature error loss function LossEV is defined as:

$$LossEV={\sum }_{n=1}^{N_I}{\sum }_{i=1}^{N_o}{\sum }_{m=0}^{M_d}\left\{{\left({Er}_{m,i,n}\right)}^2\right\}$$

(22)

where $N_o$ is the number of nodes in the output layer, $N_I$ is the number of nodes in the input layer and $M_d$ is the highest order of the partial derivatives.

The sample error loss function LossSp is defined as:

$$LossSp={\sum }_{s=1}^{N_t}{\sum }_{i=1}^{N_o}{\left[{\mathit{t}}_{i,s}-{\mathit{a}}_{i,s}^O\left(\mathit{x}\right)\right]}^2$$

(23)

where ${\mathit{t}}_{i,s}$ is the $i\mathrm{t}\mathrm{h}$ component of the $sth$ training sample; ${\mathit{a}}_{i,s}^O\left(\mathit{x}\right)$ is the $i\mathrm{t}\mathrm{h}$ node’s output of the output layer corresponding to the $s\mathrm{t}\mathrm{h}$ training sample; $N_t$ is the number of training samples.

The total loss function of the EDNN is defined as:

$$Loss=S_S·LossSp+S_E·LossEV$$

(24)

where $S_S$ is the weight of the sample error loss function LossSp; $S_E$ is the weight of the extreme points distribution feature error loss function LossEV.

The optimization criterion is to minimize the total loss function. Additionally, the stopping criterion is when the total loss function is less than 10⁻⁴ or the gradient is less than 10⁻⁷.

4. Application of the EDNN in Denoising

The research on signal denoising with EDNN was conducted. Firstly, a single-input, single-output, single-hidden-layer EDNN was realized. Then the EDNN is applied in the denoising of a second-order damped free oscillation signal and an internal combustion engine cylinder pressure trace signal.

4.1. Realization of EDNN

The single-input, single-output EDNN is shown in Figure 3. There are eight nodes in the hidden layer, and the activation function of the hidden layer is a sigmoid function. The output of the sample error calculator (SEC) is:

$$a^o(x)=f\left[({\sum }_{i=1}^{8}f\left(W_{i,1}^{1,I}\ast x+b_i^1\right))·W_{1,i}^{\mathrm{2,1}}+b_1^2\right]$$

(25)

where $W_{i,1}^{1,I}$ is the weight of the input to the $i\mathrm{t}\mathrm{h}$ node of the first hidden layer; $W_{1,i}^{\mathrm{2,1}}$ is the weight of the $i\mathrm{t}\mathrm{h}$ node in the first hidden layer to output; $b_{\mathrm{i}}^1$ is the bias of the $i\mathrm{t}\mathrm{h}$ node of the first hidden layer; $b_1^2$ is the bias of the output.

The $k\mathrm{t}\mathrm{h}$ derivative of the output with respect to the input is shown in (26).

$$\frac{d^k{a}^o(x)}{d{x}^k}={\sum }_{i=1}^n{f}^{(k)}\left(W_{i,1}^{1,I}\ast x+b_i^1\right)·W_{1,i}^{\mathrm{2,1}}\ast W_{i,1}^{1,I}$$

(26)

where $f^{(k)}\left(W_{i,1}^{1,I}\ast x+b_i^1\right)$ represents the $k\mathrm{t}\mathrm{h}$-order derivative of the activation function corresponding to the $i\mathrm{t}\mathrm{h}$ node of the hidden layer.

The highest derivative order of the voltage with respect to the time of the second-order damped free oscillation signal is two. Thus, the highest derivative order of the EDNN could be set to four, which satisfies the noise reduction demand. Therefore, the maximum value of k in expression (26) is four. Additionally, only the first- through fourth-order derivatives of the activation function are required [41]. The first- through fourth-order derivatives of the activation function are shown in expressions (27)–(30):

$$f^{\prime }\left(x\right)=f\left(x\right)-{\left[f\left(x\right)\right]}^2$$

(27)

$$f^{″}\left(x\right)=f\left(x\right)-3{\left[f\left(x\right)\right]}^2+2{·\left[f\left(x\right)\right]}^3$$

(28)

$$f^{(3)}\left(x\right)=f\left(x\right)-7·{\left[f\left(x\right)\right]}^2+12·{\left[f\left(x\right)\right]}^3-6·{\left[f\left(x\right)\right]}^4$$

(29)

$$f^{(4)}\left(x\right)=\left\{f\left(x\right)-15·{\left[f\left(x\right)\right]}^2+50·{\left[f\left(x\right)\right]}^3-60·{\left[f\left(x\right)\right]}^4+24·{\left[f\left(x\right)\right]}^5\right\}$$

(30)

where $f\left(x\right)=\frac{1}{1+e^{-x}}$ is the sigmoid function; $f^{\prime }\left(x\right),f^{″}\left(x\right),f^{(3)}\left(x\right)$,$f^{(4)}\left(x\right)$ are the first, second, third, and fourth derivatives of $f\left(x\right)$ with respect to $x$.

The $k\mathrm{t}\mathrm{h}$ derivative of the output with respect to the input is $\frac{d^k{a}^o(x)}{d{x}^k}$. Additionally, the solution of $\frac{d^k{a}^o(x)}{d{x}^k}=0$ is:

$${Sol}_{k-1}=\left\{x\left|\frac{d^k{a}^o(x)}{d{x}^k}=0\right.\right\}$$

(31)

The number of extreme points $R_{k-1}$ corresponding to the derivative $\frac{d^{k-1}{a}^o(x)}{d{x}^{k-1}}$ is the rank of the set ${Sol}_{k-1}$, i.e.:

$$R_{k-1}=Rank\left(x\left|\frac{d^k{a}^o(x)}{d{x}^k}=0\right.\right)$$

(32)

when the number of extreme points of the real physical process corresponding to the derivative $\frac{d^{k-1}{a}^o(x)}{d{x}^{k-1}}$ is ${TCE}_{k-1}$.

Therefore, the extreme points distribution feature error corresponding to the $(k-1)\mathrm{t}\mathrm{h}$ derivative $\frac{d^{k-1}{a}^o(x)}{d{x}^{k-1}}$ is:

$${Er}_{k-1}=R_{k-1}-{TCE}_{k-1}$$

(33)

where ${TCE}_{k-1}$ is determined using known physical process information.

The total loss function of the EDNN is:

$$Loss=S_S·LossSp+S_E·LossEV=S_S·{\sum }_{s=1}^{N_t}{\left[t_s-a_s^o(x)\right]}^2+S_E·{\sum }_{k=0}^{k=4}{\left({Er}_{k-1}\right)}^2$$

(34)

where LossEV is the extreme points distribution feature error loss function; LossSp is the sample error loss function; $S_S$ is the weight of the sample error loss function LossSp; $S_E$ is the weight of the extreme points distribution feature error loss function LossEV; $t_s$ is target of the $sth$ training sample; $a_s^o(x)$ is the output corresponding to the $sth$ training sample; $N_t$ is the number of training samples.

4.2. Denoising of Second-Order Damped Free Oscillation Signal via EDNN

4.2.1. Acquisition of Second-Order Damped Free Oscillation Signal

The measured second-order damped free oscillation (SODFO) signal has errors, and the effect of the errors on the derivatives is difficult to evaluate. Therefore, a SODFO signal is obtained via simulation. High-frequency noise and random noise were superposed on the simulated signal. Then, the performance of the EDNN on denoising was studied.

The simulation model of SODFO signal is shown in expression (35). Additionally, the simulated signal is a cosine-shape curve with decreasing amplitude (shown in Figure 4a). The simulation model of the high-frequency noise is shown in expression (36). Moreover, the simulated high-frequency noise is a cosine signal with a frequency 10 times the frequency of the SODFO signal (shown in Figure 4b). The simulation model of the random noise is shown in expression (37). Further, random noise is a uniformly distributed noise on [−0.01, 0.01] (shown in Figure 4c). The superposition of the higher-order noise and random noise on the SODFO signal does not change the amplitude, apparently (shown in Figure 4d).

$$y_c=10·e^{(-0.3·t)}·cos(3·t)$$

(35)

$$h_{noise}=0.02·cos(30·t)$$

(36)

$$r_{noise}~U(-0.01,0.01)$$

(37)

$$y=y_c+h_{noise}+r_{noise}$$

(38)

In expressions (35)–(38), the range of time t is [0.4, 6] seconds; y is the noisy second-order damped free oscillation (NSODFO) signal. The time t, signal y and the signal $y_c$ are vectors of length 491.

4.2.2. Comparative Research of EDNN with Shallow Neural Network on Denoising of Second-Order Damped Free Oscillation Signal

Comparative research on denoising performance of the EDNN was conducted. The NSODFO signal processed using the shallow neural network and the EDNN were compared. The SODFO signal, the noise-removed signal from the shallow neural network (NRSNN signal) and the noise-removed signal from the EDNN (NREDNN signal) are shown in Figure 5.

It can be found in Figure 5a that the SODFO signal, NRSNN signal and the NREDNN signal are well coincident. In Figure 5b, the differences between the denoising signals and the SODFO signal show that the errors of the NRSNN signal and the NREDNN signal are of the same magnitude. The maximum error of the NRSNN signal is 0.09 percent of the maximum amplitude of the SODFO signal. The maximum error of the NREDNN signal is 0.08 percent of the maximum amplitude of the SODFO signal. It indicates that the shallow neural networks and the EDNN have similar performance in approximating the amplitude of the signal.

The first- through fourth-order derivatives and the derivative errors are shown in Figure 6.

Figure 6a is the first-order derivative and the errors. The first-order derivatives of the NRSNN signal and the NREDNN signal coincide with the first-order derivative of the SODFO signal. The maximum error of the first-order derivative of the NREDNN signal is 0.16 percent of the maximum amplitude of the first-order derivative of the SODFO signal. The maximum error of the first-order derivative of the NRSNN signal is 0.49 percent of the maximum amplitude of the first-order derivative of the SODFO signal.

Figure 6b is the second-order derivatives and the errors. The second-order derivatives of the NRSNN signal and the NREDNN signal coincide with the second-order derivative of the SODFO signal. The maximum error of the second-order derivative of the NREDNN signal is 0.42 percent of the maximum amplitude of the second-order derivative of the SODFO signal. The maximum error of the second-order derivative of the NRSNN signal is 2.22 percent of the maximum amplitude of the second-order derivative of the SODFO signal.

Figure 6c is the third-order derivatives and the errors. The third-order derivative of the NREDNN signal coincides with the third-order derivative of the SODFO signal. Additionally, no trend discrepancy occurs in the third-order derivative of the NREDNN signal. However, there is a trend discrepancy occurs on the third-order derivative of the NRSNN signal (Figure 6d). The maximum error of the third-order derivative of the NREDNN signal is 4.02 percent of the maximum amplitude of the third-order derivative of the SODFO signal. The maximum error of the third-order derivative of the NRSNN signal is 12.05 percent of the maximum amplitude of the third-order derivative of the SODFO signal.

Figure 6e is the fourth-order derivatives and the errors. The fourth-order derivatives of the NRSNN signal and the NREDNN signal coincide with the fourth-order derivative of the SODFO signal. The maximum error of the fourth-order derivative of the NREDNN signal is 39.92 percent of the maximum amplitude of the fourth-order derivative of the SODFO signal. The maximum error of the fourth-order derivative of the NRSNN signal is 114.24 percent of the maximum amplitude of the fourth-order derivative of the SODFO signal. The error of NREDNN signal is less than that of the NRSNN signal (Figure 6f).

The standard deviation of the first- through fourth-order derivatives are shown in

Table 1. The standard deviation of the derivatives.

	1st Derivative	2nd Derivative	3rd Derivative	4th Derivative
SNN_18 neural network	0.016	0.215	2.642	28.692
EDNN neural network	0.010	0.074	0.751	8.414

. The standard deviation error between the derivatives of the SODFO signal and that of the NREDNN signal is less than 62.5 percent of the standard deviation errors between the derivatives of the SODFO signal and the NRSNN signal.

In summary, the noise reduction performance of the NREDNN is better than that of the shallow neural networks. Additionally, no trend discrepancy occurred in the first- through fourth-order derivatives of the NREDNN signal. Those are the advantages of the EDNN. However, the EDNN requires more memory to store the derivatives and more computational time to calculate the derivative and the extreme feature error.

4.3. Denoising of the Cylinder Pressure Signal with EDNN

The internal combustion engine cylinder pressure trace is the variation of cylinder pressure with crank angle. Generally, the cylinder pressure trace is recorded using a piezoelectric high-temperature pressure sensor and smoothed via cyclic averaging [42]. The cyclically averaged cylinder pressure trace can satisfy the demands of calculation of indicated mean effective pressure. However, when the analysis involves derivatives beyond the second order, the noise often leads to deviation from the physical trend. The denoising performance of the EDNN on cylinder pressure trace was studied.

The cylinder signal is the in-cylinder pressure which varies with the crank angle. The pressure signal is between the −120 degree CA and 120 degree CA, and the sample interval is 0.1 degrees CA. Additionally, the cylinder pressure signal is a vector of length 2400 whose maximum value is 1.6946 × 10⁶ Pa.

The single-input, single-hidden-layer, single-output EDNN with eight nodes in the hidden layer was applied to reduce the noise of the cylinder pressure trace. Moreover, the noise reduction results were compared with that from a shallow neural network with 8 nodes (SNN_8), a shallow neural network with 18 nodes (SNN_18) and the smoothing splines that are traditionally adopted in smoothing cylinder pressure trace.

The noise reduction results are shown in Figure 7. The noise-reduced signals of the four techniques are all consistent with the measured cylinder pressure (Figure 7a). Additionally, the errors between the noise-reduced signal and the measured signal in Figure 7b are all less than 400 Pa, and the relative error is less than 1.2 × 10⁻⁶%, which satisfies the accuracy requirements.

Figure 8 shows the first- through fourth-order derivatives. It can be seen in Figure 8a that the first-order derivatives of the noise-reduced signal from the four techniques are all consistent with each other, and there are no fluctuations deviating from the physical trend. The first-order derivative of the measured signal shows a small fluctuation near the zero-degree crank angle, which deviates from the physical trend.

It can be seen in Figure 8b that the second-order derivative of the noise-reduced signal from the EDNN and the SNN_18 neural network is consistent with real trend and there is no fluctuations deviating from the physical trend. The second-order derivatives of the noise-reduced signal from the SNN_8 neural network and the smoothing splines showed small fluctuations. Further, the second-order derivative of the measured signal shows frequent fluctuations deviating from the physical trend in the whole crank angle range.

It can be seen in Figure 8c that there are no fluctuations deviating from the physical trend occur on the third-order derivative of the noise-reduced signal, obtained from the EDNN. There are small fluctuations on the third-order derivative of the noise-reduced signal from the SNN_18 neural network. In addition, there are obvious fluctuations on the third-order derivative of the noise-reduced signal from the SNN_8 neural network. There are frequent fluctuations on the third-order derivative of the noise-reduced signal, obtained from the smoothing spline. Additionally, the third-order derivative of the measured signal is submerged in the noise.

It can be seen in Figure 8d that there are no fluctuations deviating from the physical trend on the fourth-order derivative of the noise-reduced signal, obtained from the EDNN. There are obvious fluctuations on the fourth-order derivative of the noise-reduced signal from the SNN_18 neural network and the SNN_8 neural network. Additionally, the fourth-order derivative of the measured signal and the noise-reduced signal obtained from the smoothing splines are submerged in the noise.

In summary, the noise reduction performance of the EDNN is better than that of the SNN_8 neural network, the SNN_18 neural network and the smoothing splines. Further, there is no trend discrepancy in the first- through fourth-order derivatives of the noise-reduced cylinder pressure signal from the EDNN.

The EDNN could derive higher derivatives consistent with the real physical process in the absence of the process’ detailed mathematical model. However, the EDNN needs more computational resources due to the recording and calculation of the higher derivatives.

5. Conclusions

The effect of the error on the higher derivatives of the measured signal was analyzed. The feasibility of using the extreme points distribution as constraints in data fitting was studied. Then, the extreme-points-distribution-based neural network (EDNN) was proposed. Finally, the superiority of the EDNN on signal denoising was verified. The details are as follows:

• The error’s deviation effect on the higher derivatives of the measured signal was analyzed and a possible way of applying the extreme points distribution as constraints on data fitting was studied.

A mathematical model was established for the analysis of the error’s effect on higher derivatives. It was found that even though the Taylor series of the noise only has the $k\mathrm{t}\mathrm{h}$-order term and the coefficients of other terms are zero, the $k\mathrm{t}\mathrm{h}$-order derivative of the measured signal could deviate from the real physical process greatly as long as the $k\mathrm{t}\mathrm{h}$-order derivative of the noise is large enough. The necessity of finding a way to fit data with higher derivatives compatible to real process trends was clarified.

The higher derivative’s extreme points distribution of typical process functions were analyzed, and the pattern of the extreme points distribution of higher derivatives was investigated. The pattern provides a theoretical basis for applying the extreme points distribution as a constraint in denoising with data fitting techniques.

2.

The extreme points distribution pattern was adopted as a constraint and the EDNN was established. The EDNN consists of an input layer, hidden layers, an output layer, an automatic differentiation layer and an extreme points distribution feature layer. A recursive formulation was established for calculating the derivatives. Additionally, a novel loss function, embedded with the extreme feature error, was proposed.

3.

The effectiveness of the EDNN on signal denoising was verified. The proposed EDNN was applied to reduce the noise in the second-order damped free oscillation signal and the internal combustion engine cylinder pressure signal. Compared with shallow neural networks and smoothing splines, the EDNN could obtain higher derivatives that are consistent with the real physical process in the absence of a detailed mathematical model. Therefore, data fitting for higher derivatives conforming to real physical process trends could be realized with EDNN, which provides a novel approach for analyzing physical processes with higher derivatives. Additionally, it could be used in understanding the real process with higher derivatives.

In summary, the advantage of the EDNN is that it could fit the measured signal with higher derivatives compatible to real physical process trends, which provides a novel tool to mine information of the real physical process through higher derivatives. However, it needs more computational resources and needs the knowledge of extreme distribution of the real process. The aim of future research is to study the computational efficiency of the EDNN and apply it in other engineering fields.