Nonlinear Dynamic System Identification in the Spectral Domain Using Particle-Bernstein Polynomials

Abstract

System identification (SI) is the discipline of inferring mathematical models from unknown dynamic systems using the input/output observations of such systems with or without prior knowledge of some of the system parameters. Many valid algorithms are available in the literature, including Volterra series expansion, Hammerstein–Wiener models, nonlinear auto-regressive moving average model with exogenous inputs (NARMAX) and its derivatives (NARX, NARMA). Different nonlinear estimators can be used for those algorithms, such as polynomials, neural networks or wavelet networks. This paper uses a different approach, named particle-Bernstein polynomials, as an estimator for SI. Moreover, unlike the mentioned algorithms, this approach does not operate in the time domain but rather in the spectral components of the signals through the use of the discrete Karhunen–Loève transform (DKLT). Some experiments are performed to validate this approach using a publicly available dataset based on ground vibration tests recorded from a real F-16 aircraft. The experiments show better results when compared with some of the traditional algorithms, especially for large, heterogeneous datasets such as the one used. In particular, the absolute error obtained with the prosed method is 63% smaller with respect to NARX and from 42% to 62% smaller with respect to various artificial neural network-based approaches.

1. Introduction

System identification (SI) is a discipline concerned with finding the mathematical models of a dynamic system based on observations of its inputs and outputs. Due to applications of SI to a variety of fields, such as the analysis and simulation of complex systems or the control of dynamic processes, a rich body of research is available on SI [1,2].

However, the intrinsic nonlinear nature of real world phenomena makes the linear hypothesis simply an approximation of real system behavior. Therefore, nonlinear system identification (NSI) is of great interest and one of the key issues in the modeling of signals generated by artificial systems and natural phenomena. The main applicative fields of NSI are automatic control, neurosciences, communications (e.g., echo cancellation, channel equalization and amplifier nonlinearity evaluation) and signal and image processing, to just mention a few [3].

The algorithms classically used for NSI depend on the selection of a suitable model to represent the data. Some valid and widely used models are the Volterra series expansion [4,5,6], Hammerstein model [7,8,9,10,11], Wiener model [12,13,14,15] and the nonlinear auto-regressive moving average model with exogenous inputs (NARMAX) and its derivatives [16,17,18,19,20,21].

In the discrete time domain, one of the most successful approaches for nonlinear system identification is the NARMAX model (and its derivatives NARX [22,23] and NARMA [24]), in which the system is modeled in terms of a nonlinear functional expansion of lagged inputs, outputs and prediction errors. NARMAX models have been shown to be very effective in many real-world applications [17,19,25,26,27,28,29], as they are powerful, efficient and unified representations of a wide variety of nonlinear systems.

The choice of the nonlinear estimator function is an important part of NARMAX models. Various estimators can be used, such as polynomials [30,31], multilayer perceptrons (MLPs) [32,33], artificial neural networks (ANNs) [34,35,36,37] and wavelet networks [38,39].

While traditional algorithms operate on time domain representations of the data, the method used in this paper takes advantage of the spectral representation of the signals through the use of the discrete Karhunen–Loève transform (DKLT), thus simplifying the complexity of the problem, especially for large temporal series. The nonlinear relationship in the spectral domain thus becomes a multivariate static function to be approximated, and an alternative algorithm based on particle-Bernstein polynomials, a variation of the classic Berstein polynomials [40,41] extended to continuous variables, has been applied for this purpose.

To assert the effectiveness of the proposed algorithm, it has been tested on a public dataset acquired from a real system, namely the F-16 aircraft benchmark dataset, which consists of measures of the airplane’s ground vibrations, and compared to other widely used ones.

This article is organized as follows. Section 2 introduces the mathematical theory and algorithms used in the paper. Section 3 describes the dataset used for the experiments. Section 4 explains how the data have been processed prior to the actual SI phase. The experimental set-up and results are shown in Section 5, together with a discussion of the results in Section 6. Finally, some conclusions are drawn in Section 7.

2. Proposed Method

The goal of this article is the identification of nonlinear dynamic systems, given a set of inputs $u\left(t\right)$ applied to the system and the corresponding outputs $y\left(t\right)$ generated by the system itself.

In the most common case of discrete time dynamic systems, the output of the system at a given time depends on the input and output values at a certain number of previous time points in what represents the memory of the system:

$$y\left(t\right)=h\left(y(t-1),\dots ,y(t-p),u(t-1),\dots ,u(t-q)\right),$$

(1)

with $t=1,\dots ,n$.

The proposed algorithm operates in the spectral domain, thus converting the system to a function that does not depend on time, but only on a set of static parameters. To this end, the discrete Karhunen–Loève transform (DKLT) [42] is used, which has the well-known property of separating the time-dependent components from fixed parameters.

Finally, the resulting nonlinear relation is estimated by particle-Bernstein polynomials.

2.1. Spectral Representation of Nonlinear Systems

Starting with the general case of time-dependent inputs $u\left(t\right)$ and outputs $y\left(t\right)$, as used in Equation (1), it is convenient to represent them in matrix form, namely U and Y, respectively, with $U\in {\mathbb{R}}^{m\times N}$, $Y\in {\mathbb{R}}^{n\times N}$ and N being the number of realizations. It is worth noticing that the original data are not always in this suitable form, likely being composed of a few realizations of different lengths. Nonetheless, they can be turned into this form by means of windowing, or splitting the input and output files into equally sized windows whose length must be chosen so it is compatible with the properties of the data.

Then, the DKLT is applied to the resulting matrices. A possible realization of the DKLT is using singular value decomposition (SVD), which factorizes a matrix (U for example) as

$$U=V\Sigma W^T$$

(2)

where $\Sigma \in {\mathbb{R}}^{N\times N}$ is a diagonal matrix containing the singular values of U and V and W contain vectors forming the bases for the U components such that U can be represented as

$$U=VX$$

(3)

where V contains the time-dependent components and X the desired static components (features) of the signals. The correspondence is biunivocal such that X can be computed as

$$X=V^{T}U\in {\mathbb{R}}^{N\times N}$$

(4)

for the properties of SVD.

Similarly, the outputs Y can be represented in their feature space as a new matrix K such that the nonlinear system in Equation (1) can be represented in a new form as

$$k=f\left(x\right),x\in {\mathbb{R}}^m,k\in {\mathbb{R}}^n$$

(5)

with no dependency on time.

With the two transformations being biunivocal, the identification problem can be solved in the feature space, and the computed outputs can then be transformed back to the time domain.

Moreover, it is customary to apply principal component analysis (PCA) to the DKLT to reduce the dimensionality of the problem [43]. By retaining only the components of V associated with the most significant singular values, the problem can be simplified with no significant loss in signal representation. In Equation (2), assuming the singular values in $\Sigma$ are sorted in descending order, this is equivalent to truncating $\Sigma$ to the upper $d\times d$ part and V to the first d rows so that Equation (4) becomes

$$X_d=V_d^{T}U,X_d\in {\mathbb{R}}^{d\times N}$$

(6)

with $d<m$. An analog reduction can be applied to the output matrix K.

In most cases, the estimation of $f(·)$ in Equation (5) is performed through supervised learning, where an experimental set S of corresponding input/output realizations is given:

$$S=\{(x_j,k_j),j=1,\dots ,N\}$$

(7)

Thus, this reduces to a regression problem in Equation (5) through a suitable technique. Usually, a part of S is used for the actual system estimation (training), while a separate subset is used to validate the approximated function (testing).

2.2. Bernstein Polynomials

Polynomials can effectively provide a basis to represent a function to be estimated, as stated by the well-known Weierstrass approximation theorem, according to which every continuous real function $f:[0,1]\to \mathbb{R}$ can be approximated by a polynomial with arbitrary accuracy. (The $[0,1]$ interval can easily be generalized to an arbitrary domain.)

Bernstein polynomials (BPs) are a convenient choice for such a basis [40,41]. Given the space of polynomials of a degree m, the $k^{th}$ BP is defined as

$$b_k^{\left(m\right)}\left(x\right)=\left(\genfrac{}{}{0pt}{}{m}{k}\right)x^k{(1-x)}^{m-k},k=0,1,\dots ,m$$

(8)

where $x\in [0,1]$ and $\left(\genfrac{}{}{0pt}{}{m}{k}\right)$ is the usual binomial coefficient.

BPs are a convenient choice for representing a function to be estimated given an input/output relationship as in Equations (5) and (7), because the associated coefficients do not need complex algorithms to be computed but depend only on the value of $f(·)$ in a series of points. More precisely, say we are given the approximation sequence $B_m\left(x\right)$, expressed as

$$B_m\left(x\right)=\sum _{k=0}^{m}f(k/m)b_k^{\left(m\right)}\left(x\right)$$

(9)

Then, it can be shown that this sequence converges uniformly to $f\left(x\right)$ as $m\to \infty$ (therefore representing a constructive proof of the Weierstrass theorem). The coefficients are simply given by the function evaluated at points $\{k/m,k=0,\dots ,m\}$.

This representation can easily be extended to the case of multivariate functions. If $x=(x_1,\dots ,x_d)$ and $k=(k_1,\dots ,k_d)$, then Equations (8) and (9) can be redefined as

$$B_m(x_1,\dots ,x_d)=\sum _{k_1=0}^m\dots \sum _{k_d=0}^{m}f\left(\frac{k_1}{m},\dots ,\frac{k_d}{m}\right)b_k^{\left(m\right)}\left(x\right)$$

(10)

and

$$b_k^{\left(m\right)}\left(x\right)=\left(\genfrac{}{}{0pt}{}{m}{k_1}\right)\dots \left(\genfrac{}{}{0pt}{}{m}{k_d}\right)x_1^{k_1}{(1-x_1)}^{m-k_1}\dots x_d^{k_d}{(1-x_d)}^{m-k_d}$$

(11)

In this case, computing the coefficients requires evaluating the function at points

$$x=(k_1/m,\dots ,k_d/m),k_1=0,\dots ,m,k_d=0,\dots ,m$$

(12)

which is a grid of points in ${\mathbb{R}}^d$. Provided that such a grid must be sufficiently dense to obtain a good approximation of $f(·)$, this has the drawback that the computational cost increases dramatically with the dimensionality of the input space, thus becoming prohibitive even for moderately complex systems.

2.3. Particle-Bernstein Polynomials

To overcome the computational cost of BPs in the multivariate function case, recently, a new class of functions named particle-Bernstein polynomials (PBPs) has been proposed [40,42,44], which has been shown to achieve good results in regression problems while maintaining a lower computational cost with respect to classic BPs.

Meanwhile, in Equation (8), the k index must be the integer for the presence of the binomial coefficient, and thte PBPs relax this constraint by replacing k with a real parameter $\xi$, where $\xi \in [0,m]$. The PBP functions are then defined as

$$C_{\xi }^{\left(m\right)}\left(x\right)={\alpha }_{\xi }^{\left(m\right)}{x}^{\xi }{(1-x)}^{m-\xi }={\alpha }_{\xi }^{\left(m\right)}{k}_{\xi }^{\left(m\right)}\left(x\right)$$

(13)

The ${\alpha }_{\xi }^{\left(m\right)}$ terms are computed so that

$${\int }_0^1{C}_{\xi }^{\left(m\right)}\left(x\right)dx=1$$

(14)

Aside from them all having the same area, PBP functions have a maximum at $x=\xi /m$ in $[0,1]$, a property similar to traditional BPs. Together with the fact that they can be considered mostly concentrated around their maximum for $m\gg 1$, the value of $f(\xi /m)$ can be approximated as

$$f(\xi /m)\cong {\int }_0^{1}f\left(x\right)C_{\xi }^{\left(m\right)}\left(x\right)dx$$

(15)

If a given set of arbitrary values $\{x^{\left(j\right)},j=1,2,\dots ,N\}$ is given together with the evaluated function $f\left(x^{\left(j\right)}\right)$ at the same points, as is usual in supervised learning, the following approximation for $f(·)$ in an arbitrary point holds:

$$f(\xi /m)\cong \frac{{\sum }_{j=1}^{N}f\left(x^{\left(j\right)}\right)k_{\xi }^{\left(m\right)}\left(x^{\left(j\right)}\right)}{{\sum }_{j=1}^N{k}_{\xi }^{\left(m\right)}\left(x^{\left(j\right)}\right)}$$

(16)

It follows that the main advantage of PBPs is that $f(·)$ can be estimated at an arbitrary point given its values in a set of N randomly chosen points, according to the most convenient criterion, and not on a fixed grid as in traditional BPs.

The advantage is even more evident when extended to multivariate functions. Given $x=(x_1,\dots ,x_d)$ and $\xi =({\xi }_1,\dots ,{\xi }_d)$, the $k_{\xi }^{\left(m\right)}\left(x\right)$ term in Equation (13) can be redefined as

$$k_{\xi }^{\left(m\right)}\left(x\right)=x_1^{{\xi }_1}{(1-x_1)}^{m-{\xi }_1}\dots x_d^{{\xi }_d}{(1-x_d)}^{m-{\xi }_d}$$

(17)

and the estimation of $f(\xi /m)$ is given by the same expression in Equation (16), replacing the scalar x and $\xi$ with their vector equivalents. It can be seen from Equation (17) that the estimation complexity does not increase exponentially with the dimension d of the input space as in Equation (11), thus sensibly reducing the complexity with respect to traditional BPs.

Figure 1 shows the several steps of the proposed algorithm in a concise way.

3. Dataset

The F-16 Ground Vibration Test Benchmark Dataset [45,46] contains experimental data acquired from a full-scale F-16 aircraft on the occasion of the Siemens LMS Ground Vibration Testing Master Class, held in September 2014 at the Saffraanberg military base in Sint-Truiden, Belgium.

During the experiment, two dummy payloads were mounted at the wing tips to simulate the mass and inertia of real devices typically equipped by an F-16 in flight (Figure 2a). The aircraft structure was instrumented with accelerometers, and as the input signal, one shaker was attached underneath the right wing. The dominant source of nonlinearity in the structural dynamics was expected to originate from the mounting interfaces of the two payloads. These interfaces consisted of T-shaped connecting elements on the payload side slid through a rail attached to the wing’s side (Figure 2b).

The measurements were acquired at a sampling frequency of 400 Hz. Two distinct input signals were made available: (1) the voltage measured at the output of the signal generator amplifier, acting as a reference input, and (2) the actual force provided by the shaker and measured by an impedance head at the excitation location.

Three acceleration signals were provided as output quantities. They were measured (1) at the excitation location, (2) on the right wing next to the nonlinear interface of interest and (3) on the payload next to the same interface. Measurements were performed with three different excitation types that yielded the three subsets of data described in the following sections.

3.1. Sine-Sweep Excitation with a Linear, Negative Rate

Sine-sweep excitations with a linear, negative rate of 0.05 Hz/s (sweep down) were applied. The covered input frequency range was 15–2 Hz. Seven different levels of excitation were provided as benchmark data. The lowest level at a 4.8-N input amplitude could be considered a linear dataset. Three higher excitation levels were given to function as estimation data in the nonlinear regimes of vibration, namely dataset numbers 3, 5 and 7 corresponding to 28.8, 67.0 and 95.6 N, respectively. Dataset numbers 2, 4 and 6 at 19.2, 57.6 and 86.0 N, respectively, were to be used for testing the models estimated using datasets 3, 5 and 7.

3.2. Multisine Excitation with a Full Frequency Grid

The data recorded under multisine excitations with a full frequency grid from 2 to 15 Hz were provided. At each force level, nine periods were acquired, considering a single realization of the input signal. The number of points per period was 8192. Note that transients were present in the first period of measurement. Similar to the sine-sweep case, 7 excitation levels were considered, starting from the linear data at 12.4 N RMS (dataset 1). In addition, three nonlinear estimation datasets (numbers 3, 5 and 7 at 36.8, 73.6 and 97.8 N RMS, respectively) were accompanied by their corresponding test sets (numbers 2, 4 and 6 at 24.6, 61.4 and 85.7 N RMS, respectively).

3.3. Multisine Excitation with a Random Frequency Grid

In the third set of tests, sine-sweep excitations were applied with only odd frequencies excited in the range of 1–60 Hz. Moreover, within each group of four successive excited odd lines, one frequency line was randomly rejected to act as a detection line for odd nonlinearities. In this setting, the frequency band from 1 to 60 Hz was excited. Three periods per level were recorded, considering 10 input realizations per level. The number of points per period was 16384. Note that only the last two periods of each realization were in a steady state. The datasets were originally sampled at 200 Hz. They were upsampled to 400 Hz in the frequency domain, processing period per period and assuming the data were periodic and in a steady state.

4. Data Processing

For the system identification and its testing, the data subset with multisine excitation on a full frequency grid was used, with separate experiments for each of the three output sensors. The full data available for the mentioned input excitation were used for a total of 1290 s and at different excitation amplitudes.

To represent the input and output data in matrix form (Section 2.1) starting from a set of time series of different lengths and amplitudes, the original signals were concatenated and then split into a set of homogeneous windows. With w being the number of samples in a window and o being the number of overlapping samples between adjacent windows, the nth data window corresponded to the samples in the range

$$\left[n(w-o),n(w-o)+w-1\right]$$

(18)

with $n\ge 0$.

For the input windows, a length of 8192 was used, corresponding to the period used to generate the dataset (Section 3). To overcome the problem of a limited number of realizations, the splitting of input data wass performed with an overlap between windows. In this way, a moving window was used on the input data, while the matching output was taken as non-overlapping windows smaller than the input ones (Figure 3).

To generate a well-balanced set of data for training and a set for testing, instead of roughly using half of the data for training and testing, as meant in the original data grouping, single inputs and outputs obtained from the different signals through windowing were randomly shuffled and then divided at an 80/20 ratio for training and testing, respectively.

The previous steps generated a pair of input/output matrices for training, named $U_{\mathrm{train}}$ and $Y_{\mathrm{train}}$, and a similar pair for testing, named $U_{\mathrm{test}}$ and $Y_{\mathrm{test}}$. Then, the DKLT was applied as in Section 2.1 by computing a base as in Equations (3) and (4). To avoid data leaks from testing to training, the decomposition was computed on the training matrices only. The same base was then applied to the testing material so as to obtain input/output pairs in the feature space of $X_{\mathrm{train}}$, $K_{\mathrm{train}}$, $X_{\mathrm{test}}$ and $K_{\mathrm{test}}$.

Finally, before using the particle-Bernstein polynomials (Section 2.3), the input data had to be normalized in the range $[0,1]$. The input data $X_{\mathrm{train}}$ and $X_{\mathrm{test}}$ were normalized by single components (columns) according to the following formula:

$$x_{\mathrm{norm}}=a+(b-a)\frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)}$$

(19)

where x is the data component to be normalized, $a=0$ and $b=1$.

A prior clipping of data was performed using their mean value and standard deviation to filter out the outlier data and better distribute the resulting signal in the desired range. Specifically, $\mu$ and $\sigma$ are the mean value and standard deviation of a given component, respectively, and data were limited to the range $\mu \pm 3\sigma$.

5. Experimental Results

All the experiments on the different outputs were performed with particle-Bernstein polynomials of a degree of 40, determined experimentally as a trade-off between the resulting accuracy and numerical stability.

The input and output windows were of sizes of 8192 and 1024, respectively. For the PCA, 16 components for the input matrices and 50 for the output were used because this was experimentally determined to suffice for providing a good representation.

The absolute error of the predicted signals with respect to the true output were computed as the root mean square (RMS) of the difference of the two signals:

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum _{k=1}^N{(y_{\mathrm{predict}}\left[k\right]-y_{\mathrm{true}}\left[k\right])}^2}$$

(20)

The relative error was computed as the ratio between the RMSE and the RMS of the reference signal:

$${\mathrm{RMSE}}_{\mathrm{rel}}=\mathrm{RMSE}/\sqrt{\frac{1}{N}\sum _{k=1}^N{\left(y_{\mathrm{true}}\left[k\right]\right)}^2}$$

(21)

Figure 4 shows the true output and the error with respect to the prediction for the three output sensors, as well as a detailed comparison of the true and predicted outputs for the three output sensors.

To test the effectiveness of our approach, a comparison of the numerical results with the NARX model created with the Matlab “System Identification” tool was first conducted using the same data for training and testing as in the PBP experiment. The parameters used for the NARX model were as follows: 10 points (lags) for the input data, 20 points for the output data, a linear input regressor and a wavelet network as an output function. These parameters were chosen as a trade-off between accuracy and the resources needed by the process (time and memory).

Table 1. Comparison of errors in predicted output for testing data.

	PBP	PBP	NARX
Output	Rel. Error (%)	Abs. Error (g)	Abs. Error (g)
1	7.3	0.07	0.81
2	5.2	0.07	1.30
3	12.2	0.14	1.14

shows the error of the predicted output for the three output sensors compared with the NARX model.

As a more exhaustive comparison, our approach was compared with deep convolutional neural networks (CNNs) as performed in [47] with the same dataset (actually a subset, as explained in the following). In [47], three types of CNNs are used: a temporal convolutional network (TCN), multilayer perceptron (MLP) and long short-term memory (LSTM). As a matter of comparison, in this case, the same data as in [47] were used, namely the multisine realizations with a random frequency grid limited to the test with a 49.0-N RMS amplitude. Moreover, only the training data were used to compute the RMSE of the identification results with no independent testing data. Another difference was the number of components needed by the PCA in our algorithm to represent the input values. In this case, 25 components were used, provided that a bigger value posed stability problems for the computations of the PBPs.

Table 2. Comparison of errors in predicted output for training data. * These results are provided as averages of the three outputs.

	PBP Abs.	NARX Abs.	LSTM Abs.	MLP Abs.	TCN Abs.
Output	Error (g)	Error (g)	Error (g) *	Error (g) *	Error (g) *
1	0.41	0.82	0.74	0.48	0.63
2	0.31	0.79	0.74	0.48	0.63
3	0.28	0.75	0.74	0.48	0.63

shows the error of the predicted data for the three output sensors for the various algorithms, with the addition of the Matlab NARX implementation with the same parameters as in the previous tests.

6. Discussion

In this work, PBPs were proposed as an alternative to the widely used system identification algorithms available in the literature, such as NARMAX and its derivatives and convolutional neural networks based on different structures.

Moreover, unlike the aforementioned algorithms, this approach does not operate in the time domain but rather in the spectral components of the signals through the use of the DKLT.

Some experiments were performed to validate this approach using a publicly available dataset based on ground vibration tests recorded from a real F-16 aircraft.

In the first experiment, we applied a PBP to the data in order to test the effectiveness of our approach, and a comparison of the numerical results with the NARX model created with Matlab’s “System Identification” tool was conducted. Table 1 reports a comparison of the errors in the predicted output for the testing data between the proposed method and the NARX model. It can be seen that the PBP algorithm yielded sensibly better results.

In the second experiment, to give a more exhaustive comparison, our approach was compared with deep convolutional neural networks (CNNs), as performed in [47]. Table 2 reports a comparison of the errors in the predicted output for the training data between the various algorithms in [47] and also the Matlab NARX model. Again, the PBP algorithm yielded the best results in terms of identification error, with reductions in the absolute error ranging from 42% to 63%.

We already successfully applied this method to synthetic data, as reported in [40], and to identify two real-world nonlinear systems in the fields of speech signals and nonlinear audio amplifiers using simulated data, as shown in [42]. This is a first attempt to apply this method to real-world data, and the experimental results show that it achieved better performance, probably because the other approaches suffered from the large amount of data and their heterogeneity. Therefore, it could potentially be applied to other real datasets presenting the same complexity.

7. Conclusions

In this article, PBPs were proposed as an alternative to the widely used system identification algorithms available in the literature, such as NARMAX and its derivatives and convolutional neural networks based on different structures. The main difference with respect to the traditional algorithms is that PBPs operate on a simplified system representation in the spectral domain.

Experiments with a specific dataset showed that this approach yields better results when compared with other algorithms, proving to be effective on datasets with large amounts of data and heterogeneous compositions. In particular, the absolute error that was obtained was 63% smaller with respect to NARX and from 42% to 62% smaller with respect to various ANN-based approaches.