Comparison of Machine Learning Algorithms for Structure State Prediction in Operational Load Monitoring

Abstract

The aim of operational load monitoring is to make predictions about the remaining usability time of structures, which is extremely useful in aerospace industry where in-service life of aircraft structural components can be maximized, taking into account safety. In order to make such predictions, strain sensors are mounted to the structure, from which data are acquired during operational time. This allows to determine how many load cycles has the structure withstood so far. Continuous monitoring of the strain distribution of the whole structure can be complicated due to vicissitude nature of the loads. Sensors should be mounted in places where stress and strain accumulations occur, and due to experiencing variable loads, the number of required sensors may be high. In this work, different machine learning and artificial intelligence algorithms are implemented to predict the current safety factor of the structure in its most stressed point, based on relatively low number of strain measurements. Adaptive neuro-fuzzy inference systems (ANFIS), support-vector machines (SVM) and Gaussian processes for machine learning (GPML) are trained with simulation data, and their effectiveness is measured using data obtained from experiments. The proposed methods are compared to the earlier work where artificial neural networks (ANN) were proven to be efficiently used for reduction of the number of sensors in operational load monitoring processes. A numerical comparison of accuracy and computational time (taking into account possible real-time applications) between all considered methods is provided.

1. Introduction

Operational load monitoring (OLM) is a process that consists in investigating characteristics of a structure under mechanical loads in its operating environment, be means of acquiring and saving data from sensors mounted to the structure. The gathered data are then processed to estimate the remaining in-service life of the structure, taking into account the number and characteristics of load cycles that the structure has withstood so far [1,2,3]. OLM processes are mainly utilized in aerospace industry for monitoring aircraft structures in order to maximize their in-service life while ensuring flight safety [4,5]. Such approach reduces the costs of inspections and increases the aircraft availability [6].

A related process to operational load monitoring, is Structural Health Monitoring (SHM), where the mechanical system is monitored, often in real time, using sensor data, in order to detect potential reduction of system performance, or even damage [7,8,9]. In aerospace industry, often the both processes are performed simultaneously (Structural Health and Usage Monitoring Systems, HUMS [1,10]), where the strains and stresses at given points are measured, and a real-time system combines the data with flight parameters to determine the current state of the structure; the data are then utilized to compute fatigue factors of some components [11]. Sometimes, OLM is considered as a case of SHM [2].

Typical sensors utilized for OLM are embedded (intrinsic optical fiber) and surface mounted (strain gauges) strain sensors [12,13]. These sensors should be placed in critical points (places of expected stress and strain accumulations). Due to vicissitude nature of the loads, and therefore variable distributions of strains and stresses during operations, the number of required sensors may be high. As the accuracy of OLM depends on the number of sensors, and the number is limited, monitoring of flying parameters (like speed, acceleration and mass) is often performed to estimate the state of the structure in other locations [14]. In the author’s earlier work [15] it has been proven that by implementing artificial neural networks (ANNs), the temporary state of the whole structure could be quite accurately estimated using only a relatively small number of sensors. The neural network was trained with data obtained from finite element simulations of the structure under many different load cases, and then tested with data acquired from strain gauges in series of experiments.

The following work is the analysis of implementation possibilities of other machine learning and artificial intelligence algorithms to OLM, for estimation of the state of the monitored structure based on a reduced number of strain sensor reads. Three different techniques are considered: adaptive neuro-fuzzy inference systems (ANFIS), support-vector machines (SVM) and Gaussian processes for machine learning (GPML). Each of these techniques was used to build several models with different parameters, that were trained by simulation data. All of the models were tested with experimental data, the accuracy and computational time was measured (taking into account possible real-time OLM applications). Detailed comparison between the obtained results is provided.

As there is a potential of real-time applications of machine learning or artificial intelligence methods implemented to OLM processes, the possibilities to deploy such models to real-time systems or embedded systems are considered. Regarding ANNs, the most popular approach is to implement already trained networks (on a capable platform, like PC) to the embedded device. As ANNs are algorithms of relatively low complexity, a whole variety of such applications can be found in the literature, regarding classification of signals for arrythmia detection [16], evaluation of fiber–metal laminates [17], Secchi depth calculation [18], power factor correction [19], or hybrid simulation [20]. Implementing SVM algorithms to real-time systems or embedded systems is also a popular approach. Zhang et al. [21] implemented SVM to detect ventricular fibrillation in real time, Arsalane et al. [22] deployed SVM to an embedded system for meat freshness evaluation based on machine vision. Dey et al. [23] studied performance of SVM classification algorithms running on embedded processors and suggested parallel computations. There are also examples of SVMs implemented to FPGA-based systems, like speech recognition system [24] or melanoma detection [25]. Regarding ANFIS, there are many examples of real-time applications of these algorithms in automatic control area, like of hydraulic systems [26], active suspension system for passenger comfort improvement [27], robot arms [28], or mobile robots navigation for target seeking and obstacle avoidance [29]. Karakuzu et al. [30] and Maldonado et al. [31] deployed ANFIS algorithms to FPGAs. Danzomo et al. [32] implemented an ANFIS-based algorithm to Arduino microcontroller, Sakib et al. [33] to Raspberry Pi for flood monitoring system, and Mishra et al. [34] to a real-time system with DSP 2812. Although real-time applications of GPML models are known [35,36], examples of deploying these algorithms to embedded devices were not found, which is probably caused by the computational complexity of these models (they require all the training data to make predictions).

2. Materials and Methods

A typical structure for aerospace applications is considered—hat-stiffened composite panel made of carbon/epoxy woven laminate, presented in Figure 1a. In the author’s earlier work [15], a detailed description of the geometry, experimental measurements of the mechanical properties, as well as building and validating a finite element model, can be found.

Six strain gauges (SG1–SG6) are mounted to the panel, in positions presented in Figure 1b,c. Strain measurements are performed in longitudinal directions. Finite element model of the panel is presented in Figure 1d. It consists of 26,357 surface elements and 79,217 nodes. At the edge, BC1 displacements in directions X and Y are fixed, at the opposite edge (marked as BC2) displacements in direction Y are fixed. The panel is loaded vertically at any point on the line BC3 (the load is applied to a 5 mm circle surface, displacements along Z axis are fixed within the circle).

The experimental tests were performed using universal testing machine MTS Insight 10 with 500 N load cell (MTS Systems Corporation, Eden Prairie, MS, USA) (Figure 2). The panel was mounted in a specially designed stiff frame (to mimic the boundary conditions BC1–BC3) and loaded in points LC1–LC3, of location presented in Figure 1b. Data from strain gauges and load cell were acquired using HBM AB22A hardware system (Hottinger Baldwin Messtechnik, Darmstadt, Germany) and Catman Easy software. The force signal from the load cell was received at the analog input of the acquisition system. Data from strain gauges and load cell were acquired with 20 Hz sampling frequency. No preprocessing was applied to the measured signals.

For OLM purposes, maximum inverse safety factor of the whole structure was observed during loading with different force values, and at different points. Safety factor is usually defined as the margin to failure, the inverse safety factor ISF_Ci is denoted as the quotient of the applied load Q to the failure load Q_Ci, taking into account a given failure criterion Ci:

$$IS{F}_{\mathrm{C}i}=\frac{Q}{Q_{\mathrm{C}i}}.$$

(1)

when there are n failure criteria ($i=1,2,\dots ,n$), the global inverse safety factor ISF is calculated as the maximum value from each criteria:

$$ISF=\max (IS{F}_{\mathrm{C}i})$$

(2)

In the presented research, three failure criteria were considered: maximum stress criterion C₁, maximum strain criterion C₂ and Tsai-Wu [37] criterion C₃:

$$IS{F}_{\mathrm{C}1}=\max \left(\frac{{\sigma }_j}{{\sigma }_{\lim ,j}}\right),$$

(3)

$$IS{F}_{\mathrm{C}2}=\max \left(\frac{{\epsilon }_j}{{\epsilon }_{\lim ,j}}\right),$$

(4)

$$IS{F}_{\mathrm{C}3}=F_j {\sigma }_j + F_{jk} {\sigma }_j{\sigma }_k,$$

(5)

where j,k = 1,2,…,6, σ_j and ε_j are stresses and strains in Voight notation, respectively, F_i are parameters calculated from stress limits σ_lim,j, and F_jk are coupling coefficients. In criteria C₁ and C₂, different stress σ_lim,j and strain ε_lim,j limits were taken for tension and compression.

From the finite element (FE) model, 1241 data samples of relationship between strains in positions SG1-SG6 and the value of ISF (Supplementary S7 to [15]) were generated. When generating the data, forces of different values were applied to 31 spots at line BC3 (Figure 1d). Figure 3 illustrates the nonlinear behavior of the ISF, where it is plotted against location of the applied force (all generated data samples with the same example force value of 40 N).

The authors of [15] present a neural network, trained with 70% of the abovementioned data generated from FE simulations, that predicts the maximal value of ISF for the whole structure based on the data from strain gauges (Figure 4). Efficiency of the networks, when dealing with strain values acquired in series of experiments, was measured.

The following work presents comparison between results of the ANN presented in [15] and of implementation of three other machine learning methods (similar prediction models, of inputs and output as presented in Figure 4): ANFIS, SVM, and GPR. The comparison takes into account the accuracy of the results and the computational effort for making predictions. In the presented research, white gaussian noise was added to the input data (SG1–SG6) generated from FE simulations (Supplementary Materials S1) in order to build more robust models to noisy data.

Before building the models, a grid search approach was implemented to find the best values of hyperparameters of the models, according to the algorithm presented in Figure 5. For each set of hyperparameters values, the model was trained 10 times, each time with randomly chosen 70% of the reference data. Each time the model was tested with remaining 30% of the reference data and with experimental data. Mean squared errors (MSEs) were computed. Each set of hyperparameters values was evaluated by the average MSE from 10 iterations.

After choosing the best combination of hyperparameters values, the models were build using algorithm presented in Figure 6. One thousand attempts for training the regression model were performed, from which the one that gave the least mean squared error (MSE) of ISF, when introduced to input data acquired from series of experiments, compared to reference output obtained from FE simulations (knowing the applied force), was chosen.

The algorithms presented in Figure 5 and Figure 6 were applied to ANFIS, SVM and GPR models.

ANFIS is a fuzzy inference system implemented in the framework of adaptive networks [38]. It has the possibility to integrate the benefits of both fuzzy inference systems and adaptive neural networks. In fuzzy systems the output is mapped from the input, based on fuzzy logic (involving membership functions, fuzzy logic operators and if-then rules). Artificial neural networks, on the other hand, are collections of artificial neurons, a mathematical model of biological neurons. The artificial neurons (nodes) are connected with directional links to transmit signals. If some of the nodes’ outputs depend on a set of parameters (to minimize the error by changing the parameters during learning), than it is an adaptive neural network [39,40]. The ANFIS structure consists of five layers, as presented in Figure 7 of network with two inputs x₁ and x₂.

Layer 1 is the fuzzification layer, where membership degrees of the inputs to rules are computed using the membership functions (MFs) μ(x_i). In the second layer (the rule layer), the firing strengths of each rule is determined as the product of inputs:

$$w_i={\mu }_{\mathrm{A}i}·{\mu }_{\mathrm{B}i},$$

(6)

for i = 1, 2. In the third layer, the input values are normalized as follows:

$$\overline{w_i}=\frac{w_i}{w_1+w_2},$$

(7)

for i = 1, 2. The fourth layer is known as the defuzzification layer with adaptive nodes that perform the following function:

$$y_i=\overline{w_i}\left(p_i{x}_1+q_i{x}_2+r_i\right),$$

(8)

where p_i, q_i and r_i are consequent parameters for rule i = 1, 2. The last, fifth layer is the output layer that calculates the sum of its inputs.

ANFIS can learn from training data the same way as ANNs; however, the advantage of ANFIS over ANN is that the hidden layer in ANFIS is already determined by the fuzzy interference system, and its size does not have to be found. ANFIS techniques have been implemented for prediction tasks in many different areas of engineering applications, like biomechanics [41], geotechnics [42], production technology [43], failure analysis [44] or structural mechanics [45].

When finding the most suited ANFIS for the given problem, algorithms presented in Figure 5 and Figure 6 were implemented to MATLAB (utilizing Fuzzy Logic Toolbox [46]). The following membership functions μ(x_i) types were considered: generalized bell-shaped (9), gaussian (10), gaussian combination (where left and right curves are of type (10)), triangular (11), trapezoidal (12), difference between two sigmoidal functions (where a single function is given as (13)), product of two sigmoidal functions (where a single function is given by (13)), Pi-shaped (14):

$$\mu \left(x_i,a,b,c\right)=\frac{1}{1+{\left|\frac{x_i-c}{a}\right|}^{2b}},$$

(9)

$$\mu \left(x_i,a,b\right)=\exp \left(\frac{-{\left(x_i-b\right)}^2}{2a^2}\right),$$

(10)

$$\mu \left(x_i,a,b,c\right)=\max \left(\min \left(\frac{x_i-a}{b-a}, \frac{c-x_i}{c-b}\right),0\right),$$

(11)

$$\mu \left(x_i,a,b,c,d\right)=\max \left(\min \left(\frac{x_i-a}{b-a},1, \frac{d-x_i}{d-c}\right),0\right),$$

(12)

$$\mu \left(x_i,a,b\right)=\frac{1}{1+{\mathrm{e}}^{-a\left(x_i-c\right)}},$$

(13)

$$\mu \left(x_i,a,b,c,d\right)=\{\begin{array}{cc}0, & x_i\le a\\ 2{\left(\frac{x_i-a}{b-a}\right)}^2,& a\le x_i\le \frac{a+b}{2}\\ 1-2{\left(\frac{x_i-b}{b-a}\right)}^2,& \frac{a+b}{2}\le x_i\le b\\ 1,& b\le x_i\le c\\ 1-2{\left(\frac{x_i-c}{d-c}\right)}^2,& c\le x_i\le \frac{c+d}{2}\\ 2{\left(\frac{x_i-d}{d-c}\right)}^2,& \frac{c+d}{2}\le x_i\le d\\ 0,& x_i\ge d\end{array},$$

(14)

where a, b, c, d are parameters to fit. The type of membership functions was the hyperparameter to be optimized with algorithm from Figure 5.

Among machine learning tools, support vector machines (SVMs) are supervised learning algorithms used in classification and regression problems. SVMs are based on statistical learning theory proposed by Vapnik [47] where training data are represented as points in space divided by a gap. The points are mapped to maximize the gap size. When data for classification is introduced, it is mapped in the same space and classified based on location of each point. In order to use SVMs for nonlinear classification, kernel trick is implemented, where inputs are mapped to higher-dimensional space [48]. Points in the new space are defined by a kernel function k(x, y). A hyperplane is found, for which the distance to nearest data points (defined as margin) on both sides is as large as possible. The hyperplane is defined as:

$$wx-b=0,$$

(15)

where w is the normal vector to the hyperplane and b is constant. In SVMs used for regression, the model depends only on a subset of the training data limited by parameter ε (which defines how much error is acceptable in the model). When training a regression SVM [49], the following minimization is performed:

$$\frac{1}{2}{\Vert w\Vert }^2\to \min$$

(16)

where

$$\left|y_i-\langle w,x_i\rangle -b\right|\le \epsilon .$$

(17)

The weight vector w can be presented as the linear combination of the training data:

$$w={{\displaystyle \sum }}_{i=1}^N\left({\alpha }_i-{\alpha }_i{}^{*}\right)x_i,$$

(18)

α_i and α_i^* being the coefficients associated with training points. The regression function takes the form:

$$f\left(x\right)={{\displaystyle \sum }}_{i=1}^N\left({\alpha }_i-{\alpha }_i{}^{*}\right)k\left(x_i,x\right)+b.$$

(19)

It has been proven that SVMs are effective regression tools for nonlinear data [50,51,52,53]; therefore, it is motivating to measure the effectiveness of SVMs used for structure state predictions in OLM.

The computations were performed using MATLAB and Statistics and Machine Learning Toolbox [54]. Three different kernel functions were considered: gaussian

$$k\left(x_i,x\right)=\exp \left(-{\Vert x_i-x\Vert }^2\right),$$

(20)

linear

$$k\left(x_i,x\right)=x_i{}^{T}x,$$

(21)

and polynomial

$$k\left(x_i,x\right)={\left(1+x_i{}^{T}x\right)}^q.$$

(22)

The optimized hyperparameters were ε (half of the width of the epsilon-insensitive band) for kernels (20)–(22), kernel scale for (20), and polynomial order q for kernel function (22).

Constraint C for coefficients α_i and α_i^*

$$\forall i:0\le {\alpha }_i,{\alpha }_i{}^{*}\le C$$

(23)

was calculated as a quotient of the interquartile range of the target vector y:

$$C=\frac{\mathrm{iqr}\left(y\right)}{1.349}.$$

(24)

The utilized optimization routine was Sequential Minimal Optimization.

The third considered technique is Gaussian Process Regression (GPR), a GPML method [55]. The process is based on Bayesian framework where predictive output is provided based on prior conditional probability. The key assumption is that the probability distribution for any input vector x ∈ R (and target values y ∈ R) over the target function y = f(x) follows the Gaussian distribution:

$$f\left(x\right)\sim \mathrm{N}\left(\mu \left(x\right),K\left(x,x_i\right)\right)$$

(25)

where μ(x) is the associated mean function and K(x, x_i) is the associated covariance (kernel) function [56,57]. The model parameters are chosen in an optimization process. Output value of an unseen point is predicted from the similarity between points K [58]. The main disadvantage of GPR is that it uses the whole training data for predictions; however, it was proven to be an efficient tool for making predictions [57,58,59], also when dealing with nonlinearities and uncertainties between inputs and targets [60,61,62]. Moreover, it was reported that GPR processing often requires less input data than ANNs [63,64,65].

Four GPR models were found using the algorithms presented in Figure 5 and Figure 6. Different covariance functions were introduced to each of the model:

$$K\left(x,x_i,{\sigma }_f,{\sigma }_l\right)={\sigma }_f{}^2\exp \left(-\frac{1}{2}\frac{{\left(x-x_i\right)}^{\mathrm{T}}\left(x-x_i\right)}{{\sigma }_l{}^2}\right),$$

(26)

$$K\left(x,x_i,{\sigma }_f,{\sigma }_l\right)={\sigma }_f{}^2\exp \left(-\frac{\sqrt{{\left(x-x_i\right)}^T\left(x-x_i\right)}}{{\sigma }_l}\right),$$

(27)

$$K\left(x,x_i,{\sigma }_f,{\sigma }_l\right)={\sigma }_f{}^2\left(1+\frac{\sqrt{5{\left(x-x_i\right)}^T\left(x-x_i\right)}}{{\sigma }_l}+\frac{5{\left(x-x_i\right)}^T\left(x-x_i\right)}{3{\sigma }_l{}^2}\right)exp\left(\frac{-\sqrt{5{\left(x-x_i\right)}^T\left(x-x_i\right)}}{{\sigma }_l}\right),$$

(28)

$$K\left(x,x_i,{\sigma }_f,{\sigma }_l,\alpha \right)={\sigma }_f{}^2{\left(1+\frac{{\left(x-x_i\right)}^T\left(x-x_i\right)}{2\alpha {\sigma }_l{}^2}\right)}^{-\alpha },$$

(29)

where σ_f, σ_l and α are the parameters of the functions [66]. Equation (26) represents the squared exponential covariance function (implemented in model 1), (27) the exponential function (model 2), (28) the Matern 5/2 function (model 3), and (29) the rational quadratic function (model 4). The optimized hyperparameter for each model was the initial value for the noise standard deviation σ₀.

3. Results

All considered regression models are listed in

Table 1. Considered prediction models.

Model Designation	Description
ANF1	ANFIS with two MFs per input
ANF2	ANFIS with three MFs per input
SVM1	SVM with gaussian kernel function
SVM2	SVM with linear kernel function
SVM3	SVM with polynomial kernel function
GPR1	GPR with squared exponential covariance function
GPR2	GPR with exponential covariance function
GPR3	GPR with Matern 5/2 covariance function
GPR4	GPR with rational quadratic covariance function

, each of these models were given a short designation. Grid-search optimization was performed for all the models using algorithm presented in Figure 5. When the best hyperparameter values were revealed, the models were built using algorithm from Figure 6. Finally, all the obtained models were tested using both numerical and experimental input data, their prediction accuracy and time of computations were measured in order to provide detailed comparison.

3.1. ANFIS Models

Two ANFIS models were considered: with two membership functions per each input (64 rules) and with three membership functions per each input (729 rules). Algorithm presented in Figure 5 was applied, do determine the best type of membership function. In the result of the grid search, for both ANF1 and ANF2 models, the most accurate regression models are with triangular membership functions.

Figure 8 presents the efficiency of the models trained according to algorithm presented in Figure 6. Results of ISF when introduced to input data from numerical simulations, as well as from series of experiments, were compared to reference data (obtained from FE simulations, for experimental inputs based on measured values of applied loads).

Table 2. Comparison of accuracy and computational time for obtained ANFIS models.

Model	Computational Time [ms]	RMSE (Numerical Input)	RMSE (Experimental Input)
ANF1	144.0	0.0048	0.0093
ANF2	1539.3	0.0056	0.0084

presents the comparison of accuracy and required computational effort for the obtained models. The accuracy is measured as the root mean square error (RMSE) between the model output and the reference ISF value obtained from numerical simulations. The required computational effort is measured as the total time of computations of every point in the experimental input dataset (1308 points). The computational time was measured as the average time from 100 runs in MATLAB software, on a PC (AMD FX-6350 processor, 16 GB DDR3 RAM).

RMSE obtained from testing with data acquired during experiments are considered to be much more relevant than RSME concerning numerically generated input data. As presented in Table 2, ANF2 model requires over 10 times more computational effort than ANF1 model, while returning results of about 12% smaller RMSE. However, as one can see in Figure 8, the results obtained from ANF1 and ANF2 models do not differ substantially (both RSME values are relatively small) therefore, ANF1 model will be taken for further considerations. Possible real-time applications of ANF2 model raises doubts for its high computational complexity with only a small improvement of accuracy, compared to ANF1.

3.2. SVM Models

Three SVM models were considered (Table 1). The following hyperparameters were optimized using the grid-search approach: kernel scale s and half of the width of the epsilon-insensitive band ε for SVM1 (gaussian) model, only ε for SVM2 (linear) model, polynomial order q and ε for SVM3 (polynomial) model. The following ranges of the hyperparameters were considered:

$$s\in \left[{10}^{-3},{10}^3\right],$$

(30)

$$\epsilon \in \left[{10}^{-3},{10}^2\right]·\frac{\mathrm{iqr}\left(y\right)}{1.349},$$

(31)

$$q\in \left\{2,3,4\right\}.$$

(32)

For s and ε 50 logarithmically equally spaced points in the ranges (30) and (31) were created. For SVM1 model, the following values were obtained: s = 1 and ε = 0.0178·iqr(y)/1.349. For both SVM2 and SVM3 models, the optimal ε value was 0.3162, and the optimal polynomial order for SVM3 model was 2.

The three SVM models with optimal values of hyperparameters were obtained with algorithm presented in Figure 6 (each one is the best from 1000 trials). The models were tested with numerical and experimental input data similarly as ANFIS models, the output plots are presented in Figure 9.

The accuracy and required computational effort were measured the same way as with ANFIS models. The comparison is presented in

Table 3. Comparison of accuracy and computational time for obtained SVM models.

Model	Computational Time [ms]	RMSE (Numerical Input)	RMSE (Experimental Input)
SVM1	3.9	0.0022	0.0023
SVM2	2.3	0.0198	0.0187
SVM3	2.0	0.0147	0.0165

, and graphically (simplifying the accuracy to experimental input data) in Figure 10. Evident conclusion from Figure 10 is that model SVM2 should be ignored in further considerations, as SVM3 performs better in both considered criteria. As one can see in Figure 9, the linear model returned quite noisy results when dealing with experimentally measured inputs, although it was trained on data with white gauss noise added. The gaussian model proved to be most immune to noisy experimental input data. SVM1 will be taken for further considerations, as it is characterized by RMSE nearly seven times smaller than SVM3 and while the computational time of both models is of the same order of magnitude.

3.3. GPR Models

For four GPR models (according to Table 1) with different covariance functions, grid search of one hyperparameter was performed—initial value for the noise standard deviation σ₀ in the range [10⁻⁴, 1], where 50 logarithmically equally spaced points were created. The following optimal values were revealed: 0.0681, 0.1468, 0.0015, 0.4642 for models GPR1–GPR4, respectively.

Taking into account the found best hyperparameter values, final models GPR1–GPR4 were created with algorithm presented in Figure 6. Figure 11 contains the IRF plots of the obtained models, obtained for numerical and experimental input data, as previously.

Table 4. Comparison of accuracy and computational time for obtained GPR models.

Model	Computational Time [ms]	RMSE (Numerical Input)	RMSE (Experimental Input)
GPR1	16.8	0.0024	0.0025
GPR2	19.0	0.0011	0.0036
GPR3	24.2	0.0017	0.0032
GPR4	29.9	0.0018	0.0034

compares the accuracy and required computational effort for GPR models, measured the same way as ANFIS and SVM models. The same comparison was graphically illustrated in Figure 12, for better visibility.

When dealing with numerical input data, the results of GPR1, GPR3 and GPR4 models are of similar accuracy, GPR2 is a little more accurate. However, when introduced to data acquired from experiments, GPR1 model turned out to be most efficient, returning results 20–30% more accurate than other models and requiring the least computational time from all GPR models. Although the accuracy of all the obtained models is satisfying, it can be clearly seen in Figure 12 that GPR1 model outruns GPR2–GPR4 models in both considered criteria; therefore, GPR2–GPR4 (of very similar accuracy when introduced to experimentally acquired input data) will be excluded from further considerations.

3.4. Comparison of ANFIS, SVM, GPL and ANN Models

The chosen models of ANFIS, SVM and GPR methods were compared to each other and to the artificial neural network from the previous research [15]. There were 100,000 trials for training the ANN, each time with a random number of neurons in the hidden layer, between 3 and 20. The ANN with be smallest error when dealing with experimental input, was chosen as the final one. The final ANN had nine neurons in the hidden layer.

Figure 13 presents the models’ responses to time series of strain data SG1–SG6, generated numerically and measured experimentally, for load cases LC1–LC3. Calculated root mean square errors and measured computational times are compared in

Table 5. Comparison of accuracy and computational time for chosen models.

Model	Computational Time [ms]	RMSE (Numerical Input)	RMSE (Experimental Input)
ANF1	144.0	0.0048	0.0093
SVM1	2.4	0.0022	0.0023
GPR1	20.7	0.0024	0.0025
ANN	0.7	0.0020	0.0074

. Graphical comparison of accuracy (with experimental input) and computational time of the models is presented in Figure 14. The ANN turned out to be computationally the most efficient and SVM1 the most accurate (about three times more accurate than ANN, but requiring about 3.5 times more computational time). The worst from all compared models was the ANF1, as its accuracy and computational time were far worse than of all the rest. The GPR1 model was of very similar accuracy than SVM1 (8% worse) and about nine times slower.

Statistical significance of the obtained results from different models was also measured with series of t-tests.

Table 6. Resultant p-values of comparing different models with t-test (numerical input).

Model	ANF1	SVM1	GPR1	ANN
ANF1	X	0.0453	0.0365	0.0062
SVM1	0.0453	X	0.3818	0.0148
GPR1	0.0365	0.3818	X	0.0696
ANN	0.0062	0.0148	0.0696	X

and

Table 7. Resultant p-values of comparing different models with t-test (experimental input).

Model	ANF1	SVM1	GPR1	ANN
ANF1	X	0.3182	0.6637	0.0053
SVM1	0.3182	X	0.0217	0.0001
GPR1	0.6637	0.0217	X	0.0000
ANN	0.0053	0.0001	0.0000	X

list the results of the t-tests (p-values) when comparing the outputs of different models introduced to numerically computed and experimentally measured input data, respectively. The null hypothesis was that the performance metrics of two considered models are equal. If the significance level was assumed as 5%, when considering numerical input data, the null hypothesis was failed to be rejected for comparing GPR1 and SVM1 models, and when considering experimental input data, the null hypothesis was failed to be rejected for pairs ANF1-SVM1 and ANF1-GPR1.

The models were also tested with measured input data during real-time experiment described in [15] (Supplementary Materials S8 to [15]) where the force was applied at point LC2 (the time plot of the force and the obtained results are presented in Figure 15). When the training data were generated, the forces did not exceed the value of 40 N in the simulations. What is curious about these results is that the applied force exceeded that value (the last four peaks), and only the ANN was able to make proper ISF predictions there, the rest of the models stopped at earlier values. The ANN on the other hand did not predict well the low values of ISF (below 0.1) however, such values are often insignificant when the strength of materials is taken into account.

Finally, a statistical research was performed, regarding repeatability of the accuracy of the models during training trials. RSME of the output, when dealing with noisy experimentally measured input data, was measured in the similar way as listed in Table 2, Table 3, Table 4 and Table 5. For ANF, GPR, and SVM models, 1000 new training trials were started with identical (best) hyperparameters values. For ANN model, 1000 new training trials were started using the same training parameters as described in [15] (nine hidden neurons, and the same training routine). For each trial, accuracy of the model was measured by computing RMSE value, and a t-test of pairwise comparison of the trained model and the considered model (found with algorithm from Figure 6) was performed. Histograms of the trained models RSME values are presented in Figure 16, while

Table 8. Comparison of statistical RMSE data for training 1000 models.

Model	Minimum Value	Maximum Value	Average Value	Standard Deviation
ANF1	0.0090	0.0121	0.0107	4.98 × 10−4
SVM1	0.0023	0.0038	0.0028	2.17 × 10−4
GPR1	0.0028	0.0066	0.0040	5.06 × 10−4
ANN	0.0050	4.9595	0.0893	0.2452

contains the statistical summary of the obtained RSME values, and Figure 17 presents the amount of training iterations where the p-value of the t-test was greater than 0.05 (which indicates that the differences in results could not be significant). The best concentration (least standard deviation) of the RSME, as well as the lowest average, was observed for SVM1 model. The average RSME value of the GPR1 model was slightly worse, and the average RSMEs of ANN and ANF1 models were one order of magnitude greater than of SVM1 and GPR1 models. The RSME standard deviation of ANF1, SVM1 and GPR1 were of the same order of magnitude, while the standard deviation of the RSME of ANN was three orders of magnitude greater. Therefore, the lowest repeatability of the accuracy was observed for the ANN model, where the histogram had to be presented using logarithmic RSME scale, as the values were scattered within different orders of magnitude. The results of the t-tests (Figure 17) confirm that deduction, where for ANNs there was the greatest rejection rate of the null hypothesis (that the differences of performance of the trained models with the considered model are not significant).

4. Discussion

Three types of machine learning techniques (ANFIS, SVM, and GPR) were applied to predict nonlinear ISF of an aerostructure excited with different loads. The results were compared with each other and with ANN model obtained in previous work, for its accuracy and required computational effort, taking into account potential real-time applications in the area of operational load monitoring.

In order to find the best suited ANFIS, models with different membership function types were considered, among which, by grid search, triangular function was revealed as the type that results with the most accurate model, both for models with two and three MFs per input. The required computational effort for using the trained model for predictions grows exponentially with increasing the number of membership functions per input. Therefore, it was decided to stop at two MFs per input, as the model accuracy was considered to be sufficient, and did not improve much when introducing three MFs per input.

For SVM models, three different kernel functions were considered. For all three models grid search of hyperparameters values was performed. From the considered models, the one with gaussian kernel function (SVM1) was of accuracy by one order of magnitude higher than two others. By observing the ISF results of SVMs tests (Figure 9), it seems that SVM2 was not immune to noisy experimental input data although all the models were trained on numerical data with white gaussian noise added. This is considered to be of great importance, as in OLM one always deals with measured noisy data. SVM1 was chosen for further considerations although its computational time was higher than SVM2 and SVM3; however, the two latter did not perform well even when tested with numerical data, similar to that used in training, which is not acceptable.

Four GPR models, with different covariance functions, were considered. Their computational time and accuracy results were quite close to each other. When looking at the plots of ISF results (Figure 11) it would be hard to reveal the model with the best performance, without computing error values. Although all GPR models could be considered as acceptable, after analysis of numerical data from Table 4, GPR1 disqualified other three models, as it fitted both considered criteria (computational time and RMSE for experimental input) best.

From the abovementioned models, one of each type, considered as best, was chosen for final comparison. The accuracy of all the chosen models is acceptable and of the same order of magnitude. However, the time of computations for each model type is of different order of magnitude, ANN being the fastest, and ANFIS being the slowest. When introduced to experimentally acquired input that exceeded the training data range, only the ANN managed to return proper results outside the limits. This means that ANNs may have the strongest generalization abilities and could manage better when unusual or unpredictable event occurs.

When analyzing the statistical data of the accuracy of the trained models (study of repeatability), the ANNs fared the worst of the four considered methods. The standard deviation is of three orders of magnitude higher than for all other model types. The histogram for ANNs could not be presented with others, at it spreads on few orders of magnitude, and logarithmic scale must have been introduced for visibility. This means that training an ANN that meets the requirements for OLM is a far harder task that could require significantly more attempts than other presented methods. The other issue with ANNs it that there is no universal rule on how many neurons should be put in the hidden layer. If the number is too low, the accuracy of the model will not meet the requirements, and if the number is too high, the model could become overfitted and also of low accuracy.

Among the three other methods, SVM proved to have the lowest average value and standard deviation of RMSE from all. Moreover, its computational time is lower than ANF and GPR models. Therefore, it is recommended for OLM systems without real-time constrains, or in OLM systems where this method meets the time constraints.

For real-time systems where the computational power is not high (e.g., where the computations are performed in an embedded system with ARM architecture) or where the required time step is relatively small, one should consider two method—ANN and SVM. ANN is the fastest due to simple floating point operations in each layer; however, finding the accurate model could be hard or even impossible. In this case, SVM could be easier to implement, as it was proven to have good repeatability (by histogram and standard deviation value).