Comparison of Machine Learning Methods for Image Reconstruction Using the LSTM Classifier in Industrial Electrical Tomography

Abstract

Electrical tomography is a non-invasive method of monitoring the interior of objects, which is used in various industries. In particular, it is possible to monitor industrial processes inside reactors and tanks using tomography. Tomography enables real-time observation of crystals or gas bubbles growing in a liquid. However, obtaining high-resolution tomographic images is problematic because it involves solving the so-called ill-posed inverse problem. Noisy input data cause problems, too. Therefore, the use of appropriate hardware solutions to eliminate this phenomenon is necessary. An important cause of obtaining accurate tomographic images may also be the incorrect selection of algorithmic methods used to convert the measurements into the output images. In a dynamically changing environment of a tank reactor, selecting the optimal algorithmic method used to create a tomographic image becomes an optimization problem. This article presents the machine learning method’s original concept of intelligent selection depending on the reconstructed case. The long short-term memory network was used to classify the methods to choose one of the five homogenous methods—elastic net, linear regression with the least-squares learner, linear regression with support vector machine learner, support vector machine model, or artificial neural networks. In the presented research, tomographic images of selected measurement cases, reconstructed using five methods, were compared. Then, the selection methods’ accuracy was verified thanks to the long short-term memory network used as a classifier. The results proved that the new concept of long short-term memory classification ensures better tomographic reconstructions efficiency than imaging all measurement cases with single homogeneous methods.

1. Introduction

Industrial tank reactors serve an essential role in many processes that involve technology lines. A chemical reactor is a tank designed to carry out the reactions that occur within it. The purpose of industrial tank reactors is to ensure that the economic parameters of chemical processes are optimal [1]. It is possible due to the reactor’s optimal design and the skilful overlap of the three sub-processes occurring inside the reactor: mass, momentum, and heat transfer. Process control can thus be based on the dynamic selection of mixing intensity, temperature, pressure, substrate proportion, and others. The study described here applies to reactors that have interactions between solids and liquids and gas and liquids.

Monitoring the states of dynamic systems is done for two main reasons. The first is detecting approaching failures [2], which include damage to the technological infrastructure, excessively high deviations in crucial process parameters, or interruptions in the process’s continuity. Second, an effective monitoring system is intended to detect a problem early enough to take practical corrective steps.

The requirement to regulate the course of the industrial process is the second reason for utilizing industrial process monitoring [3]. It is critical for guaranteeing a high degree of quality. Effective monitoring methods must be employed to control multiphase processes, including chemicals that can change aggregate states dynamically. Given the aggressive circumstances under which the reactions occur inside the reactor, this is a demanding task. The problem with using invasive sensors is the inability to directly examine any part of the reactor’s interior, the accuracy of the measurements taken, the requirement to use multiple monitoring systems at the same time, and the high uncertainty in determining the dynamic state of the process based on incomplete data (indirect method). Electrical capacitance tomography (ECT) [4,5,6,7,8,9,10,11,12,13,14,15,16], electrical impedance tomography (EIT) [17,18,19,20,21,22,23,24], magnetoacoustic tomography [25], ultrasound and radio [26,27], X-ray tomography [28], optical tomography [29,30], and other non-invasive technologies are used to monitor industrial operations. Recently, an increasing number of research papers in the field of industrial system operation have included the use of various computational methods, such as intelligent predictive methods [31], fuzzy logic [32,33], machine learning [34], numerical modelling [35], deep learning [18,36], and binary programming [37].

The present non-invasive monitoring techniques utilized in industrial processes do not fully meet the current operational needs. The images obtained of the phenomena and processes under examination are frequently blurry, confusing, and challenging to interpret, and they are riddled with mistakes in terms of the number of artefacts (crystals or gas bubbles) found in the reactor as their sizes and locations. As a result, redundant systems are employed to gather exact information about the condition of the monitored process, increasing operational expenses dramatically. The challenges and flaws in monitoring chemical tank reactors stated above cause their need for modification. Implementing a better monitoring strategy will boost the reliability of the activities occurring inside the reactors while decreasing the running costs of industrial systems.

This study aims to present an improved method of monitoring and optimization of chemical processes in heterogeneous tank reactors, in which reactions occur between solid and liquid and gas and liquid. The applied method concerns electrical tomography, and the innovation is the original method of parallel use of many homogeneous machine learning methods and the long short-term memory (LSTM) classifier in selecting the optimal method for a given measurement case [38]. The advantage of the described algorithmic concept over other non-invasive methods is the ability to automatically adapt the method to a specific measurement case, increased resistance to interference arising during measurements, higher accuracy of reconstruction than with homogeneous methods, high imaging resolution, low cost of use, and high speed of operation. Furthermore, the use of the proposed concept enables automatic adjustment of the tomographic system to a specific measurement case. As a result, the system automatically responds to changes inside the tank reactor, selecting the method of converting the measurements into images depending on the measurement vector. The LSTM classifier performs this task. The other part of this study describes the original, intelligent hybrid system enabling effective monitoring of chemical reactions using process electrical tomography.

The Method Oriented Ensemble (MOE) concept presented in Figure 1 is a novelty here and, at the same time, constitutes the authors’ contribution. The vector of 96 measurements is the input to the LSTM classifier. The LSTM network generates one of five classes at the output, determining the optimal reconstruction method for a given measurement case. The values of all 2883 pixels of the image are generated by separately trained models of the selected homogeneous method. In addition, each pixel has its own specially trained machine learning model. In this way, the tomographic image is reconstructed according to the new MOE concept. Using the measurement vector as a sequence of input variables to the recurrent deep LSTM neural network is also new [39]. The research results showed the very high effectiveness of such an approach in terms of the classification/selection of the optimal homogeneous method within MOE.

The article is broken into four parts. The Introduction part offers a synopsis of issues linked to industrial tomography, a review of known methods of imaging the interior of reactors and pipes, a description of several types of tomography, and the authors’ contribution. The second part, Materials and Methods, discusses the research facility—a physical model of the reactor, the technique of gathering training data, the innovative method-oriented ensemble (MOE) idea, five homogeneous methods utilized in the MOE, and the LSTM classifier. The final part comprises test results derived using both real and simulation data. This part also contains a discussion of the results received. Finally, part four of the paper presents an overview and synthesis of the most relevant components of the research work carried out and the results and conclusions acquired. It also includes information about upcoming research endeavours.

2. Materials and Methods

2.1. Research Object

The subject of the research is the physical model of the tank reactor. The main element of the model is a plastic cylinder around which 16 electrodes are placed. The cylinder’s diameter is 200 mm. The container was filled with tap water. Empty plastic tubes with a diameter of 20 mm are placed inside the cylinder. The task of the electrical impedance tomography (EIT) system was to reconstruct the cross-section of the tank correctly. The reconstruction’s quality is determined by the visibility and clarity of inclusions (tubes) in terms of diameter, shape, number and position of inclusions to each other and with the tank wall. Figure 2a shows the test stand with an electrical impedance tomograph connected to the reservoir electrodes. The Netrix S.A. Research and Development Center made the prototype of the EIT measuring device (tomograph). Figure 2b,c shows a plastic cylinder used as a physical model of a tank reactor. Synthetic tubes filled with air are immersed in the cylinder.

2.2. Data Preparation

Based on the above physical models, a unique simulation algorithm was developed to generate learning cases used during the training of machine learning systems. Each training case was generated with the assumption of homogeneity of the distribution of electrical conductivity. For the obtained conductivity distribution, the measurement voltages are determined by the finite element method. The Eidors toolbox was used for this purpose [40]. In individual cases, the number of internal inclusions was selected randomly. It was assumed that we would get a maximum of five objects—each of a round shape as a result of the drawing. The radius of the inclusions and the electrical conductivity are such that they correspond to the actual tests performed by the EIT. In the next stage of calculations, the centre of each internal object is drawn.

Figure 3 shows one of the 35,000 generated cases used to train the predictive system. The cross-section of the tank contains one random inclusion. Inclusion is visible through the variety of colours on the 2883 finite element mesh.

The colours correspond to conventional units corrected for the electrical conductivity of the interior of the tested object. Ninety-six voltage measurements were assigned to the conductivity values (Figure 3b). These are not the voltage values, but the arbitrary units correlated with them. As the polarity of the electrodes changes during individual measurements, the voltage changes in the range (−0.3; +0.3). Each measurement’s value was boosted by the addition of Gaussian noise with a standard deviation of 4%. Finite elements (pixels) in the image have the values 1 (for background) or ${10}^{-5}$ (for plastic tubes).

2.3. The Concept of the Method Oriented Ensemble (MOE)

The proposed novel paradigm necessitates the training of several machine learning models for each pixel individually. Because the total number of pixels in the situation at hand is 2883, each of the five homogeneous approaches will require 2883 regression prediction models to be trained. Elastic Net (EN), Linear Regression with Least Squares learner (LR-LS), Linear Regression with Support Vector Machine learner (LR-SVM), Support Vector Machine (SVM), and Artificial Neural Network (ANN) are all used in the method-oriented ensemble (MOE) concept. The process of modelling the POE concept is presented in Figure 4. The presented flowchart is consistent with Algorithm 1 and Figure 1.

#

Algorithm 1. The pseudocode algorithm for training MOE

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.

m = 96  % number of measurements n = 2883 % number of finite elements in reconstruction mesh (pixels) Train n models $f_1\left(x_{1\dots m}\right)\to y_{1\dots n}$ with method # 1 (e.g., EN) Train n models $f_2\left(x_{1\dots m}\right)\to y_{1\dots n}$ with method # 2 (e.g., LR-LS) Train n models $f_3\left(x_{1\dots m}\right)\to y_{1\dots n}$ with method # 3 (e.g., LR-SVM) Train n models $f_4\left(x_{1\dots m}\right)\to y_{1\dots n}$ with method # 4 (e.g., SVM) Train n models $f_5\left(x_{1\dots m}\right)\to y_{1\dots n}$ with method # 5 (e.g., ANN) % Assigning the RMSE for each method and pixel for i = 1:5  % for 5 methods: EN, LR-LS, LR-SVM, SVM, ANN  for j = 1:n  % for n = 2883 pixels   calculate RMSE(i, j)  % assignment root mean square error for i-th method and j-th pixel  end  meanRMSE(i) = mean(RMSE(i,:)) % Calculate the mean RMSE for each of the 5 methods. end      % Assignment meanRMSE for i-th method and all 2883 pixels. Prepare the training set to train the LSTM classifier. Inputs—96 measurements. Output—5 categories/classes. Select the method with the lowest meanRMSE. Reconstruct all n pixels using the selected method.

Algorithm 1 presents pseudocode to train a method-oriented ensemble (MOE) model. Based on 96 data, a specially trained LSTM classifier determines which of the five homogenous approaches (EN, LR-LS, LR-SVM, SVM, or ANN) is best for a given measurement situation (vector 96 measurements). It is worth noting that training five predictive models for each of the 2883 pixels yield 14,415 models. To this value, add the LSTM technique classifier. High computational complexity characterizes the new MOE idea. However, compared to the enormous improvement in image reconstruction quality that can be attained, this is a minor nuisance.

2.4. Elastic Net (EN)

When reconstructing tomographic images of real objects with low conductivity, the electrode data are frequently noisy. It is due to electrode insulation imperfections, the impacts of fast-varying, low-intensity currents created by multiplexers, the effects of electromagnetic fields, and various other reasons. Industrial reactors are another example of technological devices with a high noise level as determined by tomographic data. Electrical signal interference is one of the primary impediments to developing tomographic methods for such objects [41].

The elastic net regularization technique made the input data more robust to noise and distortions [42]. We begin with a linear system described by the equation of state $Y=X\beta +\epsilon$, where $Y\in R^n$ is the matrix of output variables (reconstruction), $X\in R^{n\times \left(k+1\right)}$ is the matrix of input variables, the coefficient $\beta \in R^{k+1}$ denotes a vector with unknown parameters, and $\epsilon \in R^n$ reflects the sequence of the disturbance.

Elastic net is a compromise between L₁ and L₂ norms, or, to put it another way, between Robert Tibshirani’s LASSO (Least Absolute Shrinkage and Selection Operator) and ridge regression is known as Tikhonov regularization. The approach is also successful when there are numerous correlated predictors or when the number of discretized current elements is substantially more significant than the number of measurement points. Task (1) can be used to indicate the problem that determines the elastic net:

$$\underset{\left({\beta }_0,{\beta }^{\prime }\right)\in R^{i+1}}{\min }\frac{1}{2n}{\sum }_{i=1}^n{\left(y_i-{\beta }_0-x_i{\beta }^{\prime }\right)}^2+\lambda P_{\alpha }\left({\beta }^{\prime }\right),$$

(1)

where $\left(y_i-{\beta }_0-x_i{\beta }^{\prime }\right)$ are the linear model residuals, $x_i$ is the vector of measurements, $y_i$ is the vector of reference values, ${\beta }_0$ is the intercept equal to the mean of the response variable parameter, ${\beta }^{\prime }$ denotes unknown parameters, $\lambda$ is the parameter that specifies the penalty for regularization, and $P_{\alpha }$ is a net elastic penalty defined by (2).

$$P_{\alpha }\left({\beta }^{\prime }\right)=\left(1-\alpha \right)\frac{1}{2}{\Vert {\beta }^{\prime }\Vert }_{L_2}+\alpha {\Vert {\beta }^{\prime }\Vert }_{L_1},$$

(2)

If the linear problem (1) has a solution in which the regression line intersects the y axis, then the unit column vector is the first column of the X matrix in the linear equation $Y=X\beta +\epsilon$. The elastic net penalty $P_{\alpha }$ is a summary combination of the L₁ and L₂ norms of unknown parameters ${\beta }^{\prime }$, as shown by Equation (2). The trade-off between LASSO and ridge regression is represented by parameter $0\le \alpha \le 1$. It is pure ridge regression if the value is $\alpha =0$, but it is pure LASSO if the value is $\alpha =1$.

2.5. Linear Regression with Least-Squares Learner (LR-LS)

A linear regression-based form of the approach was employed in the investigation. The SVM learner has been substituted by the least-squares model [43]. The algorithm employed in the LR-LS method is quite similar to the methodology employed in the previously disclosed LR-SVM method. In this instance, LASSO regression with L1 regularization was also utilized, as was the LASSO regression model without regularization. The distinction between LR-SVM and LR-LS is the loss function used to calculate the likelihood ratio. For LR-LS, the loss function is mean square error, which can be calculated as $MSE=\ell \left[y,f\left(x\right)\right]=\frac{1}{2}{\left[y-f\left(x\right)\right]}^2$ with the range of answers $y\in \left(-\infty ,\infty \right)$. In LR-LS, the deviation b equals the y-weighted median of all training processes multiplied by the number of training processes.

2.6. Linear Regression with Support Vector Machine Learner (LR-SVM)

The linear regression (LR) and support vector machines (SVM) algorithms are used in the LR-SVM algorithm [44]. The algorithm has been tuned to work best with the multidimensional vectors of data provided as input. This strategy, known as the L₁ LASSO regularization technique, employs a regression model that incorporates the “absolute value of magnitude” into the loss function as a penalty component. In this study, the “learner” was a linear regression model based on the SVM approach, which was utilized [45]. $f\left(x\right)=x\beta +b$ is the loss function for a linear regression model type, where β denotes a vector of pp coefficients, x denotes an observation of p predictor variables, and b denotes a scalar bias. The mean square error (MSE) is determined as a loss function in the implemented algorithm, and it takes the form of the formula $\ell \left[y,f\left(x\right)\right]=\max \left[0,\left|y-f\left(x\right)\right|-\epsilon \right]$ where $y\in \left(-\infty ,\infty \right)$ is a reconstruction of the response value. The LASSO cost function is represented by the Equation (3)

$$\underset{{\beta }_0,\beta }{\min }\left(\frac{{\sum }_{i=1}^n{\left(y_i-b-x_i^T\beta \right)}^2}{2n}+\lambda {\sum }_{j=1}^p\left|{\beta }_j\right|\right),$$

(3)

where the number of observations is given by n. The transposed vector of length p at observation i is given by $x_i^T$. The reconstruction of a pixel at the observation vector gives $y_i$. The regularization parameter is given by $\lambda$, which must be non-negative. In this study, the value of Lambda is $\lambda =1/n$. It is worth noting that the parameters b and $\beta$ are in that order a scalar bias and a vector of length p, respectively. The number of non-zero $\beta$ parameters reduces as the value grows. Regularized support vector machines (SVM) and least-squares regression methods are both included in the LR-SVM. The model minimizes the objective function by employing stochastic gradient descent (SGD), reducing the time required for computation. A ridge penalty is applied to support vector machines in the outlined method, which is then optimized using a dual SGD for SVM. The formula represents the criterion for terminating the iteration process $\Vert \frac{B_t-B_{t-1}}{B_t}\Vert <\varkappa$, where $\varkappa$ is the relative tolerance on linear coefficients ${\beta }_t^{\prime }$, and bias term $b_t$, and $B_t=\left[{\beta }_t^{\prime },b_t\right]$.

2.7. Support Vector Machine (SVM)

Support Vector Machines (SVM) are based on the premise that a decision space may be partitioned by erecting borders that separate objects belonging to various classes. Regression and classification problems may both be solved using SVM, a standard machine learning method. Vladimir Vapnik and his colleagues first proposed this idea in 1992. As a result, there are four types of SVM: classification type 1 (C-SVM), classification type 2 ($\epsilon$-SVM), regression type 1 ($\epsilon$-SVM), regression type 2 ($\nu$-SVM). Support vector models fall into one of these four categories ($\nu$-SVM). We use SVM regression analysis to find the functional dependence between a dependent variable y and its independent variable x. $y=f\left(x\right)+noise$ is the formula used in regression analysis since it assumes the relationship is of the deterministic type $f\left(x\right)$, with some random noise added on top of it. The main goal is to determine the function f form that best provides the dependent variable’s value in new scenarios that the SVM model has not previously “seen”. The learning test trains the SVM model system by putting it through its paces with a set of cases. According to the proposed theory, each SVM subsystem is responsible for generating a single pixel value. The total number of trained SVM models in an EIT system is equal to the output image resolution, which is $\left(96\to \mathrm{SVM}\to 1\right)\times 2883$. As a result, the technique utilized in this study implements the regression type 2 problem (also known as the $\nu$-SVM). The Equation (4) ’s deviation function is minimized [44]:

$$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.w^{T}w-C\left(\nu \epsilon +\frac{1}{N}{\sum }_{i=1}^N\left({\xi }_i+{\xi }_i^{*}\right)\right),$$

(4)

under the following conditions (5):

$$\{\begin{array}{c}\left(w^T\mathsf{\Phi }\left(x_i\right)+b\right)-y_i\le \epsilon +{\xi }_i \\ y_i-\left(w^T\mathsf{\Phi }\left(x_i\right)+b_i\right)\le \epsilon +{\xi }_i^{*}\\ {\xi }_i,{\xi }_i^{*}\ge 0, i=1,\dots ,N, \epsilon \ge 0\end{array},$$

(5)

where $\epsilon ,\nu$ are the penalty parameters, C is the capacity constant, $w$ is a vector of coefficients, b is a constant, and ${\xi }_i, {\xi }_i^{*}$ are the overlapping case parameters. N learning examples are represented by index i independent variables are represented by $x_i$, and regression patterns are represented by $y_i$. The input data are converted to a new feature space via the kernel function. It is important to note that C significantly impacts the deviation, and its value must be carefully chosen to avoid overfitting the model. Each of the SVM subsystems was trained using 4000 training cases.

2.8. Artificial Neural Network (ANN)

The researchers employed artificial neural networks in the form of a multilayer perceptron for their investigation. The collection of 35,000 examples has been separated into three subsets: training, validation, and testing, with an aspect ratio of 70:15:15. The training subset contains cases that have been validated and assessed. As a result, 2883 models were trained, which is the same number of models as the resolution of the spatial tomographic image. Additionally, to optimize the models, a backward propagation of errors approach using conjugate gradients was utilized in conjunction with conjugate gradients. The structure of a single ANN dedicated to each of the 2883 pixels is 96-10-1. Ninety-six measurements are input to the network, ten neurons are in the hidden layer, and one regression neuron is in the output. The transfer function of the hidden layer is a hyperbolic tangent $\tanh (x)=\frac{e^{2x}-1}{e^{2x}+1}$. The output layer makes use of a linear activation technique.

2.9. The Long Short-Term Memory (LSTM) Network for Classification

The classification of measurement cases to homogenous methods within MOE was accomplished using a deep LSTM network. This method differs from previous methods because it can learn long-term correlations between time steps in a time series or provided sequences. Furthermore, the measurement vector is used as a sequence of input variables to the recurrent deep LSTM neural network, a novel technique. The study’s findings demonstrated that such a strategy is quite effective in selecting the ideal homogenous method within MOE, as demonstrated by the study results.

Illustration of a time series X flow with C features (channels) of length S through an LSTM layer, as depicted in Figure 5. The output (also known as the hidden state) and the cell state at time step t are represented by the symbols h_t and c_t, respectively, in the diagram [46]. For example, the first LSTM block considers both the network’s starting state and the first time step of the sequence to compute both the first output and the updated cell state. To compute the output and the updated cell state c_t at time step t, the block uses the current state of the network (c_t−1, h_t−1) and the next time step in the series (time step t).

The hidden state (also known as the output state) and the cell state are the two states that make up the state of the layer. The output of the LSTM layer for the time step t is stored in the hidden state at time step t. In each time step, the cell state contains information that has been gained from the preceding steps in time. Thus, each time step, the layer either adds information to the cell state or removes information from the cell state, depending on the situation. The layer manages these changes through the use of gates. The cell state and concealed state of the layer are controlled by the following four components: input gate (i) that controls the level of cell state update, forget gate (f) that controls the level of cell state reset (forget), cell candidate (g) that adds information to cell state, and output gate (o) that controls the level of cell state added to hidden state.

The flow of data at the time step t is depicted in Figure 6. The diagram illustrates how the gates forget, update, and output the cell and hidden states and how they interact with one another. The LSTM network with seven layers was used.

The gates provide additional information on the cell’s state. Equation (6) can be used to characterize the weights W, the recurrent weights R, and the biases b.

$$\mathit{W}=\left[\begin{array}{c}{W}_i\\ \begin{array}{c}{W}_f\\ W_g\\ W_o\end{array}\end{array}\right], \mathit{R}=\left[\begin{array}{c}{R}_i\\ \begin{array}{c}{R}_f\\ R_g\\ R_o\end{array}\end{array}\right], \mathit{b}=\left[\begin{array}{c}{b}_i\\ \begin{array}{c}{b}_f\\ b_g\\ b_o\end{array}\end{array}\right] .$$

(6)

The symbols i, f, g, and o signify input gate, forget gate, cell candidate, and output gate, respectively. The state of a cell at a particular time step t is denoted as ${\mathit{c}}_t=f_t\odot {\mathit{c}}_{t-1}+i_t\odot g_t$, where ⊙ represents the Hadamard product, or in other words, vector element-wise multiplication. At time step t, the hidden state is defined as ${\mathit{h}}_t=o_t\odot {\sigma }_c\left({\mathit{c}}_t\right)$, where ${\sigma }_c$ is the state activation function. The Equation (7) defines the LSTM layer’s components at time step t,

$$\{\begin{array}{c}{i}_t={\sigma }_g\left(W_i{\mathit{x}}_t+R_i{\mathit{h}}_{t-1}+b_i\right) \\ \begin{array}{c}{f}_t={\sigma }_g\left(W_f{\mathit{x}}_t+R_f{\mathit{h}}_{t-1}+b_f\right)\\ g_t={\sigma }_c\left(W_g{\mathit{x}}_t+R_g{\mathit{h}}_{t-1}+b_g\right)\\ o_t={\sigma }_g\left(W_o{\mathit{x}}_t+R_o{\mathit{h}}_{t-1}+b_o\right)\end{array}\end{array} ,$$

(7)

where ${\sigma }_g$ denotes the gate activation function. The sigmoidal activation function was employed in both biLSTM layers (see

Table 1. Layers of LSTM network for classification with activations, learnable and states.

#	Layer Description	Activations	Learnable Parameters (Weights and Biases)	Total Learnables	States
1	Sequence input with 96 dimensions	96	-	0	-
2	BiLSTM with 128 hidden units	256	Input weights: 1024 × 96 Recurrent weights: 1024 × 128 Bias: 1024 × 1	230,400	Hidden state 256 × 1 Cell state 256 × 1
3	Batch normalization	256	Offset: 256 × 1 Scale: 256 × 1	512	-
4	BiLSTM with 128 hidden units	256	Input weights: 1024 × 256 Recurrent weights: 1024 × 128 Bias: 1024 × 1	364,240	Hidden state 256 × 1 Cell state 256 × 1
5	Fully connected layer	5	Weights: 5 × 256 Bias: 5 × 1	1285	-
6	Softmax	5	-	0	-
7	Classification output (cross entropy)	5	-	0	-

). The Equation $\sigma \left(x\right)={\left(1+e^{-x}\right)}^{-1}$ can be used to express this type of function.

In the case of neural networks, there are no rigid criteria for picking network parameters (e.g., number of layers, number of hidden units in LSTM layers, normalization requirements, initial weights, and biases functions) for specific types of issues. Accordingly, the network and training parameters are selected empirically. It was likewise the case in this circumstance. Table 1 presents the neural network parameters for the classification of homogeneous methods, having two double LSTM layers, 128 hidden units each. The performed experiments demonstrated that a lesser number of hidden units causes a worsening of network quality while increasing the number of units and adding subsequent layers extends the learning process without creating an increase in the quality of the LSTM network.

The first layer of the LSTM model is sequence input. The sequence input layer tries to enter chronological data into the network. The next is the bidirectional layer BiLSTM. The bidirectional LSTM layer learns long-term correlations between signal time steps or sequence data in both ways (forward with feedback). These interactions are significant when there is a need for the network to learn from full-time series at each time step. The second is the batch normalization layer. The batch normalization operation normalizes the input data across all observations for each channel independently. The batch normalization employed between convolution and nonlinear operations such as BiLSTM speeds up the training of the convolutional neural network and minimizes the sensitivity to network initialization. The next layer is again the BiLSTM bi-directional layer. The fifth layer of the LSTM is fully connected. This layer multiplies the numerical input values by the weight matrix and also adds a vector of biases. Another layer is the softmax. A softmax layer applies a softmax function to the input. For classification problems, a softmax layer and a classification layer commonly follow the final fully connected layer. The last is a classification layer. This layer computes the cross-entropy loss for classification and weighted classification tasks with mutually exclusive classes. The layer infers the number of classes from the output size of the previous layer.

One or more fully linked layers are introduced following the convolution and downsampling layers in deep networks. When the input to a fully connected layer is a sequence, as with LSTM, the fully connected layer performs each stage independently. If the output of the layer preceding the fully connected layer is an array A₁ of dimension X by Y by Z, then the output of the fully connected layer is an array A₂ of size X’ by Y by Z. The proper input to A₂ at time step t is ${\mathrm{WA}}_t+b$, where A t is the time step t of A and b is the bias. Glorot initializer was used to generate the weights for this layer in this research [47]. The softmax is the penultimate layer. It is a common type of layer in deep categorization neural networks. An ultimately linked layer is always preceding the softmax layer. The formula $y_r\left(x\right)=e^{a_r\left(x\right)}/{\sum }_{j=1}^k{e}^{a_j\left(x\right)}$ denotes the softmax activation function, with $0\le y_r\le 1$, and ${\sum }_{j=1}^k{y}_j=1$.

For classification problems with mutually exclusive classes, the final layer computes the cross-entropy loss. To ensure a sufficient number of training sets, aggregation of measurement data with comparable properties was performed. The LSTM network was trained using the adaptive moment estimation (ADAM) algorithm [48]. The following parameters apply to the BiLSTM layer: Tanh function for state activation, sigmoid function for gate activation, mini-batch size = 100, starting learning rate = 0.01, sequence length = 96 (longest), gradient threshold = 1. The parameters listed above were determined empirically. With a probability range from 0.1 to 0.5, various model variants featuring the dropout layer were examined. The experiments revealed that increasing dropout layers did not affect the network’s generalization in the scenario analyzed. As a result of the preceding, we opted against incorporating this layer into the proposed prediction model. The training status of the LSTM network using a raw input is depicted in Figure 7.

Cross-entropy and accuracy were used to assess the LSTM model’s quality. Accuracy is defined as the proportion of correctly identified observations across all instances (8).

$$Accuracy=\frac{N_c}{N}·100\% ,$$

(8)

where N_c is the number of correctly rebuilt pixels, and N is the total number of pixels [49]. Equation (9) defines the cross-entropy loss between network predictions and target values

$$Loss=-\sum _{i=1}^M{T}_{i}log\left(X_i\right)/N ,$$

(9)

where N is the number of observations, M is the number of responses, T_i is the number of patterns, and X_i is the number of network outputs. The training-progress graphic demonstrates the correctness of the training. Indeed, it indicates the accuracy of each minibatch’s classification. This number increases to 100% for optimal training development. At the finish of the training procedure, the classifier’s accuracy oscillates between 99% and 100%. It took approximately 12 min to train. The computation was carried out on a personal computer configured as follows: 2.80 GHz Intel^® CoreTM i5-8400 CPU, 16 GB RAM, NVIDIA GeForce RTX 2070 GPU. Parallel computing with GPU was used [50,51].

The LSTM learning process using a measurements vector as a signal in conjunction with the Loss indicator is illustrated in Figure 8. The graph depicts the training loss, which is the cross-entropy loss for each mini-batch. When training is carried out precisely, the loss should be zero. The shape of this plot corroborates all of the information in Figure 7.

Both Figure 7 and Figure 8 show a very high quality of the LSTM classifier. Curve shapes resembling a logarithm and a hyperbola, without fluctuations, are proof of the correct course of the learning process.

Figure 9 shows the confusion matrix of the LSTM classifier on the set of 1000 test cases. As can be seen, significant problems were caused by the correct classification of the EN method because 13.6% of the answers were incorrectly categorized as the ANN method. Thus, cumulative accuracy for the entire testing set was 98.2%.

3. Results and Discussion

Experiments were carried out to reconstruct EIT measurements based on real and simulation data to verify the effectiveness of the new MOE method. The first experiment was to visualize the inside of a physical model for three different scenarios. In the first case, two tubes were immersed in the water. In the second case, the position of the tubes was changed, and their number increased to three. Finally, the fourth case included four tubes arranged symmetrically at equal distances and the walls of the water container.

Reconstructions based on real measurements do not allow for objective comparisons due to the lack of reference images. Therefore, a series of experiments based on simulation data was also carried out to allow for an objective, index-based assessment that allows the comparison of individual reconstructions. In this way, the quality of the proposed MOE method was verified using error, deviation and correlation indicators.

3.1. Visualizations of Real Measurements

Figure 10 shows a schematic diagram of the physical model of the tested tank. Figure 10a shows a dimensioned side view of the cylinder with the tube submerged. Figure 10b–d show top views of the test tank with three cases of differently distributed inclusions. The considered measurement cases contain 2, 3 and 4 tubes, respectively, immersed in the cylinder. The diameter of the tested cross-section of the tank model is 32.5 cm. When submerged in water, the diameter of the plastic tube is 28 mm. 16 electrodes were evenly spaced around the cylinder at the height of 14 cm from the bottom.

Figure 11 shows the reconstructed three measurement cases presented earlier in Figure 10. The LSTM classifier indicated the SVM method in all tested cases. As is known, there is always a certain level of data noise in the case of real measurements. It makes this type of reconstruction more difficult to reconstruct than imaging simulation measurements. In Figure 11a, an artefact appears in the central part of the tank, but it is less pronounced than the images of the two tubes. The other reconstructions Figure 11b,c are successful.

3.2. Comparison of the Reconstructions Based on Simulation Data

Four widely used metrics were used to evaluate the quality of tomographic reconstructions objectively: Root Mean Square Error (RMSE), Relative Image Error (RIE), Percentage Error (PE), Mean Absolute Percentage Error (MAPE), and Image Correlation Coefficient (ICC). The root mean square error is calculated using the Equation (10)

$$\mathrm{RMSE}=\sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{n} ,}$$

(10)

where n is the number of finite elements or the picture resolution, $y_i$ denotes the i-th pixel’s pattern conductivity, and ${\widehat{y}}_i$ denotes the reconstruction conductivity. RIE is calculated as (11)

$$\mathrm{RIE}=\frac{\Vert y^{\prime }-y\Vert }{\Vert y\Vert } ,$$

(11)

where $y$ is the ground-truth (reference) conductivity distribution and $y^{\prime }$ is the reconstructed conductivity distribution. MAPE is calculated using the following Eqaution (12)

$$\mathrm{MAPE}=\frac{1}{n}\sum _{i=1}^n\left|\frac{y-\widehat{y}}{y}\right|.$$

(12)

The ICC metric describes the function (13)

$$\mathrm{ICC}=\frac{{\sum }_{i=1}^n\left(y_i-\overline{y}\right)\left({\widehat{y}}_i-\overline{\widehat{y}}\right)}{\sqrt{{\sum }_{i=1}^n{\left(y_i-\overline{y}\right)}^2{\sum }_{i=1}^n{\left({\widehat{y}}_i-\overline{\widehat{y}}\right)}^2}} ,$$

(13)

where $\overline{y}$ denotes the mean reference ground-truth conductivity distribution and $\overline{\widehat{y}}$ denotes the mean EIT reconstruction conductivity distribution.

The lower the root mean square error, the smaller the RIE, and the greater the MAPE value, the higher the tomographic image quality. Thus, ICC = 1 indicates perfect reconstruction, whereas ICC = 0 indicates the worst-case scenario. Reconstructions based on simulation measurements are shown in Figure 11. The first line of Figure 11 illustrates the reference images of the model tested. Reconstructions using homogeneous EN, LS-SVM, LR-LS, SVM and ANN methods are shown in the following lines.

Table 2. Reconstruction quality indicators for homogenous methods—testing set. The best values of indicators are marked in blue.

Case Number	Indicator	Methods of Reconstruction					Best Homogenous Method (MOE Concept)
		EN	LR-LS	LR-SVM	SVM	ANN
1	RMSE	0.289	0.133	0.145	0.135	0.068	ANN
	RIE	0.293	0.135	0.147	0.137	0.069	ANN
	MAPE	1610.2	1880.7	2142.1	1988.4	126.6	ANN
	ICC	0.875	0.536	0.392	0.530	0.914	ANN
2	RMSE	0.299	0.1519	0.123	0.117	0.132	SVM
	RIE	0.306	0.155	0.126	0.120	0.135	SVM
	MAPE	2845.92	2659.4	2191.2	1990.1	1087.3	ANN
	ICC	0.905	0.709	0.843	0.853	0.794	EN
3	RMSE	0.304	0.157	0.077	0.084	0.140	LR-SVM
	RIE	0.318	0.164	0.081	0.087	0.147	LR-SVM
	MAPE	4467.8	2935.4	846.1	835.3	1239.5	SVM
	ICC	0.986	0.826	0.960	0.953	0.870	EN
4	RMSE	0.317	0.135	0.076	0.085	0.152	LR-SVM
	RIE	0.336	0.143	0.081	0.090	0.161	LR-SVM
	MAPE	5952.1	2536.1	916.7	988.2	1584.6	LR-SVM
	ICC	0.950	0.900	0.969	0.962	0.876	LR-SVM

corresponds with Figure 12. Table 2 presents quantitative indicators that enable the comparison of individual homogeneous methods included in the new MOE concept. The best values of indicators are marked in blue.

Visual (subjective) observation of the images in Figure 12 concludes that the best reconstructions were obtained by the EN and ANN methods for the first measurement case with a single inclusion. According to the indicators included in Table 2, the best method is ANN. The second case of measurement is not so clear-cut. In Table 2, according to the RMSE and RIE indicators, the best is the SVM reconstruction, according to MAPE-ANN and ICC-EN. Notably, any indicators did not select the LR-LS and LR-SVM methods, which visually look worse than the others. The third case, with three inclusions, according to RMSE and RIE, is best reconstructed by the LR-SVM method, according to MAPE-SVM, and according to ICC-EN. Visual observation confirms the rightness of this choice. In the last, fourth measurement case, all indicators in Table 2 indicated the superiority of the LR-SVM method. It can be seen that the SVM method fares very similarly, although it is worse. Visual observation of Figure 12 can confirm this. Since the purpose of tomography is not a precise measurement of electrical conductivity inside the tested objects but an accurate, precise, and quick visualization of inclusions, the MAPE index, as an index reflecting the absolute values of the patterns and reconstructions, should be treated as optional/supplementary. Large values of the MAPE index result from 2883 pixels included in the tomographic image and the fact that the reference background values are “1” and the inclusions are close to zero (precisely 10⁻⁵).

Figure 13 shows the 1000-item testing set broken down by quantity for each homogeneous method selected by the LSTM classifier. As can be seen, in the test set under consideration, the LR-SVM method was indicated the most times, while the LR-LS method was chosen the least frequently. It should be noted that the method quality indicator for the LSTM classifier was the RMSE. Impedance tomography is a potential field of science for identifying inclusions inside industrial reactors, containers, and pipes [52,53]. The presented research findings substantiate the preceding assertion due to the complexity of the model transferring the measurement data into the output image and the requirement to solve the incorrectly stated inverse problem, developing an effective and universal technique. The novel MOE approach described in this article uses considerable computational resources to maximize the resulting image’s quality. With the ongoing development of new mathematical homogenous methods, hybrid (ensemble) methods should not be overlooked. In this regard, the MOE technique is more advanced. MOE makes every effort from the start to fully use the data available in the measurement data collection. Because distinct training models are used for each pixel in the output image, each pixel’s value, and hence the colour, is calculated using the entire vector of 96 measurements.

Perfect results were obtained due to the unconventional deployment of a recurrent LSTM network. The most often utilized use of LSTM is to process time series and sequences. Therefore, we considered the measurement vector a collection of structured data (sequence) with a single step in our research. This technique proved correct, as indicated by the near-perfect classification of homogeneous methods using the MOE method.

Differences in image reconstruction across separate models are due to different methods, but not exclusively. For instance, when a homogeneous approach (e.g., elastic net or SVM) is used with the same input vector for all pixels, reconstructing an image with 2883 pixels requires adjusting the parameter values and hyperparameters of models trained for specific finite elements (pixels).

The objective of machine learning methods is to extract pure information from input data. It is information that can be used to create the output image successfully. There are numerous methods for obtaining the information’s essence. Preprocessing can be used to clean, standardize, encode and decode, increase or shrink, and extract features from data. It turns out that to get the highest possible reconstruction quality, the number of iterations and variation of the model can also be multiplied. This approach is superior to typical pre-processing procedures in that it is applicable to practically any problem. As a result, the tomographic images of the inclusion distribution inside the reactors are more precise, legible, and have a more excellent resolution. It is one of the benefits of the novel MOE concept described in this paper.

4. Conclusions

This publication provides a new algorithmic MOE method that optimizes EIT reconstructions in industrial applications simultaneously using many machine learning methods. Moreover, the provided method permits the visualization of the distribution of inclusions inside the tested reactor.

The fundamental aspect that distinguishes MOE from other methods is the high chance of increasing the quality of reconstructive images without using new, previously unknown homogenous methods. MOE is the concept of making greater use of computing resources and offering alternative methods. To apply the novel MOE methodology, it is enough to train a few models that use existing prediction methods, such as SVM, linear regression, neural networks, decision trees, elastic net, LASSO, k-nearest neighbours, etc. and then use them as stated in this article. Regardless of how many methods are employed in the MOE, it is almost assured that the results achieved will be better than if each homogenous approach was utilized alone. An exciting discovery is the unusual use of the LSTM network as a method classifier. This was made possible by treating the measurement vector as a succession of measurements.

Future studies will be carried out to improve the MOE approach with increased usage of deep learning and recursive neural networks.